So they’ve identified a flaw in your game. But you aren’t sure about how to fix it. What do you do? Well you can make an educated guess. But the best developers don’t guess. They know, they use A/B testing, which is testing different versions of a game at the same time. And seeing which version gets the best response from players.
It’s called AB, because the simplest test compared two versions. An A and B version. The A version is given to one-half of the games users. And the B is given to the other half, and we see how each group of users respond. These tests can help you repair any game content that you’re unsure about. From the colour scheme to game content. And you can test out two versions or as many as you want. Let’s walk through an example of creating effective AB testing scenarios. Let’s say we’ve made a platform a game, where we jump around and shoot enemies to advance levels. Our analytics have shown us that only 20% of players make it past Level 3.
We think that the level might be too hard, so let’s change it. Let’s test out different levels of difficulty and see how users respond. To see how players respond to different scenarios, we can use A/B testing, which is a method of testing that compares the performance of two or more game versions A and B and so on. The most effective A/B testing scenarios are designed to test out one main variable, X, Y, count, colour, etc. And they monitor one or two results like level completion or profitability, things like that. Our game versions are distributed amongst our players. And so there are usually only 2-4 versions tested at one time.
So for our game we want to change level 3. Let’s create three different test cases where we change the number of enemies and see how many more players will complete the level. You also want to test your original version and comparison to the new ones. So we have an A, original version. A version B with 10% fewer enemies.
And a version C with 20% fewer enemies. The next step is to distribute these versions to our players so that about a third of our players play each version. This is usually done by a percentage. So, we have 33%, 33% and 34%, which adds up to a total of 100%, distributed to versions A, B, and C. Then we get our results and see how many players complete the level. We see that only 20% of our group A players completed the level. 80% of group B completed level 3, and 70% completed level 3 for group C. So both B and C seem like a lot better options than A. And maybe the reason that C isn’t as high as B is that it’s too easy, so people get bored playing.
Either way, A/B testing let’s us know for sure that version B is our best option.