Implement A/B tests for your Unity game with Firebase Remote Config

1. Introduction

In this codelab, you'll learn how to create a Remote Config experiment using A/B Testing for the sample game, MechaHamster: Level Up with Firebase edition , which you modified in Instrument your game with Firebase Remote Config .

A/B Testing with Remote Config lets you test changes to your app's UI, features, or engagement campaigns on a targeted audience before you roll them out to a wider audience. You can also use experiment results to:

Determine which parameter value variants improve key metrics like revenue and retention.
Discover which subgroups of users prefer which variants.
Gather and store experiment data to perform additional analysis on the effects of setting different parameter values.

This is the essence of A/B Testing: it enables you to take code instrumented for Remote Config and launch experiments that control the values clients receive, based on Remote Config conditions (including Google Analytics user properties), percentage rollouts, Analytics conversion events, and some combination of these.

First, you will implement a condition that gates who will be included in the experiment by setting user properties based on user actions. Then, you will create an A/B Testing experiment that uses Google Analytics user properties to determine which clients are included in the experiment. And, finally, you'll use that data to understand more about your audience.

What you'll learn

How to set up A/B Testing using instrumented Remote Config values
How to use Google Analytics user properties as part of conditions for admission into A/B tests

Prerequisites

Completed Instrument your game with Firebase Remote Config

What you'll need

Unity 2019.1.0f1 or higher with iOS and/or Android build support

2. Enabling the Debug Menu

There is a Debug Menu hidden in the project, and the button to access this menu exists in the game but is not currently enabled. You must enable the button to access it from the MainMenu prefab.

In the Unity editor, select the Projecttab, then under Assets, expand Hamster> Prefabs> Menusand click MainMenu.

$The Unity editor Project tab showing\nAssets. Hamster, Prefabs, Menus$

In the prefab hierarchy, find the disabled sub-object named DebugMenuButtonand click it to open it in the Inspectortab.

$The Unity editor shows the Main Menu,\nwith DebugMenu disabled$

Within the Inspectortab, check the box at the upper left corner next to the text field containing DebugMenuButtonto enable it.

$The Inspector tab for DebugMenuButton\nwith checkbox$

Save the prefab.

If you run the game in either the editor or on your device, the menu should now be accessible.

3. Enable the Remote Config submenu

From the Projecttab in Unity Editor, expand Assets> Hamster> Prefabs> Menusand double-click the DebugMenuobject to open it in the editor Hierarchytab.

$DebugMenu item nested under Assets,\nHamster, Prefabs, Menus$
In the Hierarchytab, expand the hierarchy and click the sub-object under DebugMenu > Panel, which is labeled Remote Config Actions.

$Remote Config Actions nested under\nCanvas, DebugMenu$

In the Unity Inspectortab, enable Remote Config Actionsby checking the box to the left of the text field containing the object's name.

$Unity editor with Remote Config\nActions enabled under DebugMenu, Panel$

This has two GameObject children named Set Bored Of Subtitle and Set Enjoys Subtitle, which are both configured to call existing but unimplemented methods in DebugMenu.cs .

4. Reset the subtitle override to in-app default

In the previous codelab , you overrode a parameter's default value as JSON and used conditions to serve different variants. As part of this codelab, you will delete the condition you created and reintroduce the in-app default, and you will only override it with the result of A/B Testing.

To re-enable the in-app default:

Open the Remote Config page in the Firebase console and click the pencil icon next to the subtitle_override parameter to open the Edit parameterside panel.
Click the Xicon next to the condition to delete it.
Next to the remaining default value, enable the Use in-app defaulttoggle.

$Deleting a condition from the Remote\nConfig parameter editor$

Click Saveto save your changes, then click Publish changesto publish your changes. $Publish\nchanges option on the Remote Config page$

5. Set User Properties in Debug functions

You will now write function bodies for some pre-configured but unimplemented Google Analytics functions in DebugMenu.cs (which can be found in Assets > Hamster > Scripts > States).

These functions set user properties , which are ways to describe segments of your user base and are used to record how the user feels about the game's subtitle.

Implement SetUserBoredOfSubtitle and SetUserEnjoysSubtitle by finding the existing versions of them in DebugMenu.cs and overwriting them as follows:

  void 
  
 SetUserBoredOfSubtitle 
 () 
 { 
  
 Firebase 
 . 
 Analytics 
 . 
 FirebaseAnalytics 
 . 
 SetUserProperty 
 ( 
 "subtitle_sentiment" 
 , 
  
 "bored" 
 ); 
 }

  void 
  
 SetUserEnjoysSubtitle 
 () 
 { 
  
 Firebase 
 . 
 Analytics 
 . 
 FirebaseAnalytics 
 . 
 SetUserProperty 
 ( 
 "subtitle_sentiment" 
 , 
  
 "enjoys" 
 ); 
 }

If your app is correctly configured with Google Analytics, you can call one of these functions to make the property available to Remote Config's conditions. To call SetUserBoredOfSubtitle from a mobile device, start the game and press the Debug Menubutton in the main menu, then press Set Bored of Subtitle.

6. Create a Custom Dimension

Next, you will set up various subtitle variants for the subtitle_override to see which subtitle works best. But, in your A/B test, you will only serve these variants to users whose feelings about the current subtitle (as recorded in subtitle_sentiment ) include the word "bored."

You'll use a custom dimension to create and track custom parameters on Analytics events. See Custom dimensions and metrics for more information.

To create a new custom dimension:

Open the Firebase console , expand the Analyticsmenu, and select Custom definitions.
From the Custom definitions page, click Create custom dimensions.
In the New custom dimensionwindow, set the Dimension name to "Subtitle Sentiment" and, from the Scopedrop-down, select User.
For the User propertyfield, select subtitle_sentiment.

7. Set up the A/B Testing experiment

Next, create an A/B Testing experiment to set different values for subtitle_override to be tested against each other in order to optimize two to three day user retention.

First, select Editon the subtitle_override parameter on the Firebase console Remote Config page :

$The subtitle_override parameter on\nthe Remote Config page, showing the Edit option.$

From within the Edit parameterdialog that appears, click on Add new.

$The Add new experiment option on the\nRemote Config Edit parameter page.$

From the list that appears, select Experiment.

$Remote Config parameter page: Add new\nexperiment$

Enter a name and description for your experiment.

$Experiment name and description\nsection$

Next, choose targeting conditions. First, select your App from the drop-down.

$The Targeting step of an A/B Testing\nconfiguration, with an app selected$

Next, click Andto add a new condition, then select User Propertyand choose subtitle_sentiment . If it does not appear, manually enter it.
Since you only want to set a subtitle for those whose current subtitle sentiment includes "bored," choose containsand type in bored .
Optionally, choose which percentage of the audience that matches the above criteria to expose to the test. Select 100%to avoid randomness you can't easily control.

$A/B Testing Targeting section with\nsubtitle_sentiment selected$

Next, select a goal that the test will try to maximize for. Pick Retention (2-3 days).

A/B Testing Goals section

Next, set up the parameters of the experiment and create different subtitle variants . These variants are the different values A/B Testing will serve to users whose subtitle_sentiment contains ‘bored,' and A/B Testing will determine which variant is best for maximizing retention.

Enter the following parameter value for Variant A:

  { 
 "text" 
 : 
 "A: A game about a mechanical hamster" 
 , 
 "fontSize" 
 : 
 8 
 , 
 "textColor" 
 :{ 
 "r" 
 : 
 0.0 
 , 
 "g" 
 : 
 255.0 
 , 
 "b" 
 : 
 0.0 
 , 
 "a" 
 : 
 255.0 
 }}

Enter the following parameter value for Variant B:

  { 
 "text" 
 : 
 "B: A game about a mechanical hamster" 
 , 
 "fontSize" 
 : 
 14 
 , 
 "textColor" 
 :{ 
 "r" 
 : 
 255.0 
 , 
 "g" 
 : 
 0.0 
 , 
 "b" 
 : 
 0.0 
 , 
 "a" 
 : 
 255.0 
 }}

Set the variant ratios with whole numbers as follows:
- Baseline: 1
- Variant A: 100
- Variant B: 100
This will result in total weights of 0.5% for the Baseline, 49.8% for Variant A, and 49.8% for Variant B.These variants specify that one out of 201 times, A/B Testing serves the default to those who are bored of the subtitles, but 200/201 times, it will serve one of two new values and override the title screen.
Save and submit by clicking Start experiment, then click Starton the confirmation popup. $Click Start to start the\nexperiment$

8. Run the Set User Property method and refresh

You now have set various properties about the user, which then can be used to direct the logic or presentation of your game.

If you have not previously run the SetUserProperty method or have set it to enjoys , you should still see the default subtitle when opening the game.

If you have set it to bored (before re-fetching), you should see one of the new values in a roughly 50/50 ratio.

Once a device has entered into an A/B Testing experiment, the values they receive from that experiment will not change and are persistent per installation . As a consequence, to receive one of the other experimental values, you must create a new install either by reinstalling the game to the same device/simulator or installing the game to a new device/simulator.

In a real A/B test launched across your user base, you should give the baseline a similar weight to the other variants. But in this case, you assign very skewed probabilities to validate that the experiment is working. If (in the 1/201 case), you still receive the default value, try reinstalling the game to your device/simulator.

Another effect of this is that switching the user property back to enjoys will not change the value back to the baseline but again, you can do this by switching to enjoys and reinstalling.

9. Congratulations!

You've used Remote Config A/B Testing to experiment with different Remote Config values and determine how each affects Analytics metrics.

What we've covered

How to set up A/B Testing using instrumented Remote Config values
How to use Google Analytics User Properties as part of conditions for admission into A/B Testing experiments

Next Steps

Once the experiment is over, you can select one from the list of experiments in your project to decide what you want to do with it. Do you want to choose one as the "winner" or conduct more experiments?