A/B/n experiments in 3 simple steps [2023]

Updated 2023

Editor’s note: We originally published this post in July 2021, when we had just released Unleash 4.0.

In Unleash 4.7 (released in February 2022) we introduced a feature we call impression data. Impression data offers more flexibility and makes gathering data for experiments much easier. Refer to the guide How to use impression data for step-by-step instructions.

That said, the parts in this post about business goals and KPIs still apply as much as ever.

Feature management solutions like Unleash are a great for enabling A/B/n experiments. Read more to find out how.

Talking to customers, we often find that many are really eager to dive into A/B/n experimenting, but they often find it a bit difficult to get started. There might be a lot of blockers. One blocker we repeatedly hear about is inter-team dependencies, which can make it a challenge to get up and running.

We truly believe that development teams should be as autonomous as absolutely possible. What is important is to empower the team with the tools and mandate to be as independent as possible from other teams and their priorities.

Our experience is that it is common to give one dedicated team the ownership and responsibility of a shared analytics tool. This can often create a bottleneck for other teams who’d like to enable A/B/n testing.

This guide will show you how you can get started with A/B/n experiments in 3 simple steps using Unleash, all handled within the team.

Before we begin, there are some basics that you should have in place:

• Unleash. This is to create and manage the A/B/n test in your application.
• A preferred analytics tool. In this guide, we provide a step-by-step guide using Google Analytics.
• A clear understanding of the business goals of the development team. The development team need to understand the KPIs they want to improve through their A/B testing.

Our reference case is built on a joint effort we did together with the Norwegian Labor and Welfare Organization (NAV). NAV uses Unleash self-hosted as part of their software toolbox.

A/B/n experiment step 1: Define an alternative to the targeted KPI

The first enabling step for any A/B/n experiment is to define an alternative that you believe will perform better on your target KPI.

For simplicity: Let’s assume that the experiment will test the call-to-action button color. You expect a green button will perform better than the existing blue.

You then need to decide on what part of the traffic you want to perform the experiment on. In this case, you decide that you want to run the experiment on 50% of the traffic.

Here’s how to do it:

1) Log into your Unleash instance

2) Create a new feature flag by pressing the “New feature flag” button.

3) Give the feature flag a name according to your naming convention. The example shows “AB_experiment.”

4) We also recommend that you give the feature flag a short description. This is for your reference and documentation.

5) Now you need to choose an activation strategy. The example runs an experiment on 50% of the traffic.  This means a “gradual rollout” strategy makes sense.

This activation strategy means that the Unleash engine will pick out 50% of the users to be part of the experiment, as the percentage is set to “50%.”

You also want to make the experiment sticky. This means that if a user has seen the experiment once, we want this user to have a consistent experiment.

The example includes the user identifcation (UID) to create stickiness, as UID is part of the user context in the Super Insurance web application.

6) Now you want to create the variants that contain the actual experiment. Here you’re able to define a new variant, as well as a control variant. It is worth noticing that all the variants will be weighted equally.

There are many variants you’ll be able to define. In this example, we have created two: “BlueButton” is the control variant, while “GreenButton” is the alternative we want to test.

A/B/n experiment step 2 – Set up the experiment in Google Analytics

To visualize your A/B/n experiment results in Google Analytics, you can choose to use “Custom dimensions.”

You will find this by logging onto the Google Analytics admin console: Click “Custom definitions”, “Custom dimensions” and then “New custom definition”.

In this example, we’ve chosen to call the custom dimension “featureToggleName.” The name itself is irrelevant to the Unleash SDK, but the user needs to know the id of the dimension reserved for the Unleash data.

We have also chosen to use the “Session” scope on the dimension.

Usually, for a user that returns to the site after a day or two, the experiment might not be that relevant to that specific user. This article provides additional details on the available scopes for custom dimensions.

A/B/n experiment step 3 – Connecting the dots

The last step in setting up the A/B/n experiments is to connect the experiment with the metrics. The developer in the team now needs to enter a few lines of code to get the magic to happen.

In the source code below, you can see where both the Unleash SDK and Google analytics SDK are running.

const toggleName = ‘AB_feature_toggle’;

const variant = unleash.getVariant(toggleName);

console.log(variant.name);

// Map ‘dimension4’ to ‘unleash’:
gtag(‘config’, ‘UA-YOUR_ID-1’, { ‘custom_map’: {‘dimension4’: ‘unleash’}});

// Send an event that passes ‘variant’ as a parameter:
gtag(‘event’, ‘unleash_dimension’, {‘unleash’:`${toggleName}:${variant.name}`});

Your experiment is now up and running. You’ll be able to explore the results of the experiment using your already existing reports and views in Google Analytics. The Custom dimensions are available as a secondary dimension that you can easily access.

If you are using Google Tag Manager you can pass the data to the data layer, then onwards to your analytics tool such as Google Analytics.

First, create your event JSON and push it to the data layer like so:

dataLayer.push({ //send data to GTM

‘event’: ‘unleash’, //name your event

‘feature’: ‘new-logo’, //name of the feature

‘variant’: ‘control’ //name of the variant the user sees

})

Then, in Google Tag Manager, create 4 tags

• One custom event to trigger when your Unleash event is passed
• One data layer variable to store the name of your feature
• One data layer variable to store the name of your feature variant
• One Google Analytics event tag to send the data to Google Analytics when the custom event triggers

Read: Guide to feature management

Want to try Unleash?