Challenge

Yousician is a music-learning platform boasting 20 million users per month, making it the top music learning platform in the world. 

A/B test targeting is critical for Yousician for three reasons: 

• to target users by country, 
• to target users who would benefit the most from A/B tests, and 
• to target users who are eligible to take the A/B test.

“We want to offer a consistent experience throughout all these touchpoints by using the same A/B tests where logical,” says Adam Kapos, Lead Architect at Yousician.

For Adam, the consistency needed to exist across Yousician’s platforms.

“Our apps are available on Windows, macOS, Android, iOS, and also the web,” says Adam. “When we run an A/B test, it usually affects several platforms. We want to be able to offer the same user experience across all these different touch points.”

The company had been using an A/B testing tool they were quite happy with. However, they encountered challenges running tests with the same users.

“It became an issue that lots of users would be in our ‘sticky’ A/B tests who don’t actually receive the A/B test,” says Adam. “For example, a user would only be part of a test because they launched the iOS app once a month ago, when they mainly use Android.”

This reduced the effectiveness of their A/B testing tool.

“If we were not able to test the same users, then not only would it be hard to measure the impact of an A/B test, but it could lead to bad user experience like features missing for a user,” Adam shares.

Solution

Adam and his team began looking for alternative options for A/B testing.

 “We needed a solution that could integrate an SDK into Yousician’s back end,” says Adam. “At the same time, we needed to have control over when users were considered for an A/B test to make sure they would always land in the same variant test group.”

The solution wouldn’t need to be a dedicated analytics platform.

“We have our own data pipeline so we were not looking for built-in data analytics,” says Adam. “Instead, we wanted to have some sort of review feature for A/B tests where people could double-check that the settings were correct.”

Eventually, the team landed on Unleash. For Adam, it was about keeping decisions and data in-house.

“We chose Unleash because it ensures that none of our users’ data leaves our servers,” says Adam. “All targeting decisions happen on our servers in real time. We don’t have to rely on a third party service to make those decisions for us.”

 The flexibility to design their own strategies was also appealing to Adam. 

“We also like creating our own targeting strategies on our backend,” Adam shares. “This means a definition can be created by an engineer, then reused.”

Results

Yousician now uses Unleash for most of their A/B testing, through various apps and on the web. Adam’s team has found Unleash to be both flexible and simple.

“We’re not forced to create something complex on a web UI again and again. We define our own stickiness strategies so we can ensure that the same user will always receive the same experience.”

The company has been able to use Unleash to ensure the same users receive the same tests.

“For example, if a user has “Feature Flag A”, they will be considered for the A/B test, and they will always land in the same variant group,” says Adam. “ This helps us make better decisions about the outcomes of our A/B tests. It’s always clear whether a user should receive an A/B test or not.”

Adam and his team are now able to run more complex A/B tests.

“When we got started with A/B testing many years ago, we started running many small experiments,” Adam shares. Nowadays we run fewer but bigger A/B tests with several variants.”

Unleash is now used throughout Yousician.

 

“We have three live products and each of those have multiple teams working on them,” says Adam. “We have an Unleash project for each of these products and our goal is to enable the teams to work independently of each-other.”

While the team was initially wary that they could not define the same feature across multiple projects, they quickly realized this was a useful feature of Unleash.

“We have several apps, but we also have a common website which actually has some of these A/B tests,” says Adam. “By making each feature unique, we can ensure that there are no conflicts on the website. Plus the website can receive features and A/B tests from multiple products.”

Yousician has also been able to take advantage of simplified targeting.

“We can simply targeting by automatically targeting users who are eligible for an A/B test,” says Adam. “This usually means that the app or website has a certain feature defined. With our previous A/B testing solution we had to rely on things like the app version used.”

Unleash has become a critical part of how Yousician conducts A/B testing.

“We started using Unleash more and more for internal feature flagging, but when it comes to the user facing things, A/B testing is king,” says Adam.

Adam’s advice to companies who are new to feature flags

“I think feature flags are a wonderful development tool that allow developers to integrate new features early and frequently. Being able to run A/B tests is the cherry on the cake. 

“However, it’s important to keep in mind when creating a feature flag does not make sense. 

“Example: When we wanted to unify the branding of our apps, we were thinking about making it a feature flag. We’d then flip it when everything was ready. The goal was to have a unified brand rolled out completely in sync across many apps.
“However, we found that the development overhead would have been so big that in the end we decided against it. We would have needed to duplicate countless assets to make it work.”