Six A/B tests used by Duolingo to tap into habit-forming behaviour
If you’ve used Duolingo, the language education app, you’ll know that it feels like a game.
That’s no coincidence of course. Duolingo is acutely aware of the power of gamification, and in a recent talk atCanvas Conference, product manager Zan Gilani detailed some of the A/B tests it has used to tap in to habit-forming behaviour in users.
A wee bit of background to being with (then we’ll get to the A/B tests further down the post):
Duolingo has 25 languages available, with Mandarin being added soon.
It has more than 200 million users, making it arguably the most downloaded education app.
The Duolingo mission is to provide free language education across the world (the app is free to all).
The company now has billions of data points (six billion lessons are completed every month) and is constantly iterating.
Four ways to motivate learners
The big challenge for Duolingo is keeping its users motivated. Not only is learning something by yourself very tough, but studying online brings with it additional distraction. Gilani says that MOOCs (massive open online courses) have very low completion rates, somewhere around one or two percent.
To overcome this challenge, Duolingo is learning from the gaming world, with Gilani referencing four ways to design a daily habit:
Small concrete goals
Visible progress
External triggers (get them to come back)
User investment
Those familiar with Duolingo will know that small concrete goals are built in to the app, with users having to complete a unit in order to unlock the next one. Zilani draws the analogy of getting fit – improving your fitness is “a vague and nebulous goal, but running a mile every day is very straightforward.”
Streaks fulfil all four criteria
What Zilani and Duolingo wanted to concentrate on was the power of the streak, a powerful game mechanic that fulfils all four habit-forming criteria. One can see apps such as Snapchat, Facebook and Headspace alerting users to, and rewarding them for, consecutive days of activity.
So, on Duolingo, users are encouraged to set a daily goal, to make a pledge to use the app each day, and with every consecutive day you complete you will extend your streak.
Users are so conscious of continuing their streaks that Gilani even showcased an email one user had sent to Duolingo HQ explaining how they had lost internet connection (and hence their streak) during Hurricane Irma, and asking if they could please have their streak reinstated.
A/B test #1 – visible progress
Users need to see their progress and streaks weren’t always that visible in the Duolingo app. So, the team did a test to show streak days in the top of the app at all times so that on any given date, people know what there streak is.
The results were impressive. A 3% increase in daily active users (DAU) and a 1% increase in day-14 retention (D14, those still using the app after 14 days). This uplift makes a big difference in the long run.
Test #2 – emphasise the streak
Next up, emphasising the streak after every lesson. Users began to see the ring of fire shown below.
Results: 1% increase in DAU, 3% increase in D14.
Test #3 – external triggers
External triggers such as emails and app notifications can feel spammy, said Zilani, but streaks make them seem less so. Streaks give a valid reason for an external trigger, so Duolingo can say “remember you’re on a 24-day streak” rather than simply saying “come use our app”.
In this test, the team experimented by sending these external triggers at different points after a user’s last lesson in the app.
The results showed that emails sent 23.5 hours after the last lesson worked best at encouraging re-engagement. That’s unsurprising, as Zilani pointed out that doing something at the same time everyday gives the best chance of forming a habit.
One fun part of Duolingo’s external triggers is its use of a ‘passive-aggressive notification’ after five days of inactivity. The tone fits well with the playful nature of the brand, with the message beginning: “these reminders don’t seem to be working…”
This is the most passive-aggressive push notification I've ever received pic.twitter.com/YY6L285J0d
— Molly Fitzpatrick (????) (@mollyfitz) May 5, 2017
Test #4 – user investment
Duolingo has a virtual currency, the lingot, with which users can buy, amongst other things, a streak freeze (allowing inactivity without destroying a streak).
One of the things the team tested was a streak wager, with users wagering lingots that they will hit a particular length of streak. Previously nested away in the app, this feature was surfaced (specifically a seven-day wager shown once a week).
The results: 5% increase in D14 and 600% rise in IAP (in-app purchase revenue) revenue.
It should be noted that Duolingo is always free to use, but users can now pay in-app for a subscription which allows offline use (perhaps crucial for maintaining a streak) and removes ads from the interface.
This test, Gilani mentioned, was also tried with a two-day streak wager, but the increase in D14 and DAU was minimal enough that it was abandoned.
Test #5 – designing for moments of weakness
Duolingo see 9% fewer users on the weekend according to Gilani. Typically, this is a time when many users lose their streaks. Losing a streak can be very demotivating.
So, what if a user could skip a day and keep their streak? That’s what the weekend amulet allows you to do, and it can be equipped for 20 lingots, Duolingo’s virtual currency.
Offering this to users to say “you don’t have to play on the weekend” again had a big impact on app usage.
The results: 4% increase in D14, and users 5% less likely to lose their streak.
Test #6 – create multiple ways to win
It’s not just streaks that keep a user engaged. Duolingo also moved into achievement badges. These are tricky to get right, as Gilani puts it “usually these are implemented very poorly – in narratives on bad gamification, achievements are the villain, but when they are done right they have a positive affect..”
Duolingo badges are permanent (unlike streaks) and are varied, achieved for a variety of things e.g. inviting friends, logging in before 8am etc. Badges are also completable, unlike streaks – Gilani even used the phrase “gotta catch them all”.
The first test for achievements was congratulating users who had joined Duolingo. The results were underwhelming – 0% DAU, 0% D14 0% everything else.
Gilani says they had forgotten that tests had to be viable. Signing up is not enough of an achievement. “Users have to see what they have achieved”, Gilani adds, “when they start achieving, desire kicks in.”
Testing an increased range of different badge achievements did have an impact. They saw a 2% increase in DAU and a 2% increase in D14, as well as a 5% increase in in-app purchases.
This success was built on by adding tiers to the badges. For example, users could get a badge for scoring a perfect test, but then for five perfect tests, 10 and so on. Tiered badges further improved results – a 1% increase in DAU and a 1% increase in D14.
A feel-good note to end on
Gilani finished by addressing the hall of delegates and telling them that product designers are able to empower more people to go further, and to learn more. One of Duolingo’s success stories is Edilson, a Colombian security guard who taught himself a number of languages through Duolingo, and has recently got a new job at a security academy where he teaches English.
So, what can you A/B test in your apps and services to keep customers coming back?
