PM Wargame: Use AI to improve Duolingo's engagement metrics

The board

The situation is the following:

"You are the product manager in charge of Duolingo's engagement metrics. Your focus is to get users to come back as often as possible to practice and learn.

Your manager, like everyone else in the tech industry, has been following the booming AI trend. They wonder if AI, particularly language models, could be a good addition to the user experience (and your metrics).

'AI' is a vast topic. How exactly do you think language models could help your team? What are its practical applications? What do you expect the impact might be (on the up and the downside)?"

---

In ‘PM Wargames’, I take on a hypothetical PM challenge and attempt to unpack the strategic & tactical approaches I would take to solve it. Unless sources are specifically cited, all information (including the challenge itself) is purely for illustration purposes only.

Actions

With the above in mind, here are the steps I would take on this wargame.

Research

Large language what?

I have an unfair advantage in this wargame. I have been working in artificial intelligence and, more specifically, chatbots/large language models for almost 10 years.

Let's pretend I know nothing.

Technology

My first step is to look into the technology. I've heard of ChatGPT (but how does it work?). I've heard of Midjourney (is that funky computer-generated images?). I've seen the 'LLM' acronym thrown around (large something something?).

I'll quickly search the keywords I heard from my manager ("chatbot", "LLM", "AI", and "language models"). What comes up?

I want to make a list of the core components of this technology. I want to get to a point where I can explain the technology to my parents (not quite a 5-year-old, not quite a tech-savvy individual).

Key players

Next up on my research journey is to list out the key players. I want to identify who's doing what.

If the output of my previous research into the technology was a diagram of how things work, here I want to assign each part of the diagram to one or multiple players.

That helps me understand the landscape.

First order implementation

Now that I know what it does and who does what, I can explore this technology's application. First, I want to understand how the key players are making money. 

I want to list each large provider and research their product, pricing plans, revenue, and any latest news/roadmap items I can find.

That, again, helps me understand the landscape.

Second order implementations

I've done the basics. I've got a base-level understanding of what 'this' is, what's out there, who's doing what, and for what money.

That is a good place to be.

Next, I want to see what other companies are doing. Not the providers this time, but the companies using the providers. I should have already seen names and use cases in my previous research; I'm just digging further.

I want 20-30 implementations that I'll organise this way:

• Company

• Vertical

• Provider used

• Use case (long description)

• Case study/outcome (if I can find real-life data)

Inflection points

With this research in my back pocket, I'm turning to our product and, more precisely, my user journey.

I'll assume the following journey to be somewhat accurate:

Illustrative customer journey map of Duolingo

As the PM in charge of engagement, I'm particularly interested in the following inflexion points:

Starred items are particularly interesting to create & encourage retention

Ideate

Time to be creative. 

Based on my research and my (assumed) extensive knowledge of the Duolingo platform, here are a few ideas we could explore:

1. Create a support chatbot.

2. Generate more varied exercises in real time.

3. Create a coach chatbot trained on the user's flaws and recent errors.

4. Scrape online communities for new(er) slang and teach it to advanced users.

5. Create practice personas like 'Walt the Waiter' or 'Belinda the Bus Lady'. Users can practice specific scenarios in a script-free format.

6. Combine LLMs & AI-generated video to create training lessons with 'real humans'.

With these options, here is my game plan.

Game plan

I will recommend we trial this new technology by creating a script-free, scenario-specific set of practice chatbots. To ensure this initiative boosts engagement, I will:

1. Come up with a persona for the first 10 lessons of a language. Users will engage with them early in their journey, giving me precious feedback on the idea.

2. Introduce the practice chatbots to new users as soon as they've completed their first lesson. I want my users to know these practice chatbots are available.

3. Gamify the engagement with the practice chatbots. That shouldn't be difficult since the rest of the app is already heavily gamified.

4. Remind users to practice regularly with push notifications. I might frame it as 'going to the dojo' or gym reps. 

5. Design the chatbots to:

   1. Maintain context. Users will come back if they feel like they're having a text conversation with a friend.

   2. Understand the users but point out when a grammatical mistake is made. That will be a tricky balance to strike.

   3. Congratulate users for mistake-free streaks, good usage of tricky language, advanced vocabulary, etc.

6. Sporadically recommend in-app lessons as part of the conversation to keep engagement high.

For this to work, I need my practice bots to feel like friends. I may spend extra time playing with tools that have nailed parasocial interactions (e.g. Pandorabots) to understand how to truly instil personality and stickiness in a virtual friend.

Why I'm dismissing the other options (for now)

I am prioritising the idea above. Here's why I'm not picking the others on that list:

#1: Customer service is not my priority as an engagement PM.

#2: Too low-hanging fruit, likely no engagement impact.

#3: Duolingo is already excellent at this.

#4: Feels gimmicky, probably no impact. Could be a fun marketing initiative!

#6: I'd keep that one in my back pocket.

Roll-out

My managers' instructions were to come up with practical applications of AI. I've done that. I could hand this over.

But it feels a bit light.

To truly wrap this up, I want to prepare two documents.

First, a list of recommendations for tech providers. 

I know enough at this stage that I can put something basic together; enough to give a leg up to the dev team we'll eventually hand this over to.

I could look something like this:

• Provider

• Cost per token

• Link to API docs

• Point of contact (if I made any)

Second, an experiment plan.

This feature is a shot in the dark. We want to minimise risk, get a quick feedback loop, and know what good (and bad) looks like before we start.

I will put together a simple experiment plan defining:

• User segment (who will get access to this).

• Success criteria (what does good look like).

• Hypothesis (what I expect will happen).

• Duration (when will we end this experiment).

Premortem

Alright, the fun part: what could go wrong?

Since this is an experiment, we can drastically control the downside. Still, here are a few things I'd keep an eye on while this game plays out.

1. The biggest risk, arguably a tiger, is that LLM technology might, ironically, generate poor language. We don't know whether a computer-generated interaction will be mistake-free. This is a huge risk for an app focusing on teaching a language.

2. Duolingo has a distinct, clear, and unique voice. We use characters throughout the app. There's a risk we dilute the effectiveness of what we already have.

3. We're banking on long-form, long-context interactions. It might be that users would prefer to repeat the same interactions over and over.

4. The balance between 'fun conversation with a bot' and educational will be tricky. Monitoring and calibrating it? Even trickier.

5. We haven't spent any time on user research. Is this something users want? It's fine as a growth experiment but we'll want to speak to some users very soon.

This is my game plan. What do you think of this approach? How would you play this PM wargame?