PM Wargame: Measure the success of Instagram's Verified feature
The board
The situation is the following:
“You are the product manager in charge of the Instagram Verified feature. The feature is out in the wild. There are a few thousand verified profiles already. Most of the kinks are fixed. There are no major bugs or issues.
You are now tasked with the next step: figure out what success looks like. How does Instagram (or Meta) know Instagram Verified is a successful product? How is this success measured? What does good look like and why?”
In ‘PM Wargames’, I take on a hypothetical PM challenge and attempt to unpack the strategic & tactical approaches I would take to solve it. Unless sources are specifically cited, all information (including the challenge itself) is purely for illustration purposes only.
Actions
With the above in mind, here are the first steps I would take on this wargame.
Research
User research
First, I want to understand my end users. In this case, I identify three:
Verified users
Non-verified users who want to become verified
Other non-verified users
I want to identify the behaviours of each group of users as they relate to the verified feature. Below is a short user interview designed to extract insights from each group.
Verified users
How long have you been a verified user?
Why did you decide to become verified? If the verified status was granted to you by our team, (e.g. a celebrity) why did you agree to become verified?
What benefits, if any, do you get from being verified?
How would you feel if you could no longer be a verified Instagram user?
Non-verified users who want to become verified
What benefits are you hoping to gain from becoming a verified user?
In a few words, describe your impression of verified users.
In your view, who should (or shouldn't) be allowed to be a verified user?
Other non-verified users
Describe what a verified user is.
In a few words, describe your impression of verified users.
I'm aiming to gather qualitative insights on the feature. How do people feel about verified users? What words do they use to describe them? Does their trust in the user increase when they are verified?
I'll organise the feedback into themes: pain points and goals for each bucket of users.
Business research
Instagram is part of a larger company (Meta) and a network of applications (Facebook, WhatsApp, etc.).
I want to ensure the Verified Users' success aligns with Instagram's success and, finally, Meta's success. Without insider information, I make the assumption that Meta identifies success as:
Growing revenue per user
Maximising user minutes
Creating a trustful network
I will spend time identifying the key individuals in the hierarchy who can confirm the above. I'll re-use this list once I've defined what success looks like for Verified Users to ensure we are aligned. That also gives me the stakeholder buy-in I might need when I reach the implementation phase.
Competitive research
I'll wrap up my research phase with a snoop around our competitors. The following social networks have verified users:
Facebook (Pages, Business, Profiles)
Worth poking my head in the Facebook offices next door to get some insights!
Twitter/X
Twitter has famously winded down their legacy verification programme in favour of a paid premium plan. To become verified, an account must:
Be subscribed to X Premium (~£11/mo).
Have a complete account.
Have been active in the last 30 days.
Have a confirmed phone number.
Be non-deceptive.
Quora
Quora have introduced Verified profiles in 2015. The programme is limited to celebrities (e.g. Barack Obama).
Pinterest
Pinterest has introduced a Verified Merchant Program (VMP). Becoming a VMP is free assuming the account is:
At least three months old.
A claimed business account.
Complete with profile picture and 'About me' section.
Selling products that meet merchant guidelines.
LinkedIn
It's free to become a LinkedIn verified account, albeit only limited to the US, Canada, and Mexico.
Looking at my competitors, I see most don't attempt to generate revenue from the verification feature directly. I believe this is because the value-added of trusted users on networks of this scale is more significant than the small amount of money paid verifications could generate.
Ideate
I now want to start to ideate success metrics. I will use the output of my previous research to narrow down options until I've made a sensible selection.
Here's a rapid-fire set of potential metrics or goals:
Neutralising the competition
Revenue stream
Revenue per user
Number of Verified applications
Number of approved/rejected Verified applications
Verified account engagement
User minutes
Recency
Frequency
I can organise these into separate buckets
[ business ] - [ product ] - [ user ]
Assumptions
Given I haven't actually collected data from stakeholders and users, I have to make a few assumptions before I move any further. I'll assume:
Most users associate Verified Users with public figures and trust.
To most users, Verified User interactions stand out both in their feed (visually) and in their personal engagement (emotionally).
To the business stakeholders, monetising this feature is not a priority. Instead, they would rather see an uptick in product engagement, expansion and improvement of the social graph, and 'unregretted user minutes' (shoutout Elon).
Non-verified users want to become Verified mostly for clout.
Verified users would mostly want to keep their verified status (passing the PMF test).
With these assumptions, here is my game plan.
Game plan
I will define Instagram Verified as a success when:
We neutralise the competition. Most large social media platforms have some sort of verification system. Not having one makes us seem less trustworthy and less in control of our experience. The good news is this is a pretty simple success metric: ship the feature! Since the feature is already shipped, we can tick this box.
Engagement with and by Verified accounts is up. Although somewhat of a vanity metric, I want to notice an uptrend on both sides here. This indicates our research and assumptions are correct.
Core usage metrics are trending higher for users who have encountered Verified Users. I will bundle user minutes and recency under the same condition. If users who encounter Verified accounts tend to come back more often and stay longer, we're heading in the right direction.
Although not explicitly asked in this wargame's instructions, I want to spend some time thinking what an obvious failure would look like.
For Instagram Verified to completely fail, I would need to see a complete lack of trust in, or even aversion to, Verified users. A poorly implemented verification program may instil feelings of inferiority, pay-to-play, or favouritism in users.
To monitor this, I will turn my user interview questions into a continuous in-app survey.
Roll-out
Since the feature is already built and out in the wild, I should be able to collect the data so far. I will also set up a real-time dashboard for those metrics.
I will actively monitor this dashboard and run my failure surveys for six months. After six months, depending on the outcome, the feature will go into BAU under a specific team or get iterated on. Given the 'neutralise competition' aspect, it's unlikely the feature will get completely scrapped.
Premortem
Channelling my inner Shreyas, here are a few tigers, paper tigers, or elephants (aka scary things, things that sound scary but aren't, and big things we haven't talked about) with this game plan.
Tigers
I have not identified any tigers with this plan. The feature is already live. The metrics are simple to track. We are also monitoring the potential downside.
Paper tigers
I have mentioned a potential adverse effect of verification in creating a 'pay-to-play' atmosphere where Verified Users stand out more than mere mortals. I believe this is unlikely to happen and is an unfounded fear.
Elephants
We haven't defined what it takes to become Verified. This will have an impact on trust. We've assumed a Verified account is not a paid feature, which means Verified accounts must have some sort of clout. How do we define clout, how elastic are we willing to be with this description, and how are our metrics impacted depending on this elasticity?
We haven't talked about UI. For Verified Users to truly impact the experience of other users, they'll need to stand out. Is this just a UI implementation or do Verified Users also get higher reach?
This is my game plan. What do you think of this approach? How would you play this PM wargame?