In my last post, I went through 5 categories of input that you should be using to drive the best product decisions. It’s a useful checklist I like to revisit a few times a year.
This post dives much more deeply into how to learn the most important things every time you put a variation of your marketing or product experience in front of real users: (1) what those users are doing and (2) why.
Science for Product Managers
To bring some science into product management and marketing you have to observe the natural behavior of your users. Your goal is to predict the behavior of a larger market based on the behavior of a smaller sample group. If the sample group loves the product update, uses the new features, clicks through the ads, or pays for the subscription - then it’s time to invest more resources and deploy the updates much more broadly. If the sample group ignores the new stuff, is confused by the update, or generally underperforms on your metrics compared to past variations, then you likely need to change some things. The closer you can get to this ideal of observing the natural behavior of your users, the more accurate your predictions will be.
The Ideal Setup for Experiments:
If you’re working on improving a popular product, site, or service you already have the best possible setup for learning and validation - your own active users. The trick is making sure your experiments are done with an appropriate level of risk. E.g. if you’re working on Expedia flight and hotel listings, I’m sure you already have an optimization framework in place that lets you filter a fraction of 1% of your traffic through new variations. When new variations perform poorly, you’ve only risked a fraction of 1% of sales revenue. When new variations perform well, you can quickly roll them out to the other 99%. This kind of framework allows for rapid experimentation, minimizing #fails by default and allowing you to maximize #wins.
So what about the rest of us? There are all kinds of situations that are less than ideal for product and marketing experiments, but with just as great a need:
- Early stage startups that don’thave customers yet, or have too few customers to collect accurate data in a reasonable amount of time.
- Established companies going after new target markets (e.g. Amazon.com went after 3rd party sellers, advertisers, and cloud developers - all very different markets than online shoppers).
- Product delivery situations limited by partnerships, contracts, or market makers (iOS and Android apps, software bundled into other products, white labeled services for major brands, etc.).
- Products without sophisticated optimization platforms in place.
If one of these setups describes your situation - the key is to get as close to the ideal setup as practical and go forward with your experiments. Imperfect learning is light years better than none.
- Get real users (not friends, auditors, or testers) to discover your product through a real marketing channel (digital ads aimed at your target market are a clean way to do this).
- Get enough users to draw conclusions. You can start to make really useful predictions with about 1000 daily active users. But in some cases you can evaluate a new idea with as few as 150 or 200 users. The math is a little involved but free calculators can help.
- Present your product as released (not a beta or trial), so people make real decisions about purchasing and quality. I’ve been surprised by strong positive reactions to products the internal team thought weren’t ready, so we released faster than planned! As per lean methodologies, you can do this with new features inside a product and even marketing campaigns before you have a product. People who think they’re evaluating a real, finished thing act normally. Beta testers do not.
What Are They Doing? Why?
Once you have users interacting with something they think is real, the behavioral data you get from your product or marketing analytics packages can answer really important questions like these:
- Are people responding to the new thing?
(If not, present a different offer or dig deeper into user needs) - “What are users doing in my product?”
(Invest more in these areas) - “What aren’t they doing?”
(Change the approach, or cut the feature to simplify the product) - “What are the effects of our latest updates in terms of user actions?”
(Even better if you run A/B tests with new updates) - “Are people staying engaged long enough to grow this as a business?”
(Most easily viewed as cohort tables) - “Where are the biggest drop-offs in my user lifecycle?”
(Best viewed as progression funnels)
We’ll call this the WHAT data. It’s vital, and you should record everything your users do, down to the individual click stream level. You record everything for two reasons: (1) you won’t know all the important questions in the beginning, new important questions will come up as the product evolves, and (2) you don’t want to end up with metrics debt - your team should get into the habit of recording all user interactions with the product, so your metrics are always up to date with the latest changes and features.
Set up reports for the most important behavioral metrics, share these metrics with the whole team, and judge your product progress based on them. User engagement measurements are vital to most products. User progressions through product stages or a product lifecycle can also yield big insights. Social metrics (Likes, Shares, Reads, K-factor) and sales figures (Last purchase, Average purchase size, LTV) help you identify your best customers and guide marketing spending. Use charts and visualizations that work for the various kinds of data and review it at least once a week to spot problems and opportunities.
But behavioral data like this isn’t the whole picture. What your users are doing will sometimes be confusing or frustrating. If people are churning off your service after a few days the important question is WHY? When you understand WHY, you can fix it. You’ll naturally form hypotheses as to WHY on your own whenever the data is surprising, but the main ways to figure out the true WHYs are by holding Q&A sessions with users who fit the confusing behavior pattern (e.g. pop-up surveys or phone calls), or by much more closely observing the way a few people behave and react to the product to draw accurate conclusions.
Use Qualitative Research to Learn Why:
At many companies, qualitative research is underutilized or completely missing. This is a waste, because it’s the best way to gain deeper insights into what excites people, what confuses them, and what their unmet needs are. These unmet needs will become the future hallmark features and differentiators for your product. You don’t need to be an expert to do this well. Read some articles, and go do your first contextual inquiry session. You simply go to where users are, and you watch them do the things you think your product might be able to improve for them, just like a non-interfering scientific observer/ethnographer. I used to do this years ago with the Microsoft Office team. Just watch people do their jobs, whether or not they were using our products. Some of the best new feature ideas came from what people were trying to do with pen and paper (OneNote) or coordinating with their work teams (SharePoint). At Ontela, we went and watched Cellular South retail reps sell new phones, knowing that we’d have to adapt the product to their sales environment. At INRIX, we set up “drive-along” sessions to observe drivers struggling with traffic and think of new ways to meet their needs. I recommend you do this kind of in-person learning whenever you’re planning something big and important. You can do it before you even have a product.
As for usability tests, everyone’s heard of them, but few people do them regularly. I believe in the value of user testing, and some of my teams have still gotten stuck in the pattern of a usability test every 3 months. But with the newer online services available for designing, recruiting, running, and recording studies, there’s really no valid excuse anymore. It’s *so* worth it. You can now spend 1 hour and $200 to get some meaningful insights into how people are reacting to a product or planned update. At Mazlo, we incorporated this pattern into marketing optimization tests. We’d build a variant, run two hours of user tests in the evening, and if there weren’t any major surprises, we’d go live with ad campaigns the next morning to get useful conversion data. This process is lightweight enough to do multiple times a week.
Another thing that can limit the value of usability tests is guiding users too narrowly through a list of tasks. This is the traditional technique, and it’s designed to come up with success rate information. “5/8 users tested successfully completed the task of uploading a profile photo during setup.” Not so revelatory, is it? In the early stages of product development, you can learn a lot more by running usability tests without clearly defined tasks. E.g. “Your friend Mike just sent you this email, giving you $10 off a new service. He said he likes it and he thinks you would like it to. Go ahead and do what you’d normally do in this situation, and speak your thoughts aloud as you do.” Now the user you’re testing will set her own goal, maybe to figure out if this website is worth her time and attention, and you’ll likely learn deeper truths about user expectations and problems, in addition to spotting the usability gotchas along the way.
How Your What and Why Inform Your Who
You can go far understanding what your users are doing and why. But there’s one more vital question that requires another technique altogether: “Who are my best customers and how do I find more of them?” The clues you need to answer this question are hidden in your behavioral data. By looking at the set of users who have the highest lifetime value, the greatest engagement, who’ve made the most referrals, or who’ve contributed the highest quality content to your ecosystem - you’ve already identified a subset of users who are by some measure ideal for your product. Now the trick is to figure out what they have in common. Which marketing campaigns or referring web sites brought them in to the product to begin with? Do they share a demographic profile? What features do they use the most and how does this differ from the overall averages? Break out your surveys, phone follow-ups, and qualitative techniques for these users specifically. By understanding them more deeply - why they love your product, what problems of theirs it solves, what pain points they still have - you’ll be able to steer your future marketing efforts and product investments to find, onboard, and retain more people like them. And more customers like your best customers will have a supersized impact on revenue, community cohesion, and viral growth.
Wrapping Up
Using these techniques and your own creativity, you should be able to move closer to an ideal set up for your product and marketing experiments in just a few weeks. Then you’ll start getting good answers to your What and Why questions that will lead you to make your marketing and product better each iteration. Your next step is to turn this whole cycle into a natural, habitual process for your team. What metrics do you look at every week? What metrics are you aiming to improve with new feature ideas? How will you set up the right amount of process for your team to incorporate behavioral data and qualitative research into a regular rhythm?
Then when you have a good learning process and active users, take the time to figure our what your very best customers have in common. The answers will help you direct your marketing to find more people like them, and better meet their needs.
Want a quick assignment to learn something new about your product and users next week? Spend 1 hour right now to set up your first online usability test against your marketing web site. Or send out a customer email asking for volunteers who will let you come watch them in action for 2 hours in order to make the product better.
Happy experimenting!
-Mike
(Liked this post? Read more of my stuff at www.WhateverNeedsDoing.com)
(Title photo courtesy of Gabriela Talarico, usability test sign photo courtesy of Aaron Fulkerson, both underthe CC license)
