Weekly Product Insights: A/B Testing, Customer Centricity and Responsibility
A new series with insights and ideas about building products.
Hello! You signed up for the Teach Me How 2 Product mailing list. Welcome! It might have been a while since you signed up.
I’ve been working on the site and content that’s hopefully useful, makes you think, or at least is entertaining. Hope it’s helpful, feedback is welcome and, if you’re new here, please subscribe!
Here’s what’s been on my mind this week:
1: A/B testing is not as straightforward as you might believe
It sounds super simple right? Take everyone that uses your product, randomly split them into two groups, and try out something new on one group while the other group is the control. If the numbers for the new thing go up, keep it! If the numbers for the new thing go down, don’t.
Well, sure, there is the rare occasion where the answer is clear as day and it’s a simple call to make. More often, though, you end up having a lot doubt and lack of clarity due to sample size, statistical significance, success metrics pointing in different directions, user targeting and pollution of the data set.
If you’re doing a lot of A/B testing, I highly recommend taking a course or making friends/enemies with a statistician. You’ll be better off for it.
2: A/B testing is simply one tool in the tool belt.
Here’s a mistake I see a lot: The numbers say what the numbers say, so we should do what the numbers tell us to do. Right? Sort of.
There’s a significant difference between being data driven vs. being data informed. The former abdicates your decision making responsibility to the numbers. The latter reinforces that you still have to make decisions, and the numbers are there to help you, not to be in the driver seat.
Because of how delicate A/B testing can be, as mentioned above, you still have a responsibility to interpret the results and make the right decision for your customers. Numbers can’t do that for you, humans have to on their own. And when you let the numbers drive completely, that’s when you get into trouble.
3: Pretty often, there is a lot that’s out of control and not your fault.
Shit happens. Tests go out without the tracking you need to assess the result. You miss some bugs that users hit in production. A design doesn’t quite hit the mark with your users. You get negative written feedback because your pricing is wrong. Those are situations that you can impact and exert control over.
But there are also lots of things that happen that you just can’t control. Agreements between parties crumble because market conditions change. People on your team leave because their life goals shift. A global pandemic hits and completely warps the market! It’s not your fault, but…
4: Even when something is not your fault, you should still take responsibility for it and respond.
If you became a Product Manager or a Product leader of something, likely you have exhibited some leadership traits. As such, good leaders take responsibility for things that don’t go right, even when it’s not your fault or out of your control. Often, it’s a series of small decisions that exhibit leadership and take charge of a situation that are the difference between an organization crumbling and an organization that’s thriving. So take charge. Be a good leader.
(For reference, I recommend Bob Iger’s book on the subject. Amazing.)
5: Focusing on your customers is still a very solid move.
A really good example of someone that’s doing this really well right now is Peloton. Yes, that bike is crazy expensive. And yes, a lot of the success of the brand is really great marketing. But man, they really nail the interactions with customers on their platform and deliver a great product. I’m positive, for them, it starts with their customers.
I have proof of this. They did a software update recently, and I noticed a really small but monumental change. (Side note: most monumental changes are really small I’ve found.)
In the previous version of their interface, they had an up and down arrow next to the cadence and resistance. The numbers in between would show your target range, and they would light up yellow when you were in the target range. If you wanted to hide them, it was all or nothing. You had to hide guidance for both cadence AND resistance.
Was that good enough for Peloton? Nope! They changed them, and they’re so much better! Now there’s an indicator that shows where you are in the range, and you can close one indicator or the other, or both. Brilliant.
I’d bet my left arm that this came out of customer observations, feedback and research. That’s what great companies do: obsess over the customer experience. It’s not rocket science, but it almost always leads to better products.
I hope you enjoyed this first post! If you like what you’ve read, please subscribe or forward this article to someone that might find it useful and I’ll do my best to keep cranking these out!