đ§ How to tackle unknowns building products
Part of a weekly newsletter with advice for building digital products.
Hello! This is the Teach Me How 2 Product mailing list. Welcome! Iâve been working on the site and content that Iâve found useful in my career so far. I hope it makes you think or at least is entertaining. Feedback is welcome and, if youâre new here, please subscribe!
Hereâs whatâs been on my mind this week:
The things that should keep you up at night are the things you donât know.
A stress dream that I continue to have to this day is not actually graduating with my engineering degree. I donât really get a flop sweat when I have it. Usually, it involves a lot of tossing and turning at night when the same video plays over and over in my head.
I get a call from an Austin number. Itâs the engineering school, theyâre calling to tell me that there are a few credits that I actually missed as part of my degree plan. I freak out and then find myself at campus trying to muscle out some Câs to graduate all over again. Itâs a terrible, sinking feeling every time.
I tend to get that sinking feeling when I come across something I didnât know about. Sometimes itâs a wonderful surprise, so that feeling doesnât last. But in our business, it tends to be more of the not-so-wonderful variety.
ThusâŠsinking feeling.
You might know some thingsâŠ
It's not completely hopeless. There probably is already some stuff that you know about your product and your business.
Letâs say youâve started a job at a new company working on a new product. There are a few things that you have been told about your product during onboarding. Maybe some baseline conversion rates, how certain features work. There is information that you have discovered as you have gone through your onboarding process. You understand things about your architecture. Your peers and leadership team have helped you understand your current strategy perhaps. You might have also learned some things from previous products you have worked on that you anticipate using on your new product.
Itâs tempting to take it all at face value and run ahead with cranking out new features, following blindly what youâve learned so far. But, itâs critical that you understand that what you know vs. what you donât know likely looks like this:
âŠbut you certainly donât know a lot of critical stuff.
In an effort to help you accept the inevitable, that there is a ton of ice under the surface that you donât know yet, here are some examples of things yours truly has run across that I didnât know that really bit me in the keister:
Early in my career, we shipped a new feature in an app to help a property owner quote a traveler on how much their vacation rental would cost to rent. The product passed QA, we all celebrated. In a user interview with a property owner, I discovered that the method we had been using to calculate taxes was old. We were calculating taxes with old math for lots of places people lived. Doh!
When I first set up an A/B testing program, I took on face value a lot of techniques and technology we were using for calling experiments. When the dashboard lit up green, the test was called, right? WRONG. Turns out thereâs all this pesky stuff about statistical significance, Bayesian vs. Frequentist experimentation methods, test exposure, etc. thatâs really important. And by really important, I mean that it can be the difference between actual product development progress and rolling out a bunch of crap that tanks all your numbers. So, I learned it all THE FUN WAY. đ„ł
We were in the middle of a huge re-platforming of a critical page in the shopping experience. In the middle of that effort, we discovered that our instrumentation for a critical metric we were tracking was completely different in the new platform vs. the old platform. The numbers we were using to track progress turned out to be completely wrong. đ€Šđ»ââïž
FRACKING COVID. We had no idea how a global pandemic would affect the vacation rental sector, much less the travel sector. Nine months later, weâve seen how things have shaken out. But in March, it was a complete unknown. We started 2020 with a completely different set of plans. Those were destined to change.
Freaked out yet? You should be! The iceberg is lurking out there, just waiting for you to run into it. So, what to do?
Illuminating the iceberg: how to know what you donât know
Itâs probably a good idea to start understanding what lurks below the surface. In my experience, there are a few key steps that are required to do this.
Accept it
This is probably the most important thing to do. You have to accept that this part of the iceberg even exists! There are WAY too many people walking around thinking that they have figured everything out. They havenât, I promise. Itâs impossible to know everything. You are on a journey of discovery and learning. whether you acknowledge and accept that you are or not.
Itâs much better if you accept it, I promise.
Be quick to respond, triage, and troubleshoot
If youâve come to accept that there is a lot that you donât know, that will help when the inevitable happens: your ship hits the part of the iceberg under the surface. When that happens (not if!) itâs important to have the mindset of being quick to respond when things come up that you donât know or are surprised about.
Donât freeze up, donât be the deer in headlights. Start training your mind to think like first responders: run towards the burning building instead of away from it. Analyze the situation, see where there are structural issues. Start thinking about what the priority issues are to address and troubleshoot them.
This is the behavior that got us out of the COVID-19 travel ditch. It was certainly a shock when lockdowns started happening and travel demand took a nosedive. Rather than panicking and watching things fall around us, we got to work. We re-focus our efforts on improving the core product and business, triaged bumps big and small as they came in, and worked quickly to address them. As a result, we survived.
There will be time later to reflect on the event and experience the emotions. You want to survive first so you can live to reflect.
Okay, weâve talked about what to do when something you donât know jumps out and bites you. How do you try to understand what you donât know so youâre not surprised when stuff comes up?
Question assumptions
In everything thatâs shared with you in onboarding, by your teams, by industry experts, there is inherently a set of assumptions that everyone is making. Those assumptions could be about how your business works, what your users are like, how your team works to ship product, etc.
Itâs important to identify those assumptions, question them, and learn more about them to understand how what youâve learned does or does not apply to your situation. You donât have to be mean about it! Just know that they are there and that, without questioning them, they could help you run into the iceberg. This can include things like assumptions about your instrumentation strategy, how you call A/B tests, why success metrics were chosen, why the company strategy is what it is, etc. Donât forget, lots of captains have assumed there was nothing beyond the tip of the iceberg.
Identifying assumptions can also lead to opportunity. For example, maybe you have a search product and you have to rank results in search. The ranking algorithm that everyone believes is good might not be. You could research it, find the gaps in it, and voila! You have a ranking experiment that could demonstrate positive results for your business and customers.
Research, preferably with actual users
Do your homework! Sounds like school when you were a kid, right? Well, as adults too this applies. Youâd be surprised how many people donât follow this advice when doing important things like putting together a product strategy.
Research helps you understand assumptions youâre making, where there are gaps, how the market is going to respond to what youâre doing, what your customers are actually going to do when they get their hands on your new feature or design, etc. It generally gives you the opportunity to either have confidence in your approach or have fact-based reasons for why youâre experimenting with a change.
Research can be competitive research, that can help you understand the landscape. Industry research is good too, that gives you a sense of whatâs new in your field and what customers have come to expect. My personal favorite kind of research is taking your product out to actual customers. People are unpredictable, your brilliant idea could crumble in the face of talking with actual customers. This happens a lot, and is the reason why you should do it! It helps you understand what you donât know about your customers, stops you from building useless stuff that no one wants, and sparks new ideas from things your customers say they want from you. (This has happened to me. One user session a few years ago had feedback direct from customers that shaped our multi-year strategy. Super useful!)
Analyze with data
Data is your best friend when youâre out there busting up assumptions and trying to illuminate the iceberg. It can mean the difference between freaking out over something that doesnât matter or identifying a potential problem that saves you days of troubleshooting and scrambling.
When in doubt, instrument everything that you can think of. It will come in handy when potential problems arise. Make friends with an analyst that can show you how to pull your own data. If youâre lucky enough to have access to self-service tools like Mixpanel or Amplitude, learn all of the ins and out and use them every day.
A lot of good product people excel in either user research or data analysis. The great PMs learn how to do both really well. Be great.
Test hypotheses
If you have read back on other pages of this blog, youâve probably heard me talk about A/B testing a good amount. Why am I always blabbering on about that? Because itâs important!
Itâs one of my favorite tools in the tool belt because it allows me to test hypotheses that break up assumptions and helps me understand what I donât know scientifically with empirical data. I highly recommend it as a way to understand unknowns. Itâs very similar to user research in that it surprises you and your team, regularly. Customers are different than you are, if you think something is great, half the time your customers donât.
A/B testing also has some added benefits:
It helps you settle team disputes and demonstrate that a new feature has value with actual customer data
It prevents you from rolling out new things that have a negative value. Often, the most important experiments are the ones that you DONâT roll out. These are things that your customers would have had to suffer that you prevented with an A/B test. Good for you!
Testing doesnât have to be limited to A/B testing. You can give a new feature to 20% of your customers, or send an email to 10% of your email subscribers. The key idea is using experiments to understand unknowns and take educated risks. Over time, this prevents unknowns from catching you off-guard and enables success.
Be a sponge, absorb everything
Early in my career, Iâd get frustrated with some meeting invites that would get put on my calendar. Iâd always ask why these things were useful. Iâm just going to be sitting on my laptop not really paying attention, why am I going?
Of course, everyone was smarter than I was. There were definitely some meetings that were useless sure. But there were a lot of meetings where I learned new information about how the business, company, or product worked. All by just sitting in the meeting for a bit, or reading a document someone sent me about something theyâre working on.
When in doubt, be a sponge. Absorb everything, read what people send you, and generally be curious. You never know, something you learn might be the thing that saves your product.
Adjust your strategy with new information
The final piece is being flexible about your strategy, and having a decision tree for when possible events happen.
When you discuss a strategy with your team, itâs important that there is an understanding that your strategy can, and should, change as you go along. You learn new things, you need to adjust how you allocate your time and resources. So be sure to change your strategy as you learn new information. Surprisingly, this is where a lot of teams and leaders run aground. Instead of adjusting, they plow the Titanic right into the iceberg.
What about what you discover could happen, but hasnât happened yet? Good question. You donât have to change up your entire strategy per se, but it helps to contemplate the decision you would make about your strategy if the new information you discover were to affect you in some way.
So, if you find out that all of your product instrumentation is wrong, how are you going to adjust where you spend your time and effort? Whatâs your strategy for it? Even writing a note about it and storing it away in your preferred notes app works. It encourages your mind to consider it, decide what to do about it, and know that answer so you donât look like a deer in headlights if (and when) it happens.
Thatâs it for now kids! Learning how to deal with unknowns and uncertainty is a critical skill. Youâll get better at it over time, keep attacking it and youâll be an asset to any organization you are a part of.
Donât be afraid to give me a shout if thereâs anything you want to know more about. See yâall soon!
đ
I hope you enjoyed this post! If you like what youâve read, please subscribe or forward this article to someone that might find it useful and Iâll do my best to keep cranking these out!
Create your profile
Only paid subscribers can comment on this post
Check your email
For your security, we need to re-authenticate you.
Click the link we sent to , or click here to sign in.