Hello! This is the Teach Me How 2 Product mailing list. Welcome! I’ve been working on the site and content that I’ve found useful in my career so far. I hope it makes you think or at least is entertaining. Feedback is welcome and, if you’re new here, please subscribe!

Here’s what’s been on my mind this week:

The things that should keep you up at night are the things you don’t know.

A stress dream that I continue to have to this day is not actually graduating with my engineering degree. I don’t really get a flop sweat when I have it. Usually, it involves a lot of tossing and turning at night when the same video plays over and over in my head.

I get a call from an Austin number. It’s the engineering school, they’re calling to tell me that there are a few credits that I actually missed as part of my degree plan. I freak out and then find myself at campus trying to muscle out some C’s to graduate all over again. It’s a terrible, sinking feeling every time.

I tend to get that sinking feeling when I come across something I didn’t know about. Sometimes it’s a wonderful surprise, so that feeling doesn’t last. But in our business, it tends to be more of the not-so-wonderful variety. 

Thus…sinking feeling.

You might know some things…

It's not completely hopeless. There probably is already some stuff that you know about your product and your business.

Let’s say you’ve started a job at a new company working on a new product. There are a few things that you have been told about your product during onboarding. Maybe some baseline conversion rates, how certain features work. There is information that you have discovered as you have gone through your onboarding process. You understand things about your architecture. Your peers and leadership team have helped you understand your current strategy perhaps. You might have also learned some things from previous products you have worked on that you anticipate using on your new product.

It’s tempting to take it all at face value and run ahead with cranking out new features, following blindly what you’ve learned so far. But, it’s critical that you understand that what you know vs. what you don’t know likely looks like this:

…but you certainly don’t know a lot of critical stuff.

In an effort to help you accept the inevitable, that there is a ton of ice under the surface that you don’t know yet, here are some examples of things yours truly has run across that I didn’t know that really bit me in the keister:

• Early in my career, we shipped a new feature in an app to help a property owner quote a traveler on how much their vacation rental would cost to rent. The product passed QA, we all celebrated. In a user interview with a property owner, I discovered that the method we had been using to calculate taxes was old. We were calculating taxes with old math for lots of places people lived. Doh!

• When I first set up an A/B testing program, I took on face value a lot of techniques and technology we were using for calling experiments. When the dashboard lit up green, the test was called, right? WRONG. Turns out there’s all this pesky stuff about statistical significance, Bayesian vs. Frequentist experimentation methods, test exposure, etc. that’s really important. And by really important, I mean that it can be the difference between actual product development progress and rolling out a bunch of crap that tanks all your numbers. So, I learned it all THE FUN WAY. 🥳

• We were in the middle of a huge re-platforming of a critical page in the shopping experience. In the middle of that effort, we discovered that our instrumentation for a critical metric we were tracking was completely different in the new platform vs. the old platform. The numbers we were using to track progress turned out to be completely wrong. 🤦🏻‍♂️

• FRACKING COVID. We had no idea how a global pandemic would affect the vacation rental sector, much less the travel sector. Nine months later, we’ve seen how things have shaken out. But in March, it was a complete unknown. We started 2020 with a completely different set of plans. Those were destined to change.

Freaked out yet? You should be! The iceberg is lurking out there, just waiting for you to run into it. So, what to do?

Illuminating the iceberg: how to know what you don’t know

It’s probably a good idea to start understanding what lurks below the surface. In my experience, there are a few key steps that are required to do this.

Accept it

This is probably the most important thing to do. You have to accept that this part of the iceberg even exists! There are WAY too many people walking around thinking that they have figured everything out. They haven’t, I promise. It’s impossible to know everything. You are on a journey of discovery and learning. whether you acknowledge and accept that you are or not.

It’s much better if you accept it, I promise.

Be quick to respond, triage, and troubleshoot

If you’ve come to accept that there is a lot that you don’t know, that will help when the inevitable happens: your ship hits the part of the iceberg under the surface. When that happens (not if!) it’s important to have the mindset of being quick to respond when things come up that you don’t know or are surprised about.

Don’t freeze up, don’t be the deer in headlights. Start training your mind to think like first responders: run towards the burning building instead of away from it. Analyze the situation, see where there are structural issues. Start thinking about what the priority issues are to address and troubleshoot them.

This is the behavior that got us out of the COVID-19 travel ditch. It was certainly a shock when lockdowns started happening and travel demand took a nosedive. Rather than panicking and watching things fall around us, we got to work. We re-focus our efforts on improving the core product and business, triaged bumps big and small as they came in, and worked quickly to address them. As a result, we survived.

There will be time later to reflect on the event and experience the emotions. You want to survive first so you can live to reflect.

Okay, we’ve talked about what to do when something you don’t know jumps out and bites you. How do you try to understand what you don’t know so you’re not surprised when stuff comes up?

Question assumptions

In everything that’s shared with you in onboarding, by your teams, by industry experts, there is inherently a set of assumptions that everyone is making. Those assumptions could be about how your business works, what your users are like, how your team works to ship product, etc.

It’s important to identify those assumptions, question them, and learn more about them to understand how what you’ve learned does or does not apply to your situation. You don’t have to be mean about it! Just know that they are there and that, without questioning them, they could help you run into the iceberg. This can include things like assumptions about your instrumentation strategy, how you call A/B tests, why success metrics were chosen, why the company strategy is what it is, etc. Don’t forget, lots of captains have assumed there was nothing beyond the tip of the iceberg.

Identifying assumptions can also lead to opportunity. For example, maybe you have a search product and you have to rank results in search. The ranking algorithm that everyone believes is good might not be. You could research it, find the gaps in it, and voila! You have a ranking experiment that could demonstrate positive results for your business and customers.

Research, preferably with actual users

Do your homework! Sounds like school when you were a kid, right? Well, as adults too this applies. You’d be surprised how many people don’t follow this advice when doing important things like putting together a product strategy.

Research helps you understand assumptions you’re making, where there are gaps, how the market is going to respond to what you’re doing, what your customers are actually going to do when they get their hands on your new feature or design, etc. It generally gives you the opportunity to either have confidence in your approach or have fact-based reasons for why you’re experimenting with a change.

Research can be competitive research, that can help you understand the landscape. Industry research is good too, that gives you a sense of what’s new in your field and what customers have come to expect. My personal favorite kind of research is taking your product out to actual customers. People are unpredictable, your brilliant idea could crumble in the face of talking with actual customers. This happens a lot, and is the reason why you should do it! It helps you understand what you don’t know about your customers, stops you from building useless stuff that no one wants, and sparks new ideas from things your customers say they want from you. (This has happened to me. One user session a few years ago had feedback direct from customers that shaped our multi-year strategy. Super useful!)

Analyze with data

Data is your best friend when you’re out there busting up assumptions and trying to illuminate the iceberg. It can mean the difference between freaking out over something that doesn’t matter or identifying a potential problem that saves you days of troubleshooting and scrambling.

When in doubt, instrument everything that you can think of. It will come in handy when potential problems arise. Make friends with an analyst that can show you how to pull your own data. If you’re lucky enough to have access to self-service tools like Mixpanel or Amplitude, learn all of the ins and out and use them every day.

A lot of good product people excel in either user research or data analysis. The great PMs learn how to do both really well. Be great.

Test hypotheses

If you have read back on other pages of this blog, you’ve probably heard me talk about A/B testing a good amount. Why am I always blabbering on about that? Because it’s important!

It’s one of my favorite tools in the tool belt because it allows me to test hypotheses that break up assumptions and helps me understand what I don’t know scientifically with empirical data. I highly recommend it as a way to understand unknowns. It’s very similar to user research in that it surprises you and your team, regularly. Customers are different than you are, if you think something is great, half the time your customers don’t.

A/B testing also has some added benefits:

• It helps you settle team disputes and demonstrate that a new feature has value with actual customer data

• It prevents you from rolling out new things that have a negative value. Often, the most important experiments are the ones that you DON’T roll out. These are things that your customers would have had to suffer that you prevented with an A/B test. Good for you!

Testing doesn’t have to be limited to A/B testing. You can give a new feature to 20% of your customers, or send an email to 10% of your email subscribers. The key idea is using experiments to understand unknowns and take educated risks. Over time, this prevents unknowns from catching you off-guard and enables success.

Be a sponge, absorb everything

Early in my career, I’d get frustrated with some meeting invites that would get put on my calendar. I’d always ask why these things were useful. I’m just going to be sitting on my laptop not really paying attention, why am I going?

Of course, everyone was smarter than I was. There were definitely some meetings that were useless sure. But there were a lot of meetings where I learned new information about how the business, company, or product worked. All by just sitting in the meeting for a bit, or reading a document someone sent me about something they’re working on.

When in doubt, be a sponge. Absorb everything, read what people send you, and generally be curious. You never know, something you learn might be the thing that saves your product.

Adjust your strategy with new information

The final piece is being flexible about your strategy, and having a decision tree for when possible events happen.

When you discuss a strategy with your team, it’s important that there is an understanding that your strategy can, and should, change as you go along. You learn new things, you need to adjust how you allocate your time and resources. So be sure to change your strategy as you learn new information. Surprisingly, this is where a lot of teams and leaders run aground. Instead of adjusting, they plow the Titanic right into the iceberg.

What about what you discover could happen, but hasn’t happened yet? Good question. You don’t have to change up your entire strategy per se, but it helps to contemplate the decision you would make about your strategy if the new information you discover were to affect you in some way.

So, if you find out that all of your product instrumentation is wrong, how are you going to adjust where you spend your time and effort? What’s your strategy for it? Even writing a note about it and storing it away in your preferred notes app works. It encourages your mind to consider it, decide what to do about it, and know that answer so you don’t look like a deer in headlights if (and when) it happens.

That’s it for now kids! Learning how to deal with unknowns and uncertainty is a critical skill. You’ll get better at it over time, keep attacking it and you’ll be an asset to any organization you are a part of.

Don’t be afraid to give me a shout if there’s anything you want to know more about. See y’all soon!

👋

I hope you enjoyed this post! If you like what you’ve read, please subscribe or forward this article to someone that might find it useful and I’ll do my best to keep cranking these out!