Planning Under Uncertainty

"I think about wartime product management, right? You're coming in, and I think there was... this incredible humility that was needed to really understand and first diagnose what was actually happening on the platform."

"These are core product skills that we look for in terms of leadership and grit, and the ability to drive through really, really tough problems that there's no playbook for, nobody has ever really done before."

"no credit will be given for predicting rain, only credit for building an ark... You have to build the ark. It doesn't matter if you predict you're going to fail, you've still failed. It gets you nothing. So what you have to do is figure your way out of it and spend all your time on that."

"If you're a late-stage startup... you can afford the luxury to place multiple bets and spread your chips and wait it out a little bit to see who the winner is... The key question for startups is totally different. You don't have the luxury to spread your chips. You have to go all in. You have to choose one and go all in."

"If you're not going to get significance, if there's no other techniques at your disposal, then sometimes you just got to trust your intuition and ship it. And if that's what you believe, then that's what you believed and you shouldn't spend time trying to get false precision."

"One of my favorite takeaways from that is that from this way of working around this dual track agile de-risking your riskiest ideas first approach is a concept of taking the things in the top, the biggest swing and actually prioritizing those first in terms of product discovery and figuring out what can you do to start de-risking because if you constantly put those off in favor of the lower risk or more predictable smaller swings, how are you ever going to truly innovate and get to the next level."

"Andy, he always talks about slugging average, not batting average. He's like, 'I don't care if you hit the ball every time. If one in 10 times you hit a home run that's better than someone who hits it every three out of 10 times but gets out a lot.'"

"Anxiety equals uncertainty times powerlessness... 98% of anxiety comes from two sources. One is what you don't know, and number two is what you can't control or influence."

"Even if you have a sample size of 30, the data you get back, generally, does not change but its precision will. So mathematically speaking, you're going to get the same level of trends, but the precision at which you understand those trends will become more deep if you have more data. But the underlying information that you're getting out of that won't be very different at larger scales."

"You don't leave a session with me without having tried a little experiment first. The analogy there is we would call it an in the room experiment versus then get out of the building and do an experiment... anything you think is true or you want to do, it's a hypothesis until you test it. And you go out, get data, and then you can do a bigger version."

"If every single one of your initiatives that you're doing on growth is an experiment, that's a problem... It's almost like a disease, like a paralyzing disease, that slows down progress... I think that people should trust their intuition a little bit more."

"When you do this top-down, it's a TAM forecast essentially. When you do a top-down, you're closer to the truth. Now, you probably aren't going to get to the truth... but it will help you make a better decision than if you just guess."

"Take 100 different questions, half of them I will intervene, half of them I won't... you definitely want a control group, especially in Answer Engine Optimization."

"The first part of limiting beliefs, write them down, understand what they are, look at them in the cold day of light on paper, and then translate them into things that are just obstacles to be overcome. So, 'How would I fund this,' just becomes a plan, like, 'I need to design a plan where I'd get funding for this charity.' And then that just is a problem like any other problem. It's not this nebulous, scary fear. It's just literally a to-do item."

"There are going to be two types of feedback. One is you did something and it was right, but people are upset because you changed stuff. The other is you did something and it wasn't right, and people are also upset but for good reasons. And so how do you separate these two?"

"I created a tool called the confidence meter... It goes from very low confidence which is the blue area... all the way to high confidence which is the red area and you can see the numbers going from zero to 10. Where zero is very low confidence, we don't know basically anything we're just guessing in the dark and 10 is full confidence."

"You're saying that you want a quarter million dollars worth of investment, and you're going to spend it on your team who's going to run experiments... Some of these experiments are going to fail and some are going to succeed. You don't know which ones. But that's okay, you know that by the end of the quarter, enough are going to succeed that you're probably going to move the right numbers in the right direction."

"Survival is just part of the game, and most people just give up before they get their timing right... Survival is a precursor to that. So let's not put ourselves in position that could potentially compromise the enterprise along the way. It doesn't mean don't take risks, but think about how you calculate it."

"how do to be clear about the phase that you're in? ... we are explicitly going to go learn these types of things... you just have to be very, very clear with your team on what phase you're in. 'Hey, we're in the learning phase and we explicitly are trying to learn these things' versus, 'Hey, we have this really big vision and we're just going to go at it.'"

"He just stops and he's like, "You know what? Just test the extremes. Start the experiment tomorrow. We'll figure it out." Essentially. And I think his point was like, look, we can debate this forever. So I would rather us see the upper and lower bounds of how good this could be or how bad this is going to be immediately."

"Roughly 80% of the times, ORs in the time are hypotheses and the things that we believe will be true... The closer you get to something that you go bear your head in the sand or go into an attic and build something for six months and ship it, the more likely it is that you are going to ship the 80% wrong stuff."

"Generally my advice is to experiment when you need to and to primarily see it as a risk mitigation tactic when you're making dramatic changes and to let the product development process do more work. So, spend more time with customers, be more rigorous in understanding precisely what problem you're solving, get mocks in front of people and see how they react, and hopefully have more conviction than you otherwise would when you ship something that it's okay if every customer sees it tomorrow and that the experiment doesn't actually matter as much."

"Experiments are great, but they can be slow... the subtlety is that experiments are great, but they can be slow... sometimes you just need to YOLO it because it's a better product experience or you just kind of know it's going to work. And if you're YOLO-ing 40 things and three of them work and you can look at pre-post... the speed can outweigh the cost and time it takes to do experiment."

"The moment you build experimentation, you've now made it scientific. Now, somebody comes up with an idea, say, that's a bad idea. Here, this is why it's a bad idea, because we have done this experiment six times and it has failed across this user groups at this exact level of impact created."

"Any coin that says can't lose bad on one side of it might as well say, can't win big on the other side. It's your willingness to fail that lets you have breakthrough success."

"One thing is probably just really fast iteration cycles. So placing a lot of bets and then being really rigorous about just going through that cycle very soon. I have this... principles, and one of the things on there is basically saying EOP successes goes up the more bets you make, and it's sort of a function of how many bets you make and the probability of success."

"Draw four boxes on a piece of paper... the first two to the left of them write the word words. And below them, write the word data... always start with words. You do not start with data. You always start with words. And then you'll go around to a couple of people, stakeholders, managers, others, and you'll say, 'Do you agree with this? Is this actually what we're doing?'"

"Develop a reproducible testing process, and that will actually influence the probability of your success more than anything. It's so unpredictable whether a consumer product idea will work."

"If this is true, then what next needs to be true for this thing to work out? And these layers of conditional statements. And the more layers you have, the higher risk your product is, so you should try to condense it to about like four things that must be true for the thing to work."

"Experimentation was never historically in science about winners and losers... Experimentation is always very hypothesis driven. It's about, what are you learning? And that's really an important distinction because what it means is if I go with something big, risky, and it, 'fails,' meaning that doesn't win. Nevertheless, if I was being rigorous about what hypotheses that's testing about my business, I'm potentially learning a lot."

"There's ways to take the past into account, to build what's called a prior belief before I run an experiment, and now take the data from the experiment, connect it with the prior, to come up with a conclusion... that falls broadly under the category of what's called Bayesian A/B testing."

"What I find so interesting about experiments is that when you don't know something, it seems not even a question that you would allocate some of your samples to all options... After the fact you're like, 'Treatment was better. What the heck were we thinking? Why'd we give all those samples to control?' ... you have to put yourself in the frame of reference of when you didn't have the answer. And at that moment, what you're essentially saying to yourself is that it's worth paying to learn the answer."

"At a startup, you can't do that. You just don't have those users to test with. And I think a lot of startups make the mistake of trying to use an experimental approach too early... I've had to shift my mindset from an experimental-oriented approach to making decisions to much more of a conviction-oriented approach."

"You have to allocate sometimes to these high risk, high reward ideas. We're going to try something that's most likely to fail. But if it does win, it's going to be a home run. And you have to be ready to understand and agree that most will fail. ... If you go for something big, try it out, but be ready to fail 80% of the time."

"So the way I think about setting up a plan when you have limited visibility and some major headwinds is setting up a really conservative plan and then having milestones, short term milestones that unlock the ability to lean into growth and spend based on hitting those targets."

"Data is more like a compass than a GPS. If you look at data as a way of giving you the answer, you're always wrong. You're always wrong or you're slow. Wrong or slow or sometimes both, because mostly data doesn't give you the answer. It just tells you if what you just said is ridiculous or there's potentially something there."

"If you're making a decision with less than 30% of the available data, you're making a big mistake. If you're making a decision only after you have 70%... you have waited far too long."

"failure is not learning. So it's really important that you learn when you fail. And so we celebrate failure as long as you're learning and you can only learn if you've designed the right test and you failed conclusively"

"The first thing is we have to learn how to take an idea and break it into its underlying assumptions. We have to learn how to prioritize those assumptions. Then we have to learn how to run tests that are small enough that they're just testing that assumption."

"One really, really big important lesson that I learned at Snap is about risk taking. And when you have the financial support and the foundational relationship with your investors that Evan has, it really allowed him to take these really big swings, acquire a technology that he thought was game changing, build features speculatively."

"A/B testing... that's the highest bar. It proves with near absolute certainty that there's a causal relationship... But I think that it maybe misses the point in some changes or some areas where you are working towards a bigger net new thing or this specific change won't really be indicative of the greater whole you're building towards."

"I carved out two million members and I said, 'Those are my members. I'm going to focus on building that mountain peak. I'm going to build for them.' Full liberty and doing whatever, it doesn't hurt numbers, giving the scale and really focus on building a great experience for them."

"We always have to have the ability to A, pivot of course, but also have backup and buffers in those types of scenarios. So any time we're planning we build in buffers for all of that chaos that's happening on a daily basis."

"Stay firm on the goal, but flexible on the process."

"If you fail fast, you still have plenty of time to try another attempt and build another version of the product. Try another go-to-market approach. Try a different business model so you still have plenty of time to make more and more and more attempts, and the more attempts that you have, you simply increase the likelihood of being successful."

"The only thing that's worse than a channel or a tactic that you tried not working. The only thing that's worse now is when you didn't give it the appropriate shot, right? And you prematurely were erroneously concluded that it doesn't work and it's remarkable how often you find that to be the case when I talk to companies, "Oh, YouTube, we tried it. It doesn't work." I'm like, "Okay, can I see what you've tried?" And then you look at it and you're like, "Oh, this thing was not designed to even have a shot at working from the get go.""

"I think with some tactics and some channels you can fairly objectively create some test guard rails where it's like, if it's YouTube, we know kind of minimum number of impressions that you got to get. Try two to three creative angles. Here's the click through rates range that you're looking for. If you get within these ranges on these KPIs, keep going. If you don't, abandon."

"Four life lessons... three, focus on what you can control. And four, for those things never give up."

"From an operating principle, we went into two week planning mode. Greg Greeley, who was the president of Airbnb used to say, 'Look, can't plan for a year, can't plan for a quarter. We're going to plan every two weeks. We're going to react to every two weeks.'"

"How do you build conviction in a highly noisy world? ... The thing that I would take with me everywhere is, how do you build in that noise? How do you stay focused and still build what you believe is the right thing and still let the noise happen around you?"

"Resourcefulness brings you further than resources... If resources are carbs, resourcefulness is like muscle. It stays with you. It makes you stronger, and it helps you have a better intuition and better performance over time."

"What you want to do is that you want to be the first one to hit the brick wall... speed is something that you should accelerate for the organization... can you be the first one to hit the brick wall where you have the learning faster than anyone else in the market so that you can decide, 'Oh my god, the path that I was going was not the right path.'"