Ronny Kohavi

For experimentation infrastructure, a hybrid 'build and buy' approach is often best, starting with third-party tools to establish trust.

"I think the key decision is whether you want to build or buy. ... It's usually not a zero one, it's usually both. You build some and you buy some, and it's a question of do you build 10% or do you bui..."

Maintain a searchable database of all experiment results to build institutional memory and prevent repeating failed ideas.

"I've actually put a lot of effort in this idea of institutional learning, of doing the quarterly meeting of the most surprising experiments. ... Document it. We had a large deck internally of these su..."

Replace slide decks with structured narrative documents (like Amazon's 6-pagers) to facilitate deeper reviews and honest feedback.

"I took that to heart. And many of the features that the team presented instead of a PowerPoint, you start off with a structured document that tells you what you need, the questions you need to answer..."

To build an experiment-driven culture, start with high-velocity teams as a 'beachhead' to demonstrate value before scaling to slower-moving groups.

"Find a place, find a team where experimentation is easy to run. And by that, I mean they're launching often, right? Don't go with the team that launches every six months... Go with the team that launc..."

Treat high-risk product bets as a portfolio where an 80% failure rate is expected and acceptable.

"You have to allocate sometimes to these high risk, high reward ideas. We're going to try something that's most likely to fail. But if it does win, it's going to be a home run. And you have to be ready..."

Quantify the long-term cost of negative user actions to make objective trade-offs against short-term revenue gains.

"What's that cost? Well, when they unsubscribe, we can't mail them. So we did some data science study on the side and we said, 'What is the value that we're losing from an unsubscribe?' And we came up..."

Focus retrospectives on 'surprising' results—where the delta between prediction and reality is largest—to maximize organizational learning.

"To me, a surprising experiment is one where the estimated result beforehand and the actual result differ by a lot. ... So we focused not just on the winners, but also surprising losers, things that pe..."

Use an Overall Evaluation Criterion (OEC) that balances primary goals with guardrail metrics to ensure long-term health rather than short-term gains.

"The OEC or the overall evaluation criterion is something that I think many people that start to dabble in A/B testing miss. And the question is, what are you optimizing for? And it's a much harder que..."

Decompose large redesigns into 'One Factor At a Time' (OFAT) increments to avoid the risk of a single massive failure.

"Try to decompose your redesign if you can't decompose it to one factor at a time, to a small set of factors at a time. And learn from these smaller changes what works and what doesn't. ... Do them in..."