Product Experimentation Excellence

The Guide

4 key steps synthesized from 4 experts.

1

Evaluate testing necessity and feasibility

Determine if the change warrants an experiment based on risk level and required sample size. If traffic is low or the change is an industry standard, consider shipping directly or using qualitative validation instead of a formal A/B test.

Featured guest perspectives

"If you can run experiments quickly and easily (e.g. a few hours), this decision is generally easy: run the experiment. If running experiments is a pain in the butt, and the changes are relatively benign, you can probably skip the experiment."

— Lenny Rachitsky

"It turns out you’d need *over 60,000 unique users* (per variation!) before you could draw a confident conclusion. That’s 120,000 users going through the flow before you can move on. For most startups, that ends up being far too long."

— Lenny Rachitsky

2

Design the OEC and guardrail metrics

Create an Overall Evaluation Criterion (OEC) that balances your primary success metric with long-term value indicators. Include guardrails to monitor for negative impacts on performance, latency, or user complaints.

Featured guest perspectives

"It's very easy to increase revenue by doing theatrics. Displaying more ads is a trivial way to raise revenue, but it hurts the user experience. And we've done the experiments to show that. In this case, this was just a home run that improved revenue, didn't significantly hurt the guardrail metrics."

— Ronny Kohavi

3

Execute with high velocity and statistical rigour

Maintain a high cadence of low-effort experiments to maximize learning volume. Use automated checks for Sample Ratio Mismatch and apply techniques like CUPED to reduce variance and reach significance faster.

Featured guest perspectives

"I think we did well but lost our scrappy, iterative mindset too early. Even without a ton of traffic -- we were wasting traffic not running more tests."

— Lenny Rachitsky

4

Analyze results and capture institutional learnings

Push past surface-level numbers to understand the why behind the results through cohort analysis and qualitative follow-ups. Document every result, including failures, to build a predictive model of your users over time.

Featured guest perspectives

"And it was sort of done, it was very beneficial, and then it was semi forgotten, which is one of the things you learned about institutional memories. When you have winners, make sure to address them and remember them."

— Ronny Kohavi

Get this guide as an AI skill for Claude Code

Install This Skill

Add this skill to Claude Code, Cursor, or any AI coding assistant that supports Agent Skills.

Quick Install (Recommended)

Install this skill directly using npx:

npx skills add RefoundAI/lenny-skills --skill product-experiments

Or install all 76 skills:

npx skills add RefoundAI/lenny-skills

View on GitHub →

Manual Installation

1

Download the skill

Download Skill (.zip)

2

Add to your project

Create a folder in your project root and add the skill file:

.claude/skills/product-experiments/SKILL.md

3

Start using it

Claude will automatically detect and use the skill when relevant. You can also invoke it directly:

Help me with product experimentation excellence

Guest Perspectives

Deep dive into what 3 podcast guests shared about product experimentation excellence.

Archie Abrams 1 quote

Listen to episode →

"So we constantly will relook at an experiment a year later, see that the way the GMV curve for the distribution was different than we might've originally thought. And that'll actually change what we do from that previous experiment. And so there's a lot of longterm monitoring of experiments over these very long time horizons to both inform what those input metrics are and more importantly hold ourselves accountable to, did we actually move what we cared about, which is that longterm GMV, in the right way?"

Tactical:

Implement long-term holdouts to track the downstream business impact of experiments one, two, and three years later.
Regularly revisit "winning" experiments to see if initial conversion lifts translated into incremental long-term value.
Use long-term monitoring to refine short-term input metrics and ensure they are accurate leading indicators of success.

View all skills from Archie Abrams →

Lauryn Isford 1 quote

Listen to episode →

"So, with all that said, generally my advice is to experiment when you need to and to primarily see it as a risk mitigation tactic when you're making dramatic changes and to let the product development process do more work. So, spend more time with customers, be more rigorous in understanding precisely what problem you're solving, get mocks in front of people and see how they react, and hopefully have more conviction than you otherwise would when you ship something that it's okay if every customer sees it tomorrow and that the experiment doesn't actually matter as much."

Tactical:

Use A/B tests primarily for risk mitigation on dramatic product changes.
Invest more in customer research and mock testing to build conviction before shipping.
Avoid over-relying on experiments just to provide precise numbers for performance reviews.

View all skills from Lauryn Isford →

Ronny Kohavi 4 quotes

Listen to episode →

"It's very easy to increase revenue by doing theatrics. Displaying more ads is a trivial way to raise revenue, but it hurts the user experience. And we've done the experiments to show that. In this case, this was just a home run that improved revenue, didn't significantly hurt the guardrail metrics."

Tactical:

Include guardrail metrics to ensure revenue gains don't degrade the user experience
Develop an OEC that incorporates long-term user value and retention
Avoid optimizing for single metrics that can be easily manipulated through poor UX

"At Bing, which is a much more optimized domain after we've been optimizing it for a while, the failure rate was around 85%. So it's harder to improve something that you've been optimizing for a while. And then at Airbnb, this 92% number is the highest failure rate that I've observed."

Tactical:

Expect failure rates between 80% and 92% in highly optimized product domains
Calibrate team expectations by sharing industry-standard experiment success rates
Treat high failure rates as a signal of a rigorous and mature experimentation program

"And it was sort of done, it was very beneficial, and then it was semi forgotten, which is one of the things you learned about institutional memories. When you have winners, make sure to address them and remember them."

Tactical:

Address and document every experiment winner to maintain institutional memory
Reintroduce proven successful patterns to new teams and product areas
Create a searchable repository of experiment results to avoid re-learning old lessons

"We can talk later about Wyman's law, but that was the first reaction, which is, 'This is too good to be true. Let's find a bug.' And we did. And we looked for several times, and we replicated the experiment several times, and there was nothing wrong with it."

Tactical:

Treat any suspiciously large positive result with immediate skepticism
Replicate surprising results multiple times to confirm they are not flukes
Check for data logging errors like double-counting when revenue spikes unexpectedly

View all skills from Ronny Kohavi →