How to create a product experiment backlog requires a disciplined hypothesis format, a scoring system that accounts for learning value not just expected impact, and a process that prevents the backlog from accumulating experiments that never run.
Most product experiment backlogs fail in one of two ways: they are either wish lists of features disguised as experiments, or they are well-intentioned but never triaged, growing to hundreds of entries that no one ever prioritizes.
A functional experiment backlog is a living, curated queue — not an archive.
What Belongs in an Experiment Backlog
An experiment backlog is not a feature backlog. The distinction matters:
| Feature backlog item | Experiment backlog item | |---------------------|------------------------| | "Add social proof to checkout" | "Adding social proof to checkout will increase checkout completion by 8% because new users lack trust signals" | | "Improve onboarding flow" | "Reducing onboarding from 6 steps to 3 will increase 7-day activation by 15% because users abandon before reaching the value moment" | | "Send re-engagement email" | "Sending a personalized re-engagement email at day 14 will recover 12% of churned users because prior data shows this cohort still has unmet intent" |
Every experiment must have a hypothesis that specifies: the change, the expected outcome, the magnitude, and the reason it should work.
H3: The Hypothesis Format
Use this structure for every experiment:
"If we [change], then [metric] will [increase/decrease] by [magnitude] because [rationale based on evidence]."
The "because" clause is the most important part. It is what distinguishes an experiment from a guess. The rationale should be based on: user research, behavioral data, industry benchmarks, or first-principles reasoning about user psychology.
According to Lenny Rachitsky's writing on A/B testing, experiments without a clear rationale produce learning but not insight — you know what happened but not why, which means you can't generalize the learning to other decisions.
Scoring Experiments for Prioritization
H3: The ICE Score
The standard experiment scoring framework:
- Impact (1–10): How much could this move the metric if it works?
- Confidence (1–10): How sure are we that it will work, based on evidence?
- Ease (1–10): How easy is this to implement and run?
ICE score = (Impact + Confidence + Ease) / 3
H3: Adding Learning Value
ICE misses an important dimension for experiment backlogs: Learning Value — how much would this experiment teach us even if the hypothesis is wrong?
Add a Learning Value score (1–10) and weight it alongside ICE:
Priority = ((Impact × Confidence) / Effort) + (Learning Value × 0.5)
Experiments with high learning value should be prioritized even when expected impact is moderate, because the insight compounds across future decisions.
H3: Sample Scoring Table
| Experiment | Impact | Confidence | Ease | Learning Value | Priority Score | |-----------|--------|-----------|------|---------------|---------------| | Social proof on checkout | 7 | 6 | 8 | 4 | 7.0 | | Reduced onboarding steps | 8 | 7 | 5 | 9 | 9.1 | | Re-engagement email day 14 | 6 | 8 | 9 | 3 | 7.7 |
Building and Maintaining the Backlog
H3: Hypothesis Generation Sources
Experiments should come from four sources:
- User research: Interviews and usability tests that surface friction or confusion
- Behavioral data: Funnel drop-off analysis, heatmaps, session recordings
- Customer support: Common themes in support tickets represent product friction
- Team submissions: Engineering, design, and CS should have a channel to submit experiment hypotheses
A healthy backlog has diverse sources. A backlog that only contains PM-generated ideas has a blind spot problem.
H3: Backlog Hygiene
The most common experiment backlog failure is experiments that sit untouched for 6+ months. Countermeasures:
- Cap the backlog: Maximum 20–30 active experiments. Everything beyond that is archived, not queued.
- Monthly triage: Remove experiments with low scores that haven't been promoted in 2+ months. If they haven't risen to the top in 2 months, they won't.
- Rescore stale entries: An experiment added 6 months ago may have lower confidence now that market conditions have changed.
According to Shreyas Doshi on Lenny's Podcast, the sign of a healthy experimentation culture is not how many experiments are in the backlog — it is how many experiments are running per week. A backlog is only valuable if it produces running experiments at a cadence that generates learning.
H3: Sample Backlog Template
Each backlog entry should contain:
- ID: Unique identifier for tracking
- Hypothesis: If/then/because format
- Metric: Primary metric and secondary guardrail metrics
- ICE scores: Impact, Confidence, Ease individually scored
- Learning Value: 1–10 score with rationale
- Status: Hypothesis / In Design / Running / Complete / Archived
- Owner: PM responsible for moving this forward
- Date added / Date last updated
FAQ
Q: What is a product experiment backlog? A: A curated, prioritized queue of product hypotheses — each in if/then/because format — that a team plans to test to improve a specific metric, scored for impact, confidence, ease, and learning value.
Q: How do you format a product experiment hypothesis? A: "If we [change], then [metric] will [increase/decrease] by [magnitude] because [rationale based on evidence]." The because clause is the most important part — it separates an experiment from a guess.
Q: How do you prioritize experiments in a product backlog? A: Use ICE scoring (Impact, Confidence, Ease) supplemented with Learning Value. Priority equals (Impact times Confidence divided by Effort) plus (Learning Value times 0.5). Experiments with high learning value should run even when expected impact is moderate.
Q: How do you prevent an experiment backlog from becoming a graveyard? A: Cap the backlog at 20 to 30 active entries, triage monthly to remove stale experiments, and rescore entries older than 3 months. A backlog is only valuable if it produces running experiments at a weekly cadence.
Q: Where should experiment hypotheses come from? A: User research, behavioral data, customer support ticket themes, and team submissions from engineering, design, and CS. Backlogs with only PM-generated ideas have blind spots.
HowTo: Create a Product Experiment Backlog
- Define the hypothesis format every experiment must follow: if we make this change, then this metric will move by this magnitude, because this rationale based on evidence
- Build the backlog from four sources: user research findings, behavioral data analysis, support ticket themes, and team submissions with a dedicated intake channel
- Score every experiment on Impact, Confidence, and Ease individually, then add a Learning Value score to capture experiments that will generate insight even if the hypothesis is wrong
- Set a backlog cap of 20 to 30 active experiments and archive anything that hasn't been promoted to running status in 2 months
- Run a monthly triage to remove stale entries, rescore experiments older than 3 months, and reorder the queue based on updated scores
- Track running experiments per week as the primary health metric of your experimentation culture — a backlog that produces no running experiments is an archive, not a queue