Experiment design doc · Sergey Sukharevsky

You need an experiment document before you start, when the team doesn’t know the outcome yet and can honestly define the criteria. Set down the causal hypothesis, the target group, the basic metric, the constraints and the action for each outcome; after you start, don’t change the rules discreetly.

What is Experiment Design Doc?

Definition and designation

Experiment design doc is a structured document that formulates a goal, hypothesis, success metric, detailed description of change, audience, launch plan and criteria for acceptance of the result, helps reduce risks, align expectations between teams and avoid typical errors.

When you need to experiment design doc

Such a document is required when:

A/B testing and other product discovery experiments
changes in functions where the outcome cannot be predicted intuitively
Distributed teams where it is important to synchronize expectations

Example: If you make a new onboarding page and want to check if it leads to an increase in activation, without a clear description of the test, the team will get lost in interpreting the data and choosing metrics.

The main blocks of experiment design doc

Classical template

Here is a simple, basic framework used by evidence-based product management companies (e.g., Booking.com or Amplitude):

Background/Context: A brief description of the problem or opportunity
Hypothesis: The hypothesis is formulated in the style of if…
Success metrics: How is success determined (main and secondary metrics)
Experiment design: details of changes (which changes, groups compared, traffic distribution options)
Target audience/segmentation: Who is running the test, restrictions
Plan and timeline: Startup steps and duration of the experiment
Risks/Assumptions: Possible limitations, risks and important assumptions
Analysis plan: How the analysis will be conducted and which statistical criterion is selected
Decision rules: What results are considered grounds for implementation or rollback

Example

Let’s say you want to test a new design of the payment button.

background problem (many unfinished transactions)
Hypothesis (a new color of the button will increase the completion by 5%)
Metrics (conversion to payment page as the main, CTR buttons as an additional)
Test design (multiple allocation of new users to 50% of traffic)
Critical Limitation (we only check on the mobile version)
Plan (lasting 2 weeks with at least 10,000 unique users)

Major Errors and Anti-Patterns

Fuzzy formulation of the hypothesis

Without a clear hypothesis, there’s confusion and debate about the results. You can’t just test a button for the sake of a test. You always need a specific formula: if you do X, Y changes to Z.

Unspecified metrics

If the result of an experiment is not tied to the measured values, you can’t make a decision, and remember to clearly state what is considered a success and in what form.

Insufficient size or incorrectly selected audience

A small sample size or a narrow segment makes the conclusions unreliable, and in practice, they often get the wrong results at this stage and get irrelevant statistics.

There are no clear criteria for decision-making

Without a definition of the decision rules, the spontaneous interpretation of the numbers begins, decisions are slowed down and the test is repeated several times, wasting time and data.

Checklist for quick creation of experiment design doc

Before the start of the experiment

Is the task clear? What’s the goal?
Is there a hypothesis (if…, then…)?
Are there any clear primary and secondary metrics?
What exactly is changing in the product?
So, what audience is the experiment going to?
Is there a statistical criterion for analysis?
Do you have a plan in case of technical failure?
Do you have a minimum amount of traffic and time?
Are there any risks and assumptions?

For example, you just need to use a Google Docs or Confluence text template with these items, without going into too much detail, so the project moves faster and doesn’t drown in bureaucracy.

Where to look at examples and best practices

Detailed structure and recommendations: Amplitude: Experiment Design Document Template
Work playbooks and e-commerce examples: Booking.com Experiment Playbook
About control groups and metrics: Optimizely Experimentation Guide

FAQ

Why do you need a design doc?

To capture the hypothesis, the success metric, the test details and the timing, it reduces confusion among team members and speeds up decision-making after the test.

What is the difference between an experiment design doc and a simple description of an A/B test?

Design doc is not only an A/B test diagram, but a complete document: with a hypothesis, justification, metrics, a launch plan and clear rules for evaluation.

Can I do multiple tests at the same time without design doc?

It can, but it will increase the chaos and increase the risk of incorrect conclusions, and if the team doesn’t know what’s being tested and why, the quality of the decisions will drop.

How to choose a metric for an experiment?

The metric should directly reflect the desired user behavior, the main thing is not to blur the goal and not mix metrics with proxy metrics.

What are the best tools for experiment design doc?

The most common ones are standard Google Docs, Confluence, Notion. For complex products, there are built-in templates in Amplitude Experiment, Optimizely, AB Tasty.

What if the results of the experiment are not certain?

Go back to design doc: check the hypothesis, the sample correctness, the sufficiency of the data, sometimes it’s worth repeating the experiment or adjusting the parameters.