Experiments: A/A, A/B, power

A/B test answers a narrow causal question with correct randomization, rather than confirming the value of the idea at all. Before launching, record the minimum significant effect, basic metric, guardrails, randomization unit and stop rule; after that, check the quality of the data and practical, not only statistical, significance.

Block: Metrics and Analytics

Experiments are a tool for making decisions on data, not just using resources to guess, but testing a hypothesis on real users, and always keep three things in mind: Are groups different before the start (A/A), is your experiment really reliable (A/B), and can you catch the desired effect (the power of the test).

Key terms

A/A-test
A/B test
Test power
Minimum Distinct Effect (MDE)
Significance (p-value)
Confidentiality (Confidence Level)
Sample size
Conversion metrics
Control group
The experimental group

Basic definitions

What are A/A and A/B tests?

A/B test is a way of comparing two versions of a product, feature or process: “A” (control) and “B” (new variant). Two sets of users randomly get different versions. Over time, you look at the difference in your target metric, like conversion, and decide whether it’s a new functionality or a randomness. The A/A test is a special case where both groups get exactly the same product, and it doesn’t validate the hypothesis, it tests the system of experiments: is everything random, is the metrics counted correctly.

Example:

Let’s say you want to test a new button design. In A/B, you roll out the old and new buttons for 50 percent of the visitors, you compare conversions. Before that, you run an A/A test, and 100 percent of users see the old design, they fall into different groups, and if the system reports that the conversion has diverged a lot, something went wrong: the user-sharing scheme works with an error.

Why is it important to check the power

Power is the probability that the test will catch the real effect, if it is, if the power is low, even a good experiment risks not detecting anything. Power depends on sample size, minimum discernible effect (MDE), data scatter, and significance level.

Example:

If you want to know if a new onboarding page increases conversion from 10 percent to 10.5 percent, if you have few participants, that difference will get lost in the noise, and you can calculate the sample size you want in advance with a power calculator.

How to run: step-by-step logic

A/A-test: system testing

Do a test with two identical groups to make sure:

randomization works
no system bugs
There is no unexpected scattering in the data If the result shows a significant difference, look for distribution failures, bugs, unaccounted variables.

A/B Test: Creating a Reliable Experiment

Create a hypothesis and choose the metric you expect to change. Determine the minimally distinguishable effect (what difference matters to you). The sample size and power were calculated using Statsig or Google Sample Size Calculator. Conduct a uniform random distribution of participants. Monitor the progress of the test, do not look at intermediate numbers too early (avoiding the error of overtraining). Analyze only after the experiment is completed, otherwise you risk making mistakes in conclusions.

Example:

Fintech products often test landing pages, change the offer, color CTA, drive traffic on two versions, before the A/B test is launched, you must perform an A/A to make sure that the A1 and A2 groups do not differ in key metrics (age, country, card holding) even before the content changes.

Typical errors and anti-patterns

Misinterpretation of the results

Make no mistake: p-value and eye-to-eye differences are not the same thing. Small differences are random when power is not enough.

Insufficient sample size

I rushed to finish the test, which is a high risk of getting a false effect, and p-value doesn’t mean anything if the test is short and the groups are small.

Example:

In a small business campaign, 300 users are testing a new cart, and by the end of the week, p-value 0.04, but the gain is accidental: no power, the conclusion is wrong.

Intervention in the experiment

The logic of the distribution of participants has changed, the advertising campaign has started, the big customer has gone, the comparison is devalued, and experiments should be protected from such influences as much as possible.

Working with metrics and conclusions

How to choose metrics

The test is designed to have well-defined metrics, and the metric must be sensitive, business-relevant, and non-manipulable.

Example:

For online retail, the important criterion is revenue per user, not just conversion to order, and often two are checked at once: primary (conversion), secondary (average check).

Correct conclusions

Don’t focus on one-off bursts. Always draw conclusions from the final data slice (like a week later) to remove seasonal fluctuations and external noise.

Where to get benchmarks and calculators

Recommendations and useful resources

There are no universal signs for power numbers and minimum effect calculations. Rate your niche. See Statista or Data.ai industry reports for conversion benchmarks.

FAQ

How to distinguish A/A test from A/B test when running it

A/A test: both groups see the same thing. Run if you implement a new experiment framework, change the metric system, or run the first test on a new platform. A/B is always needed when you test the hypothesis of change.

Why if A/A makes a difference, the system is broken

If the A/A result shows a significant difference – your distribution system is not random, consider the test unreliable.

How to calculate the power of the test

Use online calculators like Statsig, enter your expected conversion, minimum meaningful effect, level of significance, and see the sample size you want.

What is considered good power

The most common target is 80 percent or higher, which means that if there’s an effect of the right magnitude, 4 out of 5 of these tests will detect it.

What’s the most “honest” metric?

Look for metrics that are directly related to user behavior or revenue: e-commerce is conversion to purchase and average check; SaaS is feature activation or retention.

Can I stop the test early if it is clear?

Don’t. Ending the experiment prematurely distorts the p-value, adds to the risk of false positives.