Misinterpretation of data

Most interpretation errors occur before calculation: a vague question, a changed definition of a metric, incomparable groups, or a convenient period choice. Start the analysis with the origin of the data and alternative explanations, and separate the certainty of the output from the confidence of the number in the report.

What are the errors of data interpretation

Definition and types of errors

Data interpretation error is when you misunderstand or explain metric values, experiment results, or user behavior. The problem often arises not from a data collection failure but from incorrect conclusions. The main types of errors:

1. False correlations

You see a relationship between the two, but it doesn’t mean a causal relationship.

**Example: Launching push notifications coincided with the growth of DAUs. I thought the pushi changed everything, but it actually coincided with a big update.

2. Ignoring the base rate (base rate fallacy)

You draw conclusions without considering the initial level of the metric.

**Example: CR rose from 2% to 2.4%. The difference seems big at a short distance, but over the long horizon it may just be seasonal.

3. Misunderstanding of statistical significance

You think the growth of the metric is important, but you don’t check the p-value and the size effect. The result is wrong decisions and false hypotheses.

Why is it important?

Misinterpretation of data often leads to poor product decisions. Resources are spent on developing the wrong feature or improvements that do not affect the business goals.

The main traps when working with metrics and experiments

Correlation does not equal causation

A common mistake is to accept the conclusion “saw growth, then did it right.” But the growth could have been caused by external factors, seasonality, a marketing campaign, just coincidence.

**Example: In e-commerce, CR has grown in parallel with the launch of Black Friday. The team attributed the success to the redesign of the product card, although the main reason is traffic and discounts.

Harvard Business Review - When Correlation Does Not Imply Causation (https://hbr.org/2018/04/when-correlation-doesnt-imply-causation)

Unaccounted for segmentation

The overall numbers may hide the dramatic differences between user segments. In the end, a decision that is useful to one audience can make things worse for another.

**Example: Rated Retention for all users, ignoring new and loyal users. The new Retention fell, the loyal ones grew - the average value does not teach anything.

How to avoid mistakes: proven approaches

Use statistical verification.

Always check the effect of the A/B test for statistical significance: p-value, confidence interval, minimally detectable effect.

** Case: The e-learning team saw an increase in the average session duration. After calculating the confidence interval, it turned out that the difference falls into the error zone - the main behavior has not changed.

Statistical significance basics
Stats Engine, Optimizely

Look at the dynamics, not disposable dots

Compare the metric dynamics at different times, not just the same date before and after launch.

**Example: After the launch of the new onboarding, the app only measured CR for the first week. But in the second week, users began to fall off massively - short-term success did not become long-term.

Typical anti-patterns of interpretation

Selective reporting (cherry-picking)

Show only those metrics or segments where the change is positive, ignore the negative effects.

** Case: In the presentation on the growth of MAUs hide the fall of ARPU in new users.

Mixing metrics and lack of a single goal

You use a lot of metrics without breaking them down into lead and lagging. Confusing user activity and business results.

Example: Focus on page view growth instead of CAC, CLV or Retention.

How to Verify Conclusions

Check through segmentation and additional sources

Break users into groups and compare the results. Try to collect additional data – user interviews, high-quality analytics.

**Example: You run an A/B test for B2B and B2C. For B2C, the feature went well, for B2B, the revenue slice showed zero effect.

Focus on causality

Use additional experiments and control groups to confirm the effect.

**Example: You enter a new email chain; some users remain unchanged as a control group. The increase in activity only among participants confirms the impact of communication.

FAQ on errors in data interpretation

**1. Why can’t we just rely on one metric? Often, the growth can be random or associated with other changes.

**2. How is correlation different from causation? Correlation is the joint change of metrics. Causality is the proven relationship “A affects B”.

**3. How to choose the right metrics for analysis? Focus on the relationship to product objectives and the validation of hypotheses.

**4. How often do I have A/B tests? Depends on traffic and tasks. The main thing is not to stop the test too early and do not draw conclusions from insufficient data.

**5. How not to embellish the reports? Look at segments, dynamics and be sure to discuss the minuses along with the pros.

**6. Where to find benchmarks by metrics? See industry reports, annual reports of open companies, analytical reviews of agencies and platforms (e.g. Amplitude, Mixpanel, Appsflyer).