A/B Testing Terminology: A Glossary for Marketers

Knowing these terms will help you communicate more effectively with your team, interpret results accurately, and ultimately, run more successful tests.

1. A/B testing (split testing)

A/B testing, also known as split testing, is a method used to compare multiple versions of a web page or app to determine which one performs better.

By showing different versions (variants) to different segments of users, you can see which version drives more conversions, providing data-driven insights to improve your website or app.

2. Control group

The control group is the group that sees the original version of the web page or app. This group’s performance is used as a benchmark to compare against the performance of the variant group.

3. Variant (treatment group)

The variant, or treatment group, sees the new version of the web page or app. This group’s behavior is compared to the control group to determine the impact of the changes.

4. Hypothesis

Your hypothesis is a clear, testable statement predicting the outcome of your A/B test.

It usually follows this format: “If we change [specific element], then [desired outcome] will happen.”

Crafting a solid hypothesis is essential for a successful experiment.

5. Experiment design

Experiment design involves planning your A/B test.

This includes defining your hypothesis, selecting the metrics you’ll track, determining the sample size, and setting the duration of the test.

A well-designed experiment ensures reliable results.

6. Randomization

Randomization means randomly assigning participants to either the control or variant group. 

This process minimizes biases and ensures that any observed effects are due to the changes you’re testing, not external factors.

7. Sample size

Sample size refers to the number of participants in your A/B test. A larger sample size increases the reliability of your results, making it easier to detect true differences between the control and variant groups.

8. Statistical significance

Statistical significance indicates that the results of your test are likely not due to chance.

It provides confidence that the observed effect is real and not just a random occurrence. This is crucial for validating your test results.

9. P-value

The p-value helps determine statistical significance by measuring the probability that the results of your A/B test occurred by chance.

A p-value of less than 0.05 is typically considered statistically significant, meaning there’s less than a 5% probability that the results are due to random variation.

10. Confidence level

The confidence level indicates how certain you can be that your results are reliable.

A 95% confidence level means you can be 95% sure that the observed differences are real. Higher confidence levels provide greater assurance in your findings.

11. Conversion rate

Conversion rate is the percentage of users who complete the desired action, such as making a purchase or signing up for a newsletter.

It’s a key metric for measuring the success of your A/B test and overall campaign performance.

12. Lift

Lift refers to the percentage increase in the conversion rate of the variant group compared to the control group. 

It’s a direct measure of the impact of your changes, showing how much better (or worse) the variant performed.

13. Metrics

Metrics are the data points you track to measure the success of your A/B test. Common metrics include conversion rate, bounce rate, and average order value.

Selecting the right metrics is vital for accurate and meaningful test results.

Understanding user behavior through these metrics helps you make data-backed decisions for website optimization.

14. Statistical analysis

Statistical analysis involves applying statistical methods to interpret the results of your A/B test. This process helps you understand the significance, reliability, and implications of your findings.

Tools such as t-tests, chi-square tests, and regression analysis are often used to analyze test data.