Important terms in A/B Testing
Here are a few basic terms that you need to be familiar with before venturing into A/B testing.
Statistical significance denotes your risk tolerance and confidence level on your experiment results. For example, if your experiment has a statistical significance value of 95%, it indicates that the results are 95% accurate with a possible error rate of 5%. It is almost impossible to conduct A/B tests without statistical significance.
Baseline Conversion Rate
Baseline conversion rate denotes the current conversion rate of the page that you are testing. The end goal of your A/B testing experiments would be to increase this value. This is normally expressed as a percentage and is calculated with the formula:
Baseline conversion rate = Number of conversions / Total number of visitors
Minimum detectable effect
Minimum detectable effect is a relative percentage of increase that you would like to see in your winning variant. This has to be set before you start your experiments so that you can get an estimate of how long your tests should run and how much traffic you might have to allocate.