A beginner’s guide to A/B Testing

Are you new to A/B testing and trying to figure out how it can help to improve your website conversions? Here’s a comprehensive guide to A/B testing and everything around it. In this page, you’ll find answers to these questions and more

What is A/B Testing?

A/B testing is comparing two different variations (A and B) of a web page against each other to find out which one brings more conversions. The existing web page is called the control or the original. The altered version is called the variant. The website traffic is split equally between both the variations to check the results.

A/B tests are not restricted to just testing two variations. You can also create multiple variations. Such experiments are called A/B/n testing. The number of variations can increase as long as they are compared against constant metrics. For example, you cannot change the ‘Call-to-action’ in one variation and the ‘headline’ in the other variation. Both the variations has to be tested for the same element. Moreover, the tests need not be limited to measuring website conversions alone. You can also use them to measure the effectiveness of your marketing campaigns, lead nurturing emails, application usability and more.


Why A/B testing?

While designing a website, you might have to make a lot of decisions with respect to headlines, CTAs, content and design layouts. But how do you make these calls? Just by the look and feel of it? Or by your personal preferences? Probably yes, because how can one logically choose between a red CTA button or a green CTA button? Or between two equally powerful headlines? This is where A/B testing will help you make a data-driven decision. With A/B testing, traffic is equally split and statistically shown which variation brings more conversions. You no longer have to guess or assume.

A/B testing also helps to increase conversion rate by improving user experience. By consistently A/B testing each element on your website like headlines, CTAs, design and content, you can arrive at the best version that brings maximum conversions. So, you’ll be increasing your conversions without increasing your spending. This also increases the return on investment of your paid campaigns as well.

What can you A/B Test?

There’s more to A/B testing other than changing the color of your CTA buttons. You can test anything on your website that has an impact on your visitor behaviour. Here are some of your website elements that you can A/B test

  • Text
  • Images
  • CTA buttons
  • Headlines
  • Links
  • Videos
  • Length of the content
  • Order of the content
  • Size of the images
  • Placement of the images/content
  • Pricing
  • Personalized/ tailor-made content/design
  • Design layout

These examples are pertaining to website layouts. You can also use A/B testing to improve your email marketing campaigns, product user interface/experience, onboarding/lead nurturing emails as well.

But, how do you know that these elements impact your visitor behavior? Tools like heat map or session replay software help you identify how your visitors engage with your website. They provide you with information on where your visitors get stuck, where they exit and much more. Armed with this data, you can go ahead with the A/B testing process.

Important terms in A/B Testing

Here are a few basic terms that you need to be familiar with before venturing into A/B testing.

Statistical Significance

Statistical significance denotes your risk tolerance and confidence level on your experiment results. For example, if your experiment has a statistical significance value of 95%, it indicates that the results are 95% accurate with a possible error rate of 5%. It is almost impossible to conduct A/B tests without statistical significance.

Baseline Conversion Rate

Baseline conversion rate denotes the current conversion rate of the page that you are testing. The end goal of your A/B testing experiments would be to increase this value. This is normally expressed as a percentage and is calculated with the formula: 
Baseline conversion rate = Number of conversions / Total number of visitors

Minimum detectable effect

Minimum detectable effect is a relative percentage of increase that you would like to see in your winning variant. This has to be set before you start your experiments so that you can get an estimate of how long your tests should run and how much traffic you might have to allocate.

How to A/B test?

Though A/B testing seems like a simple method of comparing the performance of two web pages, it is best to follow a standardized approach to get accurate results and hence achieve higher conversions. The A/B testing process starts with formulating your hypothesis and ends with running your A/B testing experiments for the planned time. We will talk more about this in detail in the following sections.

Formulate hypothesis

Before you start your A/B test, you will have to come up with a hypothesis. A hypothesis is a simple statement that can predict the outcome of your experiment. Your A/B test results will either prove or disprove your hypothesis. Since you already have an idea on what to test, formulating a hypothesis should be pretty simple. 

For example, if you want to change the CTA on your website, then your hypothesis should be “Changing the CTA from Subscribe Now to Join 10000+ fellow marketers will increase conversions.” Now, you can run A/B test and see which CTA button brings more conversions or check if this had any effect on the conversion rate at all.

Create Variations

Once you have your hypothesis, you can proceed with creating variations of your web pages. Consider our previous example on changing the CTA text in your website. To do this, you will have to create a new page(variant) to place the new CTA "Join 10000+ fellow marketers". You can also experiment with the design, color and the layout of the CTA button, if needed.

So how do you create this variant? Most of the A/B testing tools, have an inbuilt editor in them that helps you create web page variations within their user interface. This means you don’t have to depend on your developers or coders to make changes to your websites. You can simply use an A/B testing tool to push changes.

Set Goals

Though the ultimate goal of your A/B testing experiment is to improve conversions, you need to specify what conversion means to you. Do you want your visitors to spend more time on your page, subscribe to your blog, or make a purchase? These goals will determine the winning page, which you can later implement it in real-time. Your A/B testing tool can help you come up with revenue-based goals, engagement-based goals or event completion goals for your experiments, depending on your requirement.

Run A/B Tests

Running your tests can be accomplished by using one of the many tools that are available in the market today. Once you have completed the above steps, simply clicking on a “Run Experiment” button in your A/B testing tool will launch your tests into action. Your tests have to run for a predetermined period of time so that you can identify your winning variation. An ideal A/B test duration is dependent on your sample size, statistical significance, number of variants, traffic distribution and the efficiency of your A/B testing tool’s algorithm. We’ll discuss about different algorithms in the following sections.

How long should you run A/B tests?

Now that you are clear with the A/B testing process, let’s elaborate a bit on the duration of your A/B tests. Experts agree that the ideal amount of time to run an A/B test is two complete business cycles. A business cycle in this case indicates the cyclic nature of your business over a time period. For example you may experience a rise in sales during Christmas and a drop in sales after the first week of new year. In this case, your business cycle is two months.

Also, you cannot stop your tests before their intended time whether or not you achieve the desired results. Running A/B tests for a long time will liquidate the process and disrupt the results. To calculate your ideal test duration, there are many calculators out there that provides you with accurate data.

Check out the below duration calculator powered by Freshmarketer, to calculate the optimum time required for your A/B tests.

Calculating target samples

How do you know if you have the right amount of traffic or the right number of visitors to run an A/B test?

With Freshmarketer’s sample size calculator, you can identify the optimum samples that you require for running your A/B tests. The sample size has to be calculated before starting your experiment, so that you get accurate results.

Simply enter your baseline conversion rate, your desired minimum detectable effect percentage and your preferred statistical significance value, to know your sample size.

How to analyze A/B test results

Once you run your A/B testing experiment for the intended period of time, the next step would be to analyze the results.

You will face one of the below scenarios while interpreting your A/B test results:

i) The conversion rate of the variant is lesser than the control. This means that original version brings more conversions than that of altered one. You can either continue with original version or try A/B testing another variation to check the results.

ii) The conversion rate of the variant is higher than the control. In this case, your hypothesis is proved right. The altered variation brings more conversions than that of original. Now, you can either continue with the winning variation or try A/B test on another variation to check the results.

iii) No significant difference is observed between the two results. Sometimes, the original or the control and the variant might have almost similar conversion rates. In that case, both has same effect on the user behaviour.

Some factors that you need to keep in mind while analyzing A/B test results:
  • The type of devices that were used to access the website(mobile/desktop)
  • The type of visitors (organic, PPC, campaigns, social media etc)
  • Visitor information (age, sex, demography etc)

Most A/B testing tools can provide the above information for you. If it doesn’t you can use a platform like Google Analytics to observe this data. Tools like Freshmarketer easily integrates with Google Analytics to provide deeper insights into your A/B testing experiments.

With Freshmarketer, you can interpret the results of your A/B tests right away with Reports. A/B test reports provide you with detailed information on your experiment and identify the winning/losing variation. If your test yields inconclusive results, you will be able to identify that as well. You can also segment your test results with advanced filters to gain even more granular insights.

Once you’ve identified your winning variation, the next step would be to implement the winning variation elements in your web page and continue to achieve higher conversions. It is best to archive your A/B test results irrespective of whether they were successful or not. To know more about interpreting your A/B test results with Freshmarketer, click here.

A/B Testing Tools

There are many tools in the market today that will help you run A/B testing experiments. You can simply choose the one that best suits your requirements and your budget. Make sure to look out for a simple, intuitive A/B testing software that provides prompt support, accurate results and is scalable enough to meet your website traffic.

Some A/B testing tools also have a Chrome extension. This will help you make changes to your experiment, launch an A/B testing experiment or navigate between different experiments that you may have set up in a quick and simple way. With Freshmarketer’s chrome extension, it is now easy and convenient to create or modify A/B testing experiments swiftly. To create your first A/B test with Freshmarketer, click here.

Different A/B testing tools use different algorithms. Some of the commonly used algorithms are Bayesian, Welch’s T-test, Student’s T-test etc. Once you have zeroed in on your A/B testing tool, installing them would probably require adding snippets of code to your website. Your tool will then run your experiments and determine the winning variation for you.

How to choose an A/B testing tool?

It’s pretty common to go with tools that are already popular but it’s important to know what features to look for and check if the A/B testing tool you buy has them. Here are top 5 factors you should analyze and decide before signing up for an A/B testing tool.

Frequency and usage

A/B testing can be done to improve user experience on the website, product, marketing campaigns and lead nurturing emails as well. There are different A/B testing tools for client-side and server-side testing. Choose a client-side testing too If you want to run A/B tests to improve website conversions and server-side testing tool if you want to test product usability and interface. If you want to run both, then its advisable to use different A/B testing tool as that would reduce dependencies.

WYSIWYG Visual Editor

A visual editor is where you edit web pages and create multiple variations to run the A/B tests. An easier and intuitive visual editor helps you run A/B tests faster. It eliminates dependencies from the tech team and helps you make website changes without much technical expertise. So make sure that your A/B testing tool is easy to use.


Different A/B testing tools use different algorithms. These algorithms determines the accuracy of your A/B testing results. Some of the commonly used algorithms are Bayesian, Welch’s T-test and Student’s T-test. The most commonly used algorithm that gives reliable results is Bayesian. Earlier few tools used T-tests but this algorithm is outdated and tend to produce unreliable discoveries.

Chrome extension

With a chrome extension, you can easily edit web pages without much technical expertise, optimize pages beyond login screens and even for gated content as well. But if you’re loading your pages inside an Iframe, you can’t enjoy these privileges. Especially, If your page has an iframe busting logic, you cannot A/B test it. But it’s important to have a frame breaker for security reasons because without it, your site/pages will be vulnerable to clickjacking attacks.

Analytical tools

It’s only logical to first start with analyzing visitor behavior, before jumping to optimization. Tools like heatmaps, session replay, funnel and form analysis features provides you with meaningful insights on how visitors interact with your website and where exactly they’re dropping off. This gives better context and direction on how to optimize your web pages as well. So, look for a tool that also has analytical tools.

Impact of A/B Testing in SEO

While it might be an effective method to A/B test each web page to check what works best, you might also have duplicate pages of a same web page which are not indexed. You would want to make sure that you don’t get penalized by google for content cloaking.

“Cloaking refers to the practice of presenting different content or URLs to human users and search engines. Cloaking is considered a violation of Google's Webmaster Guidelines because it provides our users with different results than they expected.“ - By definition of Google.

Though cloaking of web content is strictly penalized by Google, there are certain best practises suggested by Google to make sure that your A/B testing variations doesn’t affect your SEO rankings.

Use of rel= “canonical” link attribute

Just use rel= “canonical” link attribute tag so that Google algorithm does not index the duplicate pages.

Use 301 redirect from original to variation page

Use 301 redirect instead of a 302 redirect. If not, Google might continue to index the old URL and mark the new ones as the content duplicate.

Avoid running an A/B test longer than necessary

If an A/B test is running for a longer time than that of recommended, Google might interpret it as a disguised method of data cloaking.

Avoid slowing down website speed

Website’s speed is also important for SEO ranking. If your A/B testing tools’ snippet is loading synchronously, make sure it doesn’t affect your site load time.

Our recent blogs