How to use split testing based on your website traffic

It might be tempting to go with your gut when making decisions about your website.

However, intuition often turns out to be wrong, so relying on it can cost you a lot of money. What should you do instead?

You should put your intuition to the test. Don’t just assume that something is going to work, use split testing to see if it actually works.

The data doesn’t lie.

(image source)

What is split testing?

A split test is a test in which you split the traffic between two variants of the same page to see which one performs better.

There are three types of split tests:

  • A/B testing. When you conduct an A/B test, you only make a single change at a page element level. For example, if you have a green “Buy Now” button, you might create a page with a red “Buy Now” button. A/B testing is great for optimizing a page. 
  • Multivariate testing. When you conduct a multivariate test, you make several changes at a page element level. For example, if you have a green “Buy now” button, you might also create pages with a red “Buy now” button, a green “Order Now” button, and a red “Order Now” button. Multivariate testing is also great for optimizing a page. It’s not as precise as A/B testing but requires less time than running an A/B test for each change. 
  • Split URL testing. When you conduct a split URL test, you make significant changes at a page level, and then use a separate URL for each page. It works best for testing big changes such as a complete website redesign.

Multivariate testing and split URL testing are often confused because they both involve making multiple changes to the page.

How they differ

The above tests differ by:

  • Methodology: When you conduct a multivariate test, all page variants have the same URL. Meanwhile, when you run a split URL test, each page variant has a separate URL.
  • Scope: Multivariate tests usually involve testing a combination of minor changes. Meanwhile, split URL tests usually involve testing major changes.

(image source)

Basically, A/B and multivariate tests help you optimize an existing page, and split URL tests allow you to try out a new page that has the same purpose.

All three split testing methods are invaluable when it comes to conversion rate optimization.

How traffic affects split testing

Split testing doesn’t work equally well for low, medium, and high traffic websites. Why?

Example:

Consider these two situations –

  • In the first scenario, you only get 100 visitors per month. You direct 50 of them to the control sales page and the other 50 to the experimental sales page. 
  • Next, you get 1,000,000 visitors per month. You then direct 500,000 of them to the control sales page and the other 500,000 to the experimental sales page.

Let’s say that in both cases the control sales page converts at 10% and the experimental sales page converts at 12%.

But are the test results equally reliable?

No, because if you only have 50 visitors, then a conversion rate of 10% means that 5 of them bought your product, and you only need one more person to buy it to get to 12%.

Meanwhile, if you have 500,000 visitors, then a conversion rate of 10% means that 50,000 of them bought your product, and you need an additional 10,000 people to buy it to get to 12%,

Now, if the control page got 5 sales with 50 visitors and the experimental page got 6 sales with 50 visitors, that doesn’t tell you much because that one person might have decided to buy your product for reasons completely unrelated to the changes you are testing (this is called statistical noise). In other words, it could have been a fluke.

However, if the control page got 50,000 sales with 500,000 visitors and the experimental page got 60,000 sales with 500,000 visitors, you can be reasonably sure that the increase in the conversion rate was the result of the changes that you made because it’s extremely unlikely to have been a fluke.

In other words, the results from the test with 100 visitors were not statistically significant, but the results from the test with 1,000,000 visitors were.

When you are conducting a split test, it’s essential to run it until it reaches statistical significance (the point at which the results can be reliably determined not to be statistical noise). Otherwise, the results might be misleading.

And the less traffic you have, the longer it takes to reach statistical significance.

(image source)

Split testing for low, medium, and high traffic websites

It’s important to understand that the time it takes to reach statistical significance will vary depending on what you decide to test.

Generally, the smaller the change, the smaller its effect, and the smaller the effect, the larger the sample size that you need to get statistically significant results.

Here’s how you should use split testing based on the amount of traffic that your website receives:

  • Low traffic (<20K): At this level, you should focus your efforts on testing major changes, and go for big wins. 
  • Medium traffic (20k – 500K): At this level, you can also test more subtle changes. 
  • High traffic (>500K): At this level, you can also test minor changes, especially given that even a tiny increase in the conversion rate can have a significant impact on your bottom line.

In short, the less traffic you have, the bigger the changes you test should be because a test for a big change will take less time to reach statistical significance (which still might take quite a while).

Conclusion

Split testing is the best way to determine what changes you should make to your website to grow your business.

However, when it comes to what exactly you should test, you need to be mindful of the amount of traffic that is available to you.

And don’t be afraid of testing drastic changes. You might come across some big wins that way.

Feature Image & Infographic Illustration: Rajesh