What is a sample size?
Sample size refers to the random selection of visitors from a large group of visitors to your website that becomes a part of your website test. You must calculate sample sizes to understand how many visitors you might need to divert to each A/B test variation to achieve statistically significant improvement in conversions for that test.
Why is selecting the right sample size important?
To determine the effectiveness of your A/B test, selecting the right sample size is one of the most important prerequisites. Here are the reasons why this is a must to run valid tests.
Make the right use of time and effort
Running a test with too small a sample size may lead to inaccurate results. But running a test with a large sample size, crossing the required threshold, is also not desirable because it leads to a waste of resources, time, and effort, which could be spent on other tests.
Win the confidence of your management
Through an A/B test, you want to prove the validity of what you’ve hypothesized. Before making any changes to your website, you probably need to prove your idea to the stakeholders and earn their support on the move. Selecting the right sample size is necessary to increase the probability of your hypothesis backing the A/B test to win. This becomes even more important to test and implement price-related changes where management-level decision-making is involved.
Ship your ideas confidently
There’s no one-sample-size-fitting-all concept in A/B testing. So, if you think the sample size that worked for the previous test should work for the next test, you’re wrong. Sample size should be based on traffic, expected conversions, and confidence level to justify the hypothesis. Even if you have a strong hypothesis, the test can fail if you select an incorrect sample size.
How to calculate the sample size
Sample size calculations require an understanding of some technical mathematics (read to know more). We shall stick to the basics and break down the formula for easy understanding.
Number of variations
As the name suggests, you have to consider the number of variations to be used to find the right sample size. This is done to ensure the adequate distribution of traffic to all the variations. And by variations, we also include the control of the test.
Conversion rate
This is pretty straightforward. You will have to consider the current conversion rate of any element or page you’re trying to improve. In other words, it stands for the current conversion rate for your control.
Minimum detectable effect (MDE)
MDE is the minimum change in the variation’s conversion rate you would like to detect. It measures the difference in conversion rate between the control and the variation to understand the impact brought on by the test.
How can VWO help you?
Thanks to technology, you can take all the guesswork out of sample size calculations without even delving into the technical understanding of how things work. This is why VWO brings you its A/B test Sample Size Calculator.
Let’s say the current conversion rate on your landing page is 5% and you want to see a minimum improvement (MDE) of 15%. You’ve to enter these details along with the total number of variations (including the control and combinations) you want to test.
Suppose you have 1 control and 2 variations to test. Enter 3 in the field for the number of variations.
That’s it. Click on the calculator sample size and get the number of visitors each of your variations should receive.
Based on the above example, the number of visitors for each variation should be 14674 using Bayesian statistics, as against 40533 based on Frequentist statistics.
Other calculators by VWO
Apart from calculating sample size, you can also count on VWO to calculate the ideal test duration – how long your test should run – to get statistically significant results out of your A/B test. Using its significance calculator, you can find out how probable it is that the difference between the control and variation is real and not random or due to an error.
To level up the overall optimization program for your website, you should go beyond the calculators and use the VWO platform to experience the myriad benefits it brings to your business. From running different types of tests to performing user behavior analysis – you can increase conversions and, more importantly, build a culture of experimentation using this integrated experience optimization platform. Sign up for a full-feature trial and embark on your brand’s optimization journey today.
