What is multivariate testing (MVT)?
Multivariate Testing (MVT) is a derivative algorithm of traditional A/B testing that allows you to test for multiple changes on the page. For instance, you might want to experiment with different headline fonts, banner colours, and various background images. The time-consuming option is where you sequentially run tests to find the best combination. Just like a designer makes various changes at once and tries to assess iteratively if the current combination looks good, you would need to run many tests sequentially to find the best combination. An MVT generates all possible combinations of various changes at different places on your website and tests them in parallel. This is equivalent to making all the combinations yourself and adding them as variations to an A/B Test.
To understand better, say you have 3 elements on your webpage – a banner, an image, and some text. Let us call the existing versions of all the elements as C1, C2, and C3 (C for Control) respectively. Let us say you made one change to each element. The new versions are called V1, V2, and V3. Note that C and V refer to the change and the subscripts refer to the element. MVTs will create 8 versions of your page as follows (separated by a bar):
C1 C2 C3 | C1 C2 V3 | C1 V2 C3 | C1 V2 V3 | V1 C2 C3 | V1 C2 V3 | V1 V2 C3 | V1 V2 V3
However, the more changes you add, the total number of combinations increases exponentially. So, if you are testing with 4 variations for all the three elements mentioned, your total combinations become 5*5*5 = 125. This means you will need 125 times more traffic to find the winning combinations in the test. Hence, MVT is suitable for websites with a large amount of traffic who want to test multiple changes at once and find the best combination. With VWO, you can test multiple changes on your landing pages, all at the same time, without worrying about any inaccuracy creeping into the data. Take a free trial and try it out for yourself.
When should you use multivariate tests?
MVTs are algorithms that take up a lot of your traffic but give you immense value in taking a complex decisions. It is a bad idea to just throw in some changes and run an MVT on them because it will frustrate you since the results will take very long to converge. You should use an MVT when:
- Your use-case has a combinatorial aspect to it: Sometimes tests by nature have a combinatorial effect in which cases it makes sense to use an MVT. For instance, you might want to test which products sell best together. It is the kind of test where you expect products to have a combinatorial effect on revenue because some product combinations are complementary. However, this might not be the case if you are testing four different versions of headlines and four different versions of button colours. An MVT will make 4*4 = 16 combinations in your test and take 16 times longer, whereas you could run two independent tests with just 4 variations each which would take half the time (4+4 = 8 assuming you run these tests sequentially).
- You have a lot of visitors that give you the luxury to run MVTs: Sometimes, you might be having so many visitors on your website that you have the luxury to test which combination of different changes works best. Such experiments might lead to finding unexpected wins if some combinations fit in together, or some unexpected findings which make you realize that two changes should not be made together. For instance, a grey font text with a black background image would be a bad combination.
In any case, one should not run an MVT without a defined use case that can be solved only by MVTs. Traditional A/B Tests are much more reliable than MVTs because they are the scientific gold standard for the discovery of causal relationships (X causes Y). With that, let us discuss the limitations of multivariate testing.
What are the limitations of multivariate testing?
Online experimentation is traditionally done by making only one change between Control and Variation and calculating the impact of the same. This is so because if you make multiple changes at once, it gets difficult to attribute which change caused the impact on the tested metric (conversion rate, revenue, page load time). Further, testing many changes introduces randomness into the test and makes the test less reliable than traditional A/B tests.
Multivariate tests suffer from the same two limitations of increased time and decreased accuracy of the test:
- Increased time of testing: Time of testing increases because the number of visitors required increases linearly with the number of combinations in your test. So, if you have 150 combinations you would need 150 times the visitors required for a traditional A/B test with only one control and one variation.
- Decreased accuracy of results: When making so many comparisons at once, the chance that you get a faulty decision (a smart decision when the difference was insignificant) increases. Hence, one should be clear that if a use case can be tested without making all combinations changes, different elements should be tested separately.
Conclusion and future scope
Multivariate testing is a useful extension of traditional A/B testing that should be judiciously used for the right use cases. With the right application, MVTs can help you find unexpected patterns in customer behaviour and use them for driving up your KPI Metrics. But MVT does not end there. MVTs allow for many further optimizations that should in turn have the potential to make MVTs much more usable in the future.
Some of these optimizations include a matrix factorization technique that equips MVTs to calculate results with much less traffic. Further, some mathematical techniques help you to dissect MVT results and diagnose which change caused the maximum uplift in a combination. Lastly, MVTs run in combination with MABs are very helpful because 80% of combinations are usually bogus and can be quickly removed from the MVT early in the test thus saving a large amount of traffic on suboptimal variations.
