Multivariate Testing

What is multivariate testing?

Multivariate testing (MVT) is a form of A/B testing wherein a combination of multiple page elements are modified and tested against the original version (called the control) to determine which permutation leaves the highest impact on the business metrics you’re tracking. This form of testing is recommended if you want to test the impact of radical changes on a webpage as compared to analyzing the impact of one particular element. 

Unlike a traditional A/B test, MVT is more complex and best suited for advanced marketing, product, and development professionals. Let’s consider an example to give you a more comprehensive explanation of this testing methodology and see how it aids in conversion rate optimization.

Let’s say you have an online business of homemade chocolates. Your product landing page typically has three important elements to attract visitors and push them down the conversion funnel – product images, call-to-action button color, and product headline. You decide to test 2 versions of all the 3 elements to understand which combination performs the best and increases your conversion rate.  This would make your test a Multivariate Test (MVT). 

A set of 2 variations for 3 page elements means a total of 8 variation combinations. The formula to calculate the total number of versions in MVT is as follows:

[No. of variations of element A] x [No. of variations of element B] x [No. of variations of element C]… = [Total No. of variations]

Now, variation combinations would be:

2 (Product image) x 2 (CTA button color) x 2 (Product headline) = 8 

Each of these combinations will now be concurrently tested to analyze which combination helps get maximum conversions on your product landing page. 

Note, a multivariate test eliminates the need to run multiple A/B tests and subsequent A/B tests to find the winning variation. Running concurrent tests with greater variation combinations not only helps you save time, money, and effort, but also draw conclusions in the shortest possible time.  

Here’s a real-life example of multivariate testing to testify the benefits of this experimentation methodology. 

Hyundai.io found that while the traffic on its car model landing pages was significant, not many people were requesting a test drive or downloading car brochures. Using VWO’s qualitative and quantitative tools, they analyzed that each of their landing pages has many different elements, including car headline, car visuals, car specifications, testimonials, and so on which might be causing friction. They decided to run a multivariate test to understand which elements were influencing a visitor’s decision to request a test drive or download a car brochure. 

They created variations of the following sections of the car landing page:

• New SEO friendly text vs old text: They hypothesized that by making the text more SEO friendly, they could reap more SEO benefits.
• Extra CTA buttons vs no extra CTA buttons: They hypothesized that by adding extra and more prominent CTA buttons on the page, they’ll be able to nudge visitors in the right direction.
• Large photo of the car versus thumbnails: They hypothesized that it’s better to have larger photographs on the page than thumbnails to create better visitor traction

Hyundai.io tested a total of 8 combinations (3 sections, 2 variations each = 2*2*2) on their website. 

Here’s a screenshot of the original page and the winning variation:

The variation with more SEO-friendly, extra CTA buttons and larger images increased Hyundai.io’s conversion rates for both, request for test drive and download brochure, by a total of  62%. They also saw a 208% increase in their click-through rate.

MVT is not just restricted to testing the performance of your webpages. You can use it across a range of fields. For instance, you can test your PPC ads, server-side codes, and so on. But, MVT should only be used on sufficiently sized visitor segments. More in-depth analysis means longer completion time. You must also not test too many variables at once as the test will take a longer time to complete and may or may not achieve statistical significance.

Understanding some basic multivariate testing terminologies

Although an integral part of A/B testing, there are a couple of terminologies specific to multivariate testing that anyone getting into this experimentation arena should know. 

1. Combination: It refers to a number of possible arrangements or unions you can create to test a collection of variable options in multiple locations. The order of selection or permutation does not matter. For instance, if you’re testing three elements on your home page, each with three variable options, then there are a total of 27 possible combinations (3x3x3) that you’ll test. When a visitor becomes a part of your test, they’ll see one combination, also referred to as an experience, when they visit your website. 
2. Content: Text, image, or any element that becomes a part of an experiment. In multivariate testing, several content options spread across a web page are compared in tandem to analyze which combination shows the best results. Content is also sometimes referred to as a level in MVT..  
3. Location: Ideally, a location refers to a page on the website or a specific area where you run optimizations. It’s essential to website activities and experiences, and display content to visitors or track visitor behavior. 
4. Control: Ideally, control refers to the original page, element, or content against which you’re planning to run a test. It also represents the “A” in the A/B testing scenario. For instance, if you want to test the performance of your homepage’s banner image, the original or existing banner image will be the “control.” Control is often also referred to as “Champion” by many seasoned optimizers. 
5. Goal: An event or a combination of events that help measure the success of a test or an experiment. For instance, a content writer’s goal is to increase visitor engagement on their content pieces and even generate content leads. 
6. Confidence Level: How positive or assertive you are about the success of your experiment. 
7. Conversion Rate: The percentage of unique visitors entering your conversion funnel and converting it into paying customers. 
8. Element: A discrete page component such as a form, block of text, an image, call-to-action button, etc.  
9. Experiment: It’s another way of assessing or evaluating the performance of one or more page elements. 
10. Hypothesis: A tentative assumption made to draft and test a logical or empirical consequence of a particular problem. An example of a hypothesis could be: Based on the previously run experiments and qualitative data gathered through heatmap and scrollmap analysis, I expect that adding banner and text CTAs on the guide page at regular intervals will help generate more content leads and MQLs.
11. Non-Conclusive Results: Not deriving any solid conclusion from the experiment(s) you’ve run. Non-conclusive results do not point to a test’s failure—instead, just the failure of deriving a learning curve. 
12. Qualitative Research: A technique of gathering and analyzing non-numerical data from existing and potential customers to understand a concept, opinion, or experience. 
13. Quantitative Research: Digging through numerical data derived from analytics to find insights around the behavior of your website visitors and draw statistical analysis.   
14. Visitors: A person or a unique entity visiting your site or landing pages. They’re termed unique because no matter how many times a person visits your site or page, they’re counted only once.

What are the different types of multivariate testing methods?

MVT is in itself an umbrella methodology. There are several different types of multivariate tests that you can choose to run. We’ve defined each of these in detail below.

1. Full factorial testing

This is the most basic and frequently used MVT method. Using this testing methodology, you basically distribute your website traffic equally among all testing combinations. For instance, if you’re planning to test 8 different types of combinations on your homepage, each testing variation will receive one-eighth of all the website traffic. 

Since each variation gets the same amount of traffic, the full factorial testing method offers all the necessary data you’d need to determine which testing variation and page elements perform the best. You’ll be able to discover which element had no effect on your targeted business metrics and which ones influenced them the most. 

Because this MVT methodology makes no assumptions with respect to statistics or testing mathematics used in the background, our seasoned optimizers highly recommend it for people running or planning to run multivariate tests.

Increase your recurring revenue by optimizing your website using VWO’s Multivariate testing methodology. Sign up and start your 1-month free trial today!

2. Partial or fractional factorial testing

Partial or fractional factorial MVT methodology exposes only a fraction of all testing variations to the website’s total traffic. The conversion rate of the unexposed testing variations is interpreted from the ones that were included in the test. 

Say you want to test 16 variations or combinations of your website’s homepage. In a regular test (or full factorial test), traffic is split equally between all variations. However, in the case of fractional factorial testing, traffic is divided between only 8 variations. The conversion rate of the remaining 8 variations is calculated or statistically deduced based on those actually tested. 

This method involves the use of advanced mathematical techniques and the use of multiple assumptions to gather insights, and has many disadvantages. One pro point of this MVT methodology is that it requires less traffic. It’s a good option for websites or pages with low traffic. 

However, regardless of how advanced mathematics techniques you use to draw statistically significant results using fractional factorial testing, hard data is any day better than speculation.

3. Taguchi testing

This is an old and esoteric MVT method. If you run a Google search, you’ll find that most tools on the market today claim to cut down on your testing time and traffic requirement by using the Taguchi testing technique. It’s more of an “off-line quality control” technique as it helps test and ensure good performance of products or processes in their design stage.  

While some optimizers consider this a good MVT methodology, we at VWO believe that this is an old-school practice which is not theoretically sound. It was initially used in the manufacturing industry to reduce the number of combinations required to be tested for QA and other experiments. 

Taguchi testing is not applicable or suitable for online testing and hence, not recommended. Use the full factorial or partial factorial MVT approach.

How is multivariate testing different?

A/B testing vs multivariate testing

Ask an experience optimizer and they’ll say the ideal use of A/B testing is to analyze the performance of two or more radically different website elements. Meanwhile, MVT is a perfect technique to test which combination of page element(s) gets maximum conversions. 

In testing terms, it’s often recommended to use A/B testing to find what’s called the “global maximum,” and MVT to refine your way towards the “local maximum.”

Let’s take an example to understand the concept of global maximum and local maximum.

Imagine for a second that you’ve never tasted even a single piece of chocolate in your life, and you’re standing in a chocolate shop looking at 25 different types of chocolates, confused about which one to purchase.

There are probably 5 different kinds of caramel chocolates, 10 different varieties of truffles, 6 different variations of lollipops, and 4 different types of exotic fruit chocolates. Are you going to taste all these 25 flavors before deciding which one to buy?

You may try one kind of chocolate from each of the above-mentioned categories, but surely not all. If you find that you like truffles the most over lollipops, caramel chocolates and exotic fruit chocolates, you’ll start tasting more truffle flavors like “coconut truffles,” “Oreo truffles,” “chocolate-fudge truffles,” and so on to decide which among the truffle flavors you like the most.

In statistical terms, we’d say that the category of chocolates you like the most will become the global maximum. This is the type of chocolate that spoke to your taste buds and tasted the best among the lot. When you get down to the specific flavors of truffles, i.e., coconut truffle, Oreo truffle, chocolate fudge, and more, you’ll discover the local maximum – the best version of the variety that you chose.

As an experience optimizer, you must approach testing in a similar manner.  Find the webpage that gives you maximum conversions (global maximum), and then test combinations of specific elements on that webpage to understand which one improves your page’s performance and makes the highest-converting page (local maximum). What you’re looking for, global maximum or local maximum will define which testing methodology you must use. 

Here’s a list of pros and cons of using A/B testing and multivariate testing.

A/B testing

Multivariate testing

Pros:  1. A comparatively simple experimentation method to design and execute. 2. Helps conclude debates around campaign tactics when there’s one hypothesis in question.  3. Helps generate statistically significant results even with lesser traffic samples. 4. Provides clear and detailed result reports which are easy for even non-technical teams to interpret and implement. Cons: 1. Limited to testing a single element with a few variations, typically 2 to 3. 2. Not possible to analyze the interaction between various page elements within the same testing campaign.

Pros: 1. Gives insights regarding the interaction between multiple page elements.  2. Provides a granular picture around which element poses impact on the performance of a page. 3. Enables optimizers to compare many versions of a campaign and conclude which one has the maximum impact.   Cons: 1. A comparatively complex experimentation methodology to design and execute.  2. It requires more traffic than an A/B test to show statistically significant results.   3. Too many combinations make result interpretation difficult.  4. Can serve as an overkill when an A/B test could have been sufficient to show results.

Split URL testing vs multivariate testing

Assuming you’re fairly clear with the definition and concept of MVT, we’ll begin by breaking down the concept of Split URL testing. Rather than testing page elements at a granular level such as in the case of MVT, a split URL test enables you to run a test on a page level. Meaning, variations in the case of Split URL test are dramatically different and hosted on separate URLs but have the same end goal. 

Let’s continue on the example of you running an online business of homemade chocolates. Imagine your current homepage has a banner that shows different offers running on your website along with a section displaying your featured products, another section highlighting different chocolate categories, brand story, and related recipes. According to your gut feeling, the page looks attractive and has the potential to convert. 

However, after looking at the qualitative results, viewing heatmaps, session recording, etc. you find that many elements on your homepage are not showing the results they should. 

If you decide to run a Split URL test, you can create an entirely new page design with elements placed in a different manner and compare the performance of this variation with the control to analyze which one’s generating more conversions. 

Meanwhile, if you decide to run a multivariate test, you can create permutations of different page elements that you want to examine, maybe test different colors of your homepage’s CTA button, banner image, sub headings, and so on, and check which combination generates maximum conversion. There can be ‘n’ number of permutations that you can test with MVT.

One of the primary reasons MVT is better than split URL testing is that the latter demands a lot of design and development team’s bandwidth and is a lengthy process. MVT, on the other hand, is comparatively less complex to run and demands lesser bandwidth as well.  

Here’s a comparison table between Split URL testing and MVT:

Split URL testing

Multivariate testing

Pros: 1. Sizeable changes such as completely new page designs are tested to check which gets maximum traction and conversions. 2. Variations are created on separate URLs to maintain distinction and clarity. 3. Helps examine different page flows and complex changes such as a complete redesign of your website’s homepage, product page, etc.   4. With Split URL testing, you test a completely new webpage. Cons: 1. A comparatively complex test to design and execute. 2. Requires a lot of design and development team bandwidth.  3. Assess the performance of a website as a whole while ignoring the performance of individual page elements.

Pros: 1. A combination of web page elements are modified and tested to check which permutation gets maximum conversions.  2. Runs on a granular level to understand the performance of each page element. 3. Comparatively test more variations.Requires less changes in terms of design and layout.  Cons: 1. Usually requires more traffic to reach statistical significance.  2. Demands more variable combinations to run and show results.  3. The traffic spread across variations is too thin. This sometimes makes the test results unreliable. 4. Since more subtle changes are tested, the impact on conversion rate may not be significant or dramatic.

Multipage testing vs multivariate testing

As the name suggests, multipage testing is a form of experimentation method wherein changes in particular elements are tested across multiple pages. For instance, you can modify your homemade chocolate eCommerce website’s primary CTA buttons (Add to Cart and Become a Member) on the homepage, replicate the change across the entire site, and run a multipage test to analyze results. 

Compared to MVT, optimizers suggest it’s best to use multipage testing when you want to provide a consistent experience to your traffic, when you’re redesigning your website, or you want to improve your conversion funnel. Meanwhile, if you want to map the performance of certain elements on one particular web page, go with A/B testing or MVT. 

Here’s a clear distinction between multipage testing and multivariate testing. 

Multipage testing

Multivariate testing

Pros: 1. Create one test to map the performance of a particular element, say site-wide navigation, across the entire website. 2. Run funnel tests to examine different voices and tones on web pages. 3. Experiment different design theories and analyze which one’s the best.  4. Helps map site-wide conversion rate. Cons:  1. Requires huge traffic to show statistically significant results.  2. Can take longer than usual to conclude.Gaining results from this form of experimentation method can be tricky.

Pros:  1. Great for making micro-conversion decisions. 2. You can validate even the minutest of choices, such as the color of a CTA button.  3. Gives in-depth insights about how different page elements play together. 4. Determine the contribution of individual page elements.  5. Eliminates the need to run multiple A/B tests.  Cons: 1. More permutations or variations means longer time for a test to reach the stage of statistical significance. 2. Unlike multipage testing, you can test changes only on one particular page at a time in MVT.

How to run a multivariate test?

The process of setting up and running MVT is not very different from a regular A/B test, except for a couple of steps in between. But we’ll start from the beginning so that the process stays afresh. Let’s deep dive. 

1. Identify a problem

The first step to running MVT and improving the performance of your web page is to dig into data and identify all the loopholes causing visitors to drop off. For instance, the link attached to your “download guide” button may be broken or the form on your product page may be asking for information more than necessary. To spot these points of ambiguities, take the following steps.   

• Conduct informal research: Take a look at customer support feedback and examine product reviews to understand how people are reacting to your products and services. Speak to your sales, support, and design teams to get honest feedback about your website from a customer’s point of view.   
• Identify the loopholes:Use qualitative and quantitative tools to spot trends that show how your website visitors interact with your website’s key pages.
  • Quantitative tools such Google Analytics to analyze bounce rate, page time spent, exit percentage, and similar metrics. 
  • Qualitative tools such as heatmaps to see where the majority of your website visitors are concentrating their attention, scrollmaps to analyze how far they’re scrolling down the page, and session recordings to visually see their entire journey.
• Explore the option of usability testing: This tool offers an insight into how people are actually using or navigating through your website. With usability testing you can gather direct feedback about visitor issues and draft necessary solutions. 

VWO Insights offers you a full suite of qualitative and quantitative tools such as heatmaps, scroll maps, click maps, session recordings, form analytics, etc. for quick and thorough analysis.

2. Define your goals and formulate a hypothesis

Successful experimentation begins by clearly defining goals. It is these goals or metrics that help prepare smart testing strategies around element selection and their variations. For example, if you’re trying to increase your average order value (AOV) or revenue per visitor (RPV), you may select elements that directly aid these metrics and create different variations. 

Once you’ve defined your goals and selected your page elements to test, it’s time to formulate a hypothesis. A hypothesis is basically a suggested solution to a problem. For instance, after looking at the heatmap of your product page you analyze that your “Add to Cart” button is not prominently visible, you’ll perhaps form a hypothesis that “based on observations gathered through heatmap and other qualitative tools, I expect that if we put the “Add to Cart” button in a colored box, we will see high visitor interaction with the button and more conversions.  

Here’s a hypothesis format that we at VWO use:

If you don’t have a good heatmap tool in your arsenal, use VWO’s AI-powered free heatmap generator to know how visitors are likely to engage on your webpage. You can also invest in VWO Insights to generate actual visitor behavior data and use it to your leverage.

3. Create variations

Post forming a hypothesis and having a good idea around which page elements you want to test, the next step is to create variations. Infuse your site with the new web elements or variations such as clearer and more prominently visible call-to-action buttons, enhanced images, text that’s more factual and resonates with what visitors expect, and so on. 

VWO provides an excellent and highly user-friendly platform to run a multivariate test. Using VWO’s Visual Editor, you can play with your website and its elements and create as many variations as you want without the help of a developer. Do watch this video on ‘Introduction to VWO Visual Editor’ to learn more. 

4. Determine your sample size

As stated in the above sections, it’s best to run MVT on a page with high traffic volume. If you choose to run it on a small traffic volume, your test is most likely to fail or take longer than usual to reach its statistically significant stage. The higher the traffic volume, the better will be the traffic split between variations, and hence, the better shall be the test results.

You can use our A/B and multivariate test duration calculator to find out how much traffic, and how long you need to run MVT based on your website’s current traffic volume, number of variations including control, and your statistical significance.  

5. Review your test setup

The next step to running an effective and successful MVT is to review your test setup in your testing app. You may be confident that you’ve taken all the necessary steps and added the variations correctly in your testing app, but there’s no harm thoroughly reviewing it one or two times more. 

One of the clearest advantages of conducting a review is that it gives you an opportunity to ensure every element has been added correctly and all the necessary test selections have been made. Taking the time to quality check your test is a critical step to ensure its success. 

6. Start driving traffic

If you think your webpage doesn’t have enough traffic to support the MVT experiment, it’s time to look for ways to do so. Your job is to ensure your page has as much traffic as possible to make sure your testing efforts don’t fail. Use paid advertising, social media promotion, and other traffic generation methods to prove or disprove the hypothesis you’re currently playing with.

7. Analyze your results

Once your test has run its due course and reached its statistically significant stage, it’s time to access it’s results and see if your hypothesis was right or wrong. 

Since you’ve tested multiple page elements at once, take some time to interpret your results. Examine each variation and analyze their performance. Note that it’s not necessary a variation that won which may be the best one to implement on your website permanently. Sometimes, these results can be inconclusive. Use qualitative tools such as heatmaps, clickmaps, session recordings, form analytics, etc. to examine the performance of each variation and draw a conclusion.  

After all, it’s important to ensure the validity of your test and implement changes on your webpage as per the preference of your audience and deliver the experience they want. 

How to run a multivariate test on a low-traffic website?

We’ve concluded time and again that MVT requires more traffic than an A/B test to show statistically significant results. But, does that mean websites with low traffic volume cannot do multivariate testing? Absolutely not! 

The theory behind MVT asking for high traffic is pretty obvious. The higher the number of variations, the higher shall be the traffic split between the variations, and hence, the longer it will take to draw conclusive results. If you’re planning to run MVT on a website with low traffic volume, all you need to do is make some modifications.

1. Only test high-impact changes

Let’s say, there are 6 elements on your product page which you believe have the potential to improve the performance of your page and even increase conversions. Do all of these have equal potential? Probably not. Some may be more impactful and have more noticeable effects than others.  

When you’re planning to run MVT on a low traffic website, focus your energies on testing those site elements that can have a significant impact on your page’s performance and goals rather than testing small modifications with low impact intensity. 

Although it may be intimidating to test radical changes, when you do, the likelihood of them showing a dramatic difference in conversion rate is also high. No matter the outcome, the learnings and valuable insights about your customer’s behavior and perception of your brand, can help run informed future tests and business decisions. 

2. Use fewer variations

Needless to say, low traffic volume means testing less number of variations. We understand that it’s tempting to test different optimization ideas to solve visitor problems. But, with every added variation in your test, the time to achieve statistical significance also increases. Don’t take that risk. Go small and go slow. This may cost you some extra efforts and resources, it will surely save you much time compared to running MVT with more number of variations. 

Use tools like heatmaps, scrollmaps, session recordings, usability testing, etc. to find high-impact page elements. Use the ICE model (impact, confidence, ease) to create a testing pipeline and follow it. 

3. Focus on micro-conversions

Your primary goal may be to increase page sign-ups, click-through rate or overall conversions, but does it make sense to use these as your primary metrics when you know it would take you much longer to gather enough conversions and even verify the test results? Surely not.

The better thing to do is test conversions on a micro-level. A level at which conversions are plentiful and can help you optimize your page quickly. For instance, focus your efforts on increasing your page engagement rate, clicks on add to cart button, clicks on images, etc. 

Other goals could be setting up a conversion goal that fires when a visitor fills up an exit-intent pop-up form, stays on your website for more than 30 minutes, or scrolls down a certain depth/folds through your long-copy page. You can also use a quantitative tool like Google Analytics to analyze which conversions to map or use as goals to optimize your website.   

4. Consider lowering down your statistical significance setting

When you don’t have the leverage to run a test on a large sample size, resort to using other methods to measure the performance of your control and variation. Do not wait for your test to reach its statistically significant level. If your testing tool allows, you can also lower your statistical significance levels. So, for instance, if you set your significance level to 70%, any version that reaches this mark will become the winner. In this case, you would also require a much smaller sample size than going for 99% significance. 

Optimizers across the industry recommend many ways to measure the performance of a test version, but the ones we recommend are as follows:

1. The 2-sample t-test: Also known as the independent sample t-test, it’s a method used to examine whether the means of two or more unknown samples are equal or not. If your sample distribution is unequal, you can use a different estimate of the standard deviation to check results. 
2. B. The Chi-Squared Test: The primary objective of this test is to examine the statistical significance of the observed relationship between two variants with respect to the expected outcome. In other words, it helps you analyze which version of your test is most likely to reach statistical significance and has better chances of winning.   
3. Confidence Interval: This method simply measures the degree of certainty or uncertainty of a variation to reach its statistical significance by observing the data at hand.  
4. Measuring sessions: This is another way to test the statistical significance of a variation. Rather than measuring your test’s performance by counting users, take into account sessions. This means that your test treats each individual as a participant in an experiment only once. 

5. Avoid niche testing

Avoid testing those sections or elements of your site that get very few hits. Instead, target page elements that get more traction. Site-wide CTA tests, landing page tests, and the like will help you take advantage of your site’s incoming traffic. Such tests are also likely to show statistically significant results in a shorter time span. 

What are the advantages and disadvantages of multivariate testing?

Instances in which multivariate testing is valuable

1. MVT helps measure the interaction between multiple page elements 

Let’s come back to evaluating the performance of your homemade chocolate eCommerce website. You’re confident that two sequential A/B tests can produce the same results as an MVT. So, you decide to first run an A/B test to compare a static banner image with a video banner, and the latter wins. Next, on the winning variation, i.e., the video banner, you do another A/B test between two possible CTAs, ‘Explore More’ and ‘Buy Now,’ and the former proved better. Don’t you think you could’ve come to the same conclusion with an MVT test?

Unlike an A/B test, a multivariate test gives you the leverage to test and measure how multiple page elements interact with each other. Just by testing a combination of various elements (let’s say your product page’s hero image, a CTA, and headline) you can not only figure out which variation performs better than others and helps increase conversions but also discover the best possible combinations of the tested page elements.

2. Provides an in-depth visitor behavior analysis

MVT enables you to conduct an in-depth analysis on visitor behavior and their preference patterns. It provides you with statistics on the performance of the variations vs their conversion effects. This helps re-orienting the visitor connection of your website. The better the re-orientation as per a visitor’s intent, the more are the chances of high conversions.   

3. Sheds light on dead or least contributing page elements. 

A multivariate test doesn’t just help you find the right combination of page elements that will help increase your conversions. It also sheds light on those that are contributing least or nothing to your site’s conversions and occupying huge page space. These elements can be anything, textual content, images, banner, etc. 

Relocate or replace them with elements that catch the attention of your visitors and channelize some conversions as well. It’s always better to have your page elements contribute something to your goals than absolutely nothing. 

4. Guides you through page structurization

The importance of placing elements at the right location can be understood from the fact that visitors today have a short view span. They devote maximum time reading and taking in information mentioned in the first fold of your web page. So, if you’re not placing the relevant content at the top, you’re reducing the chances of getting conversions by a great margin. MVT allows you to study the placement of various page elements and locate them at their right place in order to facilitate conversions for your business and make it easy for visitors to find what they came to look for on your page. 

5. Test a wide range of combinations

Unlike a regular A/B test that gives you the leverage to test one or more variations of a particular element in singularity, MVT allows you to test multiple elements at the same time by applying the concept of permutations and combinations. Such an experimentation method not only increases your testing options that you can use to tap on conversions but helps save time by avoiding running sequential A/B tests.  

Instances where  multivariate testing is not valuable

MVT is a highly sophisticated testing methodology. A single MVT test helps answer multiple questions at once. But, just because it’s a complicated testing technique doesn’t mean it’s better than other techniques  or that the data it generates is more useful. Every coin has two sides. We listed the pros of using a multivariate test in the above section. It’s time to know the cons as well. 

1. Requires more traffic to show statistically significant results

Unlike a traditional A/B test, MVT demands high traffic inflow. This means that it only works or shows statistically significant results for sites which have a ton of traffic. Even if you do run it on a site with low traffic, you’ll have to compromise on something or the other such as test fewer combinations, use other methods to calculate a winner, etc. Read the  section above on ‘How to run a multivariate test on low traffic websites’ for better clarity.  

2. They’re comparatively tricky to set up

One thing that makes people opt for an A/B test over MVT is that the former is comparatively very simple to set up. After all, you just need to change one or two elements and add variations while keeping the rest of the page design the same. Anyone who has an understanding of web design can easily set up an A/B test, and even complex A/B tests today rarely require more than a couple of minutes of a developer’s time. Tools like VWO enable even non-technical folks to set up an A/B test within a matter of minutes. 

On the contrary, MVT requires more efforts even if you’re creating a basic one, and it’s also very easy for them to go off the rails. A minute mistake in the design or while creating the variations can hamper the test results. MVT is a good option for optimizers who have a lot of experience in the arena of experimentation. 

3. There’s a hidden opportunity cost

Time is always of the essence and a valuable commodity for any business. When you run a test on your website, you’re investing time and playing with your conversion rate. There’s a hidden cost that you put on the line. Multivariate tests are comparatively complex and slow to set up, and slower to run. All the time lost during its setup phase and course of running creates an opportunity cost. 

Amidst the time MVT takes to show meaningful results, you could have run dozens of A/B tests and drawn conclusions. A/B tests are quick to set up and also provide definite answers to many specific problems.  

4. The chances of failure are comparatively high

Needless to say that testing allows you to move fast and break things to optimize your website and make it more user friendly. You get the leverage to try crazy ideas and even fail spectacularly without facing any real risk or consequences. While this approach seems effective in the case of an A/B test, we can’t say the same for multivariate testing. 

While each A/B test, despite its success or failure, provides a series of learning points to refer back, the same fails in the case of a multivariate test. It’s comparatively difficult to draw meaningful learnings as you play with a lot of elements that too in combination. More so, MVT is so slow and tedious that it doesn’t really make any sense to take such a risk in the first place and fail in the end.  

5. MVT is biased towards design

Another MVT con that most optimizers have realized over the years is that the testing method often provides answers to problems related to design. Some of the strongest advocates or supporters of MVT are also UI and UX professionals. 

Design is obviously important, but it’s surely not everything. UI and UX elements represent only a small part of all the total variables you use to enhance the performance of your website. Copy, promotional offers, and even site functionality are essential to ensure your website is liked by your target audience. These elements are often underestimated and overlooked in the case of MVT despite the fact that they have a huge impact on the conversion rate of your site.  

Machine learning and multivariate testing

The advancement in the field of technology, especially artificial intelligence, is now prominently visible in the testing arena as well. For many years, a program called a neural network was enabling computers to learn as they gather data and take necessary actions that were more accurate than humans while using less data and time. However, neural networks were able to help humans solve some specific problems only. 

Looking at the capabilities of these neural networks, many software companies thought to use their potential to develop solutions which could enhance the entire multivariate testing process. This solution is called the evolutionary neural network or a generic neural network. 

The approach uses the capabilities of machine learning to select which website elements to test and creates all possible variations on its own. It restricts the need to test all combinations and enables \ optimizers to test only those which have the ability to show highest conversions. 

The algorithms behind the solution prune the poor performing combinations to pave the way for more likely winners to participate in the test. Over time, the highest performing combination emerges as the winner and then becomes the new control. 

Once that happens, these algorithms then introduce variables called mutations in the test. Variants that were previously pruned are reintroduced as new combinations to analyze whether any one of these might turn successful amidst the better-performing combinations. 

This approach has proved to show better and faster results even with less traffic.

Evolutionary neural networks enable testing tools to learn which set of combinations will show positive results without testing all possible multivariate combinations. With machine learning, websites that have too little traffic to opt for MVT can also consider this option now without making compromises. 

Best practices to follow when running a multivariate test

MVT has the ability to empower optimizers to discover effective content that helps drive KPIs and enhance the performance of a website, but only when they follow best practices. 

1. Create a learning agenda

Before getting started, be sure that MVT is the best testing approach to your identified problems or whether a simple A/B test would best suit your needs. We, at VWO, believe that it’s important to first draft a learning agenda. This will help you define what exactly you want to test and what you hope to learn from this experiment. 

The agenda basically acts as a blueprint, helping you establish a hypothesis, define the page elements you want to test and for which audience segment, and prioritize learning objectives accordingly. It also comes in handy when you begin to set up your test and ensures that all adjustments have been made correctly. 

2. Avoid testing all possible combinations 

For most people, running a multivariate test means testing everything that comes under their radar. That should not be the case. Restrict yourself and test only those variables that you believe can have a high impact on your conversion rate. Also, the more the number of elements, the more shall be the number of permutations, the more it will be to run and gauge the final results.

Say for example, you want to test the performance of your display ad. You decide to test four possible images, two possible CTA button colors, and three possible headlines. This totals up to 24 variations of your display ad. When you test all the variations at once, each gets 1/24th traffic of the total incoming traffic. 

With such a high traffic split, the chances of any variation reaching its statistical significance is quite low. And, even if one or some of them do, the time they take will make the test insignificant.  

Furthermore, it’s not necessary that all combinations may sense from a design standpoint. For instance, an image with a blue background and blue CTA button color will make it hard for visitors to identify the CTA, especially on a mobile screen. Use good judgment and select only those variations which can show some results.  

3. Generate ideas from data for greater experimental variety and relevancy

While it’s a great practice to limit yourself from testing every possible idea that pops in your head, it’s also important to not ignore possibilities that could impact your conversion rate. To know if a variation is worth sampling, generate ideas from various data sources. These could include:

• First-hand data collected on the basis of visitor behavior, segment demographics, and interests.
• Third-party data extracted from multiple data providers for additional visitor information such as purchase behavior, transactional data, or industry-specific data.
• Historical performance data extracted from previously run campaigns targeting similar traffic.

4. Start eliminating low performers once your test reaches minimum sample size

It’s not necessary to end your multivariate test the moment it achieves adequate sample size. Rather, you should begin eliminating the non-performing variations. Shut down variations that have negligible movement compared to control once they’ve achieved the needed representative sample size. This means that more traffic will flow towards variations that are performing well, enabling you to optimize your test for higher quality results, faster.

5. Use high-performing combinations for further testing

Once you’ve discovered a few potential variations, restructure your test and fine-tune the variable elements. If a certain headline on a product page is outperforming others, come up with a fresh set of variations around that headline.  

You can even opt to run a discrete A/B/n test with limited experimental groups to analyze the performance of the new variations in a shorter time. Most experience optimizers suggest that when you learn something from an experiment, use the knowledge to enhance the performance of other page elements. Testing is not just about increasing revenue, but about understanding visitor behavior and serving to their needs.  

Best multivariate testing tools

The market today is swamping with A/B testing tools. Not all have the required capabilities to help you run successful experiments without a hassle. Neither can you take the risk of developing an in-house experimentation suite. Here’s a list of the top 5 multivariate testing tools for experience optimizers who have a passion for testing:  

1. VWO

VWO is an all-in-one, cloud-based experimentation tool that helps optimizers run multiple tests on their website and optimize it to ensure better conversions. Besides laying an easy-to-use experimentation platform, the tool allows you to conduct qualitative and quantitative research work, create test variations, and even analyze the performance of your tests via its robust dashboard. 

VWO also offers the SmartStats feature that runs on Bayesian statistics. It gives you more control of your tests and helps derive conclusions sooner. Sign up today for a free trial and get into the habit of experimentation.

2. Optimizely 

Optimizely platform offers a comprehensive suite of CRO tools and generally entertain enterprise-level customers only.   

Essentially, Optimizely primarily provides web experimentation and personalization services. However, you can use its capabilities to run experiments on mobile apps and messaging platforms as well. You can even opt to run multiple tests on the same page and rest assured of accurate results.  

3. A/B Tasty

A/B Tasty is another experimentation platform that offers a holistic range of testing features. Besides the usual A/B and Split URL testing option, it also allows you to run multivariate tests. It has an integrated platform that’s easy-to-use and offers a real-time view of your tests and their respective confidence levels. 

4. Oracle Maxymiser

An advanced A/B testing and personalization tool, Oracle Maxymiser enables you to design and run sophisticated experiments. The platform offers many powerful features and capabilities such as multivariate testing, funnel optimization, advanced targeting and segmentation, and predictive analytics. Such features make it a perfect match for data-driven optimizers with an in-house IT support team.

5. Google Optimize 360

Google Optimize 360 is a Google product that offers a broad range of services besides experimentation. Some of these include native Google Analytics integration, URL targeting, and Geo-targeting. If you opt for Google Optimize 360’s premium version, you get to explore the tool in-depth. With Google Optimize 360 you can:

• test up to 36 combinations when running MVT
• run 100+ test simultaneously
• make 100+ personalizations at a time
• get access to Google Analytics 360 Suite administration

Conclusion

Multivariate testing is an arm of A/B testing that uses the same experimentation mechanics, but compares more than one variable on a website in a live environment. It opposes the traditional scientific notion and enables you to, in a way, run multiple A/B/n tests on the same page simultaneously. At the core, it’s a highly complex process that requires more time and effort, but provides comprehensive information around how different page elements interact with each other and which combinations work the best magic on your site. 

If you still have questions around what is multivariate testing or how it can benefit your website, request a demo today! Or, get into the habit of experimentation and start A/B testing today! Sign up for VWO’s free trial.