Strategic Splits: A Guide to A/B Test Segmentation

“105% uplift in click-through rate!”

It’s not uncommon to anticipate growth in A/B testing. But how did JellyTelly, an internet-based television network with a low sign-up rate on the homepage, achieve such explosive growth? Well, the agency that handled CRO for Jelly Telly started like everyone else—formulating hypotheses and creating a simplified homepage variation for the A/B test. What set them apart was their savvy move of segmenting only new visitors for the A/B test, leading to a solid uplift in results.

You might wonder what’s the deal with segmentation in the A/B test and why it makes such a significant impact.

To put it simply, segmentation is similar to the meticulous strategies employed by teams in the Big League, where they segment opponents’ team players based on recent form and performance and experiment with their gaming approach. No one-size-fits-all strategy here — on such a professional and super-competitive level, the teams adjust their game plan for both in-form and out-of-form segments of opponent batters and pitchers. This approach ensures that a team can apply strategies in different situations, understand players effectively, and utilize teammates’ potential.

Similarly, you can’t always have a general A/B testing approach when you have multiple segments of visitors on your digital platform. For example, it’s far more effective to A/B test your checkout page for frequently abandoning segments than all the visitors. Segmentation gives specific insights into visitor behavior and optimizes the bandwidth of your A/B testing tool. 

So, if you’re just dipping your toes to explore segmentation in A/B testing, you’ll definitely want to check out this blog post. We’ll cover the how-to’s of A/B test segmentation and best practices and show you how easy segmentation is when you are using a platform like VWO.

Let’s understand how to segment A/B tests.

When you’re running an A/B test, you can carry out pre and post-segmentation. Pre-segmentation is all about digging into existing visitor data to create visitor segments for a test. And post-segmentation? That’s breaking down the results you get from the test. Here are some commonly used filters to create segments for pre- and post-segmentation in A/B testing:

• Geo: USA, India, United Kingdom.
• Device: Mobile, tablet, and desktop.
• Traffic Source: Direct, search engine, referral, affiliates, social media, paid, etc.
• Activity: New visitor, returning visitor, event abandonment, event complete, etc.
• Other: Time of activity, browser.
• Custom segments: Business-specific segments.

[For instance, consider using a country filter to create segments like the US, India, UK, etc.]

Now, let’s take a closer look at how pre- and post-segmentation can be useful in your A/B testing.

Pre-segmentation: The slice before the spice

“I am seeing a rise in cart abandonment, what should I do?”

“How do I reduce that bounce rate from Japan?”

“How can I improve the completion rate of that course?”

These questions naturally pop up after digging into the website data from our analytics platform. Solving these issues might involve A/B testing with some hypotheses you come up with. Well, here pre-segmentation can come in handy. Instead of testing everything on all visitors, here is how you can solve the above problems and similar ones that you face:

“I will segment cart abandoners and run an A/B test by positioning the promotion block above the fold.” 

“Let’s A/B test Japan traffic with a translation widget in the variation.”

“Let’s A/B test low-engagement students segment with a variation having smaller duration videos.”

With this approach of pre-segmentation, you will more likely see an increase in your conversion rate, as you are targeting segments that are a key source of the problem. Even if it fails, you can move to test the next hypothesis without subjecting a larger visitor base to the experiment. Also, pre-segmentation optimizes your use of the A/B testing tool as many of these platforms charge based on the usage-based pricing model. 

Keep in mind: The effectiveness of pre-segmentation hinges on the tool at your disposal and how adeptly you handle and assimilate visitor data. Contemporary businesses lean on sophisticated solutions, such as customer data platforms (CDPs), to store and seamlessly integrate data from diverse sources. This not only enhances the accuracy of segmentation but also boosts the results of activities driven by these segments.

Post-segmentation: Unraveling the puzzle

You have finished A/B testing, and now it’s time to delve into the reports with post-segmentation. It might seem like a bit of an optional thing, but whether your A/B test hits the jackpot or takes a detour, post-segmentation is the secret sauce—it spills the beans on why things went the way they did, laying the groundwork for future improvements.

In post-segmentation, you can filter all the visitor data for a particular segment to understand how it performed or you can square off two segments like mobile versus desktop or returning visitors against the new ones, revealing the nuances in their experiences and how it impacted the conversion rate. 

Moreover, delving into post-segmentation after a test flop can bring you closer to understanding customer expectations and needs. Take the example of Uncommon Knowledge, an online education portal. They hit a roadblock in their A/B test featuring the latest design trends. Still, the post-segmentation analysis revealed a key insight: their primary audience predominantly falls within the 45+ age group, who are not used to the latest trends.

So, when you engage in post-segmentation after an A/B test, you’re not slamming the brakes on A/B tests; you’re building on them. This approach addresses the common challenge of hitting a creativity wall with A/B testing as you generate fresh hypotheses through in-depth analysis. It is a more strategic approach—what we like to call the experimentation loop—to A/B testing. In this loop, you launch a new experiment based on a hypothesis from analyzing a completed A/B test. It’s a far superior method compared to the randomness of traditional testing.

Now, let’s dive into the practical side of things and explore how to effectively carry out segmentation using a tool like VWO.

How does VWO handle pre- and post-segmentation?

Let’s understand the segmentation functionality in VWO with an example. Say you’ve got a cool smartwatch e-commerce store. In recent analytics reports, you notice a decline in the conversion rate among visitors using the Windows browser. The analysis led to a hypothesis of adding a sitewide time-based discount offer widget to see whether it uplifts the conversion from Windows visitors. You create an A/B test variation having the widget below the main menu.

With VWO, if you are thinking of running an A/B test to see the effect of this offer on visitors with a Windows browser, you can pull it off with just a few clicks, thanks to its user-friendly dashboard. Once you’ve punched in the web URL you want to test and sort out your variations and goals, it’s a breeze to pre-segment using the standard segment option tucked away in the audience and traffic section. Here is how it’s done on VWO.

Now, for the same time-based widget, if you want to kick things up a notch and dive into some advanced pre-segmentation —like you have a hypothesis to A/B test visitors who have a 60-second time spent duration on the homepage. For such use cases custom segment is your playground. You can segment users by behavior-based segments such as exit intent, time spent, and scroll-based criteria. Additionally, you can segment by location, IP address, query parameters, or anything specific to your site.

Here’s how it works:

Also, you can include, exclude, and even mash up segments for your A/B test. For example, a combined segment of people who have 60 seconds spent on the watch store homepage from Idaho. 

You can even create segments based on attributes you’ve tracked with third parties like Clearbit, Segment, etc. If you want to know more, dive into the nitty-gritty of custom segments with all the details in a knowledge-base article.

Now, let’s say you’ve conducted the A/B test for all users, and you’re keen on diving into post-segmentation using VWO to see how control and variation fared across different segments. Beyond the standard and custom segments we explored earlier in pre-segmentation, VWO has got you covered for comparing how the A/B test played out for various segments of visitors. Like, you can see how conversions from the desktop segment fared against the mobile segment. You can even compare multiple-goal performances. 

Also, imagine you’ve got a boatload of data, and you’re itching to dig deeper into specific segments. Like, maybe you’re curious to slice and dice based on countries for the mobile segment. In that case, VWO lets you slice and dice those segments with visitor dimensions, making it a breeze to unravel the nitty-gritty details of the performance. 

If you notice, VWO is like your trusted toolkit, handing you all the controls you need to finely segment your A/B tests during both pre and post-stages. If you find this intriguing and want to give it a spin, we offer a 30-day full-feature trial for a complete hands-on experience. Remember to give it a go! It’s worth mentioning here that access to some of the above features depends on the chosen plan and pricing. 

Now that we’ve got the scoop on segmentation and the tool to wield, let’s dive into some savvy best practices to make the most out of it.

Best practices for A/B test segmentation to kick off on the right foot

a. Prioritize bigger segments for A/B testing

After applying basic filters like country, age, and device in your analytics tools, spot the segment with the most visitors. Make it your A-list for testing, and break it down even more based on conversion rates.

Now, for the high-converting sub-segment, A/B testing with new features, the latest arrivals, or step up your game with website personalization. These segments are your core visitors. If an A/B test variation clicks with them, consider integrating it into the regular experience for all visitors.

For the low-converting sub-segment, dive into the analysis using user behavior analytics tools like heatmaps, session recordings, and clickmaps to pinpoint friction points. Formulate hypotheses based on your findings and run A/B tests to observe the impact on the conversion rate. It’s about diagnosing the issues, crafting solutions, and fine-tuning for better results.

b. A/B test new experiences for small segments with high conversion rates

Using the same filters as before in your analytics, you might spot small segments with a high conversion rate. This could be a sign of an untapped visitor base. For these segments, consider crafting a new experience or even developing a variation landing page dedicated to this specific segment and conduct a Split URL test. To test the waters, increase traffic through paid and social promotion to observe if the newly found visitor base maintains a consistently high conversion rate. Such an approach to experimentation helps in exploring and capitalizing on hidden opportunities.

Say, for instance, in your analysis using a country filter, you might stumble upon a small segment of visitors from Canada (a region you don’t usually target) with surprisingly high conversion rates. In this scenario, consider crafting a landing page tailored specifically for Canadian visitors and test it against your original landing page using a Split URL test. To validate your findings, run a paid promotion to drive more traffic and ensure statistical significance in your tests. If the results are promising and show a positive trend, you might consider implementing long-term strategies like content marketing to fully tap into this previously untapped market.

c. Dive into post-segmentation only after the test reaches statistical significance

The allure of stopping an A/B test midway can be strong, especially when you see a higher conversion rate for a variation, thanks to the real-time reports most tools provide. However, this impulsive decision might lead to a false positive (type-1 error) result, and conducting post-segmentation based on this premature data could lead to inaccurate assumptions for future actions.

It’s wiser to let the A/B test reach statistical significance before diving into post-segmentation. Market-leading tools like VWO showcase significance in their reports, but if your tools don’t showcase it, then you can turn to online statistical significance calculators to make informed decisions about stopping the A/B test. Patience can save you from hasty conclusions and guide you toward more accurate insights.

d. Combine segments to get more precise with your experiments and results

As your business expands and website traffic surges, it becomes crucial—and almost mandatory—to merge segments for both pre and post-segmentation in A/B testing. For example, instead of solely running an A/B test or analyzing reports for people residing in Idaho, consider this: for pre-segmentation, it’s more effective to create a combined segment of individuals living in Idaho using the Safari browser. With a more targeted pre-segmentation, you’re not just scratching the surface; you’re delving into the nuances of a specific group, enhancing the precision of your testing strategy.

Now, when it comes to post-segmentation with combined segments, you gain a richer understanding and key insights compared to broader segments, especially when dealing with a high volume of visitors. It’s about fine-tuning your focus to extract more meaningful information from the data deluge. 

Note: VWO lets you combine segments via its custom segments feature. Here is a screenshot of it. 

Final thoughts

Segmentation is a fantastic addition to your experimentation mix, significantly elevating your understanding of visitor behavior. Incorporating both pre and post-segmentation into your strategy can be a game-changer. However, finding the right approach for your business might take some experimenting. The learning curve can be steep initially, and you might encounter deviations from your assumptions.

But here’s the key—keep building your road based on the data you acquire. Soon enough, you’ll witness the potent impact of segmentation driving your A/B testing programs. It’s a journey worth taking for the valuable insights and optimizations it brings to your A/B testing game. 

Frequently Asked Questions