Experimentation Induces Learning: 5 Examples That Are Proof

Have you ever played an outdoor sport? 

If yes, then you will agree that training is a very important component of outdoor sports. The training simulates the real scenarios of the game to identify shortcomings and make improvements. Yet, its benefits extend beyond mere performance enhancement, as it also facilitates learning in areas such as nutrition and psychology, ultimately leading to an overall improvement in one’s sportsmanship.

Similarly, conducting online experiments is like training your digital property. It not only enhances your conversion rate but also provides valuable opportunities to gain insights into various facets of the digital landscape, resulting in the overall growth and development of your company.

Don’t believe it? 

In this blog, we have aggregated five real-life examples to showcase how experimentation induces learning related to various aspects impacting your visitors’ digital experience like content, conversion funnels, experimentation approach, and design. 

Let’s begin.

1. Dell Technologies

Dell Technologies, a technology giant, places significant emphasis on experimentation. The company’s commitment to this practice is exemplified by CEO Michael Dell’s occasional communication with the Digital Analytics team regarding ongoing tests. Experimentation has not only made a positive contribution to the multi-billion dollar revenue, but it also has induced some learnings for the company. 

In one of the fireside chats with VWO, Narayan Keshavan, Head of the Design Analytics Team, shared his insights. The following learnings helped them improve their whole experimentation cycle and approach.

a. Don’t follow experimentation trends blindly

In one of their experiments for the website in China, the team at Dell tried to add a promo banner in the Navigation bar, a trend followed by a leading eCommerce website in China. But for Dell, the conversion rates dropped by adding a banner.

They learned that a trend should be followed carefully as every website is unique, and the user experience analysis should preside over each experiment.

b. Keep a neutral eye on an experimentation idea

The Dell Technologies team found that it’s natural to expect a lift in conversion rate from variations. But, the foundation of the whole effort is to run an experiment and not prove something. Thus, the optimization team must have a neutral approach toward each idea. It helps negate biases while running the experiment and post-analysis of the data.

c. Don’t fall for testing myopia

Testing myopia is a term used by the Dell Technologies team, which means you are so focused on improving a metric that you forget the effect on other metrics. 

Let’s say you aim to enhance the Click-Through Rate (CTR) of the primary Call-To-Action (CTA) button on one of your landing pages. You conduct tests using captivating banners and observe a surge in the CTR of the primary CTA. While this experiment may seem successful based on this one metric alone, a deeper analysis reveals that the bounce rate has increased due to a decrease in website speed caused by the banner’s size. As a result, even though the conversion rate has improved from people who engage, many visitors are leaving the site without engaging because the page takes too long to load.

Therefore, it’s crucial to take a holistic approach to experimentation and analysis instead of obsessing over a single metric.

Watch the complete webinar:

2. Uber Eats

Uber Eats started its operation in South Africa as an online food delivery service. But can you imagine that it is now a virtual mall that sells jewelry, clothing, and other accessories? Here is the screenshot of its product page:

Starting with the food delivery and diversifying to become a virtual mall happened because of experimentation. Uber Eats in South Africa has ingrained a culture of experimentation, where they depend less on market research and more on experimentation in the real world. Experimentation has allowed Uber Eats South Africa to grow 149% month-on-month in its non-restaurant vertices. 

Here is what the experimentation taught them and facilitated expansion outside food delivery.

a. A tool with an easy setup and automation enhances the experimentation culture

Uber Eats’ team recognized that a simplified experimentation setup means less dependency on development and fewer entry barriers, which motivates team members to partake more in experimentation. Also, by automating post-analysis and deploying the winning variation, the workload on the team is reduced, which speeds up the experimentation rate. It collectively promotes a culture of experimentation within the company. 

b. It’s important to take time while creating a hypothesis

The hypothesis is the foundation of the experimentation, and even simple-looking tests might turn complicated if there is a fault in the hypothesis. A hypothesis must be based on data insights and behavioral analysis. Also, the hypothesis must specifically mention the website element that needs to be tested and the expected improvement that you expect at the end.

For instance, if a CTA button on a landing page with a low conversion rate is identified through heatmap analysis to have fewer clicks and less attention, changing the button color to make it more prominent could be a potential solution. In this case, the hypothesis could be: “Changing the color of the CTA button to X color will result in a Z% increase in conversion.”

In addition, the team at Uber Eats advocates for testing one hypothesis at a time with various methods like A/B testing and Multivariate testing rather than testing multiple hypotheses with the same method to achieve better results.

3. MakeMyTrip.com

MakeMyTrip.com is an online travel aggregator in India that gets 44 million visitors per month. The two-decade-old company has first-hand experience with digital transformation and changing user demands. The experimentation culture has positively contributed to the 64.1% Y-O-Y growth that the company saw in 2022.

In one of VWO’s webinars, Aakash Kumar, head of the user experience at Makemytrip.com, revealed some critical experimentation-induced learnings. 

a. Fewer purchase cycle stages don’t necessarily mean more conversion

It’s a general notion that a few stages between awareness and purchase will lead to more conversion. However, the design team at MakeMyTrip.com found something interesting.

Their typical hotel search user flow started with searching for destinations on the homepage, which led to a page with a list of hotels. On clicking a hotel tab, a page opens with a hotel description followed by offered accommodation types. The team noticed that many mobile users who ended up converting were choosing the cheapest room option and there was a low conversion rate for premium rooms. 

After conducting a thorough analysis of user behavior and data collection, the team found that the hotel details page’s description was causing users to scroll excessively, leading them to select the first room type they saw after scrolling. To address this issue, the design team introduced an additional step that allowed users to select the room type before completing the transaction.

The whole optimization effort led to an increase in the booking rate for premium room types. 

b. Design changes should be gradual and not drastic

Drawing on their decades of experience in experimentation, the design team has discovered that design changes aligned with user needs should be gradual rather than drastic. Following the ideate-experiment-learn cycle during website design experimentation is crucial. Subsequent design changes can be based on the team’s previous experimentation and learnings, which helps to minimize risks and reduce negative impacts on the conversion rate.

Watch the complete webinar:

4. Netflix

Netflix has a culture of experimentation and learning, which helped it transform from a DVD mailing service to a streaming service giant with 230+ million paid subscribers worldwide. Here is what Netflix has learned about experimentation from decades of experience.

a. A successful idea can come from an unlikely source

Netflix has an open culture where anybody can see the running tests and associated results. The naturally low win rate of experiments brings humility and humbleness to the team. It opens up doors for every team member to share ideas and experiment with user experience. There are instances at the company where ideas from experts have failed, but ideas from executives and engineers have brought unexpected wins. The democratization has enabled them to get massive revenue-generating ideas from unlikely resources. 

b. A failed experiment in the past can achieve success in the present

Customer behavior and preferences are evolving, and at Netflix, they have found that revisiting past failed ideas and deploying them at present can find success. Sometimes, the analysis of the failed ideas suggests that they may achieve success in the present or the new user group. So creating a repository of each experiment along with its analysis is beneficial to redeploy and find success in the future.  

5. Frictionless Commerce

Frictionless Commerce is a 13-year-old digital marketing agency with a focus on enhancing the sales of Shopify websites by utilizing buyer psychology techniques to influence and convert new buyers. The agency has helped 118+ DTC brands and carried out A/B tests for them to improve conversion rates. Here is what the people at Frictionless Commerce have learned about buyer’s psychology due to these experimentations, which can help create copies and headings for landing pages.

a. Showcasing expertise and brand story create a sense of connection

People like to associate with brands who are experts in their domain and specialize in crafting the product or service. Additionally, a brand’s narrative, detailing its origins and the obstacles it overcame, can foster a sense of connection with the visitor and drive them further down the path to purchase.

b. Adding surprising facts can negate the midway quitting

Behavior analysis can help you understand the scroll depth at which most visitors leave. The team at Frictionless Commerce has found that adding a surprising fact at such points can reduce the midway quitting as it gives a burst of energy to a visitor to know more. 

c. If doubts are not solved, then the visitor will leave

If a landing page fails to address a visitor’s specific query, they may feel hesitant to trust the product or service being offered. The team discovered that four particular factors – the absence of features, unclear pricing, subpar features, and confusing design elements – can all contribute to a negative perception and generate doubts in the visitor’s mind.

To avoid these pitfalls, marketers should aim to view the landing page from the visitor’s perspective and ensure that any potential doubts or concerns about features, pricing, or design are addressed and resolved through testing.

This learning, along with other insights, helped the company in shaping A/B testing campaigns for their clients. 

One of their clients is into selling shampoo bars. The team at Frictionless Commerce carried out an A/B test which was influenced by the above learnings. Here are the controls and variations of the test created with VWO. You can see that the variation is trying to solve a doubt that will come naturally to a visitor’s mind. 

The test resulted in a 5.97% increase in the number of orders. 

Begin your experimentation journey with VWO

The five examples show that experimentation not only enhances the conversion rate but also generates valuable insights that can lead to new user experiences, business ventures, and cultural shifts, ultimately propelling the company to greater heights.

Also, we saw in the case of Uber Eats, one of the key contributors to success is the availability of a user-friendly experimentation tool that enables the rapid scaling of experimentation efforts. 

However, choosing an experimentation tool is not an instant decision. You have to evaluate the compatibility with your business and get hands-on experience with the tool before spending a buck. 

To facilitate this process, we have created a 30-day free trial that provides full access to all the offered capabilities without any financial obligations.

So, If you are new to experimentation or considering a switch from your current setup, we recommend reading our detailed guide on VWO’s free trial, which provides complete information about working with the world’s leading experimentation platform.