Holiday A/B Testing Strategy: How Invesp Generated $37 Million During Holidays

I take care of regional marketing for the US. I’ve been with VWO for about 4 years. It’s very fascinating that I started my career in marketing and in my first job itself, I did a lot of A/B tests and experiments. It was not at VWO, of course. a few years later, through the various organizations that I’ve worked at, I continue to do A/B testing. And incidentally, I joined an organization, a product that specializes in A/B testing and helps people conduct it.

So it has been an absolutely, wonderful journey. I felt passionate about VWO A/B testing. It’s a very scientific process, and I try to imbibe it myself too. And we do a lot of knowledge-sharing, sessions with people from the industry. Today, we have, Khalid, who has been doing conversion and optimization since 2006. I’ve been doing it for 9 years, and I can only, imagine how much, richer an experience, Khalid has, with 8 additional years of conversion and optimization.

Khalid mentioned that with Invesp he has run more than 25,000 A/B tests. That’s unbelievable for one person to have gone through so many experiments. and I’m sure that, Khalid is going to give us a gold mine of knowledge and things that we can implement right away for the upcoming holiday season where the college has been able to, generate upwards of $37,000,000 in revenue. I’m super excited to hear from Khalid, and I would love to hand over the stage to him to take this forward from here.

Khalid Saleh:

Thank you, sir. Thank you for the kind words. And, let me just share my screen. Here we go. Where is the screen share?

Oh, here. it’s funny, as people are joining, In my previous life prior to Invesp, I used to be a software architect at Go to Meetings and Go to Webinar. So, it’s always interesting to come to the software. So let’s see. And can everybody see my screen?

I’m assuming that I’m sharing my screen right now. Okay. And it should show basically how we generate the 37,000,000 dollars just the webinar title. Can you guys see if somebody in the chat can just at least confirm that to me? Because one of the worst things that could happen is, okay. Good. so I’m gonna start you guys with a quick story.

By the way, if you have any questions at any point in time, use the chat feature and just ask any questions that you might have. Back in 2017, I still recall this as if it was yesterday. I got a call from the VP of Marketing at Z Gallerie.

For those who are familiar with Z Gallerie a large furniture retailer in the US, Karim Ozguai, who was the VP of Marketing at that point in time, called me up, and he’s like, hey, Khalid. Can you guys help us? That was 2 weeks prior to Black Friday. Hey. Can you guys help us improve our conversion rates?

During Black Friday, Cyber Monday, and now 2 weeks is a very short period of time. But I was too excited because who doesn’t wanna get the gallery as a client? So I said, yes. Long story short, basically, Karim had a goal, and he knew that their sales during Black Friday showed up, but he wanted to increase it. He wants to increase his Black Friday sales by 25% compared to the previous year. So the sales will anyways increase. And the goal was to achieve a 25 percent additional increase on what they were expecting. I didn’t really know what we signed up for because we ended up spending 2 weeks living in the office. At some point, I think there were some people, who were sleeping in the office.

Holidays come in Black Friday, that Thursday night, from basically Thursday night until Tuesday night. That was the period that we focused on. We did a ton of testing for them, and we’ve managed to increase sales during that period by 39 percent, and Karim, who’s, a good personal friend at this point, did an amazing case study with us. So what we said is like, hey, this is really a huge opportunity. How can we take the learnings that we did back in 2017 and every year try and improve it more and more?

So every year, usually, we work with retailers, with SAS companies, with anybody who has an offer during the holidays, and we tell them this is the time for you to do really absolutely amazing. We’ve made some silly mistakes along the way, and we’ve learned a few things. So what I’m gonna do is I’m gonna share with you some of the lessons that we’ve learned, over the last 4 or 5 years of heavy testing during the holidays. And let me just see over here. Okay.

So what are we gonna cover today? First, I’m gonna answer the question because I hear it all the time. I was just talking to a fairly large organization yesterday, and they said, hey, we do a code for you. Does it make sense to run experiments during the holidays?

That’s the first question. Number 2, I’m gonna show you a lot of examples. I think that’s why everybody is here. You wanna see examples and that’s worked. And, hopefully, steal some of those ideas. We don’t mind that if you do, you will learn something useful. I’ll talk about the team makeup if you are gonna do any type of testing during the holidays, then we’re gonna open it up for Q and questions and answers quickly about investment. I’m gonna spend just 30 seconds on this investment’s 2nd CRO agency started back in 2006. So we’ve been doing this for about 16 years. On average, we deliver a 65% lift in conversions in 12 months. We’ve done 900+ projects. This says 24,000 as this is from the beginning of the year. So we’ve done about 25,000 A/B tests, and we’re working with companies in 12 different countries. 

Should we test during the holidays? That’s kind of the big thing. Prior to being on the conversion side of things, I used to be the software architect who used to say, code freeze 1st week of November, we don’t touch the code until the new year. We might be working on new features, but we didn’t push anything live to the website.

So I can fully understand why people are worried about that. There are a couple of things. If you think about people who come to your website during the holidays, in general, there is a segment of visitors who are highly motivated. They’re ready to buy, and that segment is usually small, anywhere between 2 to 3%. It depends on your website conversion rates. During the holidays, people are highly motivated.

So that segment naturally will increase. And if we’re able to remove the hurdles out of there if we’re able to give them the right offers, if we’re able to nudge them along, what you will see is you’re able to capture more conversions. So from a pure revenue opportunity is absolutely amazing during the holidays. But, of course, you still have something kind of the elephant in the room. Well, how about bugs?

I mean, yes. Great. You’re trying to increase our sales. But what if things actually get a lot worse and we introduce a bug? And instead of making money, we end up losing money.

And because of a bug that we introduced some production. So that’s the concern that usually management and technical teams have, around testing during the holidays. For that, there are a couple of things that I always mention. Number 1 is whether it’s the holidays or not, you should never roll anything out. You should never test anything on your website without a really solid quality-controlled quality assurance process. Otherwise, yeah, you are shooting yourself and the food, and you wanna avoid that completely.

Now in order for us to do a solid quality control process, that means when you have the experiments, the designs, and the messaging already for the holidays or at least 5 or 6 days before you launch them, you’ve tested them. You make sure that there are no bugs. And after you launch those experiments, sometimes people say, oh, I lost the experiment and then they just go away. No. Don’t do that. It’s the holidays. It’s too scary. I wouldn’t do that. What you need to do is you need to monitor the results.

Are we seeing something abnormal? Are we seeing rates for one of our variations, for one of, like, you know, the tests that were running drop or report zero conversions? Usually, that tells you that there is a bug so you need to have that team on standby so you don’t shoot yourself in the foot. That’s number 1. Number 2, if you are worried that as you are doing testing, maybe you’re gonna reduce your conversion rates. Of course, there’s the upside of improving your conversion rates, but guess what? What you would do, especially if you have the traffic is to gradually release the the test or the experiment to your wider audience. So I would start an experiment with 10% of my visitors. And if the experiment is doing well, then I’ll increase that maybe to 20%, 25%, 50%, then I go to 100%. So that also in case, one of your variations doesn’t perform really well, that way you’re gradually rolling out the experiments and you don’t cause any problems. Those are the 2 things. Now, as you saw in the webinar title, last year, we managed to generate $37,000,000 in additional revenue for companies on top of what they were planning to generate. So every company we work with, we tell them, 

‘Hey, what’s your revenue target for the holidays’? $2,000,000 great! 

What’s your revenue target for the holidays? Half a $1,000,000. 

What’s your revenue target during the holidays? $12,000,000. Okay. 

What if we do 20 or 25 percent better? How would that look like?

And that’s what we did with Z Gallerie. Let’s look into examples and lessons because that’s why you guys are here. And I’m gonna start with each one of those examples and with each one of those lessons, I’m gonna start with kind of a general theme, and then I’ll show you an example of experiments. So one of the things that I see consistently is that people go wild with the number of offers that they have. Or maybe we’ll offer a 50% discount, 20% discount, or 30% discount, and you end up with the paradox of choice.

Correct? Too many offers, too many competing offers. I’m not sure where you want me to look, and I don’t end up making a decision. So I’m gonna show you an example, and this comes actually from Z Gallerie. So when we first and this is back from 2017, this was the very first experiments that we did with them.

So they had items that were up to 50 percent off grades. Very strong offer, but they said, you know what? We also have, like, you know, 50 percent off our best deals. We have 40% off, gifts of glam. 

They had other offers Like, hey if we don’t capture them with the 50%, we’re gonna capture them with the 20%, we’re gonna capture them with the 30%. And when we came we said, guys, not very good. If we’re gonna put our, best foot forward, we are gonna focus on the 50% offer. So if you notice over here, We had this 50%, Black Friday savings up to 50% variation to 50% off. And we’ve tested 2 different variations over here. One is showing the steel. 1 is the gifts of glam that they had and by the way, notice something over here.

They had a 40% off gives up plan. We opted to remove that 40% completely. And we said we are just gonna go with the main offer of 50% off. How well do you think this test performed? Let’s see if people sometimes get bored, by the way, talking just like, you know, and hearing my voice. So what do you guys think? Which variation do you think increased conversions?

Let’s use the chat feature, guys. V1 or V2. I mean, we have a winner over here, so I’m telling you that. But what do you think? Do you think the hottest deals work better, or do you think the gift of glam works better?

Give people a chance to think about this for a second – which design? Now if you just purely ask me, and I, by the way, there are so many different case studies in this presentation that I actually even forgot the result. I told you probably V.1 because you have the hottest deals, but, how about we jump in and we take a look at the Answers? Okay. So there was, by the way, a third variation, but variation 1. So if we go back, the hottest deal is the one that did really well, and it gave them an 11.51% increase in conversions with a high chance of winning – 96.62% chance to win, absolutely killed it for them. 

And by the way, when you’re a company like Z Gallerie in this case, you run these tests and you run them are, like, you know, very quickly, and we’ll talk about this. During Black Friday and Cyber Monday, you have to be very aware of how long it takes you to run a test, but they do close to $50,000,000 just during the holidays. So for them, an 11.51 percent lift in conversions absolutely killed it.

Something else that we’ve learned in the last few years is you need to emphasize the offers that you have and add a little bit of scarcity let’s take a look at another company, and this comes from last year. So this is their kind of typical product page. Correct? You have your ads of carts, you know, the item, and they sell, as you see over here, just some bags. We looked at it, and we’re like, okay.

This is great. You have some offers, you know that you are displaying on top. How about we change this? They already have, like, you know, this 30% discount. Let’s add the offer.

Let’s add it next to the CTA so people will see it. And then, like, you know, we’re just sort of reminding them of the 30% discount. and they have a very funny way of naming their discounts, gift tanks, 30% off. So that’s kind of the campaign that they were running.

Is that enough? No. Let’s add a little bit of urgency, and that’s our V2, where we show a timer when this offer is gonna run this is gonna run out in 2 days, 22 hours. So that’s an account of the time, a counter, or a timer that basically shows how much time remains. And the offer Again, we’re adding urgency over here.

It’s very close to the add-to-cart. Sort of makes sense. And most people say, okay. Yeah.

This makes sense. Okay. So instead of having an offer will add a timer. But how about we do something a little bit interesting? How about we mention the offer below the add-to-cart?

Okay. So, again, the add-to-cart there and then the offers below. And in V4, again, we did the same thing. So it’s very similar. V1 and V2 are just all about the placement.

So your hypothesis, if you think about this, every A/B test starts with a hypothesis, correct? Showing the offers and adding urgency will increase conversions. Great. That’s a good hypothesis, but a single hypothesis could be implemented in multiple ways. And as you see over here, those 4 different designs that you saw all achieve the same hypothesis lots of times people tell me, well, yeah, like an opposite hypothesis and failed. No.

There are so many different ways that you can test the hypothesis. So I’m gonna go back over here, and I’ll show you v 1. So the 30% off on top, V 2, 30% plus the timer V 3. Oh, same.

The 30 percent off, but it’s below the CTA. And V 4, we are showing again the timer plus the 30 percent off. What do you guys think? Which one of those designs actually increased conversion? Now if you purely ask me, I would tell you most likely just based on all the different usability principles most of it’s V2.

That, I mean, you know, there’s some close proximity to the CTA. It makes a lot of sense. Let’s take a look at the actual results. And this was sort of like, you know, sort of throw us off a little bit, but we’re like, hey, we’ll take this win with a 14.7% lift, where we had the offer, and then we had the timer below the CTA. By the way, we do have a playbook that has all these examples available. Just shoot an email to convert at the Invesp, and get the playbook. You can review all these examples. There’s a lot of commentary on what we did with each one of them that we cannot cover during the webinar. Something else.

Another good rule for Black Friday and Cyber Monday for the holidays is to remind visitors of the offers throughout the website. So as you see over here, while here is our offer, no, let’s go ahead and add the banner. Give thanks 30% off while supplies last. while supplies last. Also, the language over there, hey, we might run out of things, and let’s test this out, and we see a 12 percent lift. And this is beautiful, by the way. So the previous test that you saw was on the product pages. Site-wide is even more powerful. That targets every visitor.

So if the site is getting half a million visitors, if the site is getting 100,000 visitors, guess what? We’ve achieved a twelve percent lift in conversions with a really high chance to win a 94% high chance to win very statistically, statistically significant, results. Something else that people don’t feel very comfortable with. So we’ve done kind of the basics. Let’s let’s get into something a little bit more interesting.

Test fast and discard the control. So typical A/B testing If you wanna run A/B tests, you pull out a duration calculator, you put your number of conversions, and you say, okay, well, here’s the number of conversions. Here is the lift that I expect to get from the test It tells you that you need to run the test for 5 days, for 2 weeks, for 3 weeks. Well, guess what? That works really well outside of the holidays. 

During the holidays, the mindset is completely different. My goal is to maximize the conversion rate and the revenue that I’m generating from the visitors who are coming to my website during that very limited time period. So in this case, as you see over here, we had the original, we’ve added v 1 as well, or we had an offer we went back. V2, it doesn’t show it over here, by the way, but there was a timer that was showing.

And the point that I wanna show you over here is we’ve actually discarded because the control did not perform really well. We removed it completely, and then we decided to go with the 1 which has no timer, and V2 running 2 things. So where’s the control? Well, guess what? The control now became V1.

In order for you to do something like this, you need to be monitoring results. You can’t afford well, you know, I’m gonna relax. So usually, the holidays, and then the team already knows this. So most companies take, you know, Thanksgiving day off. they might take even Friday off, and then they’ll come back.

No. During the holidays, we wanna make a ton of money. That’s just the reality of it. So we make the decision to say, you know what? The controls not performing really well.

Both V1 and V2 are performing better, but we still don’t know which one is better let’s run them against each other. And that’s what you see over here with V1, without basically no timer ends up losing, and V2, basically, with the timer generates another 21.84% lift and look at the chance to win, in this case, and, and this has 746 conversions. So again, during the holiday’s high season, you reached statistical significance really, really fast. Okay. Let’s talk about more complex experiments.

So lots of times people test things on product pages. They test things on maybe with a banner across the websites. maybe they’ll test something on the cart page. Sometimes you wanna run an experiment across the website where there’s a particular experience for visitors across the website. So I’m gonna show you, and, in this case, we’ve run the experiment, a single experiment. It was running on the product pages on the mini carton on the cart page, and you’ll see an example of that.

And we combine that with the user thresholds. Hey. You can get a free gift if you order a certain amount if you order above $175, for example. The beauty of this is, we have a consistent experience. So let’s take a look at this. So here is the product page. Very simple.

So people can take a look at the products. Again, remember, this is a multi-page test, and we had this pop-up. So, basically, spend $175 now and get our exclusive. I’m not gonna even try and pronounce this. See, I’m looking at, like, I can tell whenever I’m gonna mess up the pronunciation.

And, and guess what? We didn’t even offer it for free. We told them, like, you know, the $175 gift is available at, very high discounts, $39.

So if somebody comes in and they view this variation, they see this pop-up. And let’s say they click on the add-to-cart, and let’s see how the mini cart looks. This is the original mini cart. And again, it’s the same experiment. Now there are many instances. There is one case where somebody still did not add items, or they might have added items.

So these experiments, you need to think through these because you have multiple scenarios that you must cover. So in this case, what we’re showing up here, congratulations. You qualify for our $39 gift. Rex, that’s what we’re showing them. Let’s go to the cart page and again, it’s the same experiments. and in this case, we’re showing the variation Again, you are $50 away from getting, you know, this gift for $39. So we established a threshold.

We didn’t even give free gifts. Correct? We gave them an item at a discount. Now let’s take a look at those. Also, we’re running a single experiment on multiple pages with 50% of the traffic between the original on the product page on the mini cart and on the cart page versus V1 off of the product page, the mini carton, the cart page, and we end up generating a 7.89%. Now this is the site that gets a ton of conversions. This ran, you know, for 2992 conversions, really opt the site generates more revenue. So think about running the experiments. 

A multi-page experiment can be and the beauty of the multi-page experiments is and helped us provide a consistent experience for our visitors across multiple pages. Another lesson that we’ve learned is to help visitors understand what they are getting. Now Lots of times people tell me what we test during Black Friday or Cyber Monday, and then they start testing some usability issues.

And I tell them it’s not the time to test choose ability issues during Black Fridays or during the holidays, just not absolutely not the time. What you are focused on is the offers and how to nudge people to take action. There are a few exceptions, and you’ll see one exception. So this company, basically, sold truck beds. Okay.

Now when we look through our heat not data, session recording data, we notice then, and when we talk to customers, they are confused about how to select the right truck bed size for them. So we end up deciding, like, you know, okay, well, how do we display the information? Do we display it? The very top over here, you know, what kind of text do we need to show? Is it the pop-up or does it take them to a different page? So that’s what we were testing with this. And what you see over here is we end up generating a 31% improvement in conversions. And this is interesting because they already had the information. I’m gonna go back. Yeah. So over here, you know, bed length, they didn’t have it, and we added it. By the way, I’m not a big fan of small tests that basically don’t explain. 

Like, they’re like, oh, most people wouldn’t notice it, but when you have enough data to show you that your visitors are absolutely confused about selecting the right bed size. So you know, let’s add that information for the visitors, and that’s what you see over here where you get a 31% lift and conversions with this, you know, it’s one of the few tests that you see, a 100% chance to win. I mean, the difference was just so huge being the 2, that is what’s about a 100% chance to win. Something else that you can do during the holidays is you need to, again, remember that your visitors are highly motivated. They’re coming with a credit card in hand. I’m looking for offers.

show me the offers and let’s just get with it. So you see this on many different websites. Okay. So here is the site. And what what what do we see over here?

This SEO text is basically the SEO company. I said, hey, let’s add this, but the problem with the SEO text, in this case, what did it do? It pushed the page below the fold, all the items. And what you end up doing is, like, you know, people coming to the page and bouncing I said, you know what?

This is a really quick test. All we’re doing, this text over here that is helping you rank is actually reducing your conversions. So in one variation look at this. Very simple. Keep the text.

We’re not gonna fight with the SEO guys. We’re gonna show the initial portion. If they click on more, they can read it, no clicks, by the way, whatsoever, on the more. And then V2, we said, you know what? Can we can we get a little bit more daring?

SEO guys don’t get upset with us. Let’s just move that text from the top and put it at the bottom. It’s still there on the page, and let’s see what the impact of control versus V1 versus V2 is. V1 with more links generated a 4.27 percent lift in conversions with no notice of how this experiment ran generating 947 conversions. So it’s highly statistically significant. It has high confidence. Correct? 94 percent chance to win with a 4% lift in conversions. And, again, remember, the idea there is I wanna have people always look for those big tests that are gonna generate. And you saw an example of a 30 lift in conversions. But to me, If I think about the funnel and break the funnel down into category pages, product pages, and cart checkout, I’m able to produce a 5, 10% lift in each one of those steps. I’ve significantly increased my sales by 50, 60, and 70%, and I’ve done even better. I’m doing really well in the and liking it during the holidays, but this is even better. What else can we test during the holidays? You should, you know, test the offers plus the brand. And let’s take a look at an example over here.

So 25% off, plus free shipping. So, you know, as we said, like, you know, 25% off, free shipping, and our hottest deals. So that’s one. Look at this. I mean, 70% off.

Which one do you think when? 70% off would win easily, but do we show our hottest deals as in V2, or we should show, like, you know, how they’re entertaining in V3? Those are all possibilities. So let’s take a look at the control. I’m gonna show you, you know, the discounts and our hottest deals Black Friday savings up to 70%, but we’re gonna play with the brand and let’s see what kind of lifts we’ve generated.

So just merely, like, you know, focusing a little bit on the brand, gave us a 6% lift, 26% lift with the hottest deals, and a 14% lift with the brand. So you gotta test this. but you start with the thoughts and the idea okay, let’s see here. Especially if you are getting the the visitors and the conversions, introduce those, any of those would have been better than the control. Correct?

Any of those variations that you see over here would have done better than the control. I would take the 26% point, but 26.4 percent over over the other, the other variations. Remember? Countdowns are a powerful ally. So notice over here, the different designs of the countdowns.

Okay. So I wanna have a countdown, a counter. So in this case, we show the countdown. We combine it. It’s like, you know, with the with some offers over here. And let’s take a look again.

Above the top navigation or below the top navigation, we see a 19% and this was also run across the website, by the way. So do we show it’s here, or do we, you know, or, you see kind of the different designs over here, you know, and how it stands out. And as you can see, by the way, treat yourself to 25% or treat yourself to 25%. I mean, it’s the same messaging.

But we are using different designs to show it and kind of display it to people. And let’s take a look at the results again over here 19.24%. And, again, this ran across the website, and I think if I recall correctly, this is a test that ran that Thursday, Thursday night, 12 o’clock at night, Eastern. That’s when this test ran. We managed to call a winner in about 24 hours, and then we deployed the winning across the winner across the website. This particular website, last Black Friday Cyber Monday was their best, sale period since the company’s inception, and they were in business for about 12 years because they saw results like this. My message to you is to test, test, test, and think about different offers. Different messaging can test different designs that you can test, and we’ll talk about the team makeup shortly.

Okay. Now here’s something if you have VIP customers, treat them to something special, give them special offers if those who are, you know, especially if you have some data around your customers, people who place an order with us, typically, people who have placed an order that is 25, 30% larger than our average order value. So we need to treat them to something or maybe people have created an account with us previously. So, notice over here, VIP access, give thanks 30 percent off ends November 21st, VIP access, again, with a timer versus just Oh, nothing about the VIPs. No.

You already have VIPs coming to the website. Let’s try and capture most of those VIPs because they already know the brand. They’re already buying from us, and they broke that mental barrier of giving us their credit card. multiple variations over here, the winner generates a 9.76 percent lift in conversions. Now I wanted to know something, and this is something that you don’t see from us typically, but look at the chance to win. It’s an 87.51% chance to win. In a typical A/B test, I wouldn’t call that a winner. 

But during the holidays, and because I am trying to maximize my revenue, I will call this a winner, anything above, probably like, you know, 85% I would call it a winner and deployed. And there are algorithms out there that you can use, a multi-arm bandit, for example, that will help you kind of answer this, but we’re not gonna get into the statistics of all of this. again, if you wanna get the playbook, by the way, drop an email to convert at invesp.com, We’ll get the playbook with all these different examples so you can go through it carefully.

Now I know Black Friday Cyber Monday is next week and people are like, this is too late to do any of this stuff. No. No. No. No.

Black Friday and Cyber Monday is just merely the beginning of the holiday season, and I’ll show you what I mean by that. I want you to take a look at this company, I just pulled their data off from November 26th to November 28th. That was kind of the Black Friday that’s in 2019. Notice what they did. The 2 days after from November 29th to December 1st.

So it’s not only about Black Friday, Saturday, or Monday. During those 2 days after we finished Black Friday and Cyber Monday, They actually saw a 116% increase in their mobile traffic, and this is just for the mobile So they doubled their sales in those 2 days compared to the previous 2 days. their eCommerce conversion rates went even up further by 18%, 19% improvement in transactions. The idea here is, yes, we are focused on Black Friday Cyber Monday, but you really use Black Friday. You use Cyber Monday to learn and you continue deploying experiments during the period after that. and then you will if you’re doing things correctly, who wouldn’t wanna take something like this where, hey, it’s another $1,000,000 instead of 500,000 that we’ve done. So okay.

So I’ve told you about all this stuff, and let’s finish with this and then maybe open it up to Q&A. If anybody has any questions, what does the team make up that I need to have in order for me to execute all of this? And lots of times people get worried, and I’m gonna show you that the team makeup, you don’t have to have all the people. Sometimes it’s about the roles, so you might have a single person playing multiple roles, but you definitely wanna think about this. You wanna have almost like a team captain, the project manager, he or she will make sure that that’s everything is running on time smoothly that we have our creative done we have all the coding for the experiments done.

You can get away without having a project manager. Yes. But during the holiday, especially if you’re running testing, and you need to constantly keep an eye, on a project manager, or somebody playing that role. So sometimes I see companies that we work with where they ask their marketing lead, the marketing coordinator, to play that role, but somebody needs to have that responsibility of really managing the team, making sure that everything happens because there is no room for air when it comes to the holidays. You need to have somebody who is gonna come up with the different offers.

So that might be your marketing specialist. If you have somebody dedicated to CRO, if you are working with an agency But who is that person who’s gonna think about the different offers? and they’re the ones who are gonna coordinate into what if you have somebody merchandising, if you have somebody management, but that’s gonna be basically the person who says, okay. Here are the different offers that we were gonna have. We work with companies that are small at 10 man shop.

Well, we are working with the CEO because he or she will be making those decisions. Here are the offers that we are gonna be running during the holidays, and here’s what we are gonna be testing. You need to have a designer, basically, is gonna create all the different amazing designs that you’re gonna be testing. And the designer, you give them the hypothesis and you tell them, listen, I want the design and you saw the different counters. Like, hey, give me 2 or 3 different variations because each one of those designs is an implementation of my hypothesis. I wanna have a counter, but I can display the counter in different ways. You wanna have a front-end developer. You’re gonna run this as an A/B test again. Sometimes you will tell me, well, it makes more sense.

Like, you know, let’s just implement it as is on our website. We don’t need to be A/B testing it. And I can go back to the example that you just saw. Three different designs, all of them performed better than the control, better than the original. Correct?

A 4%, 6%, 26% increase. Which one would you rather have? That’s the reason you A/B test. You’re A/B testing because you have a data point that tells you, yes, we need to run an A/B test.

You run an A/B test because you have a hypothesis, but your visitors judge the quality of the designs that you have. So that’s the reason you have to have that designer who’s working very closely with your CRO or your marketing specialist. You need to have somebody a front-end developer to implement the AB test, and this is kind of just standard to make sure that they’re implementing it. Make sure that whenever they’re implementing any of those experiments that they are that there’s no impact on the load time, I saw experiments or just reviewed them. We’re working with a company. They said, hey, can you guys come in? And they were trying to test the main banner on the home page. It was taking so long, and it was such a silly error, but everybody missed it. And I’m like, why is your variation taking, like, almost like 15 or 17 seconds to load?

Well, somebody didn’t think through optimizing the image that they were using to offer. And then it was really large and took too long, and I’m like, this is absolutely horrible. Instead of improving your conversion rates, you’re actually slowing down the website. So your front-end developer needs to be very careful. Finally, you need to have a QA person, quality control, a quality assurance person.

Now, again, my marketing specialist could play the role of a PM they could play the role of a QA person, but somebody clearly needs to say, you know what? You are responsible for this now. Lots of times, people say, well, yeah, all of us are gonna do the quality control. No grades. Everybody can chip in, but I want somebody so that when things go bad, I wanna go back to them and tell them what happened over here. So if that’s a dedicated QA person, sometimes you can hire remotes, you can hire either companies or freelancers to just do the QA. I wouldn’t recommend that during the holidays. Sometimes, especially if they are not familiar with your websites, somebody from the company needs to validate everything. After you finish coding all of your experiments, this is a good practice that you should always apply.

What’s your IP address last month? The experiments are for your particular IP address, only for your IP address. So you’ve seen the design. You’ve discussed the idea. We’ve coded the test. Let’s run this experiment just for your IP address. So you’re actually seeing the experiment and just validating it. Don’t just rely on I see people sometimes saying, well, do we need to do that? Yes. 

You need to do that, especially with the holidays. That is all I have for you guys over here. Again, drop an email to convert at invesp.com. If you’d like to get a copy of the playbook with all these examples with a lot more explanation than I probably did over here, you can review all these examples and implement or steal some of those ideas. We don’t mind.

That’s all. You can, by the way, drop me an email. My email address is collared@invest.com, and I hang out in the land of LinkedIn, so you can connect with me on LinkedIn. You can just search for my name Salahore. It’s LinkedIn forward slash call it h. With that, if anybody has any questions, let’s open it, let’s open it up.

S:

Thank you so much for that great session. Got it. So many stories packed into 1 in just about 40-45 minutes. Thank you so much. Khalid, I wanted to ask you one question.

If you were to recommend our audience to check out some book or anything that you read to go deep dive into a certain topic, with respect to CRO or anything, otherwise also, do you have any recommendations?

KS:

Yeah. So every marketer you talk to, by the way, tells you to focus on the customer. Correct? Kind of the standard advice that we give. But how often do marketers actually focus on the customer very low? Because we do an annual survey of VPs of marketing companies, both large and small, you wouldn’t believe 93% of VPs of marketing say we don’t talk enough to our customers. Lots of these messages that you have and offers come from the customer as we know, and we learn about the customers. 

One of the books that I ask everybody who works with us to read is Competing Against Luck. It’s a very powerful book about messaging, and it’s a very easy book to read. Some books are better summarized in a blog post, you know, but publishers wouldn’t accept that, and the authors wouldn’t get paid. Competing Against Luck is a really easy read, highly recommended, and delves into the psychology of how people make decisions and how companies can compete. How do we compete against giants, very well established the Amazons of the world and the eBays of the world, although eBay is like, you know, a company that we work with, how do you as a small company, how do you compete against them, Competing Against Luck answers that question.

In terms of CRO and all honesty, the field is solace in its infancy. We’ve written a book about CRO, and it’s an Amazon.com best-selling book.

So I see, like, you know, there are some really good blogs out there. The guys at CXL, although they’re a competitor, they do a really good job. I love their blog. VWO has an amazing blog. We publish quite a bit and we have a dedicated team just to creating really deep content, as well on the Invesp blog, and LinkedIn apps. There’s so many, like, you know, people who are just giving free knowledge.

I would say, that if somebody wants to learn CRO, there are so many resources available. The trick is in learning and implementing. Some people just become professional students, and they’re just reading and reading, reading. I tell them, implement my friend, implement because one thing you learn after doing CRO for 16 years, is that people are rather humbling. You think something is gonna work one way and people think otherwise. So you’re like, oh, gosh.

Okay. Go back to the drawing board and learn something new every time with every experiment.

S:

Thank you so much, Khalid. The book sounds very interesting. I think I’m gonna check it out very soon. For folks, Competing Against Luck by Clayton M. Christensen, do check it out.

It’s gonna be on my list. There are a couple of questions on the chat as well.

KS:

Yeah. Sorry. I completely missed seeing that people are also answering my questions. By the way, dear, I’m like, I didn’t get to see any traction. I got a ton of interaction.

I just didn’t see it. So, it’s just funny, guys. I used to be a software guy with GoToMeeting GoTowebinar, it’s the same software as it was, 15, or 16 years ago. And I feel bad about saying this. We switched to Zoom, and my daughter’s like, why aren’t you switching to Zoom?

And I’m like, you know what? Leave me alone. GoToMeeting. GoToWebinar page for your diapers for many years. So eventually, we made the switch.

Why don’t you pick some of those questions? Let’s answer them.

S:

Yes. Have you seen any winners that only performed well during the holiday season, except, of course, holiday-specific visual slash coffee?

KS:

Sometimes, it’s a good practice that the winners that you have during the holidays are away from yeah, I have a special offer. I’m not going to have it anymore. Correct? So that goes away, but it’s good practice to run some of these tests after the holidays. The reason is the holidays is a highly motivated visitor solicitor with a correct card in hand.

Correct? So that’s motivation, and your barrier for somebody who’s doing conversion optimization, trying to improve conversion rates becomes harder post-holidays. So sometimes a good practice away from the holiday-specific offers is to go back and run tests. But remember, also, the holiday testing is very focused on what is on messaging and offers and urgency and scarcity. That’s what you’re testing.

We’re not I showed a couple of examples where we’re like, hey, and then this is actually a good example where for this, the car that sold the car, the truck beds where it performed really well, performed amazing during the holidays, a 30% lift, and it still increased conversions. If I recall correctly, it was about 11 or 12% post-conversion. So it’s still really good, not that the 30% that we were really, like, you know, celebrating during the holidays. most of the time, you will see was difficult to say I would run the tests again on the experiments post-holidays, to see if I am seeing consistent results or not.

S:

That’s super interesting. It’s a little more time, but, more robust growth, I’m sure. So, Khalid, how are your predictions for, this coming holiday season, given the global, I mean, there’s a curtailment in spending happening. So based on that.

KS:

It’s going to be interesting. We’re not gonna mention the R-word over here. But, yeah, everybody is sort of like everybody is anticipating. something to happen. And I think the anticipation is much worse than the actual, looking up if something happens or not.

We saw lots of retailers who are just very cautious, but in all honesty, for the majority of the companies that we’re working with, we’re starting to see, more spend. And they’re like, you know what, they’re betting a lot. This reminds me, and then we’ve done this long enough back in

2008, 2009 and there are some companies who said, you know what? We’re going to double down. And most of those companies that doubled down during 2008, and 2009 did really well. But that’s, I think, kind of the beauty, correct, of experimentation. You put in the best effort and your visitors are gonna show up.

So all your control would have generated 10% conversions. Oh, well, yeah, these other offers that you gave us that are generating 15%. That’s kind of with lots of the other marketing you do and you’re not sure worked and did not work with experimentation. Well, here it is. We split the traffic and here’s the data you can run away from that.

Thank you so much. That’s highly increasing and motivating to give it your best. Thank you so much. I think that’s about it. There were a couple of questions but regarding the specific slides, unfortunately, I wasn’t seeing them. So my apologies. I didn’t answer those. So sort of now I can’t really tell which questions. I’m gonna just kind of finish with one thing to keep everybody kind of thinking about this.

When you run an experiment, you start with a hypothesis you have a win. Perfect. We love that. Correct. Who wouldn’t love a win?

It’s when you have a losing experiment that you start asking really interesting questions, like, so, hey, why did this lose even more interesting? You have a hypothesis, v 1, v 2, v 3. Well, v 3 is a winner, but v 1, and v 2 lost. The tough, and then the real, like, you know, person wants to delve into conversion rate optimization and experimentation.

We wanna ask, so, okay, great. We love the fact that V3 won, but why did V1 and V2 lose? What can we learn from those that’s, you know, lost? And then and you’re not gonna get the right answer. Every time you ask why, you end up with a more complex question. Well, they lost because maybe we didn’t have the right offer.

Why didn’t we have the right offer while maybe people were looking for this product and why were people looking for that? And you keep on asking why. Correct? Multiple times, and then you end up with tough questions, but your bill, your good experiments starts with a really good question and ends up with amazing questions. That’s kind of like, you know, one of the takeaways that I always mention to everybody.

S:

Awesome. Thank you so much. That’s a great thought to conclude the session with. Thank you, everyone, who joined in. At VWO, we do a lot of such amazing knowledge-sharing sessions with thought leaders, folks who’ve done who’ve been there, done that practiced themselves and, who have something to share with other practitioners. Join in another time, and thank you so much, Khalid, once again, for joining us and sharing your, 15-plus years of experience. That’s amazing.

KS:

Thank you. Thank you. Thank you for having me. I don’t know if I told you this. We were using VWO since it was in beta, by the way.

Some of the technical team there, remember them helping me, like, and kind of implement some tests back then. Whereas, like, what we’re trying to, like, you know, like, pushing the product to the limits. So I absolutely love the product. Thank you, everyone, for attending.