“I Added +$3M to Client Revenue With CRO & UX”

As always, I’m your host, Vipul, and I’m a senior marketing manager at VWO, a full funnel website experimentation platform. Today, as you can see already in front of us, we have a special guest who I feel a lot of people may already know, or, if you don’t, you will know him well after this presentation because he has lots of insights to share with you guys. So, his name is Chris Marsh, and he hails from Dash of CX. So hi, Chris!  How are you?

Chris Marsh: 

Hi, Vipul. Great. Thank you for having me. Can you hear me? All good?

V: Yes. I can hear you all good, and I hope the audience members can hear you as well. Just let us know in the questions panel or in the chat option if you are not able to, you know, hear Chris well. We’ll try to make amends. I believe there are none. There are no issues.

So, that’s great. So, cool. So, I’m glad to have you here, Chris. And the day I saw your LinkedIn profile, and I read about how you added, like, $3,000,000 of revenue to one of your clients in CRO and UX, I’ve been a lot excited since that day.

And then myself, I’m going to be a close audience to your presentation today. So without any further ado, I’ll just hand over the mic to you, and I’ll jump off the stage. So, here you go.

CM: 

Super. Thank you very much, Vipul. Very happy to be here. Okay, great. Thanks everyone for joining. Hope you’re having a good day. Let’s get stuck in. So today, you’re going to see examples of what you can add to your CRO efforts.

I’m really excited for this webinar. I’ve got so many insights to share from 5 years in CRO. You’re gonna hear loads of different tips and ideas. My promise to you is that some or one of these tips will save your CRO program or even boost it to the next level.

So here we go. A quick intro about myself. I’m Chris Marsh, freelancer for the last 3 years, under Dash of CX. Before that, I was at a great agency in the UK Fresh Egg for 2 years. I’ve got 10 years in UX and data, 5 years in CRO.

I’ve worked with many brands, lots of logos here, lots of interesting work. These days, I focus on eCommerce, and I help brands learn about their audience, and deliver better customer experiences. And in turn, we measure that we make sure that we’re improving conversions and revenue as well. So here’s what we’re going to go through today in our 30 minute webinar, but with Q & A at the end. So do pop questions in as we go, and we’ll look at them at the end.

We’ll start with 2 CRO programs that failed. So this is me being humble and sharing what went wrong. So you’ll know what to look out for and how to prevent that. Then I’ll talk about thriving CRO programs. What helps them really work and why they work and what works so well. I’ve got 6 great takeaways for you from those programs.

Next thing I’m really excited to share is a framework which I use, which I’m sharing for the first time today. And it really makes CRO that much easier, and it makes sure that it’s long term. So it’s not about quick wins or hacks. This framework guides you for, like, long term success. So I’m really excited to get stuck in and share that.

And you’ll see a real test example of how I used it, what happened and how it helped so much. Then a simple wrap up, a wrap up of 6 practical tips for you and then the Q & A. So starting off, so over 5 years, I’ve worked on more than 12 different CRO programs. Lots of varied experience. And usually they thrive, and they tick along perfectly, and everything’s great.

But interestingly, there’s been 2 which kind of hit a bump or snag. And then they failed and the program’s completely stopped, which is a real shame because as CROs, we love our work, and we know there’s a huge opportunity there. And we need to inspire our clients and help these CRO programs succeed so that we can do this work and improve experiences and revenue for the business. But twice, for two different reasons these programs stopped. So I’m gonna share these stories with you.

So you can learn from my mistakes. The first story was about a lovely client in the US, and they sold nail polish, a very high-quality nail polish brand. There was so much opportunity on their site. I was working with an agency that I still work with, and we’d work with this client for about 8 months. Just helping them with measurement and insights to start with.

And then we got to a point where we had more than 30 A/B test ideas, and they were ready to start going, which was great. We were starting our A/B testing program with them. And so the first test right out the gate was actually a huge winner. I hate to say it, but it was a pop up. It was a test based on the pop-up.

And we edited that and proved that and got a huge lift there. The 3rd test, we got a 3rd test. And that’s where things hit a problem, a snag. They had a new stakeholder join their team, which was a designer. The designer said, “Oh, this test, can I add my design as a new variation?” Because if we were changing the homepage, they said, they’ve got this great idea.

We looked at it, and we knew we could fit in another variation because of the traffic. So I made a bad decision and said, yeah, come on in, we’ll try it and make it work.

But the devs weren’t used to A/B testing at the time. They’d only done 2 other tests, and they struggled, and it went on for 3 weeks. So there was more dev work for them to fulfill this new challenge. So after 3 weeks, it hit that final point.

And the CEO said, ah, we love your work, but we just don’t have the dev resources. But, man, that was it. That was the CRO program. Grinding to a halt. It was really sad.

You know, it’s not just about losing the client because jobs change, things change. That’s fine. But there was such an opportunity on this site. There was a huge opportunity, but they felt that they couldn’t do it. They couldn’t run these tests.

So I spoke with my colleague, and he said, you know what? He says, this just highlights how crazy important momentum is in tests in CRO programs. Man. Yeah. We needed to build momentum. And to do that, we should have been doing really small tests to start with. And so we reflected on this, and it taught us how crazy important momentum is in a CRO program. So if you do have to run some large tests, just be sure that you’re running small tests alongside them. Don’t let a stakeholder come in and edit things too much. 

Taking their advice on board, we need to welcome stakeholders and their opinions, but we need to be careful with them, pop them in the test plan, and prioritize their ideas. Interestingly enough, a year later, I was reading this awesome book. Career advice – I highly recommend it. It’s called So Good They Can’t Ignore You.

And he said something which really resonated with me and reminded me of this experience. So the author talks about making small bets in your career. Okay. So he says, make small bets, and it maximizes your chance to win. Small bets are great because they’re bite-sized, you know, and they’re just simple, simple, experiments, simple tests, simple things here. He says, the main thing is you’re learning as you go.

He says, one day, one of these small bets, it will pay off huge. What’s up, man? This reminds me of that CRO program. We should have been making small bets. I thought, why didn’t I hear this 2 years ago?But here you go. So to summarize, it’s so important to build and keep momentum. Now the second CRO program, which faltered and hit an end, was a US store selling discounts, daily deals, lots of items for the day on a deal. 

And they were great clients. We’d work with them for, like, 8 months or so testing for about 6 to 8 months. We were delivering, positive return on investment. So we’ve been getting wins. We’ve been getting good learnings. Things were going well.

However, this story has to do with showing value and communicating value, and that’s where we could have done better and where there was a problem. So the CEO who we were talking with, he loved our work, but he was communicating with the owners of the sites, and they didn’t get it. They didn’t get CRO. They were old-fashioned, and they loved traffic and getting more traffic. This site had tons of traffic. It was definitely time to improve customer experiences and conversion rates.

So, the owners didn’t see the value, and they closed the program, the CRO program. The CEO told us, he said the owners are old-fashioned. They don’t see the value. What can you do? And, we reflected on this.

And we thought, well, we could have done something. We should have shown value better somehow. And now I think of it as beating the value drum. So you wanna beat the value drum, just show value as regularly as you can. Perhaps try to keep it concise and bite-sized, easy to digest. Perhaps, think of how your end client likes to receive it.

Maybe it’s a Loom video recording. Maybe it’s a quick message in Slack. Here’s an example of how I PR a CRO program in Slack. Provide a summary, keep it fact-based, and add highlights. And I even log if I find a bug on the live site, then I’ll note it down.

And then 3 months later, I’ll look at those amounts of bugs and I add it to this and say, whilst we were doing our work, we found 2 or 5 bugs on the live site. So even there, we were delivering value. And you can frame this as well. A simple little, additional way to do it is to start with value delivered. We’re framing it, we’re saying this is your value.

Just crystal clear. And then they can share this with anyone that they want and should make it easy to showcase what we’re doing, why it’s so useful. Now here’s a cool tip, which I learned from Rommil Santiago, from his book Sprinting to Show Value. So this is a great stage. If you want to grow your CRO program, there’s a great chance to do it.

So you can state the hours that were required to achieve these results. So we can say – hey, this uses 20 hours of dev time per month. We would love to unlock more dev time because that might be the blocker and we can we know that we can grow the program if we unblock that dev resource. So this is where we can do it.

We can tie these resources back to the value and success and say, you know, if we add x amount of hours of dev resource, we can achieve this growth to our program and to our success. 

So that was 2 CRO program killers, losing momentum, and not showing value regularly. Now onto the better stories, the better side of things, the thriving CRO programs. But it’s good to land from failures, of course, as a CRO. But here, I’ve worked with a handful of clients over the years, And one in particular, we delivered over 3,000,000 uplift, last year.

And these 6 takeaways are from that program. Other programs have succeeded really well. So test one concept change, and you’ll never lose. Ultimately, it’s so good to have the opinion, and the mindset to test to learn. So instead of testing to win, looking for wins, and thinking of which ideas will win, now just to make sure that we can learn. And it’s a weird thing. This may not be that intuitive, but you will end up getting better win rates, which I’ll explain later on. My framework helps you do this as well. This is key. And, yeah, hopefully, I can help you do this today. 

I really like this one. So when a test finishes, don’t just cross it off and say that’s done. It’s really important to brainstorm at this point. Hopefully, you’ve got learning. Hopefully, if you’ve changed your behavior for the better or not. And it’s so useful to think, okay, what other test ideas relate to this learning? So, you know, if you’re on test one, you can start thinking, how do we double down? How do we do more of this?

We do this elsewhere, or could we, make it bolder or do it better? And if the test loses, think of the solution. You know, was it the right solution? And ultimately, your CRO road map should be very flexible. It doesn’t need to be long and set in stone.

That’s just quite fun like that. We just have to hold our hands up and say, hey, we can only see, like, 1 month in advance 1 or 2, and, it might change. 

This is a classic. You’ve probably heard this. We have to do the CRO research. It’s a simple theory, really. If you think about it, no great company was built on CRO hacks. Okay, that they’re built on a close relationship with the customers, delivering great customer experiences, and a desire to experiment. So there’s lots of research out there on how to do this.

Don’t get put off because there’s lots of different ways to do it. Just start, just do 4 hours a month or 7 hours a month, and you’ll get great test ideas from this, which are actually backed by actual data. You know, customer problems, you’ll be able to make their life better and you’ll be able to create much better test ideas using this data. So ignore the “Hey, here’s 10 test ideas – try these”.

Now you’ve gotta actually put in the work. I hate to say it, but it’s worth it. And log learnings. This is obviously important if we’re gonna test and learn, I always will forget the learnings. So you can log these in VWO. If you don’t use VWO, you can use that table.

Tim, my colleague from Exceptional Web, was the first CRO I ever met about 3 years ago, who was using Airtable, and he introduced me to it, and it blew my mind. It’s really powerful. It’s a great way to log your research insights, your experiments and your test learnings. So I’ve got a link here where you can help yourself and use our template. 

A little bonus here. If you’re running a lot of tests, say, for a month or more, you can tag the purpose of the test with the test. And then after half a year or a year, you can look at the data and look to see, which kind of purpose is actually the most effective. You know, is it price and value or usability? Then you can kind of like… It can guide you for your next tests, or you could spot areas where you haven’t been testing. Certain purposes where you haven’t. This was inspired by Growth Rock Agency who do this really well. It’s their core method. So there’s an article link down here, if you want to go and check out more about that. 

And this, you could say, is a boring one, but honestly, it’s so important to QA your tests, and you have to QA them well, you know, give it 30 to 60 minutes, maybe 2 hours if it’s a big test or many pages. You really, really need to find bugs. You will find bugs. Okay.

Devs can’t catch them all. I’ve been called a QA madman, which I took as a compliment. And luckily, it was a positive comment from my colleague. Because I just know it’s so important. So, of course, it’s not a fun task.

It could be a bit dull, but put some of your favorite music on. Get your head down. Go through, make a little list of things that you wanna check each time and make it easier for yourself and, yeah, go through it. It’s well worth it. Otherwise, there’s way more problems down the line.

Yeah, it’s worth using real devices if you can. Use an old iPhone, old Android, choose a Mac, and a Windows machine. This will cost about a grand, If you can’t do that, you can use BrowserStack. Also, if you build a redirect test, there’s an interesting situation here, which can easily cause a huge problem, because we’re creating a new page on-site. And Google might find this page, and it might list it.

It might index it. And my CRO buddy told me, he said it’s so hard to get that out of Google, you know, once Google indexes it, it’s there. So that is bad news because of aall sorts of links and everything else, which are going to your original page, but then Google might pick up your new test page. So there’s a simple way to resolve this. Check out the link here.

Involves adding a role canonical tag. It’s the new page. I see devs miss this all the time, unfortunately, but it’s a quick check. You can catch it before it goes out live.

Lastly, this is my favorite, the 6th. It’s been said that this single law is the most important law in data analysis. Really useful to save my butt probably ten times. It’s Twineman’s law. And he says that any figure that looks interesting or different is usually wrong. Wow.

A bold claim, right? But it has a lot to it. I’ve researched this. And there are different angles. Basically, any data that’s surprising is my way of thinking about it. Any data that’s surprising, check it.

You know, that’s when we need to be skeptical. That’s when we need to think, ah, it might not be accurate. Maybe there’s a tracking issue. So we had this happen. The variation was winning by over 130% after 5 days.

And some of my colleagues were saying, well, this is great. But I thought, oh, no, that’s too much. It shouldn’t be that high. You know, it would usually be plus 5% or plus 10%.

So I said we needed someone to check the tracking. And sure enough, the tracking wasn’t working correctly. And the data was inaccurate, and we had to start the test again. But this shows up all the time. Even in your own work, you might be copying, and pasting data, and then you see this data looks really weird. I need to check it, and, oh, man, I actually made a mistake there.

So this is a really good sort of sense check to remember this law. And moving on to the final stage of the webinar, I’m very excited to show you this framework and how I use it. This can help make your CRO easier for a long time. I’m gonna show an example test of how I use this. So what’s important here is to remember that we need to use CRO research data. So, you know, these are surveys and polls, user testing, which is really good to find customer problems easily. And all the rest of it. Just do what you can, and you’ll get great ideas and issues that you’ll spot through this data. And, of course, we need to test, learn, and explain, but you’ll see exactly why this is so useful. And why do we do this?

Essentially, we wouldn’t expect this, but any idea is only gonna have a 10% chance to boost your KPI. And research has shown this. And that’s tiny, just a 10% chance. But if we’re strategic and we use these pieces of data, then our win rates grow. It grows at least 20%, you might get 30% of tests winning. So test and learn.

Knowledge helps you so much, helps you move in the right direction, you get to iterate way more easily. It’s almost impossible to iterate if we don’t do it like this. And I highly recommend you check out my framework. And if you’re interested, Shiva’s got a great video here where he explains a bit more about how he uses tests to learn with his CRO. So let’s dive in and see the framework. This guides you to test and learn. So I called this the PQ framework for CRO. It’s intuitive, and I’ll tell you why it’s great. So, the problem and question sections I feel are super important, and that’s why I’ve named it the PQ framework. Now this is a method where you add these 4 items to your test idea.

So wherever you’re logging your test ideas, go in and start adding this to your test ideas, and it will make sure that you end up releasing tests that actually help you learn a lot and iterate and grow your win rates. So it involves finding a problem, which is backed by data, customer data and involves having a question. Now I love this because it’s intuitive. It’s like, what do you want to find out? You know, will this do this? It’s just simple.

And then we have a solution and talk to your team about this. It’s a great time to loop them in, and get them involved in the program. Then obviously, you need to measure it so your stats have you measure it. Now with this, it’s actually optional whether you want to use a hypothesis here or not because what I’ve noticed is, from the industry, I’ve heard people say that people aren’t using the hypothesis. Someone analyzed 250 teams, and he said they weren’t using data in their hypothesis.

They’re basically not using them correctly – the hypothesis. So I realized that hypotheses are hard to use. Some people skip them. So, I’ve been using this, and I do add hypotheses as well, but you don’t actually have to because this actually breaks it down and makes sure you’re going in the right direction anyway. The hypothesis is useful if you wanna state why you think the change will happen.

But let’s dig into this a bit more and see how it actually works. So we’ll take a look at a real test. So I have a client that sells perfume. And we’ve run surveys and found out that the most important factor for shoppers is a low price. Now, we would know that insight.

We can review the site thinking about that. And on the product page, we didn’t show the saving amount clearly. So we simply showed the current price, and the old price crossed out. So, you know, a simple sort of non-strategic CRO would say, oh, this is a simple chance to get an uplift. Let’s just add minus -15% here and test that.

Why not? We’ve actually got the data to say it’s useful to try testing this as well. But let’s move on. So the question here is, if we show the saving in pounds or percent, will it boost shopping metrics, and also which is best? Let me have the solution ideas, which are very basic, very simple, simple tests to run.

And simply how to measure, I recommend using an online calculator to check that your KPI will actually reach significance over the amount of time that you want to run the test for, which is usually approximately 2 to 4 weeks. So with this, we’ve got surprising results. We didn’t hit a winner with this, which is crazy. But it goes to show that you never know until you test. And because we tested carefully and each variation was aimed to answer one question, we are able to learn.

So here we learn that showing the pound savings produced a flat result slightly negative. Weirdly, showing a minus percent created a huge loss, so that’s not expected. What happened there?

There could be a few different reasons for that. It could be confusing. Yeah, perhaps the savings aren’t high enough. But this is the point at this stage, we get to it.

We get to brainstorm. I think the solution was wrong. You know, could we do this a different way? So the next iteration, we actually had about 3 variations for this. For one of the variations we threw in a wild card, we said, this is a new question with this variation.

If we simply show the item is on sale, perhaps that can send a message that this is a low price. This is a great value, you know, will this left keep key shopping metrics? And it did. Here we hear our winner. It’s supposed to show that you never quite know what’s gonna happen. And if you iterate, you start going on the journey towards what works.

So that’s the PQ framework, and it ensures you will test to learn and avoid tests to win. It’s all about making small bets when you can in learning, and trying to iterate when you can. And only if you have to, you might move on. So you might iterate, say, 2 or 3 times on the test. Some tests you can’t iterate because it’s clear maybe there’s no more solutions.

This is my recommendation. This helps create a great CRO program, which is long lasting. Where does this work? Well, you gain strategic insights, like how useful certain change was, you know, how much it resonated with the users. And then you get to brainstorm new ideas based on that new learning, which is really useful.

And now to wrap up, in summary, 6 practical tips for you to take away to improve your CRO program. We have to do the heavy lifting. Of course, do CRO research. Make it doable. You know, don’t get overwhelmed.

Just do what you can. Keep it ticking along, and you’ll be rewarded. Avoid letting a stakeholder change of tests and slow things down too much. You know, keep momentum so important to keep momentum, and show value, PR your program. Also, test to learn, and you’ll never lose. Think about moving on or iterating after a test finishes, try to brainstorm new ideas.

Log your test learnings. So useful. Months or years down the line you can revisit them. They can strengthen over time if they keep getting proven, and the businesses gain massive value from this. And lastly, try the PQ framework. It will help guide you on this test to land method and may help steer you in the right direction. And for a bonus, of course, QA tests, you will find bugs.

Do good QA. Use real devices if you can. If you see data that surprises you, “Oh, that’s interesting”, be careful. Check out that data.

Be skeptical. Don’t trust surprising data. 

Great. Thanks. That’s the wrap up, next onto the Q & A.

V: 

Great. Great Chris. It was indeed very lovely, all the insights. Even I was sort of surprised with a few things that I just want to highlight here. So, I mean, the first thing is, of course, you know, one thing that you mentioned was that no great business is built on CRO hacks. And something I truly believe in, and it’s not just about the CRO hacks, but any kind of hacks, right, every business is built on process, speaking especially from a, you know, from a CRO perspective. I would say, yeah, you know, if you have to incrementally improve the value of your business, you know, embrace the CRO  process and not, you know, read a blog or 2 and be like, okay.

Maybe they tried, you know, or maybe, the example that Chris shared they had put on sale, you know, as a variation on that one. It may not work for you. So don’t take it as a hack instead. You know, pay attention to the process, the thought process that went behind creating that hypothesis or hypothesizing that variation. So that’s really one.

And the other thing, of course, I wanted to highlight was you know, I, too, myself, was, when you showed, the first two variations, with, you know, percentage changes and the absolute changes And I was like, okay, variation 2 is going to win. Variation 2 is going to win. But none of them did. It turns out that people were not interested in hearing or looking at the absolute values that they are saving, but more like, hey, it’s a sale, let’s click and buy it.

So that’s very surprising, at least to me. I believe a lot of people in the audience would be surprised to hear that as well. I see a lot of comments coming in. “Enjoyed the presentation, Chris.” Thanks, Philemon. So, Philemon says that he’s enjoying the presentation by Chris.

Timur Altman, he says, “Thanks for the book recommendation”. And, we have Bandon, “Thanks for the presentation. Do you have a link where we can download the deck? Yes. Of course, Bandon.

So we will be uploading the deck on our webinar page shortly. So, we’ll also be uploading the recording of the session, and we’ll ping you once that is ready. So don’t worry about it. We’ll ping you when things are ready. 

[Indistinct] …instead 10%. Okay. That’s an interesting question. But, Diego, would you like to, you know, unmute yourself and come up on stage and ask the question? Let me see where you are.

Okay. So I’m going to unmute you. So it will be great if we can add more context to your question. Yeah. Please go ahead.

Diego: 

Of course. First of all, thanks, Chris, for the webinar. And my question is about because in the end, sometimes we see some experiments that are not good enough, maybe for 10% of the users that come into the website. So in some talks with the team, sometimes we try to identify if it’s okay to run the experiment for 100% of the users. Maybe in the production side or maybe in the tool.

But, yeah, basically, my question is: Do you think it’s possible or okay to run the experiment for 100 percent of the users?

CM: 

Hi. Yeah. Thanks for the question. Yeah, if all of the users are relevant, then you should run it to those users. Or I mean, if your program if there’s, like, high risk to the test and you know, maybe it’s on the checkout or something, then you could soft launch it and you could launch it to maybe 10% of users or 20% of users to start.

And then if the software allows, you either ramp that up to a 100% or you quit that test and relaunch it to a 100%. Because having more data just gives us more accuracy on the results. In fact, for one client, they’re UK only, and they only sell to UK shoppers. So I actually target only the UK for those experiments. Because the other data will just produce noise.

You know, it might balance out, but there is unnecessary noise. So it’s worth thinking, you know, usually it’s to a 100% of relevant traffic, and usually it’s to all audiences when I test. But, yeah, those, hopefully, those are some good thoughts for you to consider.

Diego: 

Thank you so much. Yeah.

V: 

Got it. So it just sparks the curiosity in me, Chris. Will it be called an experimentation if you’re running a test on this entire audience space? I mean, wouldn’t the foundation of a test be that there’s a certain set of audience that is interacting with something which hasn’t changed at all, and there’s a certain segment of the audience that is going to interact with something that has changed?

CM: Oh, so hold up. I might be getting a bit confused. So, the overall traffic going into the test is different to the way we split the traffic in the test. The way we split it between variation and control. Is this what? You know, the split should be equaled between the variation and the control?

Diego: 

Yeah. I mean, we will split 50% in the variation and 50% in the control to make a 100% in the treatment.

CM: 

Yeah. I mean, I wonder why you would consider not doing a 100% traffic on the experiment. What are the reasons not to? What are the fears there or reasons?

V: 

Yeah. Even I got confused. I thought that maybe he’s talking about just showing the variation to the entire 100% of the audience. And I mean, just everyone who comes on the website, opening it for just everyone, I thought. That’s what Diego might be pointing to.

Yeah, thanks, Diego. Yeah, go ahead, Chris. Go ahead.

CM: 

Yeah. No, I don’t have anything more to add to that. But just maybe be careful if you’re starting out and the stakeholders are worried, then you could soft launch the test to 25% of traffic just to be extra careful. But always aim in the end to go up to 100% of relevant traffic.

V: 

So, next up, we have an interesting question from David. David, your question is lengthy. Would you like to talk on stage and ask the question yourself? So, I unmuted you from my side.

David: 

Sure. Thank you. So, Chris, good presentation. A lot of times, we’ll wanna test things that just don’t have a ton of traffic to the pages. So we’re stuck with really long test times.

Do you have a method that you normally go to? Like use ad words or something like that to try and drive traffic, when there’s not a ton of it? But these are high value pages. We just don’t have enough visitors. 

CM: 

Yeah.

David: 

What do you do in those cases?

CM: 

So there are a few things you can do. You could try driving more traffic, but that’ll get expensive and will you be able to continue doing that afterwards? The thing is the way I look at it is I would recommend only one variation, unfortunately, because that will help. So you’re getting more data into the test. If you use more variations, you need more data into the test.

So, one variation helps. Look at your KPIs, the measurements, and you can use an initial kind of action on that page rather than measuring further down the funnel. You could measure the progression on that page. So perhaps just click on the link that’s the most high value link on that page, like add-to-cart or such or a few products. So you could use the KPIs earlier in the funnel. And also, you know, you do want sort of 300 plus conversions per variation. A daily 500 or 800 or 1000, per variation at the end of the test.

If you don’t have that, you’ll usually simply have to hold your hands up and say, we can’t test this. But that’s not so bad because if we’re doing the CRO research first, then we have signals to say that this is useful. And as I’ve explained, there’s still a 20 to 30% chance that these changes will actually improve your KPIs if we’re using research to get to these solutions. So also you could use research to validate the solution. You could use a test on it.

You could launch it and then run heat maps on it, look at sessions and things. There’s these kinds of ways. If you wanna message me on LinkedIn, I can follow-up. I’ve created a list before about my ideas – there are 5 or 6 ideas. But that’s most of them. Hope that helps?

David: 

It does help. Thank you very much, Chris. I’ll reach out to you. Thank you.

CM: 

Cool.

V: 

Yeah. Please do. Please take a screenshot of this particular slide and reach out to Chris on his Twitter or his LinkedIn. Do mention your conversion problems to Chris.

And, of course, he’ll definitely have a solution. So I see there’s one more question from Kuda. I believe I have heard your name before, Kuda. Let me find you in the chat. Yeah.

I have unmuted you. Good. Are you there?

Kuda: 

Yes. Hi, how are you doing? Hi Chrish, thank you for the presentation. And I’m a regular always here to listen to webinars. It’s quite late here. Yeah. 

V: 

I think it’s… [Indistinct]

Kuda: 

It’s AM. So my question was, one of the points that Chris mentioned, which was a very good point that we should create our hypothesis from data. So, I always have challenges with this. So my question was how do you use data to create the hypothesis, the percentage and the absolute variations? What data point did you look to come up with the hypothesis to test the percentage in the absolute hypothesis?

CM: 

For the example I used. Yeah. So we used survey data. So we polled the audience on the site and asked what are the most important factors to you when shopping for perfume realistic, like, 7 or 8 factors. And 3 were by far the most important factors.

One was low price. Sorry. What was the other part of your question? Are we relating it to our hypothesis?

Kuda: 

Yes. How to relate it to the hypothesis? Yeah.

CM: 

So have you heard of the PXL prioritization method?

Kuda: 

Yes. We have 1 in our company as well.

CM: 

Cool. So I recommend using that and you just kind of like, so how do you relate it? So well, this is why our hypothesis is quite tricky to use. I mean, it has the format of so for this example, I would say if we change the product page, to show the pound value. So it starts with “If”, “then” the shopping KPIs will increase or conversion rates will increase.

And then the final part of a simple hypothesis is “Due to the reason”, and that’s where you add your idea, your theory as to why it will happen. And I would say due to better showcasing low value or great value on the product which is important, shown by data.

Kuda: 

Okay. Yeah. That’s understandable. Yeah. And we can also add the target audience as well in that structure of the hypothesis?

CM: 

Yeah. Yeah. Exactly. You can flash it out at the beginning, perhaps you add. So it’s defining the audience at the beginning. And this is a thing with the hypothesis that they’re big and convoluted, but they are useful. So if you can use them, that’s great, but people skip them often. And they’re, you know, they don’t give a snapshot. If you just look at an A/B test idea and you have to understand the hypothesis, you have to sort of study it for a minute and think about it. 

It’s like how-to-digest it. It’s not that intuitive, which is where the PQ framework makes things intuitive and sort of flips things around and lets us have a problem-focused approach and a question-focused approach as well. But, yeah, if you can use hypotheses, if it’s working well and you’re using them correctly to create tests, which help you learn, then you can stick with that. But what I’ve found in the past is that hypotheses don’t always help you do that. It can be weirdly easy-to-go, of course, and to create a solution.

And then it’s hard because it’s hard to look at the mock-up solution and look at your hypothesis, which is quite hard to sort of conceptualize and check that they fit.

Kuda: 

Yep.

CM: 

Basically, if you’re making too many multiple changes on the variation, you’re not gonna get accurate learning. That’s where it can go awry. And it might not be that obvious that it doesn’t fit with the hypothesis, but it won’t. 

Kuda: 

Thank you very much for that advice. 

V: 

Thank you for your question, and thanks for being a loyal audience to the VWO webinars. We appreciate it. Cool. So, I saw there was one hand that was raised by Philemon. Philemon, do you have something to say? Let me unmute you. 

Philemon: 

Hello, Chris. Thank you for the presentation. Also, I wanted to find out when you’re trying to run a test, and you want to find whether you have enough traffic, should you basically base it on the number of conversions on the page or the traffic on the site?

What should be the best that you should use to decide whether this should be a test or not? Is it traffic or the number of conversions?

CM: 

Cool. Hi, Philemon. Good question. Ultimately, it’s the conversions, but they do relate. So if your conversion rate is higher, then I think it means that your test can run a bit shorter.

So they do intertwine somewhat, but ultimately, it’s really important to get enough conversions. So we want over 500 conversions, around 4 weeks per variation. And even then, I would use this kind of insight with a pinch of salt, whereas if we get over 1000 conversions, then the data is even more accurate. But you can use calculators, and it’s best to use the calculator before you run the test so you know exactly what you’re looking at. And the calculator will tell you what uplift you need, which is really useful, because you want about 5% to 10% uplift. If the calculator says you need 20% uplift to run this test, then it’s not a good test.

You could try and use a different KPI, which requires less uplift. Because it’s hard to get 20% uplift or more often.

Philemon: 

Great. Yeah. Thanks.

V: 

Yeah. Thanks, Philemon. Thanks for your question, and feel free to connect with Chris after this webinar. So I believe there are no more questions. So I think it’s time to close this webinar officially.

So thank you so much, Chris, for preparing such an amazing and insightful presentation. I don’t know why I’m fumbling. But, yeah, thank you so much for the insights. And thanks to all of our audience today for being a close listener to Chris’s presentation. Feel free to connect with Chris on his LinkedIn.

The coordinates are there on the slide itself. And, with that, it’s time to pack our bags and yet another day. So thank you so much, Chris. Thank you so much, everyone. Have a great day ahead.

Bye bye.

CM: 

Bye. Cheers!