Conversion rate optimization course

Let’s talk about A/B testing for e-commerce businesses. So A/B testing is the world’s most widely adopted data-driven approach. It’s about splitting your traffic, splitting your visitors and showing different messages to each segment of your visitor.

Usually we call the original version control and the other tested version variation or challenger version. There are multiple types of A/B testing. 

The classic A/B testing is just splitting your visitors into two groups and serving 50-50% different messages.

We also have A/A testing which is basically splitting the visitors into two and serving them the exactly same message and exactly the same content. Why do we do this? We usually just do it to test the platform and the software and the architecture if everything is working correctly. 

We also have what we call A/B/n testing.

It’s about serving like A/B/C, A/B/C/D, splitting your traffic into more than two different parts. But the technique and the mathematics behind it is exactly the same as with A/B testing. 

And we also have what we call multivariate testing which is about testing multiple things on your website at the same time and testing all kinds of combinations of these different elements on your website.

We can also categorize A/B testing depending what you are actually testing. 

When we usually say A/B testing, we often understand the page or web page A/B testing. Having a landing page for example and splitting the traffic that goes to that landing page into two different parts and serving them two different versions of that landing page.

Another part is what we call popup A/B testing, basically having different versions of the overlays of the popups and seeing which one converts better. 

And we also have what we call multi-campaign A/B test or journey A/B test when we actually split the visitors into two parts and for one part we show a totally different journey, different pop-ups, different landing pages, different messages than to the other part. 

This is the most complex A/B testing technique that you can run on a website.

We can also differentiate A/B testing based on the mathematics that you are actually using. There are two main schools of A/B testing. The one is what we call Frequentist and the other is what we call Bayesian.

I don’t consider it important to understand the underlying differences between the two approach but what I think it’s important to see is that the Bayesian is better suited for small businesses with lower traffic. 

So most tools that serve small and medium businesses usually use the Bayesian algorithms. 

And there’s another term which you will probably meet if you dig deeper into conversion optimization which is called Statistical Significance.

So A/B testing is very much about chances. When we A/B test a website or a popup for example, we will see that one version performs better than the other. But this better performance, this uplift can be due to the changes that we actually made or it can be due to chance and just pure luck and randomness.

So basically we can never have a 100% probability that one version is better than the other. 

We can usually just say that there’s a 99% chance, for example, that the challenger is better than the control. And what you will see in a tool using Bayesian algorithms, for example, OptiMonk also uses Bayesian, is a term called chance to win.

It means, for example, in this case, that there’s a 87% chance that the challenger variant is better than the control, the original version. But it means that also there’s still a 13% chance, roughly, that this uplift, this 4% uplift in this case basically was just purely due to luck and to chance. 

Now, the next question is that how many variants should I run parallelly? You can run A, B, C, D and as many variants as you want technically.

But what you need to understand is whenever you run multiple tests at the same time and test multiple versions, you fragment your traffic into multiple pieces. 

If you have a website with millions of visitors every month, then of course it doesn’t matter because even if you split your traffic into 10 different pieces, you will have a lot of visitors still left for each different variants. 

But if you are a small business with limited traffic, then what we usually recommend is that you shouldn’t split your traffic into too many pieces.

And often it makes better sense to just run an A/B test and test one version against the control. And then if that happens to win or happen to lose, then run another A/B test against the winner of the previous test. 

So rather run sequential A/B tests and not run A, B, C, D, N tests at the same time.

The next question is that should I run multivariant tests? Well, it depends how much traffic you have. 

If you have 100,000 visitors, which is quite a lot actually a month, then if you test just two elements with just two versions, a control and one challenger variant, it will split your traffic into four pieces. It will result in 25,000 visitors per variant, which is still okay.

But if you test three different elements with three different variations, it will break your visitors into 27 pieces, which will only result in 3,700 visitors. If you have four different elements with four different variations each, it will break and fragment your traffic into 256 pieces with only 400 visitors per variant, which of course doesn’t make any sense. 

So I would say that multivariant testing is really for bigger, larger enterprises.

So for small businesses, what we recommend is to run sequential A/B tests instead. Run A/B tests one after the other. So for example, let’s say that you have one A/B test with one challenger version and a control.

If the challenger is a loser, then let’s just discard this challenger version and go to the second version and test this control, the original control against the next challenger, version C for example. 

But if this challenger beats the control, then let this new challenger version become the new control and the next A/B test should test this new control basically, the previous challenger against the next challenger. 

See? And this is how you can really A/B test the fastest with a limited traffic on your website.

Now the next question is whether you can run multiple A/B tests at the same time. Now an A/B test purist would say that no, you can only run one A/B test at the same time. Yes, it’s true that that’s the ideal case.

And if you have millions of visitors each month, then you can definitely do that just one A/B test one after the each other very quickly. But again, if you are a small business, then often running one A/B test takes time. So in these cases, I think it’s definitely a must and it’s definitely possible to run A/B tests parallelly.

What you have to be careful about is not to have conflicting A/B tests running at the same time. The next question is which conversion goal should you choose for your A/B test? Basically, what you want to optimize for. While it seems obvious that you want to optimize for orders, right? And revenue, it’s not so clear in practice.

Let’s see an example. Let’s assume that you want to A/B test the key banner, for example, on your homepage. 

Let’s say that you have 30,000 homepage visitors. And out of that, like 10% of this traffic clicks on the banner, 3,000 clicks. And out of the 3,000, maybe 750 goes and adds a product to cart. And then maybe 250 actually orders that product.

If you pick the orders as a conversion goal, then you will probably need more than 30,000 homepage visitors to decide which banner version actually performs better. But if you choose an upper funnel metric to optimize for, like, say, clicks on the banner, with this 3,000, probably 10,000 homepage visitors will be enough to decide which banner version actually works the best. 

And of course, it can be misleading because banner clicks doesn’t necessarily equal revenue and orders.

What I’m just saying is that if you want to A/B test faster, then you have the chance to choose more upper funnel metrics, and then you can just simply learn faster. 

The next question that we often get from clients is that, how long should I wait to conclude an A/B test? Now, it’s a very good question, because you very much have to beware of fake losses and fake wins. 

There’s a good example which I like on the CXL blog. In this example, after just running it for one day, there was 11 conversions on the control, and there were one conversion in the variation, the challenger. 

Based on this one, the chance to beat, the chance to win, basically is 0%, which is not too much. But after a few days, the tide has turned, as you can see, and the variation has 106 conversions, while the control has 87 conversions, meaning that the chance to win became 95% for the variation.

So, to avoid the fake wins and losses, you definitely have to wait until your chance to win is, first of all, more than 90%, but ideally more than 95%. 

Second, each variant should have at least 50 plus conversions. If you have like only 10 or 20 conversions per variant, then there’s a huge chance that you conclude too early.

And the third is that you have to run your test for at least a couple of days. If you conclude your test in one day, there’s a huge chance, again, that you are running into a fake positive or negative. 

All right, and now that’s about the theory, and now let’s see in practice how to run A/B tests in an e-commerce site.

Let’s start with the easiest thing. How to just A/B test your popups? 

Let’s just assume that you have a popup, a list building popup, for example, and you want to A/B test different messages for this popup. This is as easy as it gets, because all you have to do in your OptiMonk account is to set up different variants, basically duplicate the existing variants, change the copied or the other things that you want to A/B test, and then just simply run this campaign.

You can also turn on the automatic evaluation, so basically keep only the winner variant running, meaning that when one of your variants actually loses with a high enough confidence, then the OptiMonk will automatically turn off this variant. 

I think this is very useful if you don’t want to check your campaigns on a daily basis. Now let’s see how you can test a popup versus no popups at all.

Why do you want to do this? If you want to decide if your popup actually generates you revenue and extra money, then you can just simply run versus control, which means no popup at all. 

This is how it looks like in the OptiMonk interface. This is the variant, you know, the popup, and this is the control version, where there will be actually no variant or no popups will be shown.

The methodology behind it is exactly the same as you are running popups. 50% of the traffic will see popup and the other 50% will not see. 

And of course, you can measure the performance of these campaigns, not just on leads and on campaign conversions, which is like subscription, but you can also measure on other goals like visitors with orders, how many people actually bought, how many people actually put something into the cart or view new products.

You can create custom goals if you want, but you can use the built-in goals like the visitors with orders or visitors with carts and optimize for these conversions. 

If you need deeper insights, you can always use the campaign analytics. And here in the left, in the campaign analytics, you can actually measure your performance of your campaigns even deeper.

In the visitors with orders tab, you can see and compare your versions and your campaigns based on their effectiveness of turning visitors into actual customers. Or in the visitors with carts tab, you can measure the performance based on their ability to make people add something to the cart. 

Or you can even see deeper order metrics like how much money they spent or what was the average order value for each of these segments and see the revenue uplift for each different versions.

Now let’s see the next challenge. How to run a web page A/B test? 

Let’s assume that you have a landing page or a homepage or any other important page which where you want to A/B test the copy of your website. For example, this is our homepage and let’s assume that we want to A/B test the key headlines, sub-headlines or even the CTA text.

There are multiple ways to do this, but I’m just going to show you the easiest one. 

The easiest one is basically to just go into your OptiMonk account, create a new campaign, click on Optimize Website, click on Smart AB test, just choose the domain or the URL which you want to A/B test. And then first, our AI will find all the different headlines that’s probably worth testing.

For example, here on this homepage, it found 11 different headlines and it will also recommend different versions for each of these headlines to be tested. 

Of course, these are just recommendations and you are free to change these recommendations in the next step. If you click on Edit and Preview in the next step, our dynamic content editor will load.

This is how it looks like. You will see with this purple that this element has a recommendation and there’s a change here. And that you will also see that here in the preview, you can basically open the preview and see how it actually looks like all the changes.

If you want to actually see and change the AI generated copy, you just have to click on this element. And here you will see that these are the five different versions that was actually set up as an A/B test by the AI. Again, you can just simply edit them, add new versions of your own or delete these existing versions.

And that’s it. And then just click on Next and you can basically start this A/B test. And our Smart A/B Testing will do everything for you.

Basically, it will run sequential A/B test. If you give five different versions, for example, it will start with the first version. In this case, as you can see, it started in this A/B test against the version number one.

And as you’ve seen, it ran actually 10 days, this first version. And even though there were some uplift in conversion rates, 7% actually, it hasn’t reached statistical significance. It hasn’t reached the 90% chance to win that we require a challenger version to have to be the control.

So basically, it discarded this challenger version and went to challenger version number two. And now here in this example, this second test is running. And here, this new challenger version is actually losing at the moment.

But of course, as you can see, the results are still early. And it starts, stops, evaluates, does everything again for you. So you don’t have to always come in daily and check the results.

And if you come back later, when the Smart A/B test is already finished, this is something that you will see. From here, we can see that yes, the first version was promising, but discarded. The second version actually lost.

And the third version was actually a winner with a 97% probability of beating the original version. So now this third version can be considered the winner. The smart A/B test keeps this third challenger variant running until you hardcode this version and replace the original headline with it.

Now let’s see a third and probably the most difficult example how to A/B test product pages. A/B testing product pages is quite difficult. Why? Well, because most brands have a lot of product pages.

If you want to just test one product at the same time, then of course, it’s easy. But if you want to test all the product pages together and see, for example, if adding an extra summary, short summary will improve conversion rate, it is a very difficult game, except if you have the right A/B testing tools. 

Now let’s see how you can do it within OptiMonk.

First of all, you have to open the dynamic content editor and open up just one product page. For example, this is an example product page, which I opened. Now you have to just pick the element where you want to insert the extra element.

So for example, let’s assume that we want to insert a smart element below this headline. After clicking this add the insert smart element, we will see that there are like four different smart elements already configured in this account. Tagline one, tagline two, summary and benefit list.

All we have to make is just click on the use this and this extra headline will be added. And of course, this can be formatted into the look which you prefer. And even though you’ve only added this smart element to just one product page, the system will automatically insert it to all the product pages.

So this way you can very easily A/B test it, very similarly to just one landing page or just one pop-up, having one original, the control version when there’s no AI generated summary. And there will be one version, the challenger versions, where we have the AI generated summary. And again, compare the performance of these campaigns based on any conversion metrics which you prefer.

And then of course, in campaign analytics, you can compare the performance of all these different versions and all these different variants based on any conversion goal and data you want. Based on orders, based on revenue, based on add to cart. 

And now let’s get to the next chapter and see how we can use popups to boost conversions.