In this data-driven age of marketing, we’re often told that it’s all about testing, optimising and testing again. Then, of course, there are people who seem like experts who tell us that we should never, ever, use automated bidding strategies. Why not? Do they know everything about every campaign across every industry?
Instead of taking what we hear from ‘experts’ for granted, let’s put some empirical science to the test and learn from experience. As marketers, let’s take advantage of the modern tools available, combine them with the scientific method and draw conclusions that allow us to make the right decisions for ourselves. And, actually, it’s quite straightforward…
How to make decisions [at least 95%] confidently
TL;DR. Digital marketing makes experimentation easy, so take advantage. However, making sure that you have confidence that your results aren’t simply die to random chance is important. For many A/B type split tests (landing pages, campaigns etc.) the Chi-squared test is great. Simply summarise your results into group and outcome (A convert, A did not convert, B convert & B did not convert) and use an online test such as this handy tool from Graphpad. The p-value is the percentage probability that your results are due to chance, if that’s less than 0.05, there’s a greater than 95% chance that your experiment might be revealing a genuine difference in performance…
The importance of the experiment and statistical analysis
In marketing, it seems as though it’s very easy to be caught up in ‘best-practices’. But what are they, and who decides them? Okay, high quality scores are are always something to shoot for, but Enhanced CPC vs Cost Per Acquisition? Who’s to say? The great thing about digital marketing in particular is that we have unprecedented abilities to segment and target, experiment and optimise, so we have the capability to work out the ‘best practices’ for ourselves.
Also, if you’re doing exactly the same thing as everyone else, how are you going to get any competitive advantage? Putting a floppy disk drive in a computer was probably a ‘best practice’ until Apple came along and decided to not bother with one.
That seems to have worked out okay for them.
So experiment and play with your marketing campaigns, but be wary of the results. As humans, we love to see patterns and, even more than that, we love to imagine that what we’re doing has made a difference, but it’s easy to be misled.
The odds of tossing four heads in a row with a fair coin is just over 6%. That might not sound like much, but imagine a room with a thousand people in it who each flip a fair coin four times. On average, 60 of them will flip four heads, and 60 will flip four tails. Should those people conclude that their coin is weighted?
And that’s why we need statistics: to tell us what the chances are that we can be confident that one campaign or landing page really is out-performing the other. The good news is that, these days, doing the statistics is very easy and you don’t have to understand the maths behind it to draw a conclusion.
An experiment in Google Ads
Should you use an automated bidding strategy? And, if so, which one? Well, let’s take a quick look at an experiment I ran with a client recently. Historically, I have had good luck with Google Ads’ Maximise Clicks bidding strategy. Yes, it’s a bit brute force and you don’t know what the ‘quality’ of each click will be like, but, hey, sometimes brute force wins, right?
In a different sector though, would this work, or would an alternative strategy work better? In this case, I had had some good success with a campaign, and had moved it to a CPA-based strategy, which was performing well and generating a return on advertising spend that the client was happy with. So, we had a benchmark, could we beat it?
This experiment was quite straightforward, all we wanted to do was change a single variable, the bidding strategy. If you get a lot of traffic with a lot of conversions, there is a lot to be gained from changing multiple variables (although it does make the analysis a bit more complicated) but, for most, just changing one thing at a time can make a lot more sense.
Creating a draft of the campaign, changing the bidding strategy to maximise clicks and creating the experiment was a straightforward process in Google Ads (or Adwords as it was then…). The experiment was left to run for a period of time, and the results looked at. Now, there is more to this experiment than meets the eye, but that’s for another post. For now, we’ll just look at the conversion rate.
The results, and the conclusion…
Here are the results we saw from our experiment:
Now, looking at that, I’m sure you’re all behind the backing the CPA campaign argument, and — in this case — you’re probably right, but don’t forget that coin flip warning…
In many cases, particularly with small numbers of clicks and small numbers of visitors, all it takes is one conversion here or there to make one campaign look much better than the other, so let’s see just how confident we can be that the CPA bidding strategy performance isn’t simply due to chance.
To do this, we’ll use a very useful test called a Chi-squared test. What’s great about it is that it can be applied to a lot of marketing questions, all we need to do is give it a table like the one we’ve just created, and we can use a free web-based tool such as this one to crunch the numbers for us.
In this case, I’m going to return to my trusty R to do the analysis, but that’s mostly because I’m already half-way there having used it to prepare the above table (which is a dataframe called cont_table, as this type of table is called a 2×2 contingency table).
Using the simple code chisq.test(cont_table), we get the result p-value = 0.008843, which tells us that there is about a 0.9% probability that both campaigns are as good as each other, so I’m happy to start backing the CPA campaign with a budget increase. As a rule of thumb, 95% confidence is a good place to start (p < 0.05). It might be a bit arbitrary, but research science has made some good progress over the years with that…
