Statistical Significance

• How to determine if you have a statistically significant sample size (clicks/impressions) to make a determination on a targets effectiveness.

Statistical significance will play a large part in many of the decisions you will make while reviewing and optimizing your campaigns. For example there are many instances in this guide where a target is referenced as having a specific ACoS. Let's say 30%.

The underlying assumption in those statements is that the ACoS is statistically significant. Meaning that there is a sufficient ammount of data (impressions/clicks) on those targets for that number to actually mean something.

If an item has one click and no sales. That is a 0% ACoS. However the very next click could be a sale and that could skyrocket the ACoS to 500%. In either case, it would not be smart to view that ACoS as a meaningful number. We simply don't have a large enough sample size to make a determination.

So how can we determine if we have a statistically significant sample size?

In traditional statistics a sample size of 10% is generally accepted as statistically significant. That being 10% of the audience size.

However, with Amazon Advertising we do not have access to the search volume or audience data that would be required to determine the size of the audience. Additionally tools that approximate the sizes of those audiences are not accurate enough to be used for this purpose.

As a result we have to come at this problem from another angle.

The Goal

The goal is to determine the threshhold we should use for determining if we have a statistically significant sample size of clicks or impressions.

If we have reached that threshold and the target has not generated the expected number of clicks or sales then we can assume that the target is not effective.

Approximating Statistically Significant Impressions

How can we determine how many impressions we need to be able to make a statistically significant determination on a targets effectiveness?

• Average Click Through Rate is approximately 0.4%.

The formula for click-through-rate is:

$$CTR = \frac{Clicks}{Impressions}$$

We can plug in the average CTR:

$$.04 = \frac{Clicks}{Impressions}$$

Then we need to determine how many clicks we need to get a statistically significant sample size. The Central Limit Theorem states that the sample size needed to make a determination on a population is 30. We can plug in 30 for clicks:

$$.04 = \frac{30}{Impressions}$$

and solve for impressions:

$$Impressions = 750$$

Therefore as a general rule we can say that we can expect 750 impressions to make a statistically significant number of clicks, to then make a determination a targets CTR, Sales and Conversion Rate.

Are we safe using the average CTR for this calculation? Probably not. We will discuss that in the next section.

Adjusting The CTR

While the average CTR is 0.4%, we probably already have some data on the average CTR of the ad that this target is associated with. We could use that to adjust the number of impressions needed to make a statistically significant determination. But should we?

The 750 impressions is the number of impressions that we would expect to generate 30 clicks, which is what we need to make a statistically significant determination on conversion rate.

But we also want to learn the CTR of the target. If we use the average CTR of the ad group then we are making an assumption that the target will have the same CTR as the ad group.

We want to learn the actual CTR of this particular target. What we should do instead is pick a CTR that is our minimal acceptable CTR for this ad group.

If the target reaches the baseline impressions calculated from that CTR and has not already reached the 30 click threshold then we can assume that the target is not effective.

Let's assume that our minimal acceptable CTR is 0.3%. We can plug that into the formula:

$$.03 = \frac{30}{Impressions}$$

Which would give us a baseline impressions threshold of 1000 impressions.

Either Or

We have made a determination that we need 1000 impressions to make a statistically significant determination on a targets Click Through Rate. If we have not reached 30 clicks by that point the target is innefective.

However we could also reach 30 clicks before we reach 1000 impressions. If that happens we know the target has an acceptable click through rate, and we have the clicks we need to make a determination on conversion rate.

Based on those determinations, we can use an EITHER OR approach. We can either use the 1000 impressions threshold OR the 30 clicks threshold. Whichever comes first.

Takeaways

• We can use 1000 impressions or 30 clicks, whichever comes first, to determine if we have a statistically significant sample size to make a determination on a targets effectiveness.

Relating to Profit Optimization Models

Tying this back in to profit optimization models, which would be programatically adjusting the bids on targets, we now know that we would want to exclude any targets that have not reached the 1000 impressions or 30 clicks threshold. These targets would be considered to be in a "learning/discovery/research" phase and we would not want to make any adjustments to them.