Frequently Asked Questions

We hope these FAQs help answer the many questions about email marketing testing, statistics applied to email marketing sample sizes and our calculators.

  1. What statisitcal formulae do the calculators use?
  2. What is meant by sample size?
  3. How do I pick a sample size?
  4. What is statisical significance?
  5. Why should I worry about statistical significance?
  6. What is a confidence level?
  7. What confidence level should I look for? 80%, 95%, 99% or higher?
  8. What if my sample sizes are too small?


What statisitcal formulae do the calculators use?

The concept of statistics is not hard to grasp. However the detail can need a mathematical expert to decipher. After much research and consideration we finally settled on modelling by considering the open, click or conversion streams as bernoulli trials. The bernoulli trial outcomes are then analysed with a t-Test to determine the confidence interval.

What is meant by sample size?

In email marketing test terms, the sample size is the number of people to which the email has been sent.

How do I pick a sample size?

It is normal in an A/B marketing testing to pick the same size for the number of people to which the A email version will be sent as the B. The aim is normally to pick the smallest sample size that will show a statistically significant difference in result. This is to conserve list data for a follow up email or to use for further tests.

What is statisical significance?

A result is statistically significant if there has been enough experimentation to exclude it happening by chance.

Why should I worry about statistical significance?

Using information that is not statistically significant can cause you to make wrong marketing choices and get poorer marketing results, reducing your ROI.

What is a confidence level?

A confidence level is the level of probability that a result or hypothesis is statistically significant. For example in a classic A/B marketing test the hypothesis is basically that one version provides a better user response than the other. Our calculators will work out whether given the size of difference in response and the number of people in the market test, the sample size, the confidence level. A 95% confidence level means that there is 95% probablity if the test was repeated the result would be the same.

What confidence level should I look for? 80%, 95%, 99% or higher?

Typically people look for a 95% confidence level. The appropriate level depends on the outcome of getting it wrong, if you are playing Russian Roulette you might what a high confidence level of avoiding the bullet. For a marketing test, using a 80% level to inform your approach and further test and optimise can be acceptable.

What if my sample sizes are too small?

Assuming you don't have more data to use to provide larger sample sizes then one solution is to aggregate results from several sends. This is a compromise and brings in possibilities of interference in the tests. However it is better than not testing.