The question “How many people do I need to contact to ensure my campaign result is statistically significant?” is probably one of the most common questions asked when planning direct marketing activity. And whilst an answer of “it depends” can be hugely frustrating, sadly it does depend on a 3 key factors…
- What response rate are you expecting?
This can be a difficult one to answer if you haven’t run a similar campaign before and will vary considerably by industry, product, offer etc. However where possible you should use your own historical activity performance as a guide or use a benchmark from a similar campaign or industry. - What limit of error are you willing to accept?
This is the factor than can be the most difficult to comprehend as the answer that most people want to give is understandably “None!”. Probably the easiest way to think of ‘Limit of Error’ is to put it in the context of the response rate range you’d be willing to accept. As a guide, we generally recommend using 10% of your expected response rate. So if you’re expecting a response rate of 2.00% and you select a limit of error of 0.20% (10% * 2.00%) then essentially you’re willing to accept a response rate between 1.80% and 2.20%. - How confident do you want to be in the result?
If you accept that you can never be 100% confident that if you were to repeat the activity you will get the same result (in the response rate range you’ve specified), you obviously want to be as sure as you can that your result is not susceptible to error. The general rule is to use a 95% confidence level and we would always recommend this where possible, however there are times when you may have to accept a greater level of risk to make the test viable.
Remember that calculating a required sample size is intended to give you the best chance of generating results that are statistically valid and the more you test and learn the better you’ll get at implementing robust tests and driving improved results. Ultimately though, it’s the final result of a campaign that will dictate how reliable your test was and this can fall either side of your expected response rate and one thing is for sure, until you test something you’ll never know.
If you need help understanding the significance of a test then check out our result range calculator tool.
