Improving Your Survey Response Rates

You disliked an off-service rotation and wonder if your classmates had similar experiences. What do you do? A survey!

Surveys are ubiquitous now, and response rates to surveys of all types have been steadily decreasing across the US.[1] Why? Consider the last time you received a survey and immediately threw it away or deleted it, and you have your answer. 

Although response rates have less of an impact on nonresponse bias than we previously thought[1,2] they are still important to provide an understanding of how well the sample was represented.[3]

The decision to respond has three components: delivery, acknowledgement, and cooperation.

1) Delivery:
a. Use more than one method of delivering your survey.[4] Even as recently as 2013, postal surveys outperformed email surveys if you compare them 1 on 1. However, using both email and postal surveys can increase the response rate by almost 10%. Think outside the box: phone call, text, personally handing out paper surveys, social media, etc. The more routes of delivery, the more opportunity you have to meet people in their preferred media. (As a side bonus, you can also reduce the potential for nonresponse bias.)

2) Acknowledgement: 

a. Make your correspondences clear and professional. Ten percent of postal surveys go completely unnoticed as surveys.[5] Put simply, you lose 10% of your response rate before you even get a chance to try to convince the potential respondents that it is worth their time. Send an advance notice (explained in more detail below) that looks professional, is hand addressed, and is inviting.

3) Cooperation:

a. Give cold, hard cash, up front, without requiring survey completion, and make the amount just enough to utilize the guilt factor to gain a 15-20% absolute increase in your response rate.[6-8] The sweet spot is somewhere between $1-2 for the general public and $2-5 for physicians. Non-monetary incentives like food and trinkets do not perform as well as money; neither do lotteries. You may think you’re more likely to complete a survey for an iPod lottery than for $2 in your pocket, but the literature is very strong in this area. Guilt is an extraordinarily powerful tool, much better than chance.

b. Make it short, but do not divulge the total amount of time.[9] Just say “short.” Everyone has a different idea of what “short” means, so by using that word directly, you are more likely to reach the “ok, I’ll take a short survey” thought process than if you offer the amount of time and leave it to the potential respondent whether or not that is short. Just tell them it is short, and keep it to your word—less than 10-15 minutes and less than 1,000 words.

c. Send out at least three attempts to get people to take your survey.[10] Most people who will respond do so within 24 hours, and 90% respond within two weeks, so send your reminders somewhere between 2 and 14 days. Change the method by which you send your reminders (e.g. paper versus email) and change the days and times that you send the reminders (e.g., Sunday afternoon versus Wednesday morning) to try to reach people when it is more convenient for them.[11]

d. Send an advance notice.[12] This is a short letter that introduces the potential respondent to the survey sponsor and allows potential respondents to expect to give a “short” amount of their time to the sponsor to take the survey. They are mentally prepared for it and are therefore more likely to respond.

These methods are not mutually exclusive, and you should use different methods within your means to gather the most responses as possible. More breadth in respondent demographics (by using different methods of convincing them to reply) may also reduce your nonresponse bias, but that is a topic for another day. For now, go forth and survey!

References

1. Groves RM. Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly. 2006;70(5):646–675.

2. Johnson TP, Wislar JS. Response rates and nonresponse errors in surveys. JAMA. 2012;307(17):1805–1806.

3. NRC. Nonresponse in Social Science Surveys:. National Academies Press; 2013.

4. Millar MM, Dillman DA. Improving Response To Web and Mixed-Mode Surveys.

5. Dillman DA. Mail and internet surveys: The tailored design method. 2000.

6. CHURCH AH. Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly. 1993;57:62–79.

7. Asch DA, Christakis NA, Ubel PA. Conducting physician mail surveys on a limited budget. A randomized trial comparing $2 bill versus $5 bill incentives. Medical Care. 1998;36(1):95–99. Available at: http://journals.lww.com/lww-medicalcare/Abstract/1998/01000/Conducting_Physician_Mail_Surveys_on_a_Limited.11.aspx.

8. Singer E. The Use of Incentives fo Reduce Nonresponse in Household Surveys. In: Groves RM, Dillman DA, Eltinge JL, Little RJ, eds. Survey Nonresponse. 1st ed. 2000. New York: Wiley-Interscience.

9. Dijkstra W, Smit J. Persuading reluctant recipients in telephone surveys. In: Groves RM, Dillman DA, Eltinge JL, Little RJ, eds. Survey Nonresponse. 1st ed. New York: John Wiley & Sons, Inc; 2002.

10. Lynn P, Clarke P, Martin J, Sturgis P. The Effects of Extended Interviewer Efforts on Nonresponse Bias. In: Groves RM, Dillman DA, Eltinge JL, Little RJ, eds. Survey Nonresponse. Wiley-Interscience.

11. Hamilton MB. Online Survey Response Rates and Times. Ipathia, Inc; 2014.

12. Edwards P. Increasing response rates to postal questionnaires: systematic review. BMJ. 2002;324(7347):1183–1183. doi:10.1136/bmj.324.7347.1183.