On Wednesday, January 27, President Obama gave his
State of the Union Address. Upon completion, the pollsters and
pundants were in usual form. If you happened to watch the
post-address discussion on CNN, you probably saw John King provide
the latest Twitter results. That's right. Polling using Twitter.
John King provided us another example of how technology is aiding
us in collecting survey data.
Twitter, Facebook, LinkedIn along with SurveyMonkey, SurveyPro,
Metrics-that-Matter and many other technologies provide an array of
opportunity to collect data from colleagues, customers, and the
like. While the program evaluation community has embraced
technology to make data collection more convenient, less expensive,
and more interactive, we often have such a reliance on it, that we
fail to realize the potential error in the results that can surface
from depending solely on technology. Types of error most
immediately at risk are coverage error and non-response error.
Coverage Error
Coverage error occurs when we collect data and report results only
from a group of respondents who have access to the delivery mode we
employ. While admittedly, John King's results were not
representative of the country at large, consider some of the people
he missed:
- People who follow CNN on Twitter, but choose not to tweet.
- People who don't follow CNN on Twitter.
- People who don't know about Twitter.
- People who don't have computers.
Non-Response Error
Non-response error occurs when people do not respond to a survey.
With a low response rate it becomes difficult to draw conclusions
with the survey results. People fail to respond to surveys for a
variety of reasons, including (but not limited to):
- Lack of time
- Lack of interest
- No incentive
- No access
- Too many surveys
- Too many emails
- Technology challenged
- Technology resistant
In order to take advantage of what technology has to offer in terms
of data collection and mitigate coverage and non-response error,
consider the following steps taken from the work of Don Dillman
(2009) and other experts of survey research.
1. Identify your primary mode of data collection for a
given survey project.
You may choose technology as your primary mode. If so, then steps
2-5 below will use technology. If you choose paper-based or
telephone surveys as your primary mode of data collection, you will
use whichever one of those to complete the following steps.
2. Provide pre-notice prior to administering the
survey.
This communication will come in the form of email, if you plan to
email your survey; a letter or memorandum if you plan to use
paper-based survey; or a brief telephone call if you plan to use
telephone as your primary method of data collection. The purpose of
the pre-notice is to advise potential respondents of the importance
of the survey. In addition, the pre-notice will explain to them
when they will receive the survey, what they can expect in terms of
time commitment, completion timeline, planned use of the data, and
any incentives you are willing to offer for survey completion.
3. Administer the survey.
Three days after the pre-notice has been distributed, send the
survey. As part of the survey instructions, explain again the
importance of survey, time commitment, completion timeline, planned
use of the data, and incentives.
4. Administer the survey a second time.
After a week or two, administer the survey a second time using,
again, your primary mode of delivery. This second distribution
serves as a reminder and makes it convenient for the audience by
providing the entire survey with instructions.
5. Send a follow-up reminder.
By now, you should have received a large number of surveys. But
there are still a few people who need another reminder. So, using
your primary mode of delivery, send a reminder to those who have
not yet responded.
6. Administer the survey a third time -- using a different
delivery method.
This last contact with potential respondents is your opportunity to
influence people to respond by attacking the issue from another
position. This time, change your delivery method. If your previous
contacts have been electronic, send potential respondents a
paper-based survey or place a call to them. By changing the
delivery method, you give people who have not had access to (or who
chose not to access) your survey opportunity to respond.
* * * *
Reference
Dillman, D. A., Smyth, J. D., and Christian, L. M. (2009).
Internet, Mail, and Mixed-Mode Surveys: The Tailored Design
Method, 3rd edition. Hoboken, New Jersey: John Wiley &
Sons.
Additional Resources
Alreck, P. L. and Settle, R. B. (1995). The Survey Research
Handbook, 2nd edition. New York: McGraw-Hill
Fink, A. (2002) Series Editor. The Survey Kit 2nd edition. Thousand
Oaks: Sage Publications.
Trochim, W. M. The Research Methods Knowledge Base, 2nd
Edition. Internet WWW page, at URL: http://www.socialresearchmethods.net/kb/
(version current as of
October
20, 2006).