Web Survey Guidelines

By Malcolm Conway

Although the jury is still out, several studies indicate that Web surveys produce response rates comparable with, if not better than, traditional paper surveys and telephone interviews. For example, in their paper “From Mail to Web: Improving Response Rates and Data Collection Efficiencies,” Scott Crawford and others conducted a Web survey among university students on the topic of student drug and alcohol use. Two randomly selected samples were assigned to complete either a Web survey or a mail (paper) survey. The response rate for the Web mode was 20 percent higher than for the mail mode. The overall costs for each mode showed that the Web data collection mode is more efficient than the paper mode when used with an established technology infrastructure.

In another example, Jeff Miller in his white paper “Internet vs. Telephone Data Collection: Does Method Matter? Summary of Key Findings” describes research that specifically isolated theeffect of the data collection method. People were recruited by phone to take both Web and phone surveys. Only those with access to the Internet were given the option to participate, and individuals were randomly assigned to take either the Web or telephone version of the survey. Key findings included the following:

• Thirty-three percent of people who completed the Web survey indicated that they would definitely participate in future studies. This contrasted with the finding that 18 percent of those who completed the telephone survey indicated that they would definitely participate in future studies.
  
  
• If a “Don’t Know” or “No Response” option is given in a Web survey, significantly more people will take it than do so in phone surveys, particularly if the question is about sensitive issues, such as honesty.
  
  
• The type of scale matters. Responses to Web survey questions can vary considerably from phone survey questions. Specifically, when all the response scale points are labeled compared with labeling just the end points, phone respondents are significantly more likely to choose the end points. In contrast, when a numerical scale with verbal endpoints was used, there was no difference between the Web and phone scale responses.

Selecting a Web survey tool

Make sure to select an easy-to-use survey tool and avoid tools that require steep learning curves and extensive information technology (IT) support. Consider tools that offer question and scale libraries that are also customizable. Also, consider a tool that has a security feature like Secure Socket Layer so that communications with surveys and the administration site can be encrypted to avoid unauthorized access.

Ensure that the tool is scalable, or that it will work with future as well as current surveying needs.

Finally, make sure that the software installation option fits your needs. For example, find out if you will pay a third party to host the survey on its Website or if you will host it yourself after licensing the tool.

If you decide not to use a packaged survey tool, check that the required skills and expertise are available, as needed. Required skills may include programming with HTML, JavaScript, Visual Basic, or Web survey systems or construction languages; user interface skills to design the computer screens so they are easy to fill out and navigate among; and computer networking or IT skills to manage the survey administration over the network used.

Developing the Web survey

Generally, electronic-based questionnaires should use fewer words and simpler directions because of the limitations of the technology. Limit the number of words to what can fit on a computer screen and be easily read and understood. Scrolling through screen after screen of directions before starting to answer the questions is also very tiresome. Make both the directions as well as the survey as short as possible.

When you create your Web survey, be sure to consider some of the following tips:

Writing

• Avoid leading questions that suggest an answer.
• Eliminate unnecessary questions. Keep it simple. Simplify the questions even more than on paper surveys.
• Use matrix questions sparingly. Matrix questions are questions comprising the rows of a matrix and the common scale they share is displayed as the columns of the matrix.
• Make sure to address any cultural issues, for example, in wording or the use of graphics, when the survey is deployed globally or across multiple geographic areas.
• Consider translating the survey to accommodate respondents for whom English is not their first language.
• Reduce response errors by restricting response choices.
• Be cautious with the meaning of words.
• Make error or warning messages specific.

Designing

• Introduce a Web survey with an interesting welcome screen that is motivational, emphasizes ease of response, and provides instructions on how to proceed and navigate through the survey.
• Make sure that questions and associated response options are visible on the same screen and not split between screens.
• Use page numbers or a progress bar graphic for long surveys.
• Limit the amount of information on a screen to minimize scrolling.
• Brand awareness is important. Include the company or institution or agency logo at the top of each screen.
• Don’t force the respondent to answer a question before the next one is shown on the screen.
• Automate branching from one question to another question based on the answer to the first question so the respondent doesn’t have to scroll or navigate to other screens.
• Allow respondents to interrupt and reenter the survey, as disruptions are commonplace both at home and at work.
• Be aware of how respondents may interpret questions in terms of accompanying graphics.
• Limit the use of graphic images because too many images cause delays loading Web pages.

Pilot the process and tools

When you have designed the instruments, schedule a test run to ensure that the instruments collect the desired information.

Just as software developers and manufacturers never produce software or a manufactured product without first extensively testing it to get the bugs out, so, too, the researcher should pilot-test a draft questionnaire before implementing it. In their book Asking Questions, prominent survey experts and authors Seymour Sudman and Norman Bradburn warn against sending out a survey without first testing it.

Follow these guidelines when pilot-testing a questionnaire or survey:

• Ask people who are familiar with the topic—subject matter experts—to review the content of the draft questionnaire and provide feedback to improve it.
• Use a 6- to 12-person focus group to test the survey. Ask the group to indicate start and finish times so you can estimate the time required to complete the survey. 
•             As participants complete the questionnaire, request that they circle or mark any unclear directions, confusing wording, unclear scales, and other problems. Then lead a discussion with the participants incorporating their comments.
• Have other people review the questionnaire as appropriate. These include an editor to pick up spelling errors, a graphic artist for formatting, a diversity management specialist to screen for unintentional bias (for example, stereotypical wording), or a data analyst or person with  statistical training to give input on how best to format questions to facilitate statistical analysis.
•             For a Web survey, test the survey using different Internet browsers (Internet Explorer and Netscape) as well as different computer platforms (Mac and PC).
• Test a Web survey at different connection speeds before deployment.

Further guidance on conducting pilot tests is contained in Mal Conway and Susan Thomas’s “Using Electronic Surveys,” Infoline No. 250301.

After piloting any data collection instruments that you plan to use and correcting them as needed, it is time to collect the data.

Final analysis

Before you start the process of producing a Web survey, make a realistic assessment of the data collection costs—time, money, labor, hardware, and software—and communicate these costs to sponsors up front. Also ensure that your audience has access to email, the Internet, or intranet before you consider using an electronic questionnaire and that you have enough bandwidth to deploy the survey.

Remember, data collection is a means to an end—program or project success—and not an end in itself. Avoid creating overly complex, costly data collection systems.

This article is excerpted from “Collecting Data with Electronic Tools,” Infoline No. 250404, published by ASTD.