Chapter 8 - More About Mail Surveys
So, how could I possibly run a survey? My budget is tight. I have no staff and limited facilities. It's 10 degrees below 0 outside, and I'm not going to stand on a corner intercepting angry commuters all day. The thought of sitting on the phone repeating, "Do you strongly agree, agree, neither agree nor disagree, disagree, or strongly disagree" to two hundred people is not exactly glamorous either. Are the prospects of conducting a survey completely impossible?
Certainly not. Mail surveys are a powerful, effective, and efficient alternative to their more expensive relatives- the telephone survey and the personal interview. A quality, medium-scale mail survey can be conducted with minimal cost, little staff, and no complex equipment. And a well-conducted mail survey can be just as effective and meaningful as other more resource-intensive surveys.
Our nation's postal workers don't need to tell you about the volume of mail that ends up in your mailbox each day. Encouraging participation in mail surveys is not a simple task under these circumstances. A survey that is lost in a sea of magazines, bills, and personal, business, and advertising mail may never be found.
Although there are some drawbacks to mail surveys, it is possible to obtain valid results with the right kind of questionnaire and distribution technique. Well-prepared surveys can be executed successfully when the conditions are right. This pamphlet provides an overview of some of the strengths and weaknesses of mail surveys, a basic understanding of what is needed to conduct a successful mail survey, and ways to determine if this type of survey is appropriate for your situation.
The Advantages of Mail Surveys Cost Effectiveness
The use of mail surveys has increased dramatically recently-and for good reason. In terms of time and money, they are very economical. One way to demonstrate this is to compare and contrast mail surveys with telephone and face-to-face surveys-the other two very common ways in which to conduct a survey.
With regard to human resources, mail surveys require very little manpower. It is possible for only one person to conduct a mail survey, where as the time needed to conduct the same number of telephone or face-to-face interviews is usually much too great for a single person. Mail surveys are also significantly cheaper than telephone and face-to-face interviews. The cost of a medium-scale mail survey in a single metropolitan area might run from $5,000 to $10,000. The costs of equal-sized telephone and face-to-face surveys are estimated at 50% and 150% more, respectively. If a national (or worldwide) survey is considered, these cost differentials would be even greater.
This is primarily because postage costs are relatively low and uniform, regardless of the geographic area being covered, but telephone rates and, especially, personal interviewer expenses are higher and may also differ from one area to the next. Moreover, the total cost of sending a three-question survey is the same as for one with 100 questions, assuming that you can mail both out for the same amount of postage. So, the surveyor gets more for the dollar.
Mail surveys also enable specific segments of the population to be easily targeted. For example, if you are surveying a city on the newest location for a garbage dump, you can compare the different areas of the town and the reaction to the garbage dump through mail surveys directed at certain neighborhoods.
Finally, some studies show that people provide more honest answers to mail surveys than they do to other interviewing methods. Privacy-especially if your survey is about a delicate issue-may be important to the respondent, and mail surveys may increase the credibility of the answers.
The bottom line is, that given enough time, you may want to use a mail survey, especially if you are subject to severe money constraints. But in doing so, you must also be aware of their disadvantages.
Potential Disadvantages of Mail Surveys Coverage Errors
Many people assume that the biggest disadvantage to mail surveys is a low response rate. This is not necessarily true. Good planning can lead to response rates as high as those obtained in telephone or face-to-face interviews. The main problem with mail surveys is procuring an accurate list of people in the population from which to draw the sample for your survey. Failure to do so can lead to coverage error. One important form of coverage error occurs when mailing lists are incomplete (for example, not including college students living away from home). Mailing lists may also be biased (a list of licensed drivers may under represent poor people, the very young, and the very old). Other lists may be inaccurate (containing duplicates or names that do not belong on the list) or out of date (omit people who have recently moved into the survey area or including people who have moved away).
Wording of Questions
Another area of critical importance to mail surveys is questionnaire design-poorly worded questions are a survey breaker. Questions must be simple, short, and precise. Unlike telephone and face-to-face interviews, there is no opportunity for explanation or follow-up questions like "Do you have something specific in mind?" Questions left open to interpretation will produce unusable results.
Other factors can be of particular concern in mail surveys. For example, did you ask questions that are too personal? Is the questionnaire too long? Is the questionnaire legible and easy to follow? Can the person to be surveyed read? If, for instance, you are conducting a survey in a southern border town in Texas and neglect to include a copy of the questionnaire in both Spanish and English, the chances for success are greatly reduced.
The disadvantages of mail surveys leave room for large errors. But many of these errors can be reduced significantly with overall awareness, good planning, careful wording of questions, thorough preparation, and pretesting.
Design and Format of Mail Surveys
The appearance of mail surveys can have a large impact on the percentage of responses received. A surveyor is usually asking the respondents to volunteer their time to fill out a questionnaire for which they will receive no instant response, benefit, or gratification. If the survey makes the task difficult by providing an unattractive design or format, giving poor directions, or including confusing questions, the respondent is more likely to choose not to donate their time "to the cause." Extreme care must be taken to ensure a design and form at that emphasize professionalism, quality, and attractiveness. Even factors such as poor re production, inadequate stapling (which might allow the survey to fall to pieces), or the lack of a sponsor's name will detract from your final response count.
A great deal of care must go into selecting and designing the questions for your mail survey. First, be sure the questions will yield the kind of information you are looking for. If you are conducting a customer satisfaction survey for a magazine, simply asking people if they are satisfied with the magazine is not very useful. A follow-up question on why they are dissatisfied will help to improve your publication in the future. Second, although there are a number of different formats that can be used in soliciting responses (e.g., agree/disagree; rank your answers from 1 to 5; or open-ended replies) it is best not to jump from one type of question to another. The respondent is likely to find such a survey frustrating or, worse, more trouble than it is worth.
Questions must also be examined closely for bias and fairness. Be aware of leading questions with wording that may influence your results. Writing appropriate and balanced questions is a very complicated topic in itself and requires research beyond the scope of this pamphlet, so proceed with caution. Questions should also be designed to take into account the amount of effort or burden they place on the respondent. For instance, if questions require detailed calculations, research sensitive personal information, or long-term memory recall, they are likely to go unanswered and may even discourage the respondent from replying to the survey at all.
Finally, the design of the questions must be straightforward, unambiguous, and logical. It is important that the flow and format of the survey be intuitive-the more difficult it is to follow the survey pattern, the less likely the respondent will successfully complete the questionnaire. If complex directions cannot be avoided, use formatting, indenting, and shading to assist the respondent. In the following examples, one answer to a question requires further detail.
This example causes confusion about how to answer the question if you don't own your own business.
8. Do you own your own business?
9. How many workers do you employ fulltime?
Under 20 20-49 50-100 Over 100
10. Does your spouse work full-time?
8. Do you own your own business?
8a. How many workers do you employ full-time?
Under 20 20-49 50-100 Over 100
10. Does your spouse work full-time?
In this example, shading and indentation guide the respondent and do not detract from the flow of the survey.
Overall, strive for a survey that is uncluttered, legible, easy to follow, and uncomplicated to answer. The publishing production of the survey is also a key factor in having the questionnaire opened by the respondent. Mail surveys should be printed on high-quality paper. Be sure all materials are reproduced clearly and cleanly.
A personalized cover letter should be included with the survey itself. The cover letter should explain the reasons for the survey, express assurances of confidentiality, and identify the survey sponsors. Preparatory and follow-up materials should use the same fonts, graphics, and formatting styles as the survey itself, in order to convey professionalism.
Also, a familiar logo appearing on each document you send increases the likelihood that the recipient will associate the original survey mailing with reminders that come later on.
Mail Survey Logistics
It is important to remember that there is more to a successful mail survey than simply sending out one bulk mailing of questionnaires- no matter how carefully you have selected your sample or designed your questionnaire and printed your materials. In order to get the high response rates that mail surveys are capable of achieving, the following implementation steps are strongly suggested:
Common Pitfalls of Mail Surveys
- Use multiple contacts, including
- Send a preliminary mailing announcing the survey.
- Mail the survey to all respondents at the same time, with an accompanying cover letter.
- Send a reminder, with contact information, to request a replacement questionnaire or answer general questions about the survey.
- Send replacement questionnaires by First-Class Mail.
- Send the last replacement questionnaire by two-day Priority Mail.
- Send an acknowledgment card, thanking respondents for their cooperation.
- Use printed stationery and personalized letters with logo and contact information.
- Include a stamped, pre-addressed return envelope.
- Include a token of thanks-$1 to $5- with your initial or replacement mailing of the questionnaire, as an incentive and sign of respect.
- Pitfall #1: "I can use the white pages in the city phone book to draw my sample of people who live here."
Always examine a list before assuming that it answers all of your problems. In this example, a telephone book may seem to be comprehensive, but it contains many natural flaws. Poorer families may not be able to afford their own telephone. Wealthier or larger families may have multiple phone lines and, therefore, multiple phone book listings. Certain professionals tend to have unlisted phone numbers and would be excluded from the list outright. A phone book may also be more out of date than other available lists due to deaths and families that moved. Examine your list for inherent biases, check and recheck its accuracy, and look for ways to make any list more complete and more comprehensive.
- Pitfall #2: "I don't have the money to spend on these fancy booklets. Just stick the photocopy in an envelope and send it off. That way we can spend more money on the analysis."
A bad-looking mail survey will guarantee a poor response rate. With the high volume of mail that flows into the average American home on a daily basis, your survey must be professional and eye-catching in appearance; otherwise, it could simply end up in the trash unread. Investing in an attractive survey will save you money in the long run by delivering a high initial response rate and increasing your chances for accurate and meaningful results.
- Pitfall #3: "I'll just send out this stack of surveys, and when they all come back, we'll tally the results."
Simply sending out a straight mail survey may result in a return rate so small that the results are meaningless. The key to success is follow-up. A reminder postcard, a return incentive, a second copy of the survey sent to anyone who didn't return the first copy, or a follow-up note outlining the value of the completed survey to the recipient-these seemingly small follow-up steps can significantly increase the percentage of surveys which eventually return to you. Preparatory steps, such as teaser postcards alerting the recipient that the survey is en route, also increase response.
These guidelines will help you achieve very good response rates from your mail survey- perhaps even better than the results you could obtain by telephone. The key is to show you care about getting a response- personalized surveys that include incentives, sent by First-Class Mail or Priority Mail, and with several follow-ups, send that message to the respondent far better than a one-time, photocopied, bulk mailing with no return envelope.
Where Can I Get More Information
Don Dillman has written extensively on self-administered surveys, notable mail surveys but also their logical extension Internet surveys.. A good recent source for more information from him is Dillman (2000), Mail and Internet Surveys: The Tailored Design Method.