Conducting a representative survey of American adults is an increasingly difficult proposition. Of late, technological and social changes have conspired to increase the cost of reaching and soliciting opinion from the public. These challenges have been well known to practitioners of survey research for quite some time; the American Association of Public Opinion Research has held dozens of panels at its annual meeting on the issue of cell phone interviewing alone. As we head down the final stretch of the presidential campaign, which will be awash in poll numbers, the media has also taken notice.
This type of news coverage has never been more important, not only because it identifies emerging challenges survey researchers face, but because it provides consumers of political news with the information necessary to make informed judgments about the claims made by surveys and those who cover them.
Pollsters face a daunting array of challenges. Telephone response rates are falling, the numbers of households without a landline telephone are rising, and bilingual interviewing is becoming increasingly important to adequately represent the growing number of Latinos in the country. Each of these trends significantly increases research costs. For example, largely because of legal restrictions that require cell phone numbers to be manually dialed, a cell phone interview can be twice as expensive as a landline interview.
The increasing cost of conducting traditional telephone surveys has encouraged some pollsters and entrepreneurs to develop alternative methods or modes of data collection. Online surveys are the most common alternative to telephone surveys and provide an increasingly attractive option for cost-conscious researchers. However, with a few notable exceptions, nearly all of these online options present a fundamental problem: they do not use a random probability sample, which allows researches to make inferences about the wider population. Rather than building their panel from a random sample of recruits, most online survey companies allow panelists to self-select onto panels, often in response to online advertisements or email solicitations.
This debate invariably pits the purists against the pragmatists, who tussle over the methodology and economics of opinion polling. But there are several important considerations that consumers of opinion polling should note:
- Any survey of the general public should include some percentage of respondents who are interviewed on a cell phone, which ensures representation of the increasing number of Americans who have abandoned landlines (40% of PRRI’s respondents on major surveys are reached via cell phone).
- Surveys that purport to represent the views of some larger population must be based on probability samples.
- Interviews should be conducted in both English and Spanish, and at least some reasonable proportion of interviews should be completed in Spanish.
- The survey should include an explanation of how the weighting was performed, including a list of variables that were used to balance the sample. Most surveys use basic demographic characteristics (i.e. age, gender, race/ethnicity) based on the U.S. Census Current Population Surveys, but some also use political ideology or partisanship. These decisions, which are rarely noted in the discussion of about a survey’s findings, can have significant implications.
Understanding these challenges is the first step toward being a savvy consumer of polls. Producers of survey research should make every effort to be transparent about their methods. Every survey conducted by PRRI includes the complete topline questionnaire and a methodology statement. The datasets are also publicly available for download at the Association of Religious Data Archives and the Roper Center for Public Opinion Research at the University of Connecticut.
Open, honest debates about real challenges presented by current telephone usage trends are testament to the health of the public opinion research industry. The emergence of innovative methodologies that incorporate the best of past standards while exploring new techniques provide some real reasons for optimism about the future of this profession and the invaluable knowledge it provides.