Note from Cape Wind
The Cape Codder newspaper reports the results of their online poll on Cape Wind that found a majority of local online participants opposed, as well as a reporter's survey of beachgoers at Craigville Beach that found most in support - to their credit, the Cape Codder acknowledges in these articles that neither survey was scientific.
Last year, an independent scientific survey of public opinion on the Cape and Islands on Cape Wind, commissioned by the Cape Cod Times and WCAI / WNAN, found an even split in public opinion. Earlier this year, the Civil Society Institute commissioned an independent scientific survey of public opinion on Cape Wind that found 81% support in the State of Massachusetts and their sub-sample of Cape and Islands residents also found more support than opposition.
There is an important distinction between online, self-selecting surveys (that can only provide information about those who actually choose to participate in the survey), and scientific surveys using random sampling that are designed to estimate public opinion of the general population. The three sources of this information are the National Council on Public Polls, the American Association for Public Opinion Research, and the American Statistical Association.
1) National Council on Public Polls
The National Council on Public Polls (NCPP) is an association of polling organizations established in 1969. Its mission is to set the highest professional standards for public opinion pollsters, and to advance the understanding, among politicians, the media and general public, of how polls are conducted and how to interpret poll results. Since its inception, NCPP has sponsored seminars, workshops and press conferences in Washington State and New York to promote better understanding and reporting of public opinion polls.
20 Questions A Journalist Should Ask About Poll Results
Sheldon R. Gawiser, Ph.D. and G. Evans Witt
Polls provide the best direct source of information about public opinion. They are valuable tools for journalists and can serve as the basis for accurate, informative news stories. For the journalist looking at a set of poll numbers, here are the 20 questions to ask the pollster before reporting any results. This publication is designed to help working journalists do a thorough, professional job covering polls. It is not a primer on how to conduct a public opinion survey.
The only polls that should be reported are "scientific" polls. A number of the questions here will help you decide whether or not a poll is a "scientific" one worthy of coverage – or an unscientific survey without value.
Unscientific pseudo-polls are widespread and sometimes entertaining, but they never provide the kind of information that belongs in a serious report. Examples include 900-number call-in polls, man-on-the-street surveys, many Internet polls, shopping mall polls, and even the classic toilet tissue poll featuring pictures of the candidates on each roll.
One major distinguishing difference between scientific and unscientific polls is who picks the respondents for the survey. In a scientific poll, the pollster identifies and seeks out the people to be interviewed. In an unscientific poll, the respondents usually "volunteer" their opinions, selecting themselves for the poll.
The results of the well-conducted scientific poll provide a reliable guide to the opinions of many people in addition to those interviewed – even the opinions of all Americans. The results of an unscientific poll tell you nothing beyond simply what those respondents say.
By asking these 20 questions, the journalist can seek the facts to decide how to report any poll that comes across the news desk.
The authors wish to thank the officers, trustees and members of the National Council on Public Polls for their editing assistance and their support.
10. What about polls on the Internet or World Wide Web?
The explosive growth of the Internet and the World Wide Web has given rise to an equally explosive growth in various types of online polls and surveys.
Online surveys can be scientific if the samples are drawn in the right way. Some online surveys start with a scientific national random sample and recruit participants while others just take anyone who volunteers. Online surveys need to be carefully evaluated before use.
Several methods have been developed to sample the opinions of those who have online access. The fundamental rules of sampling still apply online: the pollster must select those who are asked to participate in the survey in a random fashion. In those cases where the population of interest has nearly universal Internet access or where the pollster has carefully recruited from the entire population, online polls are candidates for reporting.
However, even a survey that accurately sampled all those who have access to the Internet would still fall short of a poll of all Americans, as about one in three adults do not have Internet access.
But many Internet polls are simply the latest variation on the pseudo-polls that have existed for many years. Whether the effort is a click-on Web survey, a dial-in poll or a mail-in survey, the results should be ignored and not reported. All these pseudo-polls suffer from the same problem: the respondents are self-selected. The individuals choose themselves to take part in the poll – there is no pollster choosing the respondents to be interviewed.
Remember, the purpose of a poll is to draw conclusions about the population, not about the sample. In these pseudo-polls, there is no way to project the results to any larger group. Any similarity between the results of a pseudo-poll and a scientific survey is pure chance.
American Association for Public Opinion Research (AAPOR)
AAPOR was founded in 1947 by a group of dedicated public opinion research pioneers.
AAPOR is an association of about 1,600 individuals who share an interest in public opinion and survey research. Members work in a wide variety of settings, including academic institutions, commercial firms, government agencies and non-profit groups, as both producers and users of survey data. Election polling, collecting statistical data, conducting market research and improving methods for surveying individuals and instituons are just a few of the diverse research interests of AAPOR members.
- AAPOR encourages quality survey methods through its official journal Public Opinion Quarterly, its annual conference, and educational opportunities
- promotes standards of professional conduct and ethics for surveys and public opinion research facilitates informal
- networking through the AAPORnet listserv and regional chapter meetings
Survey Practices That AAPOR Condemns
Representing the results of a 900-number or other type of self-selected "poll" as if they were the outcome of legitimate research.
900-number and other types of write-in, call-in, and interactive polls have become increasingly common. These "polls" report the opinions of only those people who called in, and not those of the general public. AAPOR believes that any publicizing or promotion of such activities not only damages legitimate market and survey research, but can be very misleading when used to influence public policy or simply to disseminate information about the general public.
Best Practices for Survey and Public Opinion Research
A replicable or repeatable plan is developed to randomly choose a sample capable of meeting the survey's goals. Sampling should be designed to guard against unplanned selectiveness. A survey's intent is not to describe the particular individuals who, by chance, are part of the sample, but rather to obtain a composite profile of the population. In a bona fide survey, the sample is not selected haphazardly or only from persons who volunteer to participate. It is scientifically chosen so that each person in the population will have a measurable chance of selection. This way, the results can be reliably projected from the sample to the larger population with known levels of certainty/precision.
…Virtually all surveys taken seriously by social scientists, policy makers, and the informed media use some form of random or probability sampling, the methods of which are well grounded in statistical theory and the theory of probability.
3) American Statistical Association
The American Statistical Association (ASA) is a scientific and educational society founded in 1839 with the following mission: To promote excellence in the application of statistical science across the wealth of human endeavor.
“What is a Survey” by Fritz Scheuren, published on the website of the American Statistical Assocation.
This "What is a Survey" booklet is written primarily for non-specialists and is free of charge. Its overall goal is to improve survey literacy among individuals who participate in NORC Surveys or use NORC survey results. The material is taken from the American Statistical Association (ASA) series of the same name, which I edited, that was designed to promote a better understanding of what is involved in carrying out sample surveys - especially those aspects that have to be taken into account in evaluating the results of surveys. -- Fritz Scheuren
In a bona fide survey, the sample is not selected haphazardly or only from persons who volunteer to participate. It is scientifically chosen so that each person in the population will have a measurable chance of selection. This way, the results can be reliably projected from the sample to the larger population.
The quality of a survey is largely determined by its purpose and the way it is conducted.
Most call-in TV inquiries (e.g., 900 “polls”) or magazine write-in “polls”, for example, are highly suspect. These and other “self-selected opinion polls (SLOPS)” may be misleading since participants have not been scientifically selected. Typically, in SLOPS, persons with strong opinions (often negative) are more likely to respond.
Virtually all surveys taken seriously by social scientists and policymakers use some form of random sampling.
(The Cape Codder articles are no longer online)