Home —> About Us —> History —> Presidential Addresses —> 2010 Presidential Address

2010 Presidential Address

2010 Presidential Address from the AAPOR 65th Annual Conference

The Road to Transparency in Survey Research
by Peter V. Miller

“Despite the urge to proclaim and disseminate in textbook fashion the evolving methodological bases for the newly developed art of survey research, it was increasingly recognized that no one could really prescribe what constituted acceptable practice across the range of varying uses and applications. Instead, AAPOR moved in the direction of disclosure: no rules for conducting surveys would be pronounced but, in the tradition of all science, survey researchers should be required to describe adequately just what they did so that their findings could be objectively evaluated.”
Hollander (1992:68)

After decades of effort, we have yet to reach the place where survey researchers “describe adequately just what they do so that their findings can be objectively evaluated.” AAPOR opted long ago to push for methodological disclosure rather than to write rules for research. While the latter appeared to be too difficult of a route, the path to transparency has also turned out to be rugged and uphill. At various points along the way, we have appeared to gain little ground and wondered if the climb was worth it.

But, for a variety of reasons, the summit is visible now. If we act boldly in coming months and years, AAPOR can make a significant difference in how survey research is practiced and understood. There is still much work to do, the terrain is still difficult, but a route has opened that offers the best chance we have had to achieve the goal of a transparent science of survey research. Let’s examine the new landscape.

A New Communication Environment

The first significant change on the map is the much discussed revolution in public communication. When AAPOR’s standards for disclosure were written, there were three broadcast networks and a handful of national polling organizations. Cable television did not have a significant subscriber base. Satellite television was limited mainly to those living in rural areas who had enormous dishes in the yard. Personal computers were not being sold. Mobile phones did not exist. The Internet had not been invented. A few national publications set the agenda for each day’s news.

We live now in a remarkably different communication environment. Over-the-air broadcasting claims a small portion of television audiences. Cable and satellite television is now the norm. Most households have personal computers and most of them have Internet access. The national publications that used to dominate the news agenda have been joined by a myriad of digital offerings that compete with them for audience attention. Rather than a handful of polling organizations, we now see hundreds offering measurements of public opinion.

These changes have worked for both good and ill. On one side of the ledger, the proliferation of information sources offers audiences information that used to require considerable effort to obtain, if it was available at all. For those with a message to convey, it is now immeasurably easier to reach either large audiences or devoted consumers of specialized content. A great expansion of opportunity to communicate has taken place.

The price of broader access to information is that consumers must expend more time and energy evaluating the surfeit of content. If we were unwise to believe everything we read in the newspaper, we are doubly warned about uncritically taking on board the assertions presented in the typical Blog. Correspondingly, the “downside” of expanding opportunity to communicate is that those recently empowered may not have the expertise or the motive to produce reliable information. Traditional values in public communication—e.g., objectivity, expertise—are up for debate.

A New Research Environment

The second factor shaping our opportunity is the remarkable set of transformations that survey research has experienced in recent decades. Driven by some of the same developments in communication technology that have shaped our current information environment, our field has changed theoretically, operationally, and professionally. When AAPOR’s standards for disclosure were promulgated, the face-to-face interview was the standard method for data collection. The telephone interview rapidly took over in the 1980s, spurred by developments in sampling and by significant cost advantages. More recently, the Internet has come to play a major, if not dominant, role.

The introduction of telephone surveys had an effect on our field similar to the effect of digital media on public communication in general. The technology meant that researchers did not have to rely on a geographically distributed field operation, but could build or buy the services of a centralized telephone facility. The result was a proliferation of survey “organizations,” some consisting of just one member, which contracted with a growing number of “calling centers” to conduct cheap, fast studies.

Familiar names in polling and survey research—Gallup, Roper, Harris, Michigan’s Survey Research Center, NORC, Westat—became exceptions in a field populated by entrepreneurs whose headquarters might be suburban residences. Clients who could not afford to sponsor a study in prior years found, in the telephone survey, unprecedented access to information. More recently, the growth of Internet surveys has escalated the shift in the character of research organizations and their sponsors. Researchers now may be entirely untrained individuals who use cheap software to construct “do it yourself” surveys, relying on volunteer respondents. Research organizations may simply be entities that recruit volunteer panelists with a promised incentive and sell access to them.

As in the public communication environment, we have seen a revolution in both the number and kind of producers and consumers of survey research. A similar breakdown of hierarchy has occurred. Technological shifts have enabled more and more actors to conduct surveys, and their products are now much more accessible to sponsors and other audiences. We can see this development as egalitarian—an opening of our area of study to many who could not previously participate.

Similar concerns arise, however. Many of those who have gained access to research facilities lack the expertise or the scientific attitude necessary to carry out competent research. Many of those who pay some attention to the immense flow of survey information lack the background and perspective to interpret it. In this freer marketplace of ideas, it is significantly easier to misrepresent or even to fabricate research, as some recent notorious examples have shown.

In sum, massive technological changes have transformed both the general public communication environment and the field of survey research. The transformations have opened status borders, enabled new voices, and increased access to information.

But they have not eased the task of understanding. We are faced with a welter of conflicting claims to validity, as new, “egalitarian” modes of research vie with “traditional” methods. Cacophonies of voices, many with overt political aims, assert “truths” about the state of the social world, based on “research” whose provenance is difficult to determine or evaluate.

This difficult environment has produced and will generate many challenges for AAPOR. But it also has opened the road for us to influence events in ways we could not heretofore.

A New Approach to Our Goals

In this time of flux, we, like others formerly outside the mainstream of public discourse, can achieve a strong central voice. Amid the widespread confusion, claims, and counterclaims about polls and surveys, we can be recognized as the authoritative resource on survey research. The new communication environment has given us the means to play this role. Following the example of others, we can significantly increase our influence through skilful use of digital media. The current disordered research landscape has given us the warrant to shape the discourse. In short, we can now be clearly heard, and the need for our message has never been greater.

But these conditions alone would not suffice if AAPOR did not have a sound reputation. Years of struggle to enforce survey standards have given us the credibility needed to make use of this opportunity. Years of work on resources to aid the profession (viz. Standard Definitions) have demonstrated our expertise. While it may have appeared at one time that these efforts were quixotic or invisible, the careful science we have demonstrated can now reap its deserved reward. We alone offer the dispassionate, comprehensive, informed perspective that can mold understanding of polls and surveys.

The way forward now is to augment our current efforts to enforce methodological disclosure with a proactive, positive approach that I have called the Transparency Initiative. In essence, through this program, AAPOR offers its recognition to organizations that routinely disclose essential information about their publicly-released studies. We will help to set up an archive of this information for public examination and scholarly research. Recognizing that many organizations may not have routines for documenting their work, we will assist those in need with educational programs. And finally, we will educate sponsors of research, policy makers, and the public about the importance of transparency. Committees to manage standards and archiving, professional education, and public recognition have already begun work. Scores of organizations and individual volunteers have expressed support and signed up to help. Many more are welcome and needed.

This approach to the goal of an open science of survey research complements our longstanding policy of criticizing those who flout disclosure. Criticism is an important tool—certainly more effective in the current communication environment than in the past—but recognition has a much greater potential reach. Instead of focusing on individual bad behavior, we involve an entire research community in the work of transparency. When coupled with educational programs and a public database, the initiative has the potential to transform survey research.

A New AAPOR

We could not contemplate a venture like this if AAPOR itself had not seen a remarkable transformation in the past two decades. Wise leadership and increased member involvement have laid the foundation for the initiative. We are now a professionally managed organization. We have a more determined and active public stance. We have experience with the long-range planning and methods to start and maintain significant programs. We have a large and growing membership of researchers who take our Code seriously. We have a remarkable volunteer base, willing to take on major problems and provide solutions. We are ready for this next challenge.

The Road Ahead

But we must not underestimate the magnitude of the trials ahead. Setting up the system for routine disclosure will entail defining what transparency means, what level of openness merits our commendation. The mechanism we establish will have to balance desires for information against the costs and practicalities of disclosure. We need to maximize participation in the initiative without making participation so easy as to be meaningless.

Once the significant work of setting up a system for receiving methodological information is done, key challenges will be to make participation continuous and make valuable use of the information we collect. It will be easier to gain initial compliance than to keep organizations involved. It will be easier to collect information than to teach our varied audiences how to interpret and make use of it. For, while transparency seems to be a free-standing benefit, its real value lies in what it enables us to learn about survey research. It is a necessary, but not sufficient, condition for improving survey quality. Misinterpretations of disclosed information are likely; we cannot simply display data and expect good things to follow. To the degree that we can show the theoretical and practical payoff of transparency, we will be able to sustain the initiative over the long term.

Recent events have taught us that disclosure itself can be manipulated. It is disturbingly easy to claim that polls have been conducted using particular methods, while, in truth, the work was not done or was done another way. While we must rely on the integrity of participants in the initiative, we cannot proceed on the basis of trust alone. We must develop ways to check the information we receive. The value of AAPOR’s recognition depends on it.

This is a partial list of obstacles in the road ahead. We must be realistic about these and other challenges. Our achievements may be more limited than we would like—certainly so in the near term. But walking the road will itself provide rewards. We will learn much about what information we truly value, what our profession is capable of achieving, how we can make good use of transparency. The goal of an open science of survey research will continue to urge us on. Let’s join together and continue the journey.

Member Login

Join AAPOR Donate

Donate