Home —> About Us —> History —> Presidential Addresses —> 2009 Presidential Address

2009 Presidential Address

2009 Presidential Address from the AAPOR 64th Annual Conference

Plus Ça Change, Plus C’est La Même Chose? AAPOR’s Standards Code and Procedures by Richard A. Kulka

The theme of this year's conference is “change,” specifically a plethora of “tectonic shifts” in our culture and society that have placed traditional methods of public opinion and survey research in a state of flux. This theme, its import, and the need to take these trends quite seriously is portrayed quite well by the image chosen for our program cover and all the peripherals used this year—a sandcastle on a beautiful beach whose shape and very existence is at potential peril due to the relentless and steady onslaught of ocean tides, currents, and waves that are both recurring and soothing at times, while more random and menacing at others.

Such changes affect not only our methods, however, and several chapters in our recent history, especially the past few months, have focused in particular on AAPOR's standards—our Code of Professional Ethics and Practices (2005) and our Schedule of Procedures for Dealing with Alleged Code Violations (2009)—which have always been a critical component of the Association's identity and core values, yet may themselves be subject to significant compromise or erosion from these tides of change. Although the significant threats and challenges that these major changes pose for our traditional methods and ways of doing our work are clearly phenomenally important, if these changes—either directly or indirectly—result in an erosion or gradual abandonment of these basic values, upon which our association is founded, the ultimate price to our profession could be far greater. This is not to say that the standards and procedures that AAPOR has evolved over the years have been without controversy, that they have been played out without rancor or dissent, or that they have been consistently applied. As clearly documented in Sidney Hollander's excellent chapter in AAPOR's history—A Meeting Place (1992)—they have not. To the first point, he succinctly notes in the very first sentence: “In the beginning there was controversy about standards” (p. 65).

Nevertheless, this is an especially propitious time and place to review and comment on these issues. First, in recent months we have witnessed three major standards-related issues—some controversial, some not—that have served to highlight and sensitize us to these questions, including (1) a press release citing a nonmember of AAPOR for violation of our ethics code (I would put that in the “controversial” category); (2) the successful filing of an Amicus (“friend of the court”) Brief in support of respondent confidentiality (without apparent controversy); and (3) recent release of the report of the Ad Hoc Committee on the 2008 Presidential Primary Polling, which has highlighted a number of standards-related questions that have yet to be played out. Second, when AAPOR recently cited Gilbert Burnham, a nonmember, for failure to meet our standards for minimal disclosure (http://www.aapor.org/aaporfindsgilbertburnhaminviolationofethicscode), it was the first time that we had done so since 1997, when we cited Frank Luntz, also a nonmember, for the same violation (http://www.aapor.org/aaporfindsfrankluntzinviolationofethicscode). I was the Standards Chair in the first case and President in the second.

Third, when substantial controversy emerged after the Burnham citation, I specifically suggested on AAPORnet, our listserv, that we continue this healthy exchange in various ways, including this conference. And, while everyone here is not a member, signed acceptance of our code is a fundamental condition for AAPOR membership; those of you attending this conference for the first time have been granted complimentary membership conditional on acceptance of the code. Thus, it is a particularly good setting and opportunity (“teachable moment”) to clarify our understanding of our code and the specific ways in which potential violations are identified and dealt with.

To bring these various strands together, I will use two basic sources. The first is our history as presented in A Meeting Place (Sheatsley and Mitofsky1992), because it so clearly demonstrates that most of the key questions and issues raised in recent weeks and months—especially with regard to the Burnham case—are not in fact new. Second, I attempt to use the incredibly rich discussion and dialog among our members played out over many weeks on AAPORnet, and some additional email exchanges to and from members (and others) on these questions, in which we attempted to address their concerns: (1) why we do what we do, (2) how we might continue to enhance and improve our code, (3) how we enforce it, and (4) where some significant gaps appear to exist. Because most of this material refers to the Burnham case and its aftermath, and mostly emanates from our members, a reasonable subtitle for most of the remainder of this address would be AAPOR's Survey Standards and L’Affaire Burnham: In Our Own Words.

AAPOR's Survey Standards and L’Affaire Burnham

In a press release issued by AAPOR on February 4, 2009, it was announced that the Association found that Dr. Gilbert Burnham, a faculty member at the Johns Hopkins Bloomberg School of Public Health, had violated the Association's Code of Professional Ethics and Practices by repeatedly refusing to make public essential facts about his research on civilian deaths in Iraq. When asked to provide several basic facts about this research, Burnham refused.

AAPOR holds that researchers must disclose, or make available for public disclosure, the wording of questions and other basic methodological details when survey findings are made public. This disclosure is important so that claims made on the basis of survey research findings can be independently evaluated. Section III of the AAPOR Code (AAPOR 2005) states: “Good professional practice imposes the obligation upon all public opinion researchers to include, in any report of research results, or to make available when that report is released, certain essential information about how the research was conducted.”

At that time I noted, as part of the press release:When researchers draw important conclusions and make public statements and arguments based on survey research data, and then subsequently refuse to answer even basic questions about how their research was conducted, this violates the fundamental standards of science, seriously undermines open public debate on critical issues, and undermines the credibility of all survey and public opinion research. These concerns have been at the foundation of AAPOR's standards and professional code throughout our history, and when these principles have clearly been violated, making the public aware of these violations is an integral part of our mission and values as a professional organization.

While the framing and wording of that press release was very much like that previously composed in the case of Frank Luntz 12 years ago, it resulted in a firestorm of response—much of it in protest, and especially among our members—that had not been witnessed in 1997. In response to the initial commentary, both on our listserv and direct email, I observed in a posting on AAPORnet (February 5, 2009) thatthis action by AAPOR has clearly lit a major spark on AAPORnet, and that is as it should be. I believe that we will all benefit from what I believe will be a continued healthy exchange. A key issue raised, however, is whether our standards and procedures—all very specific and posted on our website—are fully understood by our members. The issue of investigating and citing nonmembers is one such question; what and how we investigate is another. We all sign up for these when we join AAPOR, but it is worth considering these questions carefully for two reasons. First, they provide an important background for the discussion. And, second, we as members can always revisit and change them if we deem appropriate. I am confident that our procedures were carefully and completely followed in this case, and several members have commented favorably on the actions taken. But the essence of AAPOR is vigorous discussion and debate, and I think that we are all committed to that principle, both in general, and especially as it pertains to standards.

Let us consider a sampling of some of these comments, and responses by myself and others, in some detail.

The Issue of Nonmembers

First, with respect to nonmembers, many of our members, especially those most active on AAPORnet, proposed that it was simply incorrect to launch investigations or take such actions against those who are not members of the Association. While virtually everyone who commented on AAPORnet clearly understood and supported the issue of conducting/reporting ethical research, many raised significant questions and concerns about AAPOR finding a nonmember in violation of its ethics code (e.g., Jeanette Janota, February 4, 2009). In essence, since Burnham is not an AAPOR member:

he suffers no meaningful loss to his reputation that he cares about (Jonathan Brill, February 4, 2009);

has no responsibility for or accountability to our code of ethics (Ron Hinckley, February 4, 2009);

cannot be expected or required to respond to investigations by AAPOR (Marc Sapir, February 4, 2009);

would [likely] not consider [himself] part of the industry (Laurence Shiman, March 17, 2009);

might be totally without fault in all elements of research design and consider AAPOR to be irrelevant or a pest (Phillip Downs, March 17, 2009); and

was entirely entitled to make such a refusal, since [AAPOR] has no realistic claim to “jurisdiction” here (Iain Noble, March 24, 2009).

Or, in summary, as Jan Werner (AAPORnet, March 23, 2009) succinctly and repeatedly opined:I have no opinion one way or the other about Dr. Burnham or the case against him and I don't know why he has refused to reveal details of his research. It certainly does arouse suspicion and AAPOR is absolutely right to remind people that basic principles of scientific research require that he be more open about his methods or provide compelling reasons for not doing so. But he is not a member of AAPOR and has not agreed to abide by the AAPOR Code, so it is inappropriate for AAPOR to accuse him of violating the AAPOR Code of Ethics.

Others were just as vocal in arguing for the need, appropriateness, and value of investigating complaints in apparent violation of our Code against nonmembers, not the least of whom are those who preceded us in leading the organization (cf. Hollander 1992), who have consistently raised, discussed, and reaffirmed the need to include nonmembers in our purview, as an explicit and important component of our code and procedures. While agreeing that Dr. Burnham has no formal accountability to AAPOR, many contributors to this debate strongly endorsed this longstanding position and provided additional commentary and insights into why the Association continues to include nonmembers in its purview. Mary Losch, this year's Standards Chair, clarified that “the purpose of the public censure was to illuminate an important issue that affects the public and scientific community at large. The AAPOR Procedures specifically address the issue of nonmembership and do not make distinctions about behaviors viewed as violating the Code based on membership” (AAPORnet, February 4, 2009). Shortly thereafter, in response to an email, I noted thatAAPOR has always had a strong interest in promoting research standards, and does so in part by having a strong and clear code, a process by which violations can be reported as complaints either by members or nonmembers, and a very detailed process for vetting those complaints and determining which actions, if any, to take. We explicitly do not reject complaints of violations by nonmembers because our purpose is to promote standards in the field as a whole, including minimal levels of disclosure that we feel all those conducting research for a public purpose should adhere to. By publicizing such violations or failures to disclose, our goal is to encourage those who use such data to set a higher standard, and many in fact have already done so.

But by far the most comprehensive and articulate expression and defense of why we investigate and censure nonmembers came in a series of postings by Peter Miller (AAPORnet, March 17 and 23, 2009), our incoming President:Since the offending party did not agree to abide by the AAPOR Code, what business does the Association have in censuring him or her? The idea underlying the question is that AAPOR's Code specifies behavior that is special—a level of responsibility above and beyond what is normally expected of those who do public opinion research. On this view, AAPOR members take on an extra burden when they join the Association. It follows that those who choose not to assume the burden ought to be exempted from Code-based scrutiny. But the idea that the Code calls for extraordinary behavior is fundamentally wrong. Instead, the Code lays out a set of norms to which anyone engaged in public opinion research should adhere. The purpose of the Code is not just to say what AAPOR members believe, but to publicly proclaim what responsible public opinion research is. In other words, the Code seeks to make explicit what constitutes professional behavior. What provisions of the Code should NOT apply to every public opinion researcher? What aspect is so extraordinary that asking researchers generally to comply with it is an unwarranted imposition?In essence, the Code says that researchers should behave ethically and that they should document aspects of their work that have a direct bearing on data quality and may facilitate interpretation of results. The disclosure requirements—officially termed the “minimal” disclosure requirements—apply only to research that enters the public domain. The Code does not prescribe particular research methods. Disclosure is the sole mechanism by which the meaning and quality of work is judged. Perhaps more important, disclosure is the sole mechanism in the field for separating genuine research from fraud. Anyone can claim to have done a poll or survey; disclosure is our only means for validating such assertions. Does it further the profession to say to the world that eschewing AAPOR membership gives one license to behave unprofessionally? Should we ignore behavior by nonmembers that calls into question the integrity of the field? Our predecessors answered no to these questions. We owe them and ourselves fidelity to the principles set forth in the AAPOR Code.[Several have] challenged AAPOR's authority to censure a non-member. Why should we give ourselves this mission, which, it must be said, costs a great deal to fulfill? After all, we are not a “regulatory body.” We are “entitled” to take on this work because we have the expertise, experience and judgment to do it well. No other body is more capable. We “need” to take on this work because there is no other mechanism in our profession to enforce appropriate, basic standards. Our Code certainly has educational and hortatory value. Enforcement of the Code's provisions enhances this value.

Virtually every element of Peter's summary, statement, and rationale was echoed and extended by many others who weighed in on this debate:

Why a serious researcher cannot respond to an inquiry from ANYONE about their research methodology and provide the questions they are asked to arrive at their claims is beyond me. Most serious researchers expect to be asked for this basic information and include it when they release their findings (Mark David Richards, AAPORnet, February 5, 2009).

The professional ethical standards AAPOR uses are only the minimal ones that should be applied when claims are made for scientific survey research. If AAPOR will not address ethical lapses in this field of research, then who will? Shifting the responsibility to someone else hardly seems the way to go. AAPOR has strived to be a leader in the field of survey research, and its willingness to confront ethical problems is one mark of that leadership. As one of the most visible survey research organizations in the country, AAPOR's censorship obviously carries much more weight than single researchers. But by being willing to point to ethical lapses among survey researchers, I think AAPOR provides a needed voice to our political leaders and to the public (David Moore, AAPORnet, March 17, 2009).

If “authority” implies a legal right, then AAPOR does not have the authority to demand from anyone, member or nonmember, that they provide evidence that they have followed “its” scientific standards. AAPOR's authority rests on its standing in the survey research and broader scientific communities (Peter Mariolis, AAPORnet, March 18, 2009).

If we restricted our attention to members only, then anyone wishing to carry out unethical research could simply renounce their membership or let it lapse and they then would be beyond the reach of our sanctions (Tom Guterbock, AAPORnet, March 18, 2009).

All reputable researchers should adhere to the code insofar as it embodies professional standards of research, and this applies both to how surveys are conducted and how they are reported, regardless of the membership in the organization or whether one has explicitly claimed to adhere to the code's standards. We should not be more upset that he disregarded a request from AAPOR than if he had refused to answer legitimate questions from anyone in the public or broader research community. But we should also not excuse his refusal to divulge simply because he is not an AAPOR member (Donald Ferree, AAPORnet, March 24, 2009).

Perhaps the most interesting commentary on our debate, however, came from neither a member of our association nor even a member of our profession, but rather from Thorley Winston's comment posted in response to a column by Megan McArdle in The Atlantic on February 4, 2009:If I had to guess, I would say it's because a professional organization has an interest in how the public sees members of its profession even if they’re not members of the actual formal organization. When Burnham published a statistical “study” that was perceived as being flawed/biased and intended to sway public opinion, it may have lead many people to believe that this is true of all or the majority of similar statistic studies. … By being able to say “as a profession, we hold ourselves to a certain standard of behavior and Burnham did not meet that standard of behavior,” it enables those members of the profession who do follow those standards to distinguish themselves from Burnham. So “censuring” Burnham (if this is indeed what they did) may be a way of protecting the members of their organization who are members of [the] profession from his actions and say that they hold themselves out to a higher standard than Burnham did and here's why.

As the debate wound down in the weeks leading up to the conference (AAPORnet, March 23, 2009), Mary Losch posted the following brief summary and observation: “Clearly, the value and cost/benefit of criticizing nonmembers can and should be debated periodically. Many AAPOR members are very supportive of the current procedures which include nonmembers as legitimate targets of ethics/professional practice violations and some are less so—continuing the dialogue is always healthy and the issues … raised will be conveyed to full council for their consideration.” On balance, however, the instincts and reasoning of our predecessors on the value and importance of making no distinction between members and nonmembers as potential targets of complaint seem as or perhaps even more relevant today.

Our Procedures for Dealing with Alleged Violations

The second basic set of concerns that surfaced in the aftermath ofL’Affaire Burnham were observations about the wisdom and effectiveness of the procedures established for dealing with alleged code violations. These are inextricably linked in many cases to the nonmember issue, but a brief examination of the give and take in this particular chapter of our standards history leads to two basic conclusions. First, in spite of the fact that these have been developed with great attention to detail and, after considerable discussion over many years, are well and consistently documented, many of our members are not very familiar with our standards procedures. Second, whether or not that is the case, several feel that there may be more effective or appropriate actions that we might take, although our current procedures do not really allow for such actions.

As background, in response to an initial flurry of questions about our standards procedures and their application in the case, Mary Losch, Standards Chair, succinctly summarized the AAPOR Code of Professional Ethics and Practice and the Schedule of Procedures for Investigating Code Violations (documented on www.aapor.org), first outlining the eight elements of minimal disclosure described in Section III of the Code (AAPOR2005) and then the basic procedures themselves and how they were followed in this instance (AAPORnet, March 16, 2009):The AAPOR Code of Professional Ethics and Practice outlines eight elements of minimal disclosure:

who sponsored the survey, and who conducted it;

the exact wording of questions asked, including the text of any preceding instruction or explanation to the interviewer or respondents that might reasonably be expected to affect the response;

a definition of the population under study, and a description of the sampling frame used to identify this population;

a description of the sample design, giving a clear indication of the method by which the respondents were selected by the researcher, or whether the respondents were entirely self-selected;

sample sizes and, where appropriate, eligibility criteria, screening procedures, and response rates computed according to AAPOR Standard Definitions. At a minimum, a summary of disposition of sample cases should be provided so that response rates could be computed;

a discussion of the precision of the findings, including estimates of sampling error, and a description of any weighting or estimating procedures used;

which results are based on parts of the sample, rather than on the total sample, and the size of such parts; and

method, location, and dates of data collection.The Schedule of Procedures explicitly indicates that any AAPOR member (or non-member) may submit a written complaint alleging a violation of the AAPOR Code. Per the Procedures, these investigations are confidential to provide protection to both the complainant and the target of the complaint. The allegation may be made against a member of AAPOR or a non-member. In this case, an allegation was made that Dr. Burnham violated the AAPOR Code in the survey he conducted in Iraq and about which he published results in the October 2006 issue of the Lancet. As is true for every allegation of Code violation, the Standards Chair, Associate Chair and Council followed the AAPOR Schedule of Procedures and in this case, launched a formal investigation. In the course of the investigation, the evaluation committee sought specific information related to the study. Dr. Burnham refused to provide the elements of minimal disclosure or any other information. Based on this unambiguous violation of the Code, the evaluation committee recommended a public censure and Council agreed.Commentary on these procedures raised a large number of specific themes or questions with regard to how this case was handled. Some are fairly specific and straightforward. For example, the possibility that the protection of confidentiality of respondents (Eleanor Singer, AAPORnet, February 4, 2009) or requirements by sponsoring agencies that certain information be kept confidential for reasons of safety (Ron Hinckley, AAPORnet, February 4, 2009) might underlie Burnham's refusal to disclose requested information was raised. However, as noted by Mary LoschDr. Burnham provided no indication that he was under any confidentiality constraints regarding the release of the information requested. To the contrary, he simply stated that he would not provide the specific information requested. We have no insights regarding his motivation for refusing our request for basic elements of his study such as the full questionnaire, sampling protocols, and dispositions, etc. (AAPORnet, February 4, 2009)

Nevertheless, in a press release issued on February 23, 2009, the Johns Hopkins Bloomberg School of Public Health (2009) reported that it had also investigated the Burnham Survey, including an examination of something that the AAPOR Standards Committee had been unable to obtain: a data-entry form. The responses on these forms apparently included—contrary to the IRB protocol—identifying information, an action judged unethical by the JHU committee which led to a five-year suspension of Dr. Burnham as a principal investigator on grants involving human subjects. While the elements of minimal disclosure requested as specified by the Code do not include identifying information, this parallel investigation, at least in part, possibly explains Burnham's non-cooperation with AAPOR.

Other more complicated and far-reaching concerns expressed in this discussion related to

timing, such as going public with our concerns when the article appeared rather than censoring the author two years later;

alternatives to the specific approach taken, such as academic or peer review or public comment, rather than using a complaint-based system involving formal investigation, adjudication, and possible censure;

targeting only Burnham, rather than other co-investigators/collaborating organizations and/or those who published the study;

possible perceptions of political motivation or bias in case selection, including failure to also investigate others who reported research and estimates yielding results in conflict with those of Burnham's study;

lack of transparency in the complaint/investigation protocols, including prohibitions against identifying complainants and revealing the basis and details of their complaints; and

jurisdiction, e.g., by what authority does AAPOR have a right to take such actions.

While in some ways these issues and concerns are distinct, in practice many are inextricably linked or dependent in much of the commentary that emerged following the Burnham press release, and virtually all of them reflect misunderstanding and/or frustrations with regard to a very specific set of prescribed rules and protocol codified in AAPOR's (2009) Schedule of Procedures for Dealing with Alleged Code Violations for many years.

For example, Ron Hinckley (AAPORnet, February 4, 2009) suggested thatpublicly speaking out against the article as lacking proper information to validate the findings and requesting the author, through the publication (Lancet), to produce said information would be a more timely and maybe a more proper way to approach the challenge. Investigating and finding in violation of common scientific standards … can come later. [And] what about the actual group that conducted the poll? I’m sure Burnham did not do this himself. Was that part of the information he refused to divulge? Can this entity be held accountable, too?

Similarly, Mark David Richards suggested thatquestions about AAPOR's timing and protocol for launching and issuing a notice of violation could be useful if it drives AAPOR toward a more uniform and systematic policy for responding to violations. The organization obviously needs to avoid the appearance of being politically motivated or biased in its case selection. It does seem rare when AAPOR takes a formal position- and it seems that it takes a position when the stakes associated with unsubstantiated research claims are extremely high (from a political or public policy influence point-of-view). Perhaps it takes so long because AAPOR is so cautious. I’m sure most people have experienced violations of the code where AAPOR has not taken action. (AAPORnet, February 5, 2009)

In an effort to address these concerns and suggestions, Mary Losch noted (AAPORnet, February 4, 2009):It would be very difficult for AAPOR to make any sort of public statement without having a fair and thorough investigation of the points of contention. While always more potent in the early months following a study release, there was nothing at that time to trigger an investigation of the study methods.One practical problem (among many) with making critical public statements early after publication is that someone must be charged with deciding what makes the cut for AAPOR's criticism—who does the review of all published surveys to determine which should receive a public statement regarding their methods? There are probably hundreds of studies each year that might legitimately merit a public statement if we did not use a “complaint” system.The complaint in this case was brought against Dr. Burnham. If a complaint was brought against another firm working with/for him, it would also be investigated. In this case, only one target was specified in the formal complaint.

Related to Ron Hinckley's suggestion that approaching the publication (Lancet), rather than the author, might have been timelier and more effective are assertions that independent academic or scientific review and dialog might have been a more appropriate approach to addressing this issue. For example, Hugh Gladwin (AAPORnet, March 18, 2009) suggested that “an academic review through published articles and comments would have been a better place to start.” Similarly, in an email message to the author on February 5, 2009, Debora McKenzie, a correspondent for theNew Scientist, observed that “it does seem an unusual thing for a dispute involving scientific methodology to be settled by an anonymous complaint procedure involving an investigating committee. As I think Hopkins noted in its response to this, these things are usually done in the peer-reviewed literature. Is there something different about survey-based research that warrants this approach?” In an email response the same day, I emphasized thatour focus is on the appropriate minimal standards of disclosure for public opinion and survey research that produces results made public, and these are pretty basic. If all of these issues were disclosed to peer reviewers, then there is no obvious reason why they should not also be available to anyone that requests them, and no reason was given for not doing so (such as confidentiality). We have taken no position on the scientific methodology per se, since not enough information was provided to do so, which is the point of our concern and a violation of our standards. In the case of disputes on scientific methodology, however, it seems particularly important to disclose the basic information that we requested so that all interested parties can evaluate what has been done and the likely impact on the results presented. It is quite possible that if those methods had been disclosed there would have been no violation, even if some would still disagree with their methods.While we certainly agree that peer reviewers may have been provided all of the information that our standards require, there is no way of knowing this, and if they were not, it would appear to undermine that very process. As part of their peer review process, several journals, including our Public Opinion Quarterly, require that such information be available not only to reviewers but also to anyone that requests such information, a standard that we are trying to promote with other publications that have submissions that use such data. The information provided in the article per se is not sufficient to make appropriate judgments, and our standards hold that this seriously undermines not only the scientific process, but also the public's right to know.

I also noted that an academic review of this controversy had, in fact, already been published in Public Opinion Quarterly (Marker 2008).

Two observers noted the slight irony of suggesting that an academic or scholarly review might have been a preferable strategy to address this issue. Lance Pollack emphasized thatin the absence of knowing what the “real” number is, the validity of any estimate will turn on methodological issues (sample frame, sampling procedures and execution, recruitment procedures and execution, interviewing procedures and execution, representativeness of the sample, instrumentation, etc.). These are the very pieces of information that Burnham has declined to disclose. Thus, the validity of his “different number” cannot be determined, which means real scholarly discussion is short circuited by incomplete information. The very academic review [that] is desirable is precluded by Burnham's failure to disclose the information requested. (AAPORnet, March 18, 2009)

Stephen Blumberg, Associate Standards Chair, further observed thatthere seems to be a lack of understanding that an independent scientific review of the methods (a role seemingly encouraged for AAPOR) cannot occur unless the methods are disclosed (which AAPOR has no right to require, by some opinions). I agree that it would have been nice if the Lancet editors or the JHU leadership had been able to help us with our disclosure requests, but the current procedures don't permit it. Perhaps we should modify procedures to permit such requests for intervention? (AAPORnet, February 5, 2009)

Nevertheless, a key reason for Hugh Gladwin's (AAPORnet, March 18, 2009) suggestion that “an academic review through published articles and comments would have been a better place to start” was his concern that, “while it may be unfair … AAPOR's censure will be seen by many outside our field as having political implications and even motivations,” an observation and concern also expressed by several others, two of whom have already been cited (Ron Hinckley and Mark David Richards). In addition, Michael Sullivan, while acknowledging thatAAPOR is charged with protecting the legitimacy of discipline of survey research and as such needs to take a stand against the publication of survey evidence (especially in important public policy contexts) that does not follow the basic cannons of scientific investigation, asks … is there a way to do that without getting into ethical questions. Why not write a letter to the editor of Lancet and other publications that have published this work (derivatively) explaining that in the opinion of AAPOR the survey results presented in their publication did not comport with the basic cannons of scientific investigation and should be retracted by the editor immediately with an explanation of why this is necessary. Rather than having a mechanism that many of us would agree is capable of political manipulation and extremely dangerous (i.e., censure), wouldn't it be better to destroy the credibility of the research and punish the editor for failing to properly vet the proposed research report? (AAPORnet, March 18, 2009)

Fred Goldner was also concerned about potential political manipulation, but from a somewhat different perspective, one inherent in a complaint-based system currently in place:Did our standards committee investigate the methodology employed by all those who issued casualty figures or did they only inquire about Burnham's methodology because someone had questioned his? … It would be irresponsible to inquire about the methodology employed by one side resulting in censure without doing the same for other sources … it would not be appropriate for AAPOR to take such a one sided stand. (AAPORnet, March 25, 2009)

Perhaps the strongest such statements on the potential for political motivation, manipulation, and bias, which also indict even more directly the complaint-based system that underlies AAPOR's Schedule of Procedures—as well as its jurisdiction (even with regard to members)—were offered by Jan Werner in a series of postings (AAPORnet, March 16, 23–24, 2009). First, he asks ‘why [the Executive] Council has not provided answers to two questions essential for any fair inquiry: (1) what exactly is Dr. Burnham accused of having done that would justify AAPOR's demand for supporting information; and (2) what is the source of the allegations and/or who brought the accusation before the AAPOR Standards Committee?” He also suggested that AAPOR would be on firmer ground if it were more consistent in calling its own members on the disclosure requirements spelled out in the Code. Second, he warns of potential political and reputational consequences in taking such actions:AAPOR is not a regulatory body and has no standing to compel anyone to follow its rules, even within its own field of Public Opinion Research. Nor does it have a mandate to adjudicate every dispute that involves survey methodology or practice in some way. By claiming those powers, AAPOR has set itself up to be taken advantage of by nearly any person who wishes to use our reputation to advance his case in a scientific dispute. If so, the Standards Committee could be very, very busy in coming years.What does matter is that any perception, correct or incorrect, that AAPOR can be manipulated into appearing to support one side in a debate sets a terrible precedent. It also has the potential to severely damage our reputation as an impartial professional organization.

Third, he suggests a different set of rules and procedures that the Executive Council and the various committees might follow in handling AAPOR's affairs in cases like L’Affaire Burnham:The Code could be strengthened with respect to many aspects of our profession, most notably disclosure … AAPOR needs to be more outspoken in publicizing these standards and noting when they are not met, whether by AAPOR members or others. In the situation at hand, I would have liked to see AAPOR issue a statement saying that it had been brought to the attention of the Standards Committee that Dr. Burnham had failed or refused to provide relevant materials to other researchers attempting to evaluate his findings, and that if so, such behavior is contrary to the principles of science and survey research for which AAPOR stands and to which its members are pledged. AAPOR should take this kind of action far more often. In particular, it should call out its own members when they fail to live up to the standards of open disclosure provided by the Code. … But this is very different from launching an investigation into the conduct of a researcher and issuing a formal statement of censure for ethics violations.

All three sets of comments (and those of others preceding these) attracted significant attention, response, and clarification. Stephen Bloomberg (AAPORnet, March 16, 2009), for example, clarified that

AAPOR, through its Council, chooses to investigate allegations. The investigation is initiated by AAPOR. The person originally bringing the accusation is known to the Council and can be considered by the Council when deciding whether to investigate. But the person's identity should remain anonymous publicly to encourage anyone and everyone to bring necessary allegations to the awareness of the Council.

As for the exact allegation, this cannot be disclosed as the investigation failed to find evidence to support all of the points raised in the initial allegation. Unfounded accusations should not be made public.

As for being more proactive against our own members, I fear that the need for the Council to pick and choose investigations in the absence of accusations could lead to accusations of various biases. But perhaps there is a fair way to establish a more proactive system.

Blumberg (March 24, 2009) also acknowledged as a potential “middle ground” the suggestion made by Werner, while also expressing caution regarding its possible implementation:I believe that Jan Werner offers a “middle ground” here that is worth further discussion, especially as Council considers possible modifications to the Code and the Schedule of Procedures. He suggests that, when a non-member survey researcher has engaged in conduct that is inconsistent with the AAPOR Code of Professional Ethics and Practices, AAPOR should not censure for “violation of the Code,” but rather should publicly announce that the individual's actions are inconsistent with the standards of ethics and practices of professional survey researchers. In my opinion, a formal investigation would still be necessary to determine if the standards were inconsistent and to provide the non-member with an opportunity for due process. And … such a public statement [seems] akin to saying that the non-member violated our Code of Professional Ethics and Practices.

Richard Rockwell (AAPORnet, March 23, 2009) emphasized thatthe “Schedule of Procedures for Dealing with Alleged AAPOR Code Violations” does not provide for release by AAPOR of the name(s) of the complainant(s). The Procedures do not permit anything other than what AAPOR did. … Council has not identified him publicly in any of the statements issued by AAPOR [and] would have been properly reprimanded had it done so. In fact, keeping the name of the complainant(s) confidential would seem to be a sovereign remedy against “By claiming those powers, AAPOR has set itself up to be taken advantage of by nearly any person who wishes to use our reputation to advance his case in a scientific dispute.” One can hardly advance one's reputation by being anonymous.

Similarly, Eleanor Singer (AAPORnet, March 17, 2009) succinctly summarized and validated the procedures followed, while also highlighting the inherent tension and burden in carrying them out:

There are two issues being debated on AAPORNET at the same time. One is whether the action taken by the Council in this particular case was appropriate, and the other is whether the Code or the procedures to implement the Code should be changed. The Standards Committee and the Council appear to have followed precisely the procedures prescribed when a complaint is brought. Those procedures protect the complainant from having his or her identity disclosed. They also specify that anyone, member or not, may bring a complaint, against anyone conducting a survey and failing to adhere to the standards laid out in the Code, which … are minimal standards.

I happen to agree … but it potentially imposes a huge burden on the organization, one that as a result it may fulfill only selectively. That in turn may cause a problem or the appearance of a problem. Apparently, the organization is going to re-examine the AAPOR Code; but for the moment, it is what it is, and was followed faithfully.

Peter Miller (AAPORnet, March 17, 2009) emphasized several key points with regard to code enforcement that are not well known throughout our membership as a whole:

AAPOR asks for methodological details only for studies whose results have been put into the public domain, and about which a standards complaint has been made.

Enforcing the Code is an arduous, time-consuming job. It is not a responsibility that AAPOR or any organization assumes without a careful thought. A look at the history of the Association, Paul Sheatsley and Warren Mitofsky, A Meeting Place gives a sense of how we arrived at our current position.

Code enforcement involves responding to complaints. Complaints can have many different motivations. Regardless of motivation, the merit of the complaint is judged on whether a violation of the Code has occurred.

In disclosure cases, the Standards Chair is empowered to try to get requested information before an official case is begun. Once a case is begun, the complaint is vetted by an independent committee, charged with judging whether the Code has been violated. The committee makes a recommendation to the Council about what action should be taken. Committees can and have decided that complaints have been resolved or were unfounded.

Given the time and effort involved in standards cases, one does not take them on lightly.

With regard to this particular case, Miller (March 24, 2009) further reminded us that

The censure of Gilbert Burnham was a public announcement that he did not disclose basic information about his research. The censure did not address the findings of the research. The censure did not take a side in the dispute between Spagat and Burnham. The inference, apparently drawn by several contributors to this discussion, that AAPOR was “used” by Spagat or that it endorsed Spagat's position is erroneous.

In standards cases, the Association responds to complaints about violations of the Code. … It does not seek out parties to punish for their views. It “does” respond to complaints about violations of the Code. Those charged with handling complaints are not dupes or people “with an agenda.”

Complaints can come from any source. They may, indeed, come from people who have “an axe to grind.” That they may do so does not mean that the Association's finding of a Code violation implies that the Association endorses the viewpoints of the complainant. The issue is whether the Code was violated or not. The Code does not mandate disclosure of information just to “people who agree with me” or even to “people of good will.” It mandates disclosure regardless of the source of the query.

Readers might have noticed … that there have been [only] two cases of public censure over the past 12 years. During this period, there have been many more standards complaints. The inference from these facts is that public censure is rare. Many standards complaints are determined to be out-of-scope (concerning something not in the Code) or unfounded. Many others are resolved when the accused party discloses information on response to the Association's inquiry. Censure is usually not the outcome of a standards case. [However], the “possibility” of censure may lead to disclosure of methodological information that was not disclosed in response to queries from individuals.

Tom Guterbock, a former AAPOR Standards Chair who led the most recent task force appointed to revise the standards procedures, emphasized that, after substantial debate, they reached a consensus that we should continue AAPOR's tradition of standards enforcement despite its cost and difficulty. Most important was the need for the protection of the complainant's identity in the process (as well as protection of any person accused of a violation, up until such time as public censure occurs). And, while initially worried that our rules would allow people with political or other motives to bring accusations (e.g., gaining competitive advantage over a rival), he concluded (consistent with Peter Miller's assertion) that, in the end, the accuser's motivation or political affiliation is not relevant to deciding the merit of the accusation itself (AAPORnet, March 18, 2009).

Summary and Conclusion

In summary, as I noted at the outset, “In the beginning there was controversy about standards” (Hollander 1992, p. 66), and there still is. But our code and procedures have also been the heart and soul of our association since the very beginning. As Hollander (1992) notes in his conclusion on “unfinished business,”the history of the Standards Committee has been riddled with discontinuities and punctuated with tabula rasa appraisals—sometimes with indifference to, if not ignorance of, what has gone before. Consequently, we have sometimes been condemned to repeat our history. Such are the hazards of a membership organization dependent on volunteers, whose officers in charge of dealing with standards issues are constantly changing with each election. Then, too, the world in which AAPOR members operate is not that of four [or certainly six] decades ago, when the practice of opinion research was dominated by the same few conspicuous leaders who founded the organization. The practice of opinion [and survey] research has proliferated and dispersed, and its practitioners are no longer a tight knit and collegial fraternity. (pp. 102–3)

Arguably, the situation is worse, perhaps even far worse, since that was written. So why bother? Several of those who have commented on the most recent chapter of our history in that regard have already more than hinted at the answer. For example, Patrick Murray emphasized:With the plethora of polls out there, our industry needs a watchdog. Especially when those polls are influencing important policy debates. Who better to take on this role than AAPOR? I view AAPOR's activity as “protecting the brand.” I wish they did so more often, although I appreciate the caution needed when choosing to act. (AAPORnet, March 18, 2009)

In that spirit, let me close by calling attention to a column posted by David Hill in the daily congressional newspaper The Hill on February 10, 2009:Dick Wirthlin, the renowned Republican pollster, once groused that the polling industry was being ruined by “low barriers to entry.” Wirthlin got his start, of course, in the days when privileged pollsters could access mainframe computers at universities. Then the personal computer hit the market, followed by the broad dissemination of SPSS and similar canned software packages to crunch poll data. “Field service” call centers sprang up, too, offering cheap WATS long distance. Suddenly, some guy operating a “polling firm” out of a van down by the river was ostensibly offering the same service as Dr. Dick Wirthlin.It wasn't just the hardware and software that made Dick Wirthlin different, however. He had formal academic training, experience and judgment that the van guy didn’t. One unique element of the training of most “academic” pollsters was and is an immersion in the history and ethics of polling.Since Dr. Wirthlin's melancholy critique, things have only gotten worse. Essential tools of political polling are more accessible to the untrained than ever. Online services like SurveyMonkey (an apt label) will probably accelerate the slide of the industry. A clear majority of pollsters today has no formal academic training in the trade.There is no inherent justification for pollsters with an operative background to have coarser principles than researchers with academic training. [And], to be fair, it must be acknowledged that one of the few pollsters ever formally censured by a professional organization was an Oxford-trained Ph.D.The American Association for Public Opinion Research (AAPOR) tries to stand in the breach of this chasm between polling principles and political practice. These are the “good guys” (and gals), most of whom labor in academia or serious research organizations. They volunteer their time and effort to hear complaints regarding violation of AAPOR standards. Violations can result in censure, as when AAPOR slapped Frank Luntz 12 years ago for his behavior in relation to research supposedly undergirding the Contract with America. Political pollsters aren't the only culprits. A public health researcher at Johns Hopkins was cited by AAPOR on February 4.It's good that AAPOR is there, because the threat of censure may encourage some good behavior.Most pollsters revel in our relatively unregulated enterprise. No training required. No certificate needed. No rules in place. The vacuum of that freedom, if abuses persist, will eventually be filled.

Hollander (1992, p. 103) observed that “our influence has generally been brought to bear not by committee or organization action, but by the diffusion throughout the domain of public opinion [and survey] research and its sponsors of practitioners imbued with standards and ethics acquired from AAPOR.” That said, who is better qualified to assume this important responsibility and burden if not us? In a book that I admire a great deal—Good to Great—Jim Collins (2001) observed that companies that moved from merely good—to truly great performance—were those who achieved a clear understanding of (1) what they are deeply passionate about and (2) what they could be best in the world at. The passion is clearly there, and several of our members, leaders, and sister organizations have posited that we are indeed better positioned and well-suited—perhaps more than any other organization in the world—to take on this burden. Although our methods and procedures for doing so clearly need to evolve significantly with our changing times, is it not indeed both our legacy and destiny to do so?

Member Login

Join AAPOR Donate

Donate