Spotlight on Qualitative Methods: Do I Need Multiple Coders?

In one of my favorite qualitative methods articles, sociologist Mario Small argues that the logic and language of qualitative methodologies can be “lost in translation” when these methods are incorporated into predominantly quantitative research domains. As someone who teaches qualitative research methods in a school of public health and publishes qualitative work in public health and medical journals, I have witnessed some of these translation challenges. One area where this often occurs is in discussions about the use of multiple coders and inter-coder reliability.

The use of multiple coders has become a common, if not expected, practice in qualitative public health and medical research.  Qualitative papers published in medical and public health journals frequently describe coding teams that “double code” data and assess inter-coder agreement.  In my experience, reviewers for public health and medical journals frequently request information about coder agreement, and the absence of additional coders can lead to the perception that the work is not rigorous.

. . . many esteemed qualitative anthropologists and sociologists do not employ this practice, and many seminal qualitative methods textbooks do not even discuss multiple coding.

Despite the ubiquity of multiple coders in the qualitative public health and medical literatures, qualitative methodologists disagree about the importance or even suitability of this practice. While some have argued that having multiple coders increases the reliability of qualitative findings, others have raised concerns about this technique, suggesting that it was adopted from quantitative research without considering its appropriateness for qualitative studies. Additionally, while the use of multiple coders seems to be expected in the public health and medical literature, many esteemed qualitative anthropologists and sociologists do not employ this practice, and many seminal qualitative methods textbooks do not even discuss multiple coding. Given these differences of opinion, I would like to take a moment to reflect on the use of multiple coders.

What are some things multiple coders can add to a qualitative study?

First, multiple coders can contribute to analysis when they bring a variety of perspectives to the data, interpret the data in different ways, and thus expand the range of concepts that are developed and our understanding of their properties and relationships. For this reason, some experts recommend coding teams with varied disciplinary training or life experiences. Multiple coders can also help check interpretations against the data. While each researcher views the data through the lens of their own knowledge and experience, multiple coders may be able to help prevent this lens from obscuring participant experiences and meanings. Second, identifying places where coders do not agree can be a useful way to identify codes that aren’t sufficiently well-defined, thus producing greater conceptual clarity and strengthening the analysis. Third, when the goal of qualitative analysis is to create variables for subsequent quantitative analyses, multiple coders can help to ensure consistency of categorization, thereby minimizing measurement error.

. . .identifying places where coders do not agree can be a useful way to identify codes that aren’t sufficiently well-defined, thus producing greater conceptual clarity and strengthening the analysis. 

What are some concerns associated with the emphasis on multiple coders?

However, some qualitative methodologists have raised concerns about the emphasis on multiple coders and inter-coder reliability in public health and medical literatures. First, the increasing emphasis on multiple coders may discount other important strategies that qualitative researchers can and should employ to enhance rigor and trustworthiness, including long-standing practices such as reflexivity. Furthermore, though “coding” is sometimes perceived as synonymous with “analysis,” it is only one of the many tools that qualitative researchers can use to analyze their data. Memo writing, for example is critical to qualitative analysis, yet infrequently discussed in the qualitative public health and medical literatures.

Second, the need for agreement across coders can come at the expense of interpretative insight. As methodologist Janice Morse argues the “use of the second coder will keep the analysis superficial, obvious, insignificant and trite.”  Indeed, codes that are less nuanced are easier to apply consistently.  Relatedly, coding requires an understanding of context, and qualitative coding decisions often evolve iteratively over time as a researcher develops increasing sensitivity to this context and their data. A new coder who does not have the same knowledge could not be expected to code the data in the same way.

. . .the need for agreement across coders can come at the expense of interpretative insight.

Additionally, for many qualitative studies, attaching the “wrong” code to a piece of data does not have the same consequences as it would in a quantitative study. Most qualitative researchers do not report the number of times a code appears. Instead, they review, reflect, and write about data that are associated with a particular code. This process of iterative analysis provides an opportunity to revise codes or their assignment to data excerpts as the analytic process develops.

Finally, employing a second coder is resource intensive. The perceived obligation to do so can be a barrier for some researchers, particularly those who have fewer resources to draw upon.

In summary, there may be reasons to employ a second coder in the process of qualitative analysis. But there are also reasons not to, and the strength of one’s study does not rest solely on this decision. Journal and grant reviewers should critically consider the added value of this practice in their evaluation of qualitative research. Additionally, while the use of multiple coders may be an important strategy to navigate research gatekeepers, doing so without a genuine methodological rationale may contribute to the perception that this process is always a necessity.