Inter-rater reliability measures how consistently independent raters evaluate the same qualitative material. High agreement makes coded results more repeatable and easier for stakeholders to trust. section 6.3.1 section 6.4
Use inter-rater reliability to assess whether qualitative evaluations produce sufficiently consistent results.
- Compare which topics different raters identify in the same qualitative data. section 6.3.1
- Interpret agreement topic by topic rather than assuming every part of an evaluation is equally consistent. section 6.3.1
- Treat reliability as distinct from validity: consistent ratings do not by themselves establish that the intended concept was measured accurately. section 6.4
Topic coding example
Three raters review qualitative data. All identify “service,” while their selections of “product” and “initial setup” differ; therefore, agreement is higher for “service.” section 6.3.1
Figure 6.13 Inter-rater reliability involves evaluating how much individuals agree on the topics covered in qualitative data. In this example, all three raters identify “service” as a topic, and there is some variation in the other topics they include. As such, the inter-rater reliability for “product” and “initial setup” will be lower than for “service”.
Use the figure to inspect how shared and differing topic assignments reveal variation in inter-rater reliability. section 6.3.1
