Interrater and intrarater reliability in the diagnosis and staging of endometriosis

Karen C Schliep; Joseph B Stanford; Zhen Chen; Bo Zhang; Jessie K Dorais; Erica Boiman Johnstone; Ahmad O Hammoud; Michael W Varner; Germaine M Buck Louis; C Matthew Peterson

doi:10.1097/AOG.0b013e31825bc6cf

Research Methods · Measurement and Statistics

Interrater and intrarater reliability in the diagnosis and staging of endometriosis

Schliep KC, Stanford JB, Chen Z, Zhang B, Dorais JK, Boiman Johnstone E, Hammoud AO, Varner MW, Louis GM, Peterson CM

Published July 2012 Obstetrics and gynecology, 120(1), 104-112

DOI 10.1097/AOG.0b013e31825bc6cf PMID 22914398 PMC PMC8495766

RRM Academy Synopsis

Expert surgeons largely agree on endometriosis in surgical images

Eight expert surgeons reviewing operative images from 148 women agreed substantially on whether endometriosis was present (Fleiss κ=0.69). They agreed less on stage (κ=0.44).

Key Findings

Across all eight surgeons, agreement on whether endometriosis was present was substantial: Fleiss κ=0.69 (95% CI 0.64-0.74).
Among the four academic expert surgeons, diagnosis agreement was Fleiss κ=0.79 (95% CI 0.70-0.88). The four local, specialized expert surgeons scored κ=0.58 (95% CI 0.50-0.66).
On stage, the eight surgeons showed moderate agreement, Fleiss κ=0.44 (95% CI 0.41-0.47), and matched on stage in a mean of 61% of cases (range 52-75%).
Comparing each surgeon's own stage call with computer-assisted staging from that surgeon's checklist gave a mean weighted κ=0.95 (range 0.89-0.99), almost perfect.

Interpretation

The study is a reliability study. Eight surgeons rated operative photos or video from 148 women, a random sample from the ENDO Study operative cohort, without clinical history. Kappa measures agreement beyond chance, so results describe consistency among raters. Raters reviewed the staging criteria first and knew their ratings would be analyzed. The authors state that findings for less experienced surgeons or other locations remain to be established. Image quality varied, and the local surgeons came from a small area of Utah. Links between stage and clinical outcomes remain undeveloped.

RRM Context

Surgical visualization is the diagnostic standard this study tested. Restorative reproductive medicine starts from a named diagnosis of the underlying condition. Surgeons who agree on whether endometriosis is present give that approach a consistent starting point. Stage alone leaves open what a woman's pain or fertility will look like.

Our editorial summary of this paper, not the article's abstract.

Abstract

Objective

To estimate the interrater and intrarater reliability of endometriosis diagnosis and severity of disease among gynecologic surgeons viewing operative digital images.

Methods

The study population comprised a random sample (n=148 [36%]) of women who participated in the Endometriosis: Natural History, Diagnosis and Outcomes study. Four academic expert and four local, specialized expert surgeons reviewed the images, diagnosed the presence or absence of endometriosis for each woman, and rated severity using the revised American Society for Reproductive Medicine (ASRM) criteria. Interrater-level and intrarater-level agreement were calculated for both endometriosis diagnosis and staging.

Results

The interrater reliability for endometriosis diagnosis among the eight surgeons was substantial: Fleiss κ=0.69 (95% confidence interval [CI] 0.64-0.74). Surgeons agreed on revised ASRM endometriosis staging criteria after experienced assessment in a majority of cases (mean 61%, range 52-75%) with moderate interrater reliability: Fleiss κ=0.44 (95% CI 0.41-0.47). The intrarater reliability for experienced assessment compared with computer-assisted revised ASRM staging was almost perfect (mean weighted κ=0.95, range 0.89-0.99).

Conclusion

Substantial reliability was found for revised ASRM endometriosis diagnosis, whereas moderate reliability was observed for staging. Almost perfect reliability was observed for surgeons' rating of disease severity compared with computerized-assisted, checklist-based staging. Findings suggest that reliability in endometriosis diagnosis is not greatly altered by location or composition of surgeons, supporting the conduct of multisite studies or compilation of endometriosis data across clinical centers. Although surgeons appear to be skilled at assessing endometriosis stage intuitively, how staging of disease burden correlates with clinical outcomes remains to be developed.

Topics

Research Methods › Measurement and Statistics › Research Tooling and Reproducibility · Endometriosis › Diagnosis › Staging and Classification · Clinical Guidelines and Standards › Diagnostic Criteria and Classification › Staging and Classification Systems

Karen C Schliep, Joseph B Stanford, Zhen Chen, Bo Zhang, Jessie K Dorais, Erica Boiman Johnstone, Ahmad O Hammoud, Michael W Varner, Germaine M Buck Louis, C Matthew Peterson

K Schliep, Joe Stanford, Joey Stanford, J Stanford, Z Chen, B Zhang, J Dorais, E Johnstone, A Hammoud, Mike Varner, M Varner, G Louis

PMID 22914398 22914398 DOI 10.1097/AOG.0b013e31825bc6cf 10.1097/AOG.0b013e31825bc6cf Schliep et al. 2012, Schliep 2012

Interrater and intrarater reliability in the diagnosis and staging of endometriosis

Expert surgeons largely agree on endometriosis in surgical images

Key Findings

Interpretation

RRM Context

Abstract

Topics

By this author

Related research