Research Methods · Measurement and Statistics

Endometriosis diagnosis and staging by operating surgeon and expert review using multiple diagnostic tools: an inter-rater agreement study

Schliep KC, Chen Z, Stanford JB, Xie Y, Mumford SL, Hammoud AO, Boiman Johnstone E, Dorais JK, Varner MW, Buck Louis GM, Peterson CM

Published October 5, 2015 BJOG : an international journal of obstetrics and gynaecology, 124(2), 220-229, 2017
DOI 10.1111/1471-0528.13711 PMID 26435386 PMC PMC4821828
Read our synopsis

RRM Academy Synopsis

Reviewers agreed with surgeons on endometriosis from images, reports

Four expert reviewers matched the surgeon's endometriosis diagnosis in 93% of 148 women once they read the operative report. The women came from five Utah surgical centers. Staging agreement was lower than diagnosis agreement. In the 67 women with histology, agreement dropped after reviewers saw it.

Key Findings

  • Reviewers agreed with surgeons on diagnosis in 83% of women after viewing digital images alone, and in 93% after also reading operative reports.
  • Agreement statistics for diagnosis (κ=0.67) and algorithm staging (κ=0.64) rose after operative reports (κ=0.88 and κ=0.85).
  • Staging agreement was lower than diagnosis agreement: 49% by empiric assessment and 57% by algorithm after images alone, rising to 60% and 72% with operative reports.
  • MRI matched the surgeon's diagnosis in 24 of 36 women. MRI sensitivity was 55.6% (95% CI 35.3%, 74.5%) and specificity 100% (95% CI 66.2%, 100.0%).
  • After reviewers saw operative and histopathologic reports (histology round, 67 women), between 7 and 10 women moved from a positive to a negative diagnosis.

Interpretation

The study measures how often reviewers and surgeons gave the same answer. Agreement is a measure of consistency between raters. The 148 women came from one state and were mostly non-Hispanic white, and four reviewers took part. Only 36 women had MRI and 67 had histology, so the authors say those estimates may be imprecise. Surgeons did not see histology before surgery. The paper leaves open whether negative histology reflects less disease or a missed sample.

RRM Context

Restorative reproductive medicine starts by finding the cause, so the diagnosis matters first. The authors cite guidance that positive histology confirms endometriosis. Negative histology does not exclude it. They also call for better staging systems that track pain and fertility outcomes.

Abstract

Objective

To determine agreement on endometriosis diagnosis between real-time laparoscopy and subsequent expert review of digital images, operative reports, magnetic resonance imaging (MRI), and histopathology, viewed sequentially.

Design

Inter-rater agreement study.

Setting

Five urban surgical centres.

Population

Women, aged 18-44 years, who underwent a laparoscopy regardless of clinical indication. A random sample of 105 women with and 43 women without a postoperative endometriosis diagnosis was obtained from the ENDO study.

Methods

Laparoscopies were diagnosed, digitally recorded, and reassessed.

Main Outcome Measures

Inter-observer agreement of endometriosis diagnosis and staging according to the revised American Society for Reproductive Medicine criteria. Prevalence and bias-adjusted kappa values (κ) were calculated for diagnosis, and weighted κ values were calculated for staging.

Results

Surgeons and expert reviewers had substantial agreement on diagnosis and staging after viewing digital images (n = 148; mean κ = 0.67, range 0.61-0.69; mean κ = 0.64, range 0.53-0.78, respectively) and after additionally viewing operative reports (n = 148; mean κ = 0.88, range 0.85-0.89; mean κ = 0.85, range 0.84-0.86, respectively). Although additionally viewing MRI findings (n = 36) did not greatly impact agreement, agreement substantially decreased after viewing histological findings (n = 67), with expert reviewers changing their assessment from a positive to a negative diagnosis in up to 20% of cases.

Conclusion

Although these findings suggest that misclassification bias in the diagnosis or staging of endometriosis via visualised disease is minimal, they should alert gynaecologists who review operative images in order to make decisions on endometriosis treatment that operative reports/drawings and histopathology, but not necessarily MRI, will improve their ability to make sound judgments.

Tweetable Abstract

Endometriosis diagnosis and staging agreement between expert reviewers and operating surgeons was substantial.

Topics

By this author

Related research

Research Methods › Measurement and Statistics › Statistical Methods · Endometriosis › Diagnosis › Staging and Classification
Karen C Schliep, Zhian Chen, Germaine M Buck Louis, Sunni L Mumford, C Matthew Peterson, Joseph B Stanford, Michael W Varner
K Schliep, Z Chen, G Louis, S Mumford, Joe Stanford, Joey Stanford, J Stanford, Mike Varner, M Varner
PMID 26435386 26435386 DOI 10.1111/1471-0528.13711 10.1111/1471-0528.13711 Schliep et al. 2017, Schliep 2017

Cite this article

Schliep, K. C., Chen, Z., Stanford, J. B., Xie, Y., Mumford, S. L., Hammoud, A. O., Boiman Johnstone, E., Dorais, J. K., Varner, M. W., Buck Louis, G. M., & Peterson, C. M. (2017). Endometriosis diagnosis and staging by operating surgeon and expert review using multiple diagnostic tools: an inter-rater agreement study. BJOG : an international journal of obstetrics and gynaecology, 124(2), 220-229. https://doi.org/10.1111/1471-0528.13711