Research Methods Measurement and Statistics

Endometriosis diagnosis and staging by operating surgeon and expert review using multiple diagnostic tools: an inter-rater agreement study

Schliep KC, Chen Z, Stanford JB, Xie Y, Mumford SL, Hammoud AO, Boiman Johnstone E, Dorais JK, Varner MW, Buck Louis GM, Peterson CM

Published January 2017 BJOG : an international journal of obstetrics and gynaecology, 124(2), 220-229
DOI 10.1111/1471-0528.13711 PMID 26435386 PMC PMC4821828

RRM Academy Synopsis

Reviewers agreed with surgeons on endometriosis from images, reports

Four expert reviewers matched the surgeon's endometriosis diagnosis in 93% of 148 women once they read the operative report. The women came from five Utah surgical centers. Staging agreement was lower than diagnosis agreement. In the 67 women with histology, agreement dropped after reviewers saw it.

Key Findings

  • Reviewers agreed with surgeons on diagnosis in 83% of women after viewing digital images alone, and in 93% after also reading operative reports.
  • Agreement statistics for diagnosis (κ=0.67) and algorithm staging (κ=0.64) rose after operative reports (κ=0.88 and κ=0.85).
  • Staging agreement was lower than diagnosis agreement: 49% by empiric assessment and 57% by algorithm after images alone, rising to 60% and 72% with operative reports.
  • MRI matched the surgeon's diagnosis in 24 of 36 women. MRI sensitivity was 55.6% (95% CI 35.3%, 74.5%) and specificity 100% (95% CI 66.2%, 100.0%).
  • After reviewers saw operative and histopathologic reports (histology round, 67 women), between 7 and 10 women moved from a positive to a negative diagnosis.

Interpretation

This inter-rater agreement study compared each surgeon's live call against expert reviewers who re-checked the same cases. Reviewers assessed digital images first. Then they added the operative report, then MRI, then histology. Agreement means how often two raters give the same answer. Agreement rose from a solid match to a near-perfect one once the operative report was added. The authors note that surgeons did not have histology results until after surgery. The findings support larger studies that test more tools for confirming an endometriosis diagnosis.

RRM Context

Restorative reproductive medicine starts with finding the cause. An accurate diagnosis comes first. The authors cite guidance that positive histology confirms endometriosis. Negative histology does not rule it out. They also call for better staging systems that track pain and fertility outcomes.

Abstract

Objective

To determine agreement on endometriosis diagnosis between real-time laparoscopy and subsequent expert review of digital images, operative reports, magnetic resonance imaging (MRI), and histopathology, viewed sequentially.

Design

Inter-rater agreement study.

Setting

Five urban surgical centres.

Population

Women, aged 18-44 years, who underwent a laparoscopy regardless of clinical indication. A random sample of 105 women with and 43 women without a postoperative endometriosis diagnosis was obtained from the ENDO study.

Methods

Laparoscopies were diagnosed, digitally recorded, and reassessed.

Main Outcome Measures

Inter-observer agreement of endometriosis diagnosis and staging according to the revised American Society for Reproductive Medicine criteria. Prevalence and bias-adjusted kappa values (κ) were calculated for diagnosis, and weighted κ values were calculated for staging.

Results

Surgeons and expert reviewers had substantial agreement on diagnosis and staging after viewing digital images (n = 148; mean κ = 0.67, range 0.61-0.69; mean κ = 0.64, range 0.53-0.78, respectively) and after additionally viewing operative reports (n = 148; mean κ = 0.88, range 0.85-0.89; mean κ = 0.85, range 0.84-0.86, respectively). Although additionally viewing MRI findings (n = 36) did not greatly impact agreement, agreement substantially decreased after viewing histological findings (n = 67), with expert reviewers changing their assessment from a positive to a negative diagnosis in up to 20% of cases.

Conclusion

Although these findings suggest that misclassification bias in the diagnosis or staging of endometriosis via visualised disease is minimal, they should alert gynaecologists who review operative images in order to make decisions on endometriosis treatment that operative reports/drawings and histopathology, but not necessarily MRI, will improve their ability to make sound judgments.

Tweetable Abstract

Endometriosis diagnosis and staging agreement between expert reviewers and operating surgeons was substantial.

Read the guide

Topics

By this author

Related research

Research Methods › Measurement and Statistics › Statistical Methods · Endometriosis › Diagnosis › Staging and Classification
Karen C Schliep, Zhian Chen, Germaine M Buck Louis, Sunni L Mumford, C Matthew Peterson, Joseph B Stanford, Michael W Varner
K Schliep, Z Chen, G Louis, S Mumford, Joe Stanford, Joey Stanford, J Stanford, Mike Varner, M Varner
PMID 26435386 26435386 DOI 10.1111/1471-0528.13711 10.1111/1471-0528.13711 Schliep et al. 2017, Schliep 2017