Impact of different mammography systems on artificial intelligence performance in breast cancer screening

Clarisse F. de                             Vries; Samantha                             J. Colosimo; Roger                         T. Staff; Jaroslaw                             A. Dymiter; Joseph Yearsley; Deirdre Dinneen; Moragh Boyle; David                             J. Harrison; Lesley                             A. Anderson; Gerald Lip; Corri Black; Alison                             D. Murray; Katie Wilde; James                             D. Blackwood; Claire Butterly; John Zurowski; Jon Eilbeck; Colin McSkimming

doi:10.1148/ryai.220146

Impact of different mammography systems on artificial intelligence performance in breast cancer screening

Clarisse F. de Vries^*, Samantha J. Colosimo, Roger T. Staff, Jaroslaw A. Dymiter, Joseph Yearsley, Deirdre Dinneen, Moragh Boyle, David J. Harrison, Lesley A. Anderson, Gerald Lip, Corri Black, Alison D. Murray, Katie Wilde, James D. Blackwood, Claire Butterly, John Zurowski, Jon Eilbeck, Colin McSkimming

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

Abstract

Artificial intelligence (AI) tools may assist breast screening mammography programs, but limited evidence supports their generalizability to new settings. This retrospective study used a 3-year dataset (April 1, 2016-March 31, 2019) from a U.K. regional screening program. The performance of a commercially available breast screening AI algorithm was assessed with a prespecified and site-specific decision threshold to evaluate whether its performance was transferable to a new clinical site. The dataset consisted of women (aged approximately 50-70 years) who attended routine screening, excluding self-referrals, those with complex physical requirements, those who had undergone a previous mastectomy, and those who underwent screening that had technical recalls or did not have the four standard image views. In total, 55916 screening attendees (mean age, 60 years ± 6 [SD]) met the inclusion criteria. The prespecified threshold resulted in high recall rates (48.3%, 21929 of 45444), which reduced to 13.0% (5896 of 45444) following threshold calibration, closer to the observed service level (5.0%, 2774 of 55916). Recall rates also increased approximately threefold following a software upgrade on the mammography equipment, requiring per-software version thresholds. Using software-specific thresholds, the AI algorithm would have recalled 277 of 303 (91.4%) screen-detected cancers and 47 of 138 (34.1%) interval cancers. AI performance and thresholds should be validated for new clinical settings before deployment, while quality assurance systems should monitor AI performance for consistency.

Original language	English
Article number	e220146
Number of pages	8
Journal	Radiology: Artificial Intelligence
Volume	5
Issue number	3
Early online date	22 Mar 2023
DOIs	https://doi.org/10.1148/ryai.220146
Publication status	Published - 1 May 2023

Keywords

Breast
Screening
Mammography
Computer applications: detection/diagnosis
Neoplasms: primary
Technology assessment

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1148/ryai.220146

DeVries_etal_RAI_Impact_Of_Different_AAM
Copyright © 2023 by the Radiological Society of North America, Inc. This work has been made available online in accordance with publisher policies or with permission. This accepted manuscript is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. This is the author created accepted manuscript following peer review and may differ slightly from the final published version. The final published version of this work is available at https://doi.org/10.1148/ryai.220146
Accepted author manuscript, 633 KBLicence: CC BY

Cite this

de Vries, C. F., Colosimo, S. J., Staff, R. T., Dymiter, J. A., Yearsley, J., Dinneen, D., Boyle, M., Harrison, D. J., Anderson, L. A., Lip, G., Black, C., Murray, A. D., Wilde, K., Blackwood, J. D., Butterly, C., Zurowski, J., Eilbeck, J., & McSkimming, C. (2023). Impact of different mammography systems on artificial intelligence performance in breast cancer screening. Radiology: Artificial Intelligence, 5(3), Article e220146. https://doi.org/10.1148/ryai.220146

@article{e130106fece94a1b8256ba83d0bea591,

title = "Impact of different mammography systems on artificial intelligence performance in breast cancer screening",

abstract = "Artificial intelligence (AI) tools may assist breast screening mammography programs, but limited evidence supports their generalizability to new settings. This retrospective study used a 3-year dataset (April 1, 2016-March 31, 2019) from a U.K. regional screening program. The performance of a commercially available breast screening AI algorithm was assessed with a prespecified and site-specific decision threshold to evaluate whether its performance was transferable to a new clinical site. The dataset consisted of women (aged approximately 50-70 years) who attended routine screening, excluding self-referrals, those with complex physical requirements, those who had undergone a previous mastectomy, and those who underwent screening that had technical recalls or did not have the four standard image views. In total, 55916 screening attendees (mean age, 60 years ± 6 [SD]) met the inclusion criteria. The prespecified threshold resulted in high recall rates (48.3%, 21929 of 45444), which reduced to 13.0% (5896 of 45444) following threshold calibration, closer to the observed service level (5.0%, 2774 of 55916). Recall rates also increased approximately threefold following a software upgrade on the mammography equipment, requiring per-software version thresholds. Using software-specific thresholds, the AI algorithm would have recalled 277 of 303 (91.4%) screen-detected cancers and 47 of 138 (34.1%) interval cancers. AI performance and thresholds should be validated for new clinical settings before deployment, while quality assurance systems should monitor AI performance for consistency.",

keywords = "Breast, Screening, Mammography, Computer applications: detection/diagnosis, Neoplasms: primary, Technology assessment",

author = "{de Vries}, {Clarisse F.} and Colosimo, {Samantha J.} and Staff, {Roger T.} and Dymiter, {Jaroslaw A.} and Joseph Yearsley and Deirdre Dinneen and Moragh Boyle and Harrison, {David J.} and Anderson, {Lesley A.} and Gerald Lip and Corri Black and Murray, {Alison D.} and Katie Wilde and Blackwood, {James D.} and Claire Butterly and John Zurowski and Jon Eilbeck and Colin McSkimming",

note = "Funding: Supported by the Industrical Centre of Artificial Intelligence Research in Digital Diagnosis (iCAIRD), which is funded by Innovate UK on behalf of UK Research and Innovation (UKRI) (project no. 104690).",

year = "2023",

month = may,

day = "1",

doi = "10.1148/ryai.220146",

language = "English",

volume = "5",

journal = "Radiology: Artificial Intelligence",

publisher = "Radiological Society of North America Inc.",

number = "3",

}

de Vries, CF, Colosimo, SJ, Staff, RT, Dymiter, JA, Yearsley, J, Dinneen, D, Boyle, M, Harrison, DJ, Anderson, LA, Lip, G, Black, C, Murray, AD, Wilde, K, Blackwood, JD, Butterly, C, Zurowski, J, Eilbeck, J & McSkimming, C 2023, 'Impact of different mammography systems on artificial intelligence performance in breast cancer screening', Radiology: Artificial Intelligence, vol. 5, no. 3, e220146. https://doi.org/10.1148/ryai.220146

Impact of different mammography systems on artificial intelligence performance in breast cancer screening. / de Vries, Clarisse F.; Colosimo, Samantha J.; Staff, Roger T. et al.
In: Radiology: Artificial Intelligence, Vol. 5, No. 3, e220146, 01.05.2023.

Research output: Contribution to journal › Article › peer-review

TY - JOUR

T1 - Impact of different mammography systems on artificial intelligence performance in breast cancer screening

AU - de Vries, Clarisse F.

AU - Colosimo, Samantha J.

AU - Staff, Roger T.

AU - Dymiter, Jaroslaw A.

AU - Yearsley, Joseph

AU - Dinneen, Deirdre

AU - Boyle, Moragh

AU - Harrison, David J.

AU - Anderson, Lesley A.

AU - Lip, Gerald

AU - Black, Corri

AU - Murray, Alison D.

AU - Wilde, Katie

AU - Blackwood, James D.

AU - Butterly, Claire

AU - Zurowski, John

AU - Eilbeck, Jon

AU - McSkimming, Colin

N1 - Funding: Supported by the Industrical Centre of Artificial Intelligence Research in Digital Diagnosis (iCAIRD), which is funded by Innovate UK on behalf of UK Research and Innovation (UKRI) (project no. 104690).

PY - 2023/5/1

Y1 - 2023/5/1

N2 - Artificial intelligence (AI) tools may assist breast screening mammography programs, but limited evidence supports their generalizability to new settings. This retrospective study used a 3-year dataset (April 1, 2016-March 31, 2019) from a U.K. regional screening program. The performance of a commercially available breast screening AI algorithm was assessed with a prespecified and site-specific decision threshold to evaluate whether its performance was transferable to a new clinical site. The dataset consisted of women (aged approximately 50-70 years) who attended routine screening, excluding self-referrals, those with complex physical requirements, those who had undergone a previous mastectomy, and those who underwent screening that had technical recalls or did not have the four standard image views. In total, 55916 screening attendees (mean age, 60 years ± 6 [SD]) met the inclusion criteria. The prespecified threshold resulted in high recall rates (48.3%, 21929 of 45444), which reduced to 13.0% (5896 of 45444) following threshold calibration, closer to the observed service level (5.0%, 2774 of 55916). Recall rates also increased approximately threefold following a software upgrade on the mammography equipment, requiring per-software version thresholds. Using software-specific thresholds, the AI algorithm would have recalled 277 of 303 (91.4%) screen-detected cancers and 47 of 138 (34.1%) interval cancers. AI performance and thresholds should be validated for new clinical settings before deployment, while quality assurance systems should monitor AI performance for consistency.

AB - Artificial intelligence (AI) tools may assist breast screening mammography programs, but limited evidence supports their generalizability to new settings. This retrospective study used a 3-year dataset (April 1, 2016-March 31, 2019) from a U.K. regional screening program. The performance of a commercially available breast screening AI algorithm was assessed with a prespecified and site-specific decision threshold to evaluate whether its performance was transferable to a new clinical site. The dataset consisted of women (aged approximately 50-70 years) who attended routine screening, excluding self-referrals, those with complex physical requirements, those who had undergone a previous mastectomy, and those who underwent screening that had technical recalls or did not have the four standard image views. In total, 55916 screening attendees (mean age, 60 years ± 6 [SD]) met the inclusion criteria. The prespecified threshold resulted in high recall rates (48.3%, 21929 of 45444), which reduced to 13.0% (5896 of 45444) following threshold calibration, closer to the observed service level (5.0%, 2774 of 55916). Recall rates also increased approximately threefold following a software upgrade on the mammography equipment, requiring per-software version thresholds. Using software-specific thresholds, the AI algorithm would have recalled 277 of 303 (91.4%) screen-detected cancers and 47 of 138 (34.1%) interval cancers. AI performance and thresholds should be validated for new clinical settings before deployment, while quality assurance systems should monitor AI performance for consistency.

KW - Breast

KW - Screening

KW - Mammography

KW - Computer applications: detection/diagnosis

KW - Neoplasms: primary

KW - Technology assessment

UR - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10245180/

UR - https://europepmc.org/article/MED/37293340

U2 - 10.1148/ryai.220146

DO - 10.1148/ryai.220146

M3 - Article

VL - 5

JO - Radiology: Artificial Intelligence

JF - Radiology: Artificial Intelligence

IS - 3

M1 - e220146

ER -

Impact of different mammography systems on artificial intelligence performance in breast cancer screening

Abstract

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

ICAIRD: I-CAIRD: Industrial Centre for AI Research in Digital Diagnostics

Cite this

Impact of different mammography systems on artificial intelligence performance in breast cancer screening

Abstract

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Projects

ICAIRD: I-CAIRD: Industrial Centre for AI Research in Digital Diagnostics

Cite this