Clinically focussed evaluation of anomaly detection and localisation methods using inpatient CT head data

Anomaly detection approaches in medical imaging show promise in reducing the need for labelled data. However, the question of how to evaluate anomaly detection algorithms remains challenging, both in terms of the data and the metrics. In this work, we take a cohort of inpatient CT head scans from an elderly stroke patient population containing a variety of anomalies, and treat the associated radiology reports as the reference for clinically relevant findings which should be detected by an anomaly detection algorithm. We apply two state-of-the-art anomaly detection methods to the data, namely denoising autoencoder (DAE) and context-to-local feature matching (CLFM) models. We then extract bounding boxes from the predicted anomaly score heatmaps, which we treat as candidate anomaly detections. A clinical evaluation is then conducted in which 3 radiologists rate the candidate anomalies with respect to their detection and localisation accuracy, by assigning the corresponding report sentence where a clinically relevant anomaly is correctly detected, and rating localisation according to a 3-point scale (good, partial, poor). We find that neither method exhibits sufficiently high recall for clinical use, even at low detection thresholds, although anomaly detection shows promise as a scalable approach for detecting clinically relevant findings. We highlight that selection of the optimal thresholds and extraction of discrete anomaly predictions (e.g. bounding boxes) are underexplored topics in anomaly detection.