Simulation-based evaluation of anaesthesia residents: optimising resource use in a competency-based assessment framework

Melinda Fleming, Michael McMullen*, Theresa Beesley, Rylan Egan, Sean Field

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


Introduction Simulation training in anaesthesiology bridges the gap between theory and practice by allowing trainees to engage in high-stakes clinical training without jeopardising patient safety. However, implementing simulation-based assessments within an academic programme is highly resource intensive, and the optimal number of scenarios and faculty required for accurate competency-based assessment remains to be determined. Using a generalisability study methodology, we examine the structure of simulation-based assessment in regard to the minimal number of scenarios and faculty assessors required for optimal competency-based assessments.

Methods Seventeen anaesthesiology residents each performed four simulations which were assessed by two expert raters. Generalisability analysis (G-analysis) was used to estimate the extent of variance attributable to (1) the scenarios, (2) the assessors and (3) the participants. The D-coefficient and the G-coefficient were used to determine accuracy targets and to predict the impact of adjusting the number of scenarios or faculty assessors.

Results We showed that multivariate G-analysis can be used to estimate the number of simulations and raters required to optimise assessment. In this study, the optimal balance was obtained when four scenarios were assessed by two simulation experts.

Conclusion Simulation-based assessment is becoming an increasingly important tool for assessing the competency of medical residents in conjunction with other assessment methods. G-analysis can be used to assist in planning for optimal resource use and cost-efficacy.

Original languageEnglish
JournalBMJ Simulation and Technology Enhanced Learning
VolumeOnline First
Early online date3 Dec 2019
Publication statusE-pub ahead of print - 3 Dec 2019


  • Curriculum design
  • Design
  • Simulation-based education


Dive into the research topics of 'Simulation-based evaluation of anaesthesia residents: optimising resource use in a competency-based assessment framework'. Together they form a unique fingerprint.

Cite this