Prioritising authenticity in live lecture comprehension pre-sessional assessment task design

Where pre-sessional assessments determine student progression, authenticity is crucial to deliver construct validity and support high-stakes progress decisions. However, replicating the natural processes that occur in academic environments poses challenges relating to domain representation and test-taker engagement as well as meeting stakeholder needs such as benchmarking against commercial tests. The current study reports on a project devised to develop an authentic lecture assessment for a pre-sessional programme, informed by a university-wide needs analysis. In the study, 66 students completed a live lecture comprehension task, and data were analysed using classical test theory and many facet Rasch measurement. In addition, students completed a questionnaire which was analysed using stepwise linear regression. Analysis of the lecture transcript was completed to evaluate the suitability of the content for the target level. Results showed that speech rate and vocabulary were appropriate for the assessment objectives, student self-reported comprehension accounted for some test score variance, and the test was reliable. However, many facet Rasch measurement showed the test was only able to categorise test takers into two distinct levels, which was problematic as the pre-sessional needed to multiple levels for schools to make admissions decisions. Recommendations for balancing these challenges in assessment design are presented.