Limitations of the questionnaire method: a reply to Hodroj, Latham, and Miller, ‘The moving open future, temporal phenomenology, and temporal passage’

Hodroj, Latham, and Miller’s target article is the latest in a slew of recent studies that seek to investigate temporal experience using a method that involves participants reading vignettes then completing questionnaires. I have significant misgivings about this method, and in this commentary, I discuss its limitations. I start by discussing the kinds of proposals that can, and cannot, be tested in principle using the questionnaire method, suggesting that some prominent proposals fall outside its scope. I illustrate this through an analogy with the opponent process theory of colour experience, where I suggest that the tools of cognitive neuroscience are needed. I then draw attention to some of the pitfalls of the questionnaire method in general, and to some problems specific to the current study. One major issue concerns the question of whether it has been shown that participants really understood their task. I give reasons for doubting this. First, I show that it is too easy to answer the comprehension questions correctly without understanding. Second, I argue that it is not credible that participants actually held the world view entailed by the combination of responses given by a significant minority of them, suggesting that there was significant confusion. I raise several other objections before concluding with a modest suggestion for future work.