Abstract
A key challenge in experimental social science research is the
incentivisation of subjects such that they take the tasks presented to
them seriously and answer honestly. If subject responses can be
evaluated against an objective baseline, a standard way of incentivising
participants is by rewarding them monetarily as a function of their
performance. However, the subject area of experimental philosophy is
such that this mode of incentivisation is not applicable as participant
responses cannot easily be scored along a true-false spectrum by the
experimenters. We claim that experimental philosophers’ neglect of and
claims of unimportance about incentivisation mechanisms in their surveys
and experiments has plausibly led to poorer data quality and worse
conclusions drawn overall, potentially threatening the research
programme of experimental philosophy in the long run. As a solution to
this, we propose the adoption of the Bayesian Truth Serum, an
incentive-compatible mechanism used in economics and marketing, designed
for eliciting honest responding in subjective data designs by rewarding
participant answers that are surprisingly common. We argue that the
Bayesian Truth Serum (i) adequately addresses the issue of incentive
compatibility in subjective data research designs and (ii) that it
should be applied to the vast majority of research in experimental
philosophy. Further, we (iii) provide an empirical application of the
method, demonstrating its qualified impact on the distribution of
answers on a number of standard experimental philosophy items and
outline guidance for researchers aiming to apply this mechanism in
future research by specifying the additional costs and design steps
involved.
Original language | English |
---|---|
Number of pages | 26 |
Journal | Review of Philosophy and Psychology |
Volume | First Online |
Early online date | 14 Aug 2021 |
DOIs | |
Publication status | E-pub ahead of print - 14 Aug 2021 |