An experiment of the impact of a neonicotinoid pesticide on honeybees: the value of a formal analysis of the data

Background: We assess the analysis of the data resulting from a field experiment conducted by Pilling et al. (2013) on the potential effects of thiamethoxam on honey bees. The experiment had low levels of replication, so Pilling et al. concluded that formal statistical analysis would be misleading. This would be true if such an analysis merely comprised tests of statistical significance and if the investigators concluded that lack of significance meant little or no effect. However, an analysis that includes estimation of the size of any effects—with confidence limits—allows one to reach conclusions that are not misleading and that produce useful insights.

Main Body: For the data of Pilling et al. we use straightforward statistical analysis to show that the confidence limits are generally so wide that any effects of thiamethoxam could have been large without being statistically significant. Instead of formal analysis, Pilling et al. simply inspected the data and concluded that they provided no evidence of detrimental effects and from this that thiamethoxam poses a “low risk” to bees.

Conclusions: Conclusions derived from inspection of the data were not just misleading in this case but are unacceptable in principle, for if data are inadequate for a formal analysis (or only good enough to provide estimates with wide confidence intervals) then they are bound to be inadequate as a basis for reaching any sound conclusions. Given that the data in this case are largely uninformative with respect to the treatment effect, any conclusions reached from such informal approaches can do little more than reflect the prior beliefs of those involved.