On the variety of methods for calculating confidence intervals by bootstrapping

Researchers often want to place a confidence interval around estimated parameter values calculated from a sample. This is commonly implemented by bootstrapping. There are several different frequently used bootstrapping methods for this purpose. Here we demonstrate that authors of recent papers frequently do not specify the method they have used and that different methods can produce markedly different confidence intervals for the same sample and parameter estimate. We encourage authors to be more explicit about the method they use (and number of bootstrap resamples used). We recommend the bias corrected and accelerated method as giving generally good performance; although researchers should be warned that coverage of bootstrap confidence intervals is characteristically less than the specified nominal level, and confidence interval evaluation by any method can be unreliable for small samples in some situations.