A method for identifying spatially divergent selection in structured populations

Species occupy diverse, heterogeneous environments, which expose populations to spatially varied selective pressures. Populations in different environments can diverge due to local adaptation. However, neutral evolution can also drive population divergence. Thus, testing for local adaptation requires a neutral baseline for population differentiation. The classical Q_ST -F_ST comparison was developed for this purpose. Yet, Q_ST -F_ST frequently fails to account for the complexities of population structure because the theory underlying this comparison assumes that all subpopulations are equally related, resulting in inflated false positive rates in metapopulations that deviate from the island model. To address this limitation we use estimates of between- and within-population relatedness to model population structure. Using those relatedness matrices, we infer the between- and within-population ancestral additive genetic variances under a mixed-effects model. Under neutrality, these inferred variances are expected to be equal. We propose here a test to detect selection based on the comparison of these two estimates of the ancestral variance and we compare its performance with earlier solutions. We find our method is well calibrated across various population structures and has high power to detect adaptive divergence.