Mathematics  Mental Health  Science Policy

When One Size Does Not Fit All: A Simple Statistical Method to Deal with AcrossIndividual Variations of Effects
Published:
Monday, June 18, 2012
Author:
Philippe Vindras et al.
by Philippe Vindras, Michel Desmurget, Pierre Baraduc
In science, it is a common experience to discover that although the investigated effect is very clear in some individuals, statistical tests are not significant because the effect is null or even opposite in other individuals. Indeed, ttests, Anovas and linear regressions compare the average effect with respect to its interindividual variability, so that they can fail to evidence a factor that has a high effect in many individuals (with respect to the intraindividual variability). In such paradoxical situations, statistical tools are at odds with the researcherâ€™s aim to uncover any factor that affects individual behavior, and not only those with stereotypical effects. In order to go beyond the reductive and sometimes illusory description of the average behavior, we propose a simple statistical method: applying a KolmogorovSmirnov test to assess whether the distribution of pvalues provided by individual tests is significantly biased towards zero. Using MonteCarlo studies, we assess the power of this twostep procedure with respect to RM Anova and multilevel mixedeffect analyses, and probe its robustness when individual data violate the assumption of normality and homoscedasticity. We find that the method is powerful and robust even with small sample sizes for which multilevel methods reach their limits. In contrast to existing methods for combining pvalues, the KolmogorovSmirnov test has unique resistance to outlier individuals: it cannot yield significance based on a high effect in one or two exceptional individuals, which allows drawing valid population inferences. The simplicity and ease of use of our method facilitates the identification of factors that would otherwise be overlooked because they affect individual behavior in significant but variable ways, and its power and reliability with small sample sizes (<30â€“50 individuals) suggest it as a tool of choice in exploratory studies.
More...


