A simulation comparison of two methods for controlling the experiment-wise Type I error rate of correlated tests for contrasts in one-way completely randomized designs

Abstract

A Bonferroni and an ordered P-value solution to the problem of controlling the experiment-wise Type I error rate are studied and compared in terms of actual size and power when carrying out correlated tests. Although both of these solutions can be used in a wide variety of settings, here they are only investigated in the context of multiple testing that specified pairwise comparisons of means, selected before data are collected, are all equal to zero in a completely randomized, balanced, one factor design where the data are independent random samples from normal distributions all having the same variance. Simulations indicate that both methods are very similar and effective in controlling experiment wise type error at a nominal rate of 0.05. Because the ordered P-value method has, almost uniformly, slightly greater power, it is my recommendation for use in the setting of this report.