Inference for the intrinsic separation among distributions which may differ in location and scale

Abstract

The null hypothesis of equal distributions, H0 : F1[equals]F2[equals]...[equals]FK , is commonly used to compare two or more treatments based on data consisting of independent random samples. Using this approach, evidence of a difference among the treatments may be reported even though from a practical standpoint their effects are indistinguishable, a longstanding problem in hypothesis testing. The concept of effect size is widely used in the social sciences to deal with this issue by computing a unit-free estimate of the magnitude of the departure from H0 in terms of a change in location. I extend this approach by replacing H0 with hypotheses H0* that state that the distributions {Fi} are possibly different in location and or scale, but close, so that rejection provides evidence that at least one treatment has an important practical effect. Assessing statistical significance under H0* is difficult and typically requires inference in the presence of nuisance parameters. I will use frequentist, Bayesian and Fiducial modes of inference to obtain approximate tests and carry out simulation studies of their behavior in terms of size and power. In some cases a bootstrap will be employed. I will focus on tests based on independent random samples arising from K[greater than and equals]3 normal distributions not required to have the same variances to generalize the K[equals]2 sample parameter P(X1>X2) and non-centrality type parameters that arise in testing for the equality of means.