Using prior information on the intraclass correlation coefficient to analyze data from unreplicated and under-replicated experiments

Abstract

Many studies are performed on units that cannot be replicated due to cost or other restrictions. There often is an abundance of subsampling to estimate the within unit component of variance, but what is needed for statistical tests is an estimate of the between unit component of variance. There is evidence to suggest that the ratio of the between component of variance to the total variance will remain relatively constant over a range of studies of similar types. Moreover, in many cases this intraclass correlation, which is the ratio of the between unit variance to the total variance, will be relatively small, often 0.1 or less. Such situations exist in education, agriculture, and medicine to name a few.

The present study discusses how to use such prior information on the intraclass correlation coefficient (ICC) to obtain inferences about differences among treatments in the face of no replication. Several strategies that use the ICC are recommended for different situations and various designs. Their properties are investigated. Work is extended to under-replicated experiments. The work has a Bayesian flavor but avoids the full Bayesian analysis, which has computational complexities and the potential for lack of acceptance among many applied researchers. This study compares the prior information ICC methods with traditional methods. Situations are suggested in which prior information ICC methods are preferable to traditional methods and those in which the traditional methods are preferable.