A Step-Up Multiple Test Procedure

Abstract We consider the problem of simultaneously testing k ≥ 2 hypotheses on parameters θ1, …, θk . In a typical application the θ's may be a set of contrasts, for instance, a set of orthogonal contrasts among population means or a set of differences between k treatment means and a standard treatment mean. It is assumed that least squares estimators 1, …, k are available that are jointly normally distributed with a common variance (known up to a scalar, namely the error variance σ 2) and a common known correlation. An independent χ 2-distributed unbiased estimator of σ 2 is also assumed to be available. We propose a step-up multiple test procedure for this problem which tests the t statistics for the k hypotheses in order starting with the least significant one and continues as long as an acceptance occurs. (By contrast, the step-down approach, which is usually used, starts with the most significant and continues as long as a rejection occurs.) Critical constants required by this step-up procedure to control the type I familywise error rate at or below a specified level α are computed for both one-sided and two-sided testing problems. Power comparisons are made for one-sided testing problems with the well-known normal theory based single-step and step-down test procedures, and also with a step-up test procedure proposed for a wider class of problems by Hochberg. (Two improvements over Hochberg's procedure by Hommel and Rom provide at best marginal increases in power, with the former being also more difficult to apply, and hence they are not included here.) Two different definitions of power are considered—the probability of rejecting all false hypotheses and the probability of rejecting at least one false hypothesis; the results are found to be qualitatively similar. The proposed step-up procedure is more powerful than the single-step procedure except when only one hypothesis is false, in which case it is slightly less powerful. Similarly, it is slightly less powerful than the step-down procedure when a few hypotheses are false, but it is more powerful when most or all hypotheses are false, and this power advantage increases with the number of such hypotheses under test. The proposed step-up procedure is uniformly more powerful than the Hochberg procedure and its improvements. A disadvantage of the proposed step-up procedure is the greater difficulty of computing its critical points. These are given for one-sided and two-sided tests for 5% level of significance.