Multiple-comparison procedures for steady-state simulations

Suppose that there are $k \geq 2$ different systems (i.e., stochastic processes), where each system has an unknown steady-state mean performance and unknown asymptotic variance. We allow for the asymptotic variances to be unequal and for the distributions of the k systems to be different. We consider the problem of running independent, single-stage simulations to make multiple comparisons of the steady-state means of the different systems. We derive asymptotically valid (as the run lengths of the simulations of the systems tend to infinity) simultaneous confidence intervals for each of the following problems: all pairwise comparisons of means, all contrasts, multiple comparisons with a control and multiple comparisons with the best. Our confidence intervals are based on standardized time series methods, and we establish the asymptotic validity of each under the sole assumption that the stochastic processes representing the simulation output of the different systems satisfy a functional central limit theorem. Although simulation is the context of this paper, the results naturally apply to (asymptotically) stationary time series.