When performing a meta-analysis, different methods for calculating the summary effect, heterogeneity (between-study) variance and their allied confidence intervals may yield different, or even conflicting, results. The performance of these methods may vary depending on the characteristics of the meta-analysis (e.g., the number and size of the included studies). The Wald-type confidence interval and the DerSimonian and Laird (DL) estimator may be misleading, particularly in small meta-analyses. The Cochrane's Statistical Methods Group recommended the following alternative methods to be implemented in RevMan based on the literature, including statistical simulations and empirical evaluations.
- The restricted maximum likelihood (REML) method to estimate the heterogeneity variance.
- Methods to calculate confidence intervals for the heterogeneity variance using the generalized Q-statistic method.
- The Hartung and Knapp, Sidik and Jonkman (HKSJ) method for calculating confidence intervals for the summary effect.
- Methods to calculate prediction intervals (i.e. using the t-distribution or the normal distribution)
September 2022: Cochrane endorsed these new statistical methods for fitting random effects models following a recommendation from the Methods Executive, which was informed by the relevant Methods Group, Cochrane's Methods Support Unit, the Head of Methods and Evidence Synthesis/Deputy Editor-in-Chief, a Cochrane Review Group representative and the Methods Executive representative on Cochrane's Editorial Board. Implementation developments are underway.