Bristol Meeting - Statistical Issues in Network Meta-Analysis

The Comparing Multiple Interventions Methods Group (CMIMG) has received funding from the Cochrane Methods Innovation Fund to address specific issues related to the comparison of multiple interventions in Cochrane reviews. The project involves three meetings to bring together investigators, methodologists, authors, consumers, managing editors and other interested parties to help CMIMG develop consensus guidance for carrying out reviews of multiple intervention. 

For a full list of the products of this project please see Comparing Multiple Interventions in Cochrane Reviews

The second meeting was held by Group 2 in Bristol, UK on July 16 and 17, 2013 and addressed statistical issues in network Meta-analysis

The meeting was informed by a A discussion document that describes all relevant methodologies suggested in the scientific literature to date. 

Slide Sets from presentations at the meeting:

Session 1: Models for Network Meta-Analysis and evaluation of assumptions

Assumptions of Network Meta-Analysis Models

Session 2: Presenting data from the network and results from Network Meta-Analysis

Presenting data and results

Novel presentational approaches for reporting network meta-analysis

Session 3: Selecting the appropriate effect size for Network Meta-Analysis

Overview of empirical evidence

Session 4: Evaluation of inconsistency

Overview of methods to estimate inconsistency - Part 1

Overview of methods to estimate inconsistency - Part 2

                           See also - Donegan, S., Williamson, P., D’Alessandro, U., & Tudur Smith, C. (2013). Assessing key assumptions of network meta-analysis: a review of methods.Research Synthesis Methods, doi:10.1002/jrsm.1085

Graphs for inconsistency

Session 5: What to do with inconsistent networks

Summary of Minutes from the meeting

Click here to view the complete minutes from the meeting

Conventional Cochrane intervention reviews with pairwise meta-analyses were criticized for not addressing sufficiently the assumptions underlying principled analysis of data, e.g. the study inclusion/exclusion criteria are often unclear. Moreover, several published Cochrane reviews include informal indirect comparisons without stating the assumptions being made. Consequently, enhancing Cochrane reviews to include network meta-analysis (NMA) needs very careful consideration with respect to acknowledging and evaluating the underlying assumptions. The authors should built up their network very carefully and consider the implications of the inclusion criteria for studies and treatments. Cochrane protocols and reviews with multiple interventions should include a marked section where they will defend the transitivity assumption that underlies NMA; this can be reflected on the concepts that treatments are jointly randomizable and that missing arms in studies are missing at random.

Guidance to reviewers is needed about how to deal with inconsistency and there is a wealth of methods that can be recommended. Both local tests (that identify whether a part of the network e.g. a specific comparison or ‘evidence loop’ is associated with inconsistency) and global tests (that infer about inconsistency in the entire network) are useful. Guidance needs to highlight that in the case that only indirect evidence is available, the model assumptions can be still evaluated by comparing the distribution of effect modifiers. Cochrane reviewers are familiar with strategies to address and account for heterogeneity; Cochrane guidance for inconsistency should draw in the analogy with heterogeneity.

Network graphs offer a useful options to present the evidence base but they need to be quite flexible in terms of shape and when adding/excluding treatments. Reviewers can use tables to present the summary relative treatment effects estimated in NMA; if few treatments are compared a ‘league table’ with all treatment comparisons can be presented. Otherwise reviewers can select a reference treatment (e.g placebo) and present the treatment effects compared to that reference. Ranking measures such as mean or median ranks and SUCRAs can be presented optionally along with the relative effects providing also confidence intervals for ranking if possible. The order of presenting treatment results could follow their ranking on the most important outcome.

Written guidance in Cochrane Handbook should be short and does not need to provide statistical details. A ‘good practice’ and/or ‘good reporting’ checklist drawing on ongoing works from various teams (e.g the NICE, ISPOR and PRISMA) would be helpful to guide reviewers while they perform and report NMA.