Following the development of the Risk of Bias 2 (RoB 2) tool, Cochrane will be rolling it out on new reviews in 2020. Here, Ella Flemyng, Methods Implementation Coordinator at Cochrane, highlights plans for the implementation of RoB 2 in Cochrane. 

Posted on 21 June 2019.

Our confidence in the results of research studies is affected by how they are designed, conducted and reported. This impacts on how certain we can be in the evidence we summarise in our reviews when people use it to make decisions about healthcare.  

The first structured tool to assess the Risk of Bias was launched in 2008, with guidance included in the fifth version of the Cochrane Handbook for Systematic Reviews of Interventions and integration in Review Manager 5 software. Since then Cochrane has required all reviews to assess the risk of bias in the studies they include (see the MECIR standards for assessing risk of bias here). 

Why did we need a new version of the Risk of Bias tool?

A number of evaluations of the original Risk of Bias tool have identified challenges in its reliability and consistency (see examples here, here and here). This evidence base has helped an expert group of researchers* to develop a second tool to address these issues.

The Risk of Bias 2 tool (RoB 2) has four main aims:  

1. Improve the accuracy of risk of bias assessments – RoB 2 has more comprehensive guidance and structure to improve consistency and will have specific versions for cluster RCTs and crossover trials.
2. Improve the usability of the tool – RoB 2 has clearer guidance and help built into the tool to help reviewers reach risk of bias judgements.
3. Reflect up-to-date science – RoB 2 has incorporated recent scientific developments, including evaluation of the previous tool and reflects current thinking on missing data and unblinded trials.
4. Improve the usefulness of the tools – RoB 2 includes overall risk of bias judgements that feed into sensitivity analyses and explorations of heterogeneity, and the tool is also allied to ROBINS-I for non-randomized studies.

This post mainly details plans for the pilot and roll-out of RoB 2 in Cochrane so if you are interested in more details on the tool itself, please see this webinar titled ‘RoB 2: A revised tool to assess risk of bias in randomized trials webinar’ from Matthew Page, or check out the information on the RoB 2 website: https://sites.google.com/site/riskofbiastool/welcome/rob-2-0-tool. 

When will we need to start using it in Cochrane Reviews?

Earlier this year, as part of the Strategy to 2020, Cochrane announced a deliverable of ‘RoB 2 to become the default form of risk of bias assessment for new reviews’ from December 2019 (see page 8 of the ‘Strategy to 2020: 2019 Targets’). 

But don’t panic! As we have mapped out what this project will involve, and the dependencies on different systems and processes, we have decided to step back slightly and use a pilot in 2019 to determine the definite timelines in 2020. 

However, for any of you eager to use RoB 2 in your Cochrane Reviews now, you can! RoB 2 has been built into RevMan Web; though it is not yet publicly available we can switch it on for specific reviews. If you are an author who would like to start using it, please start by contacting your Cochrane Review Group (CRG) or if you are part of a CRG with an author team who would like to use RoB 2, please contact us. 

The RoB 2 pilot 

We are currently finalising plans to run a pilot in 2019. This will involve the recruitment of a cohort of volunteer author teams to use RoB 2 in RevMan Web. We are aiming for representation across all the Networks to ensure we have diversity in review questions. As the author teams progress with RoB 2 we will hold regular check-ins with dedicated methods and technology support from the Editorial and Methods Department and Informatics and Technology Services. We welcome CRG Editors and Network Senior and Associate Editors to these discussions too and hope it will provide an opportunity for CRGs, Networks and the Central Executive Team to work through the challenges of adopting the new method.

The pilot will help us with optimising the process and technology. It will also give us some pointers on the challenges, how to manage the outputs from the signalling questions and the implications for RevMan Web. We are also keen to explore what sort of considerations need to inform protocols and to plan updates. Once published, these pilot reviews will also act as examples for others. 

The roll-out in 2020

Following discussions with the Editorial Board and other interested parties at the 2019 mid-year Governance Meetings, we have decided that a collaborative approach is needed. Working with Networks and building on the pilot will be the best way to roll-out RoB 2. This approach gives us the opportunity to learn as we go. 

For reviews in progress and updates, switching from the original risk of bias tool to RoB 2 is not mandatory. Where the protocol is published and for updates being planned after RoB 2 rollout in 2020, the switch to RoB 2 can be made on a case-by-case basis. The earlier you are in the review process, the easier the switch will be. For example, if you are yet to publish a protocol changing to RoB 2 should be more straightforward than if the review has progressed to the analysis stages. If a published review only has a handful of RCTs and you are expecting many more when you update, switching to RoB 2 would probably be best for the quality of the review. However, if the published review includes many RCTs and you are not expecting many new studies the gains will be marginal, and it would be reasonable to use the existing tool. 

Some final thoughts

During the mid-year Cochrane Governance Meetings we held a RoB 2 Strategic Session, which outlined these plans for the pilot and roll-out and called for feedback from the Cochrane community on what tools, guidance, training and support they feel are needed for the successful implementation of RoB 2. A full overview of the session and feedback is included in a post titled ‘The Risk of Bias 2 Strategic Session at the mid-year Cochrane Governance Meeting’.

We plan to keep you up-to-date with periodic posts on RoB 2 implementation so watch this space for more updates to come! 

*Contributors:

Core group: Julian Higgins, Jelena Savović, Matthew Page, Asbjørn Hróbjartsson, Isabelle Boutron, Barney Reeves, Roy Elbers, Jonathan Sterne

Working Group members: Doug Altman, Natalie Blencowe, Mike Campbell, Christopher Cates, Rachel Churchill, Mark Corbett, Nicky Cullum, Francois Curtin, Amy Drahota, Sandra Eldridge, Jonathan Emberson, Bruno Giraudeau, Jeremy Grimshaw, Sharea Ijaz, Miguel Hernán, Sally Hopewell, Asbjørn Hróbjartsson, Peter Jüni, Jamie Kirkham, Toby Lasserson, Tianjing Li, Stephen Senn, Sasha Shepperd, Ian Shrier, Nandi Siegfried, Lesley Stewart, Kate Tilling, Ian White, Penny Whiting

And: Henning Keinke Andersen, Vincent Cheng, Mike Clarke, Jon Deeks, Daniela Junqueira, Alexandra McAleenan, Geraldine Macdonald, Richard Morris, Mona Nasser, Nishith Patel, Jani Ruotsalainen, Holger Schünemann, Jayne Tierney