What we’ve learnt from the Risk of Bias 2 Pilot in Cochrane

Posted on 19 November 2020 by Ella Flemyng (Methods Implementation Manager)

A year ago, at the 2019 Cochrane Governance Meetings, we hosted a Strategic Session on the implementation of the Risk of Bias 2 (RoB 2) tool to gain feedback from the community. In this post, we highlight how that feedback has been integrated into the plans for RoB 2 and what we have learnt during the initial pilot phase. You can find what you need to know about RoB 2 in Cochrane in an accompanying post and from a position statement from the Cochrane Editorial Board.  

Cochrane Reviews are required to assess the risk of bias in the studies that they identify and summarise. A new version of the tool to assess risk of bias in randomised trials, Risk of Bias 2 (RoB 2), was included in version 6 of the Cochrane Handbook of Systematic Reviews of Interventions, which published in 2019 (details on why a new version was needed were covered in a previous post).

In 2019, the Cochrane Central Executive Team, with the support of the Governing Board, decided that the RoB 2 roll-out process should begin with a Pilot with authors and Cochrane Review Groups (CRGs) able to opt-in to use the tool.  The aim was for a supported and gradual roll-out of RoB 2 in Cochrane to observe common issues and develop training and support before scaling up to wider implementation. We have used every opportunity available to obtain feedback from the Cochrane community on the piloting process, including, the 2019 Governance Meetings, the 2019 UK and Ireland Cochrane Symposium and CRG Network teleconferences.  

The pilot so far

As of 19 November 2020, there are over 80 Cochrane Reviews across all eight of the CRG Networks in the RoB 2 Pilot. The Pilot has been led by Ella Flemyng (Methods Implementation Manager) and Kerry Dwan (Methods Support Unit Lead and Statistical Editor), working closely with Tess Moore (Systematic Review Methodology Editor in the Methods Support Unit). 

Support 

Since the launch of the Methods Support Unit in 2019, they have been working closely with the Network Associate Editors and CRGs participating in the Pilot to transfer the knowledge and editorial tools that have been developed to build the capacity needed as uptake of the tool increases. CRGs have received hands-on support and training for protocols and reviews using RoB 2. The Pilot team hosts monthly Web Clinics, where authors or CRG staff can submit questions on RoB 2 or RevMan Web for discussion: 

“Just listened back to this [RoB 2 web clinic recording] and found it incredibly helpful. The notes were essential. Thank you so much. I’m sure this will be a helpful resource to many others.”
Roses Parker, Network Support Fellow, Cochrane Musculoskeletal, Oral, Skin and Sensory Network

The monthly Web Clinics have also been a useful platform to ask questions directly to those using RoB 2 to help understand gaps in guidance and support in real-time so that we can ensure they are addressed.

Training

In 2019, we organised a three-day, in-person course on risk of bias in Bristol (UK) for CRG editorial staff, which had 50 attendees from across 33 CRGs, Network editorial teams and Cochrane Trainers. Other training updates include ensuring the Cochrane Interactive Learning Module on risk of bias and Standard Author Training Materials are RoB 2 compliant.

A nine-part Learning Live webinar series on RoB 2, which started in May 2020 is underway, which will include a mix of training on the method and on using the tool, finishing with editorial considerations for RoB 2 specifically dedicated for CRG editors and editorial staff.

Guidance

As mentioned, the new Cochrane Handbook for Systematic Reviews of Interventions includes Chapter 8 dedicated to ‘Risk of bias in randomised trials’, and full updates have been made to the MECIR to reflect all changes (see a list of changes made in October 2019 and March 2020 here). These updates are also included in the RevMan support (both 5 and Web). Full guidance on RoB 2 is also available via the Risk of Bias tools website and we have created an introductory leaflet with pre-recorded videos, a Starter Pack and FAQs, all available via Cochrane Methods.

The RoB 2 Starter Pack is a key resource for authors and editors, which includes guidance, support, training and tools, as well as checklists for reporting protocols and reviews.  

Tools

The developers of RoB 2 (team led by Julian Higgins and Jonathan Sterne from the University of Bristol) have created a Word template and Excel tool for completing the risk of bias assessment (see here – the Excel tool is Cochrane's recommended version). The Bristol team are also building a web-based online version of the RoB 2 tool, which is at the testing stage and Covidence have initiated developments to support RoB 2 too. We’re also working with the RevMan Web team to streamline the work flow between the systems.

What’s next?

The Pilot has helped us understand and overcome dependencies, gather evidence on the usability of the tool and understand the impact it has on technology, publishing and presentation. However, there is currently no date for when RoB2 will be mandatory for new intervention reviews. The implementation of RoB 2 is tied to the wider development and uptake of RevMan Web because functionality will not be implemented in RevMan 5. It is therefore not feasible to make RoB 2 mandatory for all new intervention reviews including RCTs until planned RevMan Web developments are in place and tested, and CRGs are using it as standard. Furthermore, projects that were initiated to complement the Pilot by quantifying the usability and value of RoB 2 compared with RoB 2 in Cochrane Reviews have been delayed by the pandemic. [read more about this in the position statement from the Editorial Board on the status and expectations of implementation of Risk of Bias 2 in Cochrane intervention reviews]

With the publication of the first review from the Pilot last week, which used the RoB 2 functionality in RevMan Web and includes the interactive, results-level tables, we’ve collated some additional resources for you:

If anyone has any questions about RoB 2 or the phased implementation, please contact methods@cochrane.org. 

19 Nov 2020