Risk of Bias 2 FAQs OLD

Last updated 22 February 2022

About Risk of Bias 2 (RoB 2)

RoB 2 and editing Cochrane Reviews

Using RoB 2 in Cochrane Reviews 

Updating Cochrane Reviews 

RoB 2 and variants of RCTs

RoB 2 RevMan Web and RevMan 5

Reporting RoB 2

RoB 2 and other review development software

 

About Risk of Bias 2 (RoB 2)

What is RoB 2? Why should we use it? What was wrong with the old way of assessing risk of bias? 

A: RoB 2 is an updated version of the Cochrane risk of bias tool and has been created by the same team. The main differences are: each domain has a set of questions to help you decide if there is bias or not; you assess bias for specific results within an RCT, rather than for the RCT as a whole; algorithms embedded in the tool  propose judgements; there is an overall bias decided across all the domains for each result. A short webinar describes the new tool and explains why the team decided to update it and what the advantages are. Full details about the tool can be found via the riskofbias.info website. 

I am new to using RoB 2 is there anything I need to think about for my protocol? 

A: RoB 2 is very different to other tools used for assessing bias in RCTs. We have a set of protocol considerations listed in the RoB 2 Starter Pack. These will help you think through what you need to know. Also, in the Starter Pack are links to useful webinars including an introduction to RoB 2.  

RoB 2 is “results-based” what does this mean? 

A: This means that authors assess the bias, in an RCT, for a specific result rather than assessing the RCT as whole. This is important because one result, for example mortality at end of treatment, may be assessed to be at low risk of bias. But mortality assessed at 2-year follow-up might be assessed to be at high risk of bias, this could happen for example if the trialists have lost a lot of people from the study through attrition.  Authors using RoB 2 need to specify the outcomes they plan to assess for bias. And then assess bias using RoB 2 for each result, in that study, that contributes to those outcomes. 

Why does RoB 2 not include affiliation bias assessments?

A: RoB 2 focusses on mechanisms through which bias may arise (e.g. randomisation processes; influences on outcome assessments due to knowledge of intervention received; processes of selecting which outcome data to report).  Affiliation bias may lead to bias in trial results through one or more of the mechanisms covered by RoB 2 but also leads to problems such as  inappropriate choice of comparators, or suppression of results, neither of which is covered by RoB 2. A new tool titled Tool for Addressing Conflicts of Interest in Trials (TACIT) is currently under development that will assess affiliation bias. More information on this will be available in due course see the TACIT website.

I heard the RoB2 is a “results-based” tool or results-level tool but have seen it is sometimes called “Outcome-based” which is it and what do these mean? 

A: RoB 2 is called “results-based” or “results-level” because it is used to assess bias for a specific result reported in the RCT report whereas the first risk of bias tool, and most other tools assess bias across all outcomes and results for an entire study. Assessing bias for a specific result means that it is more precise. For example for the domain “Missing outcome data” one result in a study may have minimal missing data at “End of treatment” and could be judged at “Low” risk of bias and the same outcome measured 6 months later might have considerable missing data and be judged at “Some concerns” or “High” risk of bias. By assessing bias for each of these results separately we can present a more accurate assessment of bias.  In some cases, the phrase “outcome-based” or “outcome-level” is used. They are used when the bias assessments of several results, for the same outcome are drawn together. For example, in the outcome-level risk of bias tables that feature in RevMan Web.

Editing Cochrane Reviews that use RoB2

I am a Managing Editor – do all new intervention reviews in my Cochrane Review Group need to use RoB 2? 

A: No. RoB 2 will result in better quality reviews so we do encourage its use, but it is not yet mandated across all new Cochrane intervention reviews. As Managing Editor, you can decide whether or not to approve RoB 2 methods for reviews in your Cochrane Review Group.

I am a Managing Editor and want to check if a review has used the RoB 2 tool in the right way – how can I tell? 

A:The Starter Pack has a set of protocol considerations that authors will find helpful to ensure their review is set up to use RoB 2 correctly.  In addition, it also has recommendations for reporting RoB 2 in the completed review. These are closely aligned with MECIR and The Cochrane Handbook. We are asking editors to check that all Cochrane Reviews and Protocols are following these recommendations.   More information can be found in the Starter Pack. If you have any other concerns or queries you can contact the Methods Support Unit (MSU) – through your CRG .

I am a Managing Editor – How do I “Switch on” RoB 2 in a review in RevMan Web? 

A: You can enable advanced features to switch on RoB2. Please see here for more information on when and how to do this.

 

Using RoB 2 in Cochrane Reviews

What outcomes should I choose for RoB 2 assessments?

A: It is advised that at least the outcomes listed in your Summary of Findings (SoF) table should have RoB 2 assessments as this will be used to inform GRADE. 

Do I need to complete a risk of bias assessment for all of the outcomes from all of the studies in my review? 

A: If there is no numerical result for an outcome from a specific study, then you do not need to complete a risk of bias assessment as it will not be contributing to the review.

Are we supposed to produce reviews that focus only on a few primary outcomes? 

A: The numbers of outcomes a review team selects to assess for bias should be based on their review question(s), and data needed for decision making, clinical need and research.  The familiarity review teams will develop for the RCTs in their review will mean answering signalling questions for other results becomes easier.  Since risk of bias can be different for different results from the same RCT, we need to be able to produce different assessments of bias for different results. 

Ideally, when should the risk of bias assessments be completed in the review process – before or after data analysis?

A: Risk of bias can (and should) be assessed before the synthesis itself is conducted. 

Is there any way to help organise our assessments using RoB 2? Where should I store all the answers to the signalling questions? 

A: There are different means to store your data.  The Risk of Bias development team have developed both Word and Excel documents to assist review teams. For Cochrane Reviews, we are encouraging review teams to use the RoB 2 Excel tool to store their data. Link to ROB.info 

The instruction to "Specify the numerical result" on the first page of the RoB 2 excel sheet form. Does this relate to a specific time point? 

A: Yes. The specific result is the data reported by the study for the outcome, timepoint and measurement method that you are assessing. The exception is if the result could relate to several timepoints, for example if it is estimated from a regression analysis across multiple time points (based on the trajectory of each individual over time) in which case, the result will not relate to a specific time point. 

What can authors do in the early stages to prepare for data extraction and risk of bias screening? 

A: Author teams could run a calibration exercise for all authors in a review (both methodological and clinical). This would involve completing risk of bias assessment for three or so RCTs with group discussions about how best to answer the signalling questions. For the calibration exercise the team should have all documentation (e.g. trial reports, protocols, and trial registry reports, etc) available at the start of the RoB 2 assessments, together with a copy of the BMJ paper and the detailed guidance. 

If the default is a “high” RoB rating overall if one of the five domains is assessed as high-risk rating, why continue to do a full assessment of the remaining domains? 

A: Having all domains completed is an important factor in the move towards transparency in review production. In the same way we would ask for transparency in RCT reporting following CONSORT guidelines, so it is important to be transparent when producing reviews.  In addition, and just as important, as a review author, it is important to know the detail of each study in the review. So that is one reason to complete all domains for all outcomes being assessed. 

Examination of bias across the outcomes in a review can reveal if trialists are making decisions that are likely to cause outcomes to be judged at “High” risk of bias or, what aspects of trial design and analysis could reduce bias for an outcome. This is useful and important for both the “implications for research” section of your review and for transparency in review production.   It would be important to have data from all domains to see the pattern of bias emerges across outcomes.  

What should I do if my review does not have a meta-analysis?

A: RoB 2 aims to assess risk of bias in a quantitative result that feeds into a synthesis, whether the synthesis uses meta-analysis or another synthesis without meta-analysis (SWiM) or ‘narrative’ approach. Authors should include their bias assessments for outcomes that are summarised using SWiM or ‘narrative’ methods and should include them in their GRADE assessments for the Summary of Findings tables too.  

Should only studies with outcomes with ‘low’ risk of bias be included in the meta-analysis? 

A: Restricting to ‘Low’ is one option and obviously a sensible one only if there are sufficient studies at low risk of bias. Authors can choose to run their primary analysis on studies at ‘Low’ or those with ‘Low’ or with ‘Some concerns’ about risk of bias. It is also be appropriate to pool data from studies at all levels of bias including high risk of bias and use a sensitivity analysis to assess the effects of restricting the analysis to RCTs ‘Low’ or with ‘Some concerns’. Ideally decisions should be made at the protocol stage and pre-specified.

Is it OK to stratify analyses by domain? Or do we have to use overall RoB? 

A: RoB 2 was designed so the overall risk of bias represents the risk of bias for that result. If authors add additional subgroup analyses and sensitivity analyses, it will increase risk of false results by chance. Therefore, we strongly encourage all authors to use overall risk of bias for any subgroup or sensitivity analyses.  Any decision to use specific domains must be justified.

I have already registered my Cochrane Review title and was planning to use the original risk of bias tool. However, after reading about RoB 2, can I switch to that instead?

A: If you want to switch from the original risk of bias tool to RoB 2, the earlier you are in the review process, the easier the switch will be. Once a protocol has been published that includes the original risk of bias tool, authors should not switch RoB 2 without getting approval from their Managing Editor. We strongly recommend that you do not switch tool after protocol publication as there are key considerations for RoB 2 that must be prespecified and the authors will be at high risk of using RoB 2 incorrectly.  However, if you do want to switch you can, you just need to add a difference between the protocol and the review. 

Updating Cochrane Reviews

When updating a review, should we move from the original risk of bias tool to RoB 2?

A: It will never be mandatory to switch to RoB 2 during an update if the previous version of the review used the original risk of bias tool. However, authors and CRGs can make decisions on whether to switch to RoB 2 on a case-by-case basis. If a published review only has a handful of RCTs and you are expecting many more when you update, switching to RoB 2 would probably be best for the quality of the review. However, if the published review includes a large number of RCTs and you are not expecting many new studies the gains will be marginal, and it would be reasonable to continue using the original tool. If you do switch to RoB 2, the relevant methods will need updating before you begin your update to cover all of the protocol considerations detailed in the Starter Pack and this can be included in the version history. If authors do not pre-specify these RoB 2 considerations, they will be at high risk of using RoB 2 incorrectly. More information can be found in the Starter Pack. 

I am updating my review; can I use the original risk of bias tool for the old trials and RoB 2 for the new trials?

A: No. If you to choose to use RoB 2 all results will need to be assessed in the same way. If you are updating your review, there will be a choice to keep the original risk of bias tool or switch to the new tool. The decision to switch can be made on the basis of the number of studies in the review before it is updated, and likely number after the update. If a published review only has a handful of RCTs and you are expecting many more when you update, switching to RoB 2 would probably be best for the quality of the review. However, if the published review includes a large number of RCTs and you are not expecting many new studies the gains will be marginal, and it would be reasonable to continue using the original tool. If you do switch to RoB 2, the relevant methods will need updating before you begin your update The reason for this is you will need to cover all of the protocol considerations detailed in the Starter Pack. Changes to the original protocol can be included in the section “Differences between the protocol and the review” . If authors do not pre-specify these RoB 2 considerations, they may apply RoB 2 incorrectly. More information can be found in the Starter Pack. 

RoB 2 and variants of RCTS

For quasi randomised trials, will it be appropriate to use RoB 2 for RCTs? 

A: “Quasi-randomized trials” (or “pseudo-randomized trials”) lie on a spectrum from studies that look very similar to randomized trials (e.g. using alternation, perhaps even with the sequence concealed at the point of allocation) to things that don’t look much like randomized trials at all.  For these reasons we strongly advise that reviewers define the precise nature of the design of studies they are labelling as “quasi-RCTs” to avoid ambiguity. The definition should use the features of the study design to describe the studies you plan to include.

It is the reviewers’ decision whether to include studies other than RCTs or not. If you know you will have enough RCTs to answer a question we would advise excluding studies that have not used randomisation. If you do include them, we would advise a secondary analysis to see how data from studies without randomisation quasi RCTs influences the treatment effect.

The second part of our answer covers which risk of bias tool to use: Studies that are reasonably similar to randomized trials could go through either tool. 

The Cochrane Handbook advises:

  • reviews including “randomized controlled trials or “quasi” randomized trials (trials that closely mimic RCTs) should use RoB 2 for all studies.
  • reviews including “non-randomized studies including “quasi” randomized trials (trials that closely mimic RCTs) should use ROBINS-I for all studies.

We have included cluster RCTs, is RoB 2 able to assess cluster RCTs?  

A: Yes. There is version of RoB 2 specific to cluster RCTs and it was released in March2021. It contains an additional domain to assess bias associated with timing and recruitment of participants. The new version is available from riskofbias.info website. 

We have included cross-over RCTs, can I use the tool for cross-over RCTs?  

A: Yes. There is a version of RoB 2  specific to cross-over RCTs and it was released in March 2021. It contains an additional domain to assess bias associated with period and carryover effects. The new version is available from riskofbias.info website. 

Are there Excel RoB 2 tools for the cluster and crossover versions of the RoB 2 tool?

A: Yes, the excel tools for the cluster and cross-over versions of the RoB 2 tool are available from the riskofbias.info website.

RoB 2 RevMan Web and RevMan 5

We plan to use RoB 2 can we use the desktop version of RevMan (RevMan 5)? 

A: No. It is not possible for RevMan 5 to incorporate and display RoB 2. Authors can start their review in RevMan 5 and switch to RevMan Web when they get to the point of adding their risk of bias assessments. Here is a link to the RevMan Web knowledge base.

How can I “switch on” RoB 2 in RevMan Web?

A: Authors are not able to switch on the RoB 2 features in RevMan Web themselves. We advise the switch is made after authors have input their results data to RevMan as the ‘switch’ will break compatibility for the review to be worked on in the desktop RevMan 5. Please inform your Managing Editor when you are ready to switch on the RoB 2 features in your review and they can enable this advanced feature for you.

Can I edit my review in RevMan 5 after it has been “Switched” to use RoB 2 in RevMan Web?  

A: No, to use RoB 2 you will need to only use RevMan Web once your ME has arranged to switch on the RoB 2 features. You will not be able to check out the review when you have turned on these features. See Starter Pack and RevMan Web knowledge base.

Does the RoB 2 Excel tool link into RevMan Web? 

A: The Excel file cannot be imported into RevMan Web (at this point in time). The RoB 2 team are working on an online tool which should be available in 2022.

Our advice is that detailed risk of bias assessments (with consensus responses to the signalling questions) may be presented in supplementary materials via an online repository such as datadryad.org or figshare.com (see the Editorial Policy here). Then authors need to manually copy over the decisions for each domain, the overall risk of bias and the support for judgements into RevMan Web.  The domain ratings from all authors assessing risk of bias do not need to be included (only the agreed, consolidated ratings). 

If we need to use one of the tools and decide to use the Excel tool, do we then need to manually enter only the domain judgements into RevMan Web, or also the “decision information”? 

A: Only the final, consensus agreed judgements and support for judgements (for each result assessed, each domain and overall) should be included in RevMan Web. See this step-by-step guide on what to input into RevMan Web. 

How will RoB 2 be displayed in RevMan Web if we have a bias assessment per outcome? 

A: Authors are expected to include one figure per outcome and one table per outcome. Once bias assessments have been entered into RevMan Web the software can generate forest plots with traffic lights and risk of bias tables for each outcome. Visual representations of risk of bias assessments for outcomes in ‘narrative’ synthesis will need to be created by the authors. For example by using robvis software. For outcomes with a meta-analysis we recommend that in the main text of the review authors cite the relevant Analysis. Doing this will link to the relevant forest plot with traffic lights plot. You will not need to create additional Figures. The risk of bias outcome-level tables are interactive and show the judgment for each domain, for each study, for each outcome. These judgements “expand” to reveal the “justification” or support for judgement on the html version of the review. For full guidance please see the Starter Pack and RevMan Web knowledge base.

Would you recommend that we create a separate Excel file for each outcome in order to be able to create the summary tables and outputs required for RevMan Web?

A: Some reviewers are using Excel for all outcomes. Other reviewers create a new excel file for each outcome. 

Is it possible to do the RoB 2 assessments directly in RevMan Web or do we need to use one of the tools (e.g. Excel tool) and then add this to RevMan Web?

A: We advise that you use the Excel tool to complete the RoB 2 assessments. This is available on the riskofbias.info website. We recommend this because it will store the answers to all the signalling questions for each reviewer in a succinct format, allows authors manage the reaching of consensus agreements and can be used to create an excel file of consensus agreements to upload and make available for readers and peer reviewers.  

Once your author team confirms the final, consensus agreed judgements and support for judgements (for each outcome) these should be input into RevMan Web. At the moment this needs to be done manually by copy and pasting from the tool to RevMan Web.

We ask that review teams make the consensus decisions and answers for each signalling question available as a link to data repository as peer reviewers, their CRG may want to check a few of these. And in the finished review they will need to be available for all readers.  Detailed guidance is available in the Starter Pack and RevMan Web knowledge base.

I have started my review in RevMan 5, what changes do I need to make to switch to RevMan Web to use RoB 2?

A: RoB 2 cannot be displayed using RevMan 5 software. Authors can transfer their review from RevMan 5 to RevMan Web. This is described in the RevMan Web knowledge base.  Once you switch on the RoB 2 features in RevMan Web some specific functions will no longer be available. 

  • Compatibility with the desktop RevMan (RevMan 5)
  • Covidence import is not available in RevMan Web (but RIS import is)
  • Study-centric data cannot be used together with RoB 2 at the moment, but we do plan to have that in place. 

 

I have a study with two subgroups with different ROB 2 assessments for each of the subgroups, can I plot these on the same Forest plot?  

A: Unfortunately, it is not possible, in RevMan web, to store two sets of ROB 2 assessments for a single study. This means for subgroups you can only input one set of ROB 2 assessments.  

 

 

 

 

 

Reporting RoB 2

Should we be using the fixed headings for ‘Domains’ in the ‘Risk of bias in included studies’ section in RevMan Web? As these do not appear to fit with the RoB 2 guidance - which is to not present the domains.  

A: The new reporting guidance encourages authors to link risk of bias discussions more closely to the effect estimates for each outcome. This is because RoB 2 assesses bias separately for each outcome. The headings for risk of bias domains are removed in RevMan Web when authors switch on the RoB 2 features. Detailed reporting guidance is included in the Starter Pack.

Our review includes RoB 2 assessments for both ‘Effect of Adherence’ and ‘Effect of Assignment’ how can we present them both in RevMan Web?  

A: At the moment it is not possible to present risk of bias for both types of “Effects” in RevMan Web. Our advice is to present the bias that is more prominent in your review, for example that related to the key research question(s) into forest plots and RoB 2 tables that are embedded in RevMan Web.  We advise that you create additional figures and tables outside of RevMan Web for the secondary bias assessment. These can be included either as “Additional table” and “Figures” in RevMan or included in an appendix.  The robvis tool allows authors to create traffic light plots. The images created can be exported in a range of high-quality image files that can be imported into the review either as figures. Guidance on how to do this is in the Starter Pack.

How do we create figures when a review includes cluster and crossover trials? 

A: RoB2 for cross-over RCTs includes an additional domain "Risk of bias arising from period and carryover effects".  RoB 2 for cluster RCTs includes an additional domain “Bias related to timing and recruitment of participants". It is not possible, at the moment, to display this using the current figures in RevMan Web. 

Therefore, we advise that you use the robvis tool to prepare a figure. The images can be exported and added to RevMan Web as additional figure. Guidance on how to do this is in the Starter Pack.

We have entered bias assessments for each outcome for our main analysis in RevMan Web. We want to do a sensitivity analysis do we have to re-enter the bias assessment reason for the judgment?

A: We advise authors create a duplicate of their main analysis, which will also duplicate the risk of bias data (you do not have to re-enter the reasons for the judgements). You can then edit this for your sensitivity analysis. Also to note that there is a feature specifically for Sensitivity analysis in RevMan Web, you can see how it works here.

Do we need to keep the answers to the signalling questions for each of the results for each study? 

A: Our  recommendation is that detailed risk of bias assessments (with consensus responses to the signalling questions) may be presented in supplementary materials via an online repository such as datadryad.org or figshare.com (see the Editorial Policy. Then authors need to manually copy over the decisions for each domain, the overall risk of bias and the support for judgements into RevMan Web.  The domain ratings from all authors assessing risk of bias do not need to be included (only the agreed, consensus judgements). 

RoB 2 and other review development software

Can we use Covidence to do RoB 2?

A: At the moment the risk of bias assessment section in Covidence is not compatible with RoB 2. 

How does RoB 2 fit with CiNEMA for network meta-analyses? 

A: RoB 2 works well with CINeMA as it expects an overall risk of bias judgement to inform the confidence considerations, which the RoB 2 tool produces. See this article on CINeMA.

Can we use Distiller to do RoB 2?  

A: At the moment there is no formal arrangement, but it might be possible in the future to export RoB2 information in a format compatible with RevManWeb, once the RoB 2 import function is working in RevManWeb.