The Cochrane Rapid Reviews Methods Group is leading an important piece of methodological research and needs your help.
Does it make a difference whether abstracts are screened by one or two people to find relevant studies for a systematic review? How many relevant studies will be missed when abstracts are reviewed by only one person?
Can the Cochrane RCT-classifier, a machine tool for classifying RCTs, increase this accuracy? The Cochrane Rapid Reviews Methods Group and the Cochrane Crowd Team have developed a study to answer these questions.
The study is made up of three ‘arms’ – three different groups that participants to the study could be assigned to. If you already have experience screening literature for a review, you will be randomly assigned to review a set of 100 abstracts and classify them as "Relevant" or "Not Relevant" to the research question. Then you will be randomly assigned to screen a set of abstracts retrieved from a search for either a pharmacological review or for a public health review that has broader study design inclusion criteria. If you have no experience screening for reviews, you will be assigned to the "RCT-classifier" arm where you can screen as many records as you like for the pharmacological review. In this arm, the "RCT Classifier" is tested by one person in combination with the literature review.
It would be fantastic if you could join this study. The study starts on Thursday, 23rd May.
To join, just follow these 4 steps:
1. If you don't have a Cochrane Crowd account yet, register here first and continue with step 2. If you already have a Cochrane Crowd account, you can go straight to step 2:
2. Click on this link that will launch the brief participant questionnaire
3. Once you have completed the questionnaire, log in to Cochrane Crowd as usual
4. Near the top of your task dashboard, you will see the task called Rapid, Rapid, Rapid. This is the task you will have been randomly assigned to (so exciting!). Complete the training and get screening your 100 records!
You don’t have to screen all 100 records in one go. We will keep the study running until Friday 14th June so there should be plenty of time for you to chip away at your 100 records.
If you are able to take part, you will be named on a conference poster about the study at the Cochrane Colloquium, receive a certificate of participation, and enter a prize draw for one of five gift vouchers worth $100.
Happy citation screening!
Rapid review and scoping reviews - 5 day introductory short-couse (Institute for Mental Health, Singapore) - January 2019.
Course Facilitators: Chantelle Garritty & Adrienne Stevens
IUMSP Institut universitaire de médecine sociale et préventive (Lausanne, Switzerland) - Nov. 2018.
Symposium Presentation: What is happening in the world of rapid reviews (C Garritty) (Nov. 29)
1-day Short Course: Introduction to Rapid Reviews (Facilitators: C Garritty, B Nussbaumer-Streit) (Nov. 30)
Global Evidence Summit (GES) (Cape Town, South Africa) - Sept. 2017
A panoramic view of rapid reviews: Uses and perspectives from global collaborations and networks
RRMG activity at GES started with Co-Convenors facilitating a session in order to exchange ideas about how rapid reviews have evolved as a useful information tool to support evidence-informed policy and practice. This session was a platform for various international groups and collaborations to convey their experiences with rapid reviews through various lenses including those of systematic review producers, guidelines developers, and those undertaking rapid reviews specific to health policy, in particular for LMICs. In summary, we had six presentations by representatives from the Cochrane Rapid Reviews Methods Group (RRMG), Cochrane Response, Campbell Collaboration/International Initiative for Impact Evaluation (3ie), Guidelines-International-Network (G-I-N), WHO Guideline Review Committee, and the WHO Collaborating Centre for Evidence-Informed Policy and Practice. Overall, this panel showcased the relevance of rapid reviews in an attempt to harness a growing network of those involved in this type of knowledge synthesis methodology. Facilitators were: In addition to the Cochrane RRMG Co-Convenors, the following individuals participated as co-facilitators: Karla Soares-Weiser, Susan L Norris, Daniel Phillips, Michel Laurence.
An introduction to rapid reviews: Developing timely evidence summaries for decision-makers (A Cochrane Rapid Reviews Methods Group Workshop)
RRMG activity at GES continued with a 90-minute workshop attended by 70+ participants. This session was a mix of presentations, and a small group exercise in which attendees were asked to assess and discuss the attributes of a sample of rapid review reports produced by different organizations in order to emphasize the spectrum of possible approaches to conducting rapid reviews vs. a one size fits all approach. Various technical issues were also discussed including report layout and customization, transparency, and involvement of decision-makers in the process. Importantly, the session emphasized the need to assess appropriateness in undertaking rapid reviews while underscoring the importance of managing scope through the various stages of the rapid review process. Facilitators were Chantelle Garritty, Valerie King, Barbara Nussbaumer, Adrienne Stevens, Gerald Garlehner, and Becky Skidmore.
24th Cochrane Colloquium (Seoul, South Korea) - Oct. 2016
Workshop (half-day) - Developing timely evidence summaries for decision-makers
RRMG activity at the Colloquium opened with a half-day workshop for 30 participants covering an introduction to rapid reviews, a highlight of methods research to date, a step-by-step on conducting rapid reviews, understanding the spectrum of RR products, and various synthesis approaches. The hands-on component involved an exploration of rapid review examples. The facilitators were Valerie King, Barbara Nussbaumer, and Adrienne Stevens.
Workshop - Rapid review programs and methods research: description and discussion of the experiences from among Convenors of the Cochrane Methods Rapid Reviews Group.
This workshop opened with rapid talks from among the Co-Convenors about their RR programs and affiliations (King, Kamel, Stevens/Garritty, Cochrane Austria), followed with two presentations from Cochrane Austria about recent methods work they have been undertaking and planning (Cochrane Austria methods projects). Thanks to those who attended and participated in the discussion!
The Cochrane Rapid Reviews Methods Group (RRMG) Open Meeting
The group’s first Colloquium open meeting! Following an overview of rapid reviews and the group’s activities by Valerie King (meeting overview), we learned about recent methods work, two RR programs, and the CEU perspective:
- Barbara Nussbaumer and Gerald Gartlehner, Cochrane Austria and RRMG Co-Convenorship, shared results from one project on surveys of guideline developers and policymakers in relation to level of acceptable risk of incorrect answers from RRs and methods for a second project that is underway regarding abbreviated searches (Cochrane Austria methods projects)
- Mala Mann, PaCERS Unit, Cardiff University, Wales described their newly formed RR unit (PaCERS progress)
- Michelle Macvean, Parenting Research Centre, Australia spoke on their experience with conducting rapid evidence assessments (PRC RRs)
- Karla Soares-Weiser, Cochrane Editorial Unit spoke on the important role that the RRMG will play for Cochrane Response in providing guidance and encouraged methods development from the group
The meeting included an opportunity to hear from participants on priority areas. Thank you to those who attended our meeting!
Rapid Review methods and uptake
Developing a rapid response system within systematic review centers to address priority needs from policymakers (long oral session, p.15)
Evolution of rapid evidence review methods to support policy decisions (long oral session, p. 15)
Trading certainty for speed: how much uncertainty are decision-makers willing to accept when using rapid review? An international survey (short oral session, p. 69)
Ultra-rapid HTAs: a survey of usefulness and influence in decision making (short oral session, p. 69)
Automation or other tools
Semi-automating citation screening: a retrospective assessment of a hybrid machine learning/crowdsourced approach using one year’s worth of human-generated data from the Embase crowdsourcing project (long oral session, p. 17)
SWIFT-Active Screener: reducing literature screening effort through machine learning for systematic reviews (long oral session, p.17)
Data Abstraction Assistant (DAA): a new open-access tool being developed and tested in a randomized controlled trial (long oral session, p.23)
Feedback on Covidence by systematic reviewers (poster, p. 154)
Previous Workshops Held (2015):
23rd Cochrane Colloquium - Vienna, Austria (Oct. 2015)
Evidence Live2015 - Oxford, UK (April 2015)