Why do we need Evidence-Based Methods in Cochrane Reviews?
Professor John Ioannidis from Stanford University presented on "Increasing value, reducing waste in research: a role for meta-epidemiology"
Co-chairs: Professor Mike Clarke and Dr Jo McKenzie (Read the recently published full editorial on the Cochrane Library)
This year’s methods symposium raised the profile of an evidence-based approach to the methods used in Cochrane reviews, and discussed ways we assess and evaluate our methods. Methods underpin systematic reviews, therefore, their evaluation and development is necessary to ensure the results from systematic reviews are not misleading. Likewise, such methods research can play a critical role in reducing waste in research.
The symposium addressed the identification and development of methods to evaluate methods for systematic reviews, how we propose to evaluate the performance of these methods, what constititues enough evidence to introduce a change to methods and provide transparency of the strength of evidence for a given method. The programme also included implementation of methods in Cochrane and decision making on best available methods.
The full programme of speakers and their presentations is listed. Slides are available where there is a link in the speaker's name.
(Speakers' biographies can be downloaded as a PDF file here)
It is ‘highly desirable’ to include a Summary of Findings (SoF) in a Cochrane systematic review to present the statistical results and the quality of evidence for the most important outcomes. This talk will review how the methods of data presentation in SoF tables are being evaluated. | ||
Fiona Alderdice | The BRIEF Trial: Do review summaries help midwifery students understand the findings of systematic reviews? The BRIEF randomised trial evaluated how midwifery students interpreted the findings from Cochrane Reviews when the evidence was presented in two different summary evidence formats, abstract and PLS, with and without the provision of the authors’ conclusion. The findings will be discussed reflecting on implications for summary development and gaps in knowledge. | |
Mike Clarke | This talk will introduce the Symposium and the need to ensure that the methods we use for all aspects of the design, conduct and dissemination of Cochrane Reviews are as reliable and robust as possible. It will highlight why we need to recognise when there are uncertainties about our methods and how we might resolve those uncertainties, including through the use of embedded methodological research. SWAR (Study Within A Review) are methodological research projects that could be embedded in any part of the review process to investigate the effects of different methods, such as searching, data extraction, analysis or reporting. They can also be used when disseminating the results of the review. The SWAR that will be presented in this session would examine the impact of including varying amounts of detail about the authors of the review in the media summary | |
Jon Deeks | There has been no shortage of evaluations of bias associated with non-randomised studies, but little agreement in what the results mean. This presentation will critique the evidence-base, and explain why we thought just one more evaluation was needed … | |
Julie Glanville | Research conducted into this question relating to the ClinicalTrials.gov database and the International Clinical Trials Registry Platform will be reported. | |
Jeremy Grimshaw | The presentation will:- · consider how Cochrane has developed and introduced methods historically · identify barriers and facilitators to introduction and uptake of methods · discuss using an ‘implementation’ lens during future development and introduction of new methods. | |
Julian Higgins | An overview of approaches to evaluation of statistical methods for meta-analysis. Taking as an example the choice of an estimator of between-study variance in a random-effects meta-analysis, The roles of statistical theory, empirical comparisons and simulation methods will be described. | |
Tom Jefferson via video | {"preview_thumbnail":"/sites/methods.cochrane.org/files/styles/video_embed_wysiwyg_preview/public/video_thumbnails/0VWA6NeJXCQ.jpg?itok=keUtln5V","video_url":"https://youtu.be/0VWA6NeJXCQ","settings":{"responsive":1,"width":"854","height":"480","autoplay":0},"settings_summary":["Embedded Video (Responsive)."]}A recent survey has shown that only 10% of Cochrane reviews make serious efforts to search for and include unpublished material. The recent series of cases of exposure of sponsor bias changing our understanding (and is some cases the registration) of important interventions (i.e. second generation antipsychotics) has shown that reliance on published material can be highly misleading. Journals (and ultimately research synthesisers and decision-makers) are usually presented with a very short summary of a selected trial which is part of a larger research programme. Given the growing realisation that these form a potentially biased evidence base, we may need to develop explicit methods for including regulatory material in systematic reviews. But is it feasible or worthwhile? Tom Jefferson will present and discuss some of these issues on the basis of his experience of working with regulatory documents. | |
Toby Lasserson | The CEU review screening programme has exposed a number of issues relating to implementation of methods. The talk will focus on how we can identify, generate and use a growing body of evidence on implementation to support better practices in the production of reviews. | |
Jo Mckenzie | The path from development to eventual implementation of methods for systematic reviews is in many ways analogous to that of a clinical intervention. Yet, the similarities are under-recognised. This presentation will outline stages in the identification, evaluation and implementation of methods for systematic reviews. | |
Carl Moons | Novel reviews require novel methods: when are we satisfied? When novel types of systematic reviews are being considered, often novel methodological guidelines for design, analysis and reporting are needed. The question arises when can we feel confident enough to endorse and implement these new methods and thus new types of reviews within Cochrane? Lessons learnt from systematic reviews of prognostic studies. | |
Mona Nasser | Towards an evidence informed approach to set priorities for topics of systematic reviews. The presentation outlines a critical evaluation of the literature on describing and evaluating different approaches to set priorities for research. It will discuss how the evidence on this topic informs decisions to prioritise topics for Cochrane reviews. | |
Jane Noyes & Andrew Booth | The CERQual (Confidence in the Evidence from Reviews of Qualitative research) Approach provides a transparent method for assessing the confidence of evidence from reviews of qualitative research, and indicating this confidence to end users, such as guideline panels or decision makers. CERQual uses a similar approach conceptually to other GRADE tools, but is intended for findings from systematic reviews of qualitative evidence. The development of CERQual is being taken forward through the GRADE-CERQual Project Group, a subgroup of the GRADE Working Group. | |
Barney Reeves | Cochrane has formalised guidance that review authors should consider including non-randomized studies of the effects of interventions (NRSi) when evidence from RCTs alone is judged insufficient or inadequate to answer the review question of interest. This guidance has important implications for review methods. This presentation will cover some of the methodological research that the guidance has already prompted. | |
Ian Roberts (via video) & Emma Sydenham (Debate - For the Motion) | {"preview_thumbnail":"/sites/methods.cochrane.org/files/styles/video_embed_wysiwyg_preview/public/video_thumbnails/65K4htyx094.jpg?itok=7m2PGFJt","video_url":"https://youtu.be/65K4htyx094","settings":{"responsive":1,"width":"854","height":"480","autoplay":0},"settings_summary":["Embedded Video (Responsive)."]} Methodological diversity and innovation is our strength – the move to central methodological mandates is the beginning of the end for Cochrane. Cochrane is a devolved organization – a family of highly motivated editorial teams innovating in response to the diverse challenges they face. Management consultants are paid millions to create situations like this. But Cochrane wants to move in the opposite direction. In the interests of consistency it is proposed that "all groups MUST implement new methods collectively." This presentation argues that to prosper COCHRANE must innovate and diversity is the lifeblood of innovation. So let a hundred flowers bloom and say no to central methodological mandates | |
Georgia Salanti (Debate - Against the Motion) | ||
Mark Simmonds | It is important to update Cochrane reviews, but repeated meta-analyses as more studies are identified may lead to false-positive findings. This talk discusses a Cochrane-funded project to identify methods to update meta-analyses, compare and evaluate their properties, and make recommendations on how they should be used. | |
Yemisi Takwoingi | Cochrane diagnostic test accuracy reviews undergo rigorous editorial process by a Cochrane Review Group and the centralised DTA Editorial Team. This talk will draw on the experience of the DTA editorial process as a channel for identifying common methodological challenges and uncertainties, and for facilitating introduction and consistent use of approved methods. | |
James Thomas | Given the time and resource pressures on reviewers, and the increasing volume of potentially relevant literature, there is considerable interest in using new technologies to reduce workload in systematic reviews. I will briefly review the evidence base, and make some suggestions as to how practice and methods development might proceed in this area. |