Insights from the first Methods Forum hosted by Cochrane Germany 

Posted on 7 April 2020 and written by Marius Goldkuhle (Research Associate, Cochrane Haematology Group) and Maria-Inti Metzendorf (Information Specialist, Cochrane Metabolic and Endocrine Disorders Group)

Clinical study reports (CSR) are known to hold extensive and less distorted study data than journal publications making them especially valuable for enhancing the quality of Cochrane reviews of interventions (see here, here and here). 

In February 2020, Cochrane Germany invited interested researchers to its first Methods Forum in Freiburg focusing on CSRs. Various stakeholder representatives, including the two Germany-based Cochrane Groups (Cochrane Metabolic and Endocrine Disorders and Cochrane Haematology), Cochrane Germany and the Institute for Quality and Efficiency in Health Care (IQWiG) presented on and discussed their experiences with the retrieval and utilization of CSRs. In this blog post, we summarize key insights gained at this event:

An excellent start to the day involved introductory presentations by Jörg Meerpohl, Director of Cochrane Germany, and Toby Lasserson, Cochrane´s Deputy Editor-in-Chief, both of whom addressed various sources of study data currently available, the overarching problem of publication bias, and concluded with the benefits of using CSR data for systematic reviews of pharmacological interventions. 

Maria-Inti Metzendorf, Information Specialist at the Cochrane Metabolic and Endocrine Disorders Group, presented the key steps required to identify CSRs. There are multiple potential sources of reports and limited prospects of being granted access to all CSRs identified, which demands a substantial degree of process systematization. It became evident that the initial step for effective identification of CSRs consists in determining a set of potentially eligible studies by using data sources beyond literature databases, such as extracting studies from the different reports of regulatory agencies, contacting pharmaceutical companies, searching trials registers, as well as different online platforms that offer access to CSR. This process should ideally lead to a matrix providing an overview of potentially existing studies relevant to a review question. Once this set of studies is determined, targeted searches and requests for CSRs are possible. Moreover, this matrix also allows identifying studies for which a respective CSR is absent. This ultimately leads to a discussion on selective availability of reports, which arises when manufacturers or regulatory agencies limit the access to reports with specific properties. 

A presentation with accompanying exercise held by Beate Wieseler, Head of the Drug Assessment Department at IQWiG, which routinely uses CSRs for their health technology reports, demonstrated to participants the extensive, but highly structured content of CSR, which (to put it in her encouraging words, "don't worry – be happy") enables the efficient identification of relevant study characteristics and outcome data. For systematic reviews, the extent of study data and its predetermined structure substantially facilitates its extraction and risk of bias judgments – CSRs include detailed descriptions of study attributes, endpoints (especially detailed data on adverse events) and participant flow. Nevertheless, further discussions elucidated that handling discrepancies in the reported data between different data sources, such as trials register results records, CSRs, and publications, represents a true challenge. Additional guidance including, e.g. a hierarchy of data sources, is therefore required if CSRs are to be used routinely as data sources for systematic reviews.

Subsequently, Bernd Richter, Co-ordinating Editor of the Metabolic and Endocrine Disorders Group, and Marius Goldkuhle, research associate at the Haematology Group, provided practical insights from both Cochrane Groups who are currently working on reviews incorporating data from CSR. It is considerably more challenging for Cochrane Groups to gain access to CSRs compared to IQWiG, which has dedicated contracts with the pharmaceutical industry. Discussions among participants confirmed that access to CSRs remains a (if not the) core challenge for Cochrane authors. Direct requests to manufactures require careful communication processes and in most cases contracts between the host institution of Cochrane authors and the manufacturer, which impose a substantial burden on review authors. Platforms that aim to ease access and allow to centrally request CSRs from manufacturers are not straightforward to use and often only provide a small proportion of studies. Some, such as Clinical Study Data Request (CSDR), include options to enquire about the availability of CSRs from manufacturers, but not for all potentially eligible studies or in a transparent and structured way. Participants pleaded for Cochrane to take a central and harmonizing role to facilitate access to and requesting CSRs for review authors.

The experiences presented made clear that access to CSR appears to be dependent on the review question of interest or, more precisely, on the manufacturers of the interventions under comparison. While the Cochrane Metabolic and Endocrine Disorders´ authors were able to identify and receive multiple reports from a single manufacturer for their pilot review on insulin analogues, Cochrane Haematology’s experience on the retrieval of CSRs for oncological treatments showed rather restricted accessibility of reports.

Contacting regulators, specifically the European Medicine Agency (EMA) or Health Canada, is a second option to gain access to CSRs originally submitted to the regulatory agencies by the manufacturers. For example, EMA offers access to a set of CSRs to European citizens after registration on their website. Despite the ongoing suspension of CSR publication (since 4/12/2018) due to EMA´s relocation to Amsterdam, access is limited to CSR submitted for marketing applications and therefore only relevant for novel substances. Given these restrictions, selective availability makes direct contact with manufacturers unavoidable in many cases. In addition to direct CSR access through EMA´s clinical data platform, direct contact with the agency is recommended. Even though experiences were still limited, requests can be submitted and are dealt with by a single responsible unit, which monitors and processes enquiries by all Cochrane representatives in a chronological order, unless otherwise stated. To ensure a timely response they require a detailed list of requested CSRs.

To date, the usage of CSR data in Cochrane reviews demands an important degree of endeavor. It may therefore only be a viable option for experienced review authors with excellent data management skills and enough time capacity allocated to their project. There are substantial advances to be expected from the inclusion of CSR data for the enhancement of review quality, justifying these efforts. Furthermore, guidance and support from within Cochrane (Cochrane’s Clinical Study Reports Working Group) and outside (Restoring Invisible and Abandoned Trials (RIAT)) are available which will also evolve and might make it easier for more review authors to … dig a little deeper.