Resources and publications

The Prognosis Methods Group has developed a Research Framework that has been used to set research priorities. The four key research questions are:

1. What is the course of the condition/disease? (descriptive)

2. What prognostic factors are associated with outcome? (Explanatory)

3. What groups of prognostic factors best predict outcome? (Outcome prediction)

4. What are the interactions between intervention and prognostic factors?

In relation to the development of your systematic review methods, the methodological issues include:
a. Definition of Prognosis Studies
b. Search strategies in prognosis
c. Risk of bias in prognosis studies
d. Data analysis in prognosis studies including assessment of impact of risk factors

The following methodological tools have been developed to support your review:

QUIPS: The QUIPS tool can be used to assess risk of bias in prognostic factor studies. 

PROBAST: The PROBAST initiative has developed a formal risk of bias tool for risk prediction modeling studies.

CHARMS: The CHARMS checklist is a template and checklist for designing the review, for data extraction and critical appraisal in systematic reviews of risk prediction modeling studies.

TRIPOD: The TRIPOD statement a set of evidence-based reporting guidelines for studies developing, validating, or updating (diagnostic and prognostic) risk prediction models. It was developed by an international group of experts comprising of statisticians, methodologists, clinicians and medical journal editors.

SEARCH strategy: A validated search strategy for the retrieval of primary studies for systematic reviews of risk prediction modelling studies was published in 2012.

PROGRESS: A series of five papers, linked below, were published in 2013 and 2014 on the essentials of the different types of prognostic studies, and a recommendation for the registration and sharing of protocols of prognostic studies.

GRADE: The GRADE Working Group have developed recommendations for prognostic evidence and provide a practical and useful approach to determining confidence in estimates of prognosis in broad populations.


Cochrane Prognosis Methods Group

Riley RD, Ridley G, Williams K, Altman DG, Hayden J, de Vet HC. Prognosis research: towards evidence-based results and a Cochrane Methods Group. [Comment]. Journal of Clinical Epidemiology 2007; 60(8): 863-865.

Prognostic Factor Exemplar Reviews

Hayden JA, Tougas ME, Riley R, Iles R, Pincus T. Individual recovery expectations and prognosis of outcomes in non-specific low back pain: prognostic factor exemplar review. Cochrane Database of Systematic Reviews 2014, Issue 9. Art. No.:CD011284. DOI: 10.1002/14651858.CD011284

Overall Prognosis

Croft P, Altman DG, Deeks JJ, Dunn KM, Hay AD, Hemingway H, LeResche L, Peat G, Perel P, Petersen SE, Riley RD, Roberts I, Sharpe M, Stevens RJ, van der Windt DAWM, von Kor M, Timmis A. The science of clinical practice: disease diagnosis or patient prognosis? Evidence about "what is likely to happen" should shape clinical practice. BMC Med 2015;13:20.

Riley RD, Steyerberg EW. Meta-analysis of a binary outcome using individual participant data and aggregate data. Journal of Research Synthesis Methods 2010; 1: 2-19.

Primary Prognosis Study Methods

Altman DG. The time has come to register diagnostic and prognostic research. Clin Chem 2014:60:580-582.

Hemingway H, Croft P, Perel PA, Hayden JA, Abrams K, Timmis AD, Briggs A, Udumyan R, Moons
KGM, Steyerberg EW, Roberts IG, Schroter S, Altman DG, Riley RD for the PROGRESS Group.
Prognosis research strategy (PROGRESS) 1: A framework for researching clinical outcomes. BMJ 2013;346:e5595.
Riley RD, Hayden JA, Steyerberg EW, Moons KGM, Abrams K, Kyzas PA, Malats N, Briggs A, Schroter S, Altman DG, Hemingway H for the PROGRESS Group. Prognosis research strategy (PROGRESS) 2: Prognostic factor research. PLoS Med 2013;10:e1001380.
Steyerberg EW, Moons KGM, van der Windt DA, Hayden JA, Perel P, Schroter S, Riley RD, Hemingway H, Altman DG for the PROGRESS Group. Prognosis research strategy (PROGRESS) 3: Prognostic model research. PLoS Med 2013;10:e1001381.
Hingorani A, van der Windt DA, Riley RD, Abrams K, Moons KGM, Steyerberg EW, Schroter S,
Sauerbrei W, Altman DG, Hemingway H for the PROGRESS Group. Prognosis research strategy
(PROGRESS) 4: Stratified medicine research. BMJ 2013;346:e5793.

Altman DG, McShane LM, Sauerbrei W, Taube SE. Reporting recommendations for tumor marker prognostic studies (REMARK): explanation and elaboration. BMC Medicine 2012; 10:51. (Also published in PLoS Med 2012; 9: e1001216).

Hemingway H, Riley RD, Altman DG. Ten steps towards improving prognosis research. British Medical Journal 2010; 339: b4184.

Hayden JA, Côté P, Steenstra IA, Bombardier C, for the QUIPS-LBP Working Group. Identifying phases of investigation helps planning, appraising and applying the results of explanatory prognosis studies. Journal of Clinical Epidemiology 2008; 61: 552-560.

Wilczynski NL, Haynes RB. Optimal search strategies for detecting clinically sound prognostic studies in EMBASE: an analytic survey. Journal of the American Medical Informatics Association 2005; 12(4): 481-485.

Wilczynski NL, Haynes RB; Hedges Team. Developing optimal search strategies for detecting clinically sound prognostic studies in MEDLINE: an analytic survey. BMC Medicine 2004; 2: 23.

Royston P, Parmar MKB, Altman DG. Visualizing length of survival in  time-to-event studies: A complement to Kaplan-Meier Plots. Journal of National Cancer Institute 2008; 100: 92-97.

Risk Prediction Models

Debray TPA, Damen JAAG, Snell KIE, Ensor J, Hooft L, Reitsma JB, Riley RD, Moons KGM. A guide to systematic review and meta-analysis of prediction model performance. BMJ 2017; 356: i6460.

Ahmed I, Debray TPA, Moons KGM, Riley RD. Developing and validating risk prediction models in an individual participant data meta-analysis. BMC Medical Research Methodology 2014; 14:3.

Debray TPA, Koffijberg H, Nieboer D, Vergouwe Y, Steyerberg EW, Moons KGM. Meta-analysis and aggregation of multiple published prediction models. Statistics in Medicine 2014.

Debray TP, Vergouwe Y, Koffiberg H, Nieboer D, Steyerberg EW, Moons KG. A new framework to enhance the interpretation of external validation studies of clinical prediction models. Journal of Clinical Epidemiology 2014: Aug 29. pii: S0895-4356(14)00275-3. doi: 10.1016/j.jclinepi.2014.06.018.

Kengne AP, Beulens JW, Peelen LM, Moons KG, van der Schouw YT, Schulze MB, Spijkerman AM, Griffin SJ, Grobbee DE, Palla L, Tormo MJ, Arriola L, Barengo NC, Barricarte A, Boeing H, Bonet C, Clavel-Chapelon F, Dartois L, Fagherazzi G, Franks PW, Huerta JM, Kaaks R, Key TJ, Khaw KT, Li L, Mühlenbruch K, Nilsson PM, Overvad TF, Palli D, Panico S, Quirós JR, Rolandsson O, Roswall N, Sacerdote C, Sánchez MJ, Slimani N, Tagliabue G, Tjønneland A, Tumino R, van der A DL, Forouhi NG, Sharp SJ, Langenberg C, Riboli E, Wareham NJ. Non-invasive risk scores for prediction of type 2 diabetes (EPIC-InterAct): a validation of existing models. Lancet Diabetes Endocrinol 2014 Jan;2(1):19-29. doi: 10.1016/S2213-8587(13)70103-7. Erratum in Lancet Diabetes Endocrinol. 2014 Apr;2(4):e11.

Debray TPA, Moons KGM, Ahmed I, Koffijberg H, Riley RD. A framework for developing, implementing and evaluating clinical prediction models in an individual participant data meta-analysis. Statistics in Medicine 2013; 32: 3158-3180.

Debray TPA, Koffijberg H, Vergouwe Y, Moons KGM, Steyerberg EW. Aggregating published prediction models with individual patient data: a comparison of different approaches. Statistics in Medicine 2012; 31: 2697-2712.

Debray TPA, Koffijberg H, Lu D, Vergouwe Y, Steyerberg EW, Moons KGM. Incorporating published univariable associations in diagnostic and prognostic modeling. BMC Medical Research Methodology 2012; 12: 121.

Moons KGM, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, Grobbee DE. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 2012; doi:10.1136/heartjnl-2011-301246.

Moons KGM, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, Woodward M. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012; doi:10.1136/heartjnl-2011-301247.

Moons KGM, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? British Medical Journal 2009; 338: b375 (1317-1320).

Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. British Medical Journal 2009; 338: b604 (1373-1377).

Altman DG, Vergouwe Y, Royston P, Moons KGM. Prognosis and prognostic research: validating a prognostic model. British Medical Journal 2009; 338: b605 (1432-1435).

Moons KGM, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. British Medical Journal 2009; b338 (1487-1490). 

Mallett S, Royston P, Waters R, Dutton S, Altman DG. Reporting performance of prognostic models in cancer: a review. BMC Medicine 2010; 8: 21.

Mallett S, Royston P, Dutton S, Waters R, Altman DG. Reporting methods in studies developing prognostic models in cancer: a review. BMC Medicine 2010; 8: 20.

Altman DG. Prognostic models: A methodological framework and review of models for breast cancer. Cancer Investigation 2009; 27: 235-243.

Vergouwe Y, Royston P, Moons KGM, Altman DG. Development and validation of a prediction model with missing predictor data: a practical approach. Journal of Clinical Epidemiology 2010; 63(2): 205-214.

Vergouwe Y, Moons KGM, Steyerberg EW. External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. American Journal of Epidemiology 2010; 172(8): 971-980.

Marshall A, Altman DG, Holder RL, Royston P. Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines. BMC Medical Research Methodology 2009; 9: 57.

Sauerbrei W, Royston P, Binder H. Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Statistics in Medicine 2007; 26: 5512-5528.

Heymans MW, Van Buuren S, Knol DL, Van Mechelenen W, de Vet HCW. Variable selection under multiple imputation using the bootstrap in a prognostic study. BMC Medical Research Methodology 2007; 7: 33.

Bosch van de J, Kalkman CJ, Vergouwe Y, Klei van WA, Bonsel GJ, Grobbee DE, Moons KGM. Assessing the applicability of scoring systems that predict postoperative nausea and vomiting. Anaesthesia 2005; 60: 323-331.

Moons KG, Donders ART, Steyerberg EW, Harrell FE. Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example. Journal of Clinical Epidemiology 2004; 57: 1262-1270.

Kalkman CJ, Bonsel GJ, Visser K, Moen J, Grobbee DE, Moons KGM. Preoperative prediction of severe postoperative pain. Pain 2003; 105: 415-423.

Moons KG, Harrell FE, Steyerberg EW. Should scoring rules be based on odds ratios or regression coefficients? Journal of Clinical Epidemiology 2002; 55: 1054-1055.

Prognostic factors

El Aidi H, Adams A, Moons KGM, Den Ruijter HM, Mali WPTM, Doevendans PA, Nagel E, Schalla S, Bots ML, Leiner T. Cardiac Magnetic Resonance Imaging findings and the risk of cardiovascular events in patients with recent myocardial infarction or suspected or known coronary artery disease – a systematic review of prognostic studies, Journal of the American College of Cardiology (2014), doi: 10.1016/j.jacc.2013.11.048.

Henriksson M, Palmer S, Chen R, Damant J, Fitzpatrick N, Abrams K, Hingorani AD, Stenestrand U, Janzon M, Feder G, Keogh B, Shipley MJ, Kaski J-C, Timmis A, Sculpher M, Hemingway H. Assessing the cost effectiveness of using prognostic biomarkers with decision models: case study in prioritising patients waiting for coronary  artery surgery. British Medical Journal 2010; 340: b5606 (1-9). 

Sutcliffe P, Hummel S, Young T, Wilkinson A, Hamdy F, Clarke C, Staffurth J. Use of classical and novel biomarkers as prognostic risk factors for prostate cancer. Health Technology Assessment  2009; 13(5): 1-219.

Riley RD, Sauerbrei W, Altman DG. Prognostic markers in cancer: The evolution of evidence from single studies to meta-analysis and beyond. British Journal of Cancer 2009; 100: 1219-1229.

Sauerbrei W, Holländer N, Riley RD, Altman DG. Evidence based assessment and application of prognostic markers: The long way from single studies to meta-analysis. Communications in Statistics- Theory and Methods 2006; 35: 1333-1342.

McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, Clark GM, for the Statisticis Subcommittee of the NCI-EORTC Working Group on Cancer Diagnostics. REporting recommendations for tumour MARKer prognostic studies (REMARK). 
British Journal of Cancer 2005; 93(4): 387-391.
European Journal of Cancer 2005; 41: 1690-1696.
Nature Clinical Practice Oncology 2005; 2(8): 416-422.
Journal of the National Cancer Institute 2005; 97(16): 1180-1184.

Mallett S, Timmer A, Sauerbrei W, Altman DG. Reporting of prognostic studies of tumour markers: a review of published articles in relation to REMARK guidelines. British Journal of Cancer 2010; 102: 173-180.

Altman DG, Riley RD. Primer: an evidence-based approach to prognostic markers. Nature Clinical Practice Oncology 2005; 2: 466-472.

Panayiotis A.K., Konstantinos T.L., Ioannidis J.P.A. Selective reporting biases in cancer prognostic factor studies. Journal of the National Cancer Institute 2005; 97(14): 1043-1055.

Malats N. Bustos A. Nascimento CM. Fernandez F. Rivas M. Puente D. Kogevinas M. Real FX. P53 as a prognostic marker for bladder cancer: a meta-analysis and review. Lancet Oncology 2005; 6(9): 678-686.

Riley RD, Abrams KR, Lambert PC, Sutton AJ, Altman DG. Where next for evidence synthesis of prognostic marker studies? Improving the quality and reporting of primary studies to facilitate clinically relevant evidence-based results. In: Molenberghs G (eds). Proceedings of the Statistics in Health Sciences Conference, Nantes 2004.  Birkhauser and Springer.

Riley RD, Abrams KR, Sutton AJ, et al. Reporting of prognostic markers: current problems and development of guidelines for evidence-based practice in the future. British Journal of Cancer 2003; 88(8): 1191-1198.

Riley RD. Evidence Synthesis of Prognostic Marker Studies. Unspecified Thesis, ID 57976. University of Birmingham.

Prognosis systematic reviews methods

Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement. Ann Intern Med. 2015;162:55-63. doi:10.7326/M14-0697.

Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, Vickers AJ, Ransohoff DF, Collins GS. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration. Ann Intern Med. 2015;162:W1-W73. doi:10.7326/M14-0698.

Moons KGM, de Groot JAH, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, Reitsma JB, Collins GS. Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist. PLos Med 2014 11(10):e1001744. doi:10.1 371/journal.pmed.1001744.

Peat G, Riley RD, Croft P, Morley KI, Kyzas PS, Moons KG, Perel P, Steyerberg EW, Schroter S, Altman DG, Hemingway H for the PROGRESS Group. Improving the transparency of prognosis research: the role of reporting, data sharing, registration and protocols. PLoS Med. 2014 Jul 8;11(7):e1001671. doi: 10.1371/journal.pmed.1001671.

Debray TPA, Moons KGM, Abo-Zaid G, Koffijberg H, Riley RD. Individual participant data meta-analysis for a binary outcome: one-stage or two-stage?. PLoS ONE 2013; 8: e60650

Bouwmeester W, Zuithoff NPA, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KGM. Reporting and Methods in Clinical Prediction Research: A Systematic Review. PLoS ONE 2012; 9 (5):  e1001221 (1-13).

Geersing G-J, Bouwmeester W, Zuithoff P, Spijker R, Leeflang M, Moons K. Search filters for finding prognostic and diagnostic prediction studies in medline to enhance systematic reviews. PLoS ONE 2012; 7 (2): e32844 (1-6).

Hayden JA, Chou R, Hogg-Johnson S, Bombardier C. Systematic reviews of low back pain prognosis had variable methods and results- guidance for future prognosis reviews. Journal of Clinical Epidemiology 2009; 62: 781-796.

Riley RD, Abrams KR, Lambert PC, Sutton AJ, Thompson JR. An evaluation of bivariate random-effects meta-analysis for the joint synthesis of two correlated outcomes. Statistics in Medicine 2007; 26(1): 78-97.

Riley RD, Abrams KR, Sutton AJ, Lambert PC, Thompson JR. Bivariate random-effects meta-analysis and the estimation of between-study correlation. BMC Medical Research Methodology 2007; 7: 3.

Hayden JA, Côté P, Bombardier C. Evaluation of the quality of prognosis studies in systematic reviews. Annals Internal Medicine 2006; 144: 427-437.

Altman DG. Systematic reviews of evaluation of prognostic variables. British medical Journal 2001; 323(7306): 224-228.

Examples of prognosis systematic reviews

Woolfenden S, Sarkozy V, Ridley G, Coory M, Williams K. A systematic review of two outcomes in autism spectrum disorder- epilepsy and mortality. Developmental Medicine & Child Nuerology 2012 Apr;54(4):306-12. doi: 10.1111/j.1469-8749.2012.04223.x. Epub 2012 Feb 21.

Woolfenden S, Sarkozy V, Ridley G, Williams K. A systematic review of the diagnostic stability of autism spectrum disorder. Research in Autism Spectrum Disorders 2012; 6(1): 354-354.

Peters SAE, den Ruijter HM, Bots ML, Moons KGM. Improvements in risk stratification for the occurrance of cardiovascular disease by imaging subclinical atherosclerosis: a systematic review. Heart 2012; 98: -184.

van Dieren S, Beulens JWJ, Kengne AP, Peelen LM, Rutten GEHM, Woodward M, van der Schouw YT, Moons KGM. Prediction models fpr the risk of cardiovascular disease in patients with type 2 diabetes: a systematic review. Heart ; December 18, 2011 doi: 10.1136/heartjnl-2011-300734.

Ettema RGA, Peelen LM, Schuurmans MJ, Nierich AP, Kalkman CJ, Moons KGM. Prediction models for prolonged intensive care unit stay after cardiac surgey: systematic review and validation study. Circulation, 2010: 122: 682-689.

 Rahbari NN, Aigner M, Thorlund K, Mollberg N, Motschall E, Jensen K, Diener MK, Buchler MW, Koch M, Weitz J. Meta-analysis shows that detection of circulating tumour cells indicates poor prognosis in patients with colorectal cancer. Gastroenterology 2010; 138: 1714-1726.

Parsons A, Daley A, Begh R, Aveyard P. Influence of smoking cessation after diagnosis of early stage lung cancer on prognosis: systematic review of observational studies with meta-analysis. British Medical Journal 2010; 340: 65569 (

Whiteley W, Chong WL, Sengupta A, Sandercock P. Blood markers for the prognosis of ischemic stroke: A systematic review. Stroke 2009; 40:e380-e389.

Ip HYV, Abrishami A, Peng PWH, Wong J, Chung F. Predictors of postoperative pain and analgesic consumption: A qualitative systematic review. Anesthesiology 2009; 111: 657-677.

Pengel LHM, Herbert RD, Maher CG, Refshauge KM. Acute low back pain: systematic review of its prognosis. British Medical Journal 2003; 327(9): 323328.

Aina A, May S, Clare H. The centralization phenomenon ofspinal symptoms? A systematic review. Manual Therapy 2004; 9: 134-143.

Counsell C. Dennis M. Systematic review of prognostic models in patients with acute stroke. Cerebrovascular Diseases 2001; 12(3): 159-170.

Consultation of PMG

Sguassero Y, Cuesta CB, Roberts KN, Hicks E, Comandé D, Ciapponi A, et al. (2015). Course of Chronic Trypanosoma cruzi Infection after Treatment Based on Parasitological and Serological Tests: A Systematic Review of Follow-Up Studies. PLoS ONE 10(10): e0139363. doi:10.1371/journal.pone.0139363