Step 1

Selecting managers of the process

A small team of people who represent investors in health research, their interests and visions (stakeholders).

Their role is to assess the likelihood that the proposed research will reduce the burden of disease within the context of the investments being made.

Step 2

Process managers to specify the context and risk management preferences

1. Context in space: what is the population in which the investments in health research should contribute to a reduction in the burden of disease and improve health?
2. Disease, disability and death burden: what is known about the problem to be addressed by the research?
3. Context in time: what is time lag between the intervention and detectable disease reduction?
4. Stakeholders: whose values and interests should be respected when setting research investment priorities?
5. Risk management preferences: how will investment risk be managed?

Step 3

Process managers to discuss criteria for setting health research priorities

Define criteria specific to the ‘context’ for discriminating between competing ‘investment options’.  For example: i) answerability ii) attractiveness iii) novelty iv) potential for translation v) effectiveness vi) affordability vii) deliverability viii) sustainability ix) public opinion x) ethical issues xi) potential impact on disease burden xii) equity xiii) community involvement xiv) cost and feasibility xv) enterprise generation. However the longer the criteria the greater the possibility of overlap reducing their usefulness as independent criteria.

Step 4

Process managers to choose a limited set of the most useful and important criteria

Using milestones which set out the aims of any health research select from the previous list,criteriathat should discriminate between competing options (merging criteria if necessary).   

See Figure 1 from Rudan et al article. 

Step 5

Process managers to develop the means to assess the likelihood that proposed health research options will satisfy selected criteria

Invite a group of technical experts (e.g. methodologist; economist; statistician; health impact assessor) to work closely with the process managers to list, check and score research options/questions using a simple yes/no question proforma addressing each of the criteria individually. An example question regarding criterionanswerabilityis:  Is the research option/question well framed and endpoints well defined?

Step 6

Systematically list a large number of proposed health research options

Whatever the funding circumstances that research priorities are responding to list and map  i) the research domain e.g., research to assess health burden to ii) the research avenue e.g., measuring the burden to iii) the research option e.g., duration of research and iv) the research question, in order to identify the most important and specific questions to be investigated.   

Step 7

Pre-score all competing research options

Using the framework in Step 4 map the research options/questions to the milestones.

Step 8

Score health research options using the chosen set of criteria

Technical experts to score the research options/questions independently against the criteria selected by the process managers in Step 4: 0= I disagree/1=I agree/0.5 neither agree nor disagree

Step 9

Calculating intermediate scores for each health research option

The scores of the technical experts from Step 8 are calculated for each research options/questions and divided by the number of received answers. The results are assigned a value of 0% and 100% and each represents a measure of collective optimism among the technical experts of the likelihood that each option/question would satisfy each priority setting criterion in turn. The scores can now be ranked.

Step 10

Obtaining further input from stakeholders

Involve stakeholders to i) define minimal score (threshold) for each criterion that needs to be achieved in order to consider any research option a funding priority ii) allocate different weights to these scores so they are not just a simple arithmetic mean, but a weighted mean.

Step 11

Adjusting intermediate scores taking into account the values of stakeholders

Calculate weighted mean of scores of stakeholders in Step 11. Discard research options that fail to reach all the suggested thresholds.

Step 12

Calculating overall priority scores and assigning marks

Calculate mean scores of technical experts in Step 8 for all criterion in Step 4 see figure from Rudan et al article. 

Step 13

Performing an analysis of agreement between scorers

For transparency, assess level of agreement between technical experts for each research option/question using Kappa calculation.

Step 14

Linking computed research priority scores with investment decisions

All decisions that need to be made must be based on i) research priority scores (RPS) and cost of each research option/question, either already supported or proposed as an alternative ii) maximising the sum of RPS values of supported research options within a given fixed budget iii) if the sum of the RPS scores within an existing program is lower than the sum of the alternative, resources should be shifted from the existing into the new research options.

Step 15

Feedback and revision

Adjust the research investment portfolio to new contexts and aim to reduce the existing disease burden in the most cost-effective and equitable way by i) adding further research options/questions to the list ii) adding additional criteria ii) re-scoring all research options in the redefined context iv) revising thresholds and weights.