We selected the case studies based on benefit-risk decisions that were of particular interest for exploring or testing systematic approaches to benefit-risk and (at least for the Wave 1 case studies) for which sufficient data were judged to be available. However, this finite number of case studies cannot explore all possibilities, and new challenges beyond those seen to date will inevitably arise. Therefore, we list below various challenges that we anticipate assessors may have to address, and have placed these under two sub-headings corresponding to the pre- and post-marketing stages of the product life cycle. We hope that the discussion below may provide some useful guidance. In general, the more that decision makers face difficulties such as those below, the more difficult it will be to derive reliable numerical estimates of the favourable and unfavourable effects of treatment. The appropriate response to this depends upon the importance of the outcomes in question and the extent to which quantitative modelling is employed. Assessors using quantitative decision models should take particular care to consider the impact of uncertainty of the treatment effects. Those employing a more narrative/descriptive approach to benefit-risk should ensure that the sources of evidence, and any associated uncertainty, are clearly communicated.

Limited Evidence

Before a treatment is marketed, many factors influencing its benefit-risk balance may remain unknown. Early clinical studies may be too small in size, short in duration, or narrow in focus to capture data on all of the key benefits and risks that may eventually occur with chronic usage in the wider patient population. An example of this is the use of metoclopramide for gastrointestinal disorders, nausea, and/or vomiting, which is associated with the emergence of tardive dyskinesia as an adverse reaction in the long term (EMA, 2013c).

This does not mean that there is no value in carrying out a benefit-risk assessment during the early stages of the product life cycle. Benefit-risk assessment is always a dynamic process to be undertaken throughout the use of a treatment, rather than as a single determination. A decision regarding the benefit-risk balance should be based on the best evidence available at that point in time and may later change as new evidence becomes available.

Relevance of outcome measures

The outcomes recorded in clinical trials may not have been measured in a way that is optimal for the purpose of a benefit-risk assessment. Surrogate measures may have been used in place of the long-term outcome measures of interest. In most cases, surrogate markers provide a good proxy but can result in less precision on the relation between the intervention and the primary outcome. In cases where hard endpoints are needed, the evidence generated from such trials may not be sufficient. There are also concerns over varying definitions and quality of measurements for different outcomes collected in clinical trials. Where such concerns exist regarding the relevance of outcome measures, these should be documented carefully and fully so that the decision makers using this evidence can make informed decisions on the relevance of certain outcome measures. This documentation should also be revised and addressed in future periodic assessments.

Long term follow-up data

Where trials have followed up participants beyond the original trial period, they can provide a useful source of data on a treatment's long term effects. However, analysts and reviewers of a benefit-risk analysis will recognise that the controlled nature of a clinical trial breaks down at the end of the original study period, and data from that point on is more akin to that from an observational study. Assuming the trial had a positive result, the control subjects will often have been switched to the active treatment after the end date, meaning that long-term control data may not be readily available. Such extensions to comparative clinical trials may also encounter more issues with compliance and confounders. As with any analysis, the sources and degrees of uncertainty, and their likely impact should be clearly documented.

New evidence of efficacy and safety

If a company becomes aware of new efficacy or safety evidence relating to an approved indication for one of its products, it is obliged to consider this new information in terms of its impact on the benefit-risk balance (ICH, 2012), documenting this assessment, e.g., in the periodic benefit-risk evaluation reports (PBRER) to the regulatory authorities, and applying the appropriate risk minimisation measures such as labeling, as needed.

Where the new evidence has come from a study that is not sponsored by the company and does not fall under the EMA's clinical data transparency regime, only the published summary results may be available. Integrating this information into a benefit-risk assessment based mainly on the company's own data may present challenges. Meta-analytical techniques such as ITC/MTC may be required in order to allow for factors such as heterogeneity between study populations. Bayesian modelling, which allows the distributions of summary data to be incorporated as prior information, may also be a theoretically sound and viable option.

Where the new evidence specifically relates to a different patient group from that for which the product was originally licensed, a separate benefit-risk assessment may be required for these patients. This is not simply a case of changing the data in the existing assessment; for different patient groups, the decision context will vary and so the entire assessment should be revisited from the bottom up. Assessors will need to consider whether it is appropriate to assume that the efficacy and safety profile is similar between the different groups.

Observational / surveillance data

Epidemiological studies, registry reports, and spontaneous reports may provide important data on emerging risks. As is true of clinical trials, there is potential bias associated with each source. The source of data can also be considered in terms of quality of evidence, e.g., CDC hierarchical system. Aggregating the evidence with that observed in clinical trials may also be problematic. Statistical methods may exist to deal with these issues, but this remains a relatively specialised field and not all assessors may have the resources for such approaches (or consider it appropriate to use such complex techniques for the decision at hand). Observational data will not contribute to the same extent in reducing uncertainty on benefit-risk balance as compared to randomised controlled trials. As has been noted elsewhere in this document, it is recommended that the complexity of the assessment be sufficient to answer the question of benefit-risk balance, with any limitations and sources of uncertainty appropriately noted, along with their potential impact.

Well-established products

Products with a long history on the market have the advantage of cumulative data. Information collected over time provides some of the answers to the questions described above regarding the impact of longer-term treatment. As noted, there are challenges regarding combining the data if a single quantitative database is needed. But a sufficiently flexible benefit-risk framework that accommodates multiple data sources offers the potential for a multi-faceted view of many aspects of the treatment and its favorable and unfavorable effects. As described for other treatments, the benefit-risk assessment of mature products should begin with robust framing to understand the questions that need to be addressed, followed by a consideration of the data sources that are appropriate to answer the questions, and the implications of including and excluding, or weighting other sources of data, e.g., those from other indications. As in other contexts, one of the advantages of using a benefit-risk framework is the transparency afforded around the construction of the analysis and the reporting of the results.

Another challenge with mature products is missing information. The regulatory paradigm was likely not as robust as it is today, resulting in less comprehensive documentation of evidence at time of approval. In addition, there are practicalities, such as the loss of archived information, that impact the ability to introduce data into a benefit-risk assessment. Some sort of sensitivity analysis may deal with this issue, but the best practices for this scenario are still evolving.