Conclusion and Dissemination

The final stage in the benefit-risk assessment process is the point at which a conclusion is reached and the results and consensus are communicated to a wider audience. This last stage makes it explicit that the findings of the benefit-risk assessment have logically led to a conclusion that could influence future actions. It emphasises the need for a transparent audit trail of the whole assessment process from the Planning stage to the Exploration stage. In other words, this last stage of the process brings everything together and sets the stage for action to be taken.

By this point, it is easy for those who are closely involved in the benefit-risk assessment process to have become so absorbed in the details of the analysis that they lose sight of the bigger picture " as the saying goes, they may find themselves' unable to see the woods for the trees." It is helpful at this point to take a step back and consider, with a fresh pair of eyes, what the overall aims of the assessment are and whether the approaches used to achieve those aims have been adequately documented. This may involve asking questions such as those set out below.

• What is the benefit-risk assessment conclusion, on what was the conclusion based, and how is it documented?

• What are the critical limitations which apply to the conclusion?

• How is the conclusion communicated and to whom?

• Has an audit trail been provided so that the benefit-risk assessment can be understood and reproduced by others?

• What would be the trigger to re-evaluate this benefit-risk conclusion (i.e., when does the process get re-initiated)?

• What are our recommendations regarding the use of visualisations at the Conclusion and Dissemination stage?

What is the benefit-risk assessment conclusion, on what was the conclusion based, and how is it documented?

The decision maker should be able to answer the following questions:

• What question(s) was the benefit-risk assessment aimed at addressing?

• What answer(s) were found?

• Is/are the answer(s) highly sensitive to the treatment effects data, the choice of analysis method, or the preference data?

• What is the supporting information on which the conclusion is based?

What are the critical limitations which apply to the conclusion?

It is vital to document any known limitations of the benefit-risk assessment. Two limitations that were encountered in several of the case studies are discussed below.

Lack of data was frequently cited as a limitation by the case study teams, who were largely restricted to using publicly available information. This meant that data could not be found on certain benefits and risks of interest for all comparators (a problem in both the Wave 1 efalizumab case study and the Wave 1 natalizumab case study), and the distribution of effects data within a population of patients could not be easily determined (with both the Wave 2 natalizumab case study and the Wave 2 rosiglitazone case study having to make "the best use possible of the statistical summaries to infer the population distributions"). The Wave 2 warfarin case study was unique in that patient-level data was used; however, there were issues with the quality of the data: "Discrimination between haemorrhagic stroke and ischaemic stroke in CPRD is limited as frequently non-specific codes are used."

Another frequently cited limitation concerned the generalisability of the results to real-world populations. For example, the Wave 1 efalizumab case study noted that "measures made on a clinical trial population may not reflecta?|off label use, misusea?|in a post-marketing setting." Several case study teams questioned whether the preferences they had elicited could be replicated by real patients. The Wave 1 rimonabant case study report makes clear that the results are dependent on "the explicit weighting and utility function set by selected decision makers. This raises the question if the results can be applied in the wider population." Given more resources, it was generally felt that this problem could be overcome to some extent. The Wave 2 rosiglitazone case study team were aware that "the value functions and weights elicited for the MCDA model were based on the individual preferences of only a few medical experts in the PROTECT WP5 team" but pointed out that it would be possible "to involve more medical experts in the weighting process to generate a more representative set of weights and thus improve the model."

How is the conclusion communicated and to whom?

Communication of a conclusion should never focus simply on the analysis results. It is equally important to present the evidence and assumptions on which a conclusion is based. This encourages transparent and robust benefit-risk assessments.

A particular difficulty in the field of benefit-risk assessment is that there are several conflicting versions of the terminology used to describe the concepts and methods involved. A particularly salient example is the word "risk" itself. In asking 50 European assessors what they or their agency meant by 'risk,' the EMA's Benefit-Risk Project team collected over 50 different words or phrases, several of which were in conflict, e.g., 'tolerance of a drug compared to serious side effects,' 'severity of side effects,' 'frequency of side effects' (European Medicines Agency, 2009). To avoid confusion and misinterpretations, clear unambiguous language that avoids the use of technical jargon should be used wherever possible. Other examples include the use of "value" versus "utility," and the many alternative terms for benefit-risk assessment itself (e.g., benefit-harm).

The choice of language and visualisations to use when communicating benefit-risk decisions should reflect the level of technical knowledge of the intended audience. The appropriate channels for communication may also vary. As yet, there is little consensus regarding how best to communicate benefit-risk models. PROTECT WP5's Visualisation Reviews (Mt-Isa et al., 2013a; Mt-Isa et al., 2013b) have identified various graphics and classified them based on their complexity and ease of interpretation, but these have not yet been tested out on real-life audiences.

One role for a benefit-risk model is to serve as a communication channel amongst different audiences. For example, a committee of assessors in a regulatory agency might create a model that could be used by the agencya??s approvals committee to test different perspectives about the clinical relevance of a new drug's effects before making a final decision. Post-marketing, that model could be used by a regulator's pharmacovigilance committee to see if new information tips the benefit-risk balance. Within a pharmaceutical company, a benefit-risk model might begin its life at Phase II and be elaborated as the product profile changes in response to scientific findings. In both of these examples, the model would be used by different people at different times in the drug's life cycle.

Has an audit trail been provided so that the benefit-risk assessment can be understood and reproduced by others?

An audit trail of the benefit-risk assessment process is clearly desirable for regulators, who are publicly accountable for approving medicines and must communicate their decisions to the public and to drug developers. Companies making submissions to regulators also have a clear incentive to ensure that benefit-risk information is transparently presented; and we would argue that even benefit-risk assessments carried out purely for internal purposes should be documented with a clear audit trail. This may help to ensure consistency of related decisions, to facilitate revisiting assessments in the light of new information, and in the event that bad decisions are made, to unpick what has gone wrong.

Use of formal methodologies like those reviewed by PROTECT WP5 " especially the benefit-risk assessment frameworks " naturally helps to create an audit trail.

Continuous use of a model throughout its life would require periodic documentation as specific milestones are reached so that it can be understood and used effectively following each milestone. Presumably the model would remain in the proprietary ownership of the pharmaceutical company, but once the drug is approved, it could be desirable to place the model in the public domain, either by the company or by the regulator, so that the reasons for approving "or disapproving" the drug are clearly communicated to the public.

What would be the trigger to re-evaluate this benefit-risk conclusion (i.e., when does the process get re-initiated) and what aspects of the assessment would need to be revisited?

During development of a medicinal product, the body of evidence regarding its effects and relevant indications changes rapidly. The benefit-risk balance may frequently need to be re-evaluated in light of any new information.

Updates may also be required from time to time after a product has been brought to market. For example, regulatory authorities typically require marketing authorisation holders (MAH) to submit periodic re-assessments of the benefit-risk balance (such as PBRERs).

In addition to periodic updates, there may be specific events in the post-marketing setting that trigger a repeat of a benefit-risk assessment, including the arrival of new comparators onto the market; safety signals or other emerging data; and overcoming the identified decision limitations.

Arrival of new comparators onto the market

The benefit-risk balance of a medicine is not considered in isolation but relative to alternative treatment options, i.e., the comparators. As time passes and new products are brought to market, it may be appropriate to expand the list of comparators to include any new drugs that are relevant to the indication. These may have side effects that were not included in previous benefit-risk assessments; in which case, a re-definition of the value tree may be necessary. Techniques such as ITC/MTC may be employed to aggregate evidence on the various treatments.

Safety signals or other emerging data

From time to time in the post-marketing setting, new evidence regarding a treatment's effects may become available. In some cases, the new evidence may simply provide revised estimates of the benefit and risk measures that have been included in a benefit-risk assessment; and in such cases, updating the assessment to reflect the new evidence should be fairly straightforward.

A more challenging scenario arises when previously unknown benefits or risks are revealed. For example, this may arise from long-term observational follow-up studies or spontaneous safety reports. Incorporating data from such sources into a quantitative benefit-risk assessment may not be straightforward. Spontaneous reports in particular may be perceived as unreliable, with potential bias in the reporting process and uncertainty regarding the extent of exposure to the treatment.

Revisiting a benefit-risk assessment in order to include additional outcomes may require some remodelling of the value tree for the reasons discussed earlier in this report. Any preference weights used in a quantitative analysis may no longer be valid in the presence of new outcomes and, therefore, will need to be re-elicited.

Overcoming the identified decision limitations

If any critical limitations of a benefit-risk assessment (e.g., a lack of relevant data) can subsequently be overcome, then clearly a more robust decision will be achieved by updating the benefit-risk assessment to reflect the latest information.

If there are any known limitations to the current benefit-risk assessment that could be expected to be overcome in the near future, such as lack of data on a particular risk, then these should be highlighted in the conclusion documentation as priorities for future study.

What are our recommendations regarding the use of visualisations at the Conclusion and Dissemination stage?

The PROTECT project found that currently the use of graphics and visual aids for communicating the benefit-risk of drugs is very limited, and what they did find was mainly in scientific journals. Some patient leaflets illustrate procedures, such as placing a patch or injecting a substance, but only words are used to communicate benefit-risk. There are currently many initiatives in the field of risk visualisation, but these are neither specifically for visualising benefit-risk balance or trade-off, nor specifically linked to the benefit-risk assessment approaches (Cammax Limited, 2011; Gapminder, 2011; IBM, 2011; Spiegelhalter, 2010). Quantitative benefit-risk models provide many opportunities for displaying results in easily-understood graphic form, so we present here the major recommendations from our review displays that are relevant to benefit-risk.

The aspects to be presented depend on the audience's level of technical knowledge, as well as being dependent on their interests. Therefore, the first step in generating visuals is to determine the intended audience. It is difficult to say which stakeholders should be presented with which information, but a survey might be able to give some information on the average visual preferences.

It is recommended to consider Wickens' Principles of Display Design, which are principles for human perception and information processing, to aid a better design of visual displays for human use. The Wickens principles provide a set of 13 principles, which include principles to promote perception and attention, and principles based on that the individual interprets visuals based on existing experience and knowledge of a visual or the world. The principles are set out in full in the Stage 2 Visual Review (Mt-Isa et al., 2013b). The Wickens' principles are concretised in GlaxoSmithKline (GSK) Graphics Principles, and it is recommended to use these as guidelines when designing graphs to communicate numerical information. The GSK Graphics Principles can be found online (https://ctspedia.org/do/view/CTSpedia/BestPractices) and are also set out in the Stage 2 Visual Review (Mt-Isa et al., 2013b).

If possible, the use of interactive displays is also recommended.