PROTECT Benefit-Risk

What are the lessons learned?

Lessons related to the benefit-risk methodology

Table 27 Assessment of appropriate frame for benefit-risk approaches through practical experience

Method	Comments
BRAT	The framework ensures that the process is documented, and that the discussions are focused on outcomes relevant to the BR problem. The steps of the BRAT framework should not be considered in a strictly linear fashion, but rather in a circular or parallel way. This was especially clear for the steps identify outcomes and identify data sources.
SMAA	The method does not define an frame for the benefit-risk assessment, but the method can be used within the frame of BRAT
NCB/individual benefit-harm method	The method does not define an frame for the benefit-risk assessment, but the method can be used within the frame of BRAT

Method

Comments

BRAT

The framework ensures that the process is documented, and that the discussions are focused on outcomes relevant to the BR problem.

The steps of the BRAT framework should not be considered in a strictly linear fashion, but rather in a circular or parallel way. This was especially clear for the steps identify outcomes and identify data sources.

SMAA

The method does not define an frame for the benefit-risk assessment, but the method can be used within the frame of BRAT

NCB/individual benefit-harm method

The method does not define an frame for the benefit-risk assessment, but the method can be used within the frame of BRAT

Table 28 Assessment of using meaningful reliable information for benefit-risk approaches through practical experience

Method	Comments
BRAT	The limitation of available data for older products affected three development of the value tree, which for practical reasons needs to be based on data that are available rather the allowing the three to be developed based on the application of format criteria. In clinical trials and in the meta-analysis used the effect is often measured in odds ratio or relative risk. We however found I to be more appropriate for benefit-risk assessment to use absolute risk difference. The impact of missing data on the value tree and consequent benefit-risk decision may depend on the benefit-risk model used. For example, standard BRAT tools have difficulty with missing data or data which are not biostatistically acceptable. When aggregating data from different sources it is important to be aware of issues such as different definitions of outcomes and different way of measuring certain effects, and also issue of bias when combining data. This lead to some exercise of data transformation or even criteria customization in order to have matching criteria across the sources of evidence. However, more qualitative use of the BRAT framework will allow the user to incorporate some degree of flexibility.
SMAA	For SMAA effect measures such as risk difference or risk ratio cannot be used.
NCB/individual benefit-harm method	For this implementation of NCB data on individualised patients is needed together with information treatment effect of drug versus no treatment in the form of risk rates. Again it can be a challenge to define the right balance between broadly defined criteria that makes data extraction easier, but preference weighting harder, and a narrow definition that makes data extraction harder, but preference weighting easier.

Method

Comments

BRAT

The limitation of available data for older products affected three development of the value tree, which for practical reasons needs to be based on data that are available rather the allowing the three to be developed based on the application of format criteria.

In clinical trials and in the meta-analysis used the effect is often measured in odds ratio or relative risk. We however found I to be more appropriate for benefit-risk assessment to use absolute risk difference.

The impact of missing data on the value tree and consequent benefit-risk decision may depend on the benefit-risk model used. For example, standard BRAT tools have difficulty with missing data or data which are not biostatistically acceptable. When aggregating data from different sources it is important to be aware of issues such as different definitions of outcomes and different way of measuring certain effects, and also issue of bias when combining data. This lead to some exercise of data transformation or even criteria customization in order to have matching criteria across the sources of evidence. However, more qualitative use of the BRAT framework will allow the user to incorporate some degree of flexibility.

SMAA

For SMAA effect measures such as risk difference or risk ratio cannot be used.

NCB/individual benefit-harm method

For this implementation of NCB data on individualised patients is needed together with information treatment effect of drug versus no treatment in the form of risk rates. Again it can be a challenge to define the right balance between broadly defined criteria that makes data extraction easier, but preference weighting harder, and a narrow definition that makes data extraction harder, but preference weighting easier.

Table 29 Assessment of the availability of clear values and trade-offs for benefit-risk approaches through practical experience

Method	Comments
BRAT	BRAT allows, but does not insist on preference values. Trade-off can be a challenge if criteria are not defined to be preference comparable. A broad definition makes data extraction easier, but preference weighting harder. A narrow definition makes data extraction harder, but preference weighting easier. Although the criteria were ranked order according to importance final conclusion still require a trade-off to be made by the decision maker. It might be helpful to reason in terms of health metrics, like mortality and disability rather than focusing on the cause of fatal or disabling events
SMAA	With SMAA it is possible to elicit weights on both a categorical and an ordinal scale, and to preform analysis with missing weights. It is also possible do sensitivity analysis around preferences (weights). The implementation of SMAA in the software JSMAA value function are set automatically, this can make it difficult to use weights from the literature.

Method

Comments

BRAT

BRAT allows, but does not insist on preference values.

Trade-off can be a challenge if criteria are not defined to be preference comparable. A broad definition makes data extraction easier, but preference weighting harder. A narrow definition makes data extraction harder, but preference weighting easier.

Although the criteria were ranked order according to importance final conclusion still require a trade-off to be made by the decision maker.

It might be helpful to reason in terms of health metrics, like mortality and disability rather than focusing on the cause of fatal or disabling events

SMAA

With SMAA it is possible to elicit weights on both a categorical and an ordinal scale, and to preform analysis with missing weights. It is also possible do sensitivity analysis around preferences (weights).

The implementation of SMAA in the software JSMAA value function are set automatically, this can make it difficult to use weights from the literature.

Lessons related to the visual representation of benefit-risk assessment results

The BRAT framework proposes the visuals value tree, key benefit-risk summary table and forest plot, for this case study the visuals was easily produced in word and excel. Other suitable visualisations can be easily introduced in the BRAT framework at any point in time, when necessary. For example, considering the need for interpretation by patients, pictogram was introduced in this case study to allow a one-dimensional benefit-risk assessment, although this can be an important composite such as all-cause mortality. This may help in interpretation and discussion with patients about their treatment options, but may risk oversimplifying a benefit-risk problem.

For visual representation of the SMAA analysis the stacked and divided bar chart was used to illustrate weights, and rank acceptability index. To represent data distribution plot was used, and value function was represented by line graphs.

For NCB/Individual harm-benefit risk method results are presented in tables and in scatter plots. The scatterplot takes the form of the benefit-risk plane known from other stochastic BR methods, and is a good way to represent uncertainty.