What are the key messages?

Planning

PrOACT-URL and BRAT frameworks are essentially the same, with an almost one to one mapping between the different steps in the process. The explicit iterative step in the BRAT framework is especially helpful. These frameworks alone are helpful for structuring and giving insights to the benefit-risk balance.

The definition of the Problem and of the Decision context proved to be rather difficult for efalizumab, because some of the options initially suggested to the 2009 assessors for their regulatory decision could not be addressed (lack of data which would have documented a Risk Minimisation measure). In addition, the balance to be assessed included short term Favourable Effects and long Tern Unfavourable one, as well as frequent mild effects versus rare serious irreversible one.

The construction of a model intended to make a medical benefit-risk problem explicit, requires some assumptions and some choices (which criteria to select for a Value Tree, which data to select for an Effects table, which perspective to adopt etc.). Whilst it may be questioned on how much is lost in complex models as compared to simple one, the medical significance and justification of these choices should take precedence on any other one.

Whilst initially all benefits and risks of potential relevance should be taken into account, an iterative process is recommended to highlight those with limited evidence or impact, and to then focus on those benefits and risks with potential impact on the benefit-risk balance. A clear record of which effects were included and excluded needs to be kept for transparency.

The Effects Table used in common with both methodologies proved to be a very valuable tool in order to represent the data that were required to be collected.However it didn't prevent possible sources of double counting (e.g. PASI 75 and PASI 50). Eliminating double counting can be a challenge but is critical to the validity of the analysis.

Patients preferences were not included in this analysis because of lack of time and resources, however would have brought significant added value in a psoriasis population where the social impact of the disease is very important

Evidence gathering and data preparation

Despite the rather large availability of efalizumab data in the public domain, the selection and extraction of data for subsequent representation in a proper Effects Table proved to be cumbersome and difficult. The same comment can be made for the set up of the Data Table within the BRAT framework, which suggests that the use of any model would require both appropriate biostatistical expertise and time/resources. This point would have to be taken into account if such B-R evaluation has to be made in an emergency or crisis situation.

The involvement at this stage of biostatistical, modelling and medical expertise proved to be extremely fruitful and is strongly recommended. This is because the available sources of evidence have to be discussed with various stakeholders in order to reach agreement as to which are most appropriate to be used in a benefit-risk assessment, which are redundant or double counting, how to address missing data etc.

In the BRAT model, comparative data are necessary for the presentation of results; therefore the customisation of the Value Tree may miss relevant Benefit and/or Risk criteria if such comparative data are not available. In the efalizumab case study, several ADR incidences could not be properly estimated, as data consisted of typically uncontrolled post-marketing observational series, or even spontaneous reports of rare serious and potentially under-reported unfavourable events.

MCDA, conversely, can accommodate with a large heterogeneity of data and measures. With more time and resources, a sound but indirect computation of the missing estimates could have been applied, or background incidences could have been derived from data collected from the general population (e.g. UK CPRD).

Creation of the Effects table, Summary tables and graphs, by definition, requires summarizing the benefit / risk outcome (favourable/unfavourable effects) data across multiple studies. There is a wide range of units of measure for individual study results that may be entered into the framework including absolute risk difference, relative risk, odds ratio, incidence, adjusted relative risk, time to onset, etc. For efalizumab, with the data available for each measure as described in the pertinent regulatory documents, 95% credible intervals point estimates, risk differences, and relative risks were calculated using some data transformation, some of which built in the software used (e.g. BRAT). Additional data transformation included a Bayesian mixed effects meta-analysis performed for PASI75, PGA and OLS.

In both quantitative methods (BRR, MCDA), the challenge of adequately representing a rare event in the post marketing setting in a manner that translates to the other effect measures is seen with PML. As we would expect with such small numbers, the relative risk metric is significant but with some uncertainty as evidenced by the wide confidence interval. This metric also bears the statistical assumption that there is a consistent rate of PML across time that may not be true for PML in this patient population.

A key concern with the frameworks is the elicitation of importance weights for the different clinical outcomes. It is difficult to put a preference weight on a rare serious event. Furthermore these issues are compounded with use of the hierarchical weighting process.

Generally the safety concerns that remove drugs from the market or require label change are rare but medically very serious. Breaking it down in these frameworks gives less importance to the rare serious than the current approach of balancing it all in the mind in one go.

A notable limitation is the fixed time horizon of the benefits and risks. Furthermore, stakeholders' attitudes to benefits and risks are likely to be time-dependent. For example, a patient may be prepared to accept an elevated risk of death in a few years' time as long as their disabling symptoms are minimised for the next year or two. To allow for this, a BR method that reflects the changing nature of benefits and risks over time would therefore be desirable.

Analysis

A fully populated Effects Table should be the starting point for a benefit-risk analysis. In the efalizumab case study, the same Effects table was used for all methods used, although some criteria were later discarded for the BRAT/BRR analysis.After completion of both qualitative frameworks (BRAT and PROACT-URL), the decision makers decided in both scenario that the qualitative analysis was not sufficient to enable a justifiable benefit-risk decision, and therefore agreed to follow-up with quantitative methodologies, BRR in one case, deterministic MCDA in the other one.

The weighting of benefit and risks was not done explicitly in the BRAT framework, because the final Value Tree included a small number of criteria, and the only comparator was placebo. However, even with a simplified final Forest Plot, the final decision would have relied on the implicit judgement of the decision maker and was not intuitively clear. BRAT would therefore require a quantitative method allowing for the integration of Benefit and Risk in a single score, should the decision-maker require or wish it.

In contrast, the MCDA quantitative method was conducted through a "Decision Conference", bringing together a panel of various expertises, and the final trade-off depended on the exhaustive weighting exercise performed by this panel. For such a methods requiring the building of a consensus (either on the criteria to be chosen, the measures applied to these criteria or the preference elicitation), the importance to be given to this "social process" is to be strongly emphasised and requires a face to face whole day meeting if not even more. In this context, the preference elicitation exercise should be well structured in order to obtain best results without biasing any option.

In this case study, it appears clearly that the perspective chosen (regulator, patient, Company) is of utmost importance as it impacts all of the subjective data used in the model, and to some extend the choice of the objective (medical) one.

Overall it seems that the BRAT model better fits the needs of pre-marketing B-R assessment based on RCTs data (for which it has been initially developed), and MCDA for emerging post-marketing issues triggered by rare serious ADRs having an impact on the B-R balance of a drug.The use of appropriate visualisations (e.g. those from Hiwiew3) proved to be very helpful, speed up and eased the consensus in the panel discussion using the MCDA model.

Exploration

The engagement of appropriate statistical, modelling and clinical expertise proved again to be critical at this stage in the efalizumab case study.

Overall, although based on the same data set, the application of 2 frameworks used with 2 different quantitative methods lead to divergent results within the applied models, as well as with the historical similar decision made in 2009 by the CHMP. This may illustrate the difference between a compensatory model (MCDA) and a non-compensatory one as BRR. This may also illustrate the degree of variability in all objective and subjective data used in these models, and therefore the major added value of sensitivity analysis. But the choice of the context for the modelling exercise didn't take into account some evolving competitive environment, which probably played a role in the historical decision made in 2009.

The overall result of the MCDA modelling showed that the benefit-risk of efalizumab is substantially better than that of the placebo, even taking into account the 4 PML cases. This conclusion is robust to substantial differences of opinion about the individual weights on the criteria. Indeed, orders of magnitude increases would be required for the unfavourable effects, except for PML, to tip the balance. Only when more weight is given to 5 cases of PML compared to 60% of patients achieving a 75% reduction in baseline PASI would the model favour the placebo over efalizumab. Overall, the MCDA model was relatively stable to significant changes in the assumptions made during the Decision Conference.

Although less sophisticated, the BRR method used in this case study also allowed for some exploration of the variability of the final result, depending on the change in some assumptions made on the exposed population. This is a criteria which always triggers large uncertainties in a post-marketing observational non-controlled setting.

The use of sensitivity analysis is key as it shows which variables are driving the assessment, and focuses the discussion onto these specific weights and incidence measures. This demonstrates the value of information of these variables and may motivate more work to be done to better estimate them.

From a visualisation point of view, the Hiview3 software provided some interesting and easily understandable ways of communicating the results of the sensitivity analysis.This sensitivity analysis can possibly address some methodological issues related to the time difference between the benefit criteria (12 weeks) and the major risk criteria (beyond 2 years), but this was not tested formally in this case study.

Conclusion and dissemination

In this efalizumab example, both qualitative frameworks (BRAT and PrOACT-URL) finally required an extension to quantitative methods able to integrate benefits and risks in a single value in order to facilitate the final decision to be made.The overall conclusions were not consistent across the methods used, nor with the historical decision made in 2009. This is due to the variability in the objective and subjective data, to differences in the underpinning models, but also to the Decision Context which was defined differently in this case study compared to the one offered to Decision Makers in 2009.This exemplifies clearly the need for methods allowing for a sensitivity analysis.

The visual methods used in this case study used mainly standard displays including table and graphs, but no sophisticated tools were tested. The most advanced visual displays were those built-in the Hiwiew3 software used for the MCDA modelling. They proved to be useful for the efalizumab team, but might be more obscure for non-specialists of the MCDA method and of this software.