What are the key messages?
Planning
PrOACT-URL and BRAT frameworks are essentially the same, with an almost one to one mapping between the different steps in the process. The explicit iterative step in the BRAT framework is especially helpful. These frameworks alone are helpful for structuring and giving insights to the benefit-risk balance. The weighted Net Clinical Benefit approach is a special case of MCDA assuming that all benefit/risk criteria can be quantified as rates/proportions and linear value functions are used. In simple cases, the weighted Net Clinical Benefit method is arguable easier to explain and to understand than MCDA. Clear and concise definitions on the decision problem and criteria must be established from the start, and must be used consistently throughout the process.
Evidence gathering and data preparation
The BRAT framework emphasizes the value tree build-up, data selection, data preparation as well as tabular and graphical data presentation. However, only a separate assessment of each benefit and risk as well as first visual comparisons is possible. The framework can increase transparency, predictability and consistency of benefit-risk assessments. This is an easy-to-implement tool to structure simple decision problems on daily basis.
A key concern with the frameworks is the elicitation of importance weights for the different clinical outcomes. It is difficult to put a preference weight on a rare serious event. Furthermore these issues are compounded with use of the hierarchical weighting process.
Generally the safety concerns that remove drugs from the market or require label change are rare but medically very serious. Breaking it down in these frameworks gives less importance to the rare serious than the current approach of balancing it all in the mind in one go.
In the natalizumab case, regulators and patients had initially very different perspectives. Patient value elicitation should always be part of modelling complex benefit-risk decisions.
It is also important to include a qualified statistician who knows the data and a qualified clinician who understands the disease area and the clinical implications of the findings to ensure any issue with evidence can be satisfactorily resolved so that consensus can be reached.
A notable limitation is the fixed time horizon of the benefits and risks. We are using a two-year time horizon since the available data on benefits and risks relates to a two-year period. Furthermore, stakeholders’ attitudes to benefits and risks are likely to be time-dependent. For example, a patient may be prepared to accept an elevated risk of death in a few years’ time as long as their disabling symptoms are minimised for the next year or two. To allow for this, a BR method that reflects the changing nature of benefits and risks over time would therefore be desirable.
Analysis
The use of a probabilistic model can further illuminate comparisons between treatments based on their benefit-risk scores – in particular, it clearly shows how robust the comparisons are to uncertainty in the data. But a probabilistic model is difficult to set up and may only add value if the decision-maker’s attitude to uncertainty is well understood. We expect that, where a decision is to be made between competing treatments, most decision-makers will continue to choose whichever treatment has the best benefit-risk score based on a deterministic analysis. More careful considerations and strategy to deal with cognitive biases due to different magnitudes of events are needed.
Exploration
The use of sensitivity analysis is key as it shows which variables are driving the assessment, and focuses the discussion onto these specific weights and incidence measures. This demonstrates the value of information of these variables and may motivate more work to be done to better estimate them. Incorporating uncertainty into visualisations can be difficult but should not be ignored; visualisations that are too simple may present the unintended message that there is no uncertainty.
Conclusion and dissemination
Visualisations are a key part of understanding probability distributions. From the point of view of the case study team, the visualisations were generally clear and appropriate, and the interactive visuals were particularly well received, but there should be further testing on external audiences. The choice of visualisation depends heavily on the target audience, the nature of the distributions being compared, and the relative importance of clinical and statistical significance.
The overall benefit-risk score for each treatment is a function of clinical parameters and preference parameters. Both kinds of estimate may not accurately reflect the true average parameter value in the decision population, for reasons including sampling error, heterogeneity between the study population and the decision population, study design factors (e.g. biased parameter estimates, confounding and reverse causality, preference elicitation designs) and approximations made in the data extraction.
Benefit-risk analysis employed by either of these methods is not an automated process where an algorithm is applied and an answer is generated. It is a framework to give clarity to decision makers. Expert opinion is central and many aspects are subjective. Benefit-risk analysis gives structure to enable discussion, ensures a wide breath of outcomes are considered and helps decision makes focus on the salient issues.