What are the key messages?

Planning

A clear and transparent plan is the key to an effective BR analysis and this can be done with an objective and transparent framework, such as PrOACT-URL or BRAT, which divides a large problem into manageable smaller criteria for assessment. While there are little differences between the frameworks and decision makers could use either of them at their preference, these frameworks should not be used rigidly and a certain degree of flexibility is needed in the planning and further stages of the evaluation. In the rimonabant case study, the issue at hand was often so complex that even by using these structured frameworks the analysis could not be done according to the plan simply because there was no data or it was impossible to implement the plan within the timeline. In this situation, it is acceptable to modify the plan but this decision and the reasons behind the modifications should be declared and documented clearly and transparently.

There is no right or wrong question to be answered or approach to be taken in a BR analysis and, consequently, all stakeholders should agree on these issues. For example, should active comparators or placebo be used as comparator? What benefits/risks should be included and what (and whose) weight should be assigned to each? These are important questions which are not always easy to answer and for which a lot of discussions are often required. Time for face-to-face discussions would be needed because we found telephone conferences might not be sufficient to disentangle any disagreement.

Evidence gathering and data preparation

Data on risk and benefit criteria should be gathered, prepared and summarised using a systematic approach, either in the form of meta-analysis in cases with multiple clinical trials of the same medication or using method of indirect comparison, e.g. mixed treatment comparison (MTC) method, in case with multiple comparators. In the rimonabant casestudy, there was not enough time to perform all analyses according to the plan and the team decided to modify the approach accordingly: patients’ perspective was not taken into account in wave 1 but it was elicited and analysed in wave 2; placebo as a comparator in wave 1 and additional active comparators in wave 2; trials in wave 1 and post-marketing data in wave 2. Also, given time limitation, effective coordination was crucial for successful and timely completion of the analysis. The team was split into different sub-groups performing different tasks in parallel.

Analysis

It is important to note that quantitative method is not always needed in a BR analysis. When there is only one single benefit and risk, a descriptive method may be sufficient. Similarly, when all benefits and risks show that one drug is more favourable than the other, a fully quantitative method is not necessary.

Decision should be supported by a decision analysis method and in the rimonabant case study ten methods were evaluated. While each method has and its strengths and weaknesses, we observed that, in a complex situation in which multiple benefits and risks should be taken into account and different stakeholders may have very different perspectives on the importance of benefits and risks, either MCDA or SMAA is the most suitable one. Both methodologies analyse available alternatives using observed data and decision maker preference.

Visualisation is an integral part of the analysis stage and different techniques may be employed. We used Hiview3 (MCDA) and JSMAA (SMAA) as visualisation tools and both were sufficiently useful. Our team also produced a different visualisation tool, by which different scores based on different data input, perspectives and assumptions were shown.

Exploration

To assess the sensitivity of the methods we, in this case study, performed MCDA analysis using different perspectives (physicians and non-physician), and observed different results for different perspectives. A different approach to account for different preferences from different approach could also be applied: a decision conference in which all stakeholders decide a specific weight for a specific benefit/risk. While this will result in a single score reflecting different perspectives, this approach is not without limitation, a single or group or stakeholders may dominate the discussion and their perspectives may mask others.

SMAA may be the answer to the challenge of accounting for different perspectives as it does not require weight in the analysis. The main weakness of the SMAA when weight is not taken into account is that the results may be less interpretable. As part of the SMAA analysis, the final average weight of each benefit/risk for based on which the score was produced can also be calculated and this average weight may not be clinically meaningful. We do not recommend the use of SMAA without weight input alone (missing weight); at the very least, experts opinions on weights should be taken into account, if not other stakeholders.

Different approaches to evaluating the robustness of the methods could be employed: data from clinical trials vs. those from post-marketing studies, different follow-up times, and different levels of severity. In the rimonabant case study we used trials data alone in wave 1 and trials data combined with post-marketing experience in wave 2.

Conclusion and dissemination

Different methods, perspectives, and assumptions may lead to different results and conclusions. At this point, after considering all alternatives and perspectives from stake holders a conclusion is made to be communicated to different audience. While the content of the communication should be tailored toward the type of intended audience (more technical, clinical or otherwise more general), one critical component for any audience is a documentation of how, when, and why decisions are made.