PrOACT URL was applied as it provided a strong detailed context to our case study. It helped to evaluate which further methodologies could, or could not be implemented based on the available data, aims and objectives.

Implementation was completed with ease. Technical demand was not taxing. However, the description of steps was a little unclear in places. Although the brevity of instructions was beneficial for fast progression, occasionally they lacked clarity.

• E.g. 1d."affected population" could refer to psoriasis patients, or patients receiving efalizumab

The anticipated time to be spent on the framework was not so much determined by the framework itself, and instead on the amount of literature necessary to read and extract data.

The steps are laid out in point form, which clearly delineate which fields are to be completed. The framework places a strong emphasis upon previously collected data and sources, ensuring the inclusion of relevant regulatory literature, data on favourable effects, and data on unfavourable effects.

Creative, doable alternatives are investigated in Step 5 Alternatives, where options to be evaluated against criteria are identified.

The method can apply to any stage of a product lifecycle, i.e. early development to post-marketing, where decision-making may occur to provide an explicit statement of objective, context, benefits and risks.

To provide a user guide that explains with greater clarity which information is being requested, and the rationale for documenting it. Some strong worked examples of how PrOACT-URL should be applied would be good.

The approach was chosen so as to provide a comparator against PrOACT-URL. This is because similarly to PrOACT-URL, it also provides a strong detailed context for the case study through the selection, inclusion and presentation of favourable and unfavourable effects.

Technical demand is necessary via statistical knowledge. For our case study, it was necessary to perform a meta-analysis of beneficial effects, and also calculate relative risks and risk differences (with confidence intervals).

The instructions are laid out within an extensive user guide with worked examples, and provide a strong and comprehensive description of methodological steps. While it offers a strong emphasis on data and sources, it also offers the additional benefit of providing an explicit audit trail, allowing for a high degree of transparency.

PhRMA BRAT softpilot software was used and relatively intuitive to learn. However, there were small glitches with the beta version of software e.g. red and green colouring of favourable and unfavourable effects was not automatic. Also, one comparator at a time can be used, although it is relatively easy to switch between the fields. Currently the software can only handle dichotomous variables which will limit its applicability in situations where continuous variables are important e.g. oncology.

Similarly to PrOACT URL, the method can apply to any stage of a product lifecycle, i.e. early development to post-marketing, where decision-making may occur to provide an explicit statement of objective, context, favourable and unfavourable effects.

Creative, doable alternatives are essential for this methodology. The method is only possible when a comparator such as placebo, background epidemiological rates, or active comparator. This is because relative risks and risk differences have to be calculated in order for visual representation on the forest plot, or inclusion in the Key Benefit-Risk Summary Table. This may present a limitation for selecting benefit-risk criteria because data must be available for a comparator.

BRR is a method when the benefit-risk analysis involves one benefit criterion and one risk criterion. In case there are multiple benefits or multiple risks, either multiple criteria are collapsed into one criterion, or a primary benefit and a primary risk are selected and ignore the others.

Few methods allow for the assessment of B-R with multiple criteria and multiple options, although this is closer to real-life situations.

No method provides clear guidance for assessing Benefit and Risk when the timeframe for outcomes is significantly different and when the outcome depends on time. In the efalizumab case study the main FE was measured at 12 weeks and the main UFE was measured beyond year 3.

Of note, the Problem was reframed from the initial mandate given to CHMP (maintain, vary, suspend or revoke the Market Authorisation) to "placebo 2004 versus efalizumab 2009"; this allowed to take into account all the observational data which had an impact on the Risk ofefalizumab after 4 years on the market; however it didn't take into account that, in 2009, the context was different with 3 new competitors with a more favourable efficacy, hence decreasing the relative Benefit of efalizumab as compared to these competitors.

The rationale for including or excluding benefit and risk criteria was to initially include every favourable and unfavourable measure from clinical trials and post-marketing surveillance for which data was publically available (e.g. EPARS, PSUR10, Scientific Discussion). This was to ensure that all of the evidence available to the regulator was accounted for in order to provide a strong context for decision-making.

Clinical judgements about the effects were available, and considered within Steps 7 to 11. MCDA is recommended.

Similarly to PrOACT URL, the rationale for including or excluding benefit and risk criteria was to initially include every benefit and risk measure taken from clinical trials and post-marketing surveillance.

Next, all effects with non numerical or missing data (e.g. no background epidemiology or placebo/comparator data) were excluded. From the remaining measures, it is possible to place filters on specific branches and change the value tree according to perception of stakeholder preferences. Major limitation for post-marketing data based on spontaneous reporting (no incidence) without background incidence rates.

PhRMA BRAT can account for criteria other than efficacy and safety,

The framework can be used to present qualitative information. However, this option is not available when using the software, and qualitative information does not appear in the forest plot or Key Benefit-Risk Summary Table.

Clinical judgements about the effects are considered in Step 5 Assess Outcome Importance, where values are assigned according to the perspective of the decision-maker. Simple methods of weighting, e.g. categories of importance, ranking, ad hoc weights, and direct assessment/point allocation are suggested. There is also the suggestion of more complex weighting via MCDA and conjoint analysis.

There were a significant number of effects without a comparator. These were excluded primarily because the necessary due diligence to identify the comparator data in the literature or other databases was not possible(time limitations). They could not be represented by the software, or resulting visualisations. It would be good to have some specified contingency plans in the event that no comparator is possible.

There could be an expansion on the software to represent additional formats of data other than numerical.

"Publicly available" information does not necessarily include the source data which would allow a precise measure for an outcome.

Meta-analyses of CTs is not always possible (different enrolled populations)

Format of Public documents (EPAR and scientific discussion) may not ease extraction of data.

Extreme heterogeneity of measures (absolute numbers, proportions with various denominators) is manageable in MCDA, which is useful in a post-marketing evaluation where measures are very heterogenic in nature and in units.

Comparators data used for the assessment are not necessarily relevant for decision making in a practical medical context (e.g. placebo instead of active treatments). This depends on the Problem and decision context chosen in the framework.

Several measures for a same medical outcome may be redundant and lead to double counting (e.g. PASI 75 and PASI 50)

Scoring and weighting are very sensitive to assessor's background, experience, and conditions of the B-R assessment (emergency or delayed)

The weight given to some outcome would deserve thorough discussion on their medical relevance (e.g. reversibility of serious risks, long term continuation of short term benefit, etc.)

Choice of outcomes for FE and UFE may be difficult (exhaustive list of outcomes or selection?, based on which medical relevance?)

When measures are missing, outcome (either FE or UFE) may not be taken into account and bias the final assessment.

The measures made on a CT population may not reflect a "real world" population (with off label use, misuse) in a post-marketing setting B-R assessment.

Unfavourable and favourable effects are defined clearly in the approach, by use of an effects table. There was no common scale, as the literature presented data in many different units, e.g. %, %/100py, etc.

It is important to note that the framework cannot be used alone although it provides a good contextual basis to use in conjunction with other benefit-risk methodology.

The results are easily interpretable; however, the responses must fit within limited table space. A fuller discussion may be possible e.g. 1d

( "Patients' and physicians' concerns"), and desirable under certain circumstances as they may significantly frame objectives. Some of the richness of discussion may be neglected in the brief description within the table.

Unfavourable and favourable effects were clearly defined, in addition to a detailed audit trail stating how effects were selected and included or excluded during specific steps. Again, there was no common scale as the regulatory documentation presented data in different units.

Similar to PrOACT URL, the framework requires use in conjunction with another benefit-risk methodology.

The final results are interpretable, but may be challenging for specific groups of stakeholders. E.g. interpreting confidence intervals, and odds ratio on a log scale. However, risk differences are presented in natural frequencies (with a denominator or 1000) which are regarded as an effective way of presenting risks to patients and the public.

The BRR is also often used together with probability simulations. In simulation approach, both ?B and ?R are taken as random variables following certain distributions. The probability that (∆R,∆B) falls over the threshold is calculated and is taken as evidence for decision making.

Clear values may be missing although potentially medically relevant for a final decision (e.g. efficiency of Risk Minimisation actions objectively measured by impact on some outcomes, or comparators post-marketing safety data).

Balancing a very rare serious effect with a relatively "modest" benefit may be challenging, even more so if the background of the serious effect is not nil.

The preference elicitation was done rather roughly and quickly despite the large scope of representatives of the Task Force; the preference elicitation was not systematically made with the chosen perspective (regulator) but as a mix of a prescriber's, patient's and regulator's perspective.

The approach can handle many different forms of data including, qualitative, quantitative, objective and subjective. However, if the software is to be used, only numeric data can be represented.

How uncertainty is accounted for is dependent upon confidence intervals, and the weighting method selected in Step 5 (Assess outcome importance).

Effects are not combined in the forest plot and Key Benefit-risk Summary Table.

BRR decision depends on the threshold choices. Threshold line implies that the tolerance of risk increase is proportional to the benefit increase. In reality the tolerance of risk increase may not be linear to the benefit increase. In this situation a threshold curve can be used instead of a straight line. The threshold elicitation can be conductedby either a decision conference or a properly designed questionnaire.

The model is not supposed to provide a decision per se, but as a help to decision-making through transparency of criteria and preference elicitation; however this transparency is not total if some criteria cannot be included in the model because unmeasurable

The approach provides insight by visually representing each unfavourable and favourable effect in isolation, while providing confidence intervals. The results are easily communicable, and highly transparent due to the documentation of an in-depth audit trail for every step. The value tree, Key Benefit-Risk Summary Table, and forest plot are easily exportable from the software into Microsoft Word and PowerPoint. Unfavourable and favourable effects are not combined into a common metric. In order to derive a final decision from the final results, the unfavourable and favourable effects must be combined.

The method provides a combined Benefit and Risk evaluation, which can subsequently be used for Sensitivity analysis, allowing for several possible decisions in situations where the balance would be sensitive to various scenarios. The MCDA model is also applicable where there are few or no objective measures but only preferences.