The BRAT Framework and pilot software provide guidance and a value tree tool for users to select and define benefits and risks.

The framework and software are intended to be accessible to project teams that want to frame the benefit / risk assessment on their own or preferably at first with the help of a facilitator. Implementation only requires access to Excel and, during the pilot phase, registration with PhRMA to gain access to the software application and guidance document. The software is user-friendly, with minimal technical demand. Time demands would depend upon the complexity of the problem and availability of data. In this case, the information was already in the EPAR, and framing was done for all methods at once. In a more typical prospective analysis, the framing is often helped by an analyst working with the team to define benefits and risks for the indication of interest.

BRAT can be used at any stage of the product lifecycle. The software tool is designed to be a data archive if desired, and so can be used early in development and updated throughout the product lifecycle. The pilot software is designed to look at one treatment v. one comparator (active or placebo) at a time for dichotomous variables.

The software will be turned over to a vendor after the pilot stage, with improvements to be made based on user feedback.

For this case study, the framework and tool were applied easily. The software does not calculate the results it displays, so the calculation of combined efficacy and the confidence intervals needed for efficacy and safety variables had to be done outside the tool. Given the complexities of combining data from different studies, this is not recommended to become an automated function.

MCDA is a natural progression of PrOACT, it divides difficult problems into manageable smaller criteria so to compare between alternatives. MCDA methodology also allows addition of information easily, when more data become available.

MCDA methodology framework is broad and we need to clear in our mind that there are many adaptions to this framework, and we have only tested one adaption using HiView3.

Many critiques in this report are based on appraisal of the programme.

Software we used in this exercise, HiView3, was easy to use and made the MCDA analysis much efficient. Comparing between alternatives is apparent in both visual and number output with current programme.

The HiView3 MCDA model requires criteria values and weights to be precisely known upfront. A detailed decision conference between stake holders is needed to discuss an agreed criteria function in each criteria and precise weight between each criterion; One would imagine it is often difficult and unrealistic to obtain an exact weighting score in real life situation, particularly when number of criteria for consideration is large. Besides, the decision maker's knowledge regarding to the question might not be sufficient to make an objective judgement in weighting. The result weighting and utility scale would be bias towards the stakeholder's own experience or possibility influenced by other participants.

1] Stake holders selected for the decision conference needed to be wide enough to accommodate views from different parties - regulators, physicians & patients

2] Information on criteria would need to be available for review prior to meeting.

3] Question regarding to criteria should be addressed by individuals independent to the decision conference.

This case study assumes regulator's perspective to review a drug decision with the data available 5 year after market approval.

SMAA is chosen because it has the flexibility in number and form of criteria; it allows the discrepancies and variations in different datasets taken into analysis; it also allows stakeholders to have different opinions on trade-off between criteria. All of these features make SMAA a choice in dealing with real world medical decision problems where uncertainty is almost inevitable.

SMAA can be realised by software 'jsmaa'.

PSM is applicable to all decision problems to which benefit-risk ratio (BRR) is calculated. It provides the necessary visualization and representation of benefit and risk information and incorporates uncertainty into analysis. The PSM ¨C including the MCS allows uncertainties in the input values, characterised by probability distributions, to be propagated through the network of evidence to the end results. In health care settings, the decision maker is faced with having to make value- or preference-based treatment decisions under uncertainty. Both the risk-benefit joint distribution plot and acceptability curves helps decision maker to trade-off risk and benefit

The approach is based soundly on probability theory, is comprehensive in the scope of inputs, provides readily interpretable results, and can be implemented using existing software, such as @Risk or Crystal Ball sitting in Excel, or Analytica, SAS, R or SPLUS. Its outputs are clear, graphical and easy to understand. Approach can display two-dimensional probability distributions for the differences between a new drug and a placebo or a comparator for either measures of favourable or unfavourable effects, or the two combined.

For the approach PSM using BRR as metric indices, benefit and risk criteria are limited to one benefit and one risk criteria. In the analysis of Ketek indication ABS this risk criteria as defined by overall incidence of adverse event of special interest (AESI), which were comprised by the sum of hepatic AEs, Cardiac AEs, Syncope and Visual AEs.

The framework of the Sarac Benefit-Risk Method if easy to follow and clearly described in the material available on e-room PROTECT WP5. However, the process of scoring criteria is not straightforward for layman ,and there exists no finished software for the methods. Yet, with some statistical and computational knowledge scoring of criteria can be done using mathematical programs, such as MATLAB with statistics toolbox. For the Ketek analysis, scoring was done using MATLAB and since all parameters were discrete, only one scoring method had to be implemented to produce the different scoring charts.

All benefit and risk criteria listed in the EPAR were used in the MCDA model.

Data source were reliable. However, transformation of data to utility score could be bias. As well as the final average weighted score. Both utility function and weighting were set based on stake holder's preference after decision conference meeting - which itself undoubtedly varies.

Besides, this HiView3 MCDA software only allows one value for every alternative in each criterion. However, medical data are often in range of mean with confidence interval so to account for the uncertainties and random error with the statistical estimates. The current software would not able to take the uncertainty with data into account, this is crucial in making medical judgements especially in rare events where there is a intrinsically considerable degree of uncertainty with the statistics estimates.

The rational for including or excluding criteria are not clearly defined. For example the total AEs and AEs by body system, if all of them are taken as risk criteria, each AE is actually counted twice. Should the weights be adjusted on criteria which are overlapping? In the current SMAA analysis only AEs by body system is included.

All data available to decision maker can be taken into analysis by SMAA, high quality or low quality, clinical trial data or observational data. Since SMAA describe performances by distributions, all the data can contribute to distribution estimation (at least in principle) in an accumulative way.

As in MCDA, SMAA assigns utility and weight for each criterion. These need both clinical judgements and stakeholders' opinions. SMAA relax the requirement for weights to be exact. The weights can be in a range or totally missing while SMAA still provides answers to help decision makers.

Efficacy data were, as mentioned above, pooled directly for the Sarac Benefit-Risk Method approach. This was done because the information material on Sarac¡¯s Benefit-Risk Assessment Methodology, as of now, only include a description of scoring discrete variables using trial population size, N, and number of events, x, as input.

With the Sarac Benefit-Risk method the clinical relevance of a difference in performance for drug versus comparator for each criterion is defined through the scoring method. This is done through the threshold of 2/3 of the patients performing better for drug versus comparator or vice versa. The threshold of 2/3 can be changed to another level, though this has to be done upfront. However the scoring method does not take in to account the magnitude of an effect of a drug on a criterion.

MCDA method allows a transparent judgement of value between risk and benefit.

By transforming data into utility score using criteria function, this produce a common scale to allow comparison between risk and benefit

Final results are easily interpretable in both graphical and numerical form using the HiView3 software.

The Sarac's Benefit-Risk Assessment Methodology approach does not make judgment of values explicit, Sarac's Benefit-Risk Assessment Methodology the drugs are scored in a objective manner by the threshold for which a drug performs better than the other drug on a criterion. The decisions makers has to make judgment about the direction of a criteria, e.g. if more events are good or bad or if an increase of a variable is good or bad. But to which extent an increase/decrease will result in one option being judged better on a criterion or not, is based on an objective scoring

Favourable and unfavourable effects are clearly defined in the Sarac Benefit-Risk Method approach. This is an attempt to define clinical significance and an opportunity to investigate and discuss the clinical relevance of data. With the Sarac's Benefit-Risk Assessment approach it is determined if there is a tendency towards the drug performing better or worse than the comparator for each benefit and risk criteria. This is done through the scoring of criteria, and if the drug is judged to have a favourable effect compared to the comparator this means that 2/3 of the patients in the drug group perform better than the comparator for the specific criteria, the drug I found to do worse 2/3 of the comparator patients perform better than the drug patients. If none of either of these thresholds can be fulfilled then the method cannot say if there is a difference of the effect for between drug and comparator on the criteria in question. The value of 2/3 can be changed if deemed appropriate. Although it is clearly defined if there is a favourable or unfavourable effect of the drug compared to comparator the method does no tell how big this effect is.

The method does not give a direct measure which trade-off of benefits and risks. The result of the analysis is presented visually and gives a clear overview on whether the drug has favourable or unfavourable effect compared to the comparator for each criterion. The method require that weights are assigned to each criterion for the process of scoring is begun, this helps when the final conclusion on the drugs benefit risk profile has to be made, since it has been judged where an effect favourable or unfavourable is considered more of less important compared to other criteria.

Each criterion can only hold one value, however in any form.

However, uncertainties within the data range are not addressed as this HiView3 MCDA software only allows one value for each criterion. Whereas medical data are not distinct.

We used random effect meta-analysis to combine results from different studies listed in our data source. This allows an objective approach to pooled data between studies before using the result in the MCDA model.

As discussed earlier, results of the HiView3 MCDA is dependent on precise weight information collected from stakeholders. And these often change dependent on the stakeholder involved and possibility not replicable with different stakeholder groups. As a result, conclusion from each analysis is conditional to the precise weighting decided by the stake holding group. It is arguable if the result is applicable to the wider public.

SMAA is an extension of MCDA. So it fits for any number and any forms of criteria as MCDA does. SMAA includes uncertainty in performances and uncertainty in choices of weights into consideration. The major concerns behind this extension are

(i) the performance of an alternative may change each time new data arrives, so it is suitable to view the performance as a distribution rather than a fixed value.

(ii) the choices of weights are hardly agreed exactly in practices. A range for weights or a distribution for weight vector is more realistic in real situations.

Under those uncertaintys, SMAA considers the chance (probability) that an alternative is the best one for each alternative as the evidence. SMAA is realised by simulation means.

MCDA and SMAA use additive utility function as value function, which implies preference independence and incurs criticism.

PSM is applicable to all decision problems to which benefit-risk ratio (BRR) is calculated. It provides the necessary visualization and representation of benefit and risk information and incorporates uncertainty into analysis. The Both the risk-benefit joint distribution plot and acceptability curves helps decision maker to trade-off risk and benefit

The approach is based soundly on probability theory, is comprehensive in the scope of inputs, provides readily interpretable results, and can be implemented using existing software, such as @Risk or Crystal Ball sitting in Excel, or Analytica, SAS, R or SPLUS. Its outputs are clear, graphical and easy to understand. The approach can display two-dimensional probability distributions for the differences between a new drug and a placebo or a comparator for either measures of favourable or unfavourable effects, or the two combined.

Variation in data is considered through the scoring method. The Sarac's Benefit-Risk Assessment Methodology does not offer any additional objective data driven method to deal with uncertainty for discrete data, but offer the possibility of assigning interval scores when subjective judge relevant. However, through re-sampling, it possible to evaluate uncertainty in a data driven way, which can be reflected through an interval score.

In the Sarac's Benefit-Risk Assessment Methodology approach benefits and risks are not integrated, all benefit and risk criteria a weighted prior to scoring. The criteria are weighted into one of three categories; most important criteria, medium important criteria and low importance criteria. All criteria have to be mutual preference independent.