Stochastic Multi-Criteria Acceptability Analysis (SMAA)

SMAA is a generalisation of MCDA to include uncertainty by calculating the performance of an alternative on a criterion as a distribution rather than a single value as that in MCDA (the mean, usually). The choices of weights across criteria, in addition to fixed values of choices, can be in a range or follow some distributions. Therefore, the benefit-risk balance for an alternative is also a distribution. SMAA uses the probability that an alternative has the largest balance among all alternatives as evidence for alternative selection. The realisation of SMAA is through simulation means. A program needs to run MCDA with sample performances and weights drawn from the corresponding distributions and repeat this procedure many times (1000000, say) to summarise the results of different runs. The only existing software is JSMAA (http://smaa.fi/jsmaa/), which currently deals with linear utility only. An example of the analysis for the AECB indication is shown below. The distribution of each alternative on each criterion is derived with a Bayesian approach. More detailed descriptions and results are available in the Telithromycin Case Study Report.

Benefit-risk criteria

All four indications (AECB, TP, ABS, and CAP) share the same criteria presented in the value tree in figure 10.

Figure 10: Value tree for SMAA analysis of telithromycin versus grouped comparators in each of the indications AECB, TP, ABS and CAP

Alternatives

Telithromycin is compared to its comparators for each indication, where comparators are taken as a single alternative combining the performance of all comparators. This is done due to data availability.

Data

Data is extracted from EPAR, see Appendix-Data tables.

A Bayesian approach is adopted to derive the distributions for all alternatives on all criteria. The distributions are updated from the non-informative prior beta(1,1) with data from RTCs.

Value functions

The value function of each criterion is a monotone function taking value in range [0, 1] with 0 for the least preferred value and 1 for the most preferred value. Outside the range of [0, 1], the value function keeps as 1 and 0. The value functions are defined automatically in the JSMAA software. The value functions are linear and the most and least preferred values are based on the lowest and highest values of the alternative 95% confidence interval. The choice of direction (descending or ascending) has to be done manual. The direction of the value function for the criteria cure rate is ascending, since increase in cure rate is preferred. While the direction of the value functions for the risk criteria are all descending, since smaller number of events are preferred.

Ranking the options

In SMAA, the so-called rank acceptability index for an alternative is the probability that this alternative has the largest value function among all alternatives. The rank acceptability index of telithromycin and its comparator is calculated through simulation means from the posterior distributions of the criteria using JSMAA software. The simulation goes through the following steps:

• From posterior distribution of each alternative on each criterion, a random sample is drawn. Each alternative thus has sampled values on all criteria.

• The value function for each alternative is calculated from these samples (converts them into utility then calculates weighted sum of utility). The alternative with larger value function is the winner.

• Repeat first 2 steps n times (1000000, say) and calculate the number of times each alternative is winner.

• Rank acceptability index for each alternative is calculated by the number of times (the alternative is the winner) divided by total run number n.

Sensitivity analysis by means of missing weights

We choose missing weights for a sensitivity analysis. Missing weight means any weight assignment to the five criteria is equally likely. In probability terminology, the weight vector is uniformly distributed in the weight space. Under the missing weight setting, the rank acceptability index of telithromycin and its comparator is calculated through simulation means from the posterior distributions of the criteria and uniform distribution of weights. The procedure is as follows:

• 1.Draw a random sample from posterior distribution of each alternative on each criterion. Each alternative then has sampled values on all criteria.

• 2.Draw a random sample of weight vector from uniform distribution on weight space (5 components sum up to 1).

• 3.Calculate the value function for each alternative from these samples and mark the alternative with a larger value function as the winner. Repeat first 2 steps n times (1000000, say) and calculate the number of times each alternative is the winner.

• 4.Rank acceptability index for each alternative is approximated by the number of times (the alternative is winner) divided by total run number 900000.

Applying SMAA to the Acute Exacerbation of Chronic Bronchitis (AECB) indication

The resulting posterior distributions of all criteria for telithromycin and its comparator are listed in figure 11, and graphical representations of the distributions are given below, figure 12 .

Figure 11: Distributions of telithromycin and its comparator on all criteria

Figure 12: Distributions of telithromycin (red) and its comparator (blue) on all criteria

The value functions for each criterion for the SMAA analysis of thelithromycin versus comparators for treatment of AECB are shown in figure 13. In figure 14 the weights assigned to each criterion in the AECB analysis are shown.

Figure 13: Value function by criteria in the indication AECB

Figure 14: To the left the contributing weights for each of the five criteria. To the right the corresponding contributing weight for benefit and risk in total, in the AECB analysis.

Figure 15: Rank acceptability of telithromycin and comparator for treatment of AECB.

Given the described SMAA model the probability of the benefit-risk balance of being in favour of telithromycin is only 0.25, correspondingly there is a 0.75 probability of comparator having a more favourable benefit-risk that telithromycin for treatment of AECB, see Figure 15. Therefor based on this model it is reasonable to conclude that the benefit-risk of telithromycin is worse than that of comparator. The sensitivity analysis, where the model is run under the assumption of missing weights, does not change this conclusion. Under missing weights the probability of telithromycin ranking first is only 0.22, see figure 16.

Figure 16: Rank acceptability of telithromycin and its comparator under missing weights

Comments on SMAA in this study

Applicability and acceptability

• SMAA extends MCDA by bringing into the analysis the sampling variation and preference uncertainty, which are almost inevitable in real practice. This method is applicable to any problem where MCDA is applicable.

• The utility used in SMAA and MCDA can be very general so if a decision maker is not sure about the 'satisfaction' (utility) on criteria and would like to use the actual measurements, he may simply use the linear utility function.

• The weights can be exact, or in a range, or missing (any assignments of weights are equally likely). After the analysis, SMAA will tell what assignment of weights favour which alternative. This flexibility is extremely helpful to decision maker.

Problems in implementation

• Software for SMAA is still in development stage and choices (e.g. utility functions, choices of most preferred and least preferred values) are limited.

• The process of preference elicitation (utilities and weights) is not entirely clear. We questioned whether decision conference enough to determine these preferences.

• In simulations, criteria are assumed to be independent of each other. It is not clear how much the correlations affect the results. This may not be a consequence of SMAA, but an aspect deserving further investigation.