What are the lessons learned?
Assessment of appropriate frame for benefitrisk approaches through practical experience
Comments  Proposed improvements and/or extensions  

BRAT  The BRAT Framework and pilot software provide guidance and a value tree tool for users to select and define benefits and risks. The framework and software are intended to be accessible to project teams that want to frame the benefit / risk assessment on their own or preferably at first with the help of a facilitator. Implementation only requires access to Excel and, during the pilot phase, registration with PhRMA to gain access to the software application and guidance document. The software is userfriendly, with minimal technical demand. Time demands would depend upon the complexity of the problem and availability of data. In this case, the information was already in the EPAR, and framing was done for all methods at once. In a more typical prospective analysis, the framing is often helped by an analyst working with the team to define benefits and risks for the indication of interest. BRAT can be used at any stage of the product lifecycle. The software tool is designed to be a data archive if desired, and so can be used early in development and updated throughout the product lifecycle. The pilot software is designed to look at one treatment v. one comparator (active or placebo) at a time for dichotomous variables. 
The software will be turned over to a vendor after the pilot stage, with improvements to be made based on user feedback. For this case study, the framework and tool were applied easily. The software does not calculate the results it displays, so the calculation of combined efficacy and the confidence intervals needed for efficacy and safety variables had to be done outside the tool. Given the complexities of combining data from different studies, this is not recommended to become an automated function. 
MCDA  MCDA is a natural progression of PrOACT, it divides difficult problems into manageable smaller criteria so to compare between alternatives. MCDA methodology also allows addition of information easily, when more data become available. MCDA methodology framework is broad and we need to clear in our mind that there are many adaptions to this framework, and we have only tested one adaption using HiView3. Many critiques in this report are based on appraisal of the programme. Software we used in this exercise, HiView3, was easy to use and made the MCDA analysis much efficient. Comparing between alternatives is apparent in both visual and number output with current programme. The HiView3 MCDA model requires criteria values and weights to be precisely known upfront. A detailed decision conference between stake holders is needed to discuss an agreed criteria function in each criteria and precise weight between each criterion; One would imagine it is often difficult and unrealistic to obtain an exact weighting score in real life situation, particularly when number of criteria for consideration is large. Besides, the decision maker's knowledge regarding to the question might not be sufficient to make an objective judgement in weighting. The result weighting and utility scale would be bias towards the stakeholder's own experience or possibility influenced by other participants. 
1] Stake holders selected for the decision conference needed to be wide enough to accommodate views from different parties  regulators, physicians & patients 2] Information on criteria would need to be available for review prior to meeting. 3] Question regarding to criteria should be addressed by individuals independent to the decision conference. 
SMAA  This case study assumes regulator's perspective to review a drug decision with the data available 5 year after market approval. SMAA is chosen because it has the flexibility in number and form of criteria; it allows the discrepancies and variations in different datasets taken into analysis; it also allows stakeholders to have different opinions on tradeoff between criteria. All of these features make SMAA a choice in dealing with real world medical decision problems where uncertainty is almost inevitable. SMAA can be realised by software 'jsmaa'. 
The 'jsmaa' may still need further development to include nonlinear utility choices. 
PSM using BRR  PSM is applicable to all decision problems to which benefitrisk ratio (BRR) is calculated. It provides the necessary visualization and representation of benefit and risk information and incorporates uncertainty into analysis. The PSM ¨C including the MCS allows uncertainties in the input values, characterised by probability distributions, to be propagated through the network of evidence to the end results. In health care settings, the decision maker is faced with having to make value or preferencebased treatment decisions under uncertainty. Both the riskbenefit joint distribution plot and acceptability curves helps decision maker to tradeoff risk and benefit The approach is based soundly on probability theory, is comprehensive in the scope of inputs, provides readily interpretable results, and can be implemented using existing software, such as @Risk or Crystal Ball sitting in Excel, or Analytica, SAS, R or SPLUS. Its outputs are clear, graphical and easy to understand. Approach can display twodimensional probability distributions for the differences between a new drug and a placebo or a comparator for either measures of favourable or unfavourable effects, or the two combined. For the approach PSM using BRR as metric indices, benefit and risk criteria are limited to one benefit and one risk criteria. In the analysis of Ketek indication ABS this risk criteria as defined by overall incidence of adverse event of special interest (AESI), which were comprised by the sum of hepatic AEs, Cardiac AEs, Syncope and Visual AEs. 

SBRAM  The framework of the Sarac BenefitRisk Method if easy to follow and clearly described in the material available on eroom PROTECT WP5. However, the process of scoring criteria is not straightforward for layman ,and there exists no finished software for the methods. Yet, with some statistical and computational knowledge scoring of criteria can be done using mathematical programs, such as MATLAB with statistics toolbox. For the Ketek analysis, scoring was done using MATLAB and since all parameters were discrete, only one scoring method had to be implemented to produce the different scoring charts. 
Assessment of using meaningful reliable information for benefitrisk approaches through practical experience
Comments  Proposed improvements and/or extensions  

BRAT  The same benefit and risk criteria were used for all methods, but BRAT allows the user to define benefit and risk, whether as direct efficacy and safety variables or as other measures or combinations of measures. The inclusion of clinical judgment and patient perspective is not limited by the tool. In fact, variables of interest to different audiences can be displayed or hidden, and there are filters that could be used to display the data from different perspectives, or to display results from observational data v. clinical trial data, for example.  Variables can be rank ordered in the BRAT software. Weighting for different audiences, or levels of severity, for example needs to be done outside the BRAT software, although a team has been looking at the feasibility of incorporating weighting in the tool after the pilot. 
MCDA  All benefit and risk criteria listed in the EPAR were used in the MCDA model. Data source were reliable. However, transformation of data to utility score could be bias. As well as the final average weighted score. Both utility function and weighting were set based on stake holder's preference after decision conference meeting  which itself undoubtedly varies. Besides, this HiView3 MCDA software only allows one value for every alternative in each criterion. However, medical data are often in range of mean with confidence interval so to account for the uncertainties and random error with the statistical estimates. The current software would not able to take the uncertainty with data into account, this is crucial in making medical judgements especially in rare events where there is a intrinsically considerable degree of uncertainty with the statistics estimates. 
1] Results range should be used in the model instead of one summary statistic value. Current programme we used in MCDA is not feasible for this type of input 
SMAA  The rational for including or excluding criteria are not clearly defined. For example the total AEs and AEs by body system, if all of them are taken as risk criteria, each AE is actually counted twice. Should the weights be adjusted on criteria which are overlapping? In the current SMAA analysis only AEs by body system is included. All data available to decision maker can be taken into analysis by SMAA, high quality or low quality, clinical trial data or observational data. Since SMAA describe performances by distributions, all the data can contribute to distribution estimation (at least in principle) in an accumulative way. As in MCDA, SMAA assigns utility and weight for each criterion. These need both clinical judgements and stakeholders' opinions. SMAA relax the requirement for weights to be exact. The weights can be in a range or totally missing while SMAA still provides answers to help decision makers. 
For utility and weight elicitation, should some standard techniques with language understandable to common people be introduced or developed in addition to decision conference? 
PSM using BRR  Potential problems in applying the techniques include collapsing benefits and risks into single measures (i.e. BRR). It is unclear how one might incorporate multiple dimensions of risks and benefits. The approach is suitable for two therapies for a binary measure of benefit and a binary measure of risk. However, additional risk and benefit criteria can also be accommodated (ref) in some situations. In this case, multiple thresholds are used to ensure the comparability of all units of benefits and risks.  Bayesian methods can easily be generalized to allow for other distributions of benefit and risk, provided one can simulate samples from the posterior distribution of interest. The Bayesian methods also allow prior information to be incorporated into the inference if such information is available. 
SBRAM  Efficacy data were, as mentioned above, pooled directly for the Sarac BenefitRisk Method approach. This was done because the information material on Sarac¡¯s BenefitRisk Assessment Methodology, as of now, only include a description of scoring discrete variables using trial population size, N, and number of events, x, as input. With the Sarac BenefitRisk method the clinical relevance of a difference in performance for drug versus comparator for each criterion is defined through the scoring method. This is done through the threshold of 2/3 of the patients performing better for drug versus comparator or vice versa. The threshold of 2/3 can be changed to another level, though this has to be done upfront. However the scoring method does not take in to account the magnitude of an effect of a drug on a criterion. 
This limitation in connection to scoring of discrete data, with input N and x could be overcome by developing an additional scoring method, which can still be based on the same principles of P(X_{Drug} > X_{Comparator}) ¡Ý α, α = 2/3. An approximation to the principle of scoring could be developed with uses the probability, p of an event for drug and comparator and the confidence interval for the probabilities. 
Assessment of the availability of clear values and tradeoffs for benefitrisk approaches through practical experience
Comments  Proposed improvements and/or extensions  

BRAT  Value judgments can be displayed through rank ordering of variables, or through inclusion and exclusion of variables. Weighting can be done outside the BRAT software and the weighted data can be displayed with the BRAT tools. Favourable and unfavourable effects are defined clearly based upon the initial definitions included in the value tree. A common scale of proportions is used in the current software, which will display either risk difference or relative risk. The denominator (e.g., events per 100 or 1,000 or 10,000 patients) is selected by the user. Final results display events of interest side by side, not combined into one metric.  If one is trying to make direct tradeoffs, the approach will facilitate the discussion. The judgment regarding the balance between the factors being traded will still have to be made by the user, but this has been the stated preference of many decision makers, rather than being handed "a number." 
MCDA  MCDA method allows a transparent judgement of value between risk and benefit. By transforming data into utility score using criteria function, this produce a common scale to allow comparison between risk and benefit Final results are easily interpretable in both graphical and numerical form using the HiView3 software. 

SMAA  By converting performance on each criterion into preference level and assessing the importance of different criteria, different criteria (benefits, risks) are directly comparable. The value judgement is through utility elicitation from stakeholders and decision conference. The results are about overall preferences (satisfactions) for all alternatives. Decision is clear from results.  
PSM using BRR  Both the riskbenefit joint distribution plot and acceptability curves helps decision maker to tradeoff risk and benefit  
SBRAM  The Sarac's BenefitRisk Assessment Methodology approach does not make judgment of values explicit, Sarac's BenefitRisk Assessment Methodology the drugs are scored in a objective manner by the threshold for which a drug performs better than the other drug on a criterion. The decisions makers has to make judgment about the direction of a criteria, e.g. if more events are good or bad or if an increase of a variable is good or bad. But to which extent an increase/decrease will result in one option being judged better on a criterion or not, is based on an objective scoring Favourable and unfavourable effects are clearly defined in the Sarac BenefitRisk Method approach. This is an attempt to define clinical significance and an opportunity to investigate and discuss the clinical relevance of data. With the Sarac's BenefitRisk Assessment approach it is determined if there is a tendency towards the drug performing better or worse than the comparator for each benefit and risk criteria. This is done through the scoring of criteria, and if the drug is judged to have a favourable effect compared to the comparator this means that 2/3 of the patients in the drug group perform better than the comparator for the specific criteria, the drug I found to do worse 2/3 of the comparator patients perform better than the drug patients. If none of either of these thresholds can be fulfilled then the method cannot say if there is a difference of the effect for between drug and comparator on the criteria in question. The value of 2/3 can be changed if deemed appropriate. Although it is clearly defined if there is a favourable or unfavourable effect of the drug compared to comparator the method does no tell how big this effect is. The method does not give a direct measure which tradeoff of benefits and risks. The result of the analysis is presented visually and gives a clear overview on whether the drug has favourable or unfavourable effect compared to the comparator for each criterion. The method require that weights are assigned to each criterion for the process of scoring is begun, this helps when the final conclusion on the drugs benefit risk profile has to be made, since it has been judged where an effect favourable or unfavourable is considered more of less important compared to other criteria. 
Assessment of the logically correct reasoning for benefitrisk approaches through practical experience
Comments  Proposed improvements and/or extensions  

BRAT  This approach can handle qualitative or quantitative, objective or subjective discrete data. Uncertainty is shown with confidence intervals. How to combine effects is not dictated by the approach. Correct reasoning and interpretation should be used in choosing to display relative risk or absolute risk, in the same manner that would be used outside this specific application.  Continuous data has to be made dichotomous. Wide variability, i.e., a long bar in the forest plot, can tend to overemphasize a less important variable if the data are not weighted. 
MCDA  Each criterion can only hold one value, however in any form. However, uncertainties within the data range are not addressed as this HiView3 MCDA software only allows one value for each criterion. Whereas medical data are not distinct. We used random effect metaanalysis to combine results from different studies listed in our data source. This allows an objective approach to pooled data between studies before using the result in the MCDA model. As discussed earlier, results of the HiView3 MCDA is dependent on precise weight information collected from stakeholders. And these often change dependent on the stakeholder involved and possibility not replicable with different stakeholder groups. As a result, conclusion from each analysis is conditional to the precise weighting decided by the stake holding group. It is arguable if the result is applicable to the wider public. 
We would recommend using metaanalysis to combine results from different studies for assessment. 
SMAA  SMAA is an extension of MCDA. So it fits for any number and any forms of criteria as MCDA does. SMAA includes uncertainty in performances and uncertainty in choices of weights into consideration. The major concerns behind this extension are (i) the performance of an alternative may change each time new data arrives, so it is suitable to view the performance as a distribution rather than a fixed value. (ii) the choices of weights are hardly agreed exactly in practices. A range for weights or a distribution for weight vector is more realistic in real situations. Under those uncertaintys, SMAA considers the chance (probability) that an alternative is the best one for each alternative as the evidence. SMAA is realised by simulation means. MCDA and SMAA use additive utility function as value function, which implies preference independence and incurs criticism. 
The performances of an alternative on different criteria are likely to be correlated. Currently they are taken as independent in SMAA simulations. 
PSM using BRR  PSM is applicable to all decision problems to which benefitrisk ratio (BRR) is calculated. It provides the necessary visualization and representation of benefit and risk information and incorporates uncertainty into analysis. The Both the riskbenefit joint distribution plot and acceptability curves helps decision maker to tradeoff risk and benefit The approach is based soundly on probability theory, is comprehensive in the scope of inputs, provides readily interpretable results, and can be implemented using existing software, such as @Risk or Crystal Ball sitting in Excel, or Analytica, SAS, R or SPLUS. Its outputs are clear, graphical and easy to understand. The approach can display twodimensional probability distributions for the differences between a new drug and a placebo or a comparator for either measures of favourable or unfavourable effects, or the two combined. 

SBRAM  Variation in data is considered through the scoring method. The Sarac's BenefitRisk Assessment Methodology does not offer any additional objective data driven method to deal with uncertainty for discrete data, but offer the possibility of assigning interval scores when subjective judge relevant. However, through resampling, it possible to evaluate uncertainty in a data driven way, which can be reflected through an interval score. In the Sarac's BenefitRisk Assessment Methodology approach benefits and risks are not integrated, all benefit and risk criteria a weighted prior to scoring. The criteria are weighted into one of three categories; most important criteria, medium important criteria and low importance criteria. All criteria have to be mutual preference independent. 