 Risk Analysis Tip (a sample ModelAssist topic)
Incorporating Differences in Expert Opinion
The following Risk Analysis Tip has been drawn from material in ModelAssistT. ModelAssist users can consult the ModelAssist-references (in the form of Mxxx) for additional information. To read more About ModelAssist and get a free download of the demo version, click here.
Introduction
Expert opinion is an important source of information for quantifying model parameters and variables. Expert estimates can produce unrealistic distributions but they are often the only source of information available. Therefore, you need to follow a number of important principles to get the most reliable and unbiased estimate (see M0300).
One common technique is to get more than one expert estimate for important parameters but this leaves the problem of incorporating any differences in expert opinion in to the model.
This Risk Analysis Tip will show you how to combine expert opinion the correct way and also give you tips on how to avoid errors.
Correct way of incorporating differences in expert opinion
Experts will sometimes produce profoundly different probability distribution estimates of a parameter. This is often because experts have estimated different things, made different assumptions or have accumulated different sets of information on which to base their opinions. Occasionally two or more experts simply genuinely disagree. How should you approach this problem?
The way to approach this problem is to treat the difference in opinion as another source of uncertainty. Therefore, the differences should not be discounted by, for example, taking the average of the opinions or the largest (or smallest) opinion. Instead, you need to create a composite distribution that reflects the range and emphasis of each opinion and confidence in the estimators.
Consider an example: Imagine that you have an important uncertain value in your model and that you ask three experts to estimate it. All three experts have the same information and it has been widely disseminated but you ask each one to estimate the parameter that you need to put in your model separately. Therefore, the experts don't sit together and decide what value they think the parameter should be. Instead, they discuss the information available together, and then separately estimate it. Of the three estimates you receive, two are PERT distributions (see M0361) and one is a General distribution (see M0203) that has a customized shape. These three distributions are plotted together below:
You can see they are different and you decide to weigh expert B (brown line) twice as much as expert A (red line) or C (blue line). You could have given them equal weightings but this is an example to show when you have more faith in B, for example, because perhaps she is closer to the project or more experienced.
The figure below shows the results of the two ways of modeling the combined expert opinion (correct and incorrect).
The correct technique to use is a Discrete distribution (see M0129), where the {xi} are the expert opinions and the {pi} are the weights given to each opinion according to the emphasis one wishes to place on them.
The spreadsheet with the correct model is provided here:
 Combining opinions (for @RISK users),
Combining opinions 5.5- (for Crystal Ball 5.5- users),
Combining opinions 7.0+ (for Crystal Ball 7.0+ users).
Incorrect way
An often used but incorrect way of combining expert opinions is to multiply the expert opinions by their weights, sum the results and divide the outcome by the sum of the weights to normalize it.
This method is wrong is because the formula calculates a weighted average of the three opinions. Therefore, the model will always pick a value in the centre and will not give the same degree of spread that is shown with the discrete distribution. The reason why you want to use the discrete distribution is that at least one person believes that the true value should be as low as three and at least one person believes that the value should be as high as ten. As we see from the figure above (red line), the incorrect distribution does not include these values.
ModelAssist
- The material within this 'Risk Analysis Tip' comes from one of the over 500 risk analysis topics available in ModelAssist, which gives a more detailed explanation of the above methods and any risk analysis techniques involved.
> Learn more about ModelAssist
top |
.