Eliciting a probability distribution is the process of extracting an expert’s beliefs about some unknown quantity of interest, and representing his/her beliefs with a probability distribution. The challenges are, firstly, to help the expert consider uncertainty carefully, without being excessively overconfident or underconfident, and secondly, to find a way of constructing a full probability distribution based on a small number of simple probability judgements from the expert. Elicitation can be used to construct prior distributions in Bayesian inference, though my interest is more in situations where we are using expert judgement because there is no data.
Tony O’Hagan and I have developed the Sheffield Elicitation Framework (SHELF), a package of protocols, templates and guidance documents for conduction expert elicitation. In support of this, I maintain an R package
SHELF which is available on CRAN (with a developmement version on GitHub).
Shiny apps for elicitation
SHELF R package includes shiny apps for implementing various elicitation methods. If you are not an R user, you can try the following apps online. Please note that access to these apps is time-limited (they are hosted on RStudio’s shinyapps.io service). If you want to use the apps in an actual elicitation workshop, I strongly encourage you to use the offline versions in the
SHELF R package.
Elicting a single distribution. Elicit a single distribution, directly specifying probabilities or quantiles, or using the roulette method.
Eliciting individual distributions from multiple experts. Includes options for fitting a weighted linear pool.
Eliciting bivariate distributions. Elicit a joint distribution for two dependent variables using a Gaussian copula.
Eliciting Dirichlet distributions. Elicit a Dirichlet distribution for a set of proportions constrained to sum to 1.
The MATCH elicitation tool
As part of the MATCH project, we produced a web-based elicitation tool which is based on an earlier version of the SHELF R code. Multiple users can log into the same session, which can be useful when the facilitator and expert(s) can’t meet in the same room.
Papers on elicitation
- Ren, S., Oakley, J. E. and Stevens, J. W. (2018). Incorporating genuine prior information about between-study heterogeneity in random effects pairwise and network meta-analyses. Medical Decision Making, 38 (4), 531-542
- Alhussain, Z. A. and Oakley, J. E. (2017). Eliciting judgements about uncertain population means and variances. In submission.
- Morris, D. E., Oakley, J. E. and Crowe, J. A. (2014). A web-based tool for eliciting probability distributions from experts. Environmental Modelling & Software, 52, 1-4.
- Ren, S. and Oakley, J. E. (2014). Assurance calculations for planning clinical trials with time-to-event outcomes. Statistics in Medicine 33(1), 31-45. Download supporting R code.
- Oakley, J. E. (2010). Eliciting univariate probability distributions, in Rethinking Risk Measurement and Reporting: Volume I, edited by Böcker, K., Risk Books, London.
- Daneshkhah A. and Oakley, J.E. (2010). Eliciting multivariate probability distributions. (Supporting R code for this chapter is available here) in Rethinking Risk Measurement and Reporting: Volume I , edited by Böcker, K., Risk Books, London.
- Nixon, R.M., O’Hagan, A., Oakley, J. E., Madan, J., Stevens, J.W. Bansback, N. and Brennan, A. (2009). The Rheumatoid Arthritis Drug Development Model: A case study in Bayesian clinical trial simulation. Pharmaceutical Statistics 8(4), 371-389
- Gosling, J.P., Oakley, J.E. and O’Hagan, A. (2007). Nonparametric elicitation for heavy-tailed prior distributions. Bayesian Analysis, 2, 693-718.
- Oakley, J. and O’Hagan, A. (2007). Uncertainty in prior elicitations: a non-parametric approach. Biometrika 94, 427-441.
- O’ Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. E., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E. and Rakow, T. (2006). Uncertain Judgements: Eliciting Expert Probabilities. Chichester: Wiley.
- O’Hagan, A. and Oakley, J. E. (2004). Probability is perfect, but we can’t elicit it perfectly. Reliability Engineering and System Safety, 85, 239-248.
- Oakley, J. (2002). Eliciting Gaussian process priors for complex computer codes. The Statistician, 51, 81-97.
I have also contributed to the following guidance document prepared by the European Food Safety Authority:
- European Food Safety Authority (2014). Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment. EFSA Journal 2014;12(6):3734, 278 pp. doi:10.2903/j.efsa.2014.3734