EJ Seminar Archives

27 June 2023 14:00 till 15:00

[SEJ] Roger Cooke: Convolutions of Continuous Ranked Probability Scores: comparison with the classical model.

Scoring rules were introduced by de Finetti in 1937 as tools for rewarding honesty in eliciting subjective probabilities: An expert receives score as a function of his/her probability assessment and the realization. The score is strictly proper if the expert maximizes (for negatively sensed rules, minimizes) his/her expected score, per variable, by and only by stating his/her true belief. Scoring rules were not designed for evaluating or combining experts; indeed, rewarding honesty is not the same as rewarding quality. A few examples illustrate this difference. Attention then focuses on Continuous Ranked Probability Scores, a continuous version of the Brier score. We introduce a scale invariant version of this score. Under the hypothesis that samples are independently drawn from the expert’s subjective distribution we derive its sampling distribution and also the distributions of sums of independent scores. The classical model uses asymptotically proper scoring rules applied to sets of variables based on the standard multinomial chi square test.. These two approaches are compared using the dataset of 49 studies with 526 experts published in 2021.

13 June 2023 14:00 till 15:00

[SEJ] Simon French: Reflections on 50 Years of MCDM: Issues and Future Research Needs

Modern discussions of multiple criteria decision-making extend back about half a century. I reflect on key developments, schools of thought and controversies that have taken place over the period, arguing that perhaps those of us in different schools focus too much on our differences and do not capitalise enough on what we share in common. Moreover, the differences between schools are indications of their respective weaknesses and can drive improvements in each. The discussion points to a number of issues and research needs that the community needs to address.

25 April 2023 14:00 till 15:00

[SEJ] Joshua Becker: Does communication improve the wisdom of crowds? It depends on the question.

esearch on the ‘wisdom of crowds’ has consistently shown that one way to improve the accuracy of numeric estimates such as economic forecasting is by using the average estimate of multiple individual contributors, rather than relying on one single person. However, decades of lab experiments have produced contradictory results about whether and when communication between group members makes the resulting average more accurate or less accurate. Thus despite the existence of over 100k results on Google scholar using the “Delphi method” form of information exchange, we lack clear evidence that this method is actually better than unstructured discussion, or that any form of communication is better than none. This talk will explain contradictions in prior research by showing how emergent network centralization interacts with pre-communication estimate distribution such that communication sometimes increases accuracy and sometimes decreases accuracy. Using a formal model of estimate formation and experimental data, I will argue that the fixed effect paradigm—i.e. the assumption that any form of communication is always either helpful or harmful—must be replaced with a model that depends on the particular estimation task under consideration. I will conclude by discussing some limitations of this model for describing true deliberation, sharing some new research under development, and opening the floor for collaborative speculation and brainstorming on how future research might address these topics.

28 March 2023 14:00 till 15:00

[SEJ] Jafar Razaei

14 March 2023 14:00 till 15:00

[SEJ] Katie Thorn and David Hartley: Expert judgement in the pharmaceutical industry: Predicting success in a world of attrition

Pharmaceutical organisations often rely on the use of expert judgment to build understanding of expected future outcomes across Research and Development. Most notably in the characterisation of a Probability of Technical and Regulatory Success - the likelihood that each medicine under consideration makes it on to market. Mechanisms for the elicitation of this data have been in place for many years across organisations, but both the techniques utilised and the expert structure considered varies by company.

The landscape of drug development is evolving and with this we see the complexity of medicine development plans increasing and companies (who historically may have had a single differentiating factor) needing to be more versatile. As a result, the requirements for, and demands on, expertise are becoming more complex.

In this presentation we will share common use cases in which expert judgment is considered in the pharmaceutical industry, the approaches used and the associated differences between organisations. We will also highlight, and be seeking input on, several constraints we face and challenges still to be tackled.

28 February 2023 14:00 till 15:00

[SEJ] Gordon Woo: Expert Judgement on Ukraine Conflict Risk

Some of the most important practical applications of the use of expert judgement are in international relations. The Russian invasion of Ukraine which started on 24 February 2022 has created the biggest threat to global peace and stability since the Cuban missile crisis of 1962. President Kennedy estimated the odds of disaster as between one out of three or even. The odds of disaster resulting from the Ukraine war are much smaller than this, but President Putin has pledged to use all powers and means to defend Russia. Misunderstandings and miscalculations are a feature of crisis diplomacy. The role of expert judgement in political crisis management is reviewed, with a focus on Ukraine war risk. Insights from counterfactual analysis of past crises are explored.

14 February 2023 14:00 till 15:00

[SEJ] Ullrika Sahlin: Semi-formal expert judgements – lessons learnt from food safety

A scientific assessment usually involves a group of scientific experts with the task to gather available evidence and answer an assessment question. Organizations carrying out scientific assessments have guidelines to support different steps in assessments and ensure good practice. The Scientific Committee at the European Food Safety Authority (EFSA) adopted in 2014 a Guidance of Expert Knowledge Elicitation in Food Safety Assessments. It recommends making using formal and probabilistic expert judgement. I will describe the progress in implementing structured expert judgement in these scientific assessments, and provide reflections on the choice of methods for EKE, the needs to adapt the requirements on formality and level of precision in probabilistic judgements to the applied context.

31 January 2023 14:00 till 15:00

[SEJ] Xin Ren: Measuring the impacts of Human and Organizational Factors on human errors in the Dutch construction industry using Structured Expert Judgment

While structural safety has long been viewed and treated with great importance, it remains a fundamental and critical issue in the construction industry. It has been widely acknowledged that the leading cause of structural failures is human error, rather than technical problems. The rise of a new system approach towards human error no longer views it as the cause of accidents or failures, but as a symptom of improper system design, organization, or other troublesome issues embedded inside the socio-technical system. Human errors arise from the working context and situations created by underlying factors such as the working environment and the organizational structure. These factors are the Human and Organizational Factors (HOFs) that can shape people’s performance at work and lead to the occurrence of errors. Therefore, it is important to gain a better understanding of these factors before structural safety can be improved in practice.

A previous study has identified the critical HOFs in the Dutch construction industry. In this study, the impacts of these HOFs on human error occurrences are measured using the Classical Model for Structured Expert Judgment. The survey data are analyzed using the ANDURYL toolbox. As a result, the critical HOFs are quantified for their contribution to the human error occurrence in structural design and construction. In addition, these results are compared with existing studies in other safe-critical industries.

Besides the results, insights into ethical requirements for performing Structured Expert Judgment will be presented.

24 May 2022 14:00 till 15:00

Kevin Wilson: Probability elicitation for a future flooding scenario in the CReDo digital twin

10 May 2022 14:00 till 15:00

Dmytro Perepolkin: Encoding and updating uncertain judgments using hybrid elicitation and quantile-parametrized distributions

26 April 2022 14:00 till 15:00

Willy Aspinall: Estimating incredibly small event probabilities with expert judgement, Bayes Nets and Importance Sampling: present-day likelihood of a repeat super-eruption of Aso volcano, Japan.

12 April 2022 14:00 till 15:00

Mark Burgman: A history and diagnosis of attempts to embed SEJ in government

29 March 2022 14:00 till 15:00

Jim Smith : Reifying Probabilistic Digital Twins for Flooding Risk Assessments using Expert Judgement.

15 March 2022 14:00 till 15:00

Oswaldo Morales Napoles : ANDURYL replacing EXCALIBUR

01 March 2022 14:00 till 15:00

Roger Cooke:Twin RFF SEJ panels supporting Social Cost of Carbon

15 February 2022 14:00 till 15:00