USAR decision-making: the role of hazard-specific expertise and risk assessment

The accepted detrimental effects of climate change and the anticipated increased frequency of cascading disasters means there is a pressing requirement to equip search and rescue teams with the capability to perform effective and complex risk assessments. This paper investigates risk-based decision-making expertise in the aftermath of the 2011 earthquake and tsunami in Japan. It compares the actual decisions made by an Urban Search and Rescue (USAR) commander, with the decisions that a cohort of people working within search and rescue made, when provided with the same decision context using 3 vignettes. Variations in the results are explored in terms of the complexity of the risk decision and the type of expertise required. The findings indicate that as the risk becomes more complex, the percentage of answers that were the same as the USAR commander (that we deem as ‘correct’ as they did not result in any adverse outcomes for the USAR team) decreased. Training entities need to provide decision-makers with the necessary human capabilities so they can perform the complex risk assessments required to make decisions in low-probability yet high-consequence disasters.

Introduction

The aftermath of a large-scale earthquake requires the deployment of Urban Search and Rescue (USAR) teams. These teams have to perform risk assessments to inform decision-making. These assessments are invariably based on the experience and expertise of individuals. This paper examines how USAR professionals use their expertise to perform risk-based decision-making using a series of vignettes. The vignettes are brief descriptions based on the actual decisions made by an Australian USAR commander deployed to Japan in the aftermath of the earthquake, tsunami and resultant Fukushima Daiichi nuclear disaster in 2011. Previous research identified that expertise in radiation protection and nuclear safety was used by the Australian USAR commander when making numerous decisions in what was an uncommonly complex and dynamic disaster environment (Curnin et al. 2020).

Due to the high-risk environments that USAR teams operate in, the management of risks is an essential skill. Risk assessments have to be properly interpreted and the key aspects of risk, uncertainty and knowledge, encompassed in the decision-making process (Aven 2016). However, conventional models of decision-making that argue for time-consuming and analytical processes can be unsuitable due to the time-sensitive nature of the decisions that must be made (Klein, Calderwood & Clinton-Cirocco 1988). Uncertainty of the situation, a lack of specific knowledge about the risk and temporal constraints, which are often encountered in high-consequence yet low-probability decision-making environments, can occur when an expert encounters a situation outside of their pattern repertoire. This can lead to delayed or deferred decisions (Curnin et al. 2020). In this context, judgements and decisions must be made rapidly, leaving little or no time for reflection (Hurteau et al. 2020) and can often depend on an individual’s levels of expertise.

Due to the high-risk environments that USAR teams operate in, the management of risks is an essential skill.
Image: Assistant Commissioner Robert McNeil (retired)

Prior experience and deliberate practice can play a critical role in risk assessment and decision-making in uncertain and dynamic environments, such as those created by disasters where experts are often called on to provide their knowledge to assess risk (Ahluwalia et al. 2021). In time-pressured situations, experts can use their accumulated knowledge and develop mental models to assist in making faster and accurate risk assessments. However, research has identified that when experts are presented with a domain-specific task outside of their previous experience, their performance is similar to that of a novice (Mumford et al. 2010). This phenomenon is known as domain-specificity and provides a useful tool for researchers to identify the differences in performance between novices and experts, as both can be presented with domain-specific tasks (Ericsson & Lehmann 1996) and differentiated based on their results in these tasks (Ericsson, Hoffman & Kozbelt 2018). Experience therefore plays an integral role in achieving expertise in a specific activity and the evaluation of individuals based on domain-specific tasks is an appropriate method to distinguish levels of expertise. This study explored this issue in the context of USAR decision-making.

Methods

The Critical Decision Method was used to elicit aspects of the Australian USAR commander's expertise following the 2011 Fukushima Daiichi nuclear disaster. This approach allowed for the use of a series of cognitive probes so the commander was able to reflect on his own strategies and the basis for their decision (Curnin et al. 2020). The commander was interviewed 2 times after the initial Critical Decision Method to clarify his decision-making rationales. All 4 of his team members were also interviewed between the second and third interviews to gather information, build a deeper understanding of the decisions and verify that the decision itself was considered ‘correct’ at the time. Ethics approval was received from the University of Tasmania Human Research Ethics Committee [Approval Code: H0008810].

Survey

A decision-making survey was created that drew on 3 of the decisions that the Australian USAR commander made during the deployment. The survey was designed to measure the effect of experience on decision-making ability. The 3 decisions made by the Australian USAR commander were translated into vignettes to provide context for those taking part in the survey. The 3 decisions were summarised as:

• the dust mask challenge
• the concrete bunker relocation
• the exit strategy.

These decisions were chosen due to the varying complexity required to manage the associated risks. The survey was tested with non-practitioners prior to being used in this study and took 15 minutes to complete. Participants began the survey by self-selecting the appropriate amount of experience they had in each of the 4 demographic categories of:

• number of years working in search and rescue
• current role in search and rescue
• number of operational SAR deployments completed (not exercises)
• number of SAR exercises or incidents experienced that involved a nuclear radiation scenario.

After completing the demographic information, participants were instructed how to answer each scenario. The survey consisted of 3 items. Each item started with a description of the scenario followed by 2 or 3 options. The participant’s responses were assessed as ‘correct’ if they were the same as the actual decision made by the Australian USAR commander in the Fukushima deployment and ‘incorrect’ if they selected any of the alternate options.

Participants

There were 56 participants who attended the Australian Conference on Disaster Management, which had a stream that focused on SAR. The participants were divided into 2 groups based on their levels of experience in each of the 4 demographic categories (see Table 1). There were no exclusion or inclusion criteria other than participants being fluent in English.

Table 1: Demographic information of the survey participants

Data analysis

The raw counts for each group were tabulated and converted to percentages of correct and incorrect answers for each of the 3 scenarios. Additionally, Group 2 in the ‘Number of SAR exercises or incidents experienced that involved a nuclear radiations’ scenario category was removed due to a lack of respondents (N = 1) in comparison to Group 1. However, data was useful to establish that there was not a bias of expertise with respect to the sample.

A chi-squared test of independence was conducted to examine the relationship between experience and the ability to correctly respond to the scenarios provided in the survey. The raw numbers of correct and incorrect responses were tallied for Group 1 (Low Experience) and Group 2 (High Experience) for 3 of the 4 demographic categories of (1) Years of experience in SAR, (2) Role in SAR an (3) Number of deployments. The ‘Number of SAR exercises or incidents experienced that involved a nuclear radiations’ scenario was again excluded due to the disparity between the number of respondents in the 2 categories of experience. Finally, all of the data was combined regardless of the level of expertise and a chi-squared test was conducted to determine if the proportion of correct to incorrect results was significantly different across the scenarios irrespective of ‘expertise’ as characterised by the demographic variables.

Results

Tables 2 and 3 summarise the results. Table 2 presents the raw numbers and percentage of correct and incorrect answers by demographic variables. Table 3 identifies the results of the chi-squared tests of independence between groups 1 and 2 for the 3 scenarios.

Table 2: Raw numbers and percentage of correct and incorrect answers by demographic variables.

Table 3: Chi-square results for scenarios by demographic variables.

The results demonstrate no significant difference in the proportion of correct to incorrect answers across the variables. When all the data are viewed as one dataset, there is a significant result. A chi-square test of independence was performed to examine the relationship between the scenario and the correctness of the decisions made by participants. The result was significant, χ2 (2, N = 164) = 26.8, p < 0.00001 and the ratio of correct to incorrect decisions for the scenarios are significantly different to each other.

Discussion

At face value, the results appear counter-intuitive. Differentiation of participants based on measures of SAR experience or expertise yielded no significant difference in correctness of outcome. Further analysis demonstrated a significant difference in correct answers based on the scenarios. This led to an examination of how the scenarios varied and to look for a rationale regarding answers that were incorrect or inconsistent with the decisions made by the USAR commander.

Simple risk assessment and control

Scenario 1 was the dust mask challenge and it had the highest percentage of correct answers. The participants were presented with the following scenario:

The USAR team has to drive through the Fukushima exclusion zone and the plume modelling has identified that there was caesium in the plume (caesium is a metal that may be stable/nonradioactive or unstable/radioactive). You have previously been assured by experts that the dust masks the team were issued with prior to deployment would capture and prevent larger particles from penetrating through the membrane. The team’s equipment was also informing you that the radiation levels were just above normal.

Scenario 1 possible answers:

• A - Immediately instruct the team to don their dust masks.
• B - Given that levels are almost normal and the associated down-side risk of creating concern in your team you do not instruct the team to don their masks.

The option chosen by the commander was to immediately instruct the team to don their dust masks (Option A).

Alternative option B was designed as a plausible outcome for a novice decision-maker. This option did not account for the fact that personnel within a USAR team are familiar with the use of personal protective equipment and unlikely to be overly concerned by an increase in risk while operating within this environment. The simple explanation of the correct decision from a risk-based perspective is that the commander identified an increase in risk and added a control they believed would mitigate this increase. The control was easily applied and reasonable in the circumstances. This answer was the most logical to choose and the easiest to comprehend.

Integrating intuition and risk management

Scenario 2 was the concrete bunker relocation. The participants had to respond to conflicting assessments of risk. The participants were presented with the following scenario:

Late one evening you are informed by a nuclear radiation detection expert in Australia that you need to find a concrete bunker and relocate the team to the new location. You are not aware of any changes in the last 12 hours due to sudden weather variations or other factors that would necessitate increased caution. You do not have the level of expertise as the nuclear radiation detection expert that has made this request, however, your intuition suggests that this is not correct.

Scenario 2 possible answers:

• A - Direct the team to move to the concrete bunker as it immediately manages the radiological risk.
• B - Trust your understanding of the current situation, your judgement that the information you received is incorrect and do not immediately locate the team to a concrete bunker.
• C - Make no decision – seek a second opinion on the radiological hazard.

The option chosen by the USAR commander was to make no decision and seek a second opinion on the radiological hazard (Option C). In their decision, the commander sought to manage the conflict between their intuition and an expert opinion by seeking an alternative opinion and evaluating all sources of risk-based information. They determined that their intuition was correct through further analysis of the risk, concluding that the control option advised by the first expert was not justified. We can align this result with research in the health sector where intuition is an essential tool for people working in critical-care areas. Practitioners in these areas draw on their analytical skills and intuition when assessing risks and making decisions that require a high level of precision (Cork 2014).

The alternative options were specifically designed to appear plausible to a moderately experienced USAR decision-maker. Option A required the decision-maker to ignore, or at least downplay, their intuition. Option B required participants to trust their intuition and not seek further sources of information. It was expected that a more experienced practitioner would recognise their intuition had value but that it should not be the sole basis for a decision. We anticipated they would be aware of the logistical challenge of finding a concrete bunker in the devastation of the tsunami, earthquake and radiological event. It would have meant placing the objectives of the team on hold. There also would have been consequences for other international teams in the vicinity.

While research increasingly demonstrates the value of intuition (Cork 2014), it is more powerful and valid when sense-checked with rational analysis. In this study, the error in decision-making for the 2 ‘wrong’ answers can be attributed to either authority bias (from the first expert) or a failure of meta-cognition (thinking about one’s thinking) when trusting intuition without further analysis.

Fukushima was the first event where internationally deployed teams had to manage risks of aftershocks as well as, equipment to manage hypothermia.
Images: Assistant Commissioner Robert McNeil (retired)

Pair-wise comparison of multiple risks and control options to determine the best solution

Scenario 3 was the exit strategy and was the most complex of the scenarios. This scenario elicited the least correct responses from the participants. The participants were presented with the following scenario:

You have to arrange for the team to travel to the final destination airport where the team will fly back to Australia. Which option do you choose?

Scenario 3 possible answers:

• A - Take the team to a nearby airport so that you can be flown by military aircraft to the final destination airport where the team will fly back to Australia. This means the team will not have to travel back through the exclusion zone of the Fukushima reactor again. However, it is possible that the team will face significant delays at the nearby airport as the military operating the aircraft are committed to providing mercy flights to the Japanese community. This would expose the team to staying in a location that is at increased risk of further earthquakes where the team would again have to stay in tents in subzero temperatures, however, radiological risks would be avoided.
• B - Take the team by bus and drive to the final destination airport where the team will fly back to Australia. This means the team will need to travel through the exclusion zone of the Fukushima reactor gain. The drive to the final destination airport would be long but the team would be warm in the bus. Delays at the nearby airport would be avoided. During recent travel near the Fukushima reactor site the team monitored only slight readings for radioactive exposure.

The USAR commander chose option B. During the interviews, the commander reflected that they drew on their previous experience to perform this risk assessment. They clarified a situation that the military responders could not provide a time when they would be transported and, with their expertise over the previous 10 days, they knew that the team could be at risk of earthquakes and hypothermia if they followed that decision. The commander considered option A, which was to travel by bus through the exclusion zone, and weighed up the risks. Based on their previous experience and knowledge, they determined that the risk was low. The commander rejected option A of staying at the airport after exploring other options. This demonstrated an alignment to the Recognition-Primed Decision model.

Option A was designed as a plausible decision for an experienced decision-maker without radiological hazard-specific expertise. We expected that the selection of this alternative could be triggered by the avoidance of the radiological hazard. We considered that people triggered by the radiological hazard and choosing this option would discount the risks associated with the cold temperatures and earthquake aftershocks.

Although USAR teams have been deployed internationally and have had to manage the risks of aftershocks and are deployed with the appropriate equipment to manage hypothermia, Fukushima was the first event where teams had to manage these risks in tandem with radiation risk. The fact that many of the participants did not possess domain-specific expertise in radiation hazards could infer that the protection of risk from radiation was uncontrollable as it was an unknown. Perko (2014) suggests that those who lack expertise in radiation risks are more likely to have higher risk perception for radiological risks, such as nuclear waste. In contrast, those who have expertise and experience in radiation risks are more likely to have a lower risk perception of radiological risks (Perko 2014). Scenarios such as Fukushima pose challenges for current risk models used in emergency management, such as the Dynamic Risk Assessment Model, as, in complex disasters, several risks are often controlled simultaneously. In these situations, and in the scenarios provided in the survey, a person’s cognitive biases may alter the perception they have of the risks inherent to a particular decision (Adam & Dempsey 2020).

The future of risk assessments

As the risk becomes more complex, people performing the risk assessments require a combination of higher-order reasoning skills, such as inductive and deductive reasoning. However, due to the rarity of deployments such as occurred in Fukushima, USAR teams must rely on exercising that simulates extreme yet plausible scenarios to practice and enhance their risk-based decision-making. Those responsible for preparing USAR teams need to conduct training exercises that provide the necessary human capabilities to perform the risk assessments required to make decisions in these rare circumstances.

Conclusion

Low-probability yet high-consequence disasters require USAR teams, and particularly their leaders, to conduct risk-based decision-making. This paper proposed that, in risk-based decision-making, as the risk increases in complexity the critical thinking skills of the people performing the risk assessment need to improve so they can determine the level of risk and link it to their proposed actions. This requires decision-makers to have skills such as the ability to combine intuition with rational analysis, to manage cognitive biases and to use metacognitive skills when performing risk assessments. Further research may establish other aspects of critical thinking that are relevant to complex risk assessment. Practically, due to the rarity of deployments to disasters such as Fukushima, those responsible for preparing USAR teams must provide personnel with the necessary and realistic training environment so they can experience risk-based decision-making for potential complex disasters. Collecting detailed accounts of decision-making after an event and translating them into learning materials for exercising is a viable option.