Training to improve emergency management decision-making: what the research literature tells us

The importance of training for effective performance in high-stakes, high-risk work settings is well-known. Successful training is the systematic acquisition of skills, rules, concepts or attitudes that result in improved work performance. Simply exposing people to training situations is not sufficient for them to develop knowledge and skills. This will only occur if the activity results in learning. While much training focuses on the development of technical skills, it is important to train people in non-technical skills, such as decision-making. This paper presents the results of a literature review of 95 peer-reviewed articles that consider the current training and exercise practices used to develop emergency management decision-making capability. The different approaches to training can be categorised into 4 types: discussion-based, operation-based, E-based and post-incident debriefs. This paper discusses current practice in emergency management decision-making training in each of these categories together with studies that have evaluated their effectiveness noting the generally limited nature of evaluation studies. To promote evaluation of training, several studies have developed tools to assess the effectiveness of training. Finally, key takeaway points related to emergency management organisational training and exercise programs are provided.

Introduction

The importance of training for effective performance in high-stakes high-risk work settings has long been recognised. Writing in AD 70, the historian Josephus described the Roman Army’s approach to training:

They do not wait for war to begin before handling their arms, nor do they sit idle in peacetime and take action only when emergency comes…Their battle-drills are no different from the real thing; every man works hard at his daily training…It would not be far from the truth to call their drills bloodless battles, their battles bloody drills.
(Josephus AD70/1981, p.195)

This glimpse into the past shows 3 essentials for effective workplace training: it is planned and organised, ongoing and task-focused.

A contemporary account is that training is ‘the systematic acquisition of skills, rules, concepts, or attitudes that result in improved performance in another environment’ (Goldstein and Ford 2002, p.1). This definition highlights that training is not only a systematic process that builds requisite skills and knowledge but that it also develops appropriate trainee attitudes and their understanding of the norms and unwritten rules of the work concerned. Simply exposing people to training situations is not sufficient for them to develop knowledge and skills; this will only occur if the activity results in learning.

Important points from the work of Goldstein and Ford (2002) relate to the design of training programs more generally. Their instructional systems model of training outlines 4 elements that follow a recursive process of:

1. assessing the needs for training
2. developing the training program to meet the needs
3. implementing the training program
4. evaluating the effectiveness of the training program (which links back to 1).

Thus, a training program is a system that requires ongoing evaluation and modification. This means that the design of a training program is never completely finished. Effective evaluation can address important aspects of:

• the desired learning outcomes being achieved (i.e. improved knowledge, skills, attitudes and other personal characteristics)
• the benefits exceeding the costs
• the learning transfer to the workplace.

Evaluation is an integral part of the design, development, delivery and implementation of a training program (Phillips and Phillips 2014). However, robust evaluation of training programs continues to be infrequently undertaken.

A learning framework commonly used to guide training and development programs of Australian Government agencies is the Center for Creative Leadership’s 70:20:10 model (e.g. Johnson et al. 2018; New South Wales Public Service Commission 2015). In Victoria this model has been implemented to guide incident management team training (Slijepcevic et al. 2012). The 70:20:10 framework identifies 3 types of learning: experiential, social and formal (McCall et al. 1988). According to the model, learning should be made up of 10% formal training (structured learning activities), 20% through learning from others (e.g. peer feedback, coaching, mentoring, managerial feedback and lessons learnt) and 70% through on-the-job practise and challenging work-based assignments. Despite its popularity, the 70:20:10 model has been criticised for its atheoretical nature and the lack of empirical evidence to demonstrate its effectiveness (Clardy 2018; Johnson et al. 2018; Kajewski and Madsen 2013).

The term ‘decision-making’ can be defined as ‘a commitment to a course of action that is taken in order to achieve a desired goal’ (Alison and Shortland 2021, p.13). Exercising sound judgement and effective decision-making are a critical capability for emergency management operational personnel. By their nature, emergency environments can be very challenging and involve uncertain, fluid and high-stakes situations. Individuals and teams may need to make time-critical decisions, often based on incomplete, poor quality, conflicting or large volumes of information. Depending on the phase of an incident, decision-makers need to make the most of inadequate resources or coordinate the performance of a complex array of interdependent resources to ensure an effective response. Decision-makers need to consider the incident at hand and the safety of their crews as well as its consequences for communities, the environment and businesses. Decision-making is a non-technical skill and is intimately linked with situation awareness. Without a good understanding of what is going on it is very difficult to make effective decisions (Mosier and Fischer 2010; McLennan et al. 2006). Decision-making effectiveness also relies on other non-technical skills, such as communication, coordination, cooperation, leadership and coping with stress and fatigue (Bearman et al. 2023; Butler et al. 2019; Hayes and Bearman 2023; Hayes et al. 2021). While much training focuses on the development of technical skills (such as the correct use of equipment or analysing hazard prediction modelling) it is important to train people in non-technical skills such as decision-making.

There is general agreement in the training literature of the important distinction between 2 types of knowing (e.g. Kole et al. 2020):

• declarative or explicit knowledge (knowing that)
• procedural or tacit knowledge (knowing how to).

Declarative knowledge is fact-based and traditionally taught via lectures, seminars, books and manuals. By itself, it is of limited use to develop competence in decision-making (Muñoz et al. 2015). Retention of content is generally poor in the absence of opportunities to make use of the material in practice settings soon after the training session.

Procedural knowledge is acquired mostly through undertaking tasks, making decisions, receiving feedback and reflecting (Lamb et al. 2021; McLennan et al. 2005). Noe and Ellingson (2017) highlight how policies and procedures (i.e. explicit knowledge) can be readily taught but learning through experience plays a key role in helping a person to decide when and how to apply, adopt or set aside those practices (Butler et al. 2021). Health care research on the training of tasks requiring procedural/tacit knowledge and skills shows that these are better attained through simulations than through lectures (Nestel et al. 2011). Training procedural knowledge is therefore highly dependent on undertaking suitably designed job-related exercises and activities. Trainers and facilitators provide appropriate opportunities for trainees to self-reflect on the quality of their decision-making and how it could be improved (Ellis and Davidi 2005). There is also opportunity to improve procedural/tacit knowledge learning from decision-making on-the-job following an incident if suitable discussion among personnel involved is facilitated or reflective practice is undertaken (Ford 2021; Hoffman et al. 2014; Tannenbaum and Cerasoli 2013).

Rather than trying to examine the broader training and development literature, this study focused on the material most pertinent to developing the procedural/tacit knowledge central to emergency management operational decision-making capability. There is extensive literature dealing with traditional classroom and online training of declarative/explicit knowledge, however, the focus here is on learning that supports acquisition of the procedural/tacit knowledge for effective decision-making. Research by Skryabina et al. (2017) and Chen (2014) considered 3 categories of activities:

• scenario and simulation-based training and exercises
• post-incident learning
• evaluation of training.

The rationale for this study including the third area of evaluation is that it is a critical component of a systematic approach to delivering an instructional system (Dipboye 2018; Goldstein and Ford 2002). The aim of this paper is to bring together key findings reported in recent emergency management decision-making training research literature.

Methods

A narrative literature review was used in this study. This approach was the most appropriate for the primary aim of identifying substantive research findings from the literature, rather than evaluating the overall state of emergency management decision-making training research. Initial searches by using ScienceDirect, Scopus, Web of Science and PSYCHINFO proved disappointing. Few relevant reports were identified and several important and well-known works were not identified.

The method adopted subsequently was to examine the first 1,000 abstracts generated by searching the Google Scholar database using ‘emergency management decision-making training’ as the descriptor. This resulted in 83 published papers being selected on the basis of their title and abstract, of which 54 were relevant. A further search of the Google Scholar data base using ‘incident command training’ as the descriptor identified 2 additional references. Using the ‘cited by’ Google Scholar search facility, papers that had cited each of the 56 references were checked and an additional 39 relevant references were identified. Searches of the reference lists of these 39 works did not find new references, resulting in a final total of 95 published papers selected for detailed reading. As an additional check, an EBSCO Ultimate search was undertaken for peer-reviewed abstracts over the period 2000 to 2023 using the search terms (emergency management OR crisis management OR disaster management OR public safety) AND (decision-making OR decision-making OR decision-making process OR decision-making process) AND (training OR education OR development OR learning). No additional references were located. For data extraction, content analysis of the selected papers was used to identify the training activities used to develop emergency management decision-making capability and their effectiveness.

Results

Based on Chen (2014) and Skryabina et al. (2017), training and exercise activities can be categorised into 3 groups: discussion-based, operation-based and intermediate electronic-(E-) based exercises. These intermediate forms of training fall between discussion-based and operation-based activities and include hybrid computer-supported activities and virtual/augmented reality activities (Chen 2014). In addition, another important learning opportunity is presented by post-incident debriefs. Table 1 lists the types of training and the exercise activities and objectives. The current practice in emergency management decision-making training in each of these categories and the studies that evaluate effectiveness are discussed, noting the generally limited nature of the evaluation studies. To promote better evaluation of training, several studies are identified that have developed tools to assess the effectiveness of training. Lastly, we make some general points about training and organisational programs.

Discussion-based training and exercises

Workshop exercises

Alexander (2000) recommended workshop exercises as a low-cost way to bridge the gap between theoretical classroom-based instruction and practical experience in the field. Scenarios can be postdictive reconstructions of previously occurring events or hypothetical constructions of possible future emergencies. Alexander (2000) suggested building blocks for constructing and conducting scenario exercises and emphasised the importance of scenarios that encouraged participants to think through the consequences of their decisions and actions.

Alison et al. (2022) suggested workshops should incorporate ‘grim storytelling’: imagining negative situations in which all courses of action result in bad outcomes. The purpose is to support decision-makers’ to imagine rare, high-impact events and make ‘least-worst’ decisions to help them manage such situations actively and constructively. This suggestion is based on a frequent criticism levelled at emergency services organisations following disasters and large-scale critical incidents of a failure to act in time or even to act at all (Alison et al. 2022; Waring et al. 2020). This is attributed to redundant deliberation leading to decision inertia. Redundant deliberation is a pathological hesitation arising from overthinking choices among difficult options and is likely to occur where there is no standard operating procedure or similar to provide guidance. It also occurs where decision-makers have not been exposed to enough of these events to build up a repository of expert knowledge. Grim storytelling is similar to the ‘pre-mortem’ proposed by Klein (2007) and ‘worst-case scenario thinking’ proposed by Johnson (2014). No evaluation studies of emergency services workshop exercises could be located during this study.

Table 1: Types of training and exercise activities and objectives.¹

Tactical decision games

Tactical decision games (TDGs) (Schmitt 1996) are simulations that provide low-fidelity, low-cost emergency management decision-making training. Originally developed for the military, these are scenario-based games, typically brief and conducted in small group settings. They can be a postdictive reconstruction of an aspect of a previous emergency event or a hypothetical scenario devised to challenge particular aspects of participant decision-making competence (Crichton 2009). Objectives of TDGs:

• Exercising and practising decision-making skills in the context of agency operating principles.
• Assisting participants to develop a shared understanding and recognition of possible problems they may encounter.
• Building a repertoire of problematic situations that can be quickly recognised and acted on during emergency situations.

Crichton (2009) provided a detailed account of planning and conducting TDGs and a suggested protocol for conducting post-TDG debriefs. No evaluation studies of TDGs in emergency management decision-making contexts could be located in this study.

Tabletop exercises

Tabletop exercises can be considered as extensions of TDGs in that they are more complex simulations of longer duration and involve a greater number of participants with a wider range of operational roles. Dausey et al. (2007) evaluated 31 tabletop exercises in the US involving simulated responses to human-made and naturally occurring public health threats. They proposed 6 lessons learnt for designing and conducting tabletop exercises being that exercises should:

• be designed to achieve a specific objective
• be as realistic in content as possible while remaining logically feasible
• be designed around problematic issues rather than scenarios
• be conducted so that decision-making is forced, targeted and time-delineated
• involve a limited number of participants
• be designed and executed to benefit from collaborative engagement of representatives from other likely participating agencies and external (to the sponsoring agency) developers and facilitators.

Operations-based training and exercises

Drills

Skyrabina et al. (2017) described drills as coordinated and supervised exercises to test a single operation or function. Drills are widely used to provide training on new equipment or systems, to develop or evaluate new protocols or procedures and to practise and maintain current skills (AIDR 2023). There is limited published research that considers the effectiveness of drills in emergency management.

Skyrabina et al. (2017) outlined some research that measured changes in individual performance. However, these studies relied on self-reports and did not evaluate improvements in decision-making skills.

Table 2: Berlin and Carlstrom (2015) Three-Level Collaboration Exercise.

Source: Berlin and Carlstrom (2015), p.260.

Emergency management centre exercises

Emergency management centre exercises involve personnel who work in teams at strategic levels of emergency management. The exercises can include people from multiple agencies. In such exercises, the emergency management centre (e.g. a regional coordination centre) manages a simulated incident or set of incidents as if they were managing a real operation. Actors may be employed to play external personnel (such as police or the media) or to simulate radio traffic on the fire-ground (cf. Bearman et al. 2023). Like drills, there is limited research published on the effectiveness of emergency management centre exercises. The studies located relied on self-report measures and do not evaluate changes to decision-making skills (e.g. Perry 2004).

Field exercises

Field exercises have a long history of use for training personnel and are the main vehicle for training staff in multi-agency responses. Berlin and Carlstrom (2014) evaluated 19 Swedish multi-agency (police, ambulance, fire and rescue) collaborative field exercises and found only limited evidence that the exercises improved inter-agency collaboration. Many of those interviewed reported learning little from the exercises. Following these findings, Berlin and Carlstrom (2015) developed a Three-Level Collaboration Exercise model. The model has 6 activities as shown in Table 2. The model was evaluated over 7 exercises involving 178 personnel. Data were returned by 147 participants who rated the levels of collaboration, learning and usefulness positively overall.

Resources for operations-based exercises

The US Federal Emergency Management Agency (FEMA) published a Preparedness Toolkit that provides detailed guidance and templates to assist in the planning, conduct and evaluation of disaster preparedness exercises (FEMA n.d.). While these are tailored specifically to the US situation they could be adapted for use by Australasian emergency management organisations. The Australian Institute for Disaster Resilience released an updated version of its Managing Exercises handbook (AIDR 2023). The handbook provides an overview of exercising, exercise types, exercise documentation, conducting exercises and exercise evaluation.

E-based training and exercises

Computer-supported simulation exercises

A training platform used widely by law enforcement agencies but also by some fire, military and humanitarian agencies is the Hydra Foundation Critical Incident training and debriefing (Alison et al. 2013; Eyre et al. 2011; Hydra Foundation 2022). The only research that could be located about Hydra and training was a PhD thesis (Davies 2013) that examined the effects of simulation-based training on decision-making by New South Wales Police recruits. Davies (2013) found that, in terms of aiding transfer of learning, the most important element of the simulation for experienced police officers was the nature of the task. However, for novice police students, perceived realism was the most important element.

XVR is another computer simulation platform that is widely used for incident management training in the emergency management sector. Over 80 mainly emergency management organisations in 50 countries use XVR to train their personnel (LearnPro 2022). Lamb et al. (2014) outlined how a fire and rescue service in the United Kingdom uses XVR in conjunction with Hydra to develop and maintain incident command skills. No evaluation studies of XVR could be located in this study.

Virtual reality (VR) simulation exercises

Several VR systems have been developed for use in emergency management. The On-Line Virtual Environment (OLIVE) allows users to create persistent virtual worlds where participants can collaborate over networks to train in strategic response to complex emergency scenarios (Chen 2014). However, in an evaluation of OLIVE following an exercise involving responses to a flood emergency, the majority of participants reported that they did not learn as much as they did during traditional field simulation exercises. Prasolova-Forland et al. (2017) provided a detailed account of the development of a VR system for operational-level emergency management training (tasks and judgements). Participants reported satisfaction with the experience; novices more so than experienced practitioners. Tena-Chollet et al. (2017) conducted a survey of VR systems providing training in emergency management at tactical and operational levels in several countries. They concluded that a major advantage of using virtual environments was the repeatability of scenarios allowing participants to see the consequences of alternative courses of action.

In a review of evidence for training effectiveness using VR technology, Abich et al. (2021) evaluated research spanning domains of safety and emergency response (although research from the medical field dominated the review). VR was defined as a system that presented 3D computer-generated graphics requiring the user to fully interact with a virtual environment. Three forms of VR technology were distinguished: head-worn display (HWD), head-mounted display (HMD), and cave automatic virtual environment (CAVE). Training effectiveness was assessed across 3 learning domains of psycho-motor skills, spatial ability and knowledge acquisition. Across all 3 domains, use of VR technology was found to be generally more effective than alternative training methods such as manuals and multimedia presentations. Potential limitations included individual vulnerability to motion sickness, time required for trainees to become familiar with the technology and the limited number of scenarios available because of the development costs.

Khanal et al. (2022) reviewed the literature on VR, augmented reality (AR) and mixed reality (MR) technology used in emergency management. The review covered applications including hazard modelling, intelligence gathering and training. The literature on VR-related platforms for emergency management decision-making training is limited, but the overall conclusion is that successful current applications have been aimed at developing individual skills in particular operational settings (e.g. railway operations) and familiarising novices with visual aspects of hazards in emergency settings (e.g. mining hazards). No examples of use of the technologies to develop emergency management decision-making skills at the tactical or strategic levels were cited apart from the OLIVE platform discussed by Chen (2014).

General principles for simulations

Crichton (2017) distilled 5 principles for using simulation-based training exercises to improve team effectiveness based on extensive experience in simulation-based training to improve operational safety in emergencies. These principles are:

• develop appropriate learning objectives and expected performance standards
• train the team as a whole
• develop and use appropriate structured observation tools
• use the observation tools to provide feedback during a structured debrief
• repeat the simulation-based training exercise regularly to enhance expertise and retain performance standards.

The Australasian National Council for Fire and Emergency Services (AFAC) published ‘Building capability through simulation: Research insights into good practice’ that outlines some of the key issues to be considered in planning, designing and evaluating simulation-based exercises for training incident management personnel (Hayes 2015).

A review of simulation-based training in the US Army by Strauss et al. (2019) concluded that the Army’s training emphasis was on simulation equipment and platforms rather than on learning design. Strauss et al. (2014) observed that this issue has continued for over 20 years. Salas et al. (2012, p.199) concluded that:

…challenge to training developers and simulator designers is to develop systems that use technology to promote learning. To achieve this goal, there will need to be a shift in focus from the designing of simulation for realism (and hope that learning occurs) to the design of human-centred training systems that support the acquisition of complex skills.

Recent analyses of the US Army’s use of simulation and virtual training shows that these issues continue to be a problem. While the Army continues to increase its use of simulation, it has not developed the performance measures required to understand the right mixture of training methods not to ascertain the return on investment for this type of training (Strauss et al. 2019; GAO 2013, 2016).

Post-incident learning activities—debriefs and reviews

Debriefs are a type of work meeting in which teams discuss, interpret, and learn from a recent event during which they collaborated (Allen et al. 2018). Debriefs provide a mechanism by which individuals and teams use post-incident discussion to learn and improve future performance. Across a range of work domains there is compelling evidence that well-conducted debriefs can improve team performance significantly (Owen et al. 2015). In relation to fire, rescue and other emergency response services, (Allen et al. 2019, p.507) indicated that an important goal of debriefs is to promote a positive safety climate:

The debrief allows teams to reduce ambiguity about an event when proper response to an incident is critical. This retrospective sensemaking is needed in order for team members who may have been physically distributed during an incident to develop a consensus about why and how the incident was managed more or less effectively, and how individual and collective action contributed to its success, failure or near failure.

Allen et al. (2018) cautioned that debriefs do not necessarily occur automatically nor in a well-designed fashion and the conditions that make team debriefs effective are not easy to achieve. They summarised evidence-based practices for effective debriefing in medical teams, which they deemed likely to be applicable for debriefing in most work domains. Similar points were covered by Owen et al. (2015) in the AFAC handbook summarising the evidence for effectiveness of debriefings and after-action reviews. The handbook also provides a checklist and recommended structure for conducting them. In their review of sensemaking and critical decision research, Penney et al. (2022) concluded that the relevant literature supports the usefulness of debriefing and subsequent coaching in developing decision-making expertise. Feedback and self-reflection ‘…appear vital to the development and maintenance of expertise by facilitating the restructuring of knowledge as experiences and outcomes are appraised and measured against outcomes’ (p.10).

Evaluation of training

It is clear that while some evaluation of training is performed, this is often quite limited. In a review of the literature reporting evaluations of disaster preparedness exercises (the majority of which was concerned with public health emergency preparedness), Beerens and Tehler (2016) identified that there was often a narrow focus on the reactions of participants (favourable/unfavourable) rather than on demonstrable improvements in the capability of the agencies.

To improve the evaluation of emergency management training, a number of tools are proposed. Thielsch and Hadzihalilovic (2020) reported the development of an evaluation survey tool to assess the effectiveness of tactical and strategic command unit training exercises for fire service personnel in Germany. The authors based their approach on the work of Kirkpatrick (1979) who proposed 4 levels of evaluation of workplace training programs:

• Level I: Reactions of trainees (favourable/unfavourable).
• Level II: Learnings by trainees.
• Level III: Changes in trainees’ subsequent on-the-job behaviours.
• Level IV: The impact of training on the organisation’s level of performance.

Thielsch and Hadzihalilovic (2020) proposed that a positive outcome at each level was a prerequisite for a positive outcome at the next level. They also noted that while evaluations at Levels I and II should be carried out as soon as practicable after conclusion of the training activity, evaluations at Levels III and IV could only be undertaken ‘down the track’ sometime after the training activity had concluded. They developed a 25-item evaluation tool focused on Levels I and II: the FIRE-CPX (Feedback Instrument for Rescue forces Education – Command Post eXercise scale).

Working in the Netherlands, van der Haar et al. (2013) developed a 21-item scale for members of on-scene incident management teams to self-assess their performance effectiveness on 5 dimensions of (i) situation assessment, (ii) decision-making, (iii) quality of actions, (iv) goal achievement and (v) errors. Also, Janssen and Vreugdenhil (2015) described an observers rating scheme based on behavioural markers as an evaluation tool for emergency response training exercises (TARCK-it). The rating scheme has 5 aspects of observed team performance during the exercise:

• T - Timeliness – whether activities are completed timely enough to be successful.
• A - Accuracy – whether activities are completed correctly to be successful.
• R - Relevance – whether the activities are relevant for completing one’s task.
• C - Completeness – whether activities are completed to a sufficient extent.
• K - Kosteneffectiviteit (cost effectiveness) – whether the cost of activities that are carried out are in proportion to the gain.

Important points made about training

Salas et al. (2012) make 2 important points about training. The first is that appropriate training works. The second is that the design, delivery and implementation of training programs are all-important. Training program design will be critical to develop expertise in the complex task of decision-making. To develop an effective training program, it is important for organisations to consider how to systematically provide a suitable range of experiences that will enable a practitioner or team to develop (and maintain) a sound understanding of their capabilities and a strong appreciation of the contexts in which they are likely to operate. Salas et al. (2012) emphasised that training should be a systematic process and that organisations need to pay close attention to what happens before, during and after training.

Large-scale emergencies occur infrequently so emergency management personnel are not required to manage them very often (Lamb et al. 2014; Skryabina et al. 2017). This leads to 2 main challenges: retention and generalisation (Ford and Schmidt 2000). Retention issues stem from the deterioration of knowledge and skills over time if they are not used or practised (Woodman et al. 2021). Generalisation issues come from the necessarily limited scope of training exercises that are unlikely to encompass all the demands likely to be posed by actual emergencies. Ways organisations could counter these threats to decision effectiveness have been proposed including (a) providing staff with a range of self-directed and other learning opportunities to maintain knowledge and (b) structuring post-exercise debriefings in ways that encourage development of self-reflective appraisal (metacognitive) skills (e.g. Lamb et al. 2014).

It is clear that organisations need to have systematic approaches to training that includes an understanding of skill retention and generalisation. Woodman et al. (2021) have argued that a systematic approach to training needs to include a robust analysis of what skills need to be trained, based on task decomposition methods and training needs analysis. Part of the analysis includes the rate at which existing skills decay and appropriate skills maintenance schedules.

Discussion

This study examined the practices used in the delivery of emergency management decision-making training and focused on the learning methods used to develop procedural/tacit knowledge and skills. It considered the literature concerned with enabling learning post-incident though debriefing and reviews. The review covered the use of evaluation of decision-making training and exercises.

An important finding of this study was that there were very few published studies that evaluated emergency management decision-making learning activities. For almost all the learning activities reviewed, there was little, if any, published evaluation of the effectiveness of the intervention. In a few cases there was participant self-report data or suggestions on how the specific training or exercise activity could be enhanced. This is not to say that the current training and exercise activities do not support learning. However, without robust evaluation of training and exercise activities it is more difficult for emergency management organisations to justify their decisions on the training systems and technologies they invest in and continue to use. This finding is consistent with the evaluation literature that recognised the significant organisational barriers that undermine the adoption and implementation of robust evaluation practices (e.g. Phillips and Phillips 2016; Russ-Eft and Preskill 2009).

Developments in new technology have enabled the rapid adoption of simulation and VR platforms for training and exercising. These platforms offer advantages for emergency management settings and various emergency management organisations have invested in these. A particular strength of these platforms is the repeatability of scenarios providing the opportunity for participants to see the consequences of alternative courses of action. These technologies provide training for high-risk, low-frequency events, and access to training for regionally located personnel (e.g. Victoria Ambulance online triage simulations and FLAIM fire trainer). However, there appears to be preoccupation with simulation fidelity and limited investment in the learning design to help the acquisition of complex skills such as decision-making. Crichton (2017) provided helpful guidance on the use of simulations and Hayes (2015) highlighted important points for planning, designing and evaluating simulations. However, there is little literature that assesses the effectiveness of simulation and VR and no real guidance on how to best integrate these with other learning methods (see Marlow et al. 2018).

Post-incident debriefing has received significant interest from the emergency management sector over the last 15 years. There is clear evidence that well-conducted debriefs can improve team performance but also an acknowledgment that debriefs do not automatically occur and that enabling an effective debrief can be difficult. Penney et al. (2002) emphasised the usefulness of debriefing and subsequent coaching. Their observation of the importance of debriefing, its links to coaching and the value of reflective practice highlight a further issue. Evidence from Hayes (2018) suggests that the majority of Australian emergency management organisations may not have a formal coaching or mentoring program. Of the 26 emergency management organisations surveyed, Hayes (2018) reported that only 13.5% had a formal coaching program and 26% had a formal mentoring program.

This study found that the published research focused on a particular type of learning activity. Literature that studied the learning system or the effectiveness of integrating different learning activities to support decision-making capabilities could not be located. Training is a systematic process and there is a need to link the various training activities, exercises and experiences to a coherent and planned program to support effective learning. This approach is certainly not new and Ford (2021) noted the foundations for a systematic approach to enabling learning goes back over 90 years to Viteles (1932). There is evidence that emergency management organisations use some tools to support a systematic approach. Hayes (2018) found that 90% of the 26 Australian emergency management organisations surveyed used individual development plans. Emergency management organisations varied as to whether both staff and volunteers (30%) or only paid staff used these plans (35%). A limitation of development plans is that they can be quite general and not necessarily focused on developing specific decision-making capability. Maintaining a robust instructional system requires ongoing evaluation and refinement and this can present a challenge to many emergency management organisations.

Developing capability should be a systematic process that integrates various forms of learning. Developing complex capabilities such as judgement and decision-making needs to be built over time using various forms of learning, requires exposure and practice in a variety of situations and will be strengthened by reflective practice. Thus, it is unlikely that any single learning method will equip an individual with the requisite broad set of declarative/explicit and procedural/tacit knowledge. The dynamic and challenging nature of many incidents makes it very difficult to formulate a fixed set of training protocols for training decision-makers (Cesta et al. 2014). From a human resource development perspective, this could be framed as learning the requisite knowledge, skills, abilities and other personal characteristics (i.e. KSAOs). The acquisition of KSAOs will be enabled through a curated, coherent and integrated development journey using various forms of learning and work experiences.

Based on this review some important points for organisations to consider can be made:

• A range of options for training emergency management decision-making is available using discussions, operational exercises, intermediate methods (E-based exercises, including hybrid computer-supported exercises and virtual/augmented reality exercises) and post-incident learning.
• It cannot be assumed that participation in emergency management decision-making training will result in improved performance. That can only be determined by appropriate training outcomes evaluations.
• Training activities will not, of themselves, result in improved emergency management decision-making: that will only occur if the activity results in learning. Practice does not necessarily make perfect; it may merely make the imperfect permanent.
• The endeavours of trainers and facilitators are crucial to promote learning through stimulating and guiding trainees’ self-reflections on the quality of situation assessments, decisions and actions during an exercise and how these might be improved.
• Decision-making competence is a depreciating asset. It needs to be maintained by opportunities to use it through exercises and sustained by an organisational culture that values and supports learning.
• Emergency management activities will require the involvement of other organisations and appropriately planned and conducted training activities that involve participants from other agencies.

The most constructive action for emergency management organisations to improve decision-making effectiveness is to undertake critical reviews of current arrangements for decision-making training to ascertain:

• Is there an organisation-wide program of emergency management decision-making training, development and maintenance covering all the 4 levels of emergency management decision-making?
• Is the emergency management decision-making program adequately resourced in terms of training staff expertise and material resources?
• Does the emergency management decision-making training program match the operational emergency management decision-making responsibilities?