The demands on teams coordinating emergency management at state and regional levels can be considerable. These teams may be supporting multiple incidents and are prioritising resources, liaising with other organisations and managing public interests. Also, during large-scale emergencies, teams will be working under conditions of stress and fatigue, which are known to impair cognitive processes such as memory and decision-making. This paper describes a checklist-based cognitive aid that can be used by teams to help retain their focus on tasks that need to be completed. This checklist is based on a hierarchical task analysis that was developed with emergency management agencies using observations, subject matter expert advice and prototype piloting. The checklist is a simple, straightforward set of prompts that help managers keep track of operational tasks and, thus, helps to reduce mental workload and improve cognition. The checklist can be used as a prompt to help emergency managers address the tasks they have oversight for, as a training and development resource, and as a diagnostic and monitoring tool to assess how well a control centre is operating. This can be assessed in real time and through the after-action review process. The checklist is a flexible tool that can help people better manage emergency response activities.
Based on a presentation at AFAC19 - the annual conference of AFAC and the Bushfire and Natural Hazards CRC.
Introduction
Each year, Australian and New Zealand emergency management organisations coordinate the response to thousands of incidents. State- and regional-level emergency management teams play a central role in coordinating and prioritising the response to and resourcing of more complex incidents, particularly during periods of heightened incident activity. The demands on these state and regional teams can be considerable, requiring the coordination of multiple incidents, liaising with various organisations, assisting in the provision of emergency and public information, and monitoring and sense-making from a range of channels and information sources. Moreover, larger-scale incidents attracts political interest that requires careful management.
Emergency management organisations have generally responded to these challenges by providing clear role statements for key positions and corresponding guidance on responsibilities. However, these guidance materials have lacked systematic development. Scrutiny of emergency management activities over the last 20 years has at various points in time criticised performance. Coronial inquests following the 1998 Linton and 2005 Wangary bushfires highlighted problems with coordination between regional and incident management team (Johnstone 2002, Schapel 2007). In the case of the Wangary fire, there was also coordination issues between the regional and state authorities. The 2009 Victorian Bushfires Royal Commission identified various coordination issues at the state level noting ‘confusion about responsibilities and accountabilities’ (Teague, McLeod & Pascoe 2010, p.8).
An important question for emergency management organisations is how might they provide support to assist their personnel working in state and regional emergency management teams to operate more effectively? One possible approach is to take a systematic approach by using hierarchical task analysis to identify the tasks central to effective state- and regional-level emergency management. This analysis can be used to develop a checklist or aide-mémoire.
Paton and Owen (2013) describe the three layers of incident management in Australia and New Zealand. Each layer has different types of demands and decisions are bounded by differing time scales (Owen 2012). The first layer is the incident layer where first responders and frontline personnel work directly on the incident (e.g. flood or fire). The second is the tactical layer and involves a local incident management team (IMT) coordinating the response to contain and mitigate the incident. The third is the strategic layer that incorporates the activities that occur above the local operational and tactical levels and is undertaken by state or regional teams. These state and regional emergency management teams address issues that are strategic in nature and concern whole-of-government as well as communities. In addition, the state- and regional-level teams are required to consider consequence management for longer-term recovery.
In an address at the 2011 AFAC and Bushfire Cooperative Research Centre conference, the then Queensland Fire Commissioner, Lee Johnson, said that ‘local incident management was well defined and supported by the AIIMS framework [Australasian Inter-service Incident Management System]. However, the strategic emergency domain is less well understood’ (Owen et al. 2014, p.2). Since that time, further research has been undertaken at the strategic incident management level in Australia including Bearman and co-authors (2015), Brooks and co-authors (2018), Owen (2012) and Owen and co-authors (2014). Such research investigated how networks, information flows, coordination breakdowns and errors occur within the strategic levels of emergency management. State and regional teams need to operate in a structured and deliberate manner. At times, because of operational requirements, the team and individual resources are severely stretched and can break down leading to disruptions to team processes (Bearman et al. 2015). As such, Owen (2012) concluded that state and regional emergency management teams would benefit from appropriate tools to help maintain focus and keep track of activities.
Given the demanding nature of emergency management, a suitable approach is to provide cognitive aids to help teams identify tasks and the likely ordering and interdependencies between these tasks. Rosenthal and Downs (1985) describe cognitive aids as tools and techniques that ‘help people detect, interpret, store and retrieve information efficiently’ (p.1). Using such aids helps operators undertaking complex activities to reduce omissions and errors and to improve the speed and fluidity of their performance (Reason 1987, Roth, Mumaw & Lewis 1984).
The use of cognitive aids is also beneficial for people who are working under conditions of stress and fatigue. Working under pressure negatively affects an individual’s thinking and perceptual (i.e. cognitive) processes (McLennan et al. 2014). Memory, a cognitive process central to performance in complex activities, is adversely affected by the pressure and fatigue inherent to incident management. Memory is important for allowing quick retrieval of appropriate knowledge and procedures as well as to remember to undertake tasks and activities in the future (known as prospective memory) (Matthews et al. 2000). The development of a cognitive aid provides a visual checklist that incident managers can use to remind them of the tasks that help to reduce mental workload and support prospective memory. This increases cognitive ability by partly embedding memory in the world rather than relying on mental processes.
A further advantage of cognitive aids (such as checklists) is that they frequently serve to make tacit knowledge that people have about a set of tasks explicit and able to be converted to procedures. Creating procedures allows others to gain insight into what is occurring versus what should be occurring in state and regional coordination centres. While it is reasonably easy to critique tasks that are observed, it can be difficult to identify things that are not occurring. Checklists have proven to be a valuable tool for observers who need to constantly and reliably assess the performance of teams against a standard set of criteria derived from best practice. This is an important tool for system management and continuous improvement.
Emergency management team operations can be assisted by cognitive aids to maintain focus and track key activities.
Image: Country Fire Service, South Australia
Checklist-based cognitive aids have been used in aviation since the 1930s when growing concerns about the complexity of aircraft prompted their introduction (Mellinger 2004). These tools help people make the most of their cognitive capabilities and can be used to enhance an individual’s or team’s decision-making abilities (Engel 2002). In addition to aviation, checklists have been widely adopted in acute medicine where research has shown that checklists improve patient outcomes by reducing time and errors (e.g. Chaparro et al. 2019, Marshall et al. 2016, Stiegler & Tung 2014). Checklists are also used extensively in the nuclear industry (Brooks et al. 2019).
To date there has been limited research on the use of checklists in emergency management. Brooks and colleagues (2019) considered how checklists used in other domains might inform the development and use of checklists in emergency management. The checklists commonly used in aviation, medicine and the nuclear industry tend to follow a prescribed sequence that users step though, completing one task before progressing to the next (Brooks et al. 2019). In contrast, emergency management operations tend to be more dynamic, less structured and non-linear in the way that incidents evolve and develop. Moreover, state and regional teams may be coordinating responses to multiple incidents. Some incidents may be well defined and under effective management, while others may be more chaotic, uncontained and less well understood. This means that checklists developed for emergency management should be guidelines rather than be too prescriptive with tasks able to be carried out in any order.
This study developed a cognitive aid (in the form of a checklist) that defined the key tasks to be carried out in state and regional emergency management organisations. As the checklist was designed to support state and regional management teams, particularly when the team is under pressure, it needs to meet the unique characteristics of individual environments.
Method
A hierarchical task analysis (HTA) was used to develop the checklist. HTA is an analytical tool that can be used for purposes including job design, interface design, error prediction and workload assessment (Stanton 2006). HTA assists organisations to understand fundamental goals, information processing and the cognitive activities that underpin complex activities such as those found in emergency management (Hoffmann & Militello 2014).
Shepherd (1998, p.1537) described HTA as:
a strategy for examining tasks aimed at refining performance criteria, focusing on the constituent skills, understanding task contexts and generating useful hypotheses for overcoming performance problems.
HTA can play an important role in eliciting a deeper understanding of the expertise and cognitive processes at play within a team or system (Shepherd 1998).
Task analyses of state and regional coordination centre (SCC and RCC) helped identify the key tasks to be performed by teams to ensure their responsibilities are effectively considered and managed. The task analyses follows on from the work of Bearman and Bremner (2013) who identified the key tasks that needed to be performed at the incident-control level in a volunteer fire brigade. Bearman and Bremner (2013) used an incident controller task analysis to determine the high-risk activities that are carried out during incident control and identified some of the pressures that may result in poor decisions.
This research received Central Queensland University Human Ethics Research Committee ethics approval, reference no. H15/10-226. Preliminary state and regional tasks analyses were constructed and were developed from observations of state and regional coordination centres, the expertise of the authors and through discussions with agency personnel with experience working at the state and regional level.
The preliminary task analyses were translated into an observation tool, which was further developed and evaluated using an iterative human-centred design cycle approach in a set of four regional control centre exercises. The exercises were based on a full activation of the coordination centre and required the centre to response to one or more large-scale fires. Actors simulated external stakeholders and the radio traffic from the fire ground. Outputs (such as maps and warnings) were produced in the software packages set in training mode. State-level observers evaluated the performance of RCC participants throughout the exercise.
Two observers used the regional coordination centre task analysis to evaluate the performance of the RCC members. This evaluation contributed to the overall performance evaluation conducted by the state observation team. The two observers considered the extent to which each of the tasks in the task analysis were carried out and made comments alongside items where something noteworthy was observed. At the end of each exercise, the two observers met to discuss the tool and how it could be improved. This involved reviewing each of the activities, considering the notes and comments made during the observation, adding aspects that were not being captured and amending the wording of existing activities to better capture the underlying concept. In this way, the tool was improved through an iterative cycle of evaluation and development.
During operations, coordination centre personnel undertake a range of planning, monitoring, and reporting activities. Cognitive aids are used to evaluate and improve these processes.
Image: Country Fire Service, South Australia
Figure 1: Checklist of the key activities required for state and regional-level incident coordination.
Results
The preliminary task analyses identified 75 tasks and subtasks at the state-level and 72 tasks and subtasks at the regional level. Two task analyses (one for state and one for regional) were developed with five phases of activity: Alert, Escalation, Manage Incident, De-escalation and Termination (or Close RCC). Under each of these phases, key tasks were defined that must be carried out to effectively coordinate an emergency at state and regional levels. Each phase has between 3 and 25 tasks or subtasks. These tasks and subtasks have been distilled into the checklist shown in Figure 1.
The first phase is the Alert Phase when the state or regional team is in place because there is an elevated threat of incidents. This period includes ensuring the SCC or RCC team is aware of and monitoring weather conditions, resources and has plans in place to scale up if required. During the Escalation phase the focus shifts to responding to developing incidents, ensuring that the state or regional teams anticipate likely developments and review appropriate resourcing. The next phase, Coordinate Incidents, is the most active period and has the most tasks with the requirement to coordinate multiple operations and to liaise with other agencies and to coordinate public information. The De-escalation phase covers the period of decreasing intensity of incident management activities. Although incident management operations are reducing, this phase requires careful sequencing of decisions to gradually wind down activities and resourcing. The final phase is Termination or Close of the coordination centre. This phase has the fewest number of tasks and focuses on wrapping up the centre’s activation.
Figure 1 provides a checklist based on the key tasks and subtasks in the agencies that were studied. The actual tasks and subtasks required in regional and state coordination centres will be different depending on the agency to which the checklist is being applied. Figure 1 suggests a logical order in which to undertake the tasks and subtasks for each phase. However, given the evolving nature of incidents, it is most likely that managers will cycle through the checklist a number of times during each phase, especially if the situation is fluid or still emerging. Although the checklist suggests a logical sequencing of tasks and activities, the order in which some of these are tackled may vary depending on the particular circumstances. Checklist users may find it helpful to identify the status of each task by using a traffic light coding system of green (G) for good or in-hand, amber (A) for marginal or incomplete and red (R) for not yet addressed.
Discussion
The checklist-based cognitive aid presented in this paper assists incident managers by providing a framework of the key tasks required to coordinate emergency management activities at the state and regional levels. Further research could validate the checklist, however, it can be used by agencies in at least three ways.
Aide-mémoire
The simplest use of the checklist is as a prompt to help emergency managers check that they are addressing the tasks required to coordinate the control centre and the incidents they have oversight of. This is particularly important when the team is working under conditions of stress and fatigue and helps to reduce mental workload and increase cognitive ability.
The checklist is also useful for personnel developing their incident management capabilities and for personnel who have not worked in these roles recently. The experienced practitioners who used the checklist during the pilot phase identified its value in helping to stay on track with tasks and activities required.
It is evident that such tools are helpful in improving performance of individuals and teams (Chaparro et al. 2019, Marshall et al. 2016). This is especially so for complex tasks such as those required in state and regional-level emergency management (Brooks et al. 2019).
An important difference between emergency management and other sectors that use checklists is the fluid nature of an emergency situation. Emergency management teams operate in dynamic environments that are likely to have less structure. For example, the number, scale and complexity of incidents may rapidly change. Also, an emergency management team may be required to concurrently manage multiple incidents that may be at different points of development. These incidents may be the same hazard type or they may be different (e.g. a bushfire and a flood). Such conditions mean teams must work simultaneously across varying temporal and spatial scales (Brehmer & Svenmarck 1994). This means that some tasks within a phase of the checklist will be revisited multiple times and the various incidents may be concurrently managed using different checklist phases. It is strongly recommended that each incident has a separate checklist to allow for careful tracking of the phases of each incident. Such high tempo, complex and demanding workload conditions create an environment where important tasks might be overlooked or there is difficulty in sequencing interdependent tasks.
Emergency incidents can occur with no or little warning, which requires the emergency management team to operate from a ‘cold’ start. In such cases the incident starts from the Escalation phase rather than the Readiness phase. When this occurs, teams could overlook some of the tasks that are usually undertaken in the Alert phase. To address this issue the checklist can be used to identify the tasks in the Alert Phase not considered in the Escalation phase such as reviewing the precautions or restrictions in place and checking for existing information relevant to the current incidents.
Training and development resource
The checklist outlines several important aspects of emergency management and coordination.
- It outlines the phases of an incident and maps the tasks required.
- It captures the tasks required to coordinate the control centre and the incidents.
- It provides a suggested hierarchy of the likely sequencing and priorities for the tasks.
These aspects of the checklist can be used to improve instruction in regional and state coordination functions and in face-to-face and online training settings. Emergency managers, trainers and coaches can use the checklist as a diagnostic tool and to help structure feedback and discussion with personnel during exercises, warm starts and on the job. This can help new personnel to quickly transition through developing the skills and expertise required in their roles.
Continuous improvement
The checklist can be used to help consider how well an SCC or RCC is operating as part of continuous improvement programs. For after-action reviews, the checklist can be used to facilitate review and guide discussion of the arrangements made during a shift or period of activity for a control centre. The checklist can provide structure to discussion about the various aspects of a control centre’s operation.
The checklist-based cognitive aid presented in this paper is a useful tool, however, there are a number of limitations. Brooks and colleagues (2019) highlight that while there is good evidence for the utility of checklists and other cognitive aids, effective implementation can be challenging. Highly skilled practitioners may feel that consulting a checklist might undermine how others view their competency and see no need to use checklists (Catchpole & Russ 2015). Brooks and co-authors (2019) suggest that it is important to distinguish between cognitive aids and the decision-making processes of users (Kim & Reeves 2007), noting that cognitive aids help facilitate decision-making that is based on the expertise of the practitioner, such as the intended use of the checklists presented here.
It has also been observed that some aide-mémoires may be overcomplicated or lead to a superficial tick-and-flick approach (Brooks et al. 2019). These observations can be addressed by good checklist design that is based on empirical investigation of the domain of intended use and an iterative design and evaluation method. Investigations by Alidina and colleagues (2018) of the organisational and contextual factors influencing the adoption of checklists during surgical crisis events also identified several barriers. These included factors such as a limited appreciation of the vulnerability of decision-making in stressful situations and organisational factors such as limited leadership support and inadequate training in the use of the aids.
The checklist presented here has received emergency management organisational support and has been incorporated into the South Australian Country Fire Service (CFS) standard operating procedures for conducting and managing real-time evaluations (SOP 12.4). The checklist has also been used to identify the functions of a CFS State and Regional Control Centre specified in Standard Operating Procedure 1.05 and 1.06. However, more work is required before the checklist is widely accepted and used across the organisation.
Conclusion
This paper describes the rationale for and the development of a checklist-based cognitive aid that was designed to support state and regional emergency management teams. The checklist is a description of the key tasks that must be carried out in state and regional coordination centres during an emergency. As such, it is a list of things ‘that you just can’t forget to do’. The checklist is designed to assist teams working under conditions of stress and fatigue. It can be used for training and development, it will benefit people who are new to working in state or regional coordination centres and can be used for the purpose of continuous improvement.