The climate fire weather outlooks for the new Australian Fire Danger Rating System (AFDRS) are the first of their kind in Australia. We engaged with critical stakeholders to guide the product development process for the graphical communication of fire danger risk for climate timescales (monthly and seasonal).
The many challenges
The main goal of the AFDRS climate service is to present complex datasets as actionable information. This is not without its challenges, which the AFDRS product development team encountered when producing accessible, graphical depictions of statistical data for a non-specialised audience.
While surveys showed that there is a strong cross-industry demand for fire-weather outlook products, not all sectors have the capacity or expertise to effectively adopt such products into decision-making processes. This posed the first challenge: obtaining ‘use cases’. Knowing what type of information would be used, how and when it would be used (use cases), is foundational to the product-design process so that the products would be suitable for end users. Obtaining use cases was not straightforward given how novel the products are. Many sectors did not have the specialist expertise available to discuss interpretation and operational use.
The second challenge was managing expectations around climate data for groups who regularly use weather data. Most stakeholders with an interest in fire-weather outlook products regularly make decisions based on short-timescale weather information. While weather and climate products describe similar phenomena, they are based on different types of data due to the nature of the modelling used to produce them. This subtle but significant difference often clouded and confused discussions about how products could be incorporated into operational decision-making processes.
The third challenge was producing graphical, non-dynamic products that meet information needs. The necessary trade-off between information content and simplicity was complicated by the limitations inherent in the AFDRS dataset, which restricted what information could be drawn from the data in a scientifically sound manner.
A baseline for user needs
Stakeholder guidance was essential to the design and development process. The aim was to tailor output products to best meet user needs and encourage successful adoption of the fire-danger outlook service. Stakeholder engagement occurred through regular meetings, surveys and targeted workshops. We commenced the design process with a set of desirable attributes of potential fire-weather outlook products:
- Relevant and actionable: the information should be relevant to decision makers and be provided in ways that allow it to influence decisions and improve outcomes.
- Scientifically sound: the forecasting methods should be peer reviewed, scientifically documented and supported by sound science.
- Verifiable: the performance of outlooks should be assessed against subsequently observed outcomes.
- Probabilistic: forecasts should faithfully represent future uncertainty.
- Clearly defined: the meaning of forecasts should be clear and understandable.
Some prototypes (e.g. Figure 1) were developed through a previous project that focused on the forest fire danger rating (ERP-14). The lessons from that project were adopted by the AFDRS product development team. The spatial, map-based products based on existing Bureau of Meteorology products were generally received well. Products that depict temporal information (such as trends over time), however, were not particularly well received, despite being explicitly requested by participating stakeholders. These products required significant refinement.
Lessons learnt
Feedback from the survey was used to improve the design of temporal outlook prototypes. The new prototypes have been readily welcomed in initial stages of introduction and are being assessed by a larger stakeholder group through a survey.
The 3 main lessons from the initial stakeholder survey:
- ‘Familiar and simple’ is better understood and more likely to be adopted by users.
- What customers request and what they can adopt and use operationally may be quite different.
- Many potential users are focused on tactical and near-term operations (next few days), using weather forecasts to support decision-making process; interpreting and using climate information is very different to interpreting and using weather information as the temporal scale (how the information is portrayed) and the use cases are different.
Applying lessons to guide design
Each of the lessons were crafted into guidelines to aid the development process. The 3 resulting guidelines were:
Development guideline 1: Build incrementally on a familiar knowledge base to create new products. These new products require some statistical interpretation skills. If the same kind of analysis could be applied to each product, the necessary learning required by end users to effectively use the full product suite will be reduced. Simple, correctly interpreted plots will be more effective than complex plots.
Development guideline 2: Understand what users want to know and why users want to know it. The specialised statistical understanding required to develop and interpret products cannot be justly assumed of the user. To ensure the products have the desired characteristics, those with statistical expertise need to apply it to the context of the dataset and the operational needs of the target groups to develop products that are scientifically sound and accessible without extensive end user training, while striking the balance between information content and usability.
By understanding the 'what' and 'why' that users need, we can draw on the data to present it in a way that supports decision-making on these timescales. This means that the family of products are relevant, fit-for-purpose and adopted into operational processes. It is envisaged that the successful take up of these products will rely on support to interpret them during the initial stages of the service.
Development guideline 3: Create a specific focus group with those involved in planning and decision-making given the monthly to 3-monthly lead times of the climate products. In most cases, these products are completely new to this audience, so it is difficult for many users to imagine how they might be applied operationally. It is particularly difficult when potential users are used to making decisions based on weather information (from 1–7 days in advance). Given this, the expectations of these users need to be managed around what kind of scientifically supported information could be drawn from the new AFDRS climate dataset. While it is important that the new products are broadly useful, the needs of the most relevant ‘product user groups’ must be prioritised in the engagement process.
Figure 1: Chance above median forest fire danger rating, similar to Bureau of Meteorology operational rainfall and temperature products, data from 1 Nov 2019.
Image: Bureau of Meteorology
Prototype development
We applied these guidelines to create the product suite. The direct application of each guideline resulted in a specific, but interlinked, set of product characteristics.
Prototype result 1: Based on user requirements, we created tercile temporal plots (Figure 2). Although tercile data is quite technical in its creation, it has a relatively simple interpretation of 3 categories of fire behaviour index: below, about and above average (normal). This ‘product pull’ by the users was built on the reception to and acceptance of the need for tercile spatial plots in the initial survey. They were not as popular (nor as easily interpreted) as the ‘chance above median’ products, but the general stakeholder group was regarded as having a basic ability to interpret and use this sort of information in the plots. Tercile information is more appropriate than chance above median for displaying temporal information and variations over time.
Figure 2: Tercile plot and timeseries for a NSW location, illustrated using Forest Fire Danger Index data from 1 Nov 2019.
Image: Bureau of Meteorology
Prototype result 2: We developed a broad understanding of what information users need and why. In general, this cohort needed to answer the following question: ‘should our operations and planning activities deviate from the usual?’ Given this, we developed readily interpretable plots that could aid in broad operational interpretation of the following:
| Above average fire behaviour index |
→ |
Escalate operations |
| About average fire behaviour index |
→ |
Maintain operations |
| Below average fire behaviour index |
→ |
De-escalate operations |
A more nuanced approach is necessary for each specific agency decision, but this broadly applicable idea links the outlooks into an operational decision-making and planning process.
Prototype result 3: A focused target group was nominated for initial feedback consisting of at least one longer-term strategic planner from each jurisdiction. Two workshops provided first-impression feedback and then considered feedback on the products. Both sets of feedback offered unique and valuable insights into how the temporal product could be improved for operations.
Results
It is necessary to adopt scientific advances for best practice in delivering outcomes to support the safety of communities and efficient emergency services planning. Stakeholder engagement allowed us to design products that align with user requirements and skills.