A monitoring and evaluation framework for disaster recovery programs

In recent years, there has been substantial research into how we can plan for and more effectively recover from disasters. This national discussion on better recovery outcomes is particularly important given the increasing frequency, severity and cost of disasters.

A key aspect of planning for recovery outcomes is evaluating past recovery with a view to understanding what was done well and what could be improved. The Review of Effectiveness of Commonwealth and State/Territory Relief and Recovery Payments: Report to COAG/SCPEM from the National Emergency Management Committee Recovery Sub Committee¹ highlighted that ‘jurisdictions and the Commonwealth do not measure or report on the effectiveness of their [disaster assistance] programs’.

The Recovery Sub Committee recommended the development of a ‘measure of effectiveness’ that could be used to evaluate disaster recovery interventions.

To develop a nationally consistent understanding of ‘effective’ or ‘good’ recovery the subcommittee developed national recovery outcomes of sustainability and resilience:

1. Sustainability: enabling the affected community to eventually manage its own recovery.
2. Resilience: enabling the community to better withstand a future disaster.

These outcomes highlight that effective disaster recovery intervention contributes to the community’s capacity and capability to manage their recovery once government assistance ends. Determining the extent recovery program activities have built sustainability and resilience in a community post-disaster reflects an underlying theory of change in Australian disaster recovery practice that can be summarised as ‘community-led, government-assisted’ recovery.²

Under the auspices of the Recovery Sub Committee, the Australian Government and all state and territory governments developed a National Monitoring and Evaluation Framework that provides a nationally consistent method for assessing whether a recovery program has achieved its intended outcomes. In the long-term, the framework will build understanding on what is and isn’t good recovery practice through evaluative thinking to improve planning and processes.

The aim of the framework

The framework improves disaster recovery programs through the learning that can be obtained by rigorous evaluation (Winkworth 2007, Ryan 2016). In the past, few recovery programs have been evaluated. Where evaluations occurred, there was no consistency and no systematic attempt to build the lessons learnt into subsequent recovery programs. The monitoring and evaluation framework tries to overcome this by providing a common set of recovery outcomes that provides a consistent focus for evaluations and also provides a structure for feeding back the learnings from evaluations. 

The framework can be applied:

• in any type of disaster; natural or human-induced, rapid or slow onset, and is scalable to all disasters regardless of severity
• to individual components of a recovery program, such as a grant scheme to support local businesses, or for the whole program
• in the early stages of a recovery program to assess the extent to which progress is occurring and appropriate processes are in place, or at the end to assess whether overall community recovery has effectively occurred
• by those with little evaluation expertise and by experienced consultants, non-government organisations and other recovery agents.

A program logic for disaster recovery

The framework is organised using a program logic of how the recovery process occurs. A program logic explains the way in which recovery activities are expected to lead to recovery outcomes. It captures the key aspects of a community-led, government-assisted approach, whereby outcomes are realised in the four domains of built, social, economic and environment. Activities to achieve these outcomes are guided by the National Principles for Disaster Recovery, although specific recovery programs will implement these  principles in different ways.

The program logic in the framework emphasises the need for constant feedback between recovery activities and recovery outcomes. It recognises that recovery programs are complex and the nature of ‘successful’ recovery cannot be specified in detail at the start, but partially emerges out of the recovery process itself. The program logic in the framework provides an archetype by which specific program logics for any given disaster recovery program can be based.

Recovery outcomes

The main objective of the framework is to shift the focus of disaster recovery programs and evaluations towards outcomes achieved and away from activities and outputs.

The program logic identified four broad domains of recovery outcomes based on the Community Recovery Handbook (2009).³ A list of recovery outcomes was developed to capture what it would ‘mean’ for sustainability and resilience to be achieved in each of these domains. An example of sustainability outcomes from the social domain is provided in Table 1.

Table 1: An example of recovery outcomes from the social domain.

After extensive discussions with recovery agents and experts across Australia, as well as a comprehensive literature review and review of previous evaluations (Ryan et al. 2016), 61 mid-level outcomes were identified across the four domains.

It must be kept in mind that:

• not all outcomes or domains will be relevant to every disaster; for example, an urban epidemic may not require any recovery outcomes within the environmental domain
• the framework does not organise outcomes into any order or sequence leaves; rather the evaluation determines whether there is any hierarchy or temporal ordering to the outcomes
• ‘below’ the mid-level outcomes will be disaster specific outcomes that operationalise the statements according to the  characteristics of the affected community or communities; for example, ‘existing health clients receive continuity of their care’ may bespecified in terms of the types of health needs, client subgroups or vulnerable groups that characterise an affected community.

The framework provides guidance for evaluation

On the basis of the agreed list of recovery outcomes, the framework provides general guidance for selecting indicators to measure progress toward these outcomes, methodologies for collecting data, especially in a way that includes the community in the evaluation process, and also suggestions for how to monitor and communicate progress toward recovery.

The evidence base

The other significant element of the framework is an online and searchable database of critical elements of past recovery programs and their evaluations. This evidence base is under development and will be hosted at the Australian Disaster Resilience Knowledge Hub.

It will provide a searchable database of the recovery outcomes, past evaluations, indicators of recovery outcomes and activities undertaken as part of past recovery programs. The database will allow recovery specialists to record lessons learnt from the recovery process. The evidence base can be used prospectively by recovery agents to design recovery programs that are outcomes-focused and also by evaluators of recovery programs to plan evaluations. But, insofar as these evaluations will be guided by the framework, their findings can then be retrospectively added to the evidence base so that the knowledge is captured in a way that can inform future recovery planning. It will thereby assist communities, non-government organisations, emergency services providers and governments to understand what does or does not work in recovery. It will also assist in drawing links between mitigation, preparedness and improved disaster outcomes.

Implementing the framework

To ensure the framework meets real-world demands, it is being trialled using actual disaster events in two states; the Pinery bushfire in South Australia and the Mildura storm in Victoria. The framework will be assessed in terms of the extent to which it provided support to up-front planning of recovery programs and also the extent to which it facilitated best-practice approaches to evaluating these recovery programs.

These trials are coordinated by the Evidence and Evaluation Hub at the Australia and New Zealand School of Government and involve two consulting firms to conduct the actual trials. The feedback from these trials will be incorporated into the next version of the
framework, but some preliminary findings are already emerging:

• The framework needs to clarify, by possibly providing examples, how program logics can be constructed for given disasters that capture both the specific characteristics of that disaster recovery program and also the general characteristics represented in the framework’s overarching program logic.
• The framework needs to make clearer that only a subset of outcomes will be relevant to any given recovery program.
• The framework needs to be clearer about how it applies to different types of disasters and also to different scales of evaluative activity.
• Evaluation reporting requirements need to be structured in a way that facilitate easy input into the evidence base.
• The interrelationship between relief and response efforts and recovery efforts as evaluation issues needs further elaboration.
• The list of indicators for recovery outcomes needs to be refined based on the quality of the indicators and group indicators into searchable ‘themes’ that may be of relevance to recovery agents and evaluators.
• Examples need to be provided in the framework that illustrate the principles it embodies.

Next steps for implementation

While, the framework is a significant step to propagating outcomes-based recovery thinking, the framework alone (including the evidence base) will not necessarily change practice. For evaluative thinking and effective learning to take place, the framework needs to be supported by other elements of a coordinated and national approach. The framework needs to be, and to some extent already is, referenced in key government documents such as the National Principles for Disaster Recovery. Appropriate funding for recovery evaluations also needs to be factored into recovery planning.

Of primary importance is the need to build capability for evaluative thinking among disaster recovery agents. This means an ability to think in terms of outcomes rather than outputs, an ability to draw on past experiences and examples and an anticipation of data needs for monitoring and reporting. To this end, an implementation workshop was held in December 2017 that brought together recovery agents from all jurisdictions. The workshop allowed the development of evaluation plans based on the framework by experts in the field of evaluation who provided guidance. This takes the framework from paper to practice where it can materially improve the challenges of improving disaster recovery efforts.

Footnotes 

1. The Recovery Sub Committee was a sub-committee of the Australia New Zealand Emergency Management Committee.
2. This community-led, government-assisted approach is distinct from other possible theories of change that could have been adopted. For example, a ‘wellbeing’ approach could have been used, which would see successful recovery as contingent upon community members reaching an appropriate level of wellbeing across several generally recognised dimensions.
3. The handbook is part of the Australian Disaster Resilience Handbook Collection. The series is available at: www.knowledge.aidr.org.au/collections/handbook-collection