A formative evaluation of the Triple Zero Kids’ Challenge Teacher’s Guide

Abstract

The Triple Zero Kids’ Challenge is an online, interactive safety game and a mobile application. It provides young children in Australia with essential information on how to identify and report legitimate emergencies by calling Triple Zero (000). As a companion resource, the Triple Zero Kids’ Challenge Teacher’s Guide provides educators of lower-primary school students with a series of structured learning activities to consolidate and extend the key messages of the online game. To ensure that the learning activities in the guide are both feasible and appropriate for the target age group, a formative evaluation was conducted with lower-primary students and their teachers. This paper reports on the results of the evaluation and highlights the importance of formative evaluation to the development of safety education programs for children. While the evaluation indicated that the learning activities were feasible and appropriate for lower-primary school students, it also identified the need for numerous modifications and improvements that have been incorporated into a revised 2017 version of the guide.

Introduction

Triple Zero (000) is Australia’s emergency number for requesting urgent assistance from police, fire or ambulance services. When an emergency situation is life-threatening or time-critical, calling Triple Zero facilitates the rapid dispatch of the appropriate emergency service. Given that young children are sometimes the sole bystanders during an emergency situation, educating them on the effective and responsible use of the Triple Zero emergency number is an essential component of inclusive community safety. While scholarly research on children’s use of Triple Zero is scarce, the annual Junior Triple Zero Hero Awards provide many examples where children as young as four years old have saved lives by calling Triple Zero (Emergency Services Telecommunications Authority 2017). Through the delivery of quality education programs that effectively teach young children how to use the Triple Zero emergency number, there is the potential for more lives to be saved.

Identifying a distinct gap in the provision of Triple Zero education for Australian children aged between four and seven years, the Triple Zero Awareness Working Group (TZAWG)¹ developed the Triple Zero Kids’ Challenge. The Triple Zero Kids’ Challenge is an interactive online game and mobile application (app) that provides children with the knowledge and skills they need to identify and report legitimate emergencies by calling Triple Zero (TZWAG 2017). To further support children’s learning, the TZAWG developed the Triple Zero Kids’ Challenge Teacher’s Guide that consists of structured learning activities to consolidate and extend the key messages of the online game (TZAWG 2017). While the learning activities directly address the serious topics of major accidents, medical emergencies, fires and serious crimes, they are designed to be enjoyable and engaging for young children. Taken together, the learning activities cover all of the key concepts and procedures involved in calling Triple Zero, including the kinds of situations that constitute legitimate emergencies, the correct number to dial, the information to provide to operator and the consequences for making hoax calls.

Classroom delivery of the core learning activities in the Teacher’s Guide takes approximately six to eight hours and, for this reason, every activity has been designed to align directly with the Health and Physical Education component of The Australian Curriculum (ACARA 2017) (see Table 1).

Table 1: Alignment to the Health and Physical Education component of The Australian Curriculum.

To ensure that the learning activities in the Teacher’s Guide were feasible and acceptable for lower-primary students, the design process incorporated a formative evaluation. Formative evaluation is undertaken during the design of a new program or when an existing program is implemented in a new context (International Federation of Red Cross and Red Crescent Societies 2011). It allows for modifications to be made to the program before full implementation begins and maximises the likelihood that the program will succeed (Administration for Children and Families 2010). In the context of emergency education, formative evaluation is especially important because poorly designed learning activities can have life or death consequences. Indeed, recent research has demonstrated that emergency education programs can unintentionally increase risk by creating confusion or reinforcing misconceptions about appropriate emergency response procedures (see Johnston et al. 2014 for examples from the context of children’s earthquake education). Despite this, emergency education programs for children are rarely subjected to formative evaluation during the design phase (Towers et al. 2016).

For the development of the Teacher’s Guide, a formative evaluation was especially important because empirical research that could be applied to the design of the learning activities was distinctly lacking. While various organisations around the world have developed programs and resources for educating children about their national emergency call service, an extensive search of the literature located only four published program evaluations. Two of these evaluations (Jones 1980, Rosenbaum, Creedon & Drabman 1981), both conducted in the USA, found that children aged between four and five years could be effectively trained to make an emergency call. However, the content validity of those studies has been eroded by dramatic advancements in telecommunications over the last several decades (i.e. from the demise of rotary dial landlines and rise of smart phone technology). Two more recent evaluations, one conducted in Australia (Wilks et al. 2016) and one conducted in the USA (Morgenstern et al. 2007), were both focused on programs for children aged between 11 and 13 years. However, results in both studies indicated negligible program effects because the child participants already knew how to make an emergency call.

This paper presents the findings of the formative evaluation of the Triple Zero Kids’ Challenge Teacher’s Guide. In doing so, it demonstrates the fundamental importance of formative evaluation to the design of quality safety education programs for children. Given the lack of any relevant recent research in this area, it also addresses a significant gap in the literature and provides a foundation for future evaluations of program effectiveness.

The Triple Zero Kid’s Challenge Teacher’s Guide can be downloaded from the website (http://kids.triplezero.gov.au/).

Methods

The evaluation of the Teacher’s Guide was conducted in a primary school in the south-eastern region of metropolitan Melbourne. Three Foundation Year teachers were provided with a pilot version of the Teacher’s Guide and asked to deliver the lessons during class time. Upon completion of the lessons, students from each class were invited to volunteer for an interview. Of the 60 students who were invited to participate, 22 students (11 male and 11 female) returned a consent form that had been signed by their parent or guardian and were included in the evaluation. All student volunteers were aged between five and six years.

The evaluation employed a predominantly qualitative approach to data collection. While formative evaluation can involve quantitative or qualitative methods, it lends itself strongly to the latter (George & Cowan 2004). In contrast to quantitative or experimental approaches, a qualitative approach allows for the emergence of unexpected outcomes and processes, which increases opportunities for program modification and improvement (Patton 2014). A qualitative approach is particularly important for understanding how young children are interpreting key safety concepts and provides a strong basis for redesigning learning activities that directly address common misconceptions and knowledge gaps (Johnson et al. 2014, Towers 2015).

Interviews with the children mostly involved semi-structured open-ended questions that allowed them to articulate their knowledge from their own perspectives. However, to supplement the qualitative data and provide a quantitative indication of the accuracy of children’s knowledge, several structured closed-ended questions were included. To help create a relaxed environment in which the children felt comfortable communicating their perspectives in their own words, they were interviewed in pairs or groups of three (Eder & Fingerson 2002).

The three classroom teachers (all female) who delivered the lessons were invited to provide feedback on the Teacher’s Guide via open-ended questions in a written survey. This survey asked the teachers to identify the key strengths and weaknesses of the learning activities and share their perceptions of student learning, engagement and enjoyment. The survey sought teacher recommendations for improving the structure and content of the guide.

Approval for the evaluation was granted by the RMIT Human Research Ethics Committee and the Victorian Department of Education.

Results and discussion

Children’s knowledge and skills

If children are to be competent and responsible users of the Triple Zero emergency call service they need to correctly identify legitimate emergencies when they occur. The Teacher’s Guide includes learning activities aimed at increasing children’s knowledge and skills in this area. To assess the viability of these activities, the children were asked to describe an emergency. They responded with various examples of legitimate emergencies, including major accidents, medical emergencies, fires and serious crime. Box 1 provides a selection of representative responses.

To explore the children’s knowledge of what constitutes an emergency, they were presented with a series of pictures depicting a range of emergency scenarios, including medical emergencies (e.g. a man collapsed and he can’t be woken up), fires (e.g. a fire in the kitchen can’t be extinguished) and serious crimes (e.g. a robbery taking is taking place next door).2 For each scenario, the children were asked to judge whether the scenario depicted was an emergency. Children judged correctly with a high degree of accuracy (98 per cent of responses were correct). Moreover, when asked what should be done in the event of such an emergency, all children responded with ‘call triple zero’.

The Teacher’s Guide also contains activities designed to help children distinguish between ‘big accidents’ that constitute legitimate medical emergencies and ‘small accidents’ that constitute minor injuries that can be adequately treated at home. To assess the viability of those activities, children were asked to describe the difference between a ‘big accident’ and a ‘small accident’. In their responses, some children made the distinction by providing examples of each, while other children made the distinction by suggesting the appropriate level of response. Box 2 provides a selection of representative responses.

To explore the extent to which the children could distinguish between ‘big accidents’ and ‘small accidents’, they were presented with a series of scenarios that were representative of each (e.g. ‘Dad has fallen off the roof and he can’t move’, ‘The boy has fallen off his skateboard and scraped his knee’). For each scenario, the children were asked to identify whether it was a ‘big accident’ or a ‘small accident’. They were able to do this with a high degree of accuracy (98 per cent of responses were correct). When prompted, the children were able to correctly identify that a ‘big accident’ would require a call to Triple Zero, whereas a small accident would not.

Lesson four of the guide is about big and small accidents.

The Teacher’s Guide also includes activities for teaching children how to determine whether or not someone is unconscious. It does so for two reasons. First, if a child is alone with a parent or guardian who is unconscious they must quickly identify this as a medical emergency and call Triple Zero. Second, if a child can make an accurate assessment and relay it to the Triple Zero operator, the call can be triaged more effectively. When asked to explain the meaning of ‘unconscious’ most children accurately identified the two key signs that are:

• the person is not awake
• the person cannot be woken up.

Box 3 provides a selection representative responses.

Of note, two children suggested that if someone is awake and cannot breathe, they are unconscious. In light of this finding, the Teacher’s Guide was revised so that teachers are made aware of the potential for children to misunderstand the meaning of unconscious.

While the children could identify the scenarios that require a call to Triple Zero with a high degree of accuracy, the interviews revealed some confusion concerning the actual number that should be input into the phone. When asked to dial Triple Zero into the keypad of a toy phone, all of the children entered the correct number of zeros (three). However, when asked to verbally recite the number, at least three children responded incorrectly with ‘zero zero’. The Teacher’s Guide was revised so that teachers are aware to consistently reinforce that the Triple Zero emergency number includes three zeros.

A further issue was revealed when one child said they would be able to call Triple Zero because they knew the password for their parent’s mobile phone. While it is possible to call Triple Zero without unlocking the keypad or keying-in a personal identification number, this had not been made explicit in the pilot version of the Teacher’s Guide. Indeed, a subsequent scan of various educational resources from other countries found that this issue is frequently overlooked. Given that 5.78 million Australians (about 31 per cent) have no fixed landline at home (Australian Communications and Media Authority 2016), it is increasingly likely that if a child needs to call Triple Zero, they will need to do so from a mobile phone. Hence, the Teacher’s Guide was modified to provide specific advice on how to call Triple Zero from a locked mobile phone.

Teaching young children how to call Triple Zero comes with a risk that they will test out their newly acquired knowledge by making nuisance or hoax calls. Hoax calls place a significant burden on the emergency call service. People making falsified, mischievous or hoax calls can face significant fines or prison sentences. Given that children are responsible for a significant proportion of hoax calls, the Teacher’s Guide has a full lesson devoted to this topic. While only a few children were familiar with the term ‘hoax call’, when asked what would happen if someone called Triple Zero in the absence of a real emergency, all children demonstrated a clear understanding of the serious repercussions for both the caller and the emergency services. Box 4 provides a selection of representative responses.

In the event of a legitimate emergency, children need to understand the circumstances under which they should call Triple Zero: namely, when no other adults or older siblings are present. The Teacher’s Guide explicitly addresses this by teaching children that the first step in an emergency is to check if an adult or an older sibling is available to help. Most children demonstrated a clear understanding of the circumstances under which they should make the call themselves. Only one child exhibited some confusion when he stated ‘But we’re still kids, and kids are not allowed to call Triple Zero until Grade 1’. Box 5 provides a selection of representative responses.

When calling Triple Zero, the caller needs to provide the operator with several pieces of key information. In the first instance, they must be able to inform the operator of which emergency service they require; police, fire or ambulance. The Teacher’s Guide includes activities designed to increase children’s knowledge of which emergency service to ask for. In the interviews, when asked which emergency service they would ask for in the event of fire, a medical emergency or a serious crime, all of the children responded correctly with ‘fire brigade’, ‘ambulance’ and ‘police’, respectively. Additionally, throughout the interview, when presented with types of emergency scenarios, the children correctly identified which emergency service would be required. They did so with a high degree of accuracy (97 per cent of responses were correct).

When calling Triple Zero, the caller also needs to inform the Triple Zero operator of the precise location of the emergency, including the state, town and street address. It is also helpful if the caller can provide the phone number they are calling from. The pilot version of the Teacher’s Guide included a ‘My Phone Number and Address’ card for children to take home and complete with their parents or guardians as a homework activity. However, the interviews revealed that this approach had been largely ineffective. While 13 of the 22 children interviewed knew what town they lived in, only five knew their full street address. Furthermore, only one child knew which state they lived in and only one child could recite their phone number. These findings indicated that relegating this learning to a homework activity represented a significant gap in the Teacher’s Guide. This lesson has been revised to include class-based activities to help children learn their state, town, street address and phone number.

Calling Triple Zero (000) from mobile phones does not require the phone to be unlocked.

Teacher feedback

The teachers judged the Teacher’s Guide and associated lessons to be of a high quality. They felt the lessons were well structured and that the learning objectives were clear. They also reported that the content was age-appropriate and the learning activities were easy for the children to understand. When asked about their perceptions of student learning, the teachers reported that the activities in the Teacher’s Guide had been highly effective in building children’s knowledge of what constitutes an emergency and how to respond appropriately. Box 6 provides a selection of representative responses. Teachers also reported a high level of student engagement in the learning activities. They attributed this to the interactive, game-based approaches, which were described as being ‘highly enjoyable for the children’. Providing further evidence of student engagement, one teacher reported that ‘the children often asked to do the program’.

The teachers strongly agreed that the learning objectives and activities in the Teacher’s Guide align with the Health and Physical Education component of The Australian Curriculum and they cited this as a key strength. Specifically, the Health and Physical Education curriculum for Foundation Year, Grade 1 and Grade 2 includes a learning strand on personal, social and community health, which is largely focused on identifying and responding to emergencies (ACARA 2017). The teachers explained that this direct alignment is essential because an overcrowded curriculum significantly constrains opportunities for the delivery of stand-alone programs. While the pilot version of the Teacher’s Guide had included an extensive table detailing the links across The Australian Curriculum, it was hidden away at the back of the resource. In an attempt to promote increased uptake, the introductory message at the beginning of the Teacher’s Guide was revised to ensure that the direct link to the health and physical education curriculum is readily apparent to teachers.

The teachers also identified the use of information and communication technology (ICT) (i.e. the online game scenarios) as a key strength. Moreover, they identified a valuable approach to integrating ICT for enhanced lesson delivery. In the pilot version of the Teacher’s Guide, teachers were encouraged to prepare their students for each lesson by encouraging them to play the corresponding online game. However, the teachers employed an alternative approach. They commenced each lesson by presenting the corresponding online game scenario to the whole class via an electronic whiteboard. In classrooms where there is limited access to computers or other digital devices, or when there isn’t time for every student to play the corresponding online game before each lesson, this approach would ensure that students can view the online game scenario before embarking on the related learning activities. Subsequently, the Teacher’s Guide was revised to inform other teachers of this approach to enhance lesson delivery.

Conclusion

The purpose of this formative evaluation was to determine the feasibility and acceptability of the learning activities in the Triple Zero Kid’s Challenge Teacher’s Guide. For the most part, the findings suggest that the learning activities provide children with the information and skills they need to correctly identify and report legitimate emergencies by calling Triple Zero. The findings also identified the need for important modifications and improvements, which have been integrated into a revised version of the guide.

While the pilot version of the guide was informed by existing research evidence and the input of technical experts from the TZAWG, the input provided by children and teachers was fundamental to the development of a quality resource. Based on this experience, emergency services agencies are strongly encouraged to incorporate formative evaluation into the design of emergency education programs for children.

While the methodological approach taken in this study has effectively established the feasibility and appropriateness of the learning activities contained in the Teacher’s Guide, future research should establish their effectiveness through the adoption of a repeated measures, pre-test post-test design with a control group. By providing insight into how children are interpreting and applying the key concepts and procedures conveyed in the guide, the results of this formative evaluation provide a useful basis for a rigorous, evidence-based approach to such work.

Footnotes 

1. The TZAWG is a national body that represents emergency call-taking agencies in Australia. The TZAWG develops and administers programs and activities to enhance community awareness of Triple Zero (000).