Developing an evaluation model to assess prevention measures (EVAPREM) Short introduction Uku Varblane and Tarmo Puolokainen // University of Tartu Brussels 2018

Outline Aim of the model Framework of the model Population survey Impact measurement Statistical background analysis Classification for the prevention activities Measurements, ideas behind the questionnaire Conclusion

The purpose of the model Systematically assess the impact and effectiveness of fire prevention activities in participating countries and regions. GENERAL OVERVIEW- to give an assessment of preventive activities as a whole; e.g. an “overall index”, which assesses whether the right things („effective“) and in the right amount („efficient“) are done; SPATIAL - to give an international comparison; the value of the “index” should be comparable between countries, therefore it should be generalizable across countries, since different countries do different activities for prevention; TEMPORAL - to give an assessment to the trends; the value of the index has to be found over the years, so it must be ensured that the input data is (or will be in the future) available annually; SPECIFIC ACTIVITIES - to give an assessment to specific preventive activities; the model must be sufficiently detailed to be able to conclude what activities and in what proportion should be changed to ensure the optimum outcome (in other words: what to change to maximize the outcome); the components of the index should also have a meaning / interpretation on their own. SPECIFIC TARGET GROUPS - to give an assessment to the activities across target groups (school children, elderly people etc., who would be more vulnerable).

The framework of the model The index consists of three interrelated components: Impact component, which will characterize the impact of each fire prevention activity Population survey component, which will characterize the general knowledge and development level of fire prevention in a country in a specific point of time General statistics component, which will characterize the long-term impact of fire prevention and development of society

The framework of the model General statistics Population survey Impact measurement Change in social development, external factors and practices through change in numbers of emergencies, fire fatalities, etc. General knowledge of fire safety Self-reported practices of the population Each activity has (should have) an impact in the practices of population

The framework of the model Inputs: resources, labour, etc. Target groups Population survey: general level of knowledge Processes: prevention activities Impact of prevention activities (result of measurement) Outcomes: (reduced) number of emergencies, fire fatalities, etc.

Population survey Population survey in each country/region would give a baseline information on: the general knowledge, attitudes and self-reported behaviour. A representative sample of each country/region was asked using the CATI method. The total sample size from all the five countries is 5669, which is composed of 2015 respondents from Lithuania, 1722 from Estonia, 1104 from Latvia, 428 from Frederiksborg County of Denmark and 400 from Southwest Finland.

Statistical background analysis Statistical background analysis would indicate the long-term impact of prevention activities as well as the general development level of the country/region. The change in the number of emergencies, fire fatalities, causes of fires (due to carelessness, arsons, ...), etc. can be considered, but the biggest difficulty is that each country defines those measures a little bit differently. Thus, it can only be used as a background information.

Classification of prevention activities There are multiple prevention activities in every country/region Different activities have various characteristics, which would take different methods to measure their impact So, one should classify each activity into a larger group, and this larger group would provide a „standard“ for impact measurement The impact measurement should be as standardized as possible and as flexible as necessary

General (and unknown) population Personified activities INDIVIDUAL – AT HOME Individuals, households, professionals Face-to-face Feedback forms, calls Standardized activities CLOSED GROUP – WORKPLACE, SCHOOL School children, work collectives Face-to-face, exercise Ex ante and ex post evaluation (tests, randomized groups) Event-based activities OPEN GROUP – PUBLIC ROOM Exhibition/event visitors Face-to-face, brochures, exhibits-boxes “exit polls”, feedback forms Large-scale media campaigns POPULATION GROUPS  – MASS&SOCIAL MEDIA Population, targeted group of population TV- and radio advertisement, social media, newspapers visibility evaluation, (social marketing campaign evaluation) General information provision ENTIRE POPULATION – MASS&SOCIAL MEDIA TV- and radio appearance, social media, newspaper articles dependent on each activity Specific individual General (and unknown) population

What aspects would a typical questionnaire capture? Knowledge, attitudes and behaviour Quality of the service (theory behind feedback surveys): Responsiveness: the degree to which the provider reacts promptly to the customer Professionalism Completeness: the degree to which the service is finished Pleasantness of support: the degree to which the provider uses suitable professional behavior and manners while working with the customer Overall satisfaction with service Background information of the responder

Interactions between the prevention activities, aggregation issues Different components are interrelated and linked closely to each other. Thus, it should be taken into account, that these components will effect each other and thus the final results. However, getting a „pure“ result for each component is impossible, so one has to accept this as a limitation of the model. Different components might have different priorities, one option to take it into account is to use expert opinions (like Delphi method) to give weights to each component.

Steps of the analysis

Discussion Any questions? Comments? Ideas? Concerns?

Conclusion Index is divided into three components: general statistics, population survey and impact measurement of each prevention activity. Each component is interlinked with each other. The prevention activities are classified into five groups, on which are based the different approaches for measuring the impact.

Thank you for your attention! Developing an evaluation model to assess prevention measures (EVAPREM) Tarmo Puolokainen (tarmo.puolokainen@ut.ee) Uku Varblane (uku.Varblane@ut.ee) Brussels 2018