Train the trainers on evaluation WHOSEFVA Working with Healthcare Organizations to Support Elderly Female Victims of Abuse Evaluation of Quality and Impact of Project Workstream 4 University of Tartu Train the trainers on evaluation Giorgi Davidovi UT Co-funded by the Rights, Equality and Citizenship (REC) Programme of the European Union under Grant Agreement no JUST/2015/RDAP/AG/VICT/9320

Agenda 1.Introduction of Evaluation 2.Evaluation Framework 3.Evaluation Methodology: participatory and formative evaluation 4.Indicator type 5.Evaluation of the trainings 5.1 Train the Trainers 5.2 Mutual Learning Workshops 5.3 Training for the health and social care settings 5.4 Implementing Protocols for Identifying and Supporting Elderly Female Victims of Abuse 6.Evaluation of the Best Practice Protocols 6.1 The Best Practice Protocols 6.2 Focus groups/Interviews 7.Evaluation of the policy recommendations 8.Logistics of evaluation: the role of local evaluator

Framework for evaluation 1 The evaluation action of WHOSEFVA aims: To evaluating the quality and impact of those actions undertaken within the project To elaborating on the final evaluation report of WHOSEFVA actions Framework for evaluation 1

1 2 3 4 UT, WLF, AOF UT, PPs UT, WLF, AOF, WAVE UT Task No. Task Task description Unit responsible 1 List of indicators, data collection plan and general instructions for data analysis Identification of relevant indicators need to evaluate the engagement process; Developing the data collection format (template); Defining goals for the preliminary analysis and providing general instructions for data analysis. Working together with the curriculum designers, an evaluation strategy is developed for soliciting feedback and analysing data collected from training and health care protocol implementation activities. Two other important elements of quality that will be monitored and evaluated are the quality of outputs and impact of activities. UT, WLF, AOF 2 Data collection Local evaluators are responsible for collecting the needed data at the local level (documenting activities, administering and translating feedback forms, etc). This data is sent to the evaluator to synthesize and analyse. Data must be collected in the pre-given format (changes must be coordinated with UT). UT train partners on how to collect local evaluation data – Pre/Post tests for training participants to demonstrate increased knowledge. UT, PPs 3 Preliminary analysis Regular updates on the evaluation is given every six months, so that corrective action can be taken if possible. UT, WLF, AOF, WAVE 4 Evaluation report The report will be prepared by the evaluator to address the key indicators: Collection of best practices on treating elderly female victims in healthcare Development of online course and training manual Evaluation of healthcare policy and recommendations in each partner country and EU All information gained from WS0-WS4 is collected into a final analytical report written in English. The closing evaluation will be carried out during the final meeting. The following questions will be addressed: - To what extent were the project objectives achieved? - To what extent were the project outcomes achieved? - How useful and beneficial are the results and outputs of the project to the target groups? - How useful and beneficial are the results and outputs of the project to the organisations involved and for national and EU-level work? The results are also evaluated against the EU regulations and recommendations. - How will the project outputs be sustained? How can we promote the use and utility of the outputs? UT

Framework for evaluation 1 The evaluation framework includes sessions with partners during the partner meetings. The progress will be monitored against expected results on a monthly basis, ensuring partners are well prepared for the meetings and the timeframes set for each task are followed. The evaluation questions are used as a tool for evaluating the projects process. In collaboration with all partners, the evaluation framework will address the following questions: - Are the appropriate methods being used to achieve the objectives? - Has this meeting and workshop helped to achieve the objectives? - Are we following the project plan/what should we change and why? - What did we learn during this meeting and workshop? - Can we see unexpected results and effects of the project; if yes, what effects do these have on the project activities? - How would we rate the quality of the partnership, communication, collaboration and working atmosphere among partners.

Evaluation Methodology The evaluation methodology used in the current evaluation plan is described through two concepts: participatory and formative evaluation. The two concepts are not contradictory, but rather touch upon different dimensions of the evaluation used in the given case. Participatory evaluation refers to how the evaluation is carried out, whereas formative evaluation refers to when the evaluation is carried out.

Participatory evaluation Participatory evaluation can be contrasted against conventional evaluation, in which trained evaluators are the experts who make the final judgement. Participatory evaluation emphasizes the role of participants in the evaluation process which on the one hand is more democratic when compared with some of the other evaluation methods Formative evaluation formative evaluation means that the evaluation is carried before the end of the evaluated process (either at the beginning of the program or during it). Formative evaluation may lead to substantial changes to the program as it enables clarification of the program goals and the true needs of the key stakeholders.

Indicator type A set of evaluation indicators (or questions) is presented in the following tables (tables 3 to 8). The tables include the evaluation indicators, the indicator type and the approximate time period when data should be collected. With regards to indicator type, five types of indicators are requested: Descriptive; Categorical; Binary; Ordinal; Numeric. Descriptive indicators require textual answers and are needed to provide in-depth information and context to other indicators. Categorical indicators are used in cases, where categorical options are pre-given and open-ended questions are unnecessary. Binary indicators are a special case of categorical indicators with only two possible categories. Ordinal indicators are used when the indicator can be measured on an ordinal scale, but the differences between different categories are difficult to quantify. Numeric indicators are used if the indicator is directly quantifiable.

Evaluation of the trainings

The trainings are the second shaping action, meant to improve the capacity of health and social care settings. Successful training program has the potential to amplify the effects of WHOSEFVA activities. Therefore, significant effort is assigned to the development of the training programs and its execution. WHOSEFVA project contains capacity building activities, including the Train the Trainers workshops, Mutual Learning Workshops, and Training for the health and social care settings.

WS/Task Name of TtT The content of the training Number of participants Target Group of the Training 1,2 Train the Trainer for DV organizations Experts train partner DV organizations on working with healthcare (M4). The important part of the training is to give awareness raising tools in challenging ageism and gender discrimination and promoting positive attitudes towards elderly and older women. 16 Staff of partner organizations 1,3 Train the Trainer on Mutual learning workshops Experts train partners on how to conduct mutual learning workshops (M4). Partners will learn which themes to address, as they specifically relate to the training program and protocols being introduced. Partners will also be shown how to document the results in a way that is useful to the experts. 1,8 Experts train DV workers on political advocacy and policy analysis The partners will assess the implementation of the regulations in their countries. DV organizations, staff of partner organizations 2,4 Train the Trainer on how to lead Elderly DV Support training for health care professionals Experts will conduct an in-depth 'train-the-trainer' session with staff from project partners who will conduct trainings with target audiences. Topics will include methods of engaging different audiences (nurses vs. doctors; emergency vs. primary care). Partner DV organizations staff 4,2 Train partners on how to collect local evaluation data Pre/Post tests for training participants to demonstrate increased knowledge (M8-9). Statistics collected from selected DV/healthcare organizations before trainings and after trainings (and before and after implementation of protocols in test sites) to assess change in number of victims identified and types of services offered (psychological, counseling, referral, legal, etc.)

Qualitative indicators Target audience of the evaluation Date for collection the evaluation data How partners identified the training needs of Trainees? Curriculum developers November 2017 Who were the trainers? How were the trainers identified? How satisfied were the participants with the 1) training venue, 2) logistics, and 3) content? TtT participants How the received feedback contributed the development of the pf the training manual? Curriculum Developers/ Trainers at TtT workshops How were the trainings developed? Who was developing the TtT programmes?

Quantiative indicators Local evaluators collect the data Date for collection the evaluation data Number of training participants (1.2 Train the Trainer for DV organizations ). November 2017 Number of training participants (1.3 Train the Trainer on Mutual learning workshops ). Number of training participants (1.8 Experts train DV workers on political advocacy and policy analysis). Number of training participants (2.4 Train the Trainer on how to lead Elderly DV Support training for health care professionals) Number of training participants (4.2 Train partners on how to collect local evaluation data) Number of workshops.

Mutual Learning Workshops WS/Task Name of action The content of the training Number of event Number of participants 1.5 MLW 1st and 2nd to discuss on challenges the growth of the elderly population (ageing population) brings especially to health care system services.   to give basic knowledge on elder abuse to raise awareness of human rights/women’s rights perspective in elder abuse to share knowledge how professionals working with traumatized older victims of violence can maintain own wellbeing 1 even per Partner 1. shelters/women’s services/elderly services: 20 -EE LV and 30 persons per country in FI, AU, UK, GR. 2.medical professionals: 5 -EE LV and 10 persons per country in FI, AU, UK, GR. 3. social worker: 5 -EE LV and 10 social workers per in rest of participating countries. 6 in total 210 participants in total from health and social care settings

Qualitative indicators Target audience of the evaluation Date for collection the evaluation data Mutual Learning Workshops Who were the participants of the MWL? Demographic information? Participants October 2017/ February 2018 Who were the facilitators? How were the facilitators identified? Project partners Which type of organizations received the training? How the attitude and knowledge of participants have changed after MLW? How the received feedback from participants was integrated in the development of the training manual? Manual developers How satisfied were the participants with the 1) training venue, 2) logistics, and 3) content? Quantitative indicators Local evaluators collect the data Date for collection the evaluation data Number of participants from shelters/women’s services/elderly services October 2017 Number of medical professionals Number of social worker

Pre-evaluation form Post-evaluation from   proposition 1 = not at all; 5 = completely What do you think is the level of problem of elder abuse in (add your country)? Serious problem Moderate problem Minor problem Not at all a problem I think elder abuse is a serious problem in my country What do you think is the priority level of recognizing and treating older victims of violence in your workplace? High priority Medium priority Low priority Not a priority In my opinion recognizing and treating older victims of violence should be at higher priority level than before in my workplace How familiar are with elder abuse in your work? Very familiar Moderately familiar Somewhat familiar Not at all familiar I am now more familiar with elder abuse How important you think is to recognize the older victims of violence in your work? Very important Moderately important Slightly important Low importance It is very important to recognize the older victims of violence in my work How often do you ask about violence experienced by your clients/patients if you suspect a case? Every time Almost every time Sometimes Never I feel I should ask more often about violence experienced by my clients/patients if I suspect a case

Training for the health and social care settings (II Mutual Learning Workshop) WS/Task Name of action The content of the training Number of event Number of participants 2.7 Conduct local health and social care trainings   2 session for 4x45 minute lectures in total. 2 sessions are divide into 2x45 minutes + 2x45 minutes in separate dates 1. healthcare professionals: 25 -EE; 20 - LV, and 40 person per country in FI, AU, UK, GR. 2. Social workers: 5 person per country will participate from Estonia and Latvia, and 10 person per country in FI, AU, UK, GR. 6 event in total 210 in total

Qualitative indicators Target audience of the evaluation Date for collection the evaluation data Training for the health and social care settings How the training was developed? Manual developers June 2018 Who was developing the training? How the cooperation worked among the training developers? What were the sources of the training program? Review of literature used? What is the structure of the training? Training manual/progra m What types of care providers were present? Participants What was the demographics of participants in terms of gender? What was the knowledge and attitude of training participants before and after the training? participants How well the participants integrated the information from the trainings into the everyday practice? Project partners/manu al developers How the training participants were involved in the training development? How satisfied were the participants with the 1) training venue, 2) logistics, and 3) content?

Quantitative indicators Local evaluator s collect the data Date for collection the evaluation data How many people were reached out? June 2018 How many participated? How many trainings and session was conducted?

WS/Task Name of action The content of the training Number of event Number of participants 2.8 Implementing Protocols for Identifying and Supporting Elderly Female Victims of Abuse Training will be provided for the health care staff and dv organizations on the protocols, best practices and tools developed. 1 per partner 5 persons will participate in EE and LV and 10 person in, AU, UK, GR, UK.   6 event in total 50 in total

Qualitative indicators Target audience of the evaluation Date for collection the evaluation data Training for the health and social care settings What types of care provider were presented? Participants March 2018 What was the demographics of participants in terms of gender? Participant s What was the knowledge and attitude of training participants before and after the training? How well the participants integrated the information from the trainings into the everyday practice? Manual developers How satisfied were the participants with the 1) training venue, 2) logistics, and 3) content? Quantitative indicators Local evaluators collect the data Date for collection the evaluation data How many people were reached out? March 2018 How many participated? How many trainings and session was conducted?

Evaluation of the Best Practice Protocols

Evaluation of the Best Practice Protocols WHOSEFVA experts introduce the protocols based on information gathered from the Mutual Learning Workshops, earlier Daphne projects and Emergency Care Procedure (PAKE) currently used in Finland. For the detailed information see the executive plan. The evaluation data regarding the protocols will be collected before and after the implementation. The data such as how many victims were identified, consulted before the implementation will be the main pre- evaluation indicator. Local evaluator will be obliged to conduct the interviews with the health and social care setting to document the pre-implementation the situation and post-implementation. A survey will be used for collecting the statistical data regarding the victim identification and referrals.

Qualitative indicators Date Target audience of the evaluation Date What is the structure of the protocols? What is the variation among the PPs? Protocol developers/ Project partners October 2018 How the protocols are implemented? Project partners/ health care professionals Type of services offered How relevant were the BP for the health care settings? health care professionals Type of injuries reported before the BP implementation February 2018

Quantified indicators (perception) Date Target audience of the evaluatio n Date Number of victims identified before/after the BP implementation Interview conducted by local evaluators with health care professional s/Raw data collected from health care settings February 2018 Number of referrals to police before/after the BP implementation Number of victims served before/after the BP implementation Number of documented DV case before/after the implementation

Focus groups/Interviews WS/Task Name of TtT The content of the training Number of event Number of participants Target Group of the Training 1.6 Local focus groups/interviews To gain knowledge on the victims’ experiences and identification of the best treatment 3 per partner 3 set support/focus groups or interviews. In Estonia and Latvia 5 individuals will be recruited and in Finland, Austria, Greece and UK 10 individual in each set of support/focus groups or interviews. Professionals at health and social care settings, domestic violence organizations   18 event in total 150 in total

Qualitative indicators Date Target audience of the evaluation Date Focus groups/Interviews Who were the participants of the focus groups/interview? Demographics? Participants October 2017, June/October 2018 Who were the facilitators? Project partners How the collected feedback of 1st focus group/interviews was integrated in the training manual? Manual developers October 2017 How the collected feedback of 2st focus group/interviews was integrated in the training manual? June 2018 How the collected feedback of 3st focus group/interviews was integrated in the training manual? October 2018 How satisfied the participants were with the atmosphere, content and logistics of focus groups/interview? Quantitative indicators Local evaluators collect the data Date Number of participants October 2017, June/ October 2018 Number of events

Evaluation of the policy recommendations

Qualitative indicators Date Target audience of the evaluatio n Date How was the policy recommendation developed? Project partners April 2018 The structure of the policy recommendations Variation among the policy recommendations How the advocacy meetings contributed the advocating the policy recommendation? Results of advocacy meetings Quantitative indicators Local evaluators collect the data Date Number of policy recommendation April 2018 Number of people introduced with the PR Number of participants at advocacy meetings

Logistics of evaluation: the role of local evaluator

Idea of local evaluator … The selected methodology of evaluation requires to closely coordinate the evaluation action on the local level. This means that every partner needs to have a local evaluator. The decision on who should be evaluator should be made inside the organization and coordinated with the evaluator (UT). The local evaluation task can be done by the team, but there must be identifying one designated person who will be a responsible for carrying out the local evaluation actions.

First, partners are responsible for adjusting the evaluation templates to their local environment. That mainly includes translating the questionnaires and giving the feedback on developed evaluation materials based on the local context. First steps..

Following steps… Second, if the localized evaluation plan is coordinated with the UT, partners will start with active data collection phase. That includes documenting activities, administering and translating feedback forms, conducting interviews with relevant target participants on local level, etc. To ensure the synergy of evaluation the actions undertaken by the local evaluator should be coordinated with the evaluator from UT. The raw data will be used for preliminary analysis conducted by the individual project partners and it will be sent to the evaluator (UT) for further analysis. Data must be collected in the pre-given format. The collected data should be translated and summarized also into the given evaluation templates by the local evaluator.

Cell:+37256097981 Skype: davidovi333 Thank you! Contact: Giorgi Davidovi Cell:+37256097981 Skype: davidovi333