Experiment or quasiexperiment: success and failure. Lithuanian case Vilija Šemetienė 14-06-2018, Athens CoP-CIE-ESF

content The idea of evaluation Program intervention logic Limitations Lessons learned

The idea of evaluation Project Establishment of safe environment at school Lions Quest "Teenage Crossroads" Zipi‘s friends program Apple‘ friends program Program "The Second Step"; Olweus sneer Prevention Program The project „Establishment of a Safe Environmental School II„. is designed for the prevention of violence and sneer. There are five programs implemented in this project. Zipi‘s friends program; Apple‘ friends program; Program "The Second Step"; Olweus Bullying Prevention Program; Lions Quest "Teenage Crossroads". • Each of these programs has its own intervention logic and model to help achieve the goals of the program. The logic of program interventions varies according to the duration of the training, the nature of the training, the content and scope of the methodological material, and the target groups. Zipi‘s friends program; Apple‘ friends program; Program "The Second Step„ designed especially for young children / pupils, only minimal monitoring and program effectiveness analysis, and relatively small resources are used for implementation. Programs in the second group are designed for senior or all schoolchildren, in the course of the program, sneer and violence detection and monitoring are carried out and more resources are allocated to the implementation of the programs. The program had to start at the begining of school year in september 2017. The idea was to carry out the evaluation through a social policy experiment or a counter-factual analysis. At the end of 2016 we sugested for Ministry of Education to help to prepare for social experiment : to determine the procedure for selecting of schools applying randomisation. Designed especially for young children / pupils Designed for older pupils or all schoolchildren

Program intervention logic Impact II Giving skills contributes to emotional well-being better communication between pupils and students and teachers The level of violence and sneer decreases Pupils 'parents (carers) are more actively involved in solving students' problems Program activities Trainings for teachers (school staff) Provision of methodical literature, consultations, Shearing of good practice Direct result I Teachers gain knowledge and skills Teachers are provided with methodical literature School staff) are included in the target network of professionals Direct result II Teachers organize lessons or special classes at schools Methodological materials are distributed at schools Impact I Students have the skills to deal with social and / or emotional difficulties Parents (carers) are provided with materials to help pupils overcome their problems Despite the differences between the two types of programs, they are linked by common intervention logic and operational principles. In all programs, teachers (part of the programs - and administrative staff) are provided with knowledge and competences for combating violence and bullying in schools. Trained staff trains children / pupils, distributes prepared methodological materials both directly to pupils and to involve other interested groups (parents / guardians). These trainings help children to better address their social and emotional problems, while the school creates ways for more effective communication within the classroom and the ability to involve parents / caregivers more actively in solving existing problems. These steps in the long run lead to a decrease in the level of violence and bullying in school and in better institutions for the development of emotional / psychological climate.

for social experiment or counterfactual Limitations for social experiment or counterfactual Minister order: All schools have to implement violence and bullying prevention programs from the beginning of the 2017-18 school year Due to these factors, research design comparing control and target groups s not valid, accurate and relevant to policy makers. Random selection principle has not been maintained Several programs at the same time Schools are obliged to continuously develop violence and bullying prevention programs All schools will have to implement violence and sneer prevention programs from the beginning of the 2017-18 school year. In other words, all schools will have interventions in one form or another as of September 1 (i.e., they will implement preventive programs) aimed at reducing violence and sneer. This step is essentially determined by the fact that from September 1 there will be no more schools that could form a control group for the assessment of the impact of violence and bullying prevention programs in an experimental / contraceptive way. the nature of the grant contradicts another fundamental methodological assumption for experimental or counterfactual analysis - the principle of random selection. Priority is given to schools that have so far or completely failed to implement programs for the prevention of violence and bullying, or have implemented them on a small scale. In other words, schools are selectively targeted for the implementation of preventive programs. For this reason, all project participants are already de facto part of a particular target group, and it is not possible to randomly collect different groups (according to the necessary parameters) and groups matching the research assumptions. Schools often carry out several programs for the prevention of violence and bullying at the same time for certain school age groups. Some schools are experimenting until they find the most suitable program. In addition, a number of schools are taking measures to combat violence and bullying that are not intended to directly implement violence and bullying prevention programs (for example, classroom educators on the initiative of the classroom, which discuss issues of this nature). This complicates efforts to identify the "net" effect of a specific program. The effects of programs are not immediately apparent. Since schools are obliged to continuously develop violence and bullying prevention programs, determining how a specific program or its type contributed to the results can be difficult, especially if the school is experimenting with several types of programs. Due to these factors, a simple research design comparing several schools implementing different programs and looking at which schools in the program have the lowest level of bullying is not valid, accurate and relevant to policy makers.

Lessons learned Political will Costs and benefits Timely preparation 1 2 3 4 Timely preparation Methodology Why the experiment failed: Political Will. The most serious challenge is to match the efficiency of investment and political logic Cost and benefits – Small interventions and large costs of resarch Methodology timely preparation