I SSUES IN THE EVALUATION OF A PROGRAM TO PROMOTE EDUCATIONAL ACHIEVEMENTS OF ETHIOPIAN-ISRAELI HIGH SCHOOL STUDENTS "Improving Education through Accountability and Evaluation: Lessons from Around the World” Rome, Italy, October 2012 DALIA BEN-RABI, VIACHESLAV KONSTANTINOV, RUTH BARUJ-KOVARSKY, MIRIAM COHEN-NAVOT Engelberg Center for Children and Youth Myers-JDC-Brookdale Institute

 Since 1984  Most parents are illiterate  In 2012 – most children were born in Israel  In spite of efforts of government, local authorities, and voluntary organizations large gaps in many areas, including academic achievements 2 IMMIGRATION FROM ETHIOPIA TO ISRAEL

3 ACHIEVEMENTS ON NATIONAL EXAMS OF ETHIOPIAN-ISRAELI 8TH GRADERS, COMPARED TO THOSE OF ALL STUDENTS IN HEBREW SPEAKING SCHOOLS IN ISRAEL, 2004/5 (AVERAGE SCORE, ON A SCALE OF 0-100)

4 ACHIEVEMENTS ON NATIONAL MATRICULATION EXAMS OF ETHIOPIAN-ISRAELI 12TH GRADERS, COMPARED TO THOSE OF ALL STUDENTS IN HEBREW SPEAKING SCHOOLS IN ISRAEL, 2004/5 (%)

 Was first implemented in 2006  Aimed at Ethiopian-Israeli students in middle schools and high schools  Places special emphasis on matriculation exams  Instruction in small groups for about four hours weekly, in different school subjects  Social activities and personal coaching  At its peak in 2008 reached over a third of all Ethiopian-Israeli students in grades 7-12 nationwide 5 THE PROGRAM

 Providing regular feedback each year to support ongoing program improvement:  The implementation of the program  Satisfaction of the school principals and the participating students  The impact of the program on the students’ scholastic achievements 6 EVALUATION

 Eligibility for partial matriculation certificates (enables admission to advanced professional courses)  Eligibility for matriculation certificates (a prerequisite for many jobs and some colleges)  Eligibility for matriculation certificates with a high-level of English (a prerequisite for universities) 7 MEASURES OF ACHIEVEMENT

 Are the achievements of program participants better than those of non- participants?  Is the difference in achievements growing over years, as the students are exposed to the program more years before taking the exams?  Are the gaps between Ethiopian and non- Ethiopian Israeli students narrowed? 8 QUESTIONS OF IMPACT: CAN WE ATTRIBUTE THE DIFFERENCES TO THE PROGRAM?

9 ACHIEVEMENTS OF PARTICIPANTS 2009/10

10 MATRICULATION ACHIEVEMENTS OF 12 TH GRADERS, 2010 (%) Partial matriculation matriculation University matriculation

11 Boys Girls GIRLS DO MUCH BETTER THAN BOYS: % OF BOYS AND GIRLS WITH MATRICULATION CERTIFICATES, 2010

12 GIRLS DO MUCH BETTER THAN BOYS: % OF BOYS AND GIRLS WITH MATRICULATION CERTIFICATES THAT MEET UNIVERSITY ADMISSION REQUIREMENTS, 2010 Boys Girls

13 IMPACT OF THE PROGRAM ON THE STUDENTS' MATRICULATION RESULTS

 Schools and students were not randomly assigned to the program  Schools in the program are stronger than the average, and weakest students are not included  Schools came in and out of the program over the years 14 THE PROBLEM: NO CONTROL GROUP

Comparison groups: 1.Ethiopian-Israeli students from similar non-participating schools and with similar socio-demographic characteristics:  Student characteristics : country of birth (Israel/Ethiopia), gender  School characteristics :  State or State Religious school  The percentage of students eligible for matriculation certificates among non-Ethiopian school students, in the year of the analysis  The percentage of students eligible for matriculation certificates among Ethiopian-Israeli school students in 2005, prior to the implementation of the program. 2. Non-Ethiopian Israeli students, from participating schools 15 THE SOLUTION: MATCHING STUDENTS BASED ON ADMINISTRATIVE DATA

As students have had the opportunity for more years of exposure to the program:  Differences in achievements between participants and non- participants should grow  Gaps between participants and non-Ethiopian students should be narrowed The comparison is of the added value of the program as opposed to whatever form of other assistance may exist in program or comparison schools 16 ANALYSIS 1: CHANGES OVER TIME IN 28 SCHOOLS THAT PARTICIPATED IN THE PROGRAM FOR AT LEST 3 YEARS

17 ACHIEVEMENTS OF PARTICIPATING STUDENTS, ALL ETHIOPIAN- ISRAELI STUDENTS IN PARTICIPATING SCHOOLS AND STUDENTS IN COMPARISON GROUP, BY YEARS OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION CERTIFICATES) Impact (difference between program participants and comparison group, divided by % in comparison group) 27%* Years in the Program (28 schools) *p<0.01 *49% not significant

18 ACHIEVEMENTS OF PARTICIPATING STUDENTS, ALL ETHIOPIAN- ISRAELI STUDENTS IN PARTICIPATING SCHOOLS AND STUDENTS IN COMPARISON GROUP, BY YEARS OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION CERTIFICATES MEETING UNIVERSITY ADMISSION REQUIREMENTS) 20%* Years in the Program (28 schools) *p<0.05 **p<0.01 **63% Impact (difference between program participants and comparison group, divided by % in comparison group) not significant

19 THE ACHIEVEMENTS OF ETHIOPIAN AND THE NON-ETHIOPIAN ISRAELI STUDENTS IN THE SAME SCHOOLS, BY YEAR OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION CERTIFICATES) Years in the Program

20 THE ACHIEVEMENTS OF ETHIOPIAN AND THE NON-ETHIOPIAN ISRAELI STUDENTS IN THE SAME SCHOOLS, BY YEAR OF EXPOSURE TO THE PROGRAM (% OF STUDENTS ELIGIBLE FOR MATRICULATION CERTIFICATES MEETING UNIVERSITY ADMISSION REQUIREMENTS) Years in the Program

21 IMPACT OF THE PROGRAM ON THE ACHIEVEMENTS OF STUDENTS WHO PARTICIPATED IN , TAKING ACCOUNT OF THEIR ACHIEVEMENTS IN EIGHTH GRADE

22 MATRICULATION EXAM ACHIEVEMENTS OF PARTICIPATING STUDENTS AND STUDENTS IN THE COMPARISON GROUP WITH THE LOWEST PREVIOUS ACHIEVEMENTS, AND THE IMPACT OF THE PROGRAM ( , IN PERCENTAGES) *p<0.05 **p< participating students

23 **p< participating students MATRICULATION EXAM ACHIEVEMENTS OF PARTICIPATING STUDENTS AND STUDENTS IN THE COMPARISON GROUP WITH MEDIUM PREVIOUS ACHIEVEMENTS, AND THE IMPACT OF THE PROGRAM ( , IN PERCENTAGES)

24 *p< participating students MATRICULATION EXAM ACHIEVEMENTS OF PARTICIPATING STUDENTS AND STUDENTS IN THE COMPARISON GROUP WITH THE HIGHEST PREVIOUS ACHIEVEMENTS ( ), AND THE IMPACT OF THE PROGRAM ( , IN PERCENTAGES )

 Estimate the impact of a comprehensive scholastic assistance program for Ethiopian- Israeli high school students, while trying to overcome the obstacles resulting from the lack of random assignment of schools and students to the program by creating a comparison group of matched students, based on administrative data. 25 THE STUDY AIMED TO

 Better achievements compared with non- participating students from similar schools and with similar socio-demographic characteristics  The impact of the program was also reflected in the narrowing of the gap between Ethiopian-Israeli and non- Ethiopian students 26 THE STUDY SHOWED A SIGNIFICANT PROGRAM IMPACT ON THE MATRICULATION RESULTS OF PARTICIPATING STUDENTS:

 The impact has increased as students have had the opportunity for more years of exposure to the program  The most significant impact was on students with the lowest previous achievements  Nonetheless, considerable gaps still remain, among boys, and especially as regards English exams. This poses a barrier to obtaining a matriculation certificate that meets university admission requirements. 27

 Cost-effectiveness – all students or weakest students?  Boys and girls – do they require different forms of assistance?  English as a third language  The continuum between elementary school assistance and high-school assistance  What are realistic expectations for attainable student achievements, given both the large gaps between them and other students and the scope of assistance that the program provides? 28 KEY ISSUES: