GEMS: G rowth & E ffectiveness M easures for S chools State of Israel Ministry of Education Culture & Sport Office of the Director General Division of Evaluation & Measurement Formative Assessment for Schools

2  Gems is an aid to management for school principal and teachers  Gems’ major purpose is to aid school staff in making data-based decisions, in order to use resources wisely and to improve school functioning  Gems is a school oriented assessment elucidating the strengths and weaknesses of the school and thus enabling it to allocate resources where most needed  Gems perceives the school as an holistic system

3 Achievements Learning environment Teacher-student relations Staff development Curriculum Teacher-Parents relations Educational plans Teacher-Teacher relations Organization

4  Gems indicators were based on requests by 200 school principals and 200 supervisors when asked what reliable information they need in order to improve school functioning  Gems includes the following indicators: Pedagogical Environment  Action Plan  School priorities and staff consensus  Teaching resources & teaching methods  Weak students and the help they get Academic Achievements in 4 basic subjects (Language, Mathematics, Science, English) School Climate and Work Environment

5  GEMS is administered each year in 50% of elementary and junior high schools, so that each school participates in GEMS once every two years  GEMS sources of information: Tests for 5 th and 8 th grades in Language, Math, Science and English Questionnaires for all students in 5 th through 9 th grades dealing with school climate, pedagogical environment etc. Interviews with all principals and teachers dealing with work environment, teaching methods, staff development etc.  Gems is administered at the beginning of the school year so that schools may get the report during that same year, and shorten the possibly stressful period of anticipation of the assessment

6  GEMS is school oriented. Its main goal is not to provide a national “state of the union message”, but rather to enable schools themselves to plan rationally and to follow up, over time, the fruits of their efforts  However, the data also serve the staff of the Ministry of Education and its units. Thus, over 20 different aggregate reports are prepared each year to help policy makers in different branches of the Ministry to make data-based decisions

7  The rationale of GEMS is that a necessary condition for improving the educational system is to improve the interaction between teacher and students on the class level  Such an improvement is depends on schools basing their plans and practices on continuous, accurate, reliable, longitudinal data  In order to plan and make data-based decisions schools have to take the following steps: to set specific goals to know were they are to take actions directed to the goals  GEMS is a tool facilitating these steps

8 Setting Specific Goals

9  Goals have to be specific in order to be measurable and to avoid contradiction between goals  For example if the goal is “enhancing academic achievements” assessment must specify at least 4 indicators: Average grade percent of failures percent of distinctions difference in grades between students from deprived SES and those from high SES  As shown below these are not necessarily identical:

10 “enhancing academic achievements” -- 1 st measurement School CSchool BSchool A 60 Average 999S.D 33% % failure 0% % excellent “enhancing academic achievements” -- 2 nd measurement School CSchool BSchool A 67 Average 41123S.D 0%11%56% failure 0% 33% excellent Goal set for schools: raise average by 10%

11 How schools operated to reach the goal? Does assessment distinguish between them? School C Year later School B Year later School A Year later Year 1 3 schools Student Student Student Student Student Student Student Student Student 9

12 “Without data you are just another person with an opinion” Knowing where we are A few examples of data for a specific school

13 School priorities: consensus? School Example  Principal: Achievements  Teachers: language skills school climate violence prevention Math Arts school climate English Respect for teachers Keeping school clean Independent study Parental participation “Closing the Gap” language skills Don’t know National data

14 Weak students Teachers: reasons for learning difficulties learning difficulties weak students

15 Students’ Achievements

16 In order to evaluate the changes in students' achievements, the following analysis was undertaken: comparing the change in a given school (over the two years) to the change found in a Comparison Group, consisting of schools with same background characteristics and same starting point in The school may thus draw conclusions such as: Grades 2002 Grades 2004 Size of change in school Size of change in CG Hebrew Math Science English  Achievements improved more in the school than in comparable schools  The change in the school is similar to the change in the comparison group  Achievements in the school show less improvement than that in other, similar, schools Assessing Change in 2004

17 Percent of students that didn't meet standards 20% to 25% of students have private tutor paid by parents

18 Achievements by SES The gap in achievements between the two groups is ½ to 2/3 standard deviation

19 Teaching methods and Homework  10% of the teachers coordinate amount of homework  Students are required to prepare 25 minutes homework for each hour in school  Homework is mainly drill of material taught in class  10% of the teachers require learning new material or looking for sources  60% of teaching time is frontal teaching

20 Student perception of teachers’ attitudes

21 Teachers’ perception of the work environment

22 Taking actions to reach goals: Action Plan We are free to choose our actions,... but we are not free to choose the consequences of these actions. Stephen R. Covey (1996)

23 To define specific indicators and to diagnose school relative to goals indicators derived derived from school goals indicatorsderived from previous data ? ? What to diagnose ?

24 To analyze GEMS data on two levels SCHOOL CLASS ? ?GEMS

25 To ask questions  Every school supervisor, principal, teacher should have a dialogue with available data (from GEMS and from school internal evaluations) and to ask himself: Did I know… Did I ask myself… Did I check…

26 To plan according to data An example: what should be done in classes with different composition Class AClass B

27 Prepare school action plan and define (specifically) where am I and where I want to be…

28 evaluating Planning Working

29 Feedback to assessment from 1,036 principals: 2004