1. Measuring of student subject competencies by SAM: regional experience Elena Kardanova National Research University Higher School of Economics

2. Outline of presentation ›SAM based model for assessment of student subject competencies ›Regional diagnostic study: sampling and procedures of test administration ›SAM regional norms and presentation of results ›Interpretation and uses of SAM information: primary analysis of factors that influence educational results in primary school ›SAM uses for improving teaching and learning: relation between teacher’s pedagogical approaches and educational results in primary school ›SAM experience in other countries

3. SAM based model for assessment of student subject competencies

4. Approaches to results interpretation Norm-Referenced  The result of individual student is interpreted depending on the achievement of the whole population  Each student gets test score  Norms are set

5. Approaches to results interpretation Criterion-referenced  The gradual option of achievement scale is developed. It’s based on students integrated test scores and benchmarks that divide all participants of testing into groups that relevant to different proficiency levels  Each participant is assigned a proficiensy level

6. Students estimation ›Rasch model is used as a test model ›Test scores are reported on a 1000-point scale with a mean at about 500 and standard deviation of 50 ›Test scores of all participants are on the same metric scale regardless of the time of test administration and specific set of test items completed

7. Mathematical competence scale Proficiency level 3 We expect student A to successfully complete at least 50% of level 3 items Student A Proficiency level 2 Proficiency level 1 Below level 1 We expect student B to successfully complete at least 50% of level 2 items Student B We expect student C to successfully complete at least 50% of level 1 items Student C We expect student D to be unable to successfully complete even 50% of level 1 items Student D 500 570 430 Items of the 3 rd level Items of the 1 st level Items of the 2 nd level Estimation of examinees

8. Interpretation of benchmarks Benchmarks: 570 (border btw. 2 and 3 proficiency levels) 500 (border btw. 1 and 2 proficiency levels) 430 (border btw. 0 and 1 proficiency levels)

9. Regional diagnostic study: sampling and procedures of test administration Velikiy Novgorod and its area

10. Regional diagnostic study May 2012 Sample size: 4406 students of 4-th grade (the region’s whole population of fourth grade students ) No selection at the school or classroom level

11. Description of research sampling 47% boys, 53% girls 72% urban, 28% rural Number of students 4406 Number of schools 189 Number of classes 297 Number of settlements 134

12. Regional diagnostic study: procedure ›Paper&pencil form ›Administration to the whole class by the teacher ›Two 45-minute testing sessions with a 15- minute break ›Whole region in 1 week

13. SAM regional norms and presentation of results

14. SAM results  Integrated test score( relation of result to the metric scale)  Proficiency level (relation of result to grade scale)  3D profile (relation btw. results of 3 subtests)

15. Normative-referenced interpretation: Statistical norms (Mathematics) Average group norms Mean Standard deviation 517 34 Socio-cultural norms Mean 561 Average individual norms Mean Standard deviation 522 49 Percentile individual norms 10th percentile 90th percentile 459 581

16. Math profile for a sample of students

17. Distribution of test participants on proficiency levels (Mathematics)

18. Distribution of students of different schools of the region at proficiency levels (mathematics) ›Schools put in order by increasing of the mean test score ›For every school the nean test score is indicated in brackets.

19. Distribution of students of different classes within the same school by achievement levels (mathematics)

20. Interpretation and uses of SAM information: primary analysis of factors that influence educational results in primary school

21. Key questions 1.What is the efficiency of different educational programs? 2.How different factors influence on students learning? 3.What characteristics of learning environment influence on educational quality in primary school? 4.How teachers and school work can be improved?

22. Educational environment and its characteristics that can be examined Level Responsible entity Domains Regional / Federal Federal or local Government Regional educational policy, Federal educational standard, unified exams, curriculum School School principal School policy, type of school, curriculum, condition of building and classes, sports sections, school activities, etc. Recruiting of teachers and administrative personnel Class TeacherQuality of teaching, methods of teaching, pedagogical approaches. Quality of students feedback. Educational tasks and goals Outside of school, family Parents and student Out-of-school activities, additional education, social- economic status, parents education, books, computer, Internet access, personal motivation

23. Information sources for analysis of factors Contest questionnaries (for teachers, for administrative personnel) ›Set of contest characteristics can vary depending on regional research tasks ›Focus at characteristics of school environment that can be corrected to improve the quality of education Technical information The detailed information about school learning and teaching features can be collected purposefully (e.g. educational programs).

24. Relation btw. SAM math results and type of educational institution ›There are 15% of student study in gymnasium. ›Differences btw. schools are statistically significant: gymnasiums get better tests results. › Number of children at 2 nd level is the same. Difference is btw. children at 1 st and 3 rd levels.

25. Relation btw SAM math results and type of settlement Percentage of children at 3 rd level is higher in the city and decreases in towns and villages. Its vice versa for percentage of children at 1 st level – it’s higher in villages.

26. Relation btw SAM math results and type of settlement Most of students are on the 2 nd proficincy level In Velikiy Novgorod the results are slightly better – bigger percentage of children is at 2 nd and 3 rd levels.

27. Relation btw SAM math results and size of the class ›We can single out 2 types of classes – big and small ›Small classes are those that have less than 11 students, big classes have 11 and more students (maximum number of students in one class is 33) ›All together we analyzed 76 small and 152 big classes

28. Relation btw SAM math results and size of the class Results of children in small and big classes are not statistically different.

29. SAM uses for improving teaching and learning: relation between teacher’s pedagogical approaches and educational results in primary school

30. Pedagogical approaches ›Currently it is widely assumed that teachers’ beliefs about the nature of teaching and learning include both “direct transmission beliefs about learning and instruction” or, so called, “traditional beliefs” and “constructivist beliefs about learning and instruction” (OECD, 2009). ›2 educational approaches: traditional and constructivist –The traditional approach implies that teacher communicates knowledge in a clear and structured way, explains correct solutions, gives learners clear and resolvable problems and ensures peace and concentration in the classroom –The constructivist approach implies that students are active participants in acquisition of knowledge, students’ own inquiry is stressed developing problem solutions

31. Teachers survey › Special teachers questionnaire ›228 teachers in total ›56 teachers work in Velikiy Novgorod, and 172 in Novgorod region ›17 teachers work in gymnasium and 211 in comprehensive school ›186 teachers graduated from university and 42 graduated from college ›Work experience varies from 2 to 48 years. Average is 25 years

32. Correlation -,204** is significant at the 0.05 (2- tailed) Traditional Constructivist Teachers’ general pedagogical approach

33. Clusterization of classes

34. Pedagogical approaches ›Constructive approach positively relates to results of learning in math and Russian language: the higher level of constructivism of a teacher the higher test scores students have. ›Constructive approach positively relates to the number of students in class at 3 rd level and negatively with number of students at 1 st level and below 1 st level. ›Traditional approach doesn’t have significant relation with learning results – neither with test scores or distribution of children at levels.

35. SAM experience in other countries ›Kazakhstan ›Kyrgyzstan ›Tajikistan

36. Thank you for your attention Elena Kardanova ekardanova@hse.ru Center for monitoring and quality of education Institute of education Higher School of Economics http://ioe.hse.ru/monitoring/