1. SAM case history: from school level to regional level A. Vorontsov

2. Goal: to demonstrate different opportunities and potential of SAM results usage for different parties of education (policy makers, school principals, teachers, students and parents). Questions to discuss: 1)Why school teacher needs SAM? 2)How does SAM influence on improving of quality of educational process and it’s results? 3)How to conduct SAM monitoring in school? 4)How can SAM influence educational policy at regional and federal level?

3. 1. Teachers construct tool – professional development Goal: adoption of SAM building principals and functioning. 1 st stage – developing of matrix of subject content, that gives compact idea about complex of cognitive methods of actions (terms, principals, schemes, algorithms and so on), that compose the content of main parts of some school subject.

4. Samples from content matrixes of SAM tests Content areaMode of actionMeans of action Mathematical literacy Measurement of Values Direct measurement Indirect measurement The ratio between number, value and unit "part and whole“ relation Formula for the area of a rectangle Scientific literacy Processes and their conditions Establishing temporal relations between events process scheme experimental scheme Language literacy Word and its meaning Construction an array of words and word forms with alternating sounds The ratio of the "sound-letter" Audible word model

5. Structure of test tasks Mode of action (task) Task 1.1. Task 1.2. Task 1.3. 1 level - Formal 2 level - Object-related 3 level - Functional

6. Structure of SAM test Task 1Task 2Task N … Task 1.1 Task 1.2 Task 1.3 Task 2.1 Task 2.2 Task 2.3 Task n.1 Task n.2 Task n.3 Test subject is a measure of cultural appropriation of means and their respective modes of action Content base of a test – technological matrix (System of the tools / methods of action orientation in a given subject area)

7. Characteristics of the tasks of each level within the same test item LevelsCharacteristics of the task Formal -Description of tasks shows its affiliation with particular class with standard procedures for solution (standard, model, typical tasks); -Relations, essential for task performance, are related to external characteristics of the situation that don’t need determination Reflexive -Tasks that don’t have standard solutions. Types of tasks: A) tasks that can be solved by typical solution need changing of taks’ situation ; B) tasks are presented in abstract form to avoid manipulation of data; C) involving the treatment of typical circuits action (movement from the result to conditions) when performing tasks by other people Functional A) reconstruction and testing options with a choice of one action, meet some requirements; B) on the analysis given circuits (formulas) steps to select adequate; C) to adapt the formula and steps to the specifics of the object; D) involving edge detection method and the main attraction of additional resources; E) in which the main method is a means of implementing another action.

8. Basic test results 3 key group of indexes: Integrated scores (primary and test) – characterize general success of adoption of knowledge; 3D profile – fixes the measure of adoption of cultural ways and relevant actions; Levels of achievement – based on integrated scores of test participants and benchmarks that divide all participants into groups, that relevant to different quality levels

9. THREE-DIMENSIONAL PROFILES 1 level 2 3 0 10 20 30 40 50 80 60 70 90 100 Difficulty, % Class 4V Class 4G 1 Level 2 3 0 10 20 30 40 50 80 60 70 90 100 Dificulty, % 10 th grade 8 th grade 6 th grade 4 th grade

10. Interpretation of childs’ profile: determination of zone of proximal development (ZPD) ›Levels shows the leading type of childs’ orientation in subject; ›Lets teacher see and take into account zone of proximal development:  1 level - ZPD includes additional understanding of general points of curriculum.  2 level – the broaden of the experience of solving of not standard tasks with exit to the boarders on this subject area is important.  3 level – motivation and possibilities for deepen in content of curriculum in needed.

11. Proficiency level interpretation: individual progress (Delta) Second proficiency level is a norm for primary school leavers Distribution of students of different age by proficiency levels

12. 3. Comparative assessment of student educational achievements - establishment of statistical norms Assessment of test results in frameworks of SAM based on 4 things: Average norm – average index of test accomplishment (students/schools); Socio-cultural norm – average index of leading schools; Average norm for the end of the primary school – 2 nd level (reflexive); Absolute cultural norm – adoption of knowledge at 3 rd (functional) level.

13. City level: ›At the level of regional management entities the following set of test indexes can be interesting: ›Test score - class /school /city ›Distribution of students at levels of achievement - class /school /city ›Test score for each part of the program - city ›Distribution of students at levels of achievements for each part of the program – city

14. Krasnoyarsk region - monitoring of leading schools Group of leaders is schools that have the highest indicators. Its and orienteer for the educational society. It sets socio-cultural norm as a standard for the future and additional base for assessment of test results.

15. Leading group of Krasnoyarsk region (schools) 4 th grade 5 th grade 6 th grade 7 th grade 8 th grade 9 th grade 10 th grade 11 th grade Below 1 st proficiency level 1 st proficiency level 2 nd proficiency level 3 rd proficiency level Proficiency level distribution

16. Profile is more important than percents!

17. Evaluation of educational strategies 3 1 2 80% 60% 40% 20% 0% 100%

18. Regional monitoring The Yamal-Nenets Autonomous District Level 0Level 1Level 2Level 3 7 th grade 5 th grade Level 0Level 1Level 2Level 3 Secondary general education school, 5 th grade SGES Regional average Gymnasium, 7 th grade Gymnasium Level 1 Level 2 Level 3

19. Analysis of possible mistakes Hypothesis: 1)Not enough level of adoption of some parts of the curriculum in mathematics. 2) General reason for mistakes is fail in general competencies

20. Analysis of possible mistakes Test items difficulty diagram 0 10 20 30 40 50 60 70 80 90 100 М-01 М-03 М-06М-08 М-11 C-01 C-03 C-05 C-08 D-04 D-05 R-04 R-05G-01 G-02

21. Analysis of possible mistakes 0 10 20 30 40 50 60 70 80 90 100 Measurement CalculationsRegularities Dependencies Geometry Test items difficulty (different content area)

22. Analysis of possible mistakes Test item difficulty (level 2 items) 0 10 20 30 40 50 60 70 80 90 100 М-01 М-03 М-06М-08 М-11 C-01C-03C-05 C-08 D-04 D-05 R-04 R-05 G-01G-02

23. Analysis of reasons for mistakes These data allow to give preference to the second hypothesis: General reason for mistakes is fail in general competencies and skills. First of all following skills: -Text analysis, use of information presented in different forms; -To mind a few conditions, including those that conflict; -To use modelling to determine essential relation in a task; -To identify patterns in structured objects; -To implement tentative steps in finding a solution; - To monitor the progress and the result of solving the problem.

24. Summary and recommendations For primary and post-primary school (5-6 grades): a)To include into educational process different tasks that are relevant to 2 nd and 3 rd levels. b) To analyze of test tasks and use different tools for modeling (graphic, etc); c) To engage problem solving when working on educational material including substandard situations; d) To pay more attention at forming control actions: formal and profound

25. Why SAM is needed and who needs SAM? Who?Test indicatorsPurpose and use of the results StudentIntegrated score 3D profile Level of achievements - Lets determine «problem zones» in curriculum; to rate a student; - Lets determine individual progress of a student - Level of achievement – Zone of Proximal Development TeacherIntegrated score 3D profile Level of achievements - Lets rate the class (school level); determine “problem spots” of students; - Shows leading Zone of Proximal Development, lets build individual programs for students - Shows level of adoption of curriculum of different pupils

26. Three-dimentional profile and teacher interpretation LevelsTeacher’s action 1 FormalZone of proximal development includes additional understanding of general ideas of subjects 2 ReflexiveIncreasing of atypical test items quantity that are on a border of different content areas 3 FunctionalOrganizing of motivation and possibilities to deepen into the context of curriculum.

27. Why SAM is needed and who needs SAM? Who?Test indicatorsPurpose and use of the results School principalIntegrated score 3D profile Level of achievements - Lets assess classes on quality of adoption of parts of curriculum - Lets to determine teaching approach in different classes; - To assess quality of teachers work in dynamics Regional managerIntegrated score 3D profile - Monitoring on quality characteristics of topical area, determination of special “clusters” of schools MinisterMonitoring researches- Determination of age norms of adoption of curriculum: for the primary school leavers the assimilation of the curriculum at the second level is a norm; - Determination of educational technologies, textbooks, that demonstrate best results at 2 nd and 3 rd levels in dynamics

28. Tasks for discussion 1 2 3 4 5 Level 0Level 1Level 2Level 3 SGES #1 (Muravlenko), 5 th grade SGES #2 (Purovsky), 5 th grade Boarding-school (Harshaimskaya), 5 th grade SGES (Raduga), 5 th grade SGES #1 Regional average Boarding-school 5A 5B 5V 5 th grade 7 th grade SGES #2

29. Summary: At the core of results' assessment using SAM instrument is the age and culture norm taken in their absolute form. That fact contributes to the assessment being more multidimensional SAM can change approaches to teaching methods through inclusion of teachers in the design of test materials for formative assessment; SAM can be used as an effective tool for pedagogical feedback and as a basis for formative assessment; SAM provides information on the educational process - its efficiency, targeting, target orientation; In addition to the possibility of measuring quantitative increase of competence (linear progress) SAM allows to fix qualitative increase (level progress); SAM allows you to influence education policy at all levels

30. Thank you for attention!