Multivariate analysis of the Kyrgyz data from CAPSA Presentation at the II Academic Conference „Assessment in Education: Experience and Future Development “ Astana, Kazakhstan 29 September, 2016 Prof. Dr. Rainer Lehmann Humboldt University, Berlin, Germany

Multivariate analysis of the Kyrgyz data from CAPSA 29 September 2016 In September 2012, educational experts and officials from five Central Asian countries Kasakhstan Kyrgyzstan Tajikistan Turkmenistan Uzbekistan1 agreed to conduct in their countries parallel surveys of student achievement in reading comprehension and mathematics, by way of representative random samples of fourth-grade pupils (intact classrooms) with maximum coverage. This joint effort was labelled „Central Asian Program of Student Assessment“ (CAPSA). 1 decided to withdraw at an early stage of the Program

Multivariate analysis of the Kyrgyz data from CAPSA 29 September 2016 General Aims Within this common strategy, it was further agreed to base the assessment on domain-specific frames of reference, compatible with all national curricula involved; to define these frames of reference in a hierarchical manner so as to guide the development and selection of common test items; to standardize data collection procedures across all populations (including pupils instructed in the national language and in Russian in all participating countries); to conduct pilot tests in order to ascertain nationally the quality of the tests employed, to achieve high response rates and to estimate the sample sizes required to attain an effective sample size of n* > 400

Multivariate analysis of the Kyrgyz data from CAPSA 29 September 2016 Procedures National Working Groups were trained to work with the statistical package SPSS (© IBM) and the program for probabilistic test analyses ConQuest (© ACER). These activities were significantly supported by Dipl. Psych. Julia Levin (Senior CIM Expert, Academy of Education of Kyrgyzstan). In each country, the main data collection consisted of two independent sub-samples (national language and Russian), in which both reading comprehension and mathematics was tested. Tajikistan and Turkmenistan could not adhere to the international sampling guidelines.

Multivariate analysis of the Kyrgyz data from CAPSA 29 September 2016 Procedures In Kyrgyzstan a total sample size of N ~ 4030 was achieved. Tajik and Uzbek speaking students had to be excluded. The configuration of the background questionnaire and the data collection were accomplished by the Kyrgyz working group: Samarkul Umralieva Mira Mykyeva Zeinep Zhamakeeva Dinara Dautova Bekzhan Kubanychbek uulu Julia Levin The work was supervised by a steering committee, consisting of Dr. Abakir Mamytov Artur Bakirov.

Multivariate analysis of the Kyrgyz data from CAPSA 29 September 2016 Properties of the Kyrgyz sample Total n Kyrgyztan Design effect (Deff) Effective sample size Standard-error of sampling Mathematics N(Kyrgyz) 3120 10,97 285 0,059 N(Russian) 906 7,86 115 0,093 Reading 3123 9,63 324 0,058 907 8,34 109 0,096

Multivariate analysis of the Kyrgyz data from CAPSA 29 September 2016 Intermediate Findings Significant differences in terms of mean achievement were observed between the two sub-samples (Kyrgyz vs. Russian language of instruction; see below). Consequently, the aspired sampling accuracy was not attained. Moreover, the clientele of classrooms with Kyrgyz as the language of instruction proved to be significantly more heterogeneous than that of classrooms with Russian as the language of instruction. Gender differences followed the almost universal pattern of girls performing better than boys in reading comprehension whereas the opposite held true in mathematics. It is noteworthy that the reading effect was more pronounced than the mathematics one.

Multivariate analysis of the Kyrgyz data from CAPSA 29 September 2016 Further Findings Absence of one or both parents from home for a significant period – presumably due to work – coincided with reduced achievement levels of their children. The presence of educationally relevant resources (own room for study, computer applied to learning, books) as well as instruction perceived as motivating were conducive to achievement. Regional disparities in favor of pupils from the capital (Bishkek) were striking. The only exception appeared to be the City of Osh in the domain of mathematics. These preliminary conclusions need to be tested, however, by way of multivariate analyses in which indirect effects are accounted for.

Model explaining differences in reading comprehension among fourth-grade pupils in the Republic of Kyrgyzstan (unstandardized path coeff.) Бишкек Иссык-Куль -34 -78 Reading R2 = 0,123 Ош область -29 Чуй -31 Талас -27 Нарын -55 Джалал-Абад -47 -15 2,6   5,6   4,1   25   - 4,8 Баткен   -81 г. Ош parent(s) absent (Q5) resources (Q7)   books (Q9) reading instruction (Q14) Female gender Russian language  

Model explaining differences in mathematics among fourth-grade pupils in the Republic of Kyrgyzstan (unstandardized path coefficients) Бишкек Иссык-Куль -30 -39 Mathematics R2 = 0,345 Ош область -32 Чуй -32 Талас -31 Нарын -26 Джалал-Абад -36 ,56  -7,9   -3,3   - 8,8 Баткен   + 2 г. Ош Reading Mathem. instruction (Q14) Female gender Russian language  

Multivariate analysis of the Kyrgyz data from CAPSA 29 September 2016 Conclusions The purpose of educational monitoring is to call attention to shortcomings in an educational system, such as overall or sectoral achievement deficiencies or inequalities of educational opportunities. In the case of Kyrgyzstan, two aspects stand out. There are very large regional disparities, especially in terms of reading comprehension: Already in fourth grade, the pupils of some oblasts have fallen behind by almost a standard deviation. There are also noticeable social disparities: If one or both parents work in a distant location, often in Russia, their children‘s academic performance levels are lower than would be expected under different circumstances.

Multivariate analysis of the Kyrgyz data from CAPSA 29 September 2016 Thank-you very much for your attention!