1. Achievement in Relation to Background Factors

2. 2 Background Data Gathering During every cycle of assessment, background data are gathered by questionnaire and analysis of fieldworkers’ reports. During every cycle of assessment, background data are gathered by questionnaire and analysis of fieldworkers’ reports. Questionnaires are given to students (including a section for parents), head teachers and teachers. Questionnaires are given to students (including a section for parents), head teachers and teachers. Conditions, facilities etc. in schools are observed and recorded by test administrators. Conditions, facilities etc. in schools are observed and recorded by test administrators. Background data are correlated with learning achievement to identify needs for improvement and factors associated with achievement. Background data are correlated with learning achievement to identify needs for improvement and factors associated with achievement.

3. 3 Background Data Gathering Framework Background data are classified into five areas following the Craig-Heneveld Framework: Background data are classified into five areas following the Craig-Heneveld Framework: The StudentsThe Students Supporting Inputs from Outside the SchoolSupporting Inputs from Outside the School Teaching-Learning ProcessTeaching-Learning Process Enabling ConditionsEnabling Conditions School ClimateSchool Climate Results are presented in the next section Results are presented in the next section

4. 4 The Students Data are gathered from students about: Socio-economic background (e.g., family size, father’s/mother’s education and occupation) Socio-economic background (e.g., family size, father’s/mother’s education and occupation) Factors in the home environment related to achievement (e.g., help w/ homework, space for reading, need to work outside) Factors in the home environment related to achievement (e.g., help w/ homework, space for reading, need to work outside) Attitudes towards education Attitudes towards education

5. 5 For Example: Variable % Scaled Scores MathsUrduScience Social Studies Yes23 No77 No77383403370389455473481503 DifferenceSignificantSignificantSignificantSignificant Do you work outside the home to earn money? Conclusion: about 23% of Grade 4 students reported having to work outside the home. Their achievement is significantly lower than others’.

6. 6 Supporting Inputs from Outside Information is gathered from different sources about: Parent and community support (e.g., frequent school- parent meetings, financial and other support, parental involvement in homework) Parent and community support (e.g., frequent school- parent meetings, financial and other support, parental involvement in homework) Effective support from the education system (e.g., supervision, availability of teaching materials and supplementary materials) Effective support from the education system (e.g., supervision, availability of teaching materials and supplementary materials) Adequate material support (e.g., facilities, furniture in school, school budget for supplies) Adequate material support (e.g., facilities, furniture in school, school budget for supplies)

7. 7 For Example (1): How many of the following six activities in the school does the head-teacher invite parents to participate in? Attend special events (e.g., functions, sports events) Attend special events (e.g., functions, sports events) Raise funds for the school Raise funds for the school Participate in development planning Participate in development planning Volunteer for school projects, programs, and trips Volunteer for school projects, programs, and trips Ensure that their child completes his/her homework Ensure that their child completes his/her homework Serve on school committees (e.g. personnel, finance) Serve on school committees (e.g. personnel, finance)

8. 8 For Example (2): Number of activities: ActivitiesMathsLanguage Low (0-2 Activities) 360373 High (3-6 Activities) 385401 DifferenceSignificantSignificant Conclusion: most head teachers reported inviting parents to participate in a significant number of school-related activities. This participation is positively associated with student achievement.

9. 9 Teaching - Learning Process Information is gathered from different sources about: Learning time (no. of teaching days, length of period, time on task) Learning time (no. of teaching days, length of period, time on task) Variety of teaching strategies used in class (e.g., questioning techniques, investigation, practical work) Variety of teaching strategies used in class (e.g., questioning techniques, investigation, practical work) Homework (amount given, how it is assessed and what feedback is given) Homework (amount given, how it is assessed and what feedback is given) Student assessment and what feedback on their work students receive Student assessment and what feedback on their work students receive

10. 10 For Example: How often does your teacher check your Maths homework? ResponsesMaths Twice or More Weekly 401 Once a Week or Less 347 DifferenceSignificant Conclusion: the great majority (95%) of students reported that they are taught by teachers who check their mathematics homework at least twice weekly. These students achieve significantly better than students who report their homework is checked less frequently.

11. 11 Enabling Conditions Information is gathered from different sources about: Leadership (e.g., whether the head-teacher is primarily an administrative or an instructional leader in the school) Leadership (e.g., whether the head-teacher is primarily an administrative or an instructional leader in the school) Capable teaching force (e.g., teacher education, qualifications, experience, style) Capable teaching force (e.g., teacher education, qualifications, experience, style) Flexibility and autonomy (e.g., decisions and discretion of head teacher, teacher) Flexibility and autonomy (e.g., decisions and discretion of head teacher, teacher) High level of time in school (starting/finishing time of periods, presence/absence of teachers) High level of time in school (starting/finishing time of periods, presence/absence of teachers)

12. 12 For Example (1): How many of the following six indicators of academic leadership does the head teacher have explicit policy for? Corporal punishment Corporal punishment Homework Homework Recording and reporting student progress Recording and reporting student progress Explaining students’ mistakes Explaining students’ mistakes Dealing with weak students Dealing with weak students Parental involvement Parental involvement

13. 13 For Example (2): ResponsesMathsLanguage None393375 One388370 Two or More 404403 DifferenceSignificantSignificant Conclusion: about two-thirds (63%) of all students are in schools with no explicit policy for academic matters (i.e., most head teachers see themselves as administrative rather than instructional leaders). Policy for academic matters is associated with student achievement.

14. 14 School Climate Information is gathered from different sources about: High expectations for students (school expects and rewards success) High expectations for students (school expects and rewards success) Teacher Attitudes (attitudes to students and school goals, perception of school climate) Teacher Attitudes (attitudes to students and school goals, perception of school climate) Order and Discipline (e.g., punishment policy and methods, feeling safe at school) Order and Discipline (e.g., punishment policy and methods, feeling safe at school) Organized Curriculum available to all Organized Curriculum available to all Classroom Climate (e.g., student participation, teacher- student interaction) Classroom Climate (e.g., student participation, teacher- student interaction)

15. 15 For Example (1): Students’ Perception of School Climate I like being in school I like being in school I think that students in my school try to do their best I think that students in my school try to do their best I think that teachers in my school care about the students I think that teachers in my school care about the students I think that teachers in my school want students to do their best I think that teachers in my school want students to do their best

16. 16 For Example (2): ResponsesMathsLanguage Yes to all four 401387 Yes to 3 or less 372368 DifferenceSignificantSignificant Conclusion: an overwhelming majority (92%) of students responded “yes” to all four statements. This was associated with higher achievement in both language and mathematics

17. 17 Use of Blackboard and Achievement Use of Blackboard% Scaled Scores MathsUrduScience S. Studies Yes97 No03 399350384341469431497457 DifferenceSignificantSignificantSignificantSignificant Teachers’ use of black board significantly increased students’ achievement in all the four subjects Teachers’ use of black board significantly increased students’ achievement in all the four subjects Ninety-seven percent teachers reported they daily use black boards Ninety-seven percent teachers reported they daily use black boards

18. 18 Use of Textbooks and Additional Resources 0 10 20 30 40 50 60 70 80 90 100 TextbookAdditional sources No use of textbook Percent Urdu Teacher Maths Teacher Science Teacher Social Studies Teacher Majority of the subject teachers used textbooks as major source of knowledge. Majority of the subject teachers used textbooks as major source of knowledge. Around 10% teachers reported consulting additional resources for teaching Around 10% teachers reported consulting additional resources for teaching Science teachers were more likely to consult additional resources than other subject teachers Science teachers were more likely to consult additional resources than other subject teachers Back to Slide 15

19. 19 Multi-grade Teaching and Achievement Multi-grade Teaching % Students’ Scaled Scores Maths Urdu Teachers’ Scaled Scores Maths Urdu Maths Urdu Yes 21 No 78 442466495516616644644661 DifferenceSignificantSignificantSignificantSignificant Lower frequency of Multi-grade Teaching is linked with increased performance for both students and teachers Lower frequency of Multi-grade Teaching is linked with increased performance for both students and teachers Twenty- one percent teachers reported they have to teach more than on classes in one period Twenty- one percent teachers reported they have to teach more than on classes in one period

20. 20 Rewards and Punishment in School Reward % Scaled Scores MathsUrduScience S. Studies Yes 85 No 14 403372390354474440504464 DifferenceSignificantSignificantSignificantSignificant Punishment % Frequently.03 Occasionally 38 Never 58 351388405345376391444458476452490505 DifferenceSignificantSignificantSignificantSignificant Student who Never got punished by their teachers performed significantly better Fifty-eight percent students reported they never got punished. Student who Never got punished by their teachers performed significantly better Fifty-eight percent students reported they never got punished. Only.03 % students reported frequent punishment Only.03 % students reported frequent punishment Rewarding students’ performance increased achievement all the subjects Rewarding students’ performance increased achievement all the subjects

21. 21 PTA/ SMC in School: Head Teachers’ Report Around 80% head teachers reported presence of PTA/ SMCs’ in School Presence, need, funding, or role of PTA/SMC did not increase students performance in any of the NEAS tested school Only 39% head teachers reported getting funds from Govt. Ninety-seven percent head teachers reported need for SMCs in Schools

22. 22 Teaching Resources and Achievement Teaching Resources Scaled Scores %MathsUrduScience Social Studies Curriculum Document YesNoDifference4256395392 n. s 384380 476459 382397n.s Teaching Kit YesNoDifference4456396393 n. s 369390Significant465467 483503n.s Teachers Guide YesNoDifference6534394395 n. s 372400Significant465469 485513Significant TextbooksYesNoDifference90.09396373 380389 468448 496495 AV Aids YesNoDifference5841397396 n. s 382379 468465 477483 Only 46% Head teachers reported they have the curriculum document in their school Only 46% Head teachers reported they have the curriculum document in their school Only 37% teachers reported using the curriculum document; use of curriculum document and other teaching resources did not significantly increase students achievement Only 37% teachers reported using the curriculum document; use of curriculum document and other teaching resources did not significantly increase students achievement

23. 23 Teachers General Education in the NEAS Sample General Education Rural % Urban % FemaleMaleFemaleMale Matriculation47233323 Intermediate25233418 Bachelors in Arts 18382340 Bachelors in Science.07161020 Masters.02--- Higher than Masters ---- Male teachers were better qualified than female teachers in both rural and urban areas Male teachers were better qualified than female teachers in both rural and urban areas There were more matriculate female than male teachers There were more matriculate female than male teachers In the NEAS selected schools there were no teachers with higher than masters qualification In the NEAS selected schools there were no teachers with higher than masters qualification

24. 24 Head Teachers’ and Teachers’ Professional Education 0 10 20 30 40 50 60 TeachersHead Teachers Percent PTC CT B.Ed M.Ed Higher than M.Ed About 57% teachers were PTC qualified About 57% teachers were PTC qualified There were more M. Ed head teachers than teachers There were more M. Ed head teachers than teachers There were less than 1% B. Ed Teachers and head teachers There were less than 1% B. Ed Teachers and head teachers

25. 25 Teachers Professional Training Less than 50% subject teachers secured any kind of training Less than 50% subject teachers secured any kind of training Subject curriculum and problem solving were highest reported areas of training obtained Subject curriculum and problem solving were highest reported areas of training obtained

26. 26 Teachers’ Scaled Scores Subject Teacher Scaled Mean Difference TeachersStudents Maths620404 Highly Significant Urdu564382 Difference Science660467 Social Studies 784496 Difference Highly Significant Teachers scaled scores were significantly higher than students in all four subjects Teachers scaled scores were significantly higher than students in all four subjects Teachers scores were well above the set mean of 500 (SD 100) Teachers scores were well above the set mean of 500 (SD 100)

27. 27 Assessment of Students’ Performance in School Method of Assessment Scaled Scores UrduMathsScience Social Studies Verbal Yes Yes No NoDiff.398380n.s416426n.s476479n.s513510n.s Written Yes YesNoDiff.395452Significant419390Significant480463Significant513531Significant Home work Yes Yes No NoDiff.400335Significant419382Significant481418Significant518460Significant Class Performance Yes YesNoDiff.399372Significant417406n.s478466n.s514502n.s

28. 28 Parents Educational Profiles in the NEAS Sample Education Father % Mother % RuralUrbanRuralUrban Illiterate32257253 Primary29271726 Matriculates1723.0413 Intermediate04.08.09.04 Bachelors-.02-- Masters01.01-- Urban fathers were relatively more educated than rural fathers Urban fathers were relatively more educated than rural fathers Only.02% urban fathers were graduates Only.02% urban fathers were graduates Seventy-two percent rural and 53 % urban mothers were illiterate Seventy-two percent rural and 53 % urban mothers were illiterate There were no graduate or postgraduate mothers in NEAS sample There were no graduate or postgraduate mothers in NEAS sample

29. 29 Fathers’ Education and Student Achievement Education Scaled Scores MathsUrduScience S. Studies Illiterate389376462491 Primary399385469497 Matriculates408394476506 Intermediate391407472512 Bachelors431414493569 Masters422399474536 Higher than Masters Higher than Masters398357488478 The table shows a trend that children of fathers with post- primary education performed better on different subjects The table shows a trend that children of fathers with post- primary education performed better on different subjects

30. 30 Fathers’ Occupation in the NEAS Sample Occupation Percent RuralUrban Agricultural Wage Earner.06.03 Agriculture Land Owner 35.08 Skilled Workers 1323 Small Business Owner 1730 Government Job 1215 Private Job 03.07 Majority of rural fathers were agriculture landowners Majority of rural fathers were agriculture landowners Majority of urban fathers were small business owners and skilled workers Majority of urban fathers were small business owners and skilled workers Fifteen percent urban and twelve percent rural fathers in govt. jobs Fifteen percent urban and twelve percent rural fathers in govt. jobs

31. 31 Fathers’ Occupation and Students Achievement Fathers’ occupation Scaled Scores MathsUrduScience S. Studies Agriculture wage earner 408367474507 Agriculture land owner 392375460491 Skilled workers 393389472498 Small business owner 401397477510 Government job 404402473513 Private job 384374471476 Fathers’ occupation did not significantly affect students achievement on Maths and Science tests Fathers’ occupation did not significantly affect students achievement on Maths and Science tests Children of Agricultural wage earner however, scored significantly lower as compared to govt. servants on language test Children of Agricultural wage earner however, scored significantly lower as compared to govt. servants on language test Children of private employed parents scored significantly lower on social studies test Children of private employed parents scored significantly lower on social studies test All other mean differences were non-significant All other mean differences were non-significant

32. 32 Strengthening Links between NEAS and the Educational Planning Process NEAS can support educational planning by: NEAS can support educational planning by: providing data about the actual conditions in schools; andproviding data about the actual conditions in schools; and indicating the possible impact of policy actions on student achievement.indicating the possible impact of policy actions on student achievement. Educational planners can make NEAS more effective by: Educational planners can make NEAS more effective by: proposing topics to be added to background data gathering; andproposing topics to be added to background data gathering; and suggesting relationships to be explored.suggesting relationships to be explored.

33. 33 Conclusion and Next Steps

34. 34 The presentation in summary: NEAS has assessed the achievement of Class IV students in 4 subjects: Urdu, Mathematics, Science and Social studies. NEAS has assessed the achievement of Class IV students in 4 subjects: Urdu, Mathematics, Science and Social studies. The assessment reveals large differences in average performance among provinces and between students from differing backgrounds. The assessment reveals large differences in average performance among provinces and between students from differing backgrounds. These differences in average performance can be traced back to differences in student and family characteristics, school processes and inputs. These differences in average performance can be traced back to differences in student and family characteristics, school processes and inputs. As illustrated in the next slide the factors that influence performance can be divided into four types: As illustrated in the next slide the factors that influence performance can be divided into four types:

35. 35 Additional work: The lack of performance standards limits the interpretability of the assessment results. The lack of performance standards limits the interpretability of the assessment results. Additional analysis is needed to identify causal relationships and hence where policy might yield improvement. Additional analysis is needed to identify causal relationships and hence where policy might yield improvement. The next slides illustrate how multi-level multivariate analyses can be used to reflect on the impact of different policy interventions. The next slides illustrate how multi-level multivariate analyses can be used to reflect on the impact of different policy interventions.

36. 36 Expected changes in achievement over time: All other things being equal one would not expect to see large changes in achievement levels from year to year. Evidence suggests that education reforms take at least 5 to 10 years to bear fruit. All other things being equal one would not expect to see large changes in achievement levels from year to year. Evidence suggests that education reforms take at least 5 to 10 years to bear fruit. For Pakistan one would actually expect test scores to fall over time as rising enrollment rates draw more children from relatively disadvantaged backgrounds into the system. For Pakistan one would actually expect test scores to fall over time as rising enrollment rates draw more children from relatively disadvantaged backgrounds into the system.

37. 37 Future Vision of National Assessment in Pakistan Proposed Recommendations by National Select Committee : NEAS future planning should be till 2020 at Grade 4, 5, 8, 10 and 12 levels NEAS future planning should be till 2020 at Grade 4, 5, 8, 10 and 12 levels Private schools should be included in NEAS sample on Pilot basis in 2008 National Assessment Private schools should be included in NEAS sample on Pilot basis in 2008 National Assessment Listening and speaking competencies (skills) of student assessment would be part of NEAS future activities Listening and speaking competencies (skills) of student assessment would be part of NEAS future activities Pakistan can explore the possibility to participate in the next TIMSS in 2012 Pakistan can explore the possibility to participate in the next TIMSS in 2012

38. 38 Thank You…