1. John AndersonTodd Rogers Don Klinger University of VictoriaUniversity of Alberta Queen’s University Charles UngerleiderBarry Anderson Victor Glickman University of British ColumbiaBC Ministry of Education Edudata Canada Funding Canadian Education Statistics Council Social Sciences & Humanities Research Council

2. The project focus Modeling the relationships of student, school, and home characteristics to the achievement of learning outcomes in the domains of reading, writing, mathematics and science  Utilizing hierarchical linear modeling & School Achievement Indicators Program Education Quality & Accountability Office program Alberta Provincial Language Arts & Mathematics Achievement Tests BC Foundational Skills Assessment program datasets

3. Outcomes Data issues Graduate research Findings Next Steps

4. Data issues Complexity of datasets Problem solving – age 13 Problem solving – age 16 Math content – age 13 Math content – age 16 Student achievement tests Student Questionnaires Teacher Questionnaires Principal Questionnaires

5. Data issues Organization of assessment program School-based

6. First, it should be noted that for both Language Arts and Mathematics, most of the variation in achievement was among students : 77.1% in Language Arts 75.1% in Mathematics. class level: 15.3% for Language Arts 15.7% for Mathematics. school level 10.1% for Language Arts 11.3% for Mathematics. The Alberta study

7. __________________________________________________ Test ρ _________________________________________________ SAIP Math 2001 Problem solving – age 130.18 Problem solving – age 160.15 Math content – age 130.19 Math content – age 160.15 OSSLT Reading0.13 Writing0.10 __________________________________________________ PISA average is 0.34 and ranges from.04 to 0.63

8. Data issues Data integrity Student Gender Distribution Gender: Inside questionnaire Gender/Cover MaleFemaleTotal Male 4,456 1,388 5,166 Female1,5634,689 5,589 Total 6,0196,07712,096

9. Data issues Missing Data SAIP Math Parental Educational Level (Items 24 a&b) 34% missing on mother 36% missing on father Parental Vocational Status (Items 25a&b) 53% missing on mother 40% missing on father

10. Data issues Large number of variables

11. Student beliefs about mathematics Derived variables from Student Questionnaire Math is more difficult than other school subjects I am not very interested in mathematics I learn lots of new things in mathematics Math is an important school subject Math is important for my future studies Many good jobs require math

12. Derived variables from Student Questionnaire You & your parents work on math homework You & your parents work on other homework In math course we work in pairs or small groups In math we use math books & magazines In math we had guest speakers or experts In math we use computers In math we use the internet In math we use the computer lab Instructional supports used by students

13. Derived variables from Student Questionnaire Derived variables from Student Questionnaire Instructional practices In math courses this year... The teacher gives notes The teacher shows us how to do problems We participate in math projects We are taught different ways to solve problems The teacher assigns homework We discuss quiz or tests We work alone on assigned work We work on exercises from textbook We study the textbook The teacher reads from the textbook Teachers asks questions of students Students ask teacher questions

14. Causes of math performance To do well in math you need hard work To do well in math you need encouragement - teachers To do well in math you need encouragement - parents To do well in math you need good teaching Derived variables from Student Questionnaire

15. Disciplinary climate In math courses this year... There is noise or disorder in the classroom We lose 5-10 minutes because of disruptions Derived variables from Student Questionnaire

16. Graduate research CSSE 2004 Potential and Pitfalls of Secondary Data Analyses of SAIP data. Todd Rogers & Teresa Dawber, U of Alberta CSSE 2005: The COLO Project 2005 Graduate Symposium Student and school indices in SAIP questionnaires Carmen Gress & Shelley Ross, UVic Correlates of mathematics achievement: a meta-synthesis Margot English, Shelley Ross, Carmen Gress, UVic Issues and results arising from the HLM analysis of the Ontario Secondary School Literacy Test. Chloe Soiblelman, Jinyan Huang, Cheryl Poth, & Don Klinger, Queen’s University Factors that influence writing performance Jiawen Zhou, University of Alberta Stability of SAIP Factor Analysis: Results from school questionnaire items Ally Feng, University of Alberta

17. Findings No grand general models

18. Student level (level 1) coefficients _______________________________________________________________________ Correlate CONTENT PROBLEM 13 16 13 16 _______________________________________________________________________ Student math beliefs.36.33.38.37 Instructional supports -.18 -.22 -.22 -.29 Instructional practices.03.08.04.10 Causes of math -.08 0 -.06 0 Discipline climate 0 0 0 0 Gender 0 -.09.10 0.7

19. School level (level 2) coefficients for average school math score _______________________________________________________________________ Correlate CONTENT PROBLEM 13 * 16 * 13 * 16 * _______________________________________________________________________ Limits to learning -.14 -.21 -.14 -.18 Instructional supports -.12 -.19 -.10 -.19 Causes of math 0 -.22 0 -.22 Discipline climate 0 0 0 -.17 Student math beliefs.13 0.11 0 School climate 0 0 -.04 -.05 Parent engagement 0.05 0.06 Student status.07.06.05 0 Student achievement.05 0.05 0 Instructional practices.09 0 0 0

20. Findings Perhaps no grand models, but As Lindblom (1968, 1990) has pointed out time and again, the desire that models of complex social systems such as public education have an instrumental use remains an elusive dream. Models of complex social systems are likely to be, at best, enlightening – allowing incrementally expanding understandings of complex and dynamic systems such as public schools (Kennedy, 1999)

21. The low rho suggests that Canadian schools are relatively homogeneous & Most variation in achievement results lie within classrooms and between students

22. Findings The specificity of models to grade and domain suggests that the correlates of learning outcomes have to be considered within the context of specific learning situations For example.....

23. Findings The SAIP Math models show that Student attitudes about math are related to achievement Student dependence is related to math achievement The views of school principals in regard to instructional impediments are related to average school math scores Gender tends to have a much reduced relationship to achievement when other variables are entered into the model Climate, Discipline and Parental Involvement – non-operative

24. Next Steps Linkage with other assessment programs Work with other educational partners: Teachers Parents Ministries Communications Data collection Analysis and application