STUDENT ASSESSMENT FOR INDEPENDENTS SCHOOLS: BRIEF REMARKS. J. Enrique Froemel, Ph.D. Director, Student Assessment Office.

THE STUDENT ASSESSMENT OFFICE’ S MISSION AND KEY TASKS. “To design and implement a student assessment system that fairly and comprehensively elicits the quality of Qatari education.” Design and permanently update country-wide tests to assess student achievement in key subjects. Design and refine a system for administering country-wide tests to students. Disseminate information on student achievement at the system, classroom and student levels. Develop and update a process for test maintenance and quality monitoring.

FUNCTIONAL DIAGRAM DATA COLLECTION AND MANAGEMENT OFFICE STUDENTASSESSMENTOFFICE SCHOOLEVALUATIONOFFICE EDUCATIONINSTITUTE EVALUATION INSTITUTE EDUCATIONAL REFORM SCHOOL S

OUR PERCEPTION OF EDUCATION: A DYNAMIC VENTURE. Changes generated in students are: Quantitative. Qualitative. In complexity.

OUR PERCEPTION OF EDUCATION SYSTEMS: A NESTED REALITY. 5 SYSTEM I SCHOOL A SCHOOL A SCHOOL B CLASS Y CLASS Z CLASS W CLASS 1 CLASS 2 CLASS 3 CLASS 4

OUR VIEWS ON ASSESSMENT: SEVERAL ROLES, TYPES OF APPROACHES AND INSTRUMENT FORMATS. Three roles: diagnostic, formative and summative. Two basic types of approaches: standardized and non- standardized. Several instrument formats: totally structured (multiple choice), constructed response, essays, portfolios, performance tests, observations.

DIFFERENT ASSESSMENT INFORMATION FOR DIFFERENT DECISION LEVELS. System (SAO) : – Qatar Comprehensive Educational Assessment ( QCEA ) program (every year). – PISA (15 year olds-2006 and 2009). – TIMSS (4 th and 8 th graders-2007 and 2010?). – PIRLS (4 th graders-2006 and 2010). School (SEO and schools): – School Report Cards (every year*) with data from QCEA. – School assessments (at school discretion). Classroom (SAO and schools): – QCEA (every year). – Classroom assessments (at school and teacher discretion). Individual students and/or their parents (SAO, SEO, schools and teachers): – QCEA (every year). – School Report Cards (every year*) with data from QCEA. – School assessments (at school discretion). – Classroom assessments (at school and teacher discretion).

THE QCEA : A SHORT HISTORY. Year 2004: Testing in grades Math and Science test adapted from the U.S. Terra Nova test battery. English tests written to “international standards”. Almost all multiple-choice items. Year 2005: Testing in grades 1-12 (Science 4-12). All subject matters tests aimed for alignment to Qatar Curriculum Standards. Independent alignment audit. About half of the score points came from constructed response items in most tests. Inclusion of a listening test in Arabic and English. Years 2006 and Testing in grades Further efforts to improve alignment of test items to Qatar Curriculum Standards. Performance level setting (workshop with teachers in Qatar) in Years 2008 onwards. Further adjustments in grades to be tested should be expected.

THE QCEA: POPULATION. Geographic Coverage: the whole country. Gender: boys and girls. Type of school: independent. Language of testing: Arabic and/or English (choice of language for the schools in mathematics and science). Types of items: several, more than half are open ended questions. Test elaboration: ETS (Arabic and English) and CTB McGraw-Hill (mathematics and science). Increasing localization already started. Number of subjects: – Approximately 89,000 eligible students in grades 1-12 (2005 figures). – Approximately 14,000 eligible students to be tested in grades 4-11 (2007 projection).

THE QCEA : TEST CHARACTERISTICS Arabic core test: Approx. 60 items in three strands: Listening, Reading, Writing. Longer CR and essay type items, depending on the grades. English core test: Approx. 60 items in three strands: Listening, Reading, Writing. Longer CR and essay type items, depending on the grades. Math core test: 30 items (15 MC / 15 CR). Use of calculators, rulers, protractors. Science core test: 30 items (15 MC / 15 CR). Types of scores reported: performance level location and scaled scores (IRT Three parameter model). A workshop with local teachers took place in 2005, in all subject matters. Longitudinal linkage (year to year comparability) starting Vertical scaling in Arabic and English from grades 4-11 (2006). No vertical scaling in Math and Science. Item pool replenishment : All tests include non-scored field test items. Intended to provide information for : the SEC, teachers, students and parents.

THE QCEA : PERFORMANCE LEVELS. “Meets standards”. A student at the “Meets Standards” level is a student who has completely fulfilled the expectations regarding the acquisition of content and skills required for his/her grade level according to the Qatar curriculum standards. “Approaches standards”. ….a student who has fulfilled the minimum expectations… “Below standards”. …a student who has failed to fulfill the minimum expectations …. – Level 3….He/she may reach the “Approaches Standards” level with some additional effort. – Level 2… with considerable additional effort. – Level 1….with extensive additional effort.

THE QCEA: APPLICATION TIMELINE. Fall First Pilot Test application of QCEA (in subsequent years most pilot applications are embedded in the main one). Spring First Main application of QCEA. Spring Second Main application of QCEA (Baseline). Spring Third Main application of QCEA. Spring 2007.Fourth Main application of QCEA. 2008….Subsequent main applications of QCEA.

ASSESMENT CRITICAL QUALITY ASPECTS. Validity : items should align to the Qatar Curriculum Standards. Depends on the construction on the items and the test. Reliability : tests results should be consistent over time. Depends on the quality and length of the tests. Objectivity : results should be independent of scorers. Depends on the training of scorers and the quality of the scoring process. (*) Difficulty : items and tests should be adequate to student ability. Depends on the construction on items and the scaling method (minimum number of students tested). (*) Comparability from year to year : Results in same grade levels should be comparable between years, irrespective of them coming from different students. Depends on the presence of valid “anchor items” in tests for different years, taken by enough number of students who provide valid responses. (*) Vertical scaling : A common scale should be built, so that results for students in different grade levels could be compared. Depends on the presence of valid “anchor items” in different grades, taken by enough number of students who provide valid responses. (*)