1. Oregon Content Standards and Assessment System Evaluation
Prepared for the Oregon Department of Education
by WestEd
Dr. Stanley Rabinowitz
Dr. Edynn Sato
May 18, 2007
WestEd/January 2007

2. Oregon Content Standards and Assessment System Evaluation
Background: Overview
One of several key projects funded by the Gates Foundation
Independent evaluation by WestEd, a non-profit educational research, development, and service agency
WestEd/January 2007

3. Content Standards and Assessment System Evaluation: Why?
Oregon Content Standards and Assessment System Evaluation
Content Standards and Assessment System Evaluation: Why?
Content standards signal what knowledge and skills are valued (breadth and depth)
Current state content standards have undergone periodic revision
Need to continuously revisit and re-evaluate on a set schedule
WestEd/January 2007

4. Content Standards and Assessment System Evaluation: Why?
Oregon Content Standards and Assessment System Evaluation
Content Standards and Assessment System Evaluation: Why?
Rigorous standards and assessments are major levers for improving student achievement
Signal goals
Focus instruction
Provide information for improvement to the state, districts, schools
WestEd/January 2007

5. Content Standards and Assessment System Evaluation: What?
Oregon Content Standards and Assessment System Evaluation
Content Standards and Assessment System Evaluation: What?
WestEd will:
Review the content standards (all grades, academic content areas)
Evaluate the structure of the content standards
Evaluate alignment between the state assessments and the content standards
Make recommendations for improvement of structures and systems
WestEd/January 2007

6. Oregon Content Standards and Assessment System Evaluation
What?
WestEd will address questions regarding:
the structure and quality the state content standards
the alignment of state assessments to state standards
Recommendations will be based on data from the analyses that have practice and policy implications
Research-based information related to assessment and accountability models/practices will be provided as available
WestEd/January 2007

7. Oregon Content Standards and Assessment System Evaluation
Key Deliverables
Preliminary report of an initial review of selected content standards and grades using initial protocol/criteria
Final report for the comprehensive review of the content standards
Final report for the alignment of assessments to content standards
Final report on the structure and quality of Oregon’s content standards and assessments
WestEd/January 2007

8. Content Standards and Assessment System Evaluation: How?
Oregon Content Standards and Assessment System Evaluation
Content Standards and Assessment System Evaluation: How?
WestEd’s independent evaluation will be conducted in three phases:
Preliminary Review of Selected Content Standards (already presented to Board)
Comprehensive Evaluation of Content Standards
Alignment Studies of Content Standards and Assessments
WestEd/January 2007

9. Oregon Content Standards and Assessment System Evaluation
How?
Comprehensive Evaluation of Content Standards
English language arts
Mathematics
Science
Social sciences
Arts
Second languages
Physical education
Health education
Technology
English language proficiency
WestEd/January 2007

10. Oregon Content Standards and Assessment System Evaluation
How?
Alignment Studies of Content Standards and Assessments
Content
English language arts—Reading and Literature
Mathematics
Science
Items
Oregon’s multiple-choice knowledge and skills state test items
Tests
One test blueprint/specification for English language arts, mathematics, and science
WestEd/January 2007

11. Oregon Content Standards and Assessment System Evaluation
How?
WestEd’s independent evaluation will involve analysts with expertise in
large-scale test development
standards development
alignment
measurement and statistics
the content areas
curriculum and instruction
the K-12 student population
No Child Left Behind (NCLB)
Oregon programs and history
WestEd/January 2007

12. Oregon Content Standards and Assessment System Evaluation
How?
Stakeholder engagement:
Discussions—These discussions will not influence WestEd’s independent and objective analyses, rather they will inform the analyses
Surveys (online)
Parents & Guardians (427 completed/593 total)
Educators (286 completed/535 total)
Policy Makers (18 completed/38 total)
WestEd/January 2007

13. Oregon Content Standards and Assessment System Evaluation
How?
Stakeholder groups include:
ODE staff, such as
- Management Team
- Curriculum and Assessment staff
State Board of Education
OAESD Instructional Leader Council
PK-20 Coordination Advisory Group
Literacy Leadership State Steering Committee
Critical Friends Advisory Group
WestEd/January 2007

14. Content Standards and Assessment System Evaluation: When?
Oregon Content Standards and Assessment System Evaluation
Content Standards and Assessment System Evaluation: When?
December 1, 2006 through July 31, 2007
Report of a preliminary review of selected content standards and grades using initial protocol/criteria (January 2007)
Report for the alignment studies (March/April 2007)
Report for the comprehensive review of the content standards (April-June 2007)
Final report on the structure of Oregon’s content standards and assessments (July 2007)
WestEd/January 2007

15. Status: Alignment Studies
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies
The analyses aimed to address the following key questions:
To what degree do the State assessment items reflect the concepts and skills embodied in the State’s academic content standards?
To what degree do the assessment items cover the breadth, depth, and range of complexity of content intended by the State?
WestEd/January 2007

16. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
ELA: Item Pool-Level Analyses
Categorical Concurrence
For all grades except for grade 4, the State’s assessment items had a high level of categorical concurrence overall as well as with each CCG in the grade-level standards.
At grade 4, items had a high level of categorical concurrence overall and with each CCG in the grade-level standards, except for DGUL, DAIL, and ECSI, which had a low level of categorical concurrence.
Range of Depth of Knowledge
Overall, at each grade level, items covered a range of DoK. However, the range of DoK appeared restricted for some CCGs at each grade level.
WestEd/January 2007

17. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
ELA: Item Pool-Level Analyses (continued)
Range of Knowledge (Comprehensiveness/Breadth)
Overall, items covered the range of content represented in the standards at each grade level. With the exception of Grade 3, items appeared to address the breadth of content of each CCG, where the standards were assessable according to test specifications.
Balance of Representation (Emphasis)
Overall, at each grade level there was a balance of representation. However, the actual emphasis of content related to specific CCGs in the item pool and the intended emphasis of content related to the ELA “score reporting categories” per the test specifications were not entirely consistent.
WestEd/January 2007

18. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
ELA: Test-Level Analyses 
The examination of items on the Grade 8 ELA assessment shows coverage of content across all six score reporting categories. The emphasis of content appears relatively consistent for VOCA, READ, and ECSL, less so for DGU, DAI, and ECSI.
WestEd/January 2007

19. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
Math: Item Pool-Level Analyses
 Categorical Concurrence
For all grades, the State’s assessment items had a high level of categorical concurrence overall as well as with each strand in the grade-level standards.
Range of Depth of Knowledge
Overall, at each grade level, items covered a range of DoK. However, the range of DoK appeared restricted for some strands at each grade level.
WestEd/January 2007

20. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
Math: Item Pool-Level Analyses (continued)
Range of Knowledge (Comprehensiveness/Breadth)
Overall, items covered the range of content represented in the standards at each grade, with the exception of Grade 6. At the strand level, items for Grades 7, 8, and CIM appeared to address the breadth of content of each strand; however, this was not the case for Grades 3, 4, 5, and 6.
 Balance of Representation (Emphasis)
Overall, at each grade level there was a balance of representation. However, the actual emphasis of content related to specific CCGs in the item pool and the intended emphasis of content per the test specifications are not entirely consistent.
WestEd/January 2007

21. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
Math: Test-Level Analyses
The examination of items on the Grade 5 mathematics assessments shows that overall the intended and actual distributions of content coverage at the strand level were comparable; that is, no percentage difference exceeded 4%.
WestEd/January 2007

22. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
Science: Item Pool-Level Analyses
Categorical Concurrence
For all grades, the State’s assessment items had a high level of categorical concurrence overall as well as with each CCG in the grade-level standards.
Range of Depth of Knowledge
For all grades and for all strands, except Earth and Space Science at CIM, the items represented a range of depth of knowledge levels from Recall to Strategic Thinking. In CIM, items in the Earth and Space Science strand were aligned to two DoK levels only: Recall and Basic Application.
WestEd/January 2007

23. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
Science: Item Pool-Level Analyses (continued)
Range of Knowledge (Comprehensiveness/Breadth)
For Grade 5, the items in the test pool generally address the entire breadth of the strand content as operationalized in the benchmark standards and eligible content.
For Grade 8, the items in the test pool generally address the entire breadth of the strand content as operationalized in the benchmark standards and eligible content.
For CIM, the items in the test pool generally address the entire breadth of the strand content as operationalized in the benchmark standards and eligible content.
WestEd/January 2007

24. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
Science: Item Pool-Level Analyses (continued)
Balance of Representation (Emphasis)
For Grade 5, there was generally a balanced representation of content, although Life Science had slightly more items than either Physical or Earth Sciences.
For Grade 8, each CCG had a balanced representation of content in the item pool items; however, as was the case with Grade 5, Life Science had slightly more items than either Physical or Earth Sciences.
For CIM, each CCG had a balanced representation of content in the item pool items.
The content for Physical Science, Life Science, and Earth Science were fairly evenly emphasized throughout the entire item pool.
WestEd/January 2007

25. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
Science: Test-Level Analyses
A comparison of the intended and actual distribution of items found that overall, the percentages of items at each strand were comparable; that is, no percentage difference exceeded 6%.
WestEd/January 2007

26. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
General Recommendations
The ODE should evaluate the small set of items that WestEd analysts determined had either partial or no alignment (categorical concurrence) to the standards and consider replacing or revising the items as appropriate in order to strengthen the relationships between the items and standards. Current procedures for item development and review should be analyzed and modified to limit future occurrences of partial or no alignment.
The ODE should verify that the content coverage of the assessed domains is purposeful in terms of breadth, depth, emphasis, and complexity.
WestEd/January 2007

27. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
General Recommendations (continued)
The ODE should consider the assessed content and how content interacts with item format (multiple choice) in order to articulate more clearly its intended range of depth of knowledge (i.e., whether the emphasis and range of DoK reflected in its assessments for each grade level and content area are consistent with expectations for each grade level/content area as well as across grade levels for a content area) and to ensure that the item format appropriately and effectively lends itself to students’ demonstrations of what they know and can do vis-à-vis the assessed content.
WestEd/January 2007

28. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
General Recommendations (continued)
Although the online assessment engine (TESA) is designed to select and administer to students a set of items that satisfy the test specifications, the ODE should consider examining a random sample of these assessments to ensure that the actual sets of items administered are indeed consistent with the test specifications in terms of breadth, depth, emphasis, and complexity. Additionally, documentation of the reliability and accuracy with which the test engine adheres to the state’s test specifications should be produced.
WestEd/January 2007

29. Status: Alignment Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Alignment Studies (continued)
General Recommendations (continued)
The ODE should consider examining the full range of test blueprints vis-à-vis relevant sets of assessment items.
The DoK definitions should be considered during item development to help ensure the full range of complexity of the assessed content, as appropriate.
WestEd/January 2007

30. Status: Standards Evaluation Studies
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies
The analyses aimed to address the following key questions:
Do Oregon’s content standards adequately represent the knowledge and skills that all students should know and be able to do?
Do Oregon’s content standards reflect the appropriate breadth and depth of the content area?
Do Oregon’s content standards have the clarity and consistency needed to adequately guide instruction and assessment?
WestEd/January 2007

31. Oregon Content Standards and Assessment System Evaluation
External Referents
CONTENT AREA
LEVELS
STATE REFERENT
NATIONAL REFERENT
English Language Arts
10 Grades K-8 and CIM
Indiana
Draft 2009 NAEP Reading Framework and 2011 NAEP Writing Framework (grades 4, 8, and 12), McREL Speaking and Listening (grades K-3, 5-7, CIM)
Mathematics
NCTM Principals and Standards for School Mathematics (2000) and Curriculum Focal Points for K-8 Mathematics (2006)
Science
AAAS benchmarks (2001)
WestEd/January 2007

32. External Referents (continued)
Oregon Content Standards and Assessment System Evaluation
External Referents (continued)
CONTENT AREA
LEVELS
STATE REFERENT
NATIONAL REFERENT
Social Sciences: Geography
Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM
Washington
NAEP Geography Framework (2001)
Social Sciences: History
NAEP History Framework (2006)
Social Sciences: Civics
NAEP Civics Framework (2006)
NAEP Economics Framework (2006)
WestEd/January 2007

33. External Referents (continued)
Oregon Content Standards and Assessment System Evaluation
External Referents (continued)
CONTENT AREA
LEVELS
STATE REFERENT
NATIONAL REFERENT
Arts
Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM
Washington
Compendium Arts Standards developed by McREL (2006)
Physical Education
Indiana
Moving into the Future: National Standards for Physical Education (2006)
Health
National Health Education Standards: Achieving Health Literacy
Educational Technology
1 Level (CCGs)
ISTE Standards for Students (2000)
WestEd/January 2007

34. External Referents (continued)
Oregon Content Standards and Assessment System Evaluation
External Referents (continued)
CONTENT AREA
LEVELS
STATE REFERENT
NATIONAL REFERENT
Second Languages
Benchmarks 1 (K-3), 2 (4-5), 3 (6-8), CIM
Indiana
American Council on the Teaching of Foreign Languages (1999)
English Language Proficiency: Forms and Functions
5 Levels of Language Forms and Functions NA
Current National Research on Forms and Functions
WestEd/January 2007

35. Status: Standards Evaluation Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies (continued)
Overall Quality
Oregon’s English Language Arts (ELA) standards overall are of good quality and provide breadth and depth of coverage of Reading, Literature, Writing, Speaking, and Listening. The standards generally are clearly written, focus on important skills/concepts for instruction, and are assessable.
Oregon’s mathematics standards overall are of good quality and reflect a breadth and depth of content coverage. The standards generally are clearly written, focus on important skills/concepts for instruction, and are assessable.
WestEd/January 2007

36. Status: Standards Evaluation Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies (continued)
Overall Quality (continued)
Oregon’s science standards overall are of good quality in terms of clarity and importance for instruction. However, in all grades except for grade 8, analysts determined that the standards did not reflect a range of depth of knowledge. For all grades except for grade 1, analysts determined that the standards did not reflect a breadth of knowledge. With regard to consistency, analysts determined that standards were inconsistent (language, skills, knowledge) in grades 4, 8, and CIM. And, in all grades except for grades 3, 5, 8, and CIM, there were issues of assessability.
Oregon’s social sciences standards overall are of good quality in terms of depth, breadth, consistency, importance for instruction, and assessability. However, overall these standards lack clarity.
WestEd/January 2007

37. Status: Standards Evaluation Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies (continued)
Overall Quality (continued)
Oregon’s arts standards overall are of good quality and cover a breadth and depth of content. The standards generally are clearly written, focus on important skills/concepts for instruction and are assessable.
Oregon's physical education standards overall are of good quality in terms of breadth and depth of content coverage, consistency across benchmark grades, importance for instruction, and degree of assessability. However, analysts determined that these standards are not easily understood due to lack of clarity and precision of language.
Oregon’s health standards overall are of good quality in terms of depth, consistency, importance for instruction, and assessability. For all benchmark levels except for Benchmark 2, the standards overall are of good quality in terms of breadth of content. And, overall these standards lack clarity.
WestEd/January 2007

38. Status: Standards Evaluation Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies (continued)
Overall Quality (continued)
Oregon’s educational technology standards overall are of good quality in terms of depth, breadth, consistency, importance for instruction, and assessability. However, overall these standards lack clarity.
Oregon’s second languages standards overall are of good quality in terms of depth, breadth, clarity, consistency, importance for instruction, and assessability.
Generally, Oregon’s English language proficiency (ELP) forms and functions are presented in a clear and useful format. The proficiency level descriptors clearly describe and differentiate the language skills of students at each level. However, the definitions of language function and forms of language appear incomplete. All the language functions except one are appropriate—literary analysis is typically considered English-Language Arts content rather than a language function. All the language forms except one are appropriate—the language of propaganda is not a form consistent with other elements of form presented by the state; rather it appears to be a genre.
WestEd/January 2007

39. Status: Standards Evaluation Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies (continued)
Alignment to External Referents
Oregon’s ELA standards generally aligned to the Indiana, NAEP and McREL standards in terms of overall content order/sequence and depth of content. However, while the breadth of the Oregon standards appeared comparable to the NAEP and McREL standards, they were not as comparable to the Indiana standards.
Oregon’s mathematics standards were compared to Indiana’s mathematics standards as well as to the NCTM Principles and Standards for School Mathematics (2000) and Curriculum Focal Points for K–8 Mathematics (2006). Oregon’s mathematics standards generally aligned to the NCTM standards and to the Indiana standards with regard to overall depth and breadth of content coverage. However, there was not overall congruence between Oregon and Indiana’s content ordering and sequencing.
WestEd/January 2007

40. Status: Standards Evaluation Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies (continued)
Alignment to External Referents (continued)
Oregon’s science standards were compared to Indiana’s science standards and the AAAS benchmarks (2001).
Generally, the Earth and Space Science standards aligned in terms of overall depth of content; however, they did not align in terms of content order/sequence or overall breadth of content.
Generally, the Physical Science standards aligned to the Indiana standards in terms of content order/sequence, overall depth, and overall breadth of content. The Oregon standards also aligned to the AAAS standards in terms of overall depth and breadth of content; however, they did not align in terms of content order/sequence.
Generally, the Life Science standards aligned to the Indiana standards in terms of content order/sequence, overall depth, and overall breadth of content. However, they did not align to the AAAS standards in terms of content order/sequence, overall breadth, or overall depth of content.
WestEd/January 2007

41. Status: Standards Evaluation Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies (continued)
Alignment to External Referents (continued)
Oregon’s social sciences standards were compared to Washington’s social science standards and the NAEP frameworks.
Generally, the Geography standards aligned in terms of overall content order/sequence with Washington’s standards. However, they did not align in terms of content order/sequence with the NAEP framework or in terms of the overall depth or breadth of content with the Washington standards or the NAEP framework.
Generally, the History standards did not align to the NAEP framework in terms of content order/sequence, depth or breadth of content. Nor did these standards align in terms of overall breadth with the Washington U.S. History and World History standards. However, the Oregon standards aligned to the Washington standards in terms of content order/sequence and overall depth.
WestEd/January 2007

42. Status: Standards Evaluation Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies (continued)
Alignment to External Referents (continued)
Oregon’s social sciences standards (continued).
Generally, the Civics standards aligned in terms of overall content order/sequence with Washington’s standards and the NAEP framework. However, they did not align in terms of overall depth or breadth of content with either the Washington standards or the NAEP framework.
Generally, the Economics standards aligned to the Washington standards and the NAEP framework in terms of content order/sequence, depth and breadth of content.
There was a high degree of alignment between Oregon’s arts standards and those of Washington and McREL. In terms of content order / sequence, Washington’s standards spiral content across different benchmarks in a manner similar to Oregon; many of McREL’s benchmark level standards are not spiraled across different benchmarks. Both external referents show similar depth of content to Oregon’s standards.
WestEd/January 2007

43. Status: Standards Evaluation Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies (continued)
Alignment to External Referents (continued)
Generally, Oregon's physical education standards, the National Association of Sports and Physical Education Standards (NASPE), and the Indiana standards share a common purpose. In broad terms, the nature of the content knowledge/cognitive information and the types of skills students are expected to learn and be able to demonstrate are comparable across the three sets of standards. Oregon differs in depth of content coverage from the NASPE and Indiana State standards.
Oregon’s health standards were compared to Indiana’s health standards as well as to the National Health Education Standards (NHES). Oregon’s standards generally aligned to the Indiana and NHES standards with regard to overall content order/sequence and breadth of content coverage. However, there was not overall congruence in terms of depth of content.
WestEd/January 2007

44. Status: Standards Evaluation Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies (continued)
Alignment to External Referents (continued)
Oregon’s educational technology standards generally aligned to the Washington and NETS standards in terms of overall content order/sequence, depth of content and breadth of content.
While Oregon and Indiana both have the same number of proficiency levels, a comparison of the depth and breadth of the second languages standards across levels in both states shows that the Indiana standards start at a slightly higher level than those in Oregon, and reach a higher level. Oregon’s proficiency levels appear to be based on ACTFL levels Novice-Low through Intermediate-Mid. The standards statements differ in level of detail. Generally, Oregon standards are written more broadly, and Indiana’s are more detailed and specific. Oregon’s set of Functions Supporting Standards describe a similar level of detail as Indiana’s standards.
WestEd/January 2007

45. Status: Standards Evaluation Studies (continued)
Oregon Content Standards and Assessment System Evaluation
Status: Standards Evaluation Studies (continued)
Alignment to External Referents (continued)
Oregon’s English language proficiency (ELP) forms and functions were compared to current national research and literature on forms and functions (e.g., TESOL, CALLA, Butler, et al.). All but one of Oregon’s language functions (literary analysis) are appropriate and relevant for ELD instruction. The external referents include functions that are currently not present among those in the Oregon ELP standards.
WestEd/January 2007

46. Oregon Content Standards and Assessment System Evaluation
Next Steps
Address questions regarding the state’s standards and assessment system—quality, structure, etc.
Make recommendations based on data from the analyses (standards evaluation and assessment alignment) that have practice and policy implications
Make recommendations that are research-based or based on promising/best assessment and accountability models and practices
Consider the feedback/information provided key stakeholders through surveys, meetings, etc.
WestEd/January 2007

47. Oregon Content Standards and Assessment System Evaluation
For more information about the standards and assessment evaluation:
ode.state.or.us/go/real
For more information about WestEd:
WestEd/January 2007