1. Content Analysis Workshop John L. Smithson, Director, Measures of the Enacted Curriculum Alissa Minor, Projects Manager, Measures of the Enacted Curriculum Wisconsin Center for Education Research, University of Wisconsin-Madison johns@wcer.wisc.edu Minneapolis, MN Nov 7-9, 2007 Describing the Content of Standards & Assessments

2. The Goal To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of these three domains of a standards-based approach to education reform and their relationship to one another.

3. The Goal To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of these three domains of a standards-based approach to education reform and their relationship to one another.

4. Content Descriptions Content Descriptions Alignment Analyses Needs Assessment Program Evaluation Monitoring Change Curriculum Management Teacher Reports Content Analyses SEC Taxonomy

5. Uses of Content Analysis Results Descriptive: (Tile Charts and Content Maps) Visual, curriculum-based descriptions of Instructional Targets for teacher reflection, discussion and planning. Predict student achievement gains Control for content to examine other factors As an outcome measure for change over time Examine alignment of Standards & Assessments Analytic: (Alignment)

6. To Describe Instructional Content SEC utilizes a two-dimensional taxonomy based on: Topic by Cognitive Demand 1 2 3 4 5 BCDEF

7. The Content Matrix

8. … adding levels of relative emphasis yields a 3-D construct

10. Content Map Data Displays

12. To Facilitate Comparisons

13. Uses of Content Analysis Results Descriptive: (Tile Charts and Content Maps) Visual, curriculum-based descriptions of Instructional Targets for teacher reflection, discussion and planning. Predict student achievement gains Control for content to examine other factors As an outcome measure for change over time Examine alignment of Standards & Assessments Analytic: (Alignment)

14. Alignment as a Quantity 1.00 0.000.500.250.75 0.27 (Avg. Alignment: Test to Standard) Range of Alignment: Test to Standard) (Based on results for 10 states, across grades 4, 6 and 8: SEC Collaborative 2003) 0.140.40 Aligning Tests to Standards State U Grade 8 Mathematics Alignment: Test to Standard (0.23)

15. Instructional Alignment 1.000.000.50 0.25 0.75 0.030.31 Avg. 0.050.29 Based upon results for 168 teachers, across 3 states: MSP PD Study 2004 Fine Grain Min.Max. 0.17 Avg.Min.Max. Instruction to Standards & Assessments Standards Assessments 0.19

16. Explaining variation in student learning gains

17. Alignment Index: Instruction to Standards Mathematics Across 4 Districts Alignment Analyses for School Improvement Using alignment as an outcome measure (Measuring change in alignment over time) Treatment 99 Control124 Leaders 16 Counts

18. Content Analysis Procedures Exploring the Dimensions of Content But first…. Let’s take a 10 minute break!

19. The Two Dimensions Of Content What students should know [Topics] And… Be Able to Do [Expectations for student performance]

20. Describing the Cognitive Domain 3 4 5 6 How Many Categories? Bloom’s SEC DOK (Webb) SCASS Science

21. Dimensions of Knowing & Inquiry AcquireUseExtend (From: Dimensions of Knowing and Inquiring about Science, State Collaborative on Assessments & Student Standards Science Project, Council of Chief State School Officers, 1997)

22. Depth of Knowledge * Webb, N. 1999. Alignment of Science and Mathematics Standards in Four State. NISE Research Monograph #18. Madison:Wisconsin Center for Education Research. Skill/Concept Recall Strategic Thinking Extended Thinking Level 12341234

23. Skill/ConceptRecall Strategic Thinking Extended Thinking AcquireUseExtend Exploring Cognitive Demand

24. Cognitive Demand (or Expectations for Student Performance) Skill/ConceptRecall Strategic Thinking Extended Thinking AcquireUseExtend Memorize Perform Procedures Demonstrate Understanding Conjecture, Generalize Prove Solve non- routine/ make connections Memorize Conduct Investigations Communicate Understanding Analyze Information Apply concepts /make connections Recall Perform Procedures Analyze/ Investigate Evaluate Generate /Demonstrate

25. Cognitive Demand (or Expectations for Student Performance) Skill/ConceptRecall Strategic Thinking Extended Thinking AcquireUseExtend Memorize Perform Procedures Demonstrate Understanding Analyze Information Evaluate/Apply RecallUnderstandingApplicationAnalyzeEvaluateCreate

26. Exploring Cognitive Demand CgD Immersion Activity Organize into Groups/Tables Each Table w/ CgD Pie Each Person w/ Cgd Descriptors Step 1: Place CgD cards on Pie Slices face-down Step 2: Turn cards over: ID agreements e.g. 2 cards w/ same descriptor in same slice if Group Agrees... discuss key words if not … Discuss … operational definition to distinguish Step 3: Discuss disagreements if consensus reached put in envelope / if not, set aside

27. Content Analysis Materials Cognitive Demand List Topics Lists Comments & Suggestions Worksheet Coding Forms Documents to be analyzed

28. Cognitive Demand Lists Five categories of cognitive demand Slightly different for each subject Each category is defined by a list of descriptors The list of descriptors are not exhaustive Each category stands on its own Each category has an associated letter (B-F)

32. Dimension A: Content Topics Topics List (In your packet of material) Organized at two levels: Content Areas (16 for Mathematics) (27 for Science) (14 for ELAR) Topics (identified by number) (182 Mathematics Topics) (211 Science Topics) (114 ELAR Topics) Plus: non-specific & other

33. Topics Lists Mathematics Science English Language Arts & Reading Cover grades K-12 Organized into Content Areas Topics & Content Areas have an associated #

34. Comments & Suggestion Worksheet One for each reviewer - more available Use to: Record coding conventions/decision rules Suggest/identify additional topics not listed Suggest/identify additional CGD descriptors Provide other comments & suggestions Be sure to turn in at end of workshop (and with mail-in materials, as necessary).

35. Coding Forms Assessment Coding Forms Benchmark Coding Forms Each is used to record content descriptions Each content description consists of A topic number A cognitive demand category letter

36. Assessment Coding Forms

37. Standards Coding Forms

44. Practice Coding Exercise Content Analyzing Assessments (Three code maximum)

45. Practice Coding Exercise Content Analyzing Standards (Six code maximum)

46. The Content Analysis Process ‘Coding’ Teams of 4-5 Content Experts Independent Coding by each Analyst w/ Group Discussion Goal for Process: Generalizability not Inter-rater Reliability Should not be necessary to discuss every item – select by team Pick-up and return documents / coding sheets to Alissa Sign & return to Alissa non-disclosure forms

47. A neutral content language Topics by Cognitive Demand The intended curriculum: State content standards— What students should learn The enacted curriculum: What teachers teach The learned curriculum: Student outcomes based on school learning The assessed curriculum: State (and other) assessments— tested learning Content Analysis Workshop