Copyright © 2008 Learning Point Associates. All rights reserved. Surveys of the Enacted Curriculum Cary Cuiccio Lisa Palacios March 6, 2009

2 Learning Point Associates Our Vision An education system that works for all learners. Our Mission We deliver the knowledge, strategies, and results to help educators make research-based decisions that produce sustained school improvement.

Research has found that faculty in successful schools always question existing instructional practice and do not blame lack of student achievement on external causes. — Carl Glickman, 2002

What if? … there was a powerful tool that compared teacher’s reflections of content instruction to state standards, benchmarks and assessments? …you could compare how your teachers teach content compared to how others teach across the nation? …you could have consistency across grade levels on both time teachers spend on content as well as depth of instruction?

What if? … there was a tool that compared teacher’s reflected instructional practices to other teachers’ across a school, district or the nation? …you could use data on instructional quality and content to guide professional development? …you could use anonymous teacher data to start a powerful school discussion about teachers’ curricular and professional development needs based on data and observed differences between student achievement scores and schools curriculum?

Introducing: The SEC  Online web-based surveys in Mathematics, English Language Arts, Social Studies and Science.  Groups data that are reported in user-friendly charts.  Assists in facilitating group discussions for improved instruction across classrooms, schools and districts.  A practical, reliable set of data collection tools used by teachers.

7

State Departments of Education (selected) National Organizations (College Board, ACT, NAEP, WIDA) Council of Chief State School Officers Wisconsin Center for Education Research Learning Point Associates TERC SEC Collaborative

History of the SEC  The Surveys of Enacted Curriculum (SEC) Collaborative Project was initiated in  Surveys are available K-12 for Mathematics, Science, English Language Arts and Reading, and Social Studies.

History of SEC (cont.)  Over the past ten years the Council of Chief State School Officers (CCSSO) and their partners have worked with states to implement the SEC tools for data collection, analysis, and reporting.

Research on the SEC MSP-PD Study (Blank, Smithson, Porter, Nunnaley, Osthoff, 2006) Use of SEC data for program evaluation Reform-Up-Close (Porter, Kirst, Osthoff, Smithson, Schneider, 1993) Validation of teacher self-report survey data. Upgrading Mathematics (Gamoran, Porter, Smithson, White, 1997) First content analysis of assessment using content language. Predictive validity of alignment index comparing instruction & assessments Data on Enacted Curriculum (Blank, Porter, Smithson, 2004) Use of SEC data to facilitate school improvement efforts First content analysis of state standards

Copyright © 2008 Learning Point Associates. All rights reserved. The ELA Survey Part One: Instructional Content and Cognitive Demand

Instructional Content A tool to address a key question:  Is classroom instruction aligned with state content standards and assessments? Content Emphasis Cognitive Expectations

Instructional Topics  Topics provide the framework for data collection and comparison.  Topics include the range of topics found in English Language Arts classrooms, K–12.  Stable framework—not intended for a particular state or curriculum; rather with the content area in mind generally.  Neutral matrix—intends to include all possible topics that could be taught.

100Phonemic awareness1000 Elements of Presentation (Verbal and Written) 200Phonics1100Writing applications 300Vocabulary1200Language Study 400 Text and print features 1300Listening and Viewing 500Fluency1400Speaking and Presenting 600Comprehension1500Forms of Text 700Critical Reasoning1600Genre (fiction or non-fiction) 800Author's craft1700Sources of Text 900Writing processes1800Choice K-12 English Language Arts Reading Content Areas

600Comprehension 601Word meaning from context 602Phrase 603Sentence 604Paragraph 605Main idea(s), key concepts, and sequence(s) of events 606Descriptive elements (e.g., detail, color, condition) 607Narrative elements (e.g., events, characters, setting, and plot) 608Persuasive elements (e.g. propaganda, advertisement, and emotional appeal) 609 Expository or informational elements (e.g., explanation, lists, and organizational patterns such as description, cause-effect, and compare-contrast) 610Technical elements (e.g., bullets, instruction, form, sidebars, etc.) 611Electronic elements (e.g., hypertext links, animations) 612 Strategies (e.g., activating prior knowledge, questioning; making connections, predictions; inference, imagery, summarization, re-telling) 613Self-correction strategies (e.g., monitoring, cueing systems, and fix-up) 614Metacognitive processes (e.g., reflecting about one's thinking) 615Interpreting maps, graphs, charts 616Test-taking strategies 690Other K-12 English Language Arts Reading Specific Content

Cognitive Expectations  State standards and assessments have inherent cognitive expectations.  Instructional activities also have cognitive expectations.  The SEC measures the cognitive expectations of instruction.

Cognitive Expectations English Language Arts and Reading  Memorize and Recall  Perform Procedures and Explain  Generate, Create, and Demonstrate  Analyze and Investigate  Evaluate

Copyright © 2008 Learning Point Associates. All rights reserved. Data Interpretation Part One: Instructional Content/Cognitive Demand aligned to State Standards Contour Maps Tile Charts

Survey Instrument: Alignment  Subject Content  Student Specific Topics  Time Spent on Topic  Student Expectations

Reading SEC Contour Maps

This data is from 134 Grade 3 teachers. Alignment describes how well your instruction aligns to standards or assessments. 1.0 is perfect; anything above.5 is fine. You read this like a topographical map. Darker areas are peaks and represent more time spent on that topic at that cognitive demand level. Here, the 3 rd grade teachers are spending more time on phonemic awareness and phonics, and at more levels of cognitive demand, than the standards require. Same here in terms of topographical maps. Teachers are spending less time on speaking and presenting than the standards require. This map is the NY state standards specific to your grade. You can also look at the NY state assessments. The map on this side will be your data, and only you can access it. That’s why this data is from an entire grade. You can also click on any subtopic and get another map comparing your teaching to standards and/or assessments at a “fine grain” level.

Copyright © 2008 Learning Point Associates. All rights reserved. Activity

The Survey Part Two: Instructional Practices

Instructional Practices A tool to address key questions:  Are instructional practices optimized for student achievement?  Are the professional development efforts aligned with needs and priorities?  Are teachers’ opinions and belief in line with district and school initiatives?

Instructional Practices  Homework  Classroom assessments  Opinions and beliefs  Classroom Instructional preparation  Instructional influences  Professional development  Personal characteristics  Formal course preparation

Copyright © 2008 Learning Point Associates. All rights reserved. Taking the Survey

Completing the Online Survey  Approximately minutes to compete.  High-speed Internet connection required.  May be completed in multiple sittings.  Data are saved as each section is submitted.

Copyright © 2008 Learning Point Associates. All rights reserved. Data Interpretation Part Three: Powerful Discussions

And so you just threw everything together? Matthews, a posse is something you have to ORGANIZE.

What does the SEC show? Instruction Assessment Standards Intersection of what is taught with what is tested. Intersection of what is taught with what is in standards. Intersection of what is tested with what is in the standards. Taught, tested, and in the standards.

Traditional Model for School Improvement Identify “Best Practices” for Achievement Problems Identify Shortcomings in Student Performance

Better School Improvement Identify Appropriate Programmatic Solutions Identify Shortcomings in Student Performance Review Instructional Program Data Relative to Concerns The Surveys of Enacted Curriculum

Applications for SEC  Aligning standards, assessments, and curriculum  Improvement of instruction  Interpreting student assessment results  Needs assessment and program evaluation  Indicators system for monitoring progress

Possible Data Analysis Outcomes  Targeted professional development  Changes in environmental factors  Positive climate of change  Continuous improvement process  Review of curriculum materials  Other data-driven decisions and actions

What SEC isn’t --  Individual Teacher Evaluation The “wall” of 3  A tool to judge alignment without school/district context and/or discussion with staff.  The only piece of data for measuring alignment of teacher instruction.

“Regardless of what a state policy requires or what a district curriculum spells out,” says Andrew C. Porter of Vanderbilt University, “the classroom teacher ultimately decides” what to teach. Education Week, October 8, 2003

Think, Pair, Share  Discuss what you’ve learned with the person next to you and make a few notes. What was the purpose today? Similarities in your thoughts, perspectives? Based on what you read – how could the SEC help to improve your classroom, school or district?

54 Cary Cuiccio P: Lisa Palacios P: