Evaluating the Conceptual Coherence and Rigor of Instructional Materials Florida Association of Science Supervisors 5 May 2005 Jody Bintz, BSCS Center for Professional Development

Think of a district science team working together to select or recommend instructional materials… –What are they saying? –What is influencing their decision? What are they looking at in the materials? What context issues are being considered? –How will they make their decision?

Goal Provide insights into an evidence-based process for evaluating instructional materials that: –Builds the awareness that high-quality instructional materials matter in the learning process for students. –Develops common understandings among teachers about the characteristics of high-quality instructional materials. –Requires teacher dialogue around evidence to build consensus during the decision-making process. –Serves as a professional development strategy to support implementation of instructional materials.

We believe that we found powerful evidence that textbooks exert a strong influence on what teachers teach. This seems to be true in most countries despite differences in the nature and use of textbooks. Textbook coverage is important both for what topics are taught and for the levels of performances and accomplishments expected of students. Whatever should be the case, textbooks have an impact almost everywhere, and it is perilous to ignore the ways in which they at least partially shape what is taught. Why Schools Matter, Schmidt, et. al., {date}

Premises… Student learning is directly linked to the quality of the instructional materials selected. Teachers are more effective in the classroom when they implement high quality instructional materials. Teachers make better decisions about instructional materials when the dialogue is based on evidence. The process of selecting instructional materials presents an opportunity for professional development.

Teacher Dialogue

AIM A process and tools for Analyzing Instructional Materials

Development of the AIM Process and Tools Collaboration between the BSCS Center for Professional Development and the K-12 Alliance of WestEd Adapted and field-tested as part of the NSF funded National Academy for Curriculum Leadership (NACL) Used by thousands of teachers across the country

What is AIM? It is… –A process and set of tools to select instructional materials –A professional development strategy –A collaborative process that uses consensus- building –Based on National Science Education Standards –Linked to the research findings described in How People Learn It is NOT… –A checklist; goes beyond the thumb test –Overly complicated and prescriptive

Standards and Research

Today we will… Pique Challenge Engage Model

Research on How People Learn Key Findings Key Findings for Students Key Findings for Teachers First Students come to the classroom with preconceptions about how the world works. Recognize preconceptions and adjust instruction Second Students must have a deep foundation of usable knowledge and understand facts in the context of a conceptual framework. Understand the content and conceptual framework for a discipline Third Students must be taught explicitly to take control of their own learning by monitoring their progress. Teach metacognitive skills Based on How People Learn, National Research Council, 2000

Application of Key Findings Key Findings Instructional Materials First Include structured strategies to elicit and challenge student preconceptions Incorporate background for the teacher about common preconceptions Second Be organized around a conceptual framework Connect factual information to the framework Provide relevant examples to illustrate key ideas Third Make learning goals explicit Integrate metacognitive skill development into content

Facts and Concepts (Our Definitions) Facts or definitions are pieces of information. Focus is on verifiable and discrete details. Concepts are over-arching ideas that clearly show the relationships between facts. Frequently abstract and often define the discipline.

Science Content: Gathering Evidence Scan the entire unit/chapter and note the overarching concept. Read your assigned section carefully. Write the big ideas or concepts addressed in your section on sticky notes. Write concepts in complete sentences. Limit yourself to one concept per sticky note. Note how the concepts in your section build from those in previous sections, how concepts in your section are extended in subsequent sections, and how the concepts in your section connect to the overarching concept of the unit or chapter Note the context (e.g., real-world, engaging to students) in which the concepts are learned.

Gathering Evidence Science Content Work Students Do Assessment

NACL Team’s Work Pittsburgh Team, Summer 2001

Conceptual Flow Diagram Major Concepts in a Unit from BSCS Biology: A Human Approach

Looking at the Science Content Rubric Note the criteria addressed in the rubrics. Are any of these similar to those we brainstormed previously? For your assigned row, visualize what would be present in “ideal” instructional materials that would justify a score of “5”. In other words, what would the evidence look like?

Applying the Science Content Rubric With your group, give this unit/chapter a score for the bottom three rows in this rubric. Note two or more pieces of evidence that justify each score. On the AIM Summary Sheet, note strengths and limitations for the bottom three rows of the rubric.

AIM Process: Paper Screen Score Sheet

AIM Summary Sheet

AIM includes... Pre-Screen Paper Screen Identify Criteria Gather Evidence Analyze Evidence & Apply Rubric Score Components Summarize Results Scaling-Up for Full Implementation Select Implementation Identify Criteria Gather Evidence Analyze Evidence & Apply Rubric Score Components Summarize Results

The AIM Process: Paper Screen & Implementation Merge Paper Screen and Implementation Results Select Instructional Materials Gather Evidence Identify Criteria Summarize Results Analyze Evidence and Apply Rubric Score Components Gather Evidence Analyze Evidence and Apply Rubric Score Components Gather Evidence Analyze Evidence and Apply Rubric Score Components Gather Evidence Analyze Evidence and Apply Rubric Score Components Science Content Work Students Do Assessment Work Teachers Do Gather Evidence Summarize Results Analyze Evidence and Apply Rubric Score Components Gather Evidence Analyze Evidence and Apply Rubric Score Components Student Understanding Teacher Implementation

Content Work Students Do Assessment Work Teachers Do AIM Summary Sheet

Customize AIM Establish your own criteria Adopt or adapt existing rubrics Vary the weighting system Adjust to fit your schedule and needs

Reflection What insights into instructional materials could AIM provide? What questions have been raised by this brief introduction to AIM?

Why does AIM work? Informed, justifiable, evidence-based decision- making Customized and flexible Leads to rich conversations about learning and teaching science Builds common understandings about high- quality instructional materials Informs PD for scaling-up implementation Develops teacher-leaders

What people say about AIM... “AIM really validates the money spent on curriculum.” “Profitable to team building.” “Teachers liked the process and it made sense to them.” “Very useful for conceptual teaching and understanding materials. It also plays a role in understanding inquiry.” “Vital to our ability to implement back in our district.” “Analyzing materials based on data, not on feeling.” “For the first time (in 32 years), I feel I have some insight into selecting instructional materials.” “It makes great sense. Why would you choose curriculum any other way?”

Options… FASS Leadership Team –Come to BSCS and learn to facilitate the process BSCS Staff –Come to Florida to teach leadership teams around the state how to facilitate the process –Work directly with district teams

Contact Us: Center for Professional Development BSCS 5415 Mark Dabling Blvd. Colorado Springs, CO X 119