Design Talk as a Way to Assess Student Learning of Practices Janet L. Kolodner Jennifer Holbrook, Jackie Gray, Paul J. Camp College of Computing Georgia Institute of Technology

Science Practices  Understanding a problem what might need to be investigated  Investigation with a purpose -- experimentation, modeling, learning from cases,...  Informed decision making, reporting on and justifying conclusions  Iteration towards understanding  Explaining scientifically  Teamwork, collaboration across teams, giving credit

Design Practices  Identifying criteria, constraints, problem specification  “Messing about” with materials  Investigation for the purpose of application  Informed decision making, reporting on and justifying design decisions  Tradeoff and optimization  Iteration towards a good enough solution  Explaining failures and refining solutions  Teamwork, collaboration across teams, giving credit

Learning by Design™ n A project-based inquiry approach to science education for middle school n Students learn science and technology concepts and practices in the context of attempting to achieve design challenges. n Highly collaborative n A variety of “community rituals” are embedded in the approach to promote learning of science and design practices.

Our Units n Physical Science –Apollo 13 – introduction to practices of design and science –Vehicles in Motion – motion and forces –Machines that Help – simple machines and mechanical advantage n Earth Science –Digging In -- introduction to practices –Managing Erosion – erosion and accretion –Tunneling through Georgia – geology, rocks and minerals, rock formations, underground water

A Typical Design Cycle n Challenge is presented n Students mess about to generate questions for inquiry n Investigation to address questions followed by presentation of results to class n Design planning n Pin-up session n Construction & testing n Gallery walk n Additional investigation, demo, reading, discussion of content n Iteration over last three steps to solution

Two LBD™ community rituals n Gallery walks (explanation and presentation of results) n Pin-up sessions (justification) n Ritualized public ways of participating in science and design practices n Well-articulated expectations n Repeatedly practiced and publicly discussed

Research Goals n To what extent are students learning targeted content, and what is responsible for differences? n To what extent are students learning targeted practices, and what is responsible for differences? n How do student conceptions and capabilities develop over time? n What practices by teachers are most conducive to promoting learning?

Assessment Methodologies n Content Test Pre/Post –multiple choice & some essay –what they know and development of their understanding n 20 Questions –self-assessment of capabilities n Videotaped Performance Assessments –design an experiment, run an experiment, analyze data n Videotaped Structured Interviews –interactive design, experiment design, & performance n Ethnography & Informal Data

Performance Assessments n Ask students to carry out practices in the context of an authentic, but short performance task. n Have students work in teams so as to be able to record their deliberations. n Video deliberations. n Code written products and team interactions for evidence of participation in practices. n Repeat periodically throughout experimental period.

PART I: Design and Describe a Simple Experiment in 15 Minutes Race cars and large trucks have very different types of tires. Different kinds of rubber require different amounts of force to overcome sliding friction. In this activity, your group will try to figure out and describe a simple experiment that tests the amount of force needed to overcome sliding friction for different types of rubber and under different road conditions. You have 15 minutes. Read the instructions together and begin. INSTRUCTIONS n Examine the rubber block. It has hard rubber on one side and soft rubber on the other side. n Work together to design and describe a simple experiment that compares the amount of force needed to overcome sliding friction for each kind of rubber. n Your experiment should look at sliding friction for the two types of rubber under several different conditions that might be found on a road surface. n Describe your experiment on the next page.

Coding Categories n Negotiation during collaboration n Distribution of the task n Access to prior knowledge n Adequacy of prior knowledge n Science/design talk n Science practice (experiment design) n Self checks

Scoring the Quality of Practices n Characterizing the practice –1 - didn’t do it –2 – recognized need; minimal ability –3 – novice level –4 – intermediate level –5 – expert level n Characterizing group behavior –1 – nobody did it –2 – one person recognized need and tried –3 – half the group tried –4 – done well half the time –5 – group fluency

Scoring the quality of practices n A 5 -point likert scale for each coding category reflects a quantitative continuum. –1 = Not at all: no evidence of the quality to be rated –2 = Some evidence that at least one episode or one student exhibits the quality rated –3 = The quality is exhibited by half the group –4 = The quality is exhibited for more than half the episodes –5 = The quality completely captures the nature of the episodes

Putting it all together n Code videos of performance tasks so as to document performance capabilities of students over time. n Use documentation of teacher practices to identify high, medium, and low-fidelity LBD classes and the practices of particular teachers. n Analyze across high-, medium-, and low-fidelity LBD classrooms and between LBD and non-LBD classrooms to see differences. n Use observational data to try to explain those differences in terms of the ways the classroom practices and culture and teacher facilitation differed.

Making Observations Consistent: Observational Prompt Tool n Sections focus on n Individual, small-group, whole-class activities -- what to look for in each n LBD rituals – gallery walks, messing about, wb n what to look for in student and teacher behavior when certain goals are active – generating questions for inquiry, focusing investigation, debugging, construction, …

OPT Sample Prompts n Questioning –What are teacher questions about? –What are student questions about? –What question types are being used? –Purpose of teacher questions? –How does teacher deal with off-topic questions? –(each question has a menu of types and a set of examples associated) n Gallery Walks –Who initiates the session? –Who displays the artifact? –Who asks questions? –Who gives feedback? –In what ways is feedback constructive? –What comparisons are made between groups or to previous work of the presenting group? –To what extent are students being explicit about design decisions, expectations, explanations?

LBD’s Foundations n Case-based reasoning’s model of learning from experience (Kolodner, Schank, Hammond, …) n Problem-Based Learning’s model of the classroom (Barrows, …) n Communities of Learners (Brown, Campione), Constructionism (Papert, Harel, Kafai, …), Cognitive Apprenticeship (Collins, Brown, …), architecture studio, Decision-Based Design (Mistree, …), transfer, development,... n A project-based inquiry approach with more.