Diane Ebert-May Department of Plant Biology Michigan State University Assessment of Student Learning: strategies and tools for evidence that counts HHMI

Our Team at MSU Doug Luckie - Physiology Janet Batzli - Plant Biology (University of Wisconsin) Scott Harrison - Microbiology Tammy Long - Ecology Debra Linton - Marine Biology Heejun Lim - Chemistry Education Joyce Parker - Biochemistry Duncan Sibley - Geology

“Consensogram” Directions 1. Take one color-coded post-it for each question, write the question # in the corner. 2. Write a number between on each post-it in increments of Do not share responses

“Consensogram” Questions Please respond on a scale of in increments of 10: 1. What proportion of your undergraduate courses were based on active, inquiry- based learning? 2.To what degree should undergraduate courses be based on active, inquiry-based learning? 3.To what degree do the assessments you use (or you experienced as an undergraduate) provide convincing data about student learning? 4.How important is it to use multiple kinds of data to assess your students? 5.How often do you use data to make instructional decisions? 6.In my department, teaching is as important as research and is rewarded accordingly (100 agree - 0 disagree)

Goals for This Workshop As a result of your participation in this workshop, you will... l Develop a practical and theoretical understanding about active and inquiry-based learning. l Use multiple instructional designs and strategies that promote active learning by all students. l Analyze multiple forms of assessment to gather data about students’ understanding. l Use data to identify student misconceptions and subsequently improve instructional design. l Consider teaching and learning in terms of research, recognition, and rewards.

Learning Cycle: Models for Instruction Karplus et al: BSCS »Exploration Engage »Concept IntroductionExplore »Concept ApplicationExplain » Elaborate

True or False? Faculty/graduate students are interested in assessing their students’ learning better, but have limited expertise in doing so.

True or False? Lack of meaningful assessment in undergraduate science education occurs because faculty are satisfied to be less accountable in their teaching than they are in their research.

True or False? Assessing student learning in science parallels what scientists actually do as researchers.

Assessment in Teaching Parallels Assessment in Research â Questions we ask are meaningful, interesting, fundable. â Questions are based on current knowledge and theories. â Data we collect are aligned with questions or hypotheses. â Research designs appropriate for the question and accepted in the field. â Instruments/techniques we use are calibrated, valid, repeatable. â We explain results in the context of our questions. â Results drive our next questions. â Our ideas are peer reviewed for publication/funding…

What is assessment? Data collection with a purpose »students’ understanding »students’ attitudes »students’ skills »instructional design and implementation

Graduate Education Often excellent at preparing individuals to design and carry out disciplinary research.

Graduate Education Often inadequate and haphazard in preparing future faculty/professionals to take on the increasingly complex demands of the professoriate. Teaching not - mentored, peer reviewed, based on accumulated knowledge

Solution: a new model Intergenerational teams in cooperative academic environments »Who: senior faculty, junior faculty, postdoctoral and graduate students. »What: Scholarship of science teaching and learning is fully integrated into the professional culture along with discipline-based activities.

Recognizing and Rewarding Evaluating and Improving Undergraduate Teaching in Science, Technology, Engineering, and Mathematics »National Research Council » »November 15, 2002

Assessment of Learning Curriculum development & assessment of learning are inseparable, so integrate. Do faculty claim knowledge about curriculum? Do faculty claim knowledge about assessment? Why are interdisciplinary teams so useful?

Assessment of Learning Curriculum development & assessment of learning are inseparable, so integrate. Do faculty claim knowledge about assessment? Do faculty claim knowledge about curriculum? Why are interdisciplinary teams so useful?

What are 3 central questions about learning? 1. What do we want our students to know and be able to do? 1.5. What evidence will we accept that students know and can do? 2. How does our teaching help learning?

Cognitive Theory “Learners are not simply passive recipients of information; they actively construct their own understanding.” Svinicki 1991

What Type of Learning? Bloom (1956) Major categories in the Cognitive Domain of Educational Objectives

Convergent Thinking Knowledge - remember material Comprehension - grasp the meaning of material Application - use learned material in new concrete situations –Adapted from Grolund (1970)

Divergent Thinking Analysis - break down material to understand organizational structure Synthesis - put parts together to form a new whole Evaluation - judge value of material for a purpose –Adapted from Grolund (1970)

What is assessment? Data collection with a purpose »Students’ learning »Students’ skills »Students’ attitudes »Inform instructional design and implementation

Research Question How do diagnostic assessment problems reveal student misconceptions and inform instructional design?

Background Learning theory David Hestenes - Physics, ASU ‘Force Concept Inventory’

Goal => Assessment Students will be able to demonstrate their understanding of photosynthesis and cellular respiration. Tools: multiple forms of assessment Feedback loop to instructional design

Common Misconceptions: Photosynthesis & Respiration Photosynthesis as Energy: Photosynthesis provides energy for uptake of nutrients through roots which builds biomass. No biomass built through photosynthesis alone. Plant Altruism: CO 2 is converted to O 2 in plant leaves so that all organisms can ‘breathe’. All Green: Plants have chloroplasts instead of mitochondria so they can not respire. Thin Air: CO 2 and O 2 are gases therefore, do not have mass and therefore, can not add or take away mass from an organism.

Instructional Design Active, inquiry-based learning »Cooperative learning »Questions, group processing, lecture »Homework problems including web-based modules

Radish Problem Experimental Setup: Weighed out 3 batches of radish seeds each weighing 1.5 g. Experimental treatments: »1. Seeds not moistened (dry) placed in LIGHT »2. Seeds placed on moistened paper towels in LIGHT »3. Seeds placed on moistened paper towels in DARK

Problem (cont) After 1 week, all plant material was dried in an oven overnight (no water left) and plant biomass was measured in grams. Predict the biomass of the plant material in the various treatments (use think-pair-share). »Light, No Water »Light, Water »Dark, Water

Results: Weight of Radish Seeds 1.46 g1.63 g 1.20 g Write an explanation about the results. (Remember all treatments started as 1.5g).

Students - Introductory Biology (majors) Two courses: »Group A144 (sem 1 organismal/population) » 145 (sem 2 cell/molecular) »Group B xxx (organismal/population) » 145 (sem 2 cell/molecular)

Assessment Design Multiple iterations/versions of the carbon cycle problem Multiple choice, 3 extended response Administered during instruction »Semester 1 - pretest, midterm, final exam »Semester 2 - final exam

Multiple choice question (pretest) Plants gain a tremendous amount of weight (dry biomass) as they grow from seed to adult. Which of the following substances contributes most to that weight gain a. compounds dissolved in soil water that are take up by plant roots b. water c. molecules in the air that enter through holes in the plant leaves d. organic material in the soil taken up directly by plant roots e. solar radiation

Carbon Cycle Problem (mid) Two fundamental concepts in ecology are “energy flows” and “matter cycles”. In an Antarctic ecosystem with the food web given above, how could a carbon atom in the blubber of the Minke whale become part of a crabeater seal? Note: crabeater seals do not eat Minke whales. In your response include a drawing with arrows showing the movement of the C atom. In addition to your drawing, provide a written description of the steps the carbon atom must take through each component of the ecosystem Describe which biological processes are involved in the carbon cycle.

Grandma Johnson Problem Hypothetical scenario: Grandma Johnson had very sentimental feelings toward Johnson Canyon, Utah, where she and her late husband had honeymooned long ago. Her feelings toward this spot were such that upon her death she requested to be buried under a creosote bush overlooking the canyon. Trace the path of a carbon atom from Grandma Johnson’s remains to where it could become part of a coyote. NOTE: the coyote will not dig up Grandma Johnson and consume any of her remains.

Analysis of Responses Used same scoring rubric for all three problems: Examined two major concepts: Concept 1: Decomposers respire CO 2 Concept 2: Plants uptake of CO 2 Explanations categorized into two groups: Organisms (trophic levels) Processes (metabolic)

Trace Carbon from Whale to Seal (Sem 1) (144 students, n=141) Organism Process Decomposition Respiration Release CO 2 Plant Air Root Glucose Photosynthesis % Concept 1 Decomposers respire CO 2 Concept 2 Plants uptake CO 2

Cellular Respiration by Decomposers ( students, n=63) F (1) = , p <.01 MinkeGrandma Jaguar *** Q 1 WhaleQ 2 Grandma JQ 3 Jaguar % Concept 1: Decomposers respire CO 2

Pathway of Carbon into Primary Producer ( students, n=63) F (1) = 2.700, p = Q 1 WhaleQ 2 Grandma JQ 3 Jaguar % Air Root Concept 2: Plants uptake CO 2

Trace Carbon from Spider Monkey to Jaguar (Sem 2) Respiration NA t=4.082, df=101, p < % (n=63) 0ther (n=40) Concept 1: Decomposers respire CO 2

Pathway of Carbon into Primary Producer (Sem 2) Concept 2: Plants uptake CO (n=63) 0ther (n=40) % Air Root NA t=3.028, df=101, p <.01

So What? Problem sets about major concepts »Diagnostic re: what students understand »Influence instructional design »Active learning opportunities for students »Unveil new alternative conceptions Curricular changes »Bacteria/Archaea metabolism - often skimmed/omitted »Primary production - models in lab »Source/Sink »‘Spiral’ major concepts - over/over/over

Misconceptions => Assessment => Instruction What data do you want from the assessment? What do you do when you identify student misconceptions? How will the data influence your instruction and the learning environment you create?

Gene-DNA-Chromosome l Students could explain transcription & translation but not the relation... “Gene-DNA-Chromosome.” l Concept mapping forces students to “Think different” and confront their (mis) understanding.

Concept Maps

Visual Diagrams or Models are

Concept Maps Visual Diagrams or Models are Reflection & Learning promotes Assessment Used for Organization

Concept Maps Visual Diagrams or Models are Concepts display connected with Linking Words Reflection & Learning promotes Assessment Used for Organization

Concept Maps Visual Diagrams or Models are Knowledge or Understanding represent Concepts display connected with Linking Words Hierarchy has Structure has Reflection & Learning promotes Assessment Used for Organization

Concept Maps Visual Diagrams Or Models are Knowledge or Understanding represent Concepts display connected with Linking Words Hierarchy has Structure has Reflection & Learning promotes Assessment Used for Organization Context is constructed with New Information Prior Knowledge

Multiple Choice … … Concept Maps … … Essay … … Interview high Ease of Assessment low low Potential for Assessment of Learning high Theoretical Framework Ausubel 1968; meaningful learning Novak 1998; visual representations King and Kitchner 1994; reflective judgement National Research Council 1999; theoretical frameworks for assessment Assessment Gradient

Make a draft of a concept map Take those three chapters, concepts from your class, and some post-it notes and make a concept map.

Steps to making a concept map 1.List the concepts: brain, genome, dog, plant 2.Arrange them - rank-order in terms of the top, most general, to at the bottom, most specific. 3.Add linking lines that connect the subordinate concepts under the broader ones. 4.Add linking words that indicate the relationship between two linked concepts e.g., connect to, are found in, build proteins inside.

Concepts PhotosynthesisGlucose RespirationEnergy Carbon cycleWater DecomposersOxygen Primary producers Consumers Carbon dioxide

Make a draft of a concept map Now form a group of three people, merge your post- its and make a concept map of all the science concepts.

Open-ended questions Align with learning goals Align with learning goals What thinking skills do you wish to assess, choose one questioning format What thinking skills do you wish to assess, choose one questioning format »interpret data? »write conclusions from previous work? »describe? »solve a problem?

Writing Open-ended Questions Write a description of the situation. Write a description of the situation. Write the directions for writing. Write the directions for writing. Develop a simple rubric Develop a simple rubric »Conceptual understanding »Content knowledge »Critical-thinking processes »Communication skills

C-TOOLS Concept Connector  MSU

Goal: explain evolution by natural selection

Individual Problem Explain the phenotypic changes in the tree and the animal. Use your understanding of evolution by natural selection.

How do we develop rubrics? Describe the goals for the activity, problem, task Select the assessment tasks aligned with goals Develop performance standards Differentiate levels of responses based on clearly described criteria Rate (assign value) the categories

Scoring Rubric for Quizzes and Homework

Advantages of Scoring Rubrics Improve the reliability of scoring written assignments and oral presentations Convey goals and performance expectations of students in an unambiguous way Convey “grading standards” or “point values” and relate them to performance goals Engage students in critical evaluation of their own performance Save time but spend it well

Limitations of Scoring Rubrics Problem of criteria Problem of practice and regular use Scoring Rubric website: » Sample Rubrics for Organismal Biology

Teaching Portfolio Teaching Awards and Grants Individual Student Contact Development and scholarly activity supporting teaching Evaluation of teaching - self, students, peers »Include samples of student work