1. Concept Inventories 101 Biology Scholars Program SoTL Institute July, 2011 William Cliff Department of Biology Niagara University

2. Questions to be Addressed What is a concept inventory? How have concept inventories been used in assessment? What are some field-tested inventories available in biology?

3. What is a concept inventory? from Wordle.net

4. What is a concept inventory? Set of field-tested conceptual diagnostic questions (CDQs) Validated −Importance −Authenticity −Interpretation Reliable

5. What is a conceptual diagnostic question? Based on key concepts in a field (as identified by instructors/experts) Formated as a selected response item −Correct answer −Distractors derived from research on student misconceptions Written in plain, ordinary language

6. http://www.wcer.wisc.edu/archive/cl1/flag/cat/diagnostic/diagnostic1.htm Sample Conceptual Diagnostic Question

7. http://www.wcer.wisc.edu/archive/cl1/flag/cat/diagnostic/diagnostic1.htm Sample Conceptual Diagnostic Question BRIEFLY EXPLAIN/DEFEND YOUR RESPONSE

8. http://www.wcer.wisc.edu/archive/cl1/flag/cat/diagnostic/diagnostic1.htm Sample Conceptual Diagnostic Question

9. CDQ: Jared, the Subway® man, lost a lot of weight eating a low-calorie diet. Where did all the fat/mass go? A. The mass was released as CO 2 and H 2 O. B. The mass was converted to energy and used up. C. The mass was converted to ATP molecules. D. The mass was broken down to amino acids and eliminated from the body. D’Avanzo, C. Bioscience 58:1079-1085, 2008. CDQ: When a person loses weight, what happens to the fat in the person’s body? Circle True (T) or False (F) for each option. T F Some the fat is broken down and leaves the person’s body as water and gas. T F Some the fat is converted into energy. T F Some of the fat is used up and disappears. T F Some of the fat is broken down and leaves the person’s body as feces and urine. http://www.biodqc.org/node/515 Conceptual Diagnostic Questions

10. CDQ: The selection of antibiotic-resistant, transformed bacteria is based upon a change in the: A. phenotype of the bacteria B. genotype of the bacteria C. phenotype and genotype of the bacteria D. genotype and physiology of the bacteria E. genotype and morphology of the bacteria Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009. CDQ: Consider a short polar region and a short non-polar region in a long polypeptide chain. When dissolved in water, the polypeptide will most likely fold to form a protein in which: A. the non-polar region is exposed on its surface and the polar region is interior. B. the polar region is exposed on its surface and the non-polar region is interior C. both the non-polar and the polar region are exposed on its surface. D. both the non-polar and the polar region are interior. Shi, J. et al. CBE-Life Sciences Education 9:453-461, 2010. Conceptual Diagnostic Questions

11. Host Pathogen Interaction Concept Inventory Question 1 Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009. TABLE 3A

12. Ans. “b”. genotype “When an organism becomes resistant to antibiotics (when it acquires an antibiotic- resistant gene that has been inserted as a marker), the organism’s genotype has been changed.” Ans. “b”. genotype “This must be a change in the genotype because having antibiotic resistance will not necessarily change the look of an organism (phenotype). It will merely allow it to survive in situations where the antibiotic is present” Host Pathogen Interaction Concept Inventory Question 1 “Defend Your Response” Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009.

13. 1. The selection of antibiotic-resistant, transformed bacteria is based upon a change in the: A. phenotype of the bacteria B. genotype of the bacteria C. phenotype and genotype of the bacteria D. genotype and physiology of the bacteria E. genotype and morphology of the bacteria Percentages of correct answers for question 1 on pre- and post-course surveys Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009. Host Pathogen Interaction Concept Inventory Question 1( revised ) TABLE 5

14. How have concept inventories been used in assessment? To reveal prior knowledge and misconceptions that students bring to the class room To measure conceptual gains associated with different methods of instruction To enhance professional development of faculty and curricular reform

15. Suggestions for Use Adopt already-developed, field-tested instruments −must follow the guidelines for its use for the results to be valid and reliable. −give the assessment as a pre- and post-test. −give all items in the order presented on the instrument. Adopt already-developed test items (CDQs) −cannot directly compare results to those normed from the complete instrument. −may better match course goals. Develop your own test items (CDQs) −match questions closely to course goals. −investigate the research literature before beginning. −build up a bank of well-constructed/researched items. Zeilik, M. Conceptual Diagnostic Tests. Field-Tested Learning Assessment Guide. www.wcer.wisc.edu/archive/cl1/flag/cat/cat.html

16. Limitations Limited number of biology concept inventories are currently available. Inventories may not match assessment goals. Vocabulary – e.g. use of jargon Format – close ended, multiple choice

17. Some field-tested inventories available in biology Introductory Biology (Klymkowsky and Garvin-Doxas, 2008; Klymkowsky et al. 2010) Natural Selection (Anderson et al. 2002) Genetics −Non Majors (Bowling et al. 2008) −Majors (Smith et al. 2008) Host-Pathogen Interactions (Marbach-Ad et al. 2009) Molecular and Cellular Biology (Shi et al. 2010)

18. My Work with Conceptual Diagnostic Testing Case Based Learning-mediated Misconception Repair in Respiratory Physiology Chemistry Misconceptions Associated with Body Calcium Homeostasis Teleological Thinking in Physiology Misconceptions in Renal Physiology Core Principles of Undergraduate Physiology −Developing a Physiology Concept Inventory

19. Physiology of Oxygen Transport by the Blood

20. Conceptual Diagnostic Question A Sa / P O2 Misconception Michael JA, et al.. Am. J. Physiol. 277 (Adv. Physiol. Educ. 22): S127-S135, 1999. 1 st TIER 2 nd TIER

21. Conceptual Diagnostic Question B Sa / P O2 Misconception

22. CBL Remediates Student Misconceptions Cliff, W.H. Adv. Physiol. Educ. 30: 215-223, 2006

23. CBL Improves Conceptual Understanding Cliff, W.H. Adv. Physiol. Educ. 30: 215-223, 2006.

24. Cliff, W.H. Adv. Physiol. Educ. 30: 215-223, 2006 Misconception Repair By CBL Is Progressive

25. Calcium Balance in the Body

26. Conceptual Diagnostic Test about the Chemical Equilibrium of Calcium Salts Cliff W. Adv. Physiol. Educ. 33:323-328, 2009

27. Results

28. CDQ 1 Answer C Students failed to apply the common ion effect to the solubility equilibrium established between calcium, phosphate and calcium phosphate. As a result, they did not use Le Châtelier’s principle to correctly predict the change in phosphate concentration that will occur as a result of the addition of calcium to the solution. Implications for Teaching: Students may not be able to correctly apply the common ion effect to determine changes in the equilibrium concentrations involved in physiochemical reactions. Relevant Misconceptions in Physiology: Hypercalcemia would not alter plasma phosphate concentrations. Addition of sodium bicarbonate to the body fluids would not change the concentration of hydrogen ions. Conclusions

29. Cliff W. Adv. Physiol. Educ. 33:323-328, 2009 Results

30. CDQ2 Answer A Students failed to use the principle that the concentration of a pure solid is constant for a heterogeneous equilibrium. As a result, some students mistakenly applied Le Châtelier’s principle to the situation and predicted that the addition of solid calcium phosphate would cause the reaction to shift to the left and the concentration of calcium to increase. Other students based their answer on notions of chemical dissociation or addition without explicitly considering the chemical equilibrium. Implications for Teaching: Students may mistakenly apply Le Châtelier’s principle to physiological circumstances where it is not appropriate - e.g. the contribution of bone to calcium homeostasis. Relevant Misconception in Physiology: Increased bone mass should cause an increase in the concentration of calcium ions in body fluids. Conclusions

31. Teleological Thinking in Physiology Table 1. Frequency Distribution of Student Responses on 10 Item Conceptual Diagnostic Test Figure 1. % Teleological Answers compared to Course Grade (n=72) Cliff, W.H. FASEB J. 23: 632.12, 2009.

32. Results and Conclusions 1.Overall, 39 % of the answers chosen on the ten-item test were teleological. Percentages ranged from 6% to 76% for individual items (Table 1). Considerable variance exists in student preference for teleological explanations for the cause of different physiological phenomena. 2. There was no correlation between the total number of teleological answers chosen by individual students and final course grade (Figure 1). The preference for teleological answers on the conceptual diagnostic test did not appear to interfere with student ability to think or reason mechanistically in other contexts. 3. Structured interviews indicated that some students internally translate the teleological wording of test answers into mechanistic processes. Internal restructuring indicates that students may accept teleological formulations as answers without necessarily embracing teleological reasoning as a means of explanation. Instructors need not hesitate to use teleological formulation in the class room for its heuristic value, as long as students are able to finish with satisfactory mechanistic explanations for physiological phenomena.

33. Qualitative Conceptual Diagnostic Question on Glomerular Filtration Cliff, W., M.P. Wenderoth, and J. Michael. FASEB J. 17:A816, 2003.

34. Qualitative Conceptual Diagnostic Question on Glomerular Filtration Cliff, W., M.P. Wenderoth, and J. Michael. FASEB J. 17:A816, 2003.

35. Summary  68% of respondents predicted filtrate glucose concentration would increase following increased plasma concentration and no change in GFR.  25% predicted no change.  7% predicted decrease.  Many of the students who predicted no change indicated that they believed that the GFR determined the glucose concentration in the filtrate.  Some students who answered the prediction correctly indicated that they thought both plasma concentration and GFR influenced the filtrate glucose concentration. For example: “Greater filtration means greater [ ] in filtrate.” “more glucose will be filtered so its concentration will increase” “one increases above normal and the other remains constant, so we should see an increase” “if there is a higher glucose concentration per mL, and the mL filtered is the same, there will be more glucose coming in per mL anyway.” These answers suggest that a substantial number of students have the misconception that the concentration of glucose in the filtrate is equivalent to the filtered load.

36. Quantitative Conceptual Diagnostic Question on Glomerular Filtration Cliff, W., M.P. Wenderoth, and J. Michael. FASEB J. 17:A816, 2003.

37. Summary  42 % correctly predicted the filtrate concentration will equal the final plasma concentration.  31% predicted an intermediate concentration between the initial and final plasma concentrations.  17 % predicted no change in concentration.  10% predicted a higher concentration.  Some of the students who predicted an intermediate glucose concentration reasoned that a combination of increased plasma concentration with no change in the GFR leads to an intermediate filtrate concentration.  Some students who predicted correctly indicated that both plasma concentration and GFR influenced the filtrate glucose concentration. “The GFR does not change and so blood concentration won't change as it flows out of the capillary into Bowman's capsule.” “The concentrations will be the same if the rates are the same” As with the qualitative survey, these students express the misconception that the filtrate concentration is equivalent to the filtered load..

38. Summary cont. .  A substantial fraction of students who answered incorrectly (41%) would have given a correct answer if asked to predict the qualitative change in glucose concentration. Only in response to a quantitative query did they gave evidence they held a misconception.

39. CONCLUSIONS  After instruction in renal physiology, a sizeable number of students demonstrate they hold the misconception that the concentration of glucose in the filtrate is equivalent to its filtered load.  This observation reinforces the conclusion from investigations of general models in physiology that student misconceptions about “concentration” and “amount” can have a significant impact on their understanding of renal physiology. IMPLICATIONS FOR TEACHING AND LEARNING  It is important to help students recognize the distinction between “concentration” and “amount” in order for them to avoid developing misconceptions about solute and water transport in the kidney and other areas of physiology.  A combination of qualitative and quantitative diagnostic probes can be useful to uncover different facets of student understanding of physiological concepts.

40. Defining and Assessing the Core Principles for Undergraduate Physiology NSF TUES Grant (2011-2014) PI: Jenny McFarland (Edmonds Community College) CO-PIs: William Cliff (Niagara University), Joel Michael (Rush Medical College), Harold Modell (Physiology Education Research Consortium), Ann Wright (Canisius College) RECRUITING FACULTY PARTICIPANTS to HELP DEVELOP PHYSIOLOGY CONCEPT INVENTORY!

41. More Information D’Avanzo, C. Biology Concept Inventories: Overview, Status, and Next Steps. Bioscience 58:1079-1085, 2008. Knight, J. Biology Concept Assessment Tools: Design and Use. Microbiol. Australia March, 5-8, 2010. Libarkin, J. Concept Inventories in Higher Education Science. National Research Council Promising Practices in Undergraduate STEM Education Workshop 2, Washington, DC, 13-14 October 2008. http://www7.nationalacademies.org/bose/Libarkin_CommissionedPaper.pdf Smith, J and Tanner, K. The Problem of Revealing How Students Think: Concept Inventories and Beyond. CBE-Life Sciences Education 9:1-5, 2010. Zeilik, M. Conceptual Diagnostic Tests. Field-Tested Learning Assessment Guide. www.wcer.wisc.edu/archive/cl1/flag/cat/cat.html