1. David McConnell Department of Geology & Environmental Science University of Akron May 2008 SERC Woburn Workshop Assessment of Student Learning in Geoscience Courses

2. Introductory Exercise Examine the six questions on page 2. Assume you are a student in classes where these questions would be appropriate. Rank the questions from the easiest to most challenging based on the character of the question and nature of knowledge needed to answer it correctly.

3. Teaching and learning goals can be ordered using Bloom’s Taxonomy Knowledge Comprehension Application Analysis Synthesis Evaluation memorization and recall understanding using knowledge taking apart information reorganizing information making judgements Open-ended questions can be used for all categories. Assessment of Learning Goals Higher-order thinking Content Concepts

4. 1. Knowledge 2. Comprehension 3. Application 4. Analysis 5. Synthesis 6. Evaluation What is... ? Who, what, when, where,...? Describe... What would happen if... ? What does... illustrate about...? What is analogous to... ? How could... be used to... ? What is another example of... ? How does... affect... ? What are the differences (similarities) between... ? How does.. compare or contrast with..? What is a solution for the problem of... ? How would you plan a new.. ? How does X relate to Y? Why is... important? What is the best..., and why? Do you agree/disagree that... ? Open-ended Question Stems King, A., 1995, Teaching of Psychology, v. 22, p. 13-17. Bloom’s Level Question Stems

5. Review the six questions on page 2. Rank the questions using Bloom’s Taxonomy. (See Summary of Bloom’s Taxonomy on page 4 if necessary.) Introductory Exercise

6. Knowledge Comprehension Application Analysis Synthesis Evaluation memorization and recall understanding using knowledge taking apart information reorganizing information making judgements Assessment of Learning Goals Review examples of exercises used in an Earth Science course at UA and place each exercise into one of the six Bloom’s categories using the table on page 5.

7. Bloom’s Taxonomy Exercise Examples ExerciseTaxonomy Level The Rock Cycle MC QuestionsK C Ap An S E Atmospheric Pressure & CondensationK C Ap An S E Life on Earth & MarsK C Ap An S E Earthquake Warning SystemK C Ap An S E Venn Diagram (Hurricanes vs. Tornadoes) K C Ap An S E Reading Quiz: CoastsK C Ap An S E Groundwater RubricK C Ap An S E Earth’s Layers Concept MapK C Ap An S E Relative Time DiagramK C Ap An S E Complete the scoring table on page 5

8. Assessment of SERC Woburn Site Which Bloom’s Level is represented by this exercise? Learning Module 4: Ground-Water Hydrology 1.The water-level data presented on the two maps represent information at discrete locations in the aquifer. Contouring these data requires interpolation between the known data points to produce a continuous surface known as a potentiometric surface. Using the guidelines given in the Chute reference, construct potentiometric surface maps for both sets of water- level data.Chute reference

9. Assessment of SERC Woburn Site Which Bloom’s Level is represented by this question? Learning Module 2: Investigating Cancer and Clusters After reviewing the module information on cancer and reviewing information specific to the Woburn cancer cluster, answer the following questions. 1.The Massachusetts Department of Health referred to the occurrence of cancer in the plaintiff's neighborhood as a "cluster". What is a cancer cluster and how does the medical community define a cluster?

10. Assessment of SERC Woburn Site Learning Module 1: Timeline of Events 2.From the highlighted readings (Effective Expert Witness, Matson, 1999), the author provides suggestions and guidance on methods for being an effective expert witness. Refer to court testimony from days (insert specific references). Compare the testimony presented by the plaintiffs expert hydrogeologist, Dr. George Pinder and Grace's expert hydrogeologist Dr. John Guswa. Explain how each used or ignored Matsons advice regarding expert witness testimony. Which Bloom’s Level is represented by this question?

11. Learning goals can be ordered using Bloom’s Taxonomy – but how do you assess student work? Knowledge Comprehension Application Analysis Synthesis Evaluation Degrees of correctness Right/Wrong answers Open-ended questions can be used for all categories. Assessment of Learning Goals Variation in form/content of answers More complex questions call for more sophisticated guides and responses

12. Writing Assessment with Rubrics A rubric is a scoring protocol that: Specifies criteria and a measurement scale for different levels of proficiency. Establish criteria (e.g., presentation, reasoning) tied to exercise learning goals. Determine categories (e.g., needs improvement, satisfactory, good, exemplary) and scoring scale for components of answer. Analytic vs. Holistic

13. Sample Analytic Rubric Level of AchievementGeneral PresentationReasoning, Argumentation Exemplary (10 pts) 1.Provides a clear and thorough introduction and background 2.Addresses the question 3.Presents arguments in a logical order 4.Uses acceptable style and grammar (no errors) 1.Demonstrates an accurate and complete understanding of the question 2.Uses several arguments and backs arguments with examples, data that support the conclusion Quality (8 pts) 1.Combination of above traits, but less consistently represented (1-2 errors) 2.Same as above but less thorough, still accurate 1.Uses only one argument and example that supports conclusion Adequate (6 pts) 1.Does not address the question explicitly, though does so tangentially 2.States a somewhat relevant argument 3.Presents some arguments in a logical order 4.Uses adequate style and grammar (more than 2 errors) 1.Demonstrates minimal understanding of question, still accurate 2.Uses a small subset of possible ideas for support of the argument. Needs improvement (4 pts) 1.Does not address the question 2.States no relevant arguments 3.Is not clearly or logically organized 4.Fails to use acceptable style and grammar 1.Does not demonstrate understanding of the question, inaccurate 2.Does not provide evidence to support response to the question No Answer (0 pts) Rubric by Dr. Diane Ebert-May, Botany & Plant Pathology, Michigan State University.

14. Open-ended Questions and Rubrics (CT) 2 : Critical Thinking for Civic Thinking Read scenario and answer critical thinking and civic thinking questions with help of taxonomies. Critical Thinking Question Some residents have argued that pollution from the factory is causing cancer in their community. Identify and explain strengths and weaknesses of this conclusion. Civic Thinking Question As an affected citizen of this community, identify and explain a course, or courses, of action that you personally could take to address this situation effectively and responsibly.

15. SOLO Taxonomy (Holistic Rubric) Biggs, J.B., and Collis, K.F., 1982, Evaluating the Quality of Learning: The SOLO taxonomy (Structure of Observed Learning Outcomes), Academic Press. LevelLevel Characteristics PrestructuralNo understanding demonstrated. Response does not address the question or restates the question. UnistructuralLimited understanding of topic. Response focuses on one conceptual item in a complex case. MultistructuralUnderstanding of several discrete components. Response is a collection of multiple items that are not related within the context of the exercise. RelationalUnderstanding of several components that are integrated conceptually. Response prioritizes information and is appropriate to the scale of the question. Extended Abstract Understanding demonstrated at a level extending beyond what has been dealt with in the question prompt. Response generalizes to situations beyond the scope of the question.

16. Answer Type PointsAnswer ComponentsRemarksErrors Prestructural answer 0-10 Response does not address exercise or restates exercise. Score reduced by 1 or more points per error or for incomplete or poorly written answers. Unistructural answer 11-12Only advantages or disadvantages described, key idea not discussed Up to 3 additional points for your statements for/against idea and/or additional data. 13-14One advantage and one disadvantage are partially defined, idea not discussed 15-16One advantage and one disadvantage are well defined, idea not discussed 17One advantage and one disadvantage are partially defined, idea is discussed 18 One advantage and one disadvantage are well defined, idea is discussed Multistructural answer 19-20Multiple advantages and disadvantages partially or well defined, article idea not discussed Up to 3 additional points for your statements for/against idea and/or additional data. 21-22Multiple advantages and disadvantages partially defined, article idea is discussed 23-24 Multiple advantages and disadvantages well defined, key idea is discussed Relational or Extended Abstract answer 25-27Multiple advantages and disadvantages well defined, key idea is discussed, and either answer presents case for/against ideas described or includes a description of new data needed. Additional points may be awarded for examples or analogies beyond those described in article. 28-30Multiple advantages and disadvantages well defined, key idea is discussed, and answer presents case for/against ideas described and answer includes a description of new data needed. SOLO Taxonomy (Analytic Rubric)

17. (Level 0: I have never heard of Bloom’s Taxonomy.) Level 1: I can identify how many levels there are in BT. Level 2: I can place several levels of BT in correct order. Level 3: I can describe the six categories of BT. Level 4: I can explain how I might use BT in planning a lesson. Comprehension Survey #1 Teaching Goal 1: To understand the features of Bloom’s Taxonomy

18. Comprehension Survey #2 Teaching Goal 2: To create exercises representing all categories of Bloom’s Taxonomy Level 1: I can separate out low level and high level questions on the basis of BT. Level 2: I can place simple multiple choice and essay questions using BT. Level 3: I can analyze how to classify a series of exercises based on the categories of BT. Level 4: I can make up questions representative of the six categories of BT.

19. 1.Review existing Learning Modules to assess learning goals and appropriate level of questions. 2.Ensure exercises have multiple levels of learning and sufficient scaffolding of concepts. 3.Consider time and background knowledge necessary to complete exercises. 4.Write and post new questions that integrate multiple levels of Bloom’s Taxonomy. 5.Consider the components of a thorough answer and the development of related rubrics. Assessment of Woburn Modules