Introduction to Assessment and Evaluation Jeff Froyd Texas A&M University Will be happy to share PowerPoint presentation upon request

Outline Assessment: Facilitating Learning Learning Objectives Outcome (a) Apply knowledge of mathematics, science, and engineering Outcome (i) A recognition of the need for, and an ability to engage in life-long learning Outcome (f) an understanding of professional and ethical responsibility

Froyd Informal Culture Survey Do you have any students that ask questions similar to the following? –Dear Professor, Can you tell me exactly what will be on the upcoming exam?

Froyd Informal Culture Survey Do you have any faculty members that ask questions similar to the following? –Dear Accreditation Expert, Can you tell me exactly what I have to do to prepare for the upcoming outcomes-based accreditation visit?

Purpose of Assessment The purpose of assessment is to facilitate learning.

Purpose of Assessment The purpose of assessment is to facilitate learning for: –Students –Individual faculty members –Department/program faculty (as a group) –College faculty (as a group) –University faculty (as a group)

Side Effects of Assessment Grades for Individual Students Professional Society Accreditation Regional Accreditation Happy Administration

Questions about Assessment Turn to the person sitting next to you. Together develop at least 1 question about assessment and evaluation which you would like addressed.

Questions about Assessment Successful methods to evaluate students in a class How to assess individual learning in group projects? How to evaluate in active methodologies? What is the meaning of an intermediate grade, between 0-10? Theory first, then practice or theory within cases – advantages and disadvantages How to evaluate an evaluator? How to evalute competencies in a group? How to validate the results of an assessment method? What are changes from traditional ways in basic science? Is guaranteeing quality part of the assessment? Is facilitating learning the only objective of assessment, or does it include rewards and punishments? How to evaluate teaching? How do we create a knowledge base for reliable evaluations?

Assessment and Learning Principle No. 1: Expectations for learning need to be clearly stated.

Typical Course Syllabus Faculty Thinking: I want students to be able to understand entropy. Faculty Action: Put “entropy” in course syllabus Student Response: I read the chapter on entropy and can remember many of the terms from the chapter. So I understand! Faculty Desire: In a physical setting, students will be able to recognize applications of the concept of entropy and apply appropriate tools to return the specified quantities.

Learning Objectives A learning objective must contain three basic elements: –Verb describing an observable action –Conditions of this action “When given x you will be able to..." –Level of acceptable performance It must describe learning about which you care Source -

Verbs for Learning Objectives Verbs for constructing concrete objectives: analyze computeclassify collaborate compare appreciate contrast define demonstratedirectderivedesignate discussdisplayevaluate identifyinferintegrateinterpret justifylistorganize graspreportrespondsolicit statesynthesizenameexplain Modified from

Verbs for Learning Objectives Verbs via Levels in Bloom’s Taxonomy Knowledge –Define, describe, list, reproduce, enumerate Comprehension –Classify, explain, discuss, give example, summarize Application –Determine, develop, compute, chart, utilize Analysis –Correlate, diagram, distinguish, outline, infer Evaluation –Compare & contrast, critique, justify, conclude Synthesis –Adapt, combine, compare, contrast, design, generate

Verbs for Learning Objectives Not –Understand (see earlier example) –Know –Appreciate –Value –…

ABET/EAC Learning Outcomes (c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability (d) an ability to function on multi-disciplinary teams (f) an understanding of professional and ethical responsibility Not specific enough!!

Learning Objectives: Examples Outcome (d): Distinguishes effective team process relative to ineffective team process Outcome (c): Evaluate adequacy and consistency of produced problem definition with needs statement, reality Outcome (f): Relates components of an ethical decision together in order to abide by their professional code of ethics.

Importance of Learning Objectives: Example from Physics Mechanics Learning Objective: In a physical setting, select the appropriate law (conservation of energy, conversation of momentum, F=ma …) that is appropriate for finding the specified quantity. Many exam problems rely on ability to select the appropriate law Learners often do not practice the ability, because problems at the end of a chapter provide definitive clues about which law to use

Learning Objectives Engineering Education: Assessment Methodologies and Curriculum Innovation Besterfield-Sacre, M., et al. (2000). Defining the Outcomes: A Framework for EC IEEE Transactions on Education, 43(2),

Learning Objectives Pick a course in which you are very interested. Write a learning objective for the course that has: –Verb describing an observable action –Conditions of this action “When given x you will be able to..." –Level of acceptable performance

Learning Objectives Turn to the person sitting next to you and compare your learning objectives against the requirements. Questions?

Outcome (a) Apply science, engineering, and mathematics Learning Objectives –Opportunity for discipline-specific outcomes Assessment Resources –Concept Inventory Assessment Instruments Instructional Resources –Concept-based Instruction

Physics reform through assessment using the “Force Concept Inventory” Hestenes, Wells and Swackhamer, The Physics Teacher 30, 141 (1992) Revised edition available at The FCI is a 30 question multiple-choice test on the the important concepts of physics. It involves no calculations.

The Force Concept Inventory (FCI) is not comparable to the off-the-cuff multiple choice tests that teachers construct on their own. The carefully constructed distracters for each item are not typical multiple- choice throwaways, but common sense alternatives to Newtonian concepts that amplify the significance of student responses. D. Hestenes, “Who Needs Physics Education Research,” Am. J. Phys. 66 (6), June 1998.

..we have found that nearly 80% of the students could state Newton’s Third Law at the beginning of the course, while FCI data showed that less than 15% of them fully understood it at the end. D. Hestenes, “Who Needs Physics Education Research,” Am. J. Phys. 66 (6), June 1998.

Hestenes, Wells and Swackhamer, The Physics Teacher 30, 141 (1992); Revised edition available at One of the Questions from the Force Concept Inventory

Hestenes, Wells and Swackhamer, The Physics Teacher 30, 141 (1992); Revised edition available at

Hestenes Force Concepts Inventory Results Histogram of Number of Courses vs. Their Normalized Gain ( ) Number of Courses (62 Courses Enrolling 6542 Students) Interactive Engagement Traditional Engagement Bin Centers for Normalized Gain (%) From: R. Hake, Am. J. Phys. 66, (1998), or see hitchcock.dlt.asu.edu/media2/cresmet/hake/ 100% 0% Pretest Posttest x 100 FIPE ‘98-99 FIPE ‘96-97, ‘97-98 CC class, Spring ‘99 FIPE ‘99-00

From: R. Hake, Am. J. Phys. 66, (1998), or see hitchcock.dlt.asu.edu/media2/cresmet/hake/

A “Favorite” Misconception from the MCI Nickel, and probably most other metals, cannot exist as a gas, but only as a solid or a liquid. Nickel can exist as:____ a) solid only b) liquid only c) gas only d) liquid or solid only e) liquid or solid or gas * ASU Pre TAMU Pre TAMU Post a5%6%2% b0%0.4%0% c 0.4%0% d46%57%43% e49%36%53%

Andrew Heckler, Assistant Dean, College of Mathematical and Physical Sciences, Ohio State University

FC Concept Inventories Thermodynamics Materials Strength of Materials Fluid Mechanics Dynamics Heat Transfer Chemistry Signals and Systems Electronics Electromagnetics Computer Engineering Waves Circuits

Other Concept Inventories Determining and Interpreting Resistive Electric Circuit Concepts Test (DIRECT)DIRECT Conceptual Survey on Electricity (CSE), Conceptual Survey on Magnetism (CSM), and Conceptual Survey on Electricity and Magnetism (CSEM)CSEM Test of Understanding of Kinematic Graphs (TUG-K) Statistics Concept Inventory (SCI)SCI Force and Motion Conceptual Evaluation (FMCE)FMCE

Thermal and Transport Science Concept Inventory Project staff –Ronald L. Miller, Colorado School of MinesRonald L. Miller –Ruth A. Streveler, Colorado School of MinesRuth A. Streveler –Barbara M. Olds, Colorado School of MinesBarbara M. Olds –Mary A. Nelson, University of Colorado at Boulder –Michelene Chi, University of PittsburghMichelene Chi

Calculus Concept Question x (equally spaced) f1(x)f2(x) x x x x x x Which of the two functions f1 or f2 is the function and which is its derivative? Explain your answer. Nancy Simpson, Texas A&M

Concept-based Instruction Jay Martin, John Mitchell, University of Wisconsin Focus on Reading Inclusion of Writing Continuous Assessment –Including Active Assessment Provides environment for Active (and Cooperative?) Learning Opportunity for Repetition, Reflection, and Review Built upon a structure of responsive organization and content

Concept-based Instruction Peer Instruction, Eric Mazur Regular use of concept questions –Individual answers –Pair answers –Conversations

Outcome (a) Turn to the person sitting next to you Formulate a question that you have about outcome (a) and/or concept inventories.

Outcome (a) ??

Outcome (i) a recognition of the need for, and an ability to engage in life-long learning Learning Objectives Assessment Resources Instructional Resources nts/assessment_eval/outcome_i.htmlhttp://foundationcoalition.org/home/keycompone nts/assessment_eval/outcome_i.html

Outcome (i) a recognition of the need for, and an ability to engage in life-long learning Learning Objectives: Students will be able to 1.Describe his or her learning style and its strengths and weaknesses, his or her approach to learning and its strengths and weaknesses, his or her stage of intellectual development, and his or her capabilities for self-assessment. 2.Find relevant information and perform a literature review. 3.Solve a problem by creating a hypotheses, reading literature/web, talking to experts, acquiring critical information through modeling, experimenting and discovering.

Outcome (i) a recognition of the need for, and an ability to engage in life-long learning Assessment Resources: Learning Objective 1 –Reflective Judgment Interview –Reason about Current Issues –Measure of Epistemological Reflection –Learning and Study Skills Inventory –Self-Directed Learning Readiness Survey –Study Process Questionnaire –Need for Cognition Scale –Strategic Flexibility Questionnaire –Motivated Strategies for Learning Questionnaire –??

Outcome (i) Lifelong Learning Learning Styles Learning Styles –Myers Briggs Type Indicator (MBTI) –Kolb’s Learning Style Inventory –Felder’s Index of Learning Styles –Dunn and Dunn learning styles instrument

Outcome (i) Lifelong Learning Learning Approaches Learning Approaches / Study Orientations –Surface / Reproducing –Deep / Meaning –Strategic / Achieving Instruments –Lancaster Approaches to Studying Questionnaire (LASQ) –Study Process Questionnaire (SPQ)

Outcome (i) Lifelong Learning Learning Approaches Weinstein: Strategic Learning –Skill, Will, Self-regulation –Instrument: Learning and Study Skills Inventory (LASSI) Self-Directed Learning –Instrument: Self-Directed Learning Readiness Survey (SDLRS)

Outcome (i) Lifelong Learning Intellectual Growth Models –Perry Model of Intellectual Development –Belenky et al.: Women’s Ways of Knowing, –Baxter Magolda’s Model of Epistemological Development –King-Kitchener Model of Reflective Judgment

Outcome (i) Lifelong Learning Intellectual Growth Instruments –Transcribed Interviews –Essays Measure of Intellectual Development (MID) – Perry Measure of Epistemological Reflection (MER) - Baxter Magolda –Likert-scale instruments Learning Environment Preferences (LEP) – Perry Reflective Thinking Appraisal - King-Kitchener

Outcome (i) Lifelong Learning Intellectual Growth Foundation Knowledge Identify The Problem Explore Interpretations & Connections Prioritize Alternatives Envision Strategic Innovation Confused Fact Finder Biased Jumper Perpetual Analyzer Pragmatic Performer Strategic Re-visioner Distinguish relevant & irrelevant Information Read conflicting opinions Relate assumptions & biases Analyze pros & cons Prioritize issues and information Justify assumptions Articulate vision Reinterpret information Steps for Better Thinking Performance Patterns,

Outcome (i) Lifelong Learning Intellectual Growth Felder, R. M., and Brent, R. (2004). The Intellectual Development of Science and Engineering Students. Part 1: Models and Challenges. Journal of Engineering Education, 93(4), Felder, R. M., and Brent, R. (2004). The Intellectual Development of Science and Engineering Students. Part 2: Teaching to Promote Growth. Journal of Engineering Education, 93(4),

Outcome (i) Turn to the person sitting next to you Formulate a question that you have about outcome (i) lifelong learning.

Outcome (i) ??

Outcome (f) an understanding of professional and ethical responsibility Learning Objectives Assessment Resources Instructional Resources nts/assessment_eval/outcome_f.htmlhttp://foundationcoalition.org/home/keycompone nts/assessment_eval/outcome_f.html

Outcome (f) an understanding of professional and ethical responsibility Learning Objectives –Demonstrates an ability to make informed ethical choices –Demonstrates knowledge of a professional code of ethics –Evaluates the ethical dimensions of professional engineering and scientific practice –Demonstrates ethical practice

Outcome (f) an understanding of professional and ethical responsibility Assessment Resources –Defining Issues Test II –Work to assess student ability to resolve ethical dilemmas University of Pittsburgh and Colorado School of Mines Development and testing of an assignment and a rubric