Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis.

Slides:



Advertisements
Similar presentations
Performance Assessment
Advertisements

Curriculum Development and Course Design
Intelligence Step 5 - Capacity Analysis Capacity Analysis Without capacity, the most innovative and brilliant interventions will not be implemented, wont.
Action Research Not traditional educational research often research tests theory not practical Teacher research in classrooms and/or schools/districts.
DATA TRACKING AND EVALUATION 1. Goal of the STEP program: To increase the number of STEM graduates within the five-year period of the grant. You have.
Designing Instruction Objectives, Indirect Instruction, and Differentiation Adapted from required text: Effective Teaching Methods: Research-Based Practice.
Experiential Learning Cycle
The “Highly Effective” Early Childhood Classroom Environment
Learning Objectives, Performance Tasks and Rubrics: Demonstrating Understanding and Defining What Good Is Brenda Lyseng Minnesota State Colleges.
How to Integrate Students with Diverse Learning Needs in a General Education Classroom By: Tammie McElaney.
CHAPTER 3 ~~~~~ INFORMAL ASSESSMENT: SELECTING, SCORING, REPORTING.
Russell Pimmel, Roger Seals and Stephanie Beard.  Spring 2010, NSF/DUE Engineering PDs initiate IWBW Series; Spring 2011-CS PDs join  Overall goals.
Identifying Content and Specifying Behaviors
1 CCLI Proposal Writing Strategies Tim Fossum Program Director Division of Undergraduate Education National Science Foundation Vermont.
Evaluation of Education Development Projects Barb Anderegg, Connie Della-Piana, and Russ Pimmel National Science Foundation FIE Conference October 29,
Assessing Student Learning
Project Evaluation Don Millard John Yu March 27, 2012 Guy-Alain Amoussou Lou Everett
Proposal Strengths and Weakness as Identified by Reviewers Russ Pimmel & Sheryl Sorby FIE Conference Oct 13, 2007.
Science Inquiry Minds-on Hands-on.
Interactive Science Notebooks: Putting the Next Generation Practices into Action
At the end of my physics course, a biology student should be able to…. Michelle Smith University of Maine School of Biology and Ecology Maine Center for.
March 20, 2012 Susan Finger & Sue Fitzgerald Division of Undergraduate Education National Science Foundation March 21, 2012 Sue Fitzgerald & Maura Borrego.
1 Evaluation of Education Development Projects CCLI PI Meeting August 15, 2008.
Developing Evaluation Instruments
 A set of objectives or student learning outcomes for a course or a set of courses.  Specifies the set of concepts and skills that the student must.
Rediscovering Research: A Path to Standards Based Learning Authentic Learning that Motivates, Constructs Meaning, and Boosts Success.
New Teacher Preparation: Compass Teacher Evaluation
Connie Della-Piana, Susan Finger, and Sue Fitzgerald Division of Undergraduate Education National Science Foundation CCLI/TUES PI Meeting January 27, 2011.
Learning Objectives. Objectives Objectives: By the conclusion to this session each participant should be able to… Differentiate between a goal and objectives.
TEMPUS IV- THIRD CALL FOR PROPOSALS Recommendation on how to make a good proposal TEMPUS INFORMATION DAYS Podgorica, MONTENEGRO 18 th December 2009.
Interstate New Teacher Assessment and Support Consortium (INTASC)
Evaluation of Education Development Projects Presenters Kathy Alfano, Lesia Crumpton-Young, Dan Litynski FIE Conference October 10, 2007.
Project Evaluation Don Millard John Yu March 27, 2012 Guy-Alain Amoussou Lou Everett
GOAL: To improve conceptual understanding and processing skills  In the context of course ◦ Draw free-body diagrams for textbook problems ◦ Solve 3-D.
CriteriaExemplary (4 - 5) Good (2 – 3) Needs Improvement (0 – 1) Identifying Problem and Main Objective Initial QuestionsQuestions are probing and help.
Project Evaluation Connie Della-Piana Russ Pimmel Bev Watford Workshop for Faculty from Minority Serving Intuitions Feb. 8 –10, 2006.
Monitoring through Walk-Throughs Participants are expected to purpose the book: The Three-Minute Classroom Walk-Through: Changing School Supervisory.
Understanding of the fundamentals ◦ By the end of the class, 70% of the students will be able to:  Correctly draw freebody diagrams of 2D truss structures.
Elizabeth Godfrey 1.  Periodic assessment of results Appropriateness, effectiveness, efficiency, impact, sustainability  Identifies intended and unintended.
EDU 385 Education Assessment in the Classroom
INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION working together to improve education with technology Using Evidence for Educational Technology Success.
Comp 20 - Training & Instructional Design Unit 6 - Assessment This material was developed by Columbia University, funded by the Department of Health and.
Project Evaluation Barb Anderegg and Russ Pimmel Division of Undergraduate Educxcation National Science Foundation Annual ASEE Conference June 24, 2007.
Evaluating a Research Report
T 7.0 Chapter 7: Questioning for Inquiry Chapter 7: Questioning for Inquiry Central concepts:  Questioning stimulates and guides inquiry  Teachers use.
Adriana Signorini, CRTE, SATAL Coordinator Greg Dachner, SSHA, SATAL Student Sharai Kirk, SSHA, SATAL Student How do we know our students are learning?
Teaching Thermodynamics with Collaborative Learning Larry Caretto Mechanical Engineering Department June 9, 2006.
Teaching Today: An Introduction to Education 8th edition
Project Evaluation Barb Anderegg, Connie Della-Piana, Russ Pimmel Division of Undergraduate Education National Science Foundation FIE Annual Conference.
Course and Syllabus Development Presented by Claire Major Assistant Professor, Higher Education Administration.
 An in-class quiz could be called assessment.  Using the in-class quiz results to realize the students are not achieving a learning outcome is an evaluation.
PRINCIPAL SESSION 2012 EEA Day 1. Agenda Session TimesEvents 1:00 – 4:00 (1- 45 min. Session or as often as needed) Elementary STEM Power Point Presentation.
Lecture-3.
Dr. Sande Caton. Assessments Why do we assess our students? Individually, write at least three ideas you have about assessments With one or two colleagues.
LESSON PLANNING What? Why? And How?. Goals of this session Participants will be able to identify and explain: 1.What is a lesson plan and how to develop.
Design of a Typical Course s c h o o l s o f e n g I n e e r I n g S. D. Rajan Professor of Civil Engineering Professor of Aerospace and Mechanical Engineering.
Part 1 1. March 20, 2012 Susan Finger & Sue Fitzgerald Division of Undergraduate Education National Science Foundation March 21, 2012 Sue Fitzgerald &
1.  Most of the information presented in this workshop represents the presenters’ opinions and not an official NSF position  Local facilitators will.
What Are the Characteristics of an Effective Portfolio? By Jay Barrett.
How Students Learn College Teaching Institute Presenter: Monica McCrory The Graduate School.
Course, Curriculum, and Laboratory Improvement (CCLI) Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics PROGRAM.
1. October 25, 2011 Louis Everett & John Yu Division of Undergraduate Education National Science Foundation October 26, 2011 Don Millard & John Yu Division.
Developing a Useful Instrument to Assess Student Problem Solving Jennifer L. Docktor Ken Heller Physics Education Research & Development Group
CDIO: Overview, Standards, and Processes (Part 2) Doris R. Brodeur, November 2005.
Day Two: February 25, :30-3:00. Series Goals Participants will have the opportunity to:  Work collaboratively to:  Deepen their knowledge of the.
Developing a Monitoring & Evaluation Plan MEASURE Evaluation.
IB Assessments CRITERION!!!.
Engage Cobb Conference
Chapter 4 Instructional Media and Technologies for Learning
Presentation transcript:

Project Evaluation Webinar 3 of the Transforming Undergraduate Education in Science, Technology, Engineering and Mathematics Series Scott Grissom & Janis Terpenny April 6, 2011 Sue Fitzgerald & Louis Everett April 7, 2011

2 Janis Terpenny Virginia Tech Mechanical Engineering & Engineering Education Scott Grissom Grand Valley State University Computer Science & Info Systems Sue Fitzgerald Metropolitan State University Computer Science Louis Everett Univ. Texas El Paso Mechanical Engineering

 Most of the information presented in this workshop represents the presenters’ opinions and not an official NSF position  Local facilitators will provide the link to the workshop slides at the completion of the webinar.  Participants may ask questions by “raising their virtual hand” during a question session. We will call on selected sites and enable their microphone so that the question can be asked.  Responses will be collected from a few sites at the end of each Exercise. At the start of the Exercise, we will identify these sites in the Chat Box and then call on them one at a time to provide their responses. 3

 Learning must build on prior knowledge ◦ Some knowledge correct ◦ Some knowledge incorrect – Misconceptions  Learning is ◦ Connecting new knowledge to prior knowledge ◦ Correcting misconceptions  Learning requires engagement ◦ Actively recalling prior knowledge ◦ Sharing new knowledge ◦ Forming a new understanding

 Effective learning activities ◦ Recall prior knowledge -- actively, explicitly ◦ Connect new concepts to existing ones ◦ Challenge and alter misconceptions  Active & collaborative processes ◦ Think individually ◦ Share with partner ◦ Report to local and virtual groups ◦ Learn from program directors’ responses 5

 Coordinate the local activities  Watch the time ◦ Allow for think, share, and report phases ◦ Reconvene on time -- 1 min warning slide ◦ With one minute warning, check Chat Box to see if you will be asked for a response  Ensure the individual think phase is devoted to thinking and not talking  Coordinate the asking of questions by local participants and reporting local responses to exercises 6

The session will enable you to collaborate more effectively with evaluation experts in preparing credible and comprehensive project evaluation plans …. it will not make you an evaluation expert.

After the session, participants should be able to:  Discuss the importance of goals, outcomes, and questions in the evaluation process ◦ Cognitive and affective outcomes  Describe several types of evaluation tools ◦ Advantages, limitations, and appropriateness  Discuss data interpretation issues ◦ Variability, alternative explanations  Develop an evaluation plan in collaboration with an evaluator ◦ Outline a first draft of an evaluation plan

 The terms evaluation and assessment have many meanings ◦ One definition  Assessment is gathering evidence  Evaluation is interpreting data and making value judgments  Examples of evaluation and assessment ◦ Individual’s performance (grading) ◦ Program’s effectiveness (ABET and regional accreditation) ◦ Project’s progress and success (monitoring and validating)  Session addresses project evaluation ◦ May involve evaluating individual and group performance – but in the context of the project  Project evaluation ◦ Formative – monitoring progress to improve approach ◦ Summative – characterizing and documenting final accomplishments

 Think about your favorite course. What types of in class activities could be called: ◦ Assessment versus Evaluation ◦ Formative versus Summative Evaluation  Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion, Selected local facilitators report to virtual group  With one minute warning, check Chat Box to see if you will be asked for a response 10

One Minute Look at Chat Box to see if you will be asked to respond

 Think about your favorite course. What types of in class activities could be called: ◦ Assessment versus Evaluation ◦ Formative versus Summative Evaluation  Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion, Selected local facilitators report to virtual group  With one minute warning, check Chat Box to see if you will be asked for a response 12

 An in class quiz could be called assessment.  Using the in class quiz results to realize the students do not understand how to … because they did not … is an evaluation.  Using the evaluation to change the activity you used to teach … makes it a formative evaluation.  Aha! You find your new activity produces great assessment results makes it a summative evaluation. 13

Project Goals, Expected Outcomes, and Evaluation Questions

 Effective evaluation starts with carefully defined project goals and expected outcomes  Goals and expected outcomes related to: ◦ Project management  Initiating or completing an activity  Finishing a “product” ◦ Student behavior  Modifying a learning outcome  Modifying an attitude or a perception

 Goals provide overarching statements of project intention What is your overall ambition? What do you hope to achieve?  Expected outcomes identify specific observable or measureable results for each goal How will achieving your “intention” be reflected by changes in student behavior? How will it change their learning and their attitudes?

 Goals → Expected outcomes  Expected outcomes → Evaluation questions  Questions form the basis of the evaluation process  The evaluation process consists of the collection and interpretation of data to answer evaluation questions

 Read the abstract -- Goal statement removed  Suggest two plausible goals ◦ One on student learning  Cognitive behavior ◦ One on some other aspect of student behavior  Affective behavior  Focus on what will happen to the students ◦ Do not focus on what the instructor will do  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion, Selected local facilitators report to virtual group

The goal of the project is …… The project is developing computer-based instructional modules for statics and mechanics of materials. The project uses 3D rendering and animation software, in which the user manipulates virtual 3D objects in much the same manner as they would physical objects. Tools being developed enable instructors to realistically include external forces and internal reactions on 3D objects as topics are being explained during lectures. Exercises are being developed for students to be able to communicate with peers and instructors through real-time voice and text interactions. The project is being evaluated by … The project is being disseminated through … The broader impacts of the project are … Two goals: one for student learning and one for student behavior Non engineers should substitute: “Organic chemistry” for “statics and mechanics of materials” “Interactions” for “external forces and internal reactions”

One Minute Look at Chat Box to see if you will be asked to respond

The goal of the project is …… The project is developing computer-based instructional modules for statics and mechanics of materials. The project uses 3D rendering and animation software, in which the user manipulates virtual 3D objects in much the same manner as they would physical objects. Tools being developed enable instructors to realistically include external forces and internal reactions on 3D objects as topics are being explained during lectures. Exercises are being developed for students to be able to communicate with peers and instructors through real-time voice and text interactions. The project is being evaluated by … The project is being disseminated through … The broader impacts of the project are … Two goals: one for student learning and one for student behavior Non engineers should substitute: “Organic chemistry” for “statics and mechanics of materials” “Interactions” for “external forces and internal reactions”

GOAL: To improve conceptual understanding and processing skills  In the context of course ◦ Draw free-body diagrams for textbook problems ◦ Solve 3-D textbook problems ◦ Describe the effect(s) of external forces on a solid object orally  In a broader context ◦ Solve out-of-context problems ◦ Visualize 3-D problems ◦ Communicate technical problems orally ◦ Improve critical thinking skills ◦ Enhance intellectual development

GOAL: To improve ◦ Self- confidence ◦ Attitude about engineering as a career

 Write SMART outcomes for the goals  Specific  Measurable  Attainable  Realistic  Timely  If I achieve the outcomes, I have expectation that I am closer to my goal  Goal: My students will be life long learners  Outcomes – Some of these lack SMART components  Number of books read this year is larger than last year  Seven non-fiction books are read over spring break  Number of visits to the campus library is large  Outcomes provide evidence that goals are being achieved

 Write expected measurable outcomes for each of the following goals: ◦ Improve the students’ understanding of the fundamental concepts in statics (cognitive) ◦ Improve the students’ self confidence (affective)  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion, Selected local facilitators report to virtual group Non engineers may substitute: “Organic chemistry” for “statics”

One Minute Look at Chat Box to see if you will be asked to respond

 Write expected measurable outcomes for each of the following goals: ◦ Improve the students’ understanding of the fundamental concepts in statics (cognitive) ◦ Improve the students’ self confidence (affective)  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion, Selected local facilitators report to virtual group Non engineers may substitute: “Organic chemistry” for “statics”

Understanding of the fundamentals ◦ By the end of the class, 70% of the students will be able to:  Correctly draw freebody diagrams of 2D truss structures  Correctly write Newton’s laws when given a FBD  Describe the effects on member force when one angle in a 2D truss is changed Self-Confidence ◦ By the end of the semester:  30% of the class is willing to show the solution to any homework problem on the board  Self reported test anxiety reduces to 50% of the initial amount  80% will say the class was easier than they expected it would be  50% report they are excited about taking the follow-on course

Understanding of the fundamentals ◦ Are the students better able to describe the effects of changing some variable in a simple problem ◦ Are the students better able to describe the effects of changing some variable in a simple problem as a result of the intervention Self-Confidence ◦ Do the students express more confidence in their solutions ◦ Do the students express more confidence in their solutions as a result of the intervention

Questions “Hold-up your virtual hand” and you will be called upon after we unmute your mike. 30

BREAK 15 min 31

BREAK 1 min 32

Tools for Evaluating Learning Outcomes

 Surveys ◦ Forced choice or open-ended responses  Concept Inventories ◦ Multiple-choice questions to measure conceptual understanding  Rubrics for analyzing student products ◦ Guides for scoring student reports, tests, etc.  Interviews ◦ Structured (fixed questions) or in-depth (free flowing)  Focus groups ◦ Like interviews but with group interaction  Observations ◦ Actually monitor and evaluate behavior Olds et al, JEE 94:13, 2005 NSF’s Evaluation Handbook

Surveys  Efficient  Accuracy depends on subject’s honesty  Difficult to develop reliable and valid survey  Low response rate threatens reliability, validity & interpretation Observations  Time & labor intensive  Inter-rater reliability must be established  Captures behavior that subjects are unlikely to report  Useful for observable behavior Olds et al, JEE 94:13, 2005

 Use interviews to answer these questions: ◦ What does program look and feel like? ◦ What do stakeholders know about the project? ◦ What are stakeholders’ and participants’ expectations? ◦ What features are most salient? ◦ What changes do participants perceive in themselves? The 2002 User Friendly Handbook for Project Evaluation, NSF publication REC

 Originated in physics -- Force Concept Inventory (FCI)  Several are being developed in engineering fields  Series of multiple choice questions ◦ Questions involve single concept  Formulas, calculations or problem solving skills not required ◦ Possible answers include detractors  Common errors -- misconceptions  Developing CI is involved ◦ Identify misconceptions and detractors ◦ Develop, test, and refine questions ◦ Establish validity and reliability of tool ◦ Language is a major issue

 Pittsburgh Freshman Engineering Survey ◦ Questions about perception  Confidence in their skills in chemistry, communications, engineering, etc.  Impressions about engineering as a precise science, as a lucrative profession, etc.  Validated using alternate approaches: ◦ Item analysis ◦ Verbal protocol elicitation ◦ Factor analysis  Compared results for students who stayed in engineering to those who left Besterfield-Sacre et al, JEE 86:37, 1997

 Levels of Intellectual Development ◦ Students see knowledge, beliefs, and authority in different ways  “ Knowledge is absolute” versus “Knowledge is contextual”  Tools ◦ Measure of Intellectual Development (MID) ◦ Measure of Epistemological Reflection (MER) ◦ Learning Environment Preferences (LEP) Felder et al, JEE 94:57, 2005

 Suppose you were considering an existing tool (e. g., a concept inventory) for use in your project’s evaluation of learning outcomes  What questions would you consider in deciding if the tool is appropriate?  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion  Selected local facilitators report to virtual group

One Minute Look at Chat Box to see if you will be asked to respond

 Suppose you were considering an existing tool (e. g., a concept inventory) for use in your project’s evaluation of learning outcomes  What questions would you consider in deciding if the tool is appropriate?  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion  Selected local facilitators report to virtual group

 Nature of the tool ◦ Is the tool relevant to what was taught? ◦ Is the tool competency based? ◦ Is the tool conceptual or procedural?  Prior validation of the tool ◦ Has the tool been tested? ◦ Is there information concerning its reliability and validity? ◦ Has it been compared to other tools? ◦ Is it sensitive? Does it discriminate between a novice and an expert?  Experience of others with the tool ◦ Has the tool been used by others besides the developer? At other sites? With other populations? ◦ Is there normative data?

Question s Hold up your “virtual hand” to ask a question.

Data suggest that the understanding of Concept #2 increased

 Data suggest that the understanding of Concept #2 increased  One interpretation is that the intervention caused the change  List some alternative explanations ◦ Confounding factors ◦ Other factors that could explain the change  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion  Selected local facilitators report to virtual group

One Minute Look at Chat Box to see if you will be asked to respond

 Data suggest that the understanding of Concept #2 increased  One interpretation is that the intervention caused the change  List some alternative explanations ◦ Confounding factors ◦ Other factors that could explain the change  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion  Selected local facilitators report to virtual group

 Students learned the concept out of class (e. g., in another course or in study groups with students not in the course)  Students answered with what they thought the instructor wanted rather than what they believed or “knew”  An external event distorted the pretest data  The instrument was unreliable  Other changes in the course and not the intervention was responsible for the improvement  The characteristics of groups were not similar

 Data suggest that the understanding of the concept tested by Q1 did not improve  One interpretation is that the intervention did cause a change that was masked by other factors  Think about alternative explanations  How would these alternative explanations (confounding factors) differ from the previous list?

Evaluation Plan

 List the topics that need to be addressed in the evaluation plan (a.k.a. summarize)  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion  Selected local facilitators report to virtual group

One Minute Look at Chat Box to see if you will be asked to respond

 List the topics that need to be addressed in the evaluation plan (a.k.a. summarize)  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion  Selected local facilitators report to virtual group

 Name & qualifications of the evaluation expert ◦ Get the evaluator involved early in the proposal development phase  Goals, outcomes, and evaluation questions  Instruments for evaluating each outcome  Protocols defining when and how data will be collected  Analysis & interpretation procedures  Confounding factors & approaches for minimizing their impact  Formative evaluation techniques for monitoring and improving the project as it evolves  Summative evaluation techniques for characterizing the accomplishments of the completed project.

 Workshop on Evaluation of Educational Development Projects ◦  NSF’s User Friendly Handbook for Project Evaluation ◦  Online Evaluation Resource Library (OERL) ◦  Field-Tested Learning Assessment Guide (FLAG) ◦  Student Assessment of Their Learning Gains (SALG) ◦  Science education literature

Identify the most interesting, important, or surprising ideas you encountered in the workshop on dealing with project evaluation  Take min ◦ Think individually ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 4 min  Use THINK time to think – no discussion, Selected local facilitators report to virtual group

One Minute Look at Chat Box to see if you will be asked to respond

Question s Hold up your “virtual hand” to ask a question. Proposal & Award Policies & Procedures Guide

 To download a copy of the presentation- go to:  Please complete the assessment survey-go to: