1. C16 – Tools for Assessing Learning in Engineering Teri Reed-Rhoads Purdue University School of Engineering Education
Inventions and Impact 2: Building Excellence in Undergraduate Science, Technology, Engineering, and Mathematics (STEM) Education August 2008 Washington, DC

2. Engineering Education Grant Writing Workshop
ASEE Section Meeting
April 3, 2008
Teri Reed-Rhoads
Purdue University

3. Assessment Formative – checking along the way
Summative – final analysis of achieving research goals
Consider an outside evaluator – unbiased, measurable feedback
Consider an external advisory board
Give details, actual tools, timelines, etc.

4. Assessment
Course
Curriculum
Program
Project

5. Types of Assessment Cognitive Domain
Content knowledge (Concepts)
Process knowledge, e.g., teamwork (Skills, Abilities)
Student interest, perceptions, and attitudes (Affective)
Rate change (Achievement)

6. PD’s Response – Goals on Cognitive Changes
Increase understanding of concepts
Ability to solve statics problems
Ability to construct free-body diagrams
Ability to describe verbally the effect of external forces on a solid object
Increase processing skills
Ability to solve out-of-context problems
Ability to visualize 3-D problems
Ability to communicate technical problems
Connie Della-Piana, Russ Pimmel, Bev Watford “Project Evaluation”, Workshop for Faculty from Minority Serving Institutions, Feb. 8 –10, 2006.

7. Concept Inventories
Engineering Related
20 instruments
Science and Mathematics Related
8 instruments
Physics and Astronomy Specific
12 instruments

8. Engineering Related CI

9. Science and Mathematics CI

10. Physics and Astronomy CI*
*Rebecca S. Lindell, Elizabeth Peak and Thomas M. Foster, “Are They All Created Equal? A Comparison of Different Concept Inventory Development Methodologies”, 2006 Physics Education Research Conference Proceedings, American Institute of Physics.

11. Goals on Process Knowledge
Teamwork
Team Effectiveness (Imbrie)
Team Developer (McGourty)
Communications
3-D Visualization
Etc.
Leadership

12. PD’s Response – Goals on Affective Changes
Improve students’ attitude about
Profession
Curriculum
Department
Improve students’ confidence
Improve students’ intellectual development
Connie Della-Piana, Russ Pimmel, Bev Watford “Project Evaluation”, Workshop for Faculty from Minority Serving Institutions, Feb. 8 –10, 2006.

13. Affective Measures
Pittsburgh Freshman Engineering Survey (Besterfield-Sacre)
Attitudes Toward Statistics (Wise)
Computer Science Attitude Survey (Moskal)
Structured Interviews

15. Consider a Graphic

16. PD’s Response – Goals on Achievement Rate Changes
Improve
Recruitment rates
Retention or persistence rates
Graduation rates
Connie Della-Piana, Russ Pimmel, Bev Watford “Project Evaluation”, Workshop for Faculty from Minority Serving Institutions, Feb. 8 –10, 2006.

17. Know the Motivation or Background

18. URMs 30% of population, few in labor force
Women 50% of population, 10% in labor force
# of BS Engineering Degrees
% of Freshman Engineering Class
1966
1977
2002
1975
2001
Under Represented Minorities
1,915
(4.8%)
7,971
(11.6%)
17.4
15.8
Women
146
(0.4%)
1,961
(4.9%)
14,102
(20.5%)
19.9
18.3
Chubin and Babco (2003), Walking the Talk in Retention to Graduation

19. Engineering Disciplines
Retention Rates
Engineering Discipline
Proportion Women
ProportionURM
Total URM
Non-URM
Chemical
32.4
11.5
35.0
48.1
Civil
20.1
13.2
37.9
44.2
Computer
8.8
47.6
58.7
Electrical
11.8
14.3
36.3
47.2
Industrial
29.4
17.7
51.3
Mechanical
12.0
11.0
38.0
48.7
Total
19.0
11.7
38.9
49.5
Chubin and Babco (2003), Walking the Talk in Retention to Graduation
National Science Foundation. Women, Minorities, and Person with Disabilities in Science and Engineering: Arlington, VA, 2000 (NSF )

21. Authentic Teaching Alliance Assessment Cycle

22. Group Questions
What are the known instruments/methods in each of the three areas? (Cognitive, Affective, and Rate Change)
Where have these been used successfully/unsuccessfully?
Where are the gaps in each area?

23. Process One note taker One reporter Everyone discusses
We will report out with 15 minutes left

24. There is no failure. Only feedback. -Anonymous Happy grant writing!

25. Affective
National academy of engineering website with book of surveys

26. Rate Change
Longitudinal measurements, courses, rate changes of minorities, retention, enrollment, GPA for performance measures
Use Associate Dean’s offices as resources for stats, changes, etc.

27. Wish Critical Thinking – one that is free
Definition of engineering changes for students as they go through the curriculum
Digital Systems within computer engineering
Teaming attitude by individual
Measuring how well the student can deal with an out of box situation.
Clearinghouse that explains what already exists, how long is it, how it was used in the past, cliff notes version

28. Wish ABET could sponsor (or someone) or post tools that are being used
More CI
Tools for writing and communication
Life-long learning tools (continuous learning)
Procedural knowledge on engineering design
FREE Critical Thinking

29. Wish
Meta-cognition
Societal context of knowledge, global competencies, service learning, research skills, discourse and collaborative dialogue, oral communication, problem solving, leadership (some instruments from Business schools), ability to integrate knowledge, longitudinal data, qualitative data analysis
Evaluation of case studies

30. Cognitive Math placement test English Placement test
Concept Inventories
FE, GRE – standardized tests
Critical thinking
Criticalthinking.org – has tools, and rubric developed by Gary Brown at Washington State, Watson Glazier (also not free), U of South Carolina EFFECTS (web-based)

31. California Critical Thinking
Collegiate Learning Assessment
CAAP – from ACT
SALG – Student Assessment of Learning Gains
IDEA – teaching assessments
MSLQ – Motivated Strategies for Learning Questionnaire
NSF User Friendly Guide to Assessment

32. Rogers and Sando, Rose-Hulman
Learning Styles Instruments (Kolb, Felder)
Career Instruments (Holland)
FMRI, Caps measuring brain activity
Experience sampling method – student wears a bracelet and when it goes off the student writes down what they are doing (engagement measures)
Calibrated Peer review
Teamwork effectiveness (Imbrie, Smith, Layton)
National Center for Case Studies (Buffalo)