Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Multiple Vantage Points for Employment-Related Feedback Share the Future IV Conference March 18, 2003 Joseph Hoey and Jack Marr Georgia Tech SUCCEED

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Workshop Coverage  Process for integrating information from multiple sources  Role of employer feedback in overall assessment process  Longitudinal strategy to measure value added  Methodological considerations in using assessment data  Interpreting assessment data  Making it work at Southern University

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Preliminary Questions  Going into this workshop, what are your concerns about employer feedback?  How would you use feedback data? What process would you personally find most useful?

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Relating College and the World of Work  How can those skills relevant to the world of work be assessed against those skills our students are gaining through their programs of study?  What do our students do that relates to employment?  Internships?  Major-related activities?

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Relating College and the World of Work  How can we connect academic evaluation with employment-related evaluation?  What are the similarities?  How can these be tied together to seek some sort of consistency?  Can indirect information and information derived from students or alumni play a role?  Can general education assessment be connected to the world of work?

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Georgia Tech Employer Feedback Model

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Multiple Sources of Career- Related Performance Evaluation  Co-op Employers  Recruiters  Employers of Alumni/Alumnae

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Co-op Employers  All students are evaluated every term  Can potentially evaluate “value added”  Shows “what’s happening now”

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Recruiters  First interface with post-graduation employment  Substantial Filter  Limited Information and Contact  Biased Sample---Students and Recruiters

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Alumni(ae)  Another filter  In the workplace or graduate school  “Real world” perspective  Very large sample needed to yield department-level data  Undeliverable addresses a problem  Sampling, non-response bias

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Employers of Alumni/Alumnae  Another filter  More opportunity to evaluate  Clearer perspective on what’s important  Greater investment  Other biases, e.g., Sampling is based on employee’s permission.

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Methodological Issues  Survey Methods  Sampling  Response Rate  Sources of Bias  Consistency

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education “Triangulation” Co-op Recruiter Employer Alumni(ae)

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education “Triangulation” (at best) Co-op Recruiter Employer Alumni(ae)

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Case Study: Comparing and Seeking Continuity in Results  Subject: Computer Information Systems at Very Humid University (VHU)  Just completed first round of assessment studies  Now looking at the data to figure out what they have, what it means, and how they might want to rethink their assessment process to improve its usefulness  You are there as consultants

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Case Study: Comparing and Seeking Continuity in Results  How would you use the data you have to assess the student outcomes stated?  To what extent can you use the data you have to assess the outcomes listed? Do you see any problems?  What changes to assessment methods do you recommend for Computer Info Systems at VHU?  Do you have any other recommendations?

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Employer Feedback: Development of Process  Reworked survey of recruiters to include items relevant to Criteria 2000  Reworked evaluation instrument completed by employers (supervisors) of co-op students to include items relevant to Criteria 2000  Created Alumni and Employer instruments

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Process Logistics  Project funded by SUCCEED  Negotiations with process owners  Recruiter and Co-op data collected and entered by office of origin; analyzed by Office of Assessment  Alumni and Employer surveys collected by Office of Assessment

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Findings: Co-op  All ratings moderately high or better, with some variability in spring 2001  Lifelong learning and technical skills rated highest  Written and oral communication skills rated lowest

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Findings: Co-op  Disaggregated co-op employer evaluations by department and by student class level within department  Breakdown allows clear demonstration of student knowledge and skill gain through the undergraduate experience.

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Findings: Recruiters  Importance: highest ratings on teamwork, problem solving, ability to apply knowledge, communication.  Preparation: highest ratings on using necessary techniques and skills for practice, problem solving.  Largest “performance gaps” over time: teamwork and communication skills

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Results: Bachelor’s Alumni(ae) Survey  Alumni were asked to rate a set of skills, abilities, and attributes generally expected of a Georgia Tech graduate, first rating the importance of each item relative to their personal employment experience since graduation, and then rating each item relative to how well their education had prepared them.

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Alumni Results: Importance vs. Preparation  There were 6 specific skill areas for which there was a greater than 0.50 difference between mean ratings for importance and mean ratings for preparation: The ability to…  communicate orally,  communicate in writing,  function on teams,  use computing technology in communications,  engage in lifelong learning / self-critique, and  exercise leadership skills.

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Hands-On Activity  Divide into small groups.  Discuss: How could you structure and unify employer feedback at Southern U.? Each group should put together ideas on:  what information is needed  what methods would be appropriate to use  who would have ownership/need to be involved  who would collect, enter, analyze, report data  how best to communicate and use results

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Hands-On Activity  Presentation and Discussion (10 minutes): Small groups present summarized ideas; larger group discusses.

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education What Have Previous Participants Thought?*  What information is needed  What methods would be appropriate to use  Who would have ownership/need to be involved  Who would collect, enter, analyze, report data  How best to communicate and use results * ASEE 2001, Albuquerque

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education What Information Is Needed  Criterion 3, A–K knowledge, skills, and abilities—Importance and preparation  How do our students compare with others?  How do our students compare by levels? — Degree level — Field level Are we preparing students appropriately? — What is important — Where are we succeeding — Where are we falling short What is our minimum acceptable performance? — Measure the “low end” of graduates

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education What Methods Would be Appropriate to Use  Surveys  Interviews — Telephone — Personal  Advisory committees

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Who Would Have Ownership/Need to be Involved  Students  Faculty  Recruiters/employers  Employees (grad/co-op)  Parents  Institution

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education Who Would Collect, Enter, Analyze, Report Data  Involve team (students, faculty, alumni)  Analyze and collect (survey specialists)  Report (faculty, editors, specialists)  Distribute based on type — Co-op/Career Planning and Placement — Dean’s office — Computer Services — Institutional Research — Individual departments

Curriculum 21 SUCCEED Southeastern University and College Coalition for Engineering Education How Best to Communicate and Use Results  Share survey data results and summary information with — Students — Faculty — Industry Advisory Boards — Employers — Alumni  Format data results and summary information with — Statistical analyses — Bar graphs — Trends — Tracking—year-by-year and by class  Disaggregate to department level