Slide 1 ASSESSMENTS AND CULMINATING EXPERIENCES INSTITUTIONAL WORKSHOP Mr. Bill Kohlruss-Dr. Mark Sherman CO-CHAIRS, GUIDANCE COMMITTEE FEBRUARY 21, 2013

ASSESSMENTS AND CULMINATING EXPERIENCES Where Have We Gone Astray? Based on Feedback From Accreditation Visits…

ASSESSMENTS AND CULMINATING EXPERIENCES COMPILATION OF RECOMMENDATIONS February 2009 to July 2012 Recommendations related to: Program Educational Objectives – 48 Culminating Experience – 23 Assessment – 19 Outcomes – 14 Safety – 11 Industry Advisory Boards/Committees – 10 Curriculum – 8 Faculty – 6 Facilities, Equipment & Services – 4 Thanks Ceci & Vic

ASSESSMENTS AND CULMINATING EXPERIENCES Number ONE on the list (48 Hits) is Program Educational Objectives ! Program Education Objectives was addressed in depth at the Workshop last February and we are addressing that weak area through a change in wording in the Criteria… Now called “Program Mission and Educational Goals”. New terminology, perhaps easier to understand ?

ASSESSMENTS AND CULMINATING EXPERIENCES So… Number 2 and 3 are on the agenda for today. #2 Culminating Experiences. #3 Assessments.

ASSESSMENTS AND CULMINATING EXPERIENCES To put a POSITIVE SPIN on today’s workshop… We have asked Dr. Beth Bjerke and Dr. Mark Dusenbury to do a presentation on “How They Assessed the Programs at UND”. This area was reported as a STRENGTH in the Visiting Team Report during the reaffirmation of accreditation by AABI.

Beth Bjerke and Mark Dusenbury

ASSESSMENT “Assessment is the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development” (Palomba and Banta, 1999, pg. 4)

Assessment Process  Establishing your Program Mission  Defining your Program Educational Goals  Defining your Student Learning Outcomes (AABI General, Core and Program)  Identifying your Techniques and Target Groups  Data Collection  ‘Closing the Loop’ process  Implementing changes to improve student learning  Assessing the Assessment Process!

Techniques and Target Groups  Techniques  Direct Assessment of Student Learning  Written Exams  Performance Evaluations  Assignments  Target Group – Students  Indirect Assessment of Student Learning and Program Quality  Surveys  Target Group - Students, Alumni, Industry  Interviews/Focus Groups  Target Group - Students, Alumni, Industry

Survey Data

Designing Surveys Online Survey Tools Easy to use and abuse Set up a timeline and stick to it Have a purpose for the survey Share results with faculty and students Use to improve the learning experience

Gather Indirect Perception Data (AABI a-k from Students)

Gather Indirect Perception Data (AABI a-k from Alumni)

AABI General Outcomes An ability to apply knowledge of mathematics, science and applied sciences An ability to analyze and interpret data An ability to function on multi-disciplinary teams An understanding of professional and ethical responsibilities An ability to communicate effectively, including both written and verbal communication skills A recognition of the need for, and an ability to engage in, life long learning A knowledge of contemporary issues An ability to use the techniques, skills and modern technology necessary for professional practice An understanding of the national and international aviation environment An ability to apply pertinent knowledge in identifying and solving problems An ability to apply knowledge of business sustainability to aviation 3.10

Closing the Loop with Survey Data  Take the time to have the conversation with faculty  Make recommendations and action items  Document the conversation!

Interviews and Focus Groups  Students  Have a lot to say, and many great ideas  Focus Group Discussions  Graduating Senior Interviews

Interviews and Focus Groups  Industry/Alumni  Use of Advisory Board  Other Sources  Campus Visitors  Document!

Closing the Loop with Interviews/Focus Groups  Take the time to have the conversation with faculty  Make recommendations and action items  Document the conversation!

Direct Learning  Course Prime Reports  Stage Check Analysis  Capstone

Course Mapping

Course Prime Reports

Stage Check Analysis

Capstone  Writing analysis between majors in capstone courses  Infant stages – small sample size – will get better with time  Food for thought at this time

Techniques Timeline

Faculty Outcome Discussions

MOST IMPORTANT ASPECT TO ASSESSMENT… Using what is learned through the process to improve the student learning experience in your program!

ASSESSMENTS AND CULMINATING EXPERIENCES To address our second topic we have Ms. Andrea Georgiou and Mr. Paul Carlson doing a presentation on “Our Capstone/ Culmination Experience” at MTSU. This area was reported as a STRENGTH in the Visiting Team Report during the reaffirmation of accreditation by AABI.

CULMINATING EXPERIENCES Middle Tennessee State University Andrea M. Georgiou Paul R. Carlson FOCUS Lab Project Middle Tennessee State University Andrea M. Georgiou Paul R. Carlson FOCUS Lab Project

CULMINATING EXPERIENCES FOCUS concept first proposed in 2009 by Dr. Paul A. Craig FOCUS concept first proposed in 2009 by Dr. Paul A. Craig Received NASA grant in 2010 to study team decision-making Received NASA grant in 2010 to study team decision-making Project would address best practices for communication, disruption management, teamwork, and group situational-awareness Project would address best practices for communication, disruption management, teamwork, and group situational-awareness Conceptualization and construction began in 2010 with a small research team comprised of faculty and graduate students Conceptualization and construction began in 2010 with a small research team comprised of faculty and graduate students Students began with tabletop discussions and now has evolved to include: Students began with tabletop discussions and now has evolved to include: Complex and real-world scenarios Complex and real-world scenarios Immediate feedback and ramifications to decisions Immediate feedback and ramifications to decisions Real-time performance analysis Real-time performance analysis In-depth after action reviews In-depth after action reviews FOCUS concept first proposed in 2009 by Dr. Paul A. Craig FOCUS concept first proposed in 2009 by Dr. Paul A. Craig Received NASA grant in 2010 to study team decision-making Received NASA grant in 2010 to study team decision-making Project would address best practices for communication, disruption management, teamwork, and group situational-awareness Project would address best practices for communication, disruption management, teamwork, and group situational-awareness Conceptualization and construction began in 2010 with a small research team comprised of faculty and graduate students Conceptualization and construction began in 2010 with a small research team comprised of faculty and graduate students Students began with tabletop discussions and now has evolved to include: Students began with tabletop discussions and now has evolved to include: Complex and real-world scenarios Complex and real-world scenarios Immediate feedback and ramifications to decisions Immediate feedback and ramifications to decisions Real-time performance analysis Real-time performance analysis In-depth after action reviews In-depth after action reviews P ROJECT B ACKGROUND

CULMINATING EXPERIENCES Aerospace Core TechnologyTechnology Flight Dispatch Pro Pilot Maintenance Management AdminAdmin Seminar + Simulation MTSU AEROSPACE program comprised of 5 concentrations: MTSU AEROSPACE program comprised of 5 concentrations: PROFESSIONAL PILOT PROFESSIONAL PILOT MAINTENANCE MANAGEMENT MAINTENANCE MANAGEMENT AEROSPACE ADMINISTRATION AEROSPACE ADMINISTRATION FLIGHT DISPATCH FLIGHT DISPATCH TECHNOLOGY TECHNOLOGY MTSU AEROSPACE program comprised of 5 concentrations: MTSU AEROSPACE program comprised of 5 concentrations: PROFESSIONAL PILOT PROFESSIONAL PILOT MAINTENANCE MANAGEMENT MAINTENANCE MANAGEMENT AEROSPACE ADMINISTRATION AEROSPACE ADMINISTRATION FLIGHT DISPATCH FLIGHT DISPATCH TECHNOLOGY TECHNOLOGY All concentrations take aerospace core classes All concentrations take aerospace core classes After specializing in their chosen concentrations, students return to take a capstone course After specializing in their chosen concentrations, students return to take a capstone course Prior to FOCUS lab, capstone course only addressed industry topics, résumé, and interview preparation Prior to FOCUS lab, capstone course only addressed industry topics, résumé, and interview preparation After specializing in their chosen concentrations, students return to take a capstone course After specializing in their chosen concentrations, students return to take a capstone course Prior to FOCUS lab, capstone course only addressed industry topics, résumé, and interview preparation Prior to FOCUS lab, capstone course only addressed industry topics, résumé, and interview preparation P ROJECT C ONCEPT

CULMINATING EXPERIENCES Surveys of industry experts revealed that newly-minted aviation professionals required 5-10 years to fully understand their role and impact within an organization. Surveys of industry experts revealed that newly-minted aviation professionals required 5-10 years to fully understand their role and impact within an organization. Research revealed that new aviation professionals lacked: Research revealed that new aviation professionals lacked: operational understanding of others’ roles in the organization operational understanding of others’ roles in the organization basic understanding of how their decisions and job performance impacts others basic understanding of how their decisions and job performance impacts others the ability to communicate and coordinate effectively with others to meet organizational objectives the ability to communicate and coordinate effectively with others to meet organizational objectives the use of foresight, planning, and time management in making decisions and performing tasks the use of foresight, planning, and time management in making decisions and performing tasks Research revealed that new aviation professionals lacked: Research revealed that new aviation professionals lacked: operational understanding of others’ roles in the organization operational understanding of others’ roles in the organization basic understanding of how their decisions and job performance impacts others basic understanding of how their decisions and job performance impacts others the ability to communicate and coordinate effectively with others to meet organizational objectives the ability to communicate and coordinate effectively with others to meet organizational objectives the use of foresight, planning, and time management in making decisions and performing tasks the use of foresight, planning, and time management in making decisions and performing tasks “On average, it usually takes our employees 10 years to fully understand the big picture; how the decisions they make impact others in the company.”

CULMINATING EXPERIENCES The FOCUS project is specifically designed to address teamwork deficiencies of new employees in the aviation industry The FOCUS project is specifically designed to address teamwork deficiencies of new employees in the aviation industry Students are placed in a realistic simulated airline flight operations control center Students are placed in a realistic simulated airline flight operations control center Here, students from all aviation disciplines are placed in teams to practice working with others Here, students from all aviation disciplines are placed in teams to practice working with others The FOCUS project is specifically designed to address teamwork deficiencies of new employees in the aviation industry The FOCUS project is specifically designed to address teamwork deficiencies of new employees in the aviation industry Students are placed in a realistic simulated airline flight operations control center Students are placed in a realistic simulated airline flight operations control center Here, students from all aviation disciplines are placed in teams to practice working with others Here, students from all aviation disciplines are placed in teams to practice working with others P ROJECT C ONCEPT

CULMINATING EXPERIENCES Teams work to meet organizational goals like safety, on-time performance, customer satisfaction, and disruption management Teams work to meet organizational goals like safety, on-time performance, customer satisfaction, and disruption management Teams are given various scenarios (or triggers) that require the students to work together to resolve issues quickly and effectively Teams are given various scenarios (or triggers) that require the students to work together to resolve issues quickly and effectively Teams are monitored and evaluated by graduate student and faculty observers Teams are monitored and evaluated by graduate student and faculty observers Teams work to meet organizational goals like safety, on-time performance, customer satisfaction, and disruption management Teams work to meet organizational goals like safety, on-time performance, customer satisfaction, and disruption management Teams are given various scenarios (or triggers) that require the students to work together to resolve issues quickly and effectively Teams are given various scenarios (or triggers) that require the students to work together to resolve issues quickly and effectively Teams are monitored and evaluated by graduate student and faculty observers Teams are monitored and evaluated by graduate student and faculty observers P ROJECT C ONCEPT

FOCUS light perations enter nified imulation CULMINATING EXPERIENCES FOCUS is a high-fidelity replication of a part 121 regional airline operations control center FOCUS is a high-fidelity replication of a part 121 regional airline operations control center Students participate in a 3 hour simulated work shift working for a virtual airline we call “Universal E-Lines” Students participate in a 3 hour simulated work shift working for a virtual airline we call “Universal E-Lines” Universal E-Lines operates 30 CRJ-200 aircraft on 16 flight routes throughout the southeastern United States. Universal E-Lines operates 30 CRJ-200 aircraft on 16 flight routes throughout the southeastern United States. Utilizing a hub and spoke system, Nashville and Jacksonville act as the airline’s hubs Utilizing a hub and spoke system, Nashville and Jacksonville act as the airline’s hubs Students manage up to 80 flight events in a 3 hour simulation session Students manage up to 80 flight events in a 3 hour simulation session FOCUS is a high-fidelity replication of a part 121 regional airline operations control center FOCUS is a high-fidelity replication of a part 121 regional airline operations control center Students participate in a 3 hour simulated work shift working for a virtual airline we call “Universal E-Lines” Students participate in a 3 hour simulated work shift working for a virtual airline we call “Universal E-Lines” Universal E-Lines operates 30 CRJ-200 aircraft on 16 flight routes throughout the southeastern United States. Universal E-Lines operates 30 CRJ-200 aircraft on 16 flight routes throughout the southeastern United States. Utilizing a hub and spoke system, Nashville and Jacksonville act as the airline’s hubs Utilizing a hub and spoke system, Nashville and Jacksonville act as the airline’s hubs Students manage up to 80 flight events in a 3 hour simulation session Students manage up to 80 flight events in a 3 hour simulation session WHAT IS FOCUS?

CULMINATING EXPERIENCES P ROJECT C ONCEPT

CULMINATING EXPERIENCES

P HOTOS

CULMINATING EXPERIENCES There are 10 distinct positions in the FOCUS Lab. They are: Flight Ops. Coordinator Flight Ops. Coordinator Weather & Forecasting Weather & Forecasting Crew Scheduling Crew Scheduling Fuel/Cargo Management* Fuel/Cargo Management* Flight Tracking & Schedule Management* Flight Tracking & Schedule Management* Maintenance Planning Maintenance Planning Maintenance Control Maintenance Control Ramp Tower Ramp Tower Pseudo Pilot Pseudo Pilot CRJ Simulator Crew CRJ Simulator Crew WX & Forecasting Crew Scheduling MX Control MX Planning & Scheduling P ROJECT C ONCEPT

CULMINATING EXPERIENCES P HOTOS

CULMINATING EXPERIENCES The MTSU and its industry partners pioneered the ability to electronically link to a CRJ-200 simulator located at the flight school. Pilots in this simulator can communicate with anyone in the operations center. The CRJ simulator even appears on flight tracking monitors and is displayed in the ramp tower as a photo realistic aircraft moving about the airport environment. P HOTOS

CULMINATING EXPERIENCES P HOTOS

CULMINATING EXPERIENCES P HOTOS

CULMINATING EXPERIENCES S CENARIO D ESIGN & C ONTROL Scenario Design & Control: Graduate Students Graduate Students Monitoring team performance and status of the operation Monitoring team performance and status of the operation Provide real-world feedback and consequences Provide real-world feedback and consequences Issue disruptive ‘triggers’ Issue disruptive ‘triggers’ Library Library Taxonomy Taxonomy Disruptive potential scale Disruptive potential scale Monitor response to triggers Monitor response to triggers Evaluate effectiveness Evaluate effectiveness

CULMINATING EXPERIENCES A FTER -A CTION R EVIEW The Aerospace Department collaborates with the Psychology Department on the FOCUS project. Studies include: Occupational stereotypes & attitudes Occupational stereotypes & attitudes Group dynamics Group dynamics Individual & team cognition Individual & team cognition After- Action Review: Engages students to discuss events of simulation Engages students to discuss events of simulation Feedback from team members and observers Feedback from team members and observers Discussion of unfavorable outcomes Discussion of unfavorable outcomes How to prevent in the future? How to prevent in the future? Anticipating problems Anticipating problems Communication & time management Communication & time management Reinforces positive behaviors and solutions Reinforces positive behaviors and solutions Teams establish goals and new strategies Teams establish goals and new strategies

CULMINATING EXPERIENCES A DVANTAGES Active learning experience Active learning experience Students provide performance feedback to peers Students provide performance feedback to peers Has the potential to reveal holes in curriculum Has the potential to reveal holes in curriculum Encourages faculty to incorporate a greater ‘real-world’ perspective in the classroom Encourages faculty to incorporate a greater ‘real-world’ perspective in the classroom Allows students to use knowledge gained from their coursework to creatively solve problems Allows students to use knowledge gained from their coursework to creatively solve problems High motivation to succeed and learn from mistakes High motivation to succeed and learn from mistakes Students learn to consider multiple solutions to a problem, weighing likelihood, impact, and other downstream implications Students learn to consider multiple solutions to a problem, weighing likelihood, impact, and other downstream implications Gives students a sense of time sensitivity while problem solving Gives students a sense of time sensitivity while problem solving Internship & job opportunities Internship & job opportunities Students enjoy it! Students enjoy it!

CULMINATING EXPERIENCES QUESTIONS t h a n k y o u Dr. Paul A. Craig Ms. Andrea Georgiou Mr. Paul R. Carlson with Dr. Glenn Littlepage Dr. Mike Hein Dr. Richard Moffett Mr. Gerald Hill

ASSESSMENTS AND CULMINATING EXPERIENCES  Thank You UND & MTSU  Culminating Experience  Assessment  Guidance Workshop July 2013 Kansas City