1. Copyright © 2009 by ABET, Inc. 1 May 27, 2011 EAC Pre-Visit Training 2011-2012 Evaluation Cycle

2. Copyright © 2009 by ABET, Inc. 2 May 27, 2011 University of XYZ ?? Programs to be evaluated ?? member team –?? observers, ?? co-chairs, ?? PEV’s We’ll need to stay focused! (Team Chair should modify this page to reflect the institution being evaluated and the visit team composition)

3. Copyright © 2009 by ABET, Inc. 3 May 27, 2011 Important Team Expectations Evaluators represent the EAC of ABET We are accrediting programs, certifying that they satisfy the criteria Team effort—team decisions Confidentiality Conflict of interest—every visitor should have signed a conflict-of-interest statement Observers –no evaluative statements –exit interview—thank-you only

4. Copyright © 2009 by ABET, Inc. 4 May 27, 2011 Confidentiality Do not discuss conclusions with faculty, students, and others Keep all materials and be prepared to respond to questions during the July 2012 EAC meeting, if needed. At conclusion of accreditation process, all materials are to be destroyed by September 30, 2012, unless otherwise notified by ABET HQ Information specific to the institution is to remain confidential without time limit Institutional data are confidential except with written authorization of institution ABET materials only released by ABET staff

5. Copyright © 2009 by ABET, Inc. 5 May 27, 2011 Communication Maintain open line of communication with the department head Identify deficiencies as soon as possible Discuss all issues with the department head at the debriefing Do not discuss the recommended accreditation action with anyone except team members

6. Copyright © 2009 by ABET, Inc. 6 May 27, 2011 Travel Policy READ the travel policy in the TC Workbook Highlights: –Reimbursement not to exceed Travel Destinations Management Group, Inc. cost of ticket purchased 3 weeks in advance –Meal receipts for team meals must include original itemized receipts Credit card receipts with just totals not acceptable Names of all individuals must be reported on the expense report –All expense reports should be submitted within 10 working days after return –Expense reports submitted more than 60 days after end of travel will not be reimbursed! –Be sure to use the current reimbursement form - mileage amounts are updated periodically

7. Copyright © 2009 by ABET, Inc. 7 May 27, 2011 Travel Policy Highlights (cont.): –Travel to/from your US-based domicile is covered –Taxi or bus – expenses for individual trips exceeding $25 have to have receipts –Car rental submitted for reimbursement has to be booked through Travel Destinations Management Group, Inc. Don’t accept extra insurance Coordinate team car arrangements – CARPOOL! –Note: You are encouraged to use the electronic expense report

8. Copyright © 2009 by ABET, Inc. 8 May 27, 2011 General Visit Outline Day 0 (Sunday) – –Visit campus, review curriculum materials –Tonight share what we’ve learned, plan Day 1 (Monday) – –AM: Dean’s overview, PEV’s in departments –Lunch with institutional officials –PM: Support areas, PEV’s in departments –Evening: Share what we’ve learned, tentative conclusions; draft exit statements Day 2 (Tuesday) – –Tie up loose ends –Draft exit statements by late morning –Debrief department heads before lunch –Exit meeting at XX PM

9. Copyright © 2009 by ABET, Inc. 9 May 27, 2011 Support Program Assignments Day 1 (Monday) BiologyPEV Xyz1:30 PM Chemistry1:30 PM Comp Sci1:30 PM English1:30 PM Math1:30 PM Physics1:30 PM Statistics1:30 PM Gen Ed1:30 PM Library1:30 PM Office of Info Tech1:30 PM (Team Chair should modify this page to reflect the support program assignments to the PEVs)

10. Copyright © 2009 by ABET, Inc. 10 May 27, 2011 Visit details Important forms – done by Tues AM –Transcript & curriculum analyses –Program Audit Form + Explanation of Shortcoming (two hard copies to Team Chair—one will be left with Dean after Summary Meeting with Administration, one soft copy to Team Chair) –Exit Statement—soft & hard copy to Team Chair –Program Evaluator Worksheet—soft copy to Team Chair –Program Evaluator Report—soft copy to Team Chair –Form A2a - Short form—recommended actions— prepared by Team Chair on basis of information from evaluators

11. Copyright © 2009 by ABET, Inc. 11 May 27, 2011 Visit details Important Notes –There are a lot of changes this year: criteria, definitions, self-study, and forms –Review and familiarize yourself with the changes –The PEV Workbook (E67) is an important resource for you to use –Be sure to download and use forms for the 2011-2012 evaluation cycle from the current PEV Workbook (E67)

12. Copyright © 2009 by ABET, Inc. 12 May 27, 2011 Terminology Deficiency – criterion, policy, or procedure is NOT satisfied. Weakness – lacks the strength of compliance with a criterion, policy, or procedure to ensure that the quality of the program will not be compromised. Concern – criterion policy, or procedure is satisfied; however, the potential exists for the situation to change such that the criterion, policy, or procedure may not be satisfied

13. Copyright © 2009 by ABET, Inc. 13 May 27, 2011 Working Definition of Key Terms Deficiency: assigned to any criterion, policy, or procedure that is totally or largely unmet Weakness: criterion, policy, or procedure is met to some meaningful extent, but compliance is insufficient to fully satisfy requirements Concern: criterion, policy, or procedure is fully met, but there is potential for non-compliance in the near future Observation: general commentary possibly, but not necessarily, related to criteria

14. Copyright © 2009 by ABET, Inc. 14 May 27, 2011 Limit Use of Key Terms Use Key Term only in reference to overall evaluation of each criterion The Key Term (defined in the previous slides) is the overall assessment for the criterion as a whole, not the worst finding among the sub- areas on the worksheet. Do not, for example, give a Criterion 4 deficiency to a program that lacks only an evaluation for outcome 3(e)

15. Copyright © 2009 by ABET, Inc. 15 May 27, 2011 Consistency Issues for the Team Depth and completeness of the evaluation from program to program Consistency across all programs in an institution Assignment of appropriate key terms (deficiency, weakness, concern) to describe shortcomings For weaknesses, consistency on interim recommendations—IR vs. IV

16. Copyright © 2009 by ABET, Inc. 16 May 27, 2011 Applicable Criteria Use Criteria for 2011-2012 Accreditation Cycle for a general review. These are the harmonized criteria. There are some differences with the renumbered criteria used in 2010-2011 For an interim visit, the criteria in effect at the time the prior shortcomings were identified should be used, unless it is to the program’s benefit to use the currently applicable ones, and it is the program’s choice. Note: The Program Audit Form for an interim visit reflects the old (not harmonized) criteria

17. Copyright © 2009 by ABET, Inc. 17 May 27, 2011 Harmonized Criteria I. General Criteria for Baccalaureate Level Programs 1)Students 2)Program Educational Objectives 3)Student Outcomes 4)Continuous Improvement 5)Curriculum 6)Faculty 7)Facilities 8)Institutional Support II. General Criteria for Masters Level Programs III. Program Criteria

18. Copyright © 2009 by ABET, Inc. 18 May 27, 2011 Harmonized Criteria: What Has Changed (in a nutshell)? Definition of Program Educational Objectives is modified Assessment and evaluation are still required, but have been moved from criteria 2 and 3 to criterion 4 Criterion 4 explicitly requires data from evaluation of program educational objectives and student outcomes to be used for continuous improvement Important: Self-study and forms have changed! Be sure to use the forms in the current PEV Workbook! Some information that PEVs may have seen in previous years is no longer required

19. Copyright © 2009 by ABET, Inc. 19 May 27, 2011 Criterion 2: Program Educational Objectives (PEOs) Definition & Requirements Definition: PEOs “are broad statements that describe what graduates are expected to attain within a few years of graduation” Requirements: –Published and consistent with mission of the institution, the needs of constituencies and these criteria –There is a documented and effective process, involving program constituencies, for the periodic review and revision of these PEOs

20. Copyright © 2009 by ABET, Inc. 20 May 27, 2011 Program Educational Objectives Issues Do the published PEO’s satisfy the definition? –Are they really broad statements that describe what the graduates are expected to attain within a few years of graduation? Does the program convince the team that the PEO’s are consistent with constituent needs? Is there a documented and effective review and revision process for incorporating the needs of the program’s constituencies?

21. Copyright © 2009 by ABET, Inc. 21 May 27, 2011 Scenario 1: Are These PEOs? Graduates of the program will have: A solid understanding of the basic principles of mathematics, science, and engineering and the technical competency to use the techniques, skills and modern tools for practice in engineering as well as for graduate education. The ability to work in a team and develop problem- solving skills that include oral and written communication skills to effectively communicate technical and professional information. Are they really broad statements that describe what graduates are expected to attain within a few years of graduation?

22. Copyright © 2009 by ABET, Inc. 22 May 27, 2011 Scenario 2: Are these PEOs? Graduates of the culinary engineering program are expected within a few years of graduation to have: 1.Established themselves as practicing professionals or engaged in advanced study in culinary engineering or a related area 2.Demonstrated their ability to work successfully as a member of a professional team and function effectively as responsible professionals. Are they really broad statements that describe what graduates are expected to attain within a few years of graduation

23. Copyright © 2009 by ABET, Inc. 23 May 27, 2011 Program Educational Objectives FAQs What if the PEOs really sound like a collection of student outcomes (instead of objectives)? –If PEOs are not PEOs, there is a Criterion 2 shortcoming. What if PEO’s are ambiguous or reflect student outcomes retooled to apply after graduation? –The team must determine if they meet the intent of the criterion. What if there is no process for determining the needs of the program’s constituents? –If the PEOs does not incorporate constituent needs, there is a Criterion 2 shortcoming.

24. Copyright © 2009 by ABET, Inc. 24 May 27, 2011 Criterion 3: Student Outcomes (SOs) Definition & Requirements Definition: –Student outcomes describe what students are expected to know and be able to do by the time of graduation (skills, knowledge, and behaviors) Requirements: –Student outcomes are (a) though (k) plus any additional ones articulated by the program. –The program must have documented student outcomes that prepare the graduates to attain the PEOs

25. Copyright © 2009 by ABET, Inc. 25 May 27, 2011 Criterion 3 Issues If the program has documented student outcomes, but they do not include all of (a) through (k) ), there is a Criterion 3 shortcoming. If the program does not show that documented student outcomes prepare the graduates to attain the PEOs, there is a Criterion 3 shortcoming.

26. Copyright © 2009 by ABET, Inc. 26 May 27, 2011 What Does Criterion 4 Say? The program must use appropriate, documented processes for assessing and evaluating the extent to which both the PEOs and the SOs are being attained. The results of these evaluations must be systematically utilized as input for continuous improvement of the program. Other available information may also be used to assist in continuous improvement.

27. Copyright © 2009 by ABET, Inc. 27 May 27, 2011 Criterion 4 Issues Are all PEOs and SOs (a) though (k) + being regularly assessed and evaluated? Do the assessment and evaluation demonstrate the extent of attainment of the PEOs and SOs? Are those results systematically utilized as input for the continuous improvement of the program?

28. Copyright © 2009 by ABET, Inc. 28 May 27, 2011 Criterion 4 Consistency Issues Be sure to apply this criterion in a holistic sense The process of assessment and evaluation needs to demonstrate the extent to which outcomes are attained, but there is no language that says … –all outcomes must be attained to the same degree –anything about a numeric scale measuring degree of attainment –the outcomes must be measured

29. Copyright © 2009 by ABET, Inc. 29 May 27, 2011 Criterion 4 FAQs The program has been making changes, but none are related to assessment of Criteria 2 and 3 results. –If there isn’t evidence that these results are being used as input to the improvement process, there is a Criterion 4 shortcoming. A program has rewritten the SOs and is assessing them, but their list does not include all of the Criterion 3 outcomes. –If a Criterion 3 (a-k) is not assessed there is a Criterion 4 shortcoming.

30. Copyright © 2009 by ABET, Inc. 30 May 27, 2011 Criterion 4 FAQ’s What about assessment data? What is adequate data? –Does it all have to be objective/direct? (NO) –Can it be subjective? (Some of it may be; but the evaluation should not be based only on subjective assessment) –Is the observation or conclusion of course instructor adequate? (What was his or her basis for the observation?) –Does evidence for each outcome have to be in the form of work the student has produced? (No, but the team needs to be convinced that outcome attainment has been demonstrated.)

31. Copyright © 2009 by ABET, Inc. 31 May 27, 2011 Deliverables From each PEV, the team chair must have: –A recommended action relative to the program under review that is consistent with the team’s conclusions (remember, it is a TEAM recommendation) –A PAF that accurately reflects the team findings –An Exit Statement that will be: read verbatim in the Exit Meeting used as the basis for construction of the draft statement to the institution

32. Copyright © 2009 by ABET, Inc. 32 May 27, 2011 Be sure to use the correct PAF for the 2011-2012 evaluation cycle Note that PAF for IR is different from the PAF visits (GR, IV) Program Audit Form (PAF)

33. Copyright © 2009 by ABET, Inc. 33 May 27, 2011 Statement Writing

34. Copyright © 2009 by ABET, Inc. 34 May 27, 2011 Exit Statement Format INTRODUCTION— Useful program statistics PROGRAM ISSUES –Strengths (special, unique or particularly conspicuous strengths standing above the norm) 1. 2. –Deficiencies (In order, only for those criteria where deficiencies exist) 1.XXX 2.etc. –Weaknesses (In order, only for those criteria where weaknesses exist) 1.YYY 2.etc. –Concerns (In order, where concerns exist) 1.ZZZ 2.etc. –Observations (do not have to relate to criteria) 1.etc. Refer to PEV Report Form (E74) and Institutional Exit Statement (E173)

35. Copyright © 2009 by ABET, Inc. 35 May 27, 2011 Statement Writing Program Introduction The General Description of the Program normally includes: Information about the program’s administrative location at the institution Information on the launch date of the program The date of its initial graduates Enrollment and faculty size Number of recent graduates (Refer to E74)

36. Copyright © 2009 by ABET, Inc. 36 May 27, 2011 Statement Writing Three-part Construct Each shortcoming should have three components: a) the applicable part of the criterion, using the exact language from the Criteria or APPM where possible. Each shortcoming should start with the phrase “This criterion requires that a program ……..” b) the observed facts that are inconsistent or potentially inconsistent with the stated criterion or APPM element c) the negative impact on the program of the inconsistencies or potential inconsistencies (Refer to E74)

37. Copyright © 2009 by ABET, Inc. 37 May 27, 2011 Make sure the reason for a shortcoming citation is clear and that the evidence used to identify the shortcoming is also clear - focus on WHAT is not in compliance, not HOW to fix it The words used have to make what is inadequate clear, and have to correspond to the level of shortcoming cited Use the three-part statement writing construct as described in E74 A statement that is not using the three-part construct WILL NOT BE ACCEPTED Statement Writing

38. Copyright © 2009 by ABET, Inc. 38 May 27, 2011 Statement Writing Do not have multiple shortcomings for a single criterion! –Don’t say there is a Weakness due to inadequate assessment of outcomes and a Concern that the results of evaluations are not consistently being used as input for continuous improvement –It should be one thing or another – a weakness or a concern relative to the whole criterion!

39. Copyright © 2009 by ABET, Inc. 39 May 27, 2011 Examples of poorly-written C4 citations The language must be clear: –“There is inadequate assessment of outcome 3(j).” How does the program know why its assessment of student knowledge of contemporary issues is inadequate? –“There is incomplete evidence that students attain outcome 3(i).” What is it that is incomplete about the evidence that students have a recognition of the need for and the ability to engage in life-long learning. –“It appears that outcome 3(h) is not fully assessed.” Why did the team find that the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context is not fully assessed? What does “full assessment” mean? The program has to understand what to do The next team needs to understand what was needed

40. Copyright © 2009 by ABET, Inc. 40 May 27, 2011 What Would Make It Clearer? Instead of “There is inadequate assessment of outcome 3(j).” –The only mechanism used for assessment of student knowledge of contemporary issues is through administration of a survey instrument asking students whether they have knowledge of contemporary issues. This mechanism does not provide adequate information for determining the extent to which this program outcome is attained. Instead of “There is incomplete evidence that students attain outcome 3(i).” –Although student grades in the “Introduction to the Profession” course were claimed as documentation and demonstration that students have recognition of the need for and an ability to engage in life-long learning, no clear links between the student grades and this outcome were established. Instead of “It appears that outcome 3(h) is not fully assessed.” –Even though some anecdotal assessment of whether students do have the requisite broad general education was described in the self-study report, evidence shows only preliminary plans for the development of a systematic process for assessment and evaluation of this program outcome.

41. Copyright © 2009 by ABET, Inc. 41 May 27, 2011 If it is a D, the words have to clearly say that the criterion is totally or largely unmet “there is no evidence that” “the program educational objectives are not consistent with the definition found in the Engineering Criteria” “there is no assessment and evaluation process” “not all students are required to engage in a major design experience” The language must support the citation

42. Copyright © 2009 by ABET, Inc. 42 May 27, 2011 The language must support the citation If it is a W, the words have to clearly say that the criterion is met but that strength of compliance is lacking, and how it is lacking needs to be clear –“While there is evidence that outcomes are assessed, it appears that the assessment is ad hoc. Systematic assessment and evaluation of these outcomes would strengthen compliance with this criterion.” [supporting a recommended citation of a Criterion 4 weakness relative to strength of assessment and evaluation process] –“While the program has demonstrated that graduates are able to engage in most of the activities required by the Civil Engineering program criteria, the program has not clearly demonstrated that graduates are able to apply knowledge of physics, chemistry, and at least one additional area of science.” [supporting a recommended citation of a Program Criteria weakness for a civil engineering program]

43. Copyright © 2009 by ABET, Inc. 43 May 27, 2011 The language must support the citation If it is a C, the words have to clearly say that the criterion is met and indicate exactly what leads to the potential for future noncompliance –“It is clear that all of the student outcomes are being assessed, but evaluation of some of the assessment data appears to be inconsistent. Unless evaluation of the extent to which outcomes are attained is carried out on a consistent basis, future compliance with this criterion may be jeopardized.” [supporting a recommended citation of a criterion 4 concern] –“Although all of the transcripts examined provided evidence that students meet all graduation requirements, the processes by which graduation requirements are audited prior to graduation appears to be ad hoc. This suggests that future compliance with this criterion may be jeopardized.” [supporting a recommended citation of a criterion 1 concern]

44. Copyright © 2009 by ABET, Inc. 44 May 27, 2011 The PEV Competency Model and its Application Your performance as a PEV will be evaluated against the PEV Competency Model. You are expected to be: –Technically current –Effective at communicating –Interpersonally skilled –Team-oriented –Professional –Organized Team Chair is to evaluate each PEV; each PEV is to evaluate both the team chair and other PEVs The evaluation forms are found on the ABET web site http://www.abet.org/pev.shtml

45. Copyright © 2009 by ABET, Inc. 45 May 27, 2011 The PEV Competency Rating Rating: 1.Needed Improvement 2. 3.Met Expectation 4. 5.Exceeded Expectation NANot able to evaluate The typical rating for a competent PEV is 3. Ratings of 1 & 2 are used for cases where a PEV needed improvement; explanatory comments required Ratings of 4 & 5 are reserved for rare exceptional performance; explanatory comments desired

46. Copyright © 2009 by ABET, Inc. 46 May 27, 2011 Last Words Remember – this team is “the face of ABET” The 4 big C’s go a long way in promoting a productive evaluation: –Courtesy –Consistency –Clarity –Confidentiality Reminder: After the visit, all contact with the institution must be through the Team Chair. PEVs are not to contact the program/institution directly