The Mission Hierarchy: A Model for Integrating and Assessing Institutional Outcomes Presented To: SLO Symposium By: Joachin Arias, Rhea Estoya and Edward Pai Los Angeles Harbor College 9 February 2018

Introductions: Joachin Arias, Ph.D. Rhea Estoya Edward Pai, Ph.D. Chemistry Faculty member SLO Coordinator Assessment Committee Chair HAPS co-designer Rhea Estoya Research Analyst, Office of Institutional Effectiveness Programmer/Operations Manager for HAPS Edward Pai, Ph.D. Dean, Institutional Effectiveness

Creating a Model: The “Systems” Approach Von Bertalanffy defined a system as "elements in standing relationship“ Harbor’s Assessment-based Planning System (HAPS) makes the following assumptions: Alignment of Mission, Institutional Outcomes and Assessment activities Identifying the “Elements” and “Relationships” in HAPS Establishes the “Mission Hierarchy” Roll-up assessment of all outcomes Assessment-based Planning Model Assessments result in Improvement Actions that make up a Unit’s Plan Closing the Assessment Loop Macro vs. Micro Assessment processes to improve quality Macro (Strategic) processes assure Institutional Outcomes are achieved Micro (Operational) processes assure assessment results are delivered

“Elements” of the System   Student Success / College Mission Pyramid of Student Success (with thanks to John Wooden) Institutional Outcomes: ISLOs: Communication, Cognition, Information Competency, Social Responsibility SEMP Goals: Access & Preparation, Teaching & Learning, Organizational Effectiveness, Resources & Collaboration Program-level Outcomes: PLOs: Courses in Program or Pathways that lead to a degree or certificate GELOs: Courses in a discipline that lead to a GE degree/requirement Service Areas: Student Support Administration SEMP Objectives / Strategies Course/Service-level Outcomes: SLOs: All courses that have assigned grades and services with a student learning component SEMP Measures: Student Achievement Outcomes metrics and standards (IEPI, SSSP, Strong Workforce), Federal, State or Grant requirements, SAOs: Services Area Outcomes for Student Support and Administrative Services

The “Mission Hierarchy" Outcomes Assessment-based Planning → Improvement Action Student Achievement and Course / Service-level Learning Outcomes Program / Service Area Outcomes Institution-level Outcomes Institution Mission SEMP/Goals Objectives Measures Improvement Actions ISLOs PLOs SLO/SAOs Macro (Strategic) Micro (Operational)

Assessment-based Planning All plans are based on an assessment of outcomes Two types of outcomes: Student Achievement (Strategic Goals) State scorecard Assessed in Program Review Student Learning (ISLOs) Workforce skills/soft skills Assessed in the SLO Process

Assessment-based Planning SEMP Learning Outcomes

SLO Assessment Outcomes

Conceptual/Data Roll-up Model Assessing Mission Achievement Screen shot of Institutional Outcomes Report

Assessing Mission Achievement: The Mission Hierarchy Connects the Classroom to the College Mission The Mission Hierarchy establishes the relationships (Parent-Child) between: The College Mission Institutional Outcomes Program Outcomes Classroom-based Student Achievement and Student Learning (SLOs) The Hierarchical Structure provides “roll-up capabilities” Children roll-up into Parents Direct assessments from the classroom are aggregated to assess program and institution-level outcomes

Macro/Strategic/College Level Assessment: ISLO Achievement

Program Level Assessment: PLO Achievement

HAPS Demonstration HAPS DEMO (Recording an Assessment) Institutional Outcomes Summary ISLO Assessment Summary Discipline Level Assessment Summary

Mission/SEMP Revision Framework Particpation Coverage Outcome achievement CPC meeting 1/22 and Retreat 1/29 STEP 1: Review & Evaluation of 2014-17 SEMP District SWOT District SMP (1/22) Other external initiatives/requirements CPC Retreat 1/29 STEP 2: Assess External Environment Existing SEMP + External Scan/District SMP = New/revised SEMP Proposed 1/22, discussed 1/29, approved 2/12 STEP 3: Create new/revised SEMP

CPC Responsibilities CPC: College Planning Council Annually: Apex governance committee in charge of strategic planning Annually: Oversee and maintain SEMP Oversee planning processes Integrate cluster and unit priorities to develop annual priorities for the college based on achievement of the SEMP Use Annual priorities to guide resource allocation processes SEMP Cycle (2014-17): Evaluate achievement of the college mission Renew SEMP based on college progress (annual reviews, external priorities) Identify goals, objectives and measures to add or remove

Mission/SEMP Achievement Evaluation Framework Participation Are all Operational Units of the college participating? Are Operational Units participating appropriately? Coverage Are all Goals, Objectives and Measures addressed? Is the amount of coverage appropriate? Outcome Achievement Are the activities to address SEMP goals being completed?

Mission/SEMP Achievement Measurement Methodology HAPS: Harbor’s Assessment-based Planning System Implemented using SharePoint forms and list services HAPS Records All Outcome Assessment Activities and Associated Improvement Actions (assessment-based planning) Outcomes (student achievement and student learning) Assessment Process results provide the data All results are self-reported Roll-up Model Allows Evaluation of Institutional and Program Level Outcomes

SEMP Participation Results   100% Complete Advance to next year Discontinued Abandoned Incomplete Total Cluster Count % Academic Affairs 154 41.8% 135 36.7% 38 10.3% 11 3.0% 30 8.2% 368 100.0% Administrative Services 20 76.9% 2 7.7% 0.0% 26 Student Services 119 82.6% 3 2.1% 4 2.8% 15 10.4% 144 Grand Total 293 54.5% 140 26.0% 42 7.8% 16 47 8.7% 538

SEMP Evaluation Survey Results: Participation

SEMP Goal Coverage Results   Academic Affairs Admin Services Student Services Total SEMP Goal Count Percent Goal 1: Access and Preparation for Success 17 21% 0% 63 79% 80 100% Goal 2: Teaching and Learning for Success 294 83% 61 17% 355 Goal 3: Organizational Effectiveness 34 64% 10 19% 9 53 Goal 4: Resources and Collaboration 1 4% 14 61% 8 35% 23 Other Requirement 20 80% 2 8% 3 12% 25 (blank) Grand Total 368 68% 26 5% 144 27% 538

SEMP Evaluation Survey Results: Coverage – Focus - Priorities

SEMP Outcome Achievement   100% Complete Advance to next year Discontinued Abandoned Incomplete Total SEMP Goal Count Percent Goal 1: Access and Preparation for Success 66 82.5% 5 6.3% 3 3.8% 1 1.3% 80 100.0% Goal 2: Teaching and Learning for Success 162 45.6% 118 33.2% 26 7.3% 11 3.1% 38 10.7% 355 Goal 3: Organizational Effectiveness 29 54.7% 8 15.1% 20.8% 5.7% 2 53 Goal 4: Resources and Collaboration 21 91.3% 0.0% 4.3% 23 Other Requirement 15 60.0% 7 28.0% 4.0% 25 (blank) Grand Total 293 54.5% 140 26.0% 42 7.8% 16 3.0% 47 8.7% 538

SEMP Evaluation Survey Results: Outcomes Achievement

Evaluating ISLOs Roll-up methodology (like SEMP) Three dimensions: Participation Coverage Outcome Achievement Group Exercise: Evaluate ISLOs Handouts Complete Survey (we will use these results for validating our process)

ISLO Participation Results Academic Affairs Admin Services Student Services Total Count Total Percent INSTITUTIONAL OUTCOME Count % Goal 1: Access and Preparation for Success 1 14% 0% 6 86% 7 100% Goal 2: Teaching and Learning for Success 23 70% 2 6% 8 24% 33 Goal 3: Organizational Effectiveness 33% 3 Goal 4: Resources and Collaboration 11% 78% 9 ISLO 1 - Communication 501 ISLO 2 - Cognition 981 4 985 ISLO 3 - Information and Technological Competency 265 98% 5 2% 270 ISLO 4 - Social Responsibility and Ethics 143 Other Requirement 57% 29% (blank) 30 94% 32 Grand Total 1947 14 1% 29 1990

ISLO Participation Evaluation Did all CLUSTERS participate in ISLO/SAO assessment? Did CLUSTERS meet their assessment responsibilities?

ISLO Coverage, Part I: Results Count Percent ISLO 1 - Communication 501 26% ISLO 2 - Cognition 985 52% ISLO 3 - Information and Technological Competency 270 14% ISLO 4 - Social Responsibility and Ethics 143 8% Grand Total 1899 100%

ISLO Coverage, Part 1: Evaluation Were all ISLOs assessed? Was the distribution of assessment efforts appropriate?

ISLO Coverage, Part II: Academic Divisions ISLO 1 - Communication ISLO 2 - Cognition ISLO 3 - Info and Tech Competency ISLO 4 - Social Resp. and Ethics Grand Total Academic Affairs Office 33 Business 59 115 78 32 284 Communications 81 28 1 110 Counseling 4 2 9 Health Sciences/Nursing 3 23 60 Honors Transfer Program Humanities and Fine Arts 221 386 72 73 752 Kinesiology, Health and Wellness 47 64 13 133 Math/Technology 117 120 Science/FCS 11 141 14 169 Social Science 94 25 19 216 501 981 265 143 1890

ISLO Coverage, Part 2: Evaluation Was the distribution of assessment efforts by ACADEMIC DIVISION appr0priate? Did each DIVISION assure the delivery of all college ISLOs? Were you able to identify priority areas based on this report?

ISLO Outcome Achievement Yes No N/A Blank Grand Total ISLO 1 - Communication 76% 11% 2% 100% ISLO 2 - Cognition 70% 13% 3% 14% ISLO 3 - Information and Technological Competency 77% 10% 1% 12% ISLO 4 - Social Responsibility and Ethics 73% 6% 21%

ISLO Outcome Achievement: Did the college achieve its ISLOs? Did improvement occur as a result of the SLO assessment process?

Please Complete the Evaluation Survey The college will use these results to validate our evaluation rubric We will present these results at the RP Group Conference in April THANK YOU! QUESTIONS & DISCUSSION