Using Institutional Effectiveness to Build a Culture of Performance Improvement 2007 SACS-COC Annual Meeting Department of Institutional Research and Effectiveness St. Petersburg College P.O. Box St. Petersburg, FL (727) FAX (727)

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting2 Presenters  Carol Weideman, Director of Institutional Research and Effectiveness  With SPC for the last eight years. Previously served as the Program Director for Mathematics and Natural Science at the Clearwater campus. Also teaches an online statistics course.  Jesse Coraggio, Assessment Coordinator for Academic Programs  Joined SPC during the past year. Currently completing his doctorate in Educational Measurement and Research from the University of South Florida. Research interests include instrument development, psychometrics, setting performance standards, and various item response theory applications.

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting3 SPC Background  SPC, established in 1927, is the oldest 2-year college in Florida  First Community College in Florida to offer 4 year degrees (2002)  9 Campuses throughout the county  FTE: 15,402 (LD), 1,185 (UD)  Opening Fall 2007 headcount: 26,165  Annual enrollment: 61,930

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting4 Issues with IE  SACS’s Peer Review Research Project revealed that in the area of Institutional Effectiveness (IE)  62% of institutions in off-site reviews were deemed non-compliant and  27% of institutions in on-site visits were deemed non-compliant  IE was the second most identified area behind faculty qualifications

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting5 SPC Experience with IE  As part of the recent SACS reaccreditation process, SPC spent a considerable amount of time and effort documenting and detailing their IE processes  The result…SPC had “zero” compliance issues in the area of IE during the off-site reviews

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting6 SACS Requirements for IE  Core Requirement 2.5: Integrated Planning & Evaluation Processes  “…ongoing, integrated, and institution-wide research-based planning and evaluation processes…”  Comprehensive Standard 3.3.1: Institutional Effectiveness  “…identifies expected outcomes…; assesses whether it achieves these outcomes; and provides evidence of improvement…  Comprehensive Standard 3.5.1: General Education Core  “… identifies college-level competencies within the general education core and provides evidence that graduates have attained those competencies.”

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting7 SACS Requirements for IE  Additional IE related requirements include:  Comprehensive Standard 3.1.1: Institutional Mission  Federal Requirement 4.1: Evaluating Success  Federal Requirement 4.2: Curriculum

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting8 IE Evidence  SACS Suggested Documentation for CR 2.5  Evidence of linkage of IE to institutional mission  Institutional plans and budgets that demonstrate linkage of assessment findings to planning at all levels  Minutes of appropriate (IE related) unit, committee, task force meetings…  Documentation that relates to IE, such as budget preparation instructions, minutes of budget presentation meetings, annual reports, annual assessment updates, IE reports  Samples of specific actions taken to improve the IE process and/or results from that process

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting9 Performance Improvement  From Compliance to Performance Improvement

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting10 SPC’s Definition of IE  “At SPC, Institutional Effectiveness is the integrated, systematic, explicit, and documented processes of measuring performance against the SPC mission for purposes of continuous improvement of academic programs, administrative services, and educational services offered by the college.” “Closing the Loop”

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting11 IE Phases  IE Processes are yearly processes that consist of three distinct phases:  Evaluation/assessment phase  Strategic/budget planning phase  Implementation/operation phase

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting12 Evaluation/Assessment Phase  Reflective look back at the prior year’s assessment and evaluation results.  educational outcomes  institutional surveys  departmental-specific assessments  Individual assessments results are reviewed by the respective oversight group which in turn aggregates the information and presents recommendations to the President’s Cabinet.  These recommendations are evaluated and are routinely summarized in the form of the Institutional Objectives (IOs).

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting13 Strategic/Budget Planning Phase  Transition from assessment/evaluation results to planning for the upcoming operational year  Includes assessment-driven unit planning  Unit managers state their intent for the upcoming year in measurable objectives  Unit managers ensure alignment with the institution’s mission and goals, Strategic Directions and Institutional Objectives (SD-IOs)

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting14 Implementation/Operation Phase  Execution of the developed unit plans  Unit managers are required to provide periodic reports of their progress to ensure the fidelity and stability of these unit plans  Periodic reports include a “mid-year status report” and an “end-of-year final results report.” The end-of-year report includes assessment results in addition to the unit’s use of these results in improving their unit.

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting15 Individual IE Processes  Individual IE processes are designed to result in continuous improvement and comprise the following: a.Mission statement process b.Evaluation and assessment processes for academic programs, student services, and administrative services c.Strategic Directions and Institutional Objectives (SD- IOs) process d.Unit / budget planning process e.Collegewide operating budget development process f.Budget approval, implementation, and annual operating results

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting16 Mission Statement  IE process begins and ends with the College Mission statement  College’s Mission is formally reviewed every five years by a committee representing faculty, administrators, students, community leaders, and members of the Board of Trustees (BOT)  Mission is also reviewed annually by President’s Cabinet before the new strategic planning cycle begins a. Mission Statement Process

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting17 Assessing the Mission 5 SPC Mission – Assessments  Closing the Loop a. Mission Statement Process

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting18 Evaluation and Assessment  Evaluation and assessment processes at SPC are centered on mission-driven outcomes in the following program and service areas:  (i) 26 Organizational units comprising 40 Academic programs (lower division);  (ii) 8 Colleges and Schools comprising 20 Majors (upper division), and  (iii) key administrative and educational support services.  Analysis of outcome results is on-going and captured in various assessment reports.  SPC evaluates all student services to ensure students are provided the best support possible to ensure student success. b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting19 Assessment Areas  Academic Outcomes  General Education  BAS/BS  AAS/AS  Key Administrative and Educational Support Services  Viability  Importance  Satisfaction b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting20 Assessment Focus  Continually evaluate assessment instruments and current processes  Integrate assessment with educational process  Focus on fewer, yet meaningful improvements (action items)  Use multiple assessment methods  Involve faculty in process  Integrate timing of assessments (3-year cycle) Assessments ReviewsSurveys GenGen EdEd b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting21 Direct and Indirect Measures  Direct Measures  Internally Developed General Education  Externally Developed General Education (e.g., MAPP, iSkills)  AAS/AS Academic Program Assessment Report (APAR)  BAS/BS Program Assessment  Comprehensive Academic Program Review (CAPR)  Indirect measures  Entering Student Survey  Enrolled Student Survey  Graduating Student Survey  Recent Alumni Survey  Employer Survey  Community College Survey of Student Engagement (CCSSE) b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting22 Three-year Assessment Cycle Academic Program Viability Reviews (APVRs) conducted yearly APARs CAPRs b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, 2007 Key Assessments Academic Program Viability Review (APVR) Academic Program Assessment Review (3) (APAR) Comprehensive Academic Program Review (3) (CAPR) Oversight Group Report (Multiple Programs) 1)Program Description (1) 2)Number of Graduates – by year 3)Actual Enrollments – by last six semesters 4)SSH Productivity – by last six semesters 5)Program Profitability – RPI Index by year 1)Introduction w/ use of past results (1,2) 2)Major Learning Outcomes – program specific (1, 2, 3) 3)Assessment Methodology (3) 4)Criteria for success (3) 5)Summary of assessment findings 6)Discussion & analysis 7)Action plan & time table 8)Budgetary & planning implications 9)Follow-up report 1)Key APAR findings 2)Key APVR findings 3)Environmental scan of stakeholders – program specific (1) 4)Enrollment / Graduation trends 5)Employment / wages trends (1) 6)Discussion & analysis 7)Action plan & time table 8)Annual follow-up report 9)Cabinet Review & Approval 1)Outcomes Asmt. Sources (1) 2)Previous Years Plans & Results (1,2) 3)Current Year Asmt. Results (1,2) 4)Key Areas Requiring Improvements (not meeting criteria / expectations) 5)Recommended Action Plans 6)Cabinet Review & Acknowledgement Notes:1)SACS Compliance Requirement 2)Florida DOE Recommendations 3)BOT Rule 6Hx Requirement b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting24 Academic Program Assessment  Academic Program Assessment Reports (APARs)  Prepared for the Gen Ed, A.S., and A.A.S. programs according to the three-year cycle  Include a summary of assessment findings and a discussion and analysis of those assessment findings (including use of results, action plan items, and budgetary implications)  Process was initiated college-wide in 1999  Program data is gathered and analyzed on a continuous basis b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting25 Academic Program Viability Review  Academic Program Viability Review (APVR)  Published yearly  Included measures:  Program Graduates  Course Enrollment  Student Semester Hour (SSH) productivity  Relative Profitability Index (RPI-T)  Economic Trend Data b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting26 Program Review Process  Comprehensive Academic Program Review (CAPR) is a summative evaluation that includes multiple measures such as:  program-specific performance  profitability measures  academic outcomes, and  stakeholder perceptions b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting27 Program Review Objectives  CAPR was designed to meet three objectives:  To provide a comprehensive report that summaries all elements of the program’s viability and productivity from a 360-degree perspective,  To provide comprehensive and relevant program specific information to key College stakeholders such as the President Cabinet members in order to make critical decisions regarding the continued sustainability of a program, and  To provide program leadership a vehicle to support and document actionable change for the purposes of performance improvement. b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting28 Ed Outcomes Online  To provide a medium for completing the draft educational assessment reports as well as establishing a repository for program specific information, SPC developed an Educational Assessment Web site (  College administration and instructional staff are provided access to “completed” assessment reports including the APAR (Program & Gen Ed) and the CAPR  Online access further encourages the use of assessment data as well as highlighting “best practices” across the college b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting29 College Access b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting30 Program Director Access b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting31 Student Survey Process  Data collection  Increased student awareness of survey process and accessibility  Minimized use of class time to complete surveys  Use multiple points of student contact  Online survey submission and scanning of paper forms  Quality of survey forms  Professional “look and feel” of survey format  Items designed to gather pertinent data  Included online branching models to reduce survey length b. Evaluation and Assessment Processes

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting32 Strategic Directions  College’s SD-IOs are comprised of longer-term directional statements and shorter-term, annually measurable objectives.  Strategic Directions are:  Updated on a three-year cycle and are based on achieving the College’s mission.  Developed to make improvements in learning outcomes, quality, productivity, and other environmental-scan considerations, as well as inputs from IE Team, President’s Cabinet, faculty and student representatives, the BOT, and external/community stakeholders. c. Strategic Directions and Institutional Objectives Process

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting33 Institutional Objectives  Institutional Objectives are  Reviewed and updated annually, with key on-going initiatives updated based on progress made during the prior year and carried forward, where appropriate.  Designed for the purpose of achieving the corresponding longer-term strategic direction. c. Strategic Directions and Institutional Objectives Process

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting34 SD-IO Alignment Key Components SPC Mission Evaluation/ Assessments Strategic Plan SD & IOs (87 Total; A few selected below) Operating / Results (See full details: Year-End Report) Accessible Education  BS, BAS, AA, AS Degrees  Technical Certificates  Applied Tech. Diplomas  Distance Education  Developmental Education  Continuing Education 1. Education Oversight Group (EOG) Reports 2. Academic Program Assessment Reports (APARs) 3. Academic Program Viability Reviews (APVRs) 4. Comprehensive Academic Program Reviews (CAPRs) A.2: Use educational outcome data to improve General Education Outcomes A.11: Incorporate service learning into Honors Program courses I.4: Increase baccalaureate degree opportunities w/ targeted new programs A.2: Gen. Ed. outcomes for all A.S. programs implemented A.11: Service learning now required for Honors students I.4: Int’l Business, O&P, Paralegal Studies added Mission – Assessments - Closing the Loop Examples c. Strategic Directions and Institutional Objectives Process

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting35 Unit / Budget Planning  SPC’s unit/budget planning is centered on:  more than 150 Unit / Budget Managers,  development of departmental unit plans and budget requests, and  use of the national award-winning Strategic Planning Online (SPOL) system.  Unit / Budget Managers are urged and encouraged to access, review, and make use of all available assessment outcome data (i.e., College’s IE website, oversight group reports, etc.) d. Unit / Budget Planning Process

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting36 Unit / Budget Planning  Budget guidelines are prepared by the Collegewide Budget Committee (CBC), which provides  Information about the College’s expectations for State appropriations,  Collegewide budgetary outlook, and  Salary package and other key strategic expectations.  Budget guidelines also provide details on how capital expenditure requests are to be entered into the SPOL system  SPOL system requires the unit / budget managers to select one of the SD-IOs for each of the unit plan objectives entered d. Unit / Budget Planning Process

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting37 Unit / Budget Planning  Supporting tasks, expected outcomes, and related budget funding are also captured in the SPOL system  After all unit plans and budget requests are entered, they undergo management review in which refinements and adjustments may be made based on funding availability and overall College priorities  Approved Unit plans (by College President’s direct reports) are aggregated and submitted for further review and evaluation by the CBC d. Unit / Budget Planning Process

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting38 Collegewide Budget Committee  Each year, the President confirms the membership of a CBC (SVPs, VPs, provosts, and other key functional leaders)  Certain members of the committee are rotated on an annual basis for cross-training purposes.  The charge of the CBC includes the following:  Develop budget guidelines,  Review of all budget requests approved by department heads, vice presidents, and senior vice presidents,  Ensure that the SD-IOs are funded according to their priorities and funding availability, and  Develop the College’s Operating Budget. e. Collegewide Operating Budget Development Process

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting39 Budget Approval  Operating Budget is approved by the President’s Cabinet and recommended to the BOT  Once approved by BOT, the budget is communicated and uploaded into the College’s central financial system  College financials and operating results are monitored on a monthly basis and reported quarterly to the BOT f. Budget Approval, Implementation, and Annual Operating Results

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting40 Operating Performance and Results  Operating performance and results are centered on:  Executing SD-IOs  Supporting unit plans and quality improvement initiatives  Adhering to the operating budget  Operating performance and results are accomplished through the following key steps:  Implementing budgets and action plans  Monitoring achievements and related circumstances  Evaluating mid-year unit plan reviews  Review end-of-year unit plan reports f. Budget Approval, Implementation, and Annual Operating Results

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting41 IE Next Steps  Ultimate goal is to provide stakeholders ‘timely’, ‘relevant’, ‘accurate’, and ‘interpretable’ data through:  Formatted (dashboard) style reports, and  On-demand customizable reporting, with  Valid, reliable, and standardized measures.

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting42 Links to Additional IE Materials

Using IE to Build a Culture of Performance Improvement December 10, SACS-COC Annual Meeting43 Questions