The Academic Program Review Bridging Standards 7 and 14 Middle States Annual Conference December 10, 2010

Presenters Mr. H. Leon Hill Director of Institutional Research Dr. Joan E. Brookshire Associate Vice President of Academic Affairs

Overview Framework to Address the APRs Structure/Challenges/Approach Examples of Metrics Current Action Plan Integration of End User Technology Next Steps Benefits of Our Approach Questions

Assessment Cycle-2005 Plan to meet Meet to plan Report out on planning Plan to meet Meet to plan

What we had to build on Strong focus on programs. State mandated 5-year academic program review in need of revision. Institutional Effectiveness Model (IEM) with performance indicators benchmarked through State and National data bases.

Mission Strategic Initiative: Access & Success Institutional Effectiveness

IEM Needed a way to assess how the College was performing on key metrics in relation to prior. years/semesters and compared to other institutions. Historical/Trend data Benchmark data –Pennsylvania & National Peers

Institutional Effectiveness Model

Where we started Restructured the Academic Program Review process Incorporated the use of technology

Goal of the restructuring Measure student performance as evidence by results of assessment of student learning outcomes. Measure program performance as evidenced by comparison of program performance to overall college performance on specific key indicator (current and aspirational).

Challenges Usual issues with assessment in general. Faculty had little knowledge of the Colleges performance indicators. Organizational separation of assessment of institutional and student learning outcomes.

Approach Began by building it backwards from the IEM by mapping out specific core indicators to program data, making additions where needed.

Examples of Metrics Used for APR

TARGETSCautionAcceptableAspirational Graduation Rate <19%19%-23%>23%

TARGETSCautionAcceptableAspirational Transfer Rate<29%29%-32%>32%

Definitions of Success & Retention Success=Grades of (A,B,C & P)/(A, B, C, D, P, D, F, & W) Retention=Grades of (W)/(A, B, C, D, P, D, F, & W)

Added a curricular analysis How well program goals support the colleges mission. How well individual course outcomes reinforce program outcomes. How well instruction aligns with the learning outcomes.

Specific assessment results. Changes made based on the assessment findings. Evidence of closing the loop Changes made to the assessment plan.

Action Plan Outcomes expected as a result of appropriate actions steps. Timelines and persons responsible for each action step. Resources needed with specific budget requests. Evaluation plan with expected benefits.

Bottom Line Is there sufficient evidence that the program learning outcomes are being met? Is there sufficient evidence that the program is aligned with the college on specific key indicators?

The Framework Planning and Budgeting (Standard 2) APR Action Plan APR Annual Report Annual Academic Planning Assessment Results Curriculum Committee Presidents Office Curriculum BOT & BOT

Addition of Technology Worked in concert with Information Technology to integrate iStrategy with ERP (Datatel). The implementation of this permitted end users to obtain the data needed for program assessment, without the middle man (IR and/or IT).

Next Steps in the Evolution of of College and Program Outcomes

Example of APR Report Card

Examples of Course Success

Success in ACC /FA2004/FA2005/FA 2006/F A 2007/F A 2008/F A 2009/F A % Success 61.4%57.1%55.4%55.3%51.4%44.2%48.3% # Success % Non Success 38.6%42.9%44.6%44.7%48.6%55.8%51.7% # Non Success

Success in ACC /FA2004/FA2005/FA2006/FA2007/FA2008/FA2009/FA % Female Success 63.3%57.5%58.8%57.7%57.3%51.8%58.7% Female Success % Male Success 59.8%56.8%53.2%53.6%47.2%39.1%42.1% Male Success

Success in Math /FA2004/FA2005/FA2006/FA2007/FA2008/FA2009/FA % Success53.6%46.3%47.3%45.7%44.8%43.3%47.4% Success % Non Success46.4%53.7%52.7%54.3%55.2%56.7%52.6% Non Success

Success in Math /FA2004/FA2005/FA2006/FA2007/FA2008/FA2009/FA % African American Success 42.6%37.7%38.5%25.8%26.9%29.9%34.7% African American Success % Caucasian Success58.2%51.8%50.3%52.7%53.5%48.4%52.0% Caucasian Success

Benefits Build a bridge between Standards 7 and 14. Better data. By putting data in the hands of faculty, have them actively engaged with using data in decisions/planning. IR time better used.

Annual planning cycle developed. Built a culture of assessment in several of the academic divisions. Curricular changes that align with graduation initiative. Curricular and program improvement. Created a college-wide model for improvement of student learning.

Evolution of the Dashboard Creation of a Student Success Dashboard Metrics: Course level success and retention (Developmental and College-Level) Persistence (fall to spring and fall to fall) Progression of various cohorts of students College level success in Math or English after Developmental Math or English Graduation Transfer

Graphic Representation for the SSD

Final Thoughts Its not perfect, but it works for us. Do the research on which tools are appropriate for your college Assessment of the core curriculum Launching of assessment software It all starts with asking the right question PRR 2010

Questions

Presenters Mr. H. Leon Hill Director of Institutional Research Dr. Joan E. Brookshire Associate Vice President of Academic Affairs