Dr. Ron Atwell Ms. Kathleen Connelly Poster Session Assessment Toolkit: Measuring What Matters in Student Development and Enrollment Services Dr. Ron Atwell Ms. Kathleen Connelly International Assessment and Retention Conference June 9, 2007 © 2006 Krist, Atwell, & Poisel

Program Assessment Is formative: designed to collect information that can be used for improvement ongoing OR summative: takes a picture of where you are today contributes to resource allocation infused into regular operations clear and understandable comprehensive measures your primary functions or activities cost effective time money International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Objectives of Department Performance Review Conduct comprehensive review of the department Develop historical perspective of department Identify primary areas for improvement: services offered processes used Discuss programs with division directors, vice presidents, other administrators Complement accrediting efforts International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Department Performance Review: Components of Self- Study Examine elements related to the key programs, services, or activities of the department Evaluation of: Centrality Quality Demand Distinctiveness & competitive advantage Cost International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Department Performance Review: Components of Self- Study Individual departments within a division: develop recommendations: strengths, weaknesses, opportunities reach resource decisions: eliminate, reduce, maintain, or enhance International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Phases of a Department Performance Review Department self-study and department level information completed by department Review and recommendations completed by the unit head Optional: review of the program completed by an external consultant Review of unit head (summary and department review) by appropriate administrators International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Centrality of Department Department Mission Strategic planning goals of the department Alignment of department mission with the university mission and vision Alignment of department mission with the division mission and vision International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Functional Areas of a Department Review the most important functions (activities, services, or programs) that you said your department provides Department Performance Reviews are an in-depth examination of the primary functions of your department International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Statement of Purpose and Description “The purpose of (the functional area: program, activity, or service) is to (describe the reason for the functional area) by providing (identify what this functional area does) to (your stakeholders).” *NOTE: the order of the pieces of the purpose and description may vary from the above structure For the functional area that you have selected, develop a concise statement of the purpose with a description of that area. International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Investigate Functional Areas Cost Demand Quality: Outcomes and Measures Relationship to Strategic Initiatives Distinctiveness International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Cost Expenditures for each function: program, activity or service Budget comparisons (past, present, future) Number of staff: headcount and FTE Special facilities, equipment, etc. required for each activity specified for the department Specialized delivery requirements partnerships with other areas off-site travel International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Demand for Each Function Target market: who are you trying to reach? Number of constituents you serve Ability to meet demand Comments about future anticipated demand International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Quality of Department department outcomes student learning and development outcomes students’ and other constituents’ satisfaction impact on strategic initiatives: e.g. retention; student learning and development adequacy and quality of space and facilities service optimization efficiency demand International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Defining Functional/Operational and Student Learning Outcomes Program Outcome A specific, measurable statement that describes desired performance Function operational outcome: a type of outcome that addresses operational or procedural tasks, such as efficiency or satisfaction Student learning outcome: a type of outcome that describes the intended learning outcomes that students should meet as a result of a unit’s program(s) or service(s) more precise, specific, and measurable than a goal more than one outcome can relate to each goal one outcome can support more than one goal International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Writing Outcomes: Think SMART Specific Clear and definite terms describing expected outcomes. For SLOs this includes abilities, knowledge, values, attitudes, and performance. Measurable It is feasible to get the data, data are accurate and reliable; it can be assessed in more than one way Aggressive but Attainable Has potential to move the program forward Results-oriented Describe what standards are expected Time-bound Describe where you would like to be within a specific time period International Assessment and Retention Conference, June 9, 2007, 11:30 AM From: Drucker, 1954

MATURE: Measuring Outcomes Matches directly related to the outcome it is trying to measure Appropriate methods uses appropriate direct and indirect measures Targets indicates desired level of performance Useful measures help identify what to improve Reliable based on tested, known methods Effective and Efficient characterizes the outcome concisely International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Function Assessment Measures Student learning can be assessed using a large variety of methods, ranging from exams, to performance appraisals, to observations, to surveys. Our programs are directed to choose multiple methods that are most appropriate for their discipline and to use best assessment practices. direct measure: actual evidence of something; for student learning outcomes, it means direct examination or observation of student knowledge, skills, attitudes or behaviors to provide evidence of learning outcomes indirect measure: perceived extent or value of operations or learning experiences International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Assessment Measures for Operational Outcomes Student learning can be assessed using a large variety of methods, ranging from exams, to performance appraisals, to observations, to surveys. Our programs are directed to choose multiple methods that are most appropriate for their discipline and to use best assessment practices. direct measures staff time cost materials equipment other resources cost per unit output reliability accuracy courtesy competence reduction in errors audit, external evaluator indirect measures written surveys and questionnaires: stakeholder perception students administration and staff faculty interviews focus groups International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Assessment Measures for Student Learning Outcomes Student learning can be assessed using a large variety of methods, ranging from exams, to performance appraisals, to observations, to surveys. Our programs are directed to choose multiple methods that are most appropriate for their discipline and to use best assessment practices. direct measures student records locally developed exams embedded questions external judge oral exams portfolios (with rubrics) behavioral observations simulations project evaluations performance appraisals minute papers indirect measures written surveys and questionnaires: student perception alumni perception employer perception of program exit and other interviews focus groups student records International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Distinctiveness Regional & national reputation Unique features of department Results from benchmarking with other colleges Strategic niche International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Impact on Strategic Initiatives Level of impact on appropriate strategic initiatives for the division: First year retention Student learning and development International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Assessment Results: Outcomes Results of assessment and or evaluation for each program, activity or service: Effectiveness Efficiency Quality Student Learning International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Recommendations by Function influence demand improve competitiveness achieve productivity gains achieve efficiencies reduce cost improve quality International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Action Plan and Overall Recommendations Expand, Reduce, Maintain, Eliminate Outsource, Reorganize, Re-engineer, Study further Overall comments and recommendations Planned and implemented changes International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Review by Unit Heads Review Self-Study Review Recommendations Strategic Planning: Reorganization Resource allocation: funds personnel space International Assessment and Retention Conference, June 9, 2007, 11:30 AM

Continue the Conversation Dr. Ron Atwell, Director Assessment and Planning Student Development and Enrollment Services ratwell@mail.ucf.edu Ms. Kathleen Connolly, Advisor First Year Advising and Exploration mconnoll@mail.ucf.edu Dr. Mark Allen Poisel, Associate Vice President Academic Development and Retention mpoisel@mail.ucf.edu International Assessment and Retention Conference, June 9, 2007, 11:30 AM