1. Dr. Ron Atwell Ms. Kathleen Connelly
Poster Session Assessment Toolkit: Measuring What Matters in Student Development and Enrollment Services
Dr. Ron Atwell
Ms. Kathleen Connelly
International Assessment and
Retention Conference
June 9, 2007
© 2006 Krist, Atwell, & Poisel

2. Program Assessment Is
formative: designed to collect information that can be used for improvement
ongoing OR
summative: takes a picture of where you are today
contributes to resource allocation
infused into regular operations
clear and understandable
comprehensive
measures your primary functions or activities
cost effective
time
money
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

3. Objectives of Department Performance Review
Conduct comprehensive review of the department
Develop historical perspective of department
Identify primary areas for improvement:
services offered
processes used
Discuss programs with division directors, vice presidents, other administrators
Complement accrediting efforts
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

4. Department Performance Review: Components of Self- Study
Examine elements related to the key programs, services, or activities of the department
Evaluation of:
Centrality
Quality
Demand
Distinctiveness & competitive advantage
Cost
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

5. Department Performance Review: Components of Self- Study
Individual departments within a division:
develop recommendations:
strengths, weaknesses, opportunities
reach resource decisions:
eliminate, reduce, maintain, or enhance
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

6. Phases of a Department Performance Review
Department self-study and department level information completed by department
Review and recommendations completed by the unit head
Optional: review of the program completed by an external consultant
Review of unit head (summary and department review) by appropriate administrators
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

7. Centrality of Department
Department Mission
Strategic planning goals of the department
Alignment of department mission with the university mission and vision
Alignment of department mission with the division mission and vision
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

8. Functional Areas of a Department
Review the most important functions (activities, services, or programs) that you said your department provides
Department Performance Reviews are an in-depth examination of the primary functions of your department
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

9. Statement of Purpose and Description
“The purpose of (the functional area: program, activity, or service) is to (describe the reason for the functional area) by providing (identify what this functional area does) to (your stakeholders).”
*NOTE: the order of the pieces of the purpose and description may vary from the above structure
For the functional area that you have selected, develop a concise statement of the purpose with a description of that area.
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

10. Investigate Functional Areas
Cost
Demand
Quality: Outcomes and Measures
Relationship to Strategic Initiatives
Distinctiveness
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

11. Cost Expenditures for each function: program, activity or service
Budget comparisons (past, present, future)
Number of staff: headcount and FTE
Special facilities, equipment, etc. required for each activity specified for the department
Specialized delivery requirements
partnerships with other areas
off-site travel
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

12. Demand for Each Function
Target market: who are you trying to reach?
Number of constituents you serve
Ability to meet demand
Comments about future anticipated demand
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

13. Quality of Department department outcomes
student learning and development outcomes
students’ and other constituents’ satisfaction
impact on strategic initiatives:
e.g. retention; student learning and development
adequacy and quality of space and facilities
service optimization
efficiency
demand
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

14. Defining Functional/Operational and Student Learning Outcomes
Program Outcome
A specific, measurable statement that describes desired performance
Function operational outcome: a type of outcome that addresses operational or procedural tasks, such as efficiency or satisfaction
Student learning outcome: a type of outcome that describes the intended learning outcomes that students should meet as a result of a unit’s program(s) or service(s)
more precise, specific, and measurable than a goal
more than one outcome can relate to each goal
one outcome can support more than one goal
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

15. Writing Outcomes: Think SMART
Specific
Clear and definite terms describing expected outcomes. For SLOs this includes abilities, knowledge, values, attitudes, and performance.
Measurable
It is feasible to get the data, data are accurate and reliable;
it can be assessed in more than one way
Aggressive but Attainable
Has potential to move the program forward
Results-oriented
Describe what standards are expected
Time-bound
Describe where you would like to be within a specific
time period
International Assessment and Retention Conference, June 9, 2007, 11:30 AM
From: Drucker, 1954

16. MATURE: Measuring Outcomes
Matches
directly related to the outcome it is trying to measure
Appropriate methods
uses appropriate direct and indirect measures
Targets
indicates desired level of performance
Useful
measures help identify what to improve
Reliable
based on tested, known methods
Effective and Efficient
characterizes the outcome concisely
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

17. Function Assessment Measures
Student learning can be assessed using a large variety of methods, ranging from exams, to performance appraisals, to observations, to surveys.
Our programs are directed to choose multiple methods that are most appropriate for their discipline and to use best assessment practices.
direct measure: actual evidence of something; for student learning outcomes, it means direct examination or observation of student knowledge, skills, attitudes or behaviors to provide evidence of learning outcomes
indirect measure: perceived extent or value of operations or learning experiences
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

18. Assessment Measures for Operational Outcomes
Student learning can be assessed using a large variety of methods, ranging from exams, to performance appraisals, to observations, to surveys.
Our programs are directed to choose multiple methods that are most appropriate for their discipline and to use best assessment practices.
direct measures
staff time
cost
materials
equipment
other resources
cost per unit output
reliability
accuracy
courtesy
competence
reduction in errors
audit, external evaluator
indirect measures
written surveys and questionnaires:
stakeholder perception
students
administration and staff
faculty
interviews
focus groups
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

19. Assessment Measures for Student Learning Outcomes
Student learning can be assessed using a large variety of methods, ranging from exams, to performance appraisals, to observations, to surveys.
Our programs are directed to choose multiple methods that are most appropriate for their discipline and to use best assessment practices.
direct measures
student records
locally developed exams
embedded questions
external judge
oral exams
portfolios (with rubrics)
behavioral observations
simulations
project evaluations
performance appraisals
minute papers
indirect measures
written surveys and questionnaires:
student perception
alumni perception
employer perception of program
exit and other interviews
focus groups
student records
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

20. Distinctiveness Regional & national reputation
Unique features of department
Results from benchmarking with other colleges
Strategic niche
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

21. Impact on Strategic Initiatives
Level of impact on appropriate strategic initiatives for the division:
First year retention
Student learning and development
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

22. Assessment Results: Outcomes
Results of assessment and or evaluation for each program, activity or service:
Effectiveness
Efficiency
Quality
Student Learning
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

23. Recommendations by Function
influence demand
improve competitiveness
achieve productivity gains
achieve efficiencies
reduce cost
improve quality
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

24. Action Plan and Overall Recommendations
Expand, Reduce, Maintain, Eliminate
Outsource, Reorganize, Re-engineer, Study further
Overall comments and recommendations
Planned and implemented changes
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

25. Review by Unit Heads Review Self-Study Review Recommendations
Strategic Planning:
Reorganization
Resource allocation:
funds
personnel
space
International Assessment and Retention Conference, June 9, 2007, 11:30 AM

26. Continue the Conversation
Dr. Ron Atwell, Director
Assessment and Planning
Student Development and Enrollment Services
Ms. Kathleen Connolly, Advisor
First Year Advising and Exploration
Dr. Mark Allen Poisel, Associate Vice President
Academic Development and Retention
International Assessment and Retention Conference, June 9, 2007, 11:30 AM