Program Review and Division-wide Assessment CAS Standards Program Review and Division-wide Assessment

With thanks to the generosity of Deborah A With thanks to the generosity of Deborah A. Garrett Vice Chancellor for Student Affairs Arkansas State University - Beebe Carole S. Henry, ACUHO-I CAS Liaison Executive Director of Housing & Residence Life Old Dominion University Doug Lange, CAS Secretary Dean of Student Affairs Ai The Art Institute of Tennessee-Nashville

Agenda Overview of CAS and program review Preparation for a review Self Study and Process Developing an Action Plan Assessing Student Learning Outcomes CAS Resources

CAS Overview Founded in 1979 (ACUHO-I charter member) in response to efforts to establish specialized accreditation for student affairs preparation programs CAS is a consortium of prof. association members (100,000+) in higher education who work collaboratively to promulgate standards & guidelines

CAS Overview Focus on quality programs & services for students Promulgate standards and guidelines for practice and preparation Promote assessment in educational practice Promote quality assurance within higher education

WHAT IS PROGRAM REVIEW NOT? it is not an evaluation of an individual staff member an evaluation of an individual student a vehicle for punishment

WHAT IS PROGRAM REVIEW? Formal, collaborative process for systematic review and evaluation Tied closely to mission, goals, and specified outcomes of a given department Provides a snapshot of where a department is now Identifies progress toward department priorities & university strategic planning Assesses strengths and weaknesses Makes recommendations for program improvement

ELEMENTS OF PROGRAM REVIEW A. Collect, analyze data/evidence about a program B. Compare results to some standard for purpose of evaluating services provided: effectiveness efficiency impact quality C. Tied to continuous improvement

CAS Self Assessment Components mission program leadership organization and management human resources financial resources facilities, technology, and equipment legal responsibilities equity and access campus and external relations diversity ethics assessment and evaluation

CAS Self Assessment Guides Purpose and Organization Outcomes relevant to student learning and development that must be identified and assessed: knowledge acquisition cognitive complexity intrapersonal development interpersonal competence humanitarianism and civic engagement’ practical competence Ask if they have participated in student learning outcomes development/assessment in their areas.

Key Questions for CAS Components Mission How does the unit uniquely contribute to and advance the Division’s goals, objectives, & strategic initiatives? Program What are the Unit’s core programs and services? Describe any key collaborative relationships (within and beyond Student Affairs) in which the department is involved. Please explain how these partnerships: What lines of communication exist for students’ input to be included in the mission/purpose of the dept.?  

Key Questions for CAS Components Leadership What leadership practices are used to establish and promote the unit’s values, short and long term directions, and performance expectations? How does unit leadership promote a work culture that encourages staff empowerment, innovation, risk taking to effectively meet needs of students? Philosophy of decision-making? How does planning take place?  

Key Questions continued Organization and Management Explain the unit’s organizational structure, including reporting relationships, distribution of responsibilities/authority, lines of communication, and use of management and/or work teams. What criterion does the unit use to accurately determine the correct number of personnel that are needed to effectively fulfill the mission and meet the needs of those it serves?

Key Questions continued Human Resources What strategies are used to recruit, hire, and retain staff? How are staff performance expectations established, effectively communicated, and understood? How is staff performance evaluated? How is staff being trained and developed? Financial Resources Please provide a financial overview of the unit, What is the funding model for the unit? What resource changes have occurred over the past five years and what are the anticipated changes for the future? How are budget allocation/reallocation decisions made in the unit?

Key Questions continued Facilities, Technology, and Equipment How effectively do current facilities/space and equipment support the work of the unit? How has technology been integrated into the program, service, and operating functions of the unit? How has the unit kept pace with the development of hardware, software, maintenance, and training support? What are the projected technology needs for the future? How effectively do the unit’s current facilities, space, and equipment support its operations?

Key Questions continued Legal Responsibilities How is staff trained to ensure that they are knowledgeable about the laws and regulations, and professional ethics that apply to their respective job responsibilities? What processes are in place for managing risks Equity and Access How is non-discriminatory, fair, and equitable treatment of staff and constituents ensured? Campus and External Relationships What collaborative relationships exist with campus colleagues? What short and long-term goals have been established to strengthen these types of relationships?

Key Questions continued Diversity How does the unit advance the University and Division’s diversity agenda specific to: program/service offerings recruiting, hiring, retaining, and promoting staff student learning and development outreach and engagement activities campus and work environments Ethics What is the unit’s policy/strategy for managing student/client and staff member confidentiality issues? How are they informed of these practices?

Key Questions continued Assessment and Evaluation Provide a description of the unit’s assessment and evaluation activities. How are student needs assessed and considered in the development and delivery of programs and services? How is benchmark data used to stay current with peers?

Basic Steps: Self-Assessment Process Establish the process, identify a team and train them. Understand CAS Standards and Guidelines. Determine data sources, compile and review evidence. Gather & analyze pertinent qualitative and quantitative data Judge performance. Complete the assessment process.

Step B: Understand CAS Standards and Guidelines CAS functional area standards and guidelines – basic statements that should be achievable by a program STANDARD (Bold type; Verbs MUST or SHALL) Sample: HLRP staff members must provide a variety of educational opportunities that promote academic success and the achievement of learning and student development outcomes. GUIDELINE (Regular type; Verbs SHOULD or MAY) Sample: HLRP should provide an environment that assists residents to remain in good academic standing, earn higher GPA’s, and be retained. Review a few that are in the packet

Step C: Compile and Review Documentary Evidence for Your Area Recruitment and marketing materials Participation policies and procedures Results reports; participant evaluation summaries Program documents brochures, catalogs institutional administrative documents organization charts financial statements Research, assessment, evaluation data needs assessments, follow-up studies assessment outcomes reports Staff activity reports Student activity reports Link to website if it is complete

Step D: Judging Performance Assessment criterion measures specific to your part of the review ND 1 2 3 4 NR Not Done Not met Minimally Well Fully Met Met Met Using this scale, consider each criterion statement and decide the extent to which each criterion measure has been met by the program or service

Step D: Judging Performance continued Each criterion measure focuses on a particular aspect of the standard, allowing raters to express more detailed and specific judgments In most instances, there are multiple criteria for each standard

Judging Performance Exercise Page 31: Review must statements Diversity Criterion Measures Rate 11.1, 11.2, 11.3, 11.4 independently Review in group.

Formulating an Action Plan Answer overview questions Identify areas of program strength (Work form A) Identify areas of program weakness (Work form A) Describe practices requiring follow up (Work form A) Summarize actions required for the program to meet the standards (Work form B) Summarize program enhancement activities (Work form C) Write program action plan

CAS: Work Form A Assessment, Ratings, and Significant Items INSTRUCTIONS: This work form should be completed following individual ratings of the participants. For each of the 13 Parts, identify (circle) the criterion measure item number(s) in the column labeled for which there is a substantial rating discrepancy (two or more ratings apart). Items not circled should reflect consensus among judges that practice in that area is satisfactory. Items where judgment variance occurs need to be discussed thoroughly by team members. Follow this action by determining which practices (criterion measures) can be designated as “excellent” or “unsatisfactory” and record them in the Step One column. In Step Two, list the items requiring follow-up action including any criterion measure rated as being unsatisfactory by any reviewer. Step One Part Items Excellent Unsatisfactory 1. Mission 1.1a 1.4 1.1b 1.5 1.1c 1.6 1.2 1.3 2. Program 2.1 2.4.3 2.4.8 2.4.13 2.6 2.8d 2.11 2.2 2.4.4 2.4.9 2.4.14 2.7 2.8e 2.12 2.3 2.4.5 2.4.10 2.4.15 2.8a 2.8f 2.13 2.4.1 2.4.6 2.4.11 2.4.16 2.8b 2.9 2.4.2 2.4.7 2.4.12 2.5 2.8c 2.10 When the individual ratings have been reviewed and translated into a collective rating, the team is ready to move to the interpretation phase of the assessment process. WORK FORM A For each of the 14 Parts, identify the criterion measure item number(s) for which there is substantial rating discrepancy. Items not circled should reflect consensus among raters that practice in that area is satisfactory. Items where judgment variance occurs need to be discussed thoroughly by team members. Follow this action by determining which practices can be designated as “excellent” or “unsatisfactory.” Ratings will begin to converge, show themes & trends. This will lead to the focus of the action plan. (Show example of spreadsheets developed by Texas Tech to support individual and team ratings)

CAS: Work Form A continued Step Two: List item number(s) for each Part determined to merit follow-up and describe the practice weaknesses that require attention 1. 2. 3. 4. 5. List the items requiring follow-up action, including any criterion measure rated as unsatisfactory by any reviewer. Complete the Overview Questions at the end of each Part.

CAS Work Form B: Follow Up Actions INSTRUCTIONS: The purpose of this work form is to begin the planning for action to be taken on practices judged to merit follow-up (See Step 1, Work Form A). In Step Three, transfer short descriptions of the practices requiring follow-up and detail these items using the table format provided. Step Three: Describe the current practice that requires change and actions to initiate the change Practice Description Corrective Action Sought Task Assigned To Timeline Due Dates Used to summarize actions required for the Program to Meet Standards. List specific actions identified in the self study that require implementation Prioritize the list by importance, need and achievability of the desired change

CAS Work Form C:Summary Action Plan Step Four: This form concludes the self-assessment process and calls for action to be taken as a consequence of study results. Write a brief action plan statement in the spaces below for each Part in which action is required. Part 1: Mission Part 2: Program Part 3: Leadership Part 4: Human Resources Part 5: Ethics List each specific action identified in the self study that would enhance and strengthen services Establish priorities for the action plan Identify resources that are essential to program enhancement Set dates by which specific actions are to be completed Identify responsible parties to complete the action steps Set tentative start up date for initiating subsequent self study. (show Longwood College Student Union CAS Action Plan – www.longwood.edu/studentunion/assessment.html)

Student Outcomes Assessment What is the effect of our work on students? How are they different as a result of interacting with our programs and services? How do we demonstrate their learning? How do we know? How can we show that the outcome was due to our programs, etc.?

Plan for Assessing Student Learning Outcomes Review/revise mission Identify major services, programs, activities Specify desired student outcomes Identify information needed to show student learning Identify existing and new data needed Develop process for measuring outcomes Identify ways to use findings to change/improve If you haven’t developed a plan for assessing student learning outcomes, let me recommend the following. These are the steps we took in our division of University Life. Each department: We did not attempt to measure all the outcomes we identified all at once. We started with a few key programs/activities and each year we added to the plan. Based on what we learned, we made changes and improvements to programs. We also made changes/improvements to the process and tools used to measure.

CAS Resources www.cas.edu CAS Statement of Shared Ethical Principles CAS Characteristics of Individual Excellence CAS Professional Standards, 2009 edition Prologue & Context with “How to” information Self-Assessment Guides SAG’s- tools to help conduct self-assessment activities Frameworks for Assessing Learning and Development Outcomes (FALDO’s) –contain information for measuring learning outcomes CD of Self Assessment Guides includes all 38 functional area SAG’s and Standards; also includes an e-learning course on conducting self assessment & a power point presentation that can be modified by user.

Program Review Timeline Self Study (January – April) Program review committees students faculty staff Determine adequacy of evidence Collect more data? Conduct focus group?

Timeline (continued) Report Findings Report of findings and recommendations w/emphasis on how unit can improve and contribute more fully.

Timeline (continued) External Review/Site Visit (October) Two or three external reviewers Write preliminary report based on Self Study Campus visit External Review Team Report

Timeline (continued) Writing the Unit Program Review Report (November/December) Prepare final report (generally same individuals as self study team) Share with unit director (ifs/he is not on team) to correct any factual errors Report to VP, Division Program Review Team, Unit Director, Supervisor

Timeline (continued) Develop the Unit Action Plan (November – December) Proposed actions Implementation strategies Person(s) responsible Appropriate timelines and identify resources Include in annual goal-setting

Key Questions continued Summary of Findings What are the primary strengths of the unit and how have these changed over time? What innovative programs/services/practices has the unit instituted that puts it out in front with respect to “best practices” in the field? What aspects of the unit need the most improvement and how has this changed over time? What needs to occur, primarily within existing resources to make these improvements? Please conclude the self-study report with an external review issues statement. This 1-2 page narrative should identify the key issues and questions the department and self-study team would like external reviewers to address during the campus visit.