Program Evaluation and Outcomes Assessment: Channels to Quality Assurance February 9, 2008 ACCA 4 th National Conference Savannah, GA PRESENTER Laura A. Dean, Ph.D., NCC University of Georgia ACCA Representative to CAS

Determining Quality in Higher Education Rankings and Ratings Accreditation Licensure – Certification Follow-Up Studies Self-Studies Program Review/Evaluation Outcomes Assessment

Assessment/Evaluation/Research Assessment is a snapshot of current data – what is happening? Evaluation involves a judgment – is that good or does it need improvement? Assessment & evaluation are focused on improvement of local situations Research is designed to be generalizable to other settings

Program Evaluation Is the program or service functioning effectively to achieve its mission? What evidence is available to support the determination? How is evidence used to make program decisions? CAS as one tool for self-assessment of programs and services

Outcomes Assessment What is the effect of our work on students? How are they different as a result of interacting with our programs or services? How do we demonstrate their learning? Need to develop measurable learning outcomes Sources include CAS standards, Learning Reconsidered, locally identified goals & outcomes

Comprehensive assessment plan A comprehensive plan includes assessment of programs and services and of student outcomes Outcomes are one element of a program evaluation

Why use CAS for program evaluation? Nationally developed and endorsed standards Approved by consortium of 35+ professional associations in higher ed. Don’t reinvent the wheel – the work has been done for you Materials support the process

CAS Standards Format 13 Component Parts 1.Mission 2.Program 3.Leadership 4.Organization & management 5.Human resources 6.Financial resources 7.Facilities, technology, & equipment 8.Legal responsibilities 9.Equity and access 10.Campus and community relations 11.Diversity 12.Ethics 13.Assessment & evaluation

Counseling Services Standard & Guideline Example STANDARD BOLD TYPE; AUXILIARY VERBS MUST & SHALL Counseling services must be a) intentional, b) coherent, c) based on theories and knowledge of counseling, learning, and human development, d) reflective of developmental and demographic profiles of the student population, and e) responsive to the needs of individuals within a higher education setting. GUIDELINE LIGHT- FACED TYPE; AUXILIARY VERBS SHOULD & MAY Counseling services should provide consultation, supervision, and in-service professional development for faculty members, administrators, staff and student staff members, and paraprofessionals.

The CAS SAG: A Self-Assessment Guide Translates CAS standards into an effective workbook format Promotes program self-assessment and development Informs on program strengths and weaknesses Supports professional staff development Leads to enhanced student learning and development

Conducting the Self-Assessment Process: Team Actions Decide whether to include guidelines or other measures that go beyond the standards Gather and analyze relevant quantitative and qualitative data Individuals rate each and every criterion measure Obtain additional documentary evidence if required to make an informed team decision Complete the assessment, ratings, & action plan worksheets

Criterion Measure Rating Scale ND1234NR Not Done Not met Minimally Well Fully Not Rated MetMetMet Using this scale, consider each criterion statement and decide… The extent to which each criterion measure has been met by the Counseling Service

Assessment Criteria Example Part 6: FINANCIAL RESOURCES Counseling Services (CS) must have adequate funding to accomplish its mission and goals. Funding priorities must be determined within the context of the stated mission, goals, objectives, and comprehensive analysis of the needs and capabilities of students and the availability of internal or external resources. CS must demonstrate fiscal responsibility and cost effectiveness consistent with institutional protocols. ND NR Not DoneNot Met Minimally Met Well MetFully MetNot Rated PART 6. FINANCIAL RESOURCES (Criterion Measures) Rating ScaleNOTES 6.1 The program has adequate funding to accomplish its mission and goals. ND NR 6.2 Funding priorities are determined within the context of program mission, student needs, and available fiscal resources.ND NR 6.3The program demonstrates fiscal responsibility and cost effectiveness consistent with institutional protocols. ND NR Part 6: Financial Resources Overview Questions A. What is the funding strategy for the program? B. What evidence exists to confirm fiscal responsibility and cost-effectiveness?

Examples of Evaluative Evidence STUDENT RECRUITMENT MATERIALS: Brochures & other program information Participation policies & procedures PROGRAM DOCUMENTS: Mission statements; program purpose & philosophy statements Catalogs & related materials Staff & student manuals; policies & procedures statements ADMINISTRATIVE DOCUMENTS: Organization charts; student & staff profiles Financial resource statements & budgets Annual reports STAFF ACTIVITY REPORTS: Curriculum vitae & resumes; professional activity Service to other programs, departments, or community STUDENT ACTIVITY REPORTS: Portfolios, developmental transcripts, resumes Reports of student service RESEARCH & EVALUATION DATA: Needs assessments & self-studies Program evaluation; graduate & follow-up studies

Qualitative Data Focus group information Written summaries of responses to open-ended questions in interviews and on evaluations Client satisfaction surveys, self-reports, and written comments, both solicited and unsolicited

Quantitative Data Summaries of objective responses on a questionnaire or program evaluation Statistics about use by students and various population subgroups Needs assessments, follow-up studies, and self-study reports Institutional research reports and fact books

Complete Each Work Form Work Form A: Assessment, Ratings, and Significant Items Work Form B: Follow-Up Actions Work Form C: Summary Action Plan

Action Plan Components Program Action Plans Identify responsible parties to complete the action steps & timeline Summary action plan creates an overview of the work to be done PLAN For the Future

Closing the loop Purpose & scope of study is defined Self-study team is trained Self-study is conducted Evaluative evidence is collected Discrepancies are identified Appropriate actions are determined Special actions for program enhancement are recommended Action plan is developed and communicated

Outcomes Assessment May be various kinds of outcomes Learning outcomes are the primary emphasis of the field currently, driven by pressures for accountability If goal is reaching more students, then tracking participation may be useful Important to map desired outcomes to institutional mission, division mission & goals, department mission & goals Map the linkages

Developing learning outcomes Define the broad goal Define the objective Define the criteria Identify strategies Assess the results

Learning Outcomes Students will develop intrapersonal competence Students will demonstrate enhanced self-esteem Students will recognize and demonstrate appropriately assertive behavior Workshop on assertive behavior will be offered twice this semester; attendance goal is 30 students total; attainment of the objective is defined as 80% of participants correctly identifying examples of assertiveness and being able to role-play successfully continued

Assessing Outcomes Assessment: Formative - recording % of participants who correctly identify examples and role- play successfully, as observed by workshop presenter Summative - responses to an evaluation form that includes self-report of learning Follow-up - observations, self-reports, or later survey

Program: Learning and Development Outcome Domains Intellectual growth Effective communication Enhanced self-esteem Realistic self-appraisal Clarified values Career choices Leadership development Healthy behavior Meaningful interpersonal relationships Independence Collaboration Social responsibility Satisfying and productive lifestyles Appreciating diversity Spiritual awareness Personal and educational goals Identify areas of focus for the program or service

Learning Reconsidered Outcome Domains Cognitive complexity Knowledge acquisition, integration, & application Humanitarianism & interpersonal competence Civic engagement Intrapersonal competence Practical competence

Frameworks for Assessing Learning and Development Outcomes “FALDOs” – companion to CAS standards book Chapters for each CAS learning outcome domain with a theoretical description of the domain (e. g., leadership development, social responsibility, career choices), assessment examples, list of possible instruments, and additional resources Published as book & CD set

Where to start?? Start small Start doable Start narrow Start targeted Start. One program, one workshop, one desired outcome, one measurement tool

Mapping Outcomes In a more developed approach, a department or division may adopt a set of desired outcomes Each specific department, program, etc. plays a role in contributing to the overall achievement A chart can be developed to map where each of the desired outcomes will be targeted and assessed The goal is not that everyone individually does everything; the goal is that everyone collectively addresses everything Another approach is to develop a sequence of outcomes across several years, so that all are addressed in that span of time

Additional CAS Resources Standards book – revised in 2006 Prologue & Context with “How to” information CAS Statement of Shared Ethical Principles Characteristics of Individual Excellence Glossary & FAQs FALDOs – book & CD formats as package SAGs – CD or individually