Turning ACUI/EBI Assessments into Positive Change Darlena Jones, Ph.D. Director of Education and Program Development

Who is EBI? Founded in 1994 by Joseph Pica, Ed.D and Glenn Detrick (retired in 2002) Over 1500 Colleges and Universities (U.S.A. and 15 other countries like Australia, Mexico, Spain, and Egypt) have participated in EBI’s studies Surveyed over 12 million peopleNearly 100 Assessments in 10 areas of higher education Over 500 custom assessments for schools like MIT, Ohio State, UCLA, University of Georgia, and University of Florida Produced over 17,000 customized reports Creation of WESS (Web-Enabled Survey System) that collects, analyzes, reports, disseminates, and manages assessment data

MAP- Works Student Affairs College of Nursing College of Engineering College of Business College of Education Family of EBI Assessments

ACUI-I/EBI Assessments Began in 2000, 211 institutions have participated Satisfaction measures like bookstore, dining service Learning Outcomes measures like leadership ACUI/EBI College Union/Student Center Assessment Began in 2006, 40 institutions have participated Satisfaction measures like advisor and training Learning Outcomes measures like personal and practical competencies ACUI/EBI Student Leadership Assessment Began in 2007, 44 institutions have participated Satisfaction measures like promotion of activities and programming board Learning Outcomes measures like impact and importance of student activities ACUI/EBI Student Activities Assessment

Assessment Highlights Tested and reliable instruments Mapped to CAS, NASPA/ACPA Learning Reconsidered Add institution specific questions Survey Items Online and paper reports Benchmarks against Select 6, Carnegie Class, and National Norm Longitudinal trends analysis Executive summaries, interactive reporting, Action Planning (NEW!) Reporting Annual administration (either fall or spring term) Online reporting available immediately Full comparative reporting available in summer Process

Data-Driven Decision-Making

Making Decisions Which way do you normally choose… … and which way is defendable?

Research shows that if decisions are based on assessment information, the probability is increased they will attain the desired outcomes. Likely to challenge decisions –Campus (faculty, staff, upper administration, school newspaper) –Board of regents –Department of education (state and federal) –Constituents (Parents, students, alumni) –Media (TV, newspapers, bloggers, etc.) Defending Decisions

Assessment Cycle Conduct Assessment Analyze Information Create Action Plans Write Summary Reports Implement Actions Late fall term or early spring ASAP! Mid spring Late spring Immediately!

Basic Analysis Techniques

Scale We use a 7-point scale to measure perception. This scale can be viewed as a “percentage of satisfaction”. 0% Satisfaction 100% Satisfaction 50% Satisfaction Very Dissatisfied or No Agreement Neutral GOAL VALUE: Mean of 5.50 or 75% Satisfaction Level Very Satisfied or High Agreement Group 1 and 2 together = “Very dissatisfied” Group 6 and 7 together = “Very satisfied”

Types of Survey Questions Categorical or Demographic On average, how often do you visit the College Union building? How often do you participate in activities sponsored by the College Union? Likert-Scaled Questions How satisfied are you with the extent to which the College Union: Promotes programs of interest to students To what extent do you agree or disagree with the following statements. The College Union is a student-oriented facility Text or Open- Ended Please tell us the main reason why you choose not to visit the College Union What services or activities would motivate you to visit the College Union? Institution Specific Questions Additional questions added by your institution. Can be any question type here in addition to numeric

A statistical grouping of questions that when combined provide a richer understanding of a larger concept (also called “constructs”) Factors This factor, College Enhances Life and Leadership, is a combination of 8 scaled questions

The number and percentage of respondents that answered each response value Frequency Distribution of Scaled Questions 21% were very dissatisfied 44% were very satisfied

The mean or average is obtained by adding all of the responses to a particular question or factor and dividing by the number of respondents to that question or factor.  Values of Question or Factor Number of Respondents to that Question or Factor Mean of a Scaled Question or Factor Mean = 4.97

Statistical Difference Identify areas of lower performance compared to others Identify areas of higher performance compared to others

Recommendations for Improvement Focus on these high-impact factors No/Low Impact Factors High Impact Factors!

Data Analysis: Sub-Populations Life and Leadership “crossed” by D010. Frequency of Union Activity Participation Very large differences in student perceptions Aggregate Population Results for Life and Leadership (top predictor)…

Assessment in Action

Things to think about… What is the political temperature of your campus/state? How does your organizational structure impact your ability to create change? Whose permission do you need to implement changes? How much will changes cost? Do you have the budget/staff? Can you reallocate resources? What is the time frame of change?

Creating an Action Plan Step 1 Choose a goal Step 2 Conduct a preliminary analysis Step 3 Choose a performance goal Step 4 Gather more information (if necessary) Step 5 Create manageable tasks and organize in to a plan Step 6 Monitor execution of plan and reevaluate

Examples… College Union / Student Center Assessment –Improve Overall Program Effectiveness –Increase number of students using Union services –Increase satisfaction with bookstore/dining Student Organization Leader Assessment –Improve Overall Program Evaluation –Improve satisfaction with the leadership skills learned –Improve satisfaction with collaboration Student Activity Assessment –Improve Overall Evaluation of College / University –Increase number of students involved –Increase learning associated with activities

Identify the top predictors of Overall Program Effectiveness Identify any student sub-populations with special issues on top predictor

Now that you understand your program’s performance, what improvement in performance are you working towards? Improvement in Mean: What improvement in mean score do you expect for next year’s assessment results if the actions you put in place are successful? Other Measures: What other ways will you measure success? In other words, describe what the environment will look like if you’re successful in improving this factor.

–Conflicting information? Search out more information –Consider focus groups (students, student staff, other staff etc.) to help understand the effect you’re seeing –Examples of information Student records Focus groups Internal data collected Other assessments –Things to think about: Do you have access to that information? If you don’t, how can you get access to it? What do you think that data will help prove?

How do you eat an elephant?

Describe how you intend to monitor each action You have to be prepared to change the action if you, as you monitor progress, that the results aren’t what you want

Working in groups, please create one action to improve “College Union Enhances Life and Leadership”. –To what extent do College Union activities: Expand understanding of others whose backgrounds differ from yours Expand understanding of your role as a citizen of the college community Enhance ability to interact socially Expose you to new and different ideas Provide leadership training Provide opportunities for you to assume a leadership role Enhance your appreciation of the arts Enhance your appreciation of the value of volunteerism Group Discussion –What actions did you create? –How will you monitor those actions? –How will you know when your actions are successful? Let’s Get Started!

Closing the Loop

Executive Summary Recipe Project Description Brief description of project (½ - ¾ page in length) Major Findings Good news and bad news (½ - ¾ page in length) Recommendations for Improvement Short description of the analysis with a table showing the factors in priority order highlighting the top three predictors (1 page) Other Key Findings Any political “hot topics” (i.e. GPA changes, student characteristics) that are of general interest (½ - ¾ page in length) Appendix In-depth information on survey design, participating schools (as long as necessary)

Final Results President’s Office Your Office Admissions/ Recruiting / Marketing Campus Police Campus Programming Residence Life Administrators / Faculty / Students VPs of Academic and Student Affairs Share Results!

Questions / Discussion Darlena Jones, Ph.D.