Using National Studies of Student Engagement to Support Institutional Change Nathan Marti, CCSSE Todd Chamberlain, NSSE FAIR Conference June 23, 2004

What Really Matters in College: Student Engagement “The research is unequivocal: students who are actively involved in both academic and out-of-class activities gain more from the college experience than those who are not so involved.” Ernest T. Pascarella & Patrick T. Terenzini, How College Affects Students

Lessons from the Research  What matters most to desired outcomes is what students do, not who they are  A key factor for student learning is the quality of effort students devote to educationally purposeful activities

National Survey of Student Engagement (pronounced “nessie”) Community College Survey of Student Engagement (pronounced “sessie”) College student surveys that assess the extent to which students engage in educational practices associated with high levels of learning and development

Penetrating NSSE/CCSSE Findings  Overview of Reports  Start Broad, then Dig Deeper  Identify Significant Items  Focus on What is Important to YOUR Institution’s Priorities

Customized Institutional Report Customized Institutional Report Respondent Characteristics Respondent Characteristics National benchmarks National benchmarks Institutional data Institutional data Means and frequencies Means and frequencies Subpopulations Subpopulations Comparisons by Carnegie/size, national Comparisons by Carnegie/size, national

Respondent Characteristics

Respondent Characteristics: Does This Represent Your Campus? Respondent Characteristics: Does This Represent Your Campus? Response Rate Response Rate Sampling Error Sampling Error Student Characteristics Student Characteristics Gender Gender Race/Ethnicity Race/Ethnicity Class Level Class Level Enrollment Status Enrollment Status Comparisons by Carnegie/size, national and consortium when relevant Comparisons by Carnegie/size, national and consortium when relevant

Benchmarks: High Level Views of the Data NSSE/CCSSE developed five indicators, or benchmarks, to represent the multi-dimensional nature of student engagement at the institutional, sector and national levels Level of Academic Challenge Active & Collaborative Learning Enriching Educational Experiences/ Student Effort Supportive Campus Environment/ Support for Learners Student- Faculty Interaction

Benchmark Report

Mean Summary Report Frequency Distribution Report

Means & Frequency Reports Means & Frequency Reports Look at the Items that Make Up Each Benchmark Look at the Items that Make Up Each Benchmark Which Items Have Significantly Higher/Lower Responses than Comparison Groups? Which Items Have Significantly Higher/Lower Responses than Comparison Groups? Practical Significance: Identify Standardized Effect Sizes greater than.2 Practical Significance: Identify Standardized Effect Sizes greater than.2

Activity: Highlight Key Findings Activity: Highlight Key Findings Review Benchmark and Means Summary Report Review Benchmark and Means Summary Report Small groups by benchmark Small groups by benchmark Examine a benchmark Examine a benchmark Which items differ? Which items differ?

Digging Deeper: Using the Data  Identify important subpopulations  Determine the outcome that matters  Determine the factors that influence the outcome

Principles for Data-Driven Learning-Centered Change

1. Get the ideas right  Focus on a real problem (e.g., persistence, raising expectations, success in major field courses)  Concentrate on effective educational practices

Characteristics of Educationally Effective Colleges Organizational culture valuing  High expectations  Respect for diverse talents  Emphasis on early years of study

Characteristics of Educationally Effective Colleges Curriculum  Coherence in learning  Synthesizing experiences  Integrating education and experience  Ongoing practice of learned skills

Characteristics of Educationally Effective Colleges Instruction  Active learning  Assessment and feedback  Collaboration  Adequate time on task  Out-of-class contact with faculty

2. Get grass roots buy-in  Leaders endorse, but don’t dictate  Structures not (nearly) as important as relationships  Validate pockets of quality  The 10% rule

2. Get grass roots buy-in Examples Ask deans about their concerns Ask deans about their concerns Focus groups Focus groups Get students “engaged” in the improvement effort (Illinois State, Oregon State) Get students “engaged” in the improvement effort (Illinois State, Oregon State) Faculty version of NSSE survey Faculty version of NSSE survey

3. Keep the stakes and volume low  Avoid “winners” & “losers”  Suspend disbelief  Denial management  Go public later than sooner

4. It’s the culture (stupid)  Culture is (almost) always (at least) part of the problem  Focus on “reculturing” and “revisioning”  Use familiar (or at least understandable) language

5. Think and act systemically  Link innovations and change efforts from different parts of the campus (e.g., Greater Expectations, Gen Ed reform, SOTL, NSSE, service learning, diversity)

Presenting NSSE/CCSSE Findings Potential Audiences? Potential Audiences? Consider Audiences Before Administration—It May Influence Decisions Consider Audiences Before Administration—It May Influence Decisions Internal Use & External Reporting Internal Use & External Reporting Be Strategic Be Strategic

Internal Sharing of NSSE 2003 Results% President80 Administrative Staff68 Department Chairs59 Academic Advisors51 Faculty71 Governing Board34 Students32 Other (web site, fact book, etc.)20 Internal Audiences Source: NSSE 2003 Report Card

External Audiences External Sharing of NSSE 2003 Data% Accreditation Agencies31 State Agencies11 Media13 Web Site25 Prospective Students13 Parents13 Alumni13 Other7 No External Disclosure35 Source: NSSE 2003 Report Card

Ways to Communicate Results  Customize Report to Reach Campus Audiences  Organize NSSE data by colleges, depts, programs  Focus on items of interest to stakeholder and communicate understandable percentages  Use data to start discussions  Use data to correct misunderstandings  Assemble and distribute detailed information on method and credibility of survey

Activity: Connecting Findings with Relevant Audiences Activity: Connecting Findings with Relevant Audiences What was the most interesting finding? What was the most interesting finding? Who needs to know about this? Who needs to know about this? How can you let them know? How can you let them know?

National Context Putting Your Results in Context:  How good is good enough?  What would it mean to have 80% of your students indicate that they are satisfied with the college?

Sampling Methods Sampling Consistency Across Colleges:  CCSSE takes a stratified random sample of courses  NSSE takes random samples of first- and fourth-year students  Results are representative of the populations from which they are drawn  Over-sampling for targeted populations

Comparing Your Results Sampling Consistency Across Colleges:  Because results are comparable across institutions, institutions can situate their results  Anything on the survey can potentially be benchmarked

Benchmarking Two Approaches:  Normative - compares your students’ responses to those of students at other colleges and universities.  Criterion - compares your school’s performance against a predetermined value or level appropriate for your students, given your institutional mission, size, curricular offerings, funding, and so forth.

Criterion Referenced Approach Most valued activities Most valued activities Ask faculty & staff what is most valued in institution, department, then present relevant data Eliminate “Nevers” Eliminate “Nevers” Work on reducing or eliminating reports by students of never doing specific engagement activities

Assessment Purposes  Accountability  Improvement

Activity: Identify Appropriate Benchmarks Activity: Identify Appropriate Benchmarks Internal Differences: How do part- and full-time students differ? Internal Differences: How do part- and full-time students differ? External Examples: How does your institution differ from similar colleges? External Examples: How does your institution differ from similar colleges?

Principles for Good Practice in Undergraduate Education (Chickering & Gamson, 1987)  Student-faculty contact  Active learning  Prompt feedback  Time on task  High expectations  Respect for diverse learning styles  Cooperation among students

Lessons from the Research  What matters most is what students do, not who they are  A key factor is the quality of effort students expend  Educationally effective institutions channel student energy toward the right activities

Principles For Promoting Student Engagement  Intentionality - Not leaving serendipity to chance  Alignment (mission, curriculum, student experiences)  Collaboration (pull many levers)  Assessment and feedback to guide/document improvement

How is Engagement Measured?  Switch benchmarks  How would you measure this construct?

Psychometric Findings: Reliability  Both instruments have been shown to be reliable and valid  Good construct reliability: benchmarks items measure the same construct  Structural equation models have been used to demonstrate that there is strong consistency across subpopulations (gender & part- v. full- time) and across years

Psychometric Findings: Validity

Psychographic Data  NSSE data used to form clusters of students  pdf pdf pdf  Psychographic data consistently was more effective at predicting outcomes than demographic data

The Florida Opportunity  How does engagement relate to outcome?  Persistence  Goal Attainment

Activity: Expectations and Desires in Student Engagement  How many students responded often or very often?  What would you like to see?  How did they respond?  How can you increase engagement?

Questions and Suggestions  Questions for NSSE and CCSSE  Suggestions for NSSE and CCSSE