Researchers Beware: Comparing FSSE with NSSE Can Be Messy Ed Rugg, Director Center for Institutional Effectiveness Kennesaw State University Presented.

Slides:



Advertisements
Similar presentations
Student Experiences with Information Technology and their Relationship to Other Aspects of Student Engagement Thomas F. Nelson Laird and George D. Kuh.
Advertisements

Now That They Stay, What Next?: Using NSSE Results to Enhance the Impact of the Undergraduate Experience.
Academic Advising Cathy Sanders Director of Assessment and Faculty Governance UNC Charlotte Office of Academic Affairs.
Gary Whisenand Director, Institutional Research August 26, 2011.
Gallaudet Institutional Research Report: Annual Campus Climate Survey: 2010 Pat Hulsebosch: Executive Director – Office of Academic Quality Faculty Senate.
Using the 2005 National Survey of Student Engagement in Student Affairs Indiana State University.
Indiana State University Assessment of General Education Objectives Using Indicators From National Survey of Student Engagement (NSSE)
Gallaudet University Results on National Survey of Student Engagement Office of Institutional Research August, 2007.
2012 National Survey of Student Engagement Jeremy D. Penn & John D. Hathcoat.
Shimon Sarraf, Research Analyst Center for Postsecondary Research, Indiana University Bloomington Session for NSSE “Veterans” Regional NSSE User’s Workshop.
FACULTY DEVELOPMENT INSTITUTE – 23/2/2008 THE SYLLABUS CORNERSTONE OF EFFECTIVE LEARNING FACILITATOR: Professor Pandeli Glavanis (PhD) Associate Director,
Learning Community II Survey Spring 2007 Analysis by Intisar Hibschweiler (Core Director) and Mimi Steadman (Director of Institutional Assessment)
Basic Reports and Data Dissemination Strategies Regional Users’ Workshop October 6-7, 2005.
Survey of Entering Student Engagement Survey of Entering Student Engagement SENSE: a 30,000 Ft. View.
Mind the Gap: Overview of FSSE and BCSSE Jillian Kinzie NSSE.
General Education Courses and the Promotion of Essential Learning Outcomes Thomas F. Nelson Laird Amanda Suniti Niskodé George D. Kuh Center for Postsecondary.
Urban Universities: Student Characteristics and Engagement Donna Hawley Martha Shawver.
Faculty Survey of Student Engagement Using What Faculty Say about Improving Their Teaching Thomas F. Nelson Laird, IUB Jennifer Buckley, IUB Megan Palmer,
BENCHMARKING EFFECTIVE EDUCATIONAL PRACTICE IN COMMUNITY COLLEGES What We’re Learning. What Lies Ahead.
Faculty Survey of Student Engagement Getting Faculty Involved in the Student Engagement Conversation: The Faculty Survey of Student Engagement Thomas.
Nuggets from NSSE: Evidence for the Assurance of Learning ( 216 ) Ed Rugg, Director & SACS Liaison Ed Rugg, Director & SACS Liaison Center for Institutional.
Nuggets from NSSE & FSSE on Student Learning Outcomes for Accreditation Ed Rugg PhD, Director of Institutional Effectiveness & SACS Liaison Ed Rugg PhD,
Nuggets from NSSE & FSSE on Student Learning Outcomes for Accreditation Presented by Dr. Ed Rugg, Director of Institutional Effectiveness & SACS Liaison,
Report of the Results of the Faculty Survey of Student Engagement William E. Knight and Jie Wu Office of Institutional Research Presentation to the Faculty.
DIXIE STATE UNIVERSITY The Faculty Role in Student Retention
Derek Herrmann & Ryan Smith University Assessment Services.
1 N ational S urvey & F aculty S urvey of S tudent E ngagement (NSSE) & (FSSE) 2006 Wayne State University.
The Faculty Survey of Student Engagement (FSSE) measures faculty expectations for student engagement in educational practices that are known to be empirically.
Derek Herrmann & Ryan Smith University Assessment Services.
2009 National Survey of Student Engagement (NSSE) Report Institutional Research & Information November 18, 2009.
Before & After: What Undergraduates and Alumni Say About Their College Experience and Outcomes Angie L. Miller, NSSE & SNAAP Research Analyst Amber D.
Student Engagement: 2008 National Survey of Student Engagement (NSSE) Office of Institutional Research and Planning Presentation to Senate November 2008.
Gallaudet Institutional Research Report: National Survey of Student Engagement Pat Hulsebosch: Executive Director – Office of Academic Quality Faculty.
2009 National Survey of Student Engagement (NSSE) Report Institutional Research & Information November 18, 2009.
Maryland Consortium Findings from the 2006 CCSSE Survey.
Primary Factors of Student Engagement at UTBTSC in 2002 Deborah Suzzane, Ph.D., Director Institutional Research & Planning.
2009 National Survey of Student Engagement (NSSE) Report Institutional Research & Information November 18, 2009.
An Overview.  Association of American Colleges and Universities (AAC&U)  Liberal Education and America’s Promise (LEAP)  aacu.org/leap.
1 Presentation of Results NSSE 2005 Florida Gulf Coast University Office of Planning and Institutional Performance.
National Survey of Student Engagement 2009 Missouri Valley College January 6, 2010.
National Survey of Student Engagement 2007 Results for Students in Graduate and Professional Studies.
1 Board Meeting Data Presentation August 25, 2009.
Measuring the Gap Between Faculty and Student Perceptions of Engagement: Results from the Faculty Survey of Student Engagement University Assessment Office.
1 This CCFSSE Drop-In Overview Presentation Template can be customized using your college’s CCFSSE/CCSSE results. Please review the “Notes” section accompanying.
Faculty Satisfaction Survey Results October 2009.
National Survey of Student Engagement (NSSE) Comparison on the survey results at UPRM with peers Office of Institutional Research and Planning University.
Integrating Teaching, Learning and Assessment: A Programmatic ePortfolio Process AAEEBL Conference, Boston, MA - July 28, 2011 Karen Stein, Faculty Director,
NSSE 2013 Results Report to Faculty Senate February 19, 2014.
UNDERSTANDING 2012 NATIONAL SURVEY OF STUDENT ENGAGEMENT (NSSE) RESULTS Nicholls State University October 17, 2012.
Diversity at Stetson: Perspectives of Students and Faculty John Tichenor Associate Professor of Decision and Information Science.
RESULTS OF THE 2009 ADMINISTRATION OF THE COMMUNITYCOLLEGE SURVEY OF STUDENT ENGAGEMENT Office of Institutional Effectiveness, April 2010.
RESULTS OF THE 2009 ADMINISTRATION OF THE COMMUNITYCOLLEGE SURVEY OF STUDENT ENGAGEMENT Office of Institutional Effectiveness, September 2009.
Office of Institutional Research and Effectiveness 1 The University of Texas-Pan American National Survey of Student Engagement 2003, 2004, 2005, 2006.
The University of Texas-Pan American Susan Griffith, Ph.D. Executive Director National Survey of Student Engagement 2003 Results & Recommendations Presented.
The University of Texas-Pan American National Survey of Student Engagement 2013 Presented by: November 2013 Office of Institutional Research & Effectiveness.
National Survey of Student Engagement Executive Snapshot 2007.
The University of Texas-Pan American National Survey of Student Engagement 2014 Presented by: October 2014 Office of Institutional Research & Effectiveness.
Faculty Senate Pat Hulsebosch, Office of Academic Quality 11/17/08.
Full-time Respondents
The University of Texas-Pan American
FACILITATOR: Professor Pandeli Glavanis (PhD) Associate Director, CLT
NSSE Results for Faculty
NSSE 2004 (National Survey of Student Engagement)
The University of Texas-Pan American
Derek Herrmann & Ryan Smith University Assessment Services
Your Institutional Report Step by Step
Your Institutional Report Step by Step
The University of Texas-Pan American
Learning Community II Survey
Indiana University Bloomington
Presentation transcript:

Researchers Beware: Comparing FSSE with NSSE Can Be Messy Ed Rugg, Director Center for Institutional Effectiveness Kennesaw State University Presented at the AIR Forum 2005 San Diego, California

National Surveys of Student Engagement Student Perspectives (NSSE) The College Student Report 2004 The College Student Report items for response 100 items for response Faculty Perspectives (FSSE) Faculty Survey of Student Engagement items for response 112 items for response

Suggested Comparisons The 2004 Institutional Report Supported Making Comparisons of Faculty and Student Responses on 57 Supposedly Similar FSSE and NSSE Items

FSSE & NSSE: Kin, But Not Twins FSSE is an Apple NSSE is an Orange Extra Care is Needed when Comparing Apples with Oranges

Direct Comparisons of NSSE & FSSE Responses Are Messy “Comparable items” for the two surveys are rarely worded identically or similarly. The context or focus for responses is often not the same across the two surveys. Response categories for “comparable items” are often very different for the two surveys. Substantive Differences Confound the Results of Most Comparisons

Few Well-Matched Pairs Few Well-Matched Pairs Only 10 of the 57 “comparable items” from NSSE and FSSE were nearly identical in wording—a well- matched pair Only 10 of the 57 “comparable items” from NSSE and FSSE were nearly identical in wording—a well- matched pair

Well-Matched Pairs in 2004 Category FSSE Items NSSE Items Quality of Student 2, 3, 4 8 a-c Relationships Relationships Institutional 5 a-e, g-h 10 a-g Environment Environment

Informative Direct Comparisons for Well-Matched Items When responding to equivalent items in the same context, faculty and students agreed more than disagreed, with a few notable exceptions. See “Well-Matched Comparisons of NSSE & FSSE Handout” on institutional environment and quality of student relationships

Different Response Contexts & Focuses NSSE respondents focus on the first year or senior year experience as a whole FSSE respondents focus on a selected lower or upper division course experience

Interpret Contextual Differences with Caution For example, when more students than faculty report that students often have conversations with other students of a different race or nationality (FSSE 13f vs. NSSE 1u), remember that the faculty are referencing in-class experiences only, while students are reflecting on their out-of-class as well as in-class experiences as a whole.

Different Response Categories Limit Comparability For 20 of the 57 “comparable items,” faculty used different response categories than students used

Example of a Messy Item Comparison Involving Different Response Categories Why compare an instructor’s rating of the importance of study abroad… with a student’s reported plan for completing a study abroad experience ? What would any percentage difference mean ? Enriching Educational Experiences (FSSE 1f compared to NSSE 7f)

Example of a Messy Matched Pair in Wording, Context, & Response Why compare the faculty’s report of the percentage of students who frequently asked questions in one selected course… with the student’s report of how frequently he/she asked questions in class during the current school year ? What would any percentage difference mean? Academic & Intellectual Experiences (FSSE 12a compared to NSSE 1a)

Consider Convergent Analyses When direct comparisons of FSSE with NSSE items are messy, consider parallel descriptive analyses and inspect the results for convergence. See “Nuggets from NSSE & FSSE – Handouts 1-4” on student learning outcomes-- extracted from convergent analyses of NSSE 11 & 2 and FSSE 21 & 20

Where Do We Go From Here? Proceed with caution Proceed with caution Wear boots Wear boots Get to the high ground Get to the high ground

An Opportunity to Improve FSSE FSSE has great untapped potential to capture valuable informed judgments of educational experts (the faculty) in ways that would enable cross-validation of the self-reported engagement of students in NSSE.

Another Key Opportunity NSSE and FSSE could be more centrally involved in the national conversations on documenting achievement of student learning outcomes.

Your Turn for Questions & Comments PowerPoint and handouts are available at: PowerPoint and handouts are available at: Thanks! Thanks!