2018 SOS Review Student Opinion Survey Background Big Picture Results Self-reported Learning Gains Using SOS Results for Assessment: moving from counting, to comparing, to correlating SUNY Oswego

SOS Background SUNY system-wide survey done since 1985 Administered every three years, most recently in spring 2015 and spring 2018 College impressions and plans, satisfaction with services and facilities, self-assessed learning outcomes Modifications each year In 2018, 103 items and 14 demographic questions Results include comparisons to other SUNYs of same type (12 University Colleges) Comparison Colleges: Brockport, Buff State, Cortland, Fredonia, Geneseo, New Paltz, Old Westbury, Oneonta, Plattsburgh, Potsdam, Purchase SUNY Oswego

Today’s focus Highly aggregated results from SUNY-Oswego Focus on Student Learning Outcomes, not satisfaction Will look at some differences based on college and transfer status, and based on race, gender and income, but selectively and illustratively, not comprehensively SOS over the past decade – learning contribution trends 2018 SOS to 2017 NSSE (multi-measure) Contributions of Student Affairs and Campus Environment (Community) to Student Learning Generally I’ll start off showing some pretty simple graphs of how our students answered questions – e.g. counting the number or percent who answered a certain way. Then we will dig deeper by making some comparisons – comparing how different groups of students answer questions, comparing how answers have changed over time, and comparing these results to other survey’s results. Then we go deeper still by imaging a set of relationship between sets of variables and examining how well the data reflect what we imagine might be true. SUNY Oswego

In general, how are we doing? Generally AWESOME Each is answered on a 5-point likert scale; two are satisfaction items from Very Satisfied to Very Dissatisfied. The other two use different labels (Definitely Yes….. Definitely No, Very Good…. Very Bad) What meaning does each of these capture that the others do not? Their correlations are pretty high, but NOT identical (.5 to .65 pearson’s R values) Why would so many people come back here when the percent who are very satisfied and give highest ratings is quite a bit lower? If you were the provost and you saw this graph, where would you spend your next discretionary dollar… in student affairs or faculty development? Which of these measures would you like YOUR area to most actively contribute to, and why? How? Which of your services do you think would correlate with each Big Ticket? Continuous improvement means pressing to move from good to great SUNY Oswego

15 student learning outcomes Self assessed Students rate the size of our contribution to their learning on a scale from 1, None, to 5, Very Large Results presented here are from Juniors and Seniors Averages range from 3.4 to 4.0 May be two factors of learning… “soft skills” vs. “hard skills” SUNY Oswego

Possible two “factors” of learning SUNY Oswego

Very few differences based on starting here or transferring in Definitely some differences that reflect how teaching goals and pedagogies differ from one school to another Note that student perception of learning and actual learning are correlated but not the same thing; we can learn things without realizing we have learned them (children and language), and we can think we know everything when in fact we know very little (many teenagers) All things considered, SOE students seem most aware of large gains in their learning; could reflect assessment culture – explaining to students what and why they are learning certain things Somewhat concerning that the School of Business students are least aware of or likely to report very large contributions to learning, since it houses our largest major Very few differences based on starting here or transferring in SUNY Oswego

Race and ethnicity do not clearly differentiate; Gender and Pell have some interesting patterns Average number of “Very Large” areas of learning contribution Patterns could be random, but warrant additional study These could just be random. Before making policy changes we would want to replicate these comparisons with data from 2015 and look carefully for things like a handful of outlying cases. Without a possible explanation WHY differences might exist, seeing differences in patterns has only limited usefulness. Need to have a theory about how things work. SUNY Oswego

Learning Contribution has Increased (Green line always the highest) Do we actually think students are learning more across all of these areas than they did in the earlier years? If not, what do you think explains the rising numbers? SUNY Oswego

Multi-measure Validation: NSSE items with similar wording show similar gains National Survey of Student Engagement. 4-point scales rather than 5. These are for seniors only, since the NSSE does not include juniors. But the main point is that students are definitely more likely to say their education is contributing “Very Much” compared to a decade earlier. SUNY Oswego

We’re all in this together: a model Academic Experiences Student Life Experiences Sense of Community Learning Outcomes One thing that our office does that may differ a little from yours is we try to look at outcomes on an institutional-wide basis. In the Tomorrow Plan we establish our goals Impact 2, drivers 2 and 4: Provide high-impact, inspiring and transformative learning experiences and Incorporate and infuse dynamic and effective teaching practices in curriculum across the college. Impact 1, driver 4: Encourage critical thinking, leadership, synthesis of knowledge and motivation to live active, engaged lives; all of Impact 2, the Education Ecosystem Impact 1, driver 1: Foster a college-wide culture of caring that provides supportive, effective advisement and personalized mentoring. The Tomorrow Plan implies a model that says all of these things working together will lead to the greatest learning outcomes. What do the data say? SUNY Oswego

Student Life Experiences Sense of Community Learning Academic Experiences Student Life Experiences Sense of Community Learning Intellectual stimulation Clubs and activities Sense of belonging on this campus 15 items (15 to 75; average is 56) Service learning Community service Freedom from harassment or bullying Talk with faculty outside of class Leadership development Racial harmony Innovative instruction Residence hall programs Faculty respect for students Research or creative work with faculty Providing input on college policies Non-teaching staff respect for students Student collaboration Student government Campus acceptance of individual differences Received feedback on work Recreation and intramural programs Campus openness to opinions of others Making judgements about pedagogy College social activities Social support network on campus Cultural programs Guest speakers outside of class Created scales for three sets of variables. The variables I left out usually were things that were relevant for only a few students. The scales themselves don’t mean anything specifically. It’s how they correlate with the learning scale that interests us now. SUNY Oswego

Testing the model Sense of Community goes in first, statistically, and explains 36% of the variance in Learning Next comes Academic Experience. It explains an additional 13% of the variance to bring the total to 49% Next comes Student Life. It adds an additional 6% more to bring the total to 55% Standardized betas of .31, .33 and .29 mean that these three areas of experience are almost equally important in how a student perceives his or her learning gains. It’s hard for me to explain just how well this model works. In survey research and social science work, it can be exciting to have one variable explain 10% of the variance in a dependent variable… or for three to explain 30%. So 55% is pretty massive. What’s more awesome is the balance between the three factors – the classroom, the cocurricular, and the climate. SUNY Oswego

Our Ecosystem is very robust Academic Experiences Student Life Experiences Sense of Community Learning Outcomes .33 .29 I looked at how this model changes by considering things like gender, race, college, Pell status. None of those attributes has much impact on this model. .31 SUNY Oswego

Replication: 2015 Student Life goes in first, statistically, and explains 37% of the variance in Learning Next comes Academic Experience. It explains an additional 10% of the variance to bring the total to 47% Next comes Sense of Community. It adds an additional 4% more to bring the total to 51% Standardized betas of .31, .33 and .25 mean that academics and student life experiences have similar impacts on how a student perceives his or her learning gains, and are similar to results from 2018. Sense of Community is slightly less influential than the others, and less than in 2018. Overall, the three definitely must be considered cohesively if we want to improve learning: the replication largely confirms the results from 2018. It’s hard for me to explain just how well this model works. In survey research and social science work, it can be exciting to have one variable explain 10% of the variance in a dependent variable… or for three to explain 30%. So 55% is pretty massive. What’s more awesome is the balance between the three factors – the classroom, the cocurricular, and the climate. SUNY Oswego

Replication reflection Is the higher contribution of the sense of belonging and community in 2018 compared to 2015 reflecting an actual change in our students’ experiences? It’s possible. The composition of the campus has become more diverse which means the opportunity for students to feel poorly integrated may have increased.

Using campus-wide survey data for assessment below the institution-wide level Mimicking SOS or NSSE wording when doing assessments of selected populations, services or programs This will insure that your items are well stated and may give you some benchmarking context SUNY Oswego