1. Who Are Our Students??? Assessment Forum Monday, Feb. 9, 2004 Donald J. Lerew, Jr. – Institutional Researcher Ronald Burwell – Professor of Sociology

2. Overview Welcome – Review of Agenda IR & Assessment Website & Data Sets Assessment Calendar AICUP First-Year Student Survey 2003 Senior Interviews – Some Highlights NSSE Student Engagement Survey 2002 CSS Senior Survey 2003

3. Talking Points How do seniors view their time at Messiah? How many hours do our students spend in academic pursuits in a given week? How do students’ views change regarding gender roles? Who stays and who leaves…and why? Social vs. Academic Factors? How much effort do our students put into their college experience?

4. Research/Assessment Web Site General, college-wide statistics IR Publications Listing Assessment Calendar Assessment data –CIRP, CSS, NSSE, CSEQ, SSI, FYSS Location: Messiah Home > Campus Offices > Institutional ResearchMessiah Home > Campus Offices > Institutional Research

5. Survey Data Sets Most analyses have datasets available Datasets typically created for SPSS Contact Ron or I for a copy

6. Assessment Calendar Schedule of planned institutional assessment activities over next several years Presented and reviewed by Assessment Committee Goal is to not overburden students, while gathering sufficient, meaningful data Calendar is available on IR website

8. Assessment Calendar CIRP – Cooperative Inst. Research Program –Survey of New Freshmen at orientation –Baseline picture of who students are before college AICUP First-Year Student Survey –Survey of first-year students in spring semester –Assesses academic and personal development of students during their first year of college

9. Assessment Calendar NSSE – National Survey of Student Engagement –Given to 2 nd -semester freshmen and seniors –Studies quality of UG learning on campus by way of student engagement CSS – College Student Survey –Surveys Messiah seniors –Students’ self-reported academic and campus life experiences

10. Assessment Calendar CSEQ – College Student Experiences Questionnaire –Surveys seniors –Assesses the quality of student effort in utilizing resources/opportunities at college SSI – Student Satisfaction Inventory –Surveys subset of each class level –Measures student importance and satisfaction to various campus issues

11. Assessment Calendar Other Possible Instruments YFCY – Your First College Year – Supplement to CIRP FSSE – Faculty Survey of Student Engagement Others???

12. AICUP – First-Year Student Survey Spring 2003 Survey of first-year students Surveyed during spring semester Surveyed through campus mail General and retention analyses conducted

13. AICUP – First-Year Student Survey --Self-Assessment-- MC men are significantly less prepared & are significantly more challenged in the area of study skills during the first year MC students indicate study and writing skills as their two biggest first-year skill challenges Both self-rating and first-year assessment of spirituality traits were significantly better than comparison institutions

14. AICUP – First-Year Student Survey --Transition to College-- MC women experience significantly more homesickness than men MC students more likely to be homesick than at comparison institutions MC men report significantly less perceived help in areas of: –Dealing with roommate problems –Dealing with personal problems –Working with advisor

15. AICUP – First-Year Student Survey --Academic Advising-- MC men relied significantly more on themselves for academic advice than did women (M-28.6%, W-15.4%) MC students placed more emphasis on soliciting academic advice from family (MC-25.9%, Comp-15.1%) MC students placed less reliance on the advisor (MC-21.4%, Comp-29.0%)

16. AICUP – First-Year Student Survey --Academic Advising-- MC women met with advisor more often during first year (M-2.55, W-3.26 mtgs.) –Comparison group averaged 3.80 meetings MC students rated advisors significantly better than comparison group students in: –Knowledge level, availability, & overall satisfaction

17. AICUP – First-Year Student Survey --Use of Time-- MC students spend nearly 22 hrs/wk attending class or studying/doing homework MC men spend significantly more time than women watching TV and playing video/computer games MC women spend significantly more time in prayer and meditation

18. AICUP – First-Year Student Survey --Use of Time-- MC students spend significantly more time than comparison group students in: –Prayer/meditation –Studying and completing homework –Participating in student clubs/groups

19. AICUP – First-Year Student Survey --Satisfaction-- MC women significantly more satisfied in: –Quality of academic advising, quality of academic offerings, availability of academic offerings, Internet access, faculty attitude toward students, and personal counseling MC students indicated significantly higher satisfaction than comparison students in: –Library facilities/resources, food services

20. AICUP – First-Year Student Survey --Satisfaction-- 93.7% of MC students indicated that they would “definitely” or “probably” return for the second year (Comp. Group - 90.2%) Prominent reasons for not returning –Transferring –Dissatisfaction with social atmosphere –Lack of financial resources

21. AICUP – First-Year Student Survey --Retention Analysis: General-- Attrited students reported significantly… –Higher levels of homesickness and loneliness –Lower levels of belongingness Attrited students indicated significantly less college help in the areas of: –Personal problems, student club involvement, organized social events, rec. athletic activities

22. AICUP – First-Year Student Survey --Retention: Skills Assessment-- Attrited students rated themselves significantly lower in the areas of: –Intellectual self-confidence, social self- confidence, ability to function independently, leadership ability, reading analytically, and spirituality While incoming skill levels differed, no significant differences in reported growth during first year

23. AICUP – First-Year Student Survey --Retention: Other Items-- Attrited students were significantly more likely to indicated the advisor as the primary source of academic advice; equally less on self-reliance Retained students spent more time socializing with friends (7 vs. 4 hrs/wk) Both student groups indicated similar academic, residence hall, and social life expectations –Attrited students indicated significantly lower levels of having those expectations met during the first year

24. Senior Interviews – Spring 2002 Approximately 27 completed interviews with seniors during the spring of 2002 Transcriptions created for 24 interviews (sound quality created some difficulties in transcription) Topics: Educational Experiences; Religion; Gender

25. Educational Experiences Based on questions from Richard Light –Academic regrets –Unforgettable experiences –Critical moments –Important books –Interactions in the residence hall

26. Religion Awareness of and introduction to college’s religious heritage [related to Lilly Project] Important formative faith experiences Experiences with doubt Understanding of what is a “mature Christian” [James Fowler]

27. Gender Views on male / female roles with society and the family Views on ordination of women and role in the church Impact of Messiah College on ideas regarding gender

28. Senior Voices Academic regrets Unforgettable experiences Residence Hall interactions Views on gender roles Ordination of women

29. NSSE – Student Engagement Spring 2002 Survey of 1 st -yr. and senior students Determine extent to which students engage in educational practices that are associated with learning outcomes Surveyed during spring semester Web administration General and school analyses

30. NSSE – Student Engagement Percentile Ranks BenchmarkClMC Score Bac- Gen Bac- Lib Arts NSSE 2002 Level of academic Challenge Fr Sr 57.6 62.290-100 40-50 60-70 80-90 90-90 Active/collaborative Learning Fr Sr 45.8 50.8 70-80 40-50 60-70 40-50 80-90 60-70 Student/Faculty Interactions Fr Sr 35.9 45.1 40-50 40-4010-20 50-60 60-70 Enriching Educ. Experiences Fr Sr 65.2 57.690-100 40-50 50-6080-90 Supportive Campus Environment Fr Sr 65.5 66.0 70-80 80-90 60-70 80-90 80-90 90-100

31. NSSE – Student Engagement --Acad/Intellectual Experiences-- Only 21% of MC seniors worked with other students on projects during class “often” or “very often” (Bac-Gen & NSSE – 43%) MC first-year students worked significantly more often with classmates outside of class to complete class assignments MC students discussed class ideas with others outside of class significantly more often than comparison students

32. NSSE – Student Engagement --Acad/Intellectual Experiences-- MC women were more likely than men to: –Use e-mail to communicate with instructor –Work harder than they thought they could –Discuss ideas with others outside of class –Work with classmates outside of class Students w/ a “B- or lower” average were more likely to: –Come to class unprepared –Work with others on projects during class

33. NSSE – Student Engagement --Mental Activities-- Five Levels: (1) Memorize, (2) Analyze, (3) Synthesize, (4) Make Judgments, (5) Apply MC students had similar category breakdowns to comparison students –Exception: both first-year and senior MC students indicated lesser levels of memorization MC students w/ “B- or lower” indicated significantly higher usage of memorization

34. NSSE – Student Engagement --Reading & Writing-- MC students indicated a significantly higher number of assigned textbooks/readings than comparison students MC students indicated writing significantly more 5-19 and under-5 page papers

35. NSSE – Student Engagement --Educational Experiences-- MC students indicated significantly higher participation/plan to participate rates in: –Practicums, internships, or field experiences –Community service or volunteer work –Foreign language coursework –Study abroad –Culminating senior experience (exam, capstone, project, etc.) MC women are more likely to attend

36. NSSE – Student Engagement --Time Usage-- preparingMC students spend significantly more time preparing for class than do comparison students (FY-14.5, Sr.-14 hrs/wk) MC FY students spend 11 hrs/wk relaxing/socializing; 9 hrs/wk for seniors MC students work significantly more on- campus, but significantly less off-campus

37. NSSE – Student Engagement --Time Usage-- MC first-year women spend significantly more time preparing for class; senior men, however, spend more time than women MC students with a “B- or lower” spend: –Less time preparing for class (9 vs. 14 hrs/wk)

38. NSSE – Student Engagement --Educational/Personal Growth-- MC students indicated significantly more growth in: –Acquiring a broad general education –Developing personal code of values/ethics –Contributing to welfare of community –Writing clearly and effectively –Thinking critically and analytically MC first-year students reported significantly less growth in the use of computer/info. technology than did comparison students

39. NSSE – Student Engagement --Institutional Emphasis-- MC students reported significantly more emphasis by the college on: –Spending significantly more time studying and on academic work (seniors only) –Encouraging diversity contacts/interaction –Providing support needed to thrive socially –Attending campus events and activities Messiah women indicated higher perceived emphasis toward academic and social support

40. NSSE – Student Engagement --Satisfaction-- Over 90% of MC men and 95% of women rated the educational experience as “good” or “excellent” –Significantly higher than comparison groups A high majority of students indicated they would “probably” or “definitely” return –FY (W-94.7%, M-90.6%) –Sr. (W-93.8%, M-84.6%) Students w/ “B- or lower” evaluated educational experience significantly lower than “B or better”

41. NSSE – Student Engagement --Analysis by School-- Summarizes mean responses by school Compares each school mean by the aggregate of all other schools Compares each school mean by class level

42. College Student Survey - 2003

43. Sample Sample size – 267 graduating senior 32.2% male 67.8% female Slight oversample of females

44. Plans for fall 2003 Approximately 1 in 5 seniors plan on graduate study in the fall Most of the remaining students plan on working either part time or full time

45. Academic Activities MC students report frequent peer interaction in connection with academic activities (e.g., studying with other students and discussing courses with other students) MC students were more likely to be guests in professors homes compared to other private colleges

46. Other Activities Some 38.7% report drinking beer during the last year 56.8% report consuming wine or alcohol during the past year While these are lower than comparable rates at private colleges they are higher than in 2001 About 40% of MC students begin to use alcohol while in college

47. Other experiences A majority of students reported that they occasionally or frequently were homesick (64.7%) or depressed (63.9%) About one third of the students reported frequently feeling overwhelmed (38.7%)

48. Time Usage MC students report that they spend about 25 hours per week on academic pursuits (classes/studying) A comparison sample from private colleges report about 22 hours per week on the same activities In 2001 MC students reported slightly more time spent on academic pursuits (28 hours)

49. Interactions with Professors MC students report that professors frequently provided intellectual challenge (55.6%) and emotional support (36.5%)

50. Satisfaction with college Some 91% of MC students are satisfied with their overall college experience (private – 86%) Most MC students say they would choose the same college again (MC – 89%, private – 80%)

51. Objectives of importance The top four objectives were: –Integrating spirituality into my life –Helping others in difficulty –Raising a family –Developing a meaningful philosophy of life

52. Political Identity By their senior year, MC students are more likely to say they are “middle of the road” –First year – 32.4% –Senior year – 46.3% Comparing 2001 with 2003, the 2003 seniors seem to be less conservative and more moderate (even upon entering college)

53. Questions regarding majors In both 2001 and 2003 we asked twenty supplementary questions regarding experiences in the major In general, MC students report very high levels of satisfaction with their experiences in the major; the class of 2003 is even more positive than the class of 2001

54. Comparisons: 2001 vs. 2003 In a number of areas, the 2003 seniors are more positive than those of 2001 Given the discussion regarding alcohol, it is interesting that the use of alcohol among seniors is higher in 2003 Less time spent on academic pursuits Less conservative students

55. Questions??