1. Does Orientation Promote Learning and Development? An Assessment Plan Unearths the Answers. Megan France, Steve Grande, and Sara Finney James Madison University

2. An overview  Welcome  The Orientation Program at JMU  Our Assessment Plan  Data Collection  Results  Use of results to make informed decisions about program changes  Q & A

3. Orientation  As a result of your Orientation program, how should students change?

4. James Madison University  Public; four-year  Shenandoah Valley, Virginia  3,800 first year students  Primarily residential  Center for Assessment and Research Studies

5. JMU Orientation  One-day summer program: Summer Springboard  Part of a three-phase experience 1. One Book 2. Summer Springboard 3. August Orientation  Components for students and family members

6. Orientation Overview  University welcome ~45 minutes; University President Focus on high expectations, student support, sense of community  Small group discussions led by peer mentors ~35 min; Personalizing university welcome message  Information on majors and unique academic programs ~60 min; Meetings between students and faculty  Advising and course registration ~2 hrs; Small group and 1-on-1 with advisor  Take care of university “business” and experience campus ~2.5 hrs; Staff and OPAs guide student through pre-enrollment activities and campus

7. Activities 1. Interaction with peers 2. Receive orientation materials 3. Experience the university culture 4. Inspiring speakers 5. Meet and interact with faculty and advisers 6. Interact with current JMU student mentors

8. Assessment Plan Development  One year to develop goals and objectives Labor intensive Be realistic Focus on a few goals  Finding or developing appropriate outcome measures  Data collection plan  On-going process

9. As a result of Summer Springboard...  Orientation Goal 1: Increase perceived cohesion to JMU Objective 1: Significantly increase sense of belonging Objective 2: Significantly increase feeling of morale  Orientation Goal 2: Assist in social integration to JMU Objective 1: Significantly increase college self-efficacy  Orientation Goal 3: Value a diverse campus and increase openness to diversity and challenge Objective 1: Significantly increase universal-diverse orientation  Learning Goals?

10. Perceived Cohesion  Measures the extent to which an individual experiences and reports a sense of belonging and feelings of morale towards a group (JMU) Two subscales:  3 items measure Sense of Belonging  “I see myself as part of the JMU community.”  3 items measure Feelings of Morale  “I am enthusiastic about JMU.” Scale of 1 (strongly disagree) to 9 (strongly agree) Bollen and Hoyle (1990)

11. Goals and Objectives: Perceived Cohesion ObjectiveActivityMeasures Sense of Belonging University Welcome and university business Perceived Cohesion Scale (PCS) Feelings of Morale Peer mentor meetings Perceived Cohesion Scale (PCS)

12. As a result of Summer Springboard...  Orientation Goal 1: Increase perceived cohesion to JMU Objective 1: Significantly increase sense of belonging Objective 2: Significantly increase feeling of morale  Orientation Goal 2: Assist in social integration to JMU Objective 1: Significantly increase college self-efficacy  Orientation Goal 3: Value a diverse campus and increase openness to diversity and challenge Objective 1: Significantly increase universal-diverse orientation  Learning Goals?

13. College Self-Efficacy Inventory  Measures self-efficacy or the confidence a student has with respect to a range of college-related behaviors Three subscales:  4 items measure Roommate Efficacy  “Divide chores with others you live with.”  7 items measure Course Efficacy  “Take good class notes.”  9 items measure Social/Social Integration  “Ask a professor a question.” Scale of 1 (no confidence at all) to 10 (complete confidence) Solberg, O'Brien, Villareal, Kennel, and Davis (1993)

14. Goals and Objectives: College Self- Efficacy ObjectiveActivityMeasures College Self- Efficacy Faculty, adviser, and student staff interactions College Self- Efficacy Inventory (CSEI)

15. As a result of Summer Springboard...  Orientation Goal 1: Increase perceived cohesion to JMU Objective 1: Significantly increase sense of belonging Objective 2: Significantly increase feeling of morale  Orientation Goal 2: Assist in social integration to JMU Objective 1: Significantly increase college self-efficacy  Orientation Goal 3: Value a diverse campus and increase openness to diversity and challenge Objective 1: Significantly increase universal-diverse orientation  Learning Goals?

16. Miville-Guzman Universality Diversity Scale – Short Form  Measures Universal-Diverse Orientation, which refers to an awareness and acceptance of both similarities and differences among people Three subscales:  5 items measure Diversity of Contact  “I am interested in learning about the many cultures that have existed in this world.”  5 items measure Relativistic Appreciation  “Persons with disabilities can teach me things that I could not learn elsewhere.”  4 items measure Sense of Connection  “Getting to know someone of another race is generally an uncomfortable experience for me.” (R) Scale of 1 (strongly disagree) to 6 (strongly agree) Miville (1992)

17. Goals and Objectives: Universal- Diverse Orientation* ObjectiveActivityMeasures Universal-Diverse Orientation Role modeling and interaction Miville-Guzman Universality- Diverse Scale (MGUDS)

18. Goals and Objectives ObjectiveActivityMeasures Sense of BelongingUniversity Welcome and university business Perceived Cohesion Scale Feelings of MoralePeer mentor meetings Perceived Cohesion Scale College Self-EfficacyFaculty, adviser, and student staff interactions College Self-Efficacy Inventory Universal-Diverse Orientation Role modeling and interaction Miville-Guzman Universality-Diverse Scale

19. Data Collection  Who? All first-year incoming students (not including transfer students)  What? First Year Student Survey Summer Springboard Survey  Both surveys included all three measures (PCS, CSEI, MGUDS)  Where? Surveys are on-line (i.e.,websurveyor)

20. Data Collection  When? First Year Student Survey is distributed early May and needs to be completed prior to Summer Springboard Summer Springboard Survey is distributed via e-mail after the students attend Orientation programming  Why this Design? To assess growth: The First Year Student Survey serves as a pre-test to Summer Springboard (post- test) There is no control population

21. Data Collection Issues  Missing data Assessment was not mandatory, but is expected  3,748 first year students  3,259 students completed the First Year Student Survey  1,909 completed both the First Year Student Survey and the Summer Springboard Survey Students disregard of directions  249 students completed the post-test prior to the pre-test and their data could not be used (sample reduced to 1,660 students) Invalid data  38 students from 1,660 were removed due to obvious response sets  Total sample was 1,622 out of 3,748 43.3%

22. Results: Explanation of Analyses  Significance tests Paired samples t-test used for analyses  Compare mean of scores on pre-test with mean of scores on post-test using the same students  Effect sizes Measure of practical significance  Important to investigate due to large sample Cohen’s d Examined change in the metric of the variable Examined if students were more likely to reach practical “cutoff” score after Summer Springboard.

23. Analyses: Perceived Cohesion Scale Freshman Survey (Pre) Summer Springboard (Post) MeanSDMeanSD Mean Differencep d Sense of Belonging21.304.8822.474.391.17<.001.25 Feelings of Morale22.554.9523.434.340.89<.001.18 Note. N = 1622. Scores range from 3 to 27 for both subscales.

24. Results: Perceived Cohesion  Students practically and significantly increased in sense of belonging and feelings of morale for JMU after attending Summer Springboard, but small effect size  What level of perceived cohesion would we like to see for all students after Summer Springboard?

25. Overall level of Sense of Belonging  Three items, 1 (strongly disagree) to 9 (strongly agree); range of total = 3 to 27 Wanted the majority of students to score above the midpoint of scale at end of Springboard 18 or higher  Sense of Belonging at pre-test 76.7% (1,244 students out of 1,622)  Sense of Belonging at post-test 85% (1,379 students out of 1,622)

26. Overall level of Feelings of Morale  Three items, 1 (strongly disagree) to 9 (strongly agree); range of total = 3 to 27 Wanted the majority of students to score above the midpoint of scale at end of Springboard 18 or higher  Feelings of Morale at pre-test 87.5% (1,419 students out of 1,622)  Feelings of Morale at post-test 90.3% (1,465 students out of 1,622)

27. Analyses: College Self-Efficacy Inventory Freshman Survey Summer Springboard MeanSDMeanSD Mean Differencep d Course Efficacy a 54.159.6153.729.60-0.43.02-.06 Roommate Efficacy b 33.904.8333.584.95-0.32<.001-.08 Social Efficacy c 67.4313.3868.6012.901.16<.001.13 Note. N = 1619. a = scores range from 7 to 70, b = scores range from 4 to 40, c = scores range from 9 to 90

28. Results: College Self-Efficacy  Students practically and significantly increased only for social efficacy items, but again small effect size  What level of efficacy would we like all students to score after attending Summer Springboard with respect to interacting with faculty/staff?

29. Overall levels of social self-efficacy on CSEI  Five items; 1 (no confidence) to 10 (complete confidence); range of total = 5 to 50 Wanted the majority of students to score above the midpoint of scale at end of Springboard 30 or above  Social efficacy at pre-test 78.9% (1,277 students out of 1,619)  Social efficacy at post-test 82.3% (1,333 students out of 1,619)

30. Analyses: Miville-Guzman Universality-Diversity Scale Freshman Survey Summer Springboard MeanSDMeanSD Mean Differencep d Diversity of Contact a 17.585.2417.795.38.21.01.06 Relativistic Appreciation a 22.554.1322.464.34-.08.20-.02 Sense of Connection b 20.923.9220.734.04-.19.02-.05 Note. N = 1,620. a = scores range from 5 to 30. b = scores range from 4 to 24.

31. Results: MGUDS  Students did not significantly increase in Diversity of Contact, Relativistic Appreciation, or Sense of Connection after attending Summer Springboard

32. What does this mean for Orientation at JMU?  Goal 1: Increase perceived cohesion to JMU Objective 1: Supported Objective 2: Supported  Goal 2: Assist in social integration to JMU Objective 1: Partially supported  Goal 3: Learn the value of a diverse campus Objective 1: Not supported

33. Use of Results: Perceived Cohesion  Issues to consider for future assessment Ceiling effect  Students are scoring very high overall on this measure Solution  Change directions  Find a more sensitive measure for perceived cohesion 2006: The following 6 items concern how you feel about JMU. Please read each item carefully and indicate the extent to which you agree with the statements… 2007: The following 6 items are asking how you feel about JMU. Not all students feel the same way or are expected to feel the same way. Please read each item carefully and respond according to how you feel at this current moment, not how you hope to feel as a first year student…

34. Use of Results: College Self- Efficacy  Issues to consider for future assessment Why didn’t students increase in roommate or course efficacy?  There is no programming (treatment); No need to assess constructs Minor change in social self-efficacy  Investigating new outcome measures focused on confidence to interact with faculty and staff  Kicking around the idea of assessing help-seeking

35. What about the M-GUDS?  As a result of our findings we realize that Openness to Diversity is not substantially addressed during Summer Springboard, nor will it be.  Although increased Universal-Diverse Orientation was our goal, we realized that we first needed to help students get connected to JMU and meet faculty, mentors and peers before we could fully address openness to diversity.

36. Future Assessment for 2007  Clearer focus regarding goals of program Collecting PCS and CSEI No longer assess MGUDS during Summer Springboard  Investigate ways to increase student participation in assessment endeavors  Revisiting goals and objectives

37. Thank you  Questions?  Megan France (francemk@jmu.edu)francemk@jmu.edu  Steve Grande (grandese@jmu.edu)grandese@jmu.edu  Sara Finney (finneysj@jmu.edu)finneysj@jmu.edu  Thanks to Andrew Jones for helping with data analysis and shaping this proposal.

38. Perceived Cohesion 1. I feel a sense of belonging to JMU. 2. I feel that I am a member of the JMU community. 3. I see myself as part of the JMU community. 4. I am enthusiastic about JMU. 5. I am happy to be at JMU. 6. JMU is one of the best schools in the nation.

39. CSEI 1. Make new friends at college. 2. Divide chores with others you live with. 3. Talk to university staff. 4. Manage time effectively. 5. Ask a question in class. 6. Participate in class discussions. 7. Get a date when you want one. 8. Research a term paper. 9. Do well on your exams. 10. Join a student organization. 11. Talk to your professors. 12. Join an intramural sports team. 13. Ask a professor a question. 14. Take good class notes. 15. Get along with others you live with. 16. Divide space in your residence. 17. Understand your textbooks. 18. Keep up to date with your schoolwork. 19. Write course papers. 20. Socialize with others you live with.

40. MGUDS  It is very important that a friend agrees with me on most issues.  I would like to join an organization that emphasizes getting to know people from different countries.  In getting to know someone, I like knowing both how he/she differs from me and is similar to me.  Persons with disabilities can teach me things that I could not learn elsewhere.  I would like to go to dances that feature music from other countries.  Knowing about the different experiences of other people helps me understand my own problems better.  I often listen to the music of other cultures.  Getting to know someone of another race is generally an uncomfortable experience for me.  It's really hard for me to feel close to a person from another race.  I am interested in learning about the many cultures that have existed in this world.  I can best understand someone after I get to know how he/she is both similar and different from me.  I attend events where I might get to know people from different racial backgrounds.  I am only at ease with people of my own race.  Knowing how a person differs from me greatly enhances our friendship.  I often feel irritated by persons of a different race.

41. References  Bollen, K.A. & Hoyle, R.H. (1990). Perceived cohesion: A conceptual and empirical examination. Social Forces, 69(2), 479- 504.  Solberg, V. S., O’Brien, K., Villareal, P., Kennel, R., & Davis, B. (1993). Self-efficacy and Hispanic college students: validation of the College Self-Efficacy Instrument. Hispanic Journal of Behavioral Sciences, 15, 80-95.  Miville, M.L., Gelso, C.J., Pannu, R., Liu, W., Holloway, P. & Fuertes, J.N. (1999). Appreciating similarities and valuing differences: Miville-Guzman Universality-Diversity Scale. Journal of Counseling Psychology, 46, 291-307.