Three Colleges Collaborate to Assess “Study Abroad” – Influential Factors, Experiences and Outcomes Christine Brooks Cote, Bowdoin College James C. Fergerson, Bates College Mark Freeman, Colby College North East Association for Institutional Research Portsmouth, NH November 2004

Off-Campus Study Survey Participants Bates Bowdoin Colby Amherst Smith

Project Development Genuine interest in understanding the impact upon students of a semester or a year spent abroad Genuine interest in understanding the impact upon students of a semester or a year spent abroad A need to evaluate the Colby-Bates-Bowdoin Off-Campus Study Programs (CBB) in light of the goals presented to the Mellon Foundation A need to evaluate the Colby-Bates-Bowdoin Off-Campus Study Programs (CBB) in light of the goals presented to the Mellon Foundation Seized the opportunity to use the instrument in a collaborative assessment project supported by NEASC and funded by the Mellon Foundation Seized the opportunity to use the instrument in a collaborative assessment project supported by NEASC and funded by the Mellon Foundation

With Support From: “Motivating Inquiry Regarding Teaching and Learning at Critical Transition Points in Liberal Education”

20 Liberal Arts Colleges Participated “Motivating Inquiry Regarding Teaching and Learning at Critical Transition Points in Liberal Education” 20 Liberal Arts Colleges Participated First Year Seminar study First Year Seminar study Senior/Capstone Experience study Senior/Capstone Experience study Off-Campus Study survey Off-Campus Study survey “Faculty Fellows” interviews of students about focusing on an area of study “Faculty Fellows” interviews of students about focusing on an area of study Institutions could opt in or out of each study

Comparing Study Away Programs

Survey Development Based upon a survey designed at Bowdoin Based upon a survey designed at Bowdoin Survey was reviewed and revised by the participating institutions (IR offices, off-campus study offices, faculty, deans, NEASC) Survey was reviewed and revised by the participating institutions (IR offices, off-campus study offices, faculty, deans, NEASC) Flexible administration: Common “core” of questions with some local questions; language was localized Flexible administration: Common “core” of questions with some local questions; language was localized Colby/Bates/Bowdoin also used the survey for a final evaluation its CBB Consortium programs in London, Cape Town, and Quito Colby/Bates/Bowdoin also used the survey for a final evaluation its CBB Consortium programs in London, Cape Town, and Quito

Survey Administration Bowdoin: Done in conjunction with senior exit interviews Bowdoin: Done in conjunction with senior exit interviews Bates: Stand-alone Web survey Bates: Stand-alone Web survey Colby: Web survey in conjunction with Senior Survey Colby: Web survey in conjunction with Senior Survey HTML code shared, but some questions and wording were “localized”; data collected locally HTML code shared, but some questions and wording were “localized”; data collected locally Institutions sent data to Colby, for processing/clean-up; SPSS file returned Institutions sent data to Colby, for processing/clean-up; SPSS file returned

Bates: “Calendar Wars” & “Working the Web” Paper announcement with 5 reminders. Survey appears as “to do” link in student’s personal academic affairs account until submitted. L.L. Bean gift certificates raffled as “incentives”.

Survey Components Students’ motivations for off-campus study choices Students’ motivations for off-campus study choices Preparation and advising for study away experience Preparation and advising for study away experience Student experiences while away Student experiences while away Perceptions of the impact on academic and personal growth Perceptions of the impact on academic and personal growth Senior academic activities that built upon the study abroad experience Senior academic activities that built upon the study abroad experience Several “free-response” questions Several “free-response” questions

Major Survey Components Students’ motivations for off-campus study choices Very strong influence No influence on choices on choices My desire to learn or become more fluent in the language of this country or region of the world My desire to learn or become more fluent in the language of this country or region of the world My desire to study in a country or region of the world that is very different from the one I have lived in most of my life My desire to study in a country or region of the world that is very different from the one I have lived in most of my life My desire to have a lighter academic load than I would normally have at [Bates or Bowdoin or Colby] My desire to have a lighter academic load than I would normally have at [Bates or Bowdoin or Colby]

Motivations for Study Away (All Institutions) * Not asked by Bowdoin ** “Academic Advisors or Professors” at Bowdoin

Major Survey Components Students’ experiences while studying away Strongly agree Strongly disagree While away, I was part of an intellectual community that was invigorating and thought- provoking. While away, I was part of an intellectual community that was invigorating and thought- provoking. Most of my time outside of class was spent with people who called the country or region of the world their home. Most of my time outside of class was spent with people who called the country or region of the world their home. While away, I nearly always spoke a language other than English. While away, I nearly always spoke a language other than English.

Major Survey Components Students’ perception of impact of off-campus study experience Strongly agree Strongly disagree The experiences I had while away caused me to change my understanding of how people around the world view the United States. The experiences I had while away caused me to change my understanding of how people around the world view the United States. My understanding of the culture, country or region of the world I was in changed significantly by my living there for a time. My understanding of the culture, country or region of the world I was in changed significantly by my living there for a time. The experiences I had while away taught me a great deal about gender issues facing people in other parts of the world. The experiences I had while away taught me a great deal about gender issues facing people in other parts of the world. Returning to [Bates or Bowdoin or Colby], I engaged in research or independent study that directly built upon the work I did while away. Returning to [Bates or Bowdoin or Colby], I engaged in research or independent study that directly built upon the work I did while away.

Key Experiences/Outcomes Questions Did students gain the skills that are desirable in a liberal arts context? Did students gain the skills that are desirable in a liberal arts context? How did outcomes vary by type of student, field of study, or by institution? How did outcomes vary by type of student, field of study, or by institution? Did study away change personal, academic, and career goals? Did study away change personal, academic, and career goals? Did students link study away experiences to other academic work? Did students link study away experiences to other academic work? How do they compare with those who didn’t study away? How do they compare with those who didn’t study away?

Major Study Away Outcomes Overview

Other Study Away Outcomes Overview

Study Away Reasons - Cluster Analysis Reasons for studying abroad items are the best candidates for a priori or “causal” variables. Reasons for studying abroad items are the best candidates for a priori or “causal” variables. If Reasons presuppose certain outcomes then a model focusing on them has diagnostic value in practice. If Reasons presuppose certain outcomes then a model focusing on them has diagnostic value in practice. Cluster analysis into “types” to simply interpretation. Cluster analysis into “types” to simply interpretation. A four cluster solution fit the five-school data best A four cluster solution fit the five-school data best

Selected Cluster Solution Cluster names are subjective interpretations of item response patterns Cluster characteristics

Does Cluster Membership Predict OCS Experiences? Ay-uh… Well-Roundeds and Experientialists strong on cultural immersion, language learning experiences; weakest of the four clusters on “adjusting to academic structure” Well-Roundeds and Experientialists strong on cultural immersion, language learning experiences; weakest of the four clusters on “adjusting to academic structure” Narrow Academics stand out on “keeping in touch with professors back home” and “course learning”, “part of intellectual community”; weakest on cultural immersion experiences Narrow Academics stand out on “keeping in touch with professors back home” and “course learning”, “part of intellectual community”; weakest on cultural immersion experiences Club Meds weak pretty much everywhere, especially traditional academics – but more cultural immersion than Narrow Academics Club Meds weak pretty much everywhere, especially traditional academics – but more cultural immersion than Narrow Academics

Does Cluster Membership Predict OCS Outcomes? Ay-uh… Well-Roundeds strongest on most outcomes, virtually across the board – “academic integration” as well as “personal growth” and “cross-cultural understanding” outcomes Well-Roundeds strongest on most outcomes, virtually across the board – “academic integration” as well as “personal growth” and “cross-cultural understanding” outcomes Narrow Academics weakest on “personal growth” and “cross-cultural understanding” – almost as strong as Well- Roundeds in “integration with post-OCS academics” Narrow Academics weakest on “personal growth” and “cross-cultural understanding” – almost as strong as Well- Roundeds in “integration with post-OCS academics” Club Meds bring up the rear virtually across the board, especially on post-OCS academic integration. They do, however, report that that the experience was “rewarding” Club Meds bring up the rear virtually across the board, especially on post-OCS academic integration. They do, however, report that that the experience was “rewarding”

Comparing Students By College By Cluster

Academic Division and Clusters - Bates

Location of Study – By Cluster (Bates)

Cumulative Average GPAs – By Cluster (Bates)

Changes in Goals – Bates

Core Changes – By Division

Linking OCS Survey Data With Senior Survey Outcomes Responses were linked with HEDS senior survey data (Colby only) – can leverage the OCS survey results against other institution-wide outcomes Responses were linked with HEDS senior survey data (Colby only) – can leverage the OCS survey results against other institution-wide outcomes And can compare outcomes with non- study abroad graduates And can compare outcomes with non- study abroad graduates

Regression Analysis: Are Other Factors Linked To Outcomes? So, “reasons for studying abroad” account for a healthy portion of the variance in outcomes. So, “reasons for studying abroad” account for a healthy portion of the variance in outcomes. What of other factors? What of other factors? Sex Sex Major Major Country of study Country of study Does school “uniquely” account for any variation in outcomes? Does school “uniquely” account for any variation in outcomes? If so, may help If so, may help

Regression analysis

Survey Data Useful for CBB Program Evaluation CBB Off-Campus Study Programs were funded by the Mellon Foundation CBB Off-Campus Study Programs were funded by the Mellon Foundation Two critical goals of the programs were to generally enhance the academic quality of study abroad experiences and improve the integration of study abroad experiences with the educational program at the home college. Two critical goals of the programs were to generally enhance the academic quality of study abroad experiences and improve the integration of study abroad experiences with the educational program at the home college.

Survey Data Useful for CBB Program Evaluation Motivations The academic opportunities within the program or country support my major CBB 61% All Others 47% CBB 61% All Others 47% My desire to have a lighter academic load than I would normally have at [Bates or Bowdoin or Colby] CBB 5% All Others 13% CBB 5% All Others 13%

Survey Data Useful for CBB Program Evaluation Experiences and Perception of Impact While away, I was part of an intellectual community that was invigorating and thought-provoking. CBB 55% All Others 43% CBB 55% All Others 43% I learned a great deal from the courses I took while away. CBB 81% All Others 61% CBB 81% All Others 61%

Personal Growth / Reflection “Study abroad gave me the opportunity to have my own thoughts; it provided space from academics that was filled by reflection. That's very important in college because students are confronted by so much information that is difficult to synthesize. With study abroad, there is time and space to reflect on Bates, on the USA, on yourself.” -- Anthropology major, Ghana It is difficult to put in words. I learned so much - and it is still sinking in after a year. I learned to interview powerful political figures in a different language. I learned to live - and enjoy living in a home without indoor plumbing. I even learned a lot from being mugged, getting parasites, and getting sick - all of it opened me up to a different reality from life at Bates. -- Political Science major, Madagascar

Connections to Research “Broadly speaking, it was a lesson on life, what it means to be human, and how to interact and communicate cross- culturally. I worked so hard with subsistence campesino potato farmers to produce our own food or kill our own animals, and that has made me much more conscious of the flow of food globally. At Bates I took a Social Justice Spanish class, I was rewarded an Otis Fellowship to return to Bolivia, and I understood my thesis with greater depth. The mix of Bates and Off Campus help me confront my own assumptions and romanticisms about the life in Bolivia, Vermont, and so on. --Environmental Studies Major, Bolivia

Senior Thesis

Senior Thesis / Research Links “My senior thesis related directly to my experience in Tasmania. The first week of my stay there was the most influential in my thesis. I returned to Tasmania the following summer to do research for my thesis, and having already lived there, made connections and gotten to know the culture and area was very helpful. I could hit the ground running.” -- Environmental Studies major “My thesis was not about Ecuador, but I extensively worked with Spanish texts, so that was important. Although I did not like Ecuador very much, the experience provided me with confidence to apply for a Phillips Fellowship, which allowed me to do pre-thesis research in Cuba.” -- History major

Uses of Collaborative Assessment – Advantages Labor-saving: Divides up workload and tasks Labor-saving: Divides up workload and tasks Survey design involved faculty and other administrators Survey design involved faculty and other administrators Improved analysis by sharing technical expertise and discussing results Improved analysis by sharing technical expertise and discussing results Promoted other data-sharing – Your First College Year results shared Promoted other data-sharing – Your First College Year results shared Importance of comparative data – why are we different, and what does that mean? Importance of comparative data – why are we different, and what does that mean?

Uses of Collaborative Assessment -- Concerns Different campus cultures and analytical approaches Different campus cultures and analytical approaches Timing Timing Campus calendars / office work schedules Campus calendars / office work schedules Assembling the right team(s) Assembling the right team(s) Getting “Buy-in” to exchange data Getting “Buy-in” to exchange data FERPA concerns (exchanging non-survey outcomes data) FERPA concerns (exchanging non-survey outcomes data) Integrating data from different collection methods Integrating data from different collection methods

Policy Discussions and Faculty Reactions “This is really interesting!” “This is really interesting!” At Bowdoin, the faculty Off-Campus Study Committee has reviewed the entire report, including qualitative data from one-on-one and group interviews. At Bowdoin, the faculty Off-Campus Study Committee has reviewed the entire report, including qualitative data from one-on-one and group interviews. Data from the report will be used in our final CBB report to the Mellon foundation. Data from the report will be used in our final CBB report to the Mellon foundation. Important assessment discussions would focus on reasons behind college differences...yet to happen. Important assessment discussions would focus on reasons behind college differences...yet to happen.

Questions? Contacts: Contacts: Christine Brooks Cote, Christine Brooks Cote, Jim Fergerson, Jim Fergerson, Mark Freeman, Mark Freeman,