1 Assessment at Selective Liberal Arts Colleges A Model for Motivating Inquiry Christine Brooks Cote Bowdoin College Mark Freeman Colby College Erin Lowery-Corkran Rhode Island Office of Higher Education / NEASC NEASC Annual Meeting, Boston, 2004

2 “Motivating Inquiry Regarding Teaching and Learning at Critical Transition Points in Liberal Education” Funded by the Andrew W. Mellon Foundation Supported by NEASC, strong leadership from Bob Froh One-year Pilot Project Participating colleges were Amherst, Bates, Bowdoin, Colby, Smith, Trinity, and Wellesley Many other colleges attended conferences and gave support and direction to the project

3 “Motivating Inquiry Regarding Teaching and Learning at Critical Transition Points in Liberal Education” Goal was to test out forms of collaboration intended to......motivate faculty engagement and...foster a culture of inquiry

4 Liberal Arts and Assessment Liberal Arts and Assessment --Dan Chambliss, Hamilton College We don’t like it Liberal arts education is different But it’s here to stay, so we should do it right

5 “Much of the discussion about assessment, even though focused on learning, sounds more like a managerial movement than an academic endeavor.” --Robert R. Newton, Boston College

6 Outcomes-Oriented Approach to Assessment 1. Identification of departmental/programmatic learning outcomes supporting institutional mission 2. Collecting data to determine whether outcomes have been met 3. Utilize data to improve or change departmental/programmatic operations (curriculum/pedagogy/student life)

7 Inquiry-Oriented Approach to Assessment 1. Identify topics of interest for research and examination within curriculum/pedagogy/student life; identify research questions 2. Collect data that will inform the issues; answer the questions 3. Provide opportunities for faculty to discuss the data, what has been learned about the topic, the answers to the questions; discuss educational goals and expectations 4. Use results for change and improvement

8 “Motivating Inquiry Regarding Teaching and Learning at Critical Transition Points in Liberal Education” ‘Student Reflective Reports’ – Exploring Student Intentionality in the First Year ‘Mellon Fellows’ – Integrating Study Abroad with Campus Experiences and Curriculum in the Major Course/Capstone Memos – Assessing the Accomplishment of Learning Goals in the Core Curriculum and the Major

9 Why it Worked Structure: Collaboration Through NEASC Process: Inquiry-based Assessment Outcome: Reflection on Learning Outcomes Outcomes assessment Changes in policy / practice Further discussion / reflection

10 ‘Student Reflective Reports’ – Exploring Student Intentionality in the First Year

11 Exploring Student Intentionality Exploring Student Intentionality Interview Questions Round One: After arriving on campus… – What are your goals for your education? How did you decide upon these goals? What courses are you taking? How do you think these courses will help you achieve your educational goals? What knowledge and competencies or skills do you hope to acquire or develop during your first year in college? Round Two: At the end of the first semester… – What have you learned about yourself and your academic interests during the first semester? Have your educational goals changed in any way? How? Why? What courses do you plan to take in the spring semester? How do these match with the courses you earlier planned on taking in the spring? Have you done any thinking about a possible major? What are your thoughts?

12 Exploring Student Intentionality Exploring Student Intentionality Data Student responses differed – some were more sophisticated, more connected, more thoughtful, more eloquent than others It appeared that some were more intentional and others were less intentional in their approach to college Might some kind of developmental model help us in understanding the data? Might we come up with some kind of “intentionality scale?”

13 Exploring Student Intentionality Data Analysis Read through students’ responses and recognized nine dimensions of student intentionality Students were rated on each of the dimensions Checked for inter-rater reliability

14 Exploring Student Intentionality Dimensions of Student Intentionality 1. Developing Skills 2. Exploring the Curriculum 3. Pursuing Knowledge 4. Declaring a Major 5. Declaring a Career 6. Making Connections 7. Planning and Mapping the Future 8. Reflecting on Own Ideas and Actions 9. Owning the Self

15 Exploring Student Intentionality Dimension One: Developing Skills Low Rating – Students spoke about gaining proficiency in a “laundry list” of skills. High Rating – Students spoke about their interest and purpose in developing proficiency in one or more skills. They spoke of the long-term or personal benefits of acquiring proficiency in skills.

16 Exploring Student Intentionality Dimension Two: Exploring the Curriculum Low Rating – Students spoke generally about getting a “liberal arts education” or becoming “well- rounded”. High Rating – Students spoke about wanting to explore the curriculum and explained why.

17 Exploring Student Intentionality Dimension Three: Pursuing Knowledge Low Rating – Students spoke about the importance of grades or fulfilling course requirements. High Rating – Students expressed a personal interest or purpose in learning that went beyond external factors.

18 Exploring Student Intentionality Dimension Four: Declaring a Major Low Rating – Students were undecided or were unable to explain the reasons for declaring a certain major or wanting to pursue study of a particular academic field. High Rating – Students spoke about narrowing their choice of majors and why they wanted to pursue studying a particular academic field.

19 Exploring Student Intentionality Dimension Five: Declaring a Career Low Rating – Students were ambiguous about future career plans. High Rating – Students spoke of narrowing their choice of professions based on personal interest and pursuing experience and knowledge that would prepare them for their chosen profession.

20 Exploring Student Intentionality Dimension Six: Making Connections Low Rating – Students did not see a connection between life and learning inside and outside of the classroom. High Rating – Students were able to connect and integrate their experiences outside the classroom with the knowledge they gained in the classroom.

21 Exploring Student Intentionality Dimension Seven: Planning and Mapping the Future Low Rating – Students spoke only of short-term educational plans. High Rating – Students spoke of a long-term educational plan, such as taking courses to fulfill a major that would point them towards a desired profession(s). Some students went so far as to analyze and reflect on their plans for attaining their educational and career goals.

22 Exploring Student Intentionality Dimension Eight: Reflecting on Own Ideas and Actions Low Rating – Students spoke of only one or two things they had learned about themselves, usually pertaining to their interests or aptitude. High Rating – Students spoke about many things they had learned about themselves that changed how they were thinking about their educational plans.

23 Exploring Student Intentionality Dimension Nine: Owning the Self Low Rating – Students spoke about external factors that shaped their goals. High Rating – Students demonstrated a commitment and interest in pursuing their goals by realizing that they were responsible for their education and for making decisions about how to live their lives.

24 Exploring Student Intentionality Exploring Student Intentionality The Intentional Learner...is purposeful and self-directed. Purpose implies clear goals, an understanding of process, and appropriate action. Further, purpose implies intention in one’s actions....possesses a self-awareness about the reason for study, the learning process itself, and how education is used....makes connections among seemingly disparate information and draws on a wide range of knowledge to make decisions or solve problems....connects study to personal life, formal educational to work, and knowledge to social responsibility. -- AAC&U, “Greater Expectations”

25 ‘Mellon Fellows’ – Integrating Study Abroad with Campus Experiences and Curriculum in the Major

26 Study Abroad Mellon Fellows Advising 4-6 students Advising 4-6 students Evaluating senior capstone projects Evaluating senior capstone projects Consulting with campus offices and NEASC to develop senior survey Consulting with campus offices and NEASC to develop senior survey

27 Study Abroad Advising Logs - Themes Purpose of study abroad experience Integration of study abroad experience into core Liberal Arts curriculum and/or major Language based versus non-language based programs Program policies Conducting research while abroad

28 Study Abroad Review of Senior Capstone Projects To what extent was there direct evidence of the impact of study abroad experience in the content of the capstone project? To what degree did capstone projects based upon study abroad experiences align well with students’ overall academic programs? What was the design and content of capstone projects that included referencing study abroad experiences? To what degree do the criteria for the evaluation of capstone projects address the impact of study abroad experiences?

29 Study Abroad Capstone Review Process allowed faculty to: Learn more about other types of capstone projects Develop methods of evaluation for reviewing student work Reflect and articulate standards/criteria used to evaluate such projects

30 Study Abroad Study Abroad Survey Components Students’ motivations for off-campus study choices Preparation and advising for study away experience Student experiences while away Perceptions of the impact on academic and personal growth Senior academic activities that built upon the study abroad experience Several “free-response” questions

31 Study Abroad Study Abroad Study Abroad Survey Amherst, Bates, Bowdoin, Colby, Smith Common content, multiple methods Sharing.html code facilitated merging of datasets Large sample size (n = 795) to test relationships Compare schools in terms of outcomes, processes that “work”

32 Study Abroad Study Away Reasons - Cluster Analysis Reasons for studying abroad items are the best candidates for a priori or causal variables. If reasons presuppose certain outcomes then a model focusing on them has diagnostic value in practice. Cluster analysis into types to simply interpretation. – A four cluster solution fit the five-school data best

33 Study Abroad Selected Cluster Solution Cluster names are subjective interpretations of item response patterns Cluster characteristics

34 Study Abroad Does Cluster Membership Predict OCS Experiences? Well-Roundeds and Experientialists strong on cultural immersion, language learning experiences; weakest of the four clusters on “adjusting to academic structure” Narrow Academics stand out on “keeping in touch with professors back home” and “course learning”, “part of intellectual community”; weakest on cultural immersion experiences Club Meds weak pretty much everywhere, especially traditional academics – but more cultural immersion than Narrow Academics

35 Study Abroad Does Cluster Membership Predict OCS Outcomes? Well-Roundeds strongest on most outcomes, virtually across the board – “academic integration” as well as “personal growth” and “cross-cultural understanding” outcomes Narrow Academics weakest on “personal growth” and “cross-cultural understanding” – almost as strong as Well-Roundeds in “integration with post-OCS academics” Club Meds bring up the rear virtually across the board, especially on post-OCS academic integration. They do, however, report that that the experience was “rewarding”

36 Study Abroad Variation Across Schools Large dataset, many variables, permitted powerful analysis of school differences while controlling for program differences Accounted for variation in outcomes: – Non-English speaking country – Gender (female) – Natural Science majors weak on “cross-cultural” learning outcomes – Well-rounded Internationalists – School did still account for some variation, after these factors were controlled

37 Course/Capstone Memos – Assessing the Accomplishment of Learning Goals in the Core Curriculum and the Major

38 Capstone Cross-Discipline Discussions Discipline based conference calls to: – Share information regarding the different types and requirements of capstone project/senior theses – Develop questions to be answered – Consider the format of the Capstone Memo

39 Capstone Capstone Memo Guidelines Context Evaluation of Student Work Evaluation of the Course

40 Capstone Differences Across Departments Chemistry English Social Sciences

41 Capstone Discussion Themes Role of the capstone – culminating experience or crowning achievement? Preparation of students prior to completing the capstone Different opportunities for students to demonstrate learning through a culminating experience

42 Therefore, what?

43 What We Learned and Gained Structures That Worked Collaboration: Multiple school involvement enhanced perceived value and importance of project. – Economy of scale – Added value, opportunity for peer comparisons Inquiry-based: Clarifying learning goals within each of the study areas was very much part of the project itself Defined focus: Focus on specific “transitional points”: – Increased faculty interest and involvement – Encouraged faculty to “think outside the department” NEASC sponsorship: Not a funding issue so much as a “signaling” issue.

44 Link to Presentation Materials n.edu/ir/conferences/ neasc2004.shtml

45 Next Proposal Inquiry-Based Collaborative Assessment of Student Learning in Liberal Arts College stay tuned!