1. College of Business Administration Assessing Inter- Cultural Learning: Direct Measures and Beyond Kathleen A. Krentler San Diego State University

2. College of Business Administration So what’s this assessment stuff? It’s supposed to be good for me but I’m not sure I buy it! ACCOUNTABILITY! Accrediting Agencies want it Universities want it Parents want it Legislators want it Taxpayers want it And just possibly...... Sincere interest that our students learn what we want them to learn.

3. College of Business Administration “I assess my students! I give grades!!”

4. College of Business Administration Why grades don’t equate to learning assessment... Nigel Mid-Term Exam90% Final Exam93% Presentation65% Participation50% TOTAL74.5% Klaus Mid-Term Exam50% Final Exam55% Presentation98% Participation95% TOTAL74.5% COURSE COMPONENTS Mid-Term Exam 25% Final Exam 25% Presentation 25% Participation 25%

5. College of Business Administration The Programmatic Assessment Process What do we want our students to know? How will we know if they know it? Do they know it? If not, how do we fix the problem?

6. College of Business Administration Student Learning Outcomes of the CROCUS Project 1.Gain knowledge of other cultures 2.Use knowledge, diverse cultural frames of reference, and alternate perspectives to think critically and solve problems 3.Become conscious of one’s own cultural perspective. 4.Accept cultural differences and tolerate cultural ambiguity. 5.Improve target language competency in speaking, listening, reading, and writing.

7. College of Business Administration How will we know if they know it? I.E., How could we determine whether these Student Learning Outcomes (SLOs) have been met? We could Ask... Good Idea? So Joe, did you gain knowledge of German culture while you were at University of Mannheim?

8. College of Business Administration Indirect Measures Acquires evidence about how students feel or what they believe about their learning. E.G. HELM “I know the arts and crafts of other cultures.” “I know the cultural values and religious beliefs of other cultures.” “I know the legal and economic systems of other cultures.”

9. College of Business Administration Is it wrong to measure SLOs this way? Is it sufficient to measure SLOs this way? Using Indirect Measures to assess learning

10. College of Business Administration Direct Measures Acquires evidence about student learning and the learning environment through demonstration. Calls for Assurance of Learning through assessment typically require direct measures. So Nigel, describe the legal and economic system of the U.S.

11. College of Business Administration A word about SLO language... Consider the SLO, “Gain knowledge of other cultures.” Is there a reliable method to be assured that someone has “gained knowledge”? Is it possible that Nigel could have gained knowledge but still was unable to describe that knowledge? SLOs should use “action verbs” that can be measured: “Describe the legal and economic systems of other cultures” rather than “Gain knowledge of the legal and economic systems of other cultures.”

12. College of Business Administration Student Learning Outcomes of the CROCUS Project 1.Gain knowledge of other cultures. 1.Describe other cultures. 2.Use knowledge, diverse cultural frames of reference, and alternate perspectives to think critically and solve problems. 3.Become conscious of one’s own cultural perspective. 3.Articulate one’s own cultural perspective. 4.Accept cultural differences and tolerate cultural ambiguity. 4.Acknowledge cultural differences and discuss cultural ambiguity. 5.Improve target language competency in speaking, listening, reading, and writing.

13. College of Business Administration Assessing student learning in the Crocus Project What SDSU has done... 1.Used assignments in the “Crossing the Bridge” Course 2.Completed interviews with students

14. College of Business Administration Assignments in the Course Students were asked to complete written assignments that demonstrated their accomplishment of the SLOs. SLO: “Gain knowledge of other cultures.” Assignment: “Write a two-page paper (500 words or more) about your host country. Accumulate facts about the country’s history, political situation, economy, religion, international relations etc. Ensure that you include current events in your paper (elections, economic ties etc.).”

15. College of Business Administration Assignments in the Course Rubric developed to measure whether assignment demonstrated the SLOs Set benchmark – 75% of the students should be able to achieve a Moderate level 2 independent raters read each assignment and score it using the rubric. Scores reviewed for inter-rater reliability Determination made as to whether the benchmark was achieved.

16. College of Business Administration Student Interviews Students completed in-depth interviews upon return from Study Abroad Is this a direct measure? Depends on what they are asked in interview! INDIRECT: “Did you gain a knowledge of the culture of the country you studied in?” DIRECT: “Describe the culture of the country you studied in.”

17. College of Business Administration Student Interviews Same rubric used to measure whether responses demonstrated the SLOs Set benchmark – 75% of the students should be able to achieve a Moderate level 2 independent raters listen to responses and score them using the rubric. Scores reviewed for inter-rater reliability Determination made as to whether the benchmark was achieved

18. College of Business Administration Do They Know It? Interesting Outcome – Students scored higher when assessed via interviews than when assessed via course assignments. CHALLENGE: Achieving cross-measure reliability.

19. College of Business Administration If they don’t know it, how do we fix the problem? Only 65% of students showed evidence of having gained cultural knowledge, below the 75% acceptable benchmark. Programmatic Assessment asks not “what’s wrong with these students?” but rather, “how can we strengthen the program to produce a better result?”

20. College of Business Administration CLOSING THE LOOP 1.Did the program deliver the content needed to allow students to accomplish the SLOs? Curriculum mapping can help to answer this question. 2.How does the program need to change to improve things?