1. XIV Peru-TESOL Sharing Teaching and Learning Experiences in a Global World Iquitos August 2, 2006 M. Karen Jogan, Ph.D. kjogan@alb.edu

2. Catching that elusive butterfly: assessing speaking in five easy steps

3. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 3 Today we will… Outline the process to capture oral competency of EFL students Define oral proficiency – identify proficiency levels in instructional program Design assessable speaking tasks for each level Identify or create rubrics used to assess speaking tasks Align assessment tool to a program outcome Prepare to implement an oral assessment plan

4. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 4 What are the steps in the process? 1.Determine minimal outcomes, using proficiency guidelines 2.Include speaking tasks for assessment in the classroom 3.Adapt or develop assessment tools to assess speaking 4.Promote contextualized speaking activities 5.Assess the speaking task, incorporate into overall assesment framework

5. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 5 1. Determine minimal student outcomes

6. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 6 What is proficiency? Novice Intermediate Advanced Superior

7. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 7 Specific linguistic functions

8. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 8 Superior Advanced Intermediate NoviceTell time, give one´s name, describe weather, list objects

9. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 9 Superior Advanced Intermediate Ask for directions, describe preferences, ask and answer questions, order a meal NoviceTell time, give one´s name, describe weather, list objects

10. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 10 Superior Advanced Narrate a story in the past, compare and contrast Intermediate Ask for directions, describe preferences, ask and answer questions, order a meal NoviceTell time, give one´s name, describe weather, list objects

11. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 11 SuperiorSupport opinion, hypothesize, narrate in detail, abtract topics Advanced Narrate a story in the past, compare and contrast Intermediate Ask for directions, describe preferences, ask and answer questions, order a meal NoviceTell time, give one´s name, describe weather, list objects

12. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 12 Discourse – from limited to extended

13. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 13 Superior Discourse level Multiple paragraphs Advanced Multiple sentences paragraph level Handle complex situation and transaction Sentences create with language; handle simple situation/transaction Novice Short memorized phrases lists

14. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 14 Time to think! In which level of EFL instruction are your students able to perform the following?  Complain  Summarize  Narrate a past event  Greet a friend  Compare  Ask questions to elicit personal information

15. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 15 To consider how well students perform language functions, which (assessment) activities are appropriate?  What does the student DO to demonstrate competency???

16. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 16 Describe Ask and answer questions Narrate in past

17. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 17 Describe Ask and answer questions Narrate in past Oral presentations Interview your partner Story retelling, summarizing Picture-cued descriptions

18. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 18 Ask/provide information Agree/disagree Express opinions Support opinions Persuade Complain

19. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 19 Ask/provide information Agree/disagree Express opinions Support opinions Persuade Complain Role Plays Debates Information gap

20. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 20 Get the gist Get the main idea Get specific details Summarize

21. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 21 Get the gist Get the main idea Get specific details Summarize Tell/discuss what you heard  audio file  film  news report  native speaker… Tell about what you read in  newspaper report  story, etc.

22. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 22 Summarize Describe Synthesize Analyze Summarize/describe/ anaylze  video  program  Article, etc.

23. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 23 2. Include speaking tasks for assessment in the classroom

24. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 24 List all speaking activities which are part of your course Designate or design specific speaking tasks for assessment Define all elements of the “product” which you expect your students to produce Consider which criteria you will use for assessment

25. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 25 Interpersonal mode  Roleplays among students  Interviews between students  Oral monologues  Situation cards (simulations)  Performing a play Presentational mode  Narration or description (oral presentation)  Poster session  Oral presentation with power point

26. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 26 Time to talk! Describe an oral activity which your students have performed successfully. Why do you think that activity was successful? Describe an oral activity that your students did not perform well. Why do you think that the activity was unsuccessful?

27. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 27 Rules to remember Design the task carefully. If you can´t define it, you can´t assess it.

28. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 28 Designing a speaking activity Determine level Select function or language task  What is the grammar students need to perform this task? Select a topic or situation  Vocabulary? Define a time frame (present/past/future) Define the elements of the context  Who, what, where, when??? What aspects of culture must be included as part of the activity?

29. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 29 It´s your turn! Design a speaking activity 1.Level 2.Function 3.Grammar 4.Context – who, what, where, when 5.Content – topic or situation 6.Culture

30. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 30 3. Adapt or develop assessment tools to assess speaking

31. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 31 Recognize that a variety of assessment tools can be used Survey avaliable assessment instruments Examine criteria; recognize that criteria can differ according to instructional level Collaborate to develop appropriate assessment tools Match assessment tool to appropriate activity and instructional level

32. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 32 4. Promote contextualized speaking activities

33. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 33 Move from skill building to skill using in the instructional program  What can students DO???? Move from meaningful to communicative to situational activities Provide a framework for students to structure a conversational exchange Monitor the process

34. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 34 Sample role play buildup Context:  You have just met a new student from the U.S. who is attending your university. You want to find out about this person and what you both have in common. After you establish common ground, invite your new friend to join you for an activity in which you are both interested.

35. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 35 Prepare your students for success!

36. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 36 Sample role play buildup: instructions to students Preliminary step  Key words and phrases I will use  Question words  Nouns  Verbs

37. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 37 Sample role play buildup: instructions to students 1.Write down the question words you will need to ask 2.Write down possible answers to these questions 3.Interview someone in the class 4.Write down the answers you hear 5.Let your partner interview you

38. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 38 Sample role play buildup: instructions to students 6.Decide with your partner which questions are the most appropriate 7.Decide with your partner which role each of you will play 8.Practice acting out with your partner

39. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 39 And the final step Now, act out your role play.  Teacher uses assessment tool to evaluate the student product

40. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 40 Excedes requirements Meets requirements Needs improvement Content Original and on task: excellent use of new vocabulary Content is complete Incomplete content or vocabulary Accuracy Grammar usage is exceptional Grammar usage is appropriate Grammar needs improvement Fluency Speaks without hesitations Some hesitations Halting and many hesitations

41. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 41 5. Assess the speaking task and incorporate into the overall assessment framework

42. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 42 Address the goals of your program Make known to students the criteria for assessment and the specific tasks to be assessed prior to the activity Apply an appropriate assessment instrument to the oral task Review outcomes

43. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 43 And finally… Evaluate the effectiveness of your assessment tool Consider how to improve your course or program, based on results from assesment

44. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 44 Programatic global assessment Oral interview Oral presentation – final project Essay – final product Listen or read and summarize/evaluate/analyze orally or in writing Portfolio = collections, selections, and reflections Standardized test or mock test Integrative tests tap all skills!

45. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 45 How can we catch that elusive butterfly? 1.Build a team 2.Determine goals of your assessment program. 3.Match course objective to the test. 4.Test like you teach, teach like you test! 5.Review sample assessment tools. 6.Adapt tools to meet your program’s needs

46. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 46 7.Use the assessment tool with your class. 8.Critique the tool with your colleagues. Are you consistent with your colleagues in its application? Are there ways that the tool or the activity could be improved? Should more “buildup” opportunities be provided to your students? 9.Collaborate and continue to share, review, adapt, & critique with colleagues.

47. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 47

48. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 48 Time to work…

49. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 49 Your Assignment… Define a student outcome for oral production Clearly define the task that the student will complete to demonstrate competency Identify a rubric for use in evaluating the task. Consult online resources that follow. Match rubric to task

50. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 50 Online references Rubistar  Adapt rubrics to meet the criteria you determine to evaluate student products http://rubistar.4teachers.org/index.php Rubric Bank – Chicago City Schools  Review grading guidelines for speaking, writing activities http://intranet.cps.k12.il.us/Assessments/Ideas_and_Rubri cs/ideas_and_rubrics.html http://intranet.cps.k12.il.us/Assessments/Ideas_and_Rubri cs/ideas_and_rubrics.html Landmark  Create a rubric from scratch http://landmark- project.com/classweb/tools/rubric_builder.php3 http://landmark- project.com/classweb/tools/rubric_builder.php3

51. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 51 University of Alaska sample assessment instruments  Scoring guides http://www.eed.state.ak.us/tls/frameworks/wrldlang/w linstr3.html http://www.eed.state.ak.us/tls/frameworks/wrldlang/w linstr3.html Kenyon County Schools  Assesment tools and links http://www.kenton.k12.ky.us/tr/rubrics.html

52. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 52 Discovery School Assessment tools  Comprehensive collection of assessment tools http://school.discovery.com/schrockguide/assess.htm l http://school.discovery.com/schrockguide/assess.htm l Fairfax County Virginia Public Schools  Performance Assessment for Language Students http://www.fcps.k12.va.us/DIS/OHSICS/forlang/PALS /rubrics/ http://www.fcps.k12.va.us/DIS/OHSICS/forlang/PALS /rubrics/

53. M. Karen Jogan, Ph.D. kjogan@alb.edu Peru TESOL 2006 53 References Bachman, Lyle F. 1990. Fundamental considerations in language testing. New YOrk: Oxford University Press Oller, John W. 1980. Language tests at school. New York: Longman O’Malley, J. M. and L. Valdez Pierce (1996). Authentic Assessment in English Language Learners. New York; Addison Wesley.