1. EVALUATING A TBLT SPANISH IMMERSION PROGRAM COLLOQUIUM: Evaluating task-based language programs Greg Burwell – BPA – greg.burwell@dhs.govgreg.burwell@dhs.gov Marta Gonzalez-Lloret –UHM – marta@hawaii.edumarta@hawaii.edu Katie Nielson – UMD (CASL) – knielson@casl.umd.eduknielson@casl.umd.edu

2. Overview BackgroundStudent assessmentPilot studyOrdinate studyStudent evaluationDiscussion

3. TBLT at the Border Patrol Academy (BPA)  BPA Training Environment  Spanish at the BPA (before and after)  Students, instructors, and supervisors sought practical training  Developed in 2006-2007  Needs Analysis  Genuine audio/video  Extensive input from actual Border Patrol agents

4. TBLT at the Border Patrol Academy (BPA)  First implemented in December of 2007  First cohort of 15 students graduated in March 2008  As of August 2009, 2,813 students have completed the program  To date, the program has started 180 classes with over 3,000 students  100% pass rate

5. TBLT at the BPA- Curriculum  8-week program  following Long’s (1985, 2005a) stages for designing and implementing a TBLT program 7 target tasks Several target task types Multiple pedagogic tasks

6. TBLT at the BPA - Curriculum Target Task Types: Extract subject Interview subject Determine citizenship status Inspect Vehicle

7. TBLT at the BPA - Curriculum Input (video, audio)Output (guided)Role-play tasks COMPLEXITY + Lab Activities + La Chamba interactive video game

8. TBLT at BPA- Assessment Computer-based language test Performance-based assessment where trainees interact with native Spanish speaking role-players as they complete that module’s target task One-on-one conference with instructor Weekly formative assessment: Critical scenarios with professional role-players Students role-play key course tasks over two days Criterion-referenced scoring Each scenario scored by two instructors End-of-course summative assessment:

9. TBLT at BPA - Assessment

10.  Multiple versions of end-of-module assessments  Questions considered in each iteration:  Are these elements required to complete the task?  Which are critical?  How do we assess language? Global accuracy, fluency, and pronunciation ratings Inclusion of “using appropriate Spanish” with each required task Removal of any mention of language

11. TBLT at BPA - Assessment

14.  Instructor training  Video evaluation  Role-play evaluation  Reliability  Rubric modification  Perceived grade inflation  Language versus task accomplishment  Non-obligatory steps removed  Average grade changed from 94.10% to 95.48%

15. Participants Grammar-based group (GB) (n=20) TBLT group (TBLT) (n=20) Oral narrative based on picture story Hypotheses: TBLT will perform better on measures of fluency, lexical complexity and structural complexity GB will perform better on measures of grammatical accuracy Pilot Study – Oral narration task

16. Pilot study- cont  Compared on:  Fluency [ syllables/second] (Gilabert, 2004)  Lexical complexity [lexical words/words] (Robinson, 1995; Gilabert, 2004) [1 st time use of lexical word/words] Ratio of lexical/functional words (Gilabert, 2004)  Structural complexity [ words/utterance ] (Ortega, 1999)  Grammatical accuracy TLU of noun-modifier agreement (Ortega, 1999) TLU of noun-verb agreement

17. Pilot study- Results - Fluency syllables/second

18. Pilot study- Results – Lexical complexity Lexical words/words1 st use lexical w/wordsRatio of lexical/ functional words

19. Pilot study- Results- Structural complexity #of words/ utterance

20. Pilot study- Results –Grammatical Accuracy TLU of noun-modifier agreementTLU of noun-verb agreement

21. Pilot study - Discussion Very small n-size due to difficulty obtaining samples Significant effect for fluency and structural complexity (.05) Students from TBLT course tend to produce more fluent and complex speech that is less grammatically accurate but contains a greater variety of lexical items.

22. Participants First 256 TBLT students Performance on computerized oral proficiency interview Hypotheses: TBLT will improve students’ overall proficiency TBLT will affect overall proficiency regardless of language abilities at the start of the course Ordinate study

23. Ordinate Study - continued  Ordinate (Versant) Computerized Oral Proficiency Exam  Used at BPA as a placement test  Scored between 20 – 80 Overall Sentence Mastery Vocabulary Fluency Pronunciation  Students took the test before and after the 8-week TBLT program

24. Ordinate Study - Mean Scores

25. Ordinate Study – Change Scores * Overall: Mean: 7.52 SD: 5.42 * Sentence mastery: Mean: 7.55 SD: 8.21 * Vocabulary: Mean: 7.16 SD: 10.99 * Fluency: Mean: 5.86 SD: 7.06 * Pronunciation: Mean 3.53 SD 4.92 All change scores significant at.01

26. Ordinate Study - Correlation

27. Ordinate Study - Discussion Hypothesis #1 Overall proficiency will improve as a result of TBLT Yes: Post-test means improve an average of 7.5 points overall on an unrelated, standardized proficiency test Hypothesis #2 Overall proficiency will improve regardless of starting level Yes: Almost no correlation between pre-test score and change score

28. Students’ evaluations  Electronic (anonymous) surveys  2 Groups: Pre-experienced Current students of the TBLT Spanish program n=41 Experienced Current BA Agents trained with TBLT Spanish program n=14

29. Students’ evaluation (pre-experienced)

30. In general students liked the best…  The work done with the role-players.  The direct application of the class to their job. “Role playing is very helpful. I feel that it helped me learn more spanish (sic) and become more comfortable speaking it.” “I enjoyed the hands on experience the most. Being able to read and write is important when learning a new language, but being able to incorporate it into your daily duties is beneficial.” “I have enjoyed that we get to have a lot of hands on in this program. Also helps a lot for us to be able to apply all that we have learned in basic and build apon (sic) basic.” Students’ evaluation

31. Students’ evaluations  Students suggestions:  More conversational Spanish (not for the job)  More grammar in class  More role-play work with Spanish speakers

32. Agents’ evaluation (experienced)

33. Agents’ evaluations  What they found most useful from the Spanish course was:  Commands and expressions  Knowing the procedures and forms in Spanish  What they would have liked to learn:  More conversation (outside scenarios)  More vocabulary  Grammar and verb conjugation

34. TBLT at BPA Students are all passing Overall student proficiency improves after the TBLT course Student speech is more fluent and complex than in the old course What does this mean? Students find the course practical

35. Difficulty of training instructors properly on methods and principles of TBLT Instructors reluctant to adopt new approach Lack of trained personnel to collect and analyze data Lack of support for follow-up research Difficulties with evaluating TBLT at the BPA

36. THANK YOU GRACIAS MAHALO Questions?