Keeping it real: Authenticity in testing 2018 BILC PROFESSIONAL SEMINAR Keeping it real: Authenticity in testing Benjamin C. Pim (M.A.Language Testing) AERONAUTICA MILITARE

AGENDA What is language testing? Why test? Authenticity TLU Motivation Overview of tests Role-plays JTAC Role-play STANAG Role-play Role vs. Play Known knowns…. AERONAUTICA MILITARE

TESTING AERONAUTICA MILITARE

What is language testing? "Language Testing is the practice and study of evaluating the proficiency of an individual in using a particular language effectively." Priscilla Allen, University of Washington. AERONAUTICA MILITARE

Why test? 4 basic assumptions Tasks are relevant to the decisions we want to take. Language produced is not only relevant but can be scored. Direct relationship between the score and the ability of the person. Produce a reductive score that summarizes what the person has done. ‘…the only reason we give tests for assessment is that someone else can make a decision about the person taking the test’ Prof.G.Fulcher 2017 AERONAUTICA MILITARE

Authenticity The extent to which test tasks correspond to language use in a non-test (i.e. target language use) situation. (Longman Dictionary of Language Teaching and Applied Linguistics) Test quality showing correspondence between test task characteristics and the usage of target language in real life situations. (LTS Handbook) AERONAUTICA MILITARE

TLU “a specific setting outside the test itself that requires the test taker to perform language use tasks” Target Language Use TLU,(Bachman and Palmer ) AERONAUTICA MILITARE

Motivation “If the examinees do not approach the testing situation in the expected manner, the results may be prove to be invalid. This might occur if the examinees are insincere, misinformed, or hostile with regard to the test or the testing.” Response Validity (Henning ,1987) AERONAUTICA MILITARE

Overview of tests General English Tests: End of module JFLT ESP Tests: TAE TPT JTAC M.E. ROLE PLAYS AERONAUTICA MILITARE

Role-plays The role-play is used to check whether the examinee can carry out tasks that cannot be elicited by means of a conversational exchange. It also shows how the examinee functions in a real life situation in the target culture. It can serve as a level check or a probe. It is generally introduced midway through the test. (LTS Handbook) TLU AERONAUTICA MILITARE

JTAC Role-play AERONAUTICA MILITARE

STANAG Role-play AERONAUTICA MILITARE

“Something similar happended to me…and I just took the sink apart” Role vs. Play “I’m not a qualified JTAC yet, but I feel more prepared to do the course in September” “Something similar happended to me…and I just took the sink apart” “With all respect, I fail to see the link between this and my future work” “I felt confident because I knew the right words” “I have never experienced anything like this – so to be honest I didn’t know what to say.. It felt a bit like acting” “I felt the ‘talk-on’ tested part of my real job” “It felt real speaking to a CAS pilot” “I felt stupid” AERONAUTICA MILITARE

Role vs. Play (FEWEHANS) “I got lost- it all looked the same” “I didn’t understand what the pilot was saying and what he wanted…he shouted at me” “I was able to find the target, but I am not sure if the pilot did” “I felt totally out of my comfort-zone and I confused the pilot by giving too much detail” AERONAUTICA MILITARE

Role vs. Play (FEWEHANS) “It felt unrealistic” Me: Good-evening…Good morning.. It’s Benjamin Pim here, so sorry for disturbing you at this ungodly hour Co-worker: What can I do for you??? Me: Errrrr…ummm…I am most sorry for this..but I lost my wedding ring down the sink… Co-worker: Sorry to hear that… Me…(embarrassed silence)… could you please come over and help me retrieve it please? Co-worker: Can you call back tomorrow. Me: (deathly silence)…Ok……Sleep well Co-worker: Night “I didn’t really know what to say” “It was so short I felt that it didn’t produce a rateable sample” “I spoke too much…I said things that I would not usually say in that context as I knew I was being rated for language use” AERONAUTICA MILITARE

Known knowns –or at least what we think we know They (our employers) think they know what they need They think we know what they think they need We think we know what they think they need But is what they actually need the same as what they think they need?   AERONAUTICA MILITARE

Thanks for your attention benjamin.pim@gmail.com AERONAUTICA MILITARE