1. Keeping it real: Authenticity in testing
2018 BILC PROFESSIONAL SEMINAR
Keeping it real: Authenticity in testing
Benjamin C. Pim
(M.A.Language Testing)
AERONAUTICA MILITARE

2. AGENDA What is language testing? Why test? Authenticity TLU Motivation
Overview of tests
Role-plays
JTAC Role-play
STANAG Role-play
Role vs. Play
Known knowns….
AERONAUTICA MILITARE

3. TESTING
AERONAUTICA MILITARE

4. What is language testing?
"Language Testing is the practice and study of evaluating the proficiency of an individual in using a particular language effectively." Priscilla Allen, University of Washington.
AERONAUTICA MILITARE

5. Why test? 4 basic assumptions
Tasks are relevant to the decisions we want to take.
Language produced is not only relevant but can be scored.
Direct relationship between the score and the ability of the person.
Produce a reductive score that summarizes what the person has done.
‘…the only reason we give tests for assessment is that someone else can make a decision about the person taking the test’ Prof.G.Fulcher 2017
AERONAUTICA MILITARE

6. Authenticity
The extent to which test tasks correspond to language use in a non-test (i.e. target language use) situation.
(Longman Dictionary of Language Teaching and Applied Linguistics)
Test quality showing correspondence between test task characteristics and the usage of target language in real life situations.
(LTS Handbook)
AERONAUTICA MILITARE

7. TLU
“a specific setting outside the test itself that requires the test taker to perform language use tasks”
Target Language Use TLU,(Bachman and Palmer )
AERONAUTICA MILITARE

8. Motivation
“If the examinees do not approach the testing situation in the expected manner, the results may be prove to be invalid. This might occur if the examinees are insincere, misinformed, or hostile with regard to the test or the testing.”
Response Validity (Henning ,1987)
AERONAUTICA MILITARE

9. Overview of tests General English Tests: End of module JFLT ESP Tests:
TAE
TPT
JTAC
M.E.
ROLE PLAYS
AERONAUTICA MILITARE

10. Role-plays
The role-play is used to check whether the examinee can carry out tasks that cannot be elicited by means of a conversational exchange. It also shows how the examinee functions in a real life situation in the target culture. It can serve as a level check or a probe. It is generally introduced midway through the test.
(LTS Handbook)
TLU
AERONAUTICA MILITARE

11. JTAC Role-play
AERONAUTICA MILITARE

12. STANAG Role-play
AERONAUTICA MILITARE

13. “Something similar happended to me…and I just took the sink apart”
Role vs. Play
“I’m not a qualified JTAC yet, but I feel more prepared to do the course in September”
“Something similar happended to me…and I just took the sink apart”
“With all respect, I fail to see the link between this and my future work”
“I felt confident because I knew the right words”
“I have never experienced anything like this – so to be honest I didn’t know what to say.. It felt a bit like acting”
“I felt the ‘talk-on’ tested part of my real job”
“It felt real speaking to a CAS pilot”
“I felt stupid”
AERONAUTICA MILITARE

14. Role vs. Play (FEWEHANS)
“I got lost- it all looked the same”
“I didn’t understand what the pilot was saying and what he wanted…he shouted at me”
“I was able to find the target, but I am not sure if the pilot did”
“I felt totally out of my comfort-zone and I confused the pilot by giving too much detail”
AERONAUTICA MILITARE

15. Role vs. Play (FEWEHANS)
“It felt unrealistic”
Me: Good-evening…Good morning.. It’s Benjamin Pim here, so sorry for disturbing you at this ungodly hour
Co-worker: What can I do for you???
Me: Errrrr…ummm…I am most sorry for this..but I lost my wedding ring down the sink…
Co-worker: Sorry to hear that…
Me…(embarrassed silence)… could you please come over and help me retrieve it please?
Co-worker: Can you call back tomorrow.
Me: (deathly silence)…Ok……Sleep well
Co-worker: Night
“I didn’t really know what to say”
“It was so short I felt that it didn’t produce a rateable sample”
“I spoke too much…I said things that I would not usually say in that context as I knew I was being rated for language use”
AERONAUTICA MILITARE

16. Known knowns –or at least what we think we know
They (our employers) think they know what they need
They think we know what they think they need
We think we know what they think they need
But is what they actually need the same as what they think they need?
AERONAUTICA MILITARE

17. Thanks for your attention
AERONAUTICA MILITARE