1. Test Standardization: From Design to Concurrent Validation
BILC Professional Seminar
3 October 2017
Tbilisi, Georgia
Ray Clifford

2. Did you know that September 30th was International Translation Day?
That date was selected as International Translation Day, because that is the feast day for St. Jerome, the patron saint of translators.

3. Because language is the most complex of all of the observable human behaviors…
Even professional translators can make mistakes that reveal the complexities of language.

4. Consider this translation:

5. Which is the better translation?
The restrooms are behind you.

6. Consider this translation:
If this is your first visit to Moscow, you are welcome to it.

7. Which is the better translation?
If this is your first visit to Moscow, you are welcome to it.
We especially welcome those who are first-time visitors to Moscow.

8. Consider this translation:
Ladies may have a fit upstairs.

9. Which is the better translation?
Ladies may have a fit upstairs.
Women’s tailoring is located upstairs.

10. Consider this translation:
Cooles and Heates: If you want just condition of warm in your room, please control yourself.

11. Which is the better translation?
Cooles and Heates: If you want just condition of warm in your room, please control yourself.
Cooling and heating your room: If you would like more heat, please adjust the thermostat.

12. Consider this translation:
Ladies are requested not to
have children in the bar.

13. Which is the better translation?
Ladies are requested not to have children in the bar.
Please do not take children
into the bar.

14. Language testing is a professional discipline.
Language is the most complex of observable human behaviors.
Testing language is a complex science.
Language testing is complexity2.
Every test development effort is a research project.
Every research project should be conducted in a precise, disciplined manner.

15. Our goal is to equivalent test results from all member nations’ STANAG 6001 tests.
Experts tell us that to get consistent results, we need to apply aligned Criterion-Referenced testing procedures, with:
Consistent elicitation and scoring criteria when testing Speaking and Writing skills.
Consistent test design and scoring criteria when testing Reading and Listening skills.

16. For the “full story” read:
Luecht, R. (2003). Multistage complexity in language proficiency assessment: A framework for aligning theoretical perspectives, test development, and psychometrics. Foreign Language Annals, 36,
Shrock, S. A. & Coscarelli, W. C. (2007). Criterion-referenced test development: Technical and legal guidelines for corporate training and certification. (3rd.ed.). San Francisco, CA:  John Wiley and Sons.

17. For today, here are the basics:
The tested construct must have clear Task, Context, and Accuracy (TCA) expectations.
And these three elements must be aligned:
The construct to be tested.
The test design and its development.
The scoring process.
If all of these elements are aligned, the tests are legally defensible.
It is this alignment that establishes the provenance of a Criterion-Referenced test.

18. C-R Testing Experts Stress the Need for Alignment
Construct Definition
What is to be tested
Test Design
How it is tested
Test Scoring Process
How the test is scored
Must
align
with
Must
align
with

19. For STANAG 6001 Tests: The Construct The Test Design
(Has a subtest for each level)
The Scoring Process
(yields a separate score for each level)
The Construct
(Has multiple levels)

20. Everyone accepts that STANAG 6001 has multiple levels to be tested.
What:
STANAG 6001
Level 3
Level 2
Level 1

21. For STANAG 6001 Tests: Test Design Construct
(Should have a subtest for each level)
Scoring Process
(Should yield a separate score for each level)
Construct
(Has multiple levels)

22. Not everyone designs their STANAG 6001 tests with STANAG 6001 levels.
What: ≠
How Tested:
How Scored:
STANAG 6001
Non-aligned Test Design
Report a Single Test Score
&
Estimate "cut scores"
Level 3
Hard ?s X 3  |
Uncertainty
Level 2 2
Level 1
Easy ?s 1

23. For STANAG 6001 Tests: Test Design Construct
(Has a subtest for each level)
Scoring Process
(Yields a separate score for each level)
Construct
(Has multiple levels)

24. Report a Single Test Score
Very few nations assign a separate score to each subtest in their STANAG 6001 tests.
What: =
How tested: ≠
How Scored:
STANAG 6001
Aligned Test Design
Report a Single Test Score
&
Estimate "cut scores"
Level 3
Level 3 Test X 3
Uncertainty
Level 2
Level 2 Test 2
Level 1
Level 1 Test 1

25. Pick the Highest Level Passed
To have direct evidence of a C-R test’s validity requires full alignment, including scoring.
What: =
How Tested:
How Scored:
STANAG 6001
Aligned Test Design
Score Each Level
&
Pick the Highest Level Passed
Level 3
Level 3 Test
Level 3 Score
Passed at 3?
Level 2
Level 2 Test
Level 2 Score
Passed at 2?
Level 1
Level 1 Test
Level 1 Score
Pased at 1?

26. Why does not having separate scores lead to inaccurate results?
Language learners do not completely master one proficiency level before they begin learning the skills described at the next higher level.
Usually, learners develop conceptual control or even partial control over the next higher proficiency level by the time they have attained sustained, consistent control over the lower level.

27. STANAG 6001 Levels are like buckets…
The blue arrows indicate the water (ability) level observed at each level for a Level 1 speaker.. 3 2 1
Some notes on language learning:
The buckets may begin filling at the same time.
Some Level 2 skills will develop before Level 1 is mastered and some Level 3 skills will develop before Level 2 is mastered.
But the buckets will still reach their full (or mastery) state sequentially.

28. Why does not having separate scores lead to inaccurate results?
A proficiency rating of a 1, a 2, or a 3 can only be assigned if all of the Task, Condition, and Accuracy expectations associated with that level have been met.
Therefore, it is more important to know how much “water” (ability) is in each bucket, than it is to know how much “total water” there is.
Here is an example.

29. Which learner is more proficient?
Alice received a TOTAL score of 35.
Bob received a TOTAL score of 37.

30. Which learner is more proficient?
Alice received a TOTAL score of 35.
Bob received a TOTAL score of 37.
Before we decide, let’s look at their subtest scores.

31. Example A: Alice’s total score = 35 C-R Proficiency Level = 2 Level 2 (with Random abilities at Level 3)
Level 1
Level 2
Level 3
"Almost all"
17 points,
85%
15 points,
75%
Most
Some
None
3 points,
15%

32. Example B: Bob’s total score = 37 C-R Proficiency Level = 1+ Level 1 (with Developing abilities at Level 2)
Level 1
Level 2
Level 3
"Almost all"
17 points,
85%
Most
 11 points,
55%
Some
9 points,
45%
None

33. Which learner is more proficient?
Alice is more proficient. She received a TOTAL score of 35, but met the Criterion-Referenced ,TCA requirements for both Levels 1 and Level 2 !
Bob received a total score of 37, but he only satisfied the TCA requirements for Level 1 !

34. When Things Align, Everything’s Fine
STANAG 6001 consists of a hierarchy of multiple, independent standards.
Each level in the hierarchy must be tested separately.
Each level in the hierarchy must be scored separately.
(Reporting a total or combined score hides crucial Criterion-Referenced information.)

35. This combined design was not used!
That is why both the BAT design and its scoring process use a level-by-level approach!
Difficult Items
Level 3 Items
Level 2 Items
Level 1 Items
Easy Items
This combined design was not used!

36. The BAT Reading Test Is a Criterion-Referenced Proficiency Test.
Defines the construct with a hierarchy of level-by-level TCA requirements.
Follows those level-by-level criteria in the test design and development process.
Applies the same level-by-level criteria in its scoring process.
Has been empirically validated.
Items clustered in difficulty at each level.
The clusters did not overlap.

37. Criterion-Referenced BAT Validity Statisitics With 3 Levels & 4 testlets of 5 items each/level; n = 680
STANAG 6001 Level
Logit value of Testlet A
Logit value of Testlet B
Logit value of Testlet C
Logit value of Testlet D
Standard Error of the model
in Logits 3
+ 1.8
.04 2
+ 0.3
+ 0.2 1
- 1.5
- 1.6
- 1.7
.06

38. Ascending Difficulty of Levels Testlet difficulties are within +/-
Ascending Difficulty of Levels Testlet difficulties are within +/-.02 logits of each other. Standard Error of Measurement < .06 Vertical distance between clusters > 1 logit

39. How do your Reading and Listening test results compare to the results from the validated BAT?
Were the items in the Level 2 subtest more difficult than those in the Level 1 subtest?
Were the items in the Level 3 subtest more difficult than those in the Level 2 subtest?
Did you assign a separate score for each STANAG Level?
Did you use “floor and ceiling” procedures to determine each person’s proficiency level?

40. The more “yes” answers you gave to these questions, the more likely it is that your results will align with the BAT results.

41. For more information on the empirical validation of the CR approach to the testing of reading proficiency, see:
Clifford, R. & Cox, T. (2013) “Empirical Validation of Reading Proficiency Guidelines”, Foreign Language Annals. Vol. 46, No. 1. Clifford, R. (2016) “A Rational for Criterion-Referenced Proficiency Testing”, Foreign Language Annals. Vol. 49, No. 2.

42. Thank you.