1. Teaching, Learning, and Testing: Finding Congruence
BILC Professional Seminar
Zagreb, Croatia
16 October 2018
Ray Clifford

2. An old proverb says, Time is the best teacher.

3. But it has also been noted that: Time may be the best teacher, but it eventually kills all of its students.

4. A seminal work in Education is Bloom’s Taxonomy of Educational Objectives Note: the author called this book, “The most-quoted, least-read book ever published.” There is a revised version titled, A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, Abridged Edition. Editors, Lorin W. Anderson, et al. Longman, New York. 2001

5. The revised version divides the original taxonomy into two dimensions: 1. Ways of thinking and 2. Types of knowledge .
This revision is summarized in the chart on the next slide.
Thinking.
The columns represent each of the six levels of the Cognitive Process Dimension—ranging from lower-order thinking skills at the left to higher-order thinking skills at the right.
Knowing.
The rows represent the Knowledge Dimension—ranging from concrete knowledge at the bottom through abstract knowledge at the top.

6. Bloom’s Revised Taxonomy (With added highlighting and annotation)
Common Combinations
Cognitive Dimension
Remember
Understand
Apply
Analyze
Evaluate
Create
Knowledge Dimension
Metacognitive
Procedural
Conceptual
Factual

7. Revised Bloom’s Taxonomy: Greater Accuracy, but Greater Complexity
4 levels of knowledge.
6 levels of cognition.
4 x 6 equals 24 interactions (or combinations).
Is this increased accuracy useful?
How much complexity can humans handle?

8. “How Many Variables Can Humans Process. ” Graeme S. Halford, et al
“How Many Variables Can Humans Process?” Graeme S. Halford, et al. Psychology Science, Vol 16, No. 1, pp , American Psychology Society, Page 70.
n=30 / Three quotes from the research findings:
“As the order [the number] of the interaction[s] increases, the number of variables increases.”
“Results showed a significant decline in accuracy and speed of [problem] resolution from three-way to four-way interactions.”
“Furthermore, performance on a five-way interaction was at [the] chance level.”

9. (Trained individuals)
“How Many Variables Can Humans Process?” Graeme S. Halford, et al. Psychology Science, Vol 16, No. 1, pp , American Psychology Society, 2005.
n=30
(Trained individuals)
Problem Complexity
2-way
3-way
4-way
5-way*
2 Problems Right 30 26 13 NA
1 Problem Right 4 12
0 Problems Right 10
My interpretation:
Good OK
Poor
Terrible
* One problem, n = 22. Chance responses, consistent with results extrapolated from the other study.

10. Can Bloom’s revised taxonomy be simplified to make it more useable?
The Knowledge and Cognitive Dimensions can be (re)combined into three levels:
Direct Application of factual knowledge
Near transfer of conceptual knowledge.
Far transfer of abstract knowledge.
These three levels can be applied to learning and to testing activities.

11. The expected level of learning: (3 Levels of Learning Outcomes)
Direct Application
Near Transfer
Far Transfer 11

12. The 1st Level of Learning Outcomes
With direct application learning, students…
Memorize and practice specific responses.
Focus on the content of a specific course, textbook, or curriculum.
Learn only what is taught. 12

13. The 2nd Level of Learning Outcomes
With near transfer learning, students…
Go beyond rote responses and use rehearsed and semi-rehearsed responses.
Focus on a specific set of tasks and communicative settings and respond within those domains.
Apply what they learn within a range of familiar, predictable settings. 13

14. The 3rd Level of Learning Outcomes
With far transfer learning, students…
Develop the ability to transfer what is learned from one context to another.
Acquire the knowledge and skills needed to respond spontaneously to new, unknown, or unpredictable situations.
Learn how to continue learning and to become independent learners. 14

15. The testing method used: (3 Types of Tests)
Achievement
Performance
Proficiency 15

16. Achievement tests measure:
The 1st Type of Test
Achievement tests measure:
Practiced, memorized responses.
Acquisition of what was taught.
The content of a specific textbook or curriculum. 16

17. Performance tests measure:
The 2nd Type of Test
Performance tests measure:
Rehearsed and semi-rehearsed responses.
One’s ability to respond in constrained, familiar, and predictable settings.
The transfer of learning to similar situations. 17

18. Proficiency tests measure:
The 3rd Type of Test
Proficiency tests measure:
Whether skills are transferable to new tasks.
One’s spontaneous, unrehearsed abilities.
The general ability to accomplish tasks across a wide variety of real-world settings. 18

19. Advanced Proficiency Requires Far Transfer Learning
A By-Level Proficiency Summary with Text Types
(Green = Far Transfer, Blue = Near Transfer, Red = Direct Application)
NATO LEVEL
FUNCTION/TASKS
CONTEXT/TOPICS
ACCURACY
All expected of an educated NS [Books]
Accepted as a well-educated NS
All subjects 5
Tailor language, counsel, motivate, persuade, negotiate [Chapters]
Wide range of professional needs
Extensive, precise, and appropriate 4
Support opinions, hypothesize, explain, deal with unfamiliar topics [Multiple pages]
Practical, abstract, special interests
Errors never interfere with communication & rarely disturb 3 2
Narrate, describe, give directions [Multiple paragraphs]
Concrete, real-world, factual
Intelligible even if not used to dealing with non-NS
Q & A, create with the language [Multiple sentences]
Intelligible with effort or practice 1
Everyday survival
Random
Unintelligible
Memorized [Words and Phrases] 19

20. A Personal Observation

21. Tests Define Instructional Expectations
Students (especially adults learners) don’t want to waste their time studying what is not going to be “needed”.
For students (and often commanders, teachers, and administrators); the tests used and not a course’s stated learning objectives define what is “needed”.
Therefore, tests can have a negative or a positive impact on learning.

22. “Washback” Effects of Testing
Testing has a negative impact when:
Educational goals are reduced to those that are most easily measured.
Testing procedures do not reflect course goals, for instance…
Giving multiple choice tests in speaking classes.
Using grammar tests as a measure of general proficiency.

23. “Washback” Effects of Testing
Testing has a positive impact when:
Tests reinforce course objectives.
Tests act as change agents for improving teaching and learning.
The test results are useful for students, teachers, and administrators.

24. Aligning Teaching and Testing Promotes Learning
Direct Application <=> Achievement
Memorized responses using the content of a specific textbook or curriculum.
Near Transfer <=> Performance
Rehearsed ability to communicate in specific, familiar settings.
Far Transfer <=> Proficiency
Unrehearsed general ability to accomplish real-world communication tasks across a wide range of topics and settings. 24

25. When teaching and testing are not aligned, learning suffers.
Direct Application Teaching
+ Proficiency Testing
=> Learners will fail the tests.
Learners won’t be prepared for the tests.
Motivation will be reduced.
Far Transfer Teaching
+ Achievement Testing
=> You’ll get Direct Application learning.
Students will adjust their learning down to the level of the tests. 25

26. When teaching and testing are not aligned, learning suffers.
Direct Application Teaching
+ Proficiency Testing
=> Learners will fail the tests.
Far Transfer Teaching
+ Achievement Testing
=> You’ll get Direct Application learning.
Note # 1: The type of test used can limit the level of students’ learning. 26

27. Curriculum, Instructional, and Testing Constraints Can Also Reduce the Scope of Learning…
High academic goals are set and learner outcomes are defined. 1.
Developers include only examples of the most frequently occurring or important goals in a textbook. 2.
Teachers present as much of the textbook as time allows. 3.
Students are tested on a sample of items drawn from the textbook. 4.
Instructional Goals and Learning Outcomes
Textbook
Teaching
Test

28. Curriculum, Instructional, and Testing Constraints Can Reduce the Scope of Learning…
High academic goals are set and learner outcomes are defined. 1.
Developers include only examples of the most frequently occurring or important goals in a textbook. 2.
Teachers present as much of the textbook as time allows. 3.
Students are tested on a sample of items drawn from the textbook. 4.
Instructional Goals and Learning Outcomes
Textbook
Teaching
Test
Note # 2: The reduced content of the tests used can limit the breadth of students’ learning.

29. An Alternative Teaching and Testing Model
Course developers sample from the real-world domain areas to create a textbook.
2a.
Textbook
Set instructional goals and define expected learner outcomes. 1.
Real-world Instructional Domains: cognitive understanding, psychomotor skills, and affective insights.
Teachers adapt text materials to learners’ abilities, diagnose learning difficulties, adjust activities and add supplemental materials to help students apply new knowledge and skills in constrained achievement and performance areas, and then in real-world proficiency settings. 3.
Teacher
Test developers use an independent sample of the real-world domain areas to create proficiency tests that are not based on the textbook.
2b.
Students
Students practice, expand, and then demonstrate their unrehearsed extemporaneous proficiency across a broad range of real-world settings that are not in the textbook. 4.
Test

30. Simply stated, “good” teaching (reinforced by “good” testing) can expand the scope of learning.
High academic goals are set and learner outcomes are defined. 1.
Developers include only examples of the most frequently occurring or important goals in a textbook. 2.
Teachers present as much of the textbook as time allows. 3.
Students are tested on a sample of items drawn from the textbook. 4.
Instructional Goals and Learning Outcomes
(STANAG
6001?)
Textbook
Teaching
Test

31. So, what constitutes “good” teaching and testing?
Making your teaching/learning activities match the level of your desired learner outcomes.
Using the kind of the tests that match the level of those learning outcomes.
Achievement tests for direct application objectives.
Performance tests for near transfer objectives.
Proficiency tests for far transfer objectives.

32. If those suggestions for “good” teaching are followed, a teaching and testing model will emerge that:
Will not be based on successively derived, reduced subsets of the original objectives.
Will maintain the students’ focus on the course’s true learning objectives.
Will reinforce the teacher’s role as a “facilitator” of learning, rather than as a “presenter” of information.

33. Keep it simple! 3 x 3 x 3 = 27!!! Desired Learning Outcomes
Doesn’t work. The mind can only handle 3 interactions at a time.
Desired Learning Outcomes
a. Direct application
b. Near transfer
c. Far transfer
Keep it simple!
Work with a maximum of three variables.
Reduce complexity
by aligning levels;
i.e. use all “a”, all “b”, or all “c” levels.
Teaching Activities
a. Presentation
b. Application
c. Problem solving
Testing
a. Achievement
b. Performance
c. Proficiency

34. And once again… If things align, everything’s fine! Thank you.