1. Improving and Assessing Instructional Effectiveness
Theodore Frick
Department of Instructional Systems Technology
School of Education
Indiana University Bloomington
Invited Lecture to IUB Medical Sciences Faculty
October 14, 2002

2. Improving and Assessing Instructional Effectiveness -- Ted Frick
Overview
Confounding of instructional effectiveness by other factors
Types of knowledge about education
Type 2, 3 and 4 examples
Recommendations
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

3. The Logic of Effectiveness
Before instruction
After instruction
Conclusion
1. Student nonmastery
Student mastery
Instruction appears to be effective enough, but …
2. Student nonmastery
Student nonmastery
Instruction apparently was not effective enough
3. Student mastery
Can't tell if instruction is effective, since student had achieved the goal before instruction began
4. Student mastery
??? Something wrong with the content - factual errors, incorrect procedures, bad models ??? Bad luck ???
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

4. Improving and Assessing Instructional Effectiveness -- Ted Frick
Confounding Factors
Learning occurs without intentional instruction.
Learning occurs despite poor instruction.
We’re not the only teachers a student has.
Results of instruction may occur LONG afterwards.
Bandura (1977): Social Learning Theory: We learn by imitation – others model behavior and we imitate it. No intentional instruction. E.g., child observes dad hit mom; child hits sibling.
Especially if students are intelligent and highly motivated
Obvious? Note: teachers can be anyone who intentionally tries to guide learning of another – e.g., a peer, a parent, Sesame Street.
We’re not always around when the big effects of our teaching occur. E.g., YoungHwan Kim returns 12 years later to show me the book and Web-based testing system in Korea he has developed on computer adaptive testing that he learned from me as a student in our doctoral program.
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

5. Conflation of Research Methods and Outcomes
In addition to the confounding problem in determining effectiveness of instruction, there has been a lot of debate in education about inquiry methods – e.g., quantitative vs. qualitative.
Continuing problem of bridging research and practice in education
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

6. Improving and Assessing Instructional Effectiveness -- Ted Frick
Consider this Debate
Imagine for a moment three carpenters arguing about which tool is best.  
Quanta: "Hammers and nails are clearly superior."
Qualia: "I disagree.  Screwdrivers and screws are much more effective."
Performa: "You're both wrong.  Saws are best for cutting wood."
Quanta: "Who said anything about cutting wood?  I thought we were talking about fastening wood together."
Qualia: "Right. Who needs saws?"
Performa: "I do. I need to cut this board in half."
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

7. Improving and Assessing Instructional Effectiveness -- Ted Frick
Premise
The kind of knowledge about education that we create through disciplined inquiry determines what research methods are appropriate and useful.
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

8. Types of Educology: Knowledge about Education
Methodology of Theory Building (Elizabeth Steiner, 1988) (link)
Non-axiological knowledge – what is
Axiological knowledge – pertaining to values:
Instrumental value: good for – what is effective
Intrinsic value: good in itself – what is worthwhile
The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles (Lee Cronbach, et al., 1972)
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

9. Six Types of Knowledge (Frick)
Purpose of Inquiry
Scope of Knowledge
Unique
Generalizable
What is? 1 2
What is effective? 3 4
What is worthwhile? 5 6
See Frick: R690 Syllabus for further details
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

10. Six Types of Knowledge (Frick)
Purpose of Inquiry
Scope of Knowledge
Unique
Generalizable
What is? 1 2
What is effective? 3 4
What is worthwhile? 5 6
Example: Type 1 Knowledge Claim
‘Elizabeth Steiner is an educational philosopher.’
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

11. Six Types of Knowledge (Frick)
Purpose of Inquiry
Scope of Knowledge
Unique
Generalizable
What is? 1 2
What is effective? 3 4
What is worthwhile? 5 6
Example: Type 2 Knowledge Claim ‘Students who spend more time engaged successfully in arithmetic and reading tasks score more highly on achievement tests in those content areas.’ (Fisher, et al., 1976)
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

12. Six Types of Knowledge (Frick)
Purpose of Inquiry
Scope of Knowledge
Unique
Generalizable
What is? 1 2
What is effective? 3 4
What is worthwhile? 5 6
Example: Type 3 Knowledge Claim ‘The DISTAR program for teaching arithmetic and reading to elementary students works.’
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

13. Six Types of Knowledge (Frick)
Purpose of Inquiry
Scope of Knowledge
Unique
Generalizable
What is? 1 2
What is effective? 3 4
What is worthwhile? 5 6
Example: Type 4 Knowledge Claim ‘Instruction is most effective when it is problem-based, activates prior learner knowledge, demonstrates what is to be learned, provides opportunities for guided practice, and encourages integration with everyday life.’ (Merrill, 2001)
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

14. Six Types of Knowledge (Frick)
Purpose of Inquiry
Scope of Knowledge
Unique
Generalizable
What is? 1 2
What is effective? 3 4
What is worthwhile? 5 6
Example: Type 5 Knowledge Claim
‘The corporal punishment policy in Houston schools is a bad policy.’
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

15. Six Types of Knowledge (Frick)
Purpose of Inquiry
Scope of Knowledge
Unique
Generalizable
What is? 1 2
What is effective? 3 4
What is worthwhile? 5 6
Example: Type 6 Knowledge Claim
‘Teachers and students should respect each other.’
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

16. Do not confuse ends and means
The kind of knowledge about education that we create through disciplined inquiry determines what research methods are appropriate and useful.
Outcome of research is knowledge.
Inquiry methods are a means to that end.
Criteria for evaluating adequacy of research methodology are NOT the same for each knowledge type – e.g.,
criteria for statistical inference to a broad population do not apply to Types 1, 3, and 5;
effectiveness not of concern in Types 1, 2, 5, and 6, but main concern in 3 and 4.
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

17. Improving and Assessing Instructional Effectiveness -- Ted Frick
For today…
Focus on a few examples of Types 2, 3 and 4 knowledge of education
Type 2: Academic learning time research
Type 3 outcomes: Program/product evaluation – Kirkpatrick’s 4 levels (1959)
Type 4 process: Effective Web Instruction – Frick & Boling (2002)
Type 4: Principles of 5-Star Instruction – David Merrill (2001)
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

18. Six Types of Knowledge (Frick)
Purpose of Inquiry
Scope of Knowledge
Unique
Generalizable
What is? 1 2
What is effective? 3 4
What is worthwhile? 5 6
Type 2 Example: Relationship of academic learning time and student achievement
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

19. Type 2: Academic Learning Time
ALT is positively correlated with academic achievement in the same content areas (Fisher, et al., 1976; Rieth & Frick, 1983; Berliner; 1985).
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

20. Six Types of Knowledge (Frick)
Purpose of Inquiry
Scope of Knowledge
Unique
Generalizable
What is? 1 2
What is effective? 3 4
What is worthwhile? 5 6
Type 3: Evaluation of a particular instructional program or product
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

21. Type 3: Program/Product Evaluation – Kirkpatrick’s Levels
Techniques for Evaluating Training Programs -- Donald Kirkpatrick (1959)
Reaction (satisfaction)
Learning (achievement)
Behavior (transfer to real context)
Results (impact on organization/context)
More on Kirkpatrick’s levels from Encyclopedia of Educational Technology
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

22. Six Types of Knowledge (Frick)
Purpose of Inquiry
Scope of Knowledge
Unique
Generalizable
What is? 1 2
What is effective? 3 4
What is worthwhile? 5 6
Type 4 Example: how to create effective instructional products – design theory
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

23. Improving and Assessing Instructional Effectiveness -- Ted Frick
Type 4: Effective Web Instruction: An Inquiry-Based Process – Frick & Boling
More
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

24. Type 4: First Principles of Instruction – David Merrill
Integration
Activation
Problem
Application
Demonstration
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

25. Type 4: First Principles of Instruction (cont’d)
“Learning is facilitated when:
Learners are engaged in solving real-world problems.
Existing knowledge is activated as a foundation for new knowledge.
New knowledge is demonstrated to the learner.
New knowledge is applied by the learner.
New knowledge is integrated into the learner’s world.” (Merrill, 2001, p. 2)
Does your instruction rate 5 stars? A rating scale
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

26. Six Types of Knowledge (Frick)
Purpose of Inquiry
Scope of Knowledge
Unique
Generalizable
What is? 1 2
What is effective? 3 4
What is worthwhile? 5 6
Type 3: Indiana University physician education program
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

27. Improving and Assessing Instructional Effectiveness -- Ted Frick
Recommendations
In physician education, focus on Type 3 knowledge:
Apply Merrill’s 5 principles as criteria for design. See video (requires RealPlayer).
Use Kirkpatrick’s 4 levels of evaluation for your programs at IU.
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

28. Improving and Assessing Instructional Effectiveness -- Ted Frick
Questions?
This presentation is available at:
Contact:
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick