1. ONR/NETC Planning Meeting July 18, 2003 UCLA/CRESST Los Angeles, CA ONR/NETC 7/18/03 v.4 What Works in Distance Learning Harry O’Neil University of Southern California CRESST

2. 2 OSD/NETC 7/18/03 v.4 Problem: State of the practice in Distance Learning (DL) lags the state of the art in instructional design Research-based guidelines are needed to help Naval and contractor personnel design, evaluate, and use DL The Research Problem & Situation

3. 3 OSD/NETC 7/18/03 v.4 Example Guidelines What Works: Research About Teaching and Learning U.S. Department of Education. (1987). What works: Research about teaching and learning (2nd ed., IS 87-110). Washington, DC: U.S. Government Printing Office.

4. 4 OSD/NETC 7/18/03 v.4 Example Guidelines (cont.) Summary of Research Findings With Implications for Navy Instructions and Learning Office of the Chief of Naval Education and Training. (1988). What works: Summary of research findings with implications for Navy instruction and learning (NAVEDTRA 115-1). Pensacola, FL.

5. 5 OSD/NETC 7/18/03 v.4 Example Guidelines (cont.) What Works in Alternative Assessment Advance Design Information, Inc. (1992, September). What works in alternative assessment? Sherman Oaks, CA. (Baker, E. L., O’Neil H. F., Jr., & Linn, R. L.)

6. 6 OSD/NETC 7/18/03 v.4 Technical Objectives Develop a core set of research-based DL Guidelines Instructional Design (Clark) Multimedia (Mayer) Learning Strategies (Dembo) Assessment (Baker) Management Strategies (Kazlauskas) Self-Regulation and Motivation (O’Neil,Clark)

7. 7 OSD/NETC 7/18/03 v.4 Technical Approach/Deliverables Use experts to compile knowledge Require scientific base Use standardized formats Guidelines Worked examples Create paper version

8. 8 OSD/NETC 7/18/03 v.4 Technical Approach/Deliverables (cont.) Transition to NETC portal Transition to OSD/Advanced Distributed Learning portal Create DL lesson in disc brakes as case/worked example Conduct formative evaluation Revise guidelines/lesson

9. 9 OSD/NETC 7/18/03 v.4 Strategies Based on Coherence Principle (Mayer, v.4, 02/03/03) 1. Guideline (nontechnical):Coherence effect: People learn better from multimedia messages when extraneous words, pictures, and sounds are excluded rather than included. 2. Guideline (technical): Coherence principle: If creating a multimedia message, then eliminate words, pictures, and sounds that are not directly relevant to the explanation. Guideline Examples (cont.)

10. 10 OSD/NETC 7/18/03 v.4 … Coherence Principle (Mayer, v.4, 02/03/03) 3. Guideline based on: Research 4. Degree of confidence:High for short multimedia explanations of how something works. Guideline Examples (cont.)

11. 11 OSD/NETC 7/18/03 v.4 5. Comments: The guideline is based on a collection of studies, including the following. (1) Students read an illustrated text explaining how lightning storms develop. For some students, a few sentences are added that contain interesting stories and photos about lightning such as what happened when a high school football player was struck (which can be called seductive details). Students performed better on transfer tests when interesting material was excluded rather than included. … Coherence Principle (Mayer, v.4, 02/03/03) Guideline Examples (cont.)

12. 12 OSD/NETC 7/18/03 v.4 5. Comments: (2) Students viewed a narrated animation explaining how lightning storms develop or how a car’s braking system works. For some students, the presentation included background music and environmental sounds (such as the sound of metal grinding when the brakes went on). Students performed better on transfer tests when the music and sounds were excluded rather than included. … Coherence Principle (Mayer, v.4, 02/03/03) Guideline Examples (cont.)

13. 13 OSD/NETC 7/18/03 v.4 5. Comments: (3) Students viewed a narrated animation explaining how lightning storms develop. For some students, the presentation included several interspersed video segments depicting interesting events such as a golfer being carried away after being struck by lightning (which can be called seductive details). Students performed better on transfer tests when the interesting video clips were excluded rather than included. … Coherence Principle (Mayer, v.4, 02/03/03) Guideline Examples (cont.)

14. 14 OSD/NETC 7/18/03 v.4 6. References: 7. Glossary: 8. User: 9. ID: … Coherence Principle (Mayer, v.4, 02/03/03) Guideline Examples (cont.)

15. 15 OSD/NETC 7/18/03 v.4 Progress to Date Guidelines Guidelines delivered as of 2/28/03 –Learning Strategies –Instructional Strategies –Management Strategies –Multimedia –Assessment Strategies

16. 16 OSD/NETC 7/18/03 v.4 Progress to Date (cont.) Lesson/Worked Examples Draft lesson example available 8/31/03 Final lesson example available 12/03

17. 17 OSD/NETC 7/18/03 v.4 Transition Opportunities Explore with NETC possible implementation of Guidelines in Human Performance Centers Formative evaluation, fall 03 OSD implementation Bob Wisher / ADL program

18. 18 OSD/NETC 7/18/03 v.4 What Could Be Done Next? Guidelines Team Training Team Assessment Simulation

19. 19 OSD/NETC 7/18/03 v.4 What Could Be Done Next? (cont.) Create Information Management Training System Guidelines Reach back Lessons instantiating guidelines Assessment authoring system

20. 20 OSD/NETC 7/18/03 v.4 Create Tools from Guidelines Formative evaluation tool to be used by instructional designers Deliverables tool to be used by contracting officer’s technical representative Proposal evaluation tool to be used by selection board What Could Be Done Next? (cont.)

21. 21 OSD/NETC 7/18/03 v.4 Back-up

22. 22 OSD/NETC 7/18/03 v.4 Self-Regulation Metacognition Motivation Self-Monitoring Self-EfficacyEffort Planning Anxiety Self-Regulation Components

23. 23 OSD/NETC 7/18/03 v.4 Guidelines Template Section Definition/Function 1.Guideline (non- technical): One-sentence overview of findings. What would you tell a smart but not technically oriented significant other about the area? e.g., In remedial instruction, use of learner control depends on high prior knowledge as well as motivation.

24. 24 OSD/NETC 7/18/03 v.4 Guidelines Template (cont.) Section Definition/Function 2.Guideline (technical): A one-sentence description of findings in technical terms (ideally expressed as a rule). 3.Guideline based on: Research or expert opinion.

25. 25 OSD/NETC 7/18/03 v.4 Guidelines Template (cont.) Section Definition/Function 4.Degree of confidence: Your degree of confidence in the guideline. Low, medium, high (if no empirical evidence, this high-confidence category cannot be used). 5.Comments: Two to three paragraphs re findings. More detail and how one might implement to lead to similar findings. Best evidence would be empirical.

26. 26 OSD/NETC 7/18/03 v.4 Guidelines Template (cont.) Section Definition/Function 6.References: Three to four, APA style (provide the paper copy/soft copy of the actual references). These are provided for users who want more detail. One of the references should be seminal (defined area, key terms, significant researcher, etc.). One reference should be a literature review in the area. A meta-analysis would be the best evidence. The remaining references should be recent journal articles (or references someone could easily find, e.g., no conference presentations, no dissertations, no technical reports, etc.).

27. 27 OSD/NETC 7/18/03 v.4 Guidelines Template (cont.) Section Definition/Function 7.Glossary: One-sentence definition (with paper copy/soft copy reference) of all jargon on the pages. 8.User: For whom did you write the two-pager? Instructional designer or program manager? 9.ID: Author name, What Works in Distance Learning: [Key Area], title of guideline, version #, day, month and year of most recent update.