1. 1 DLXXI Courseware Guidelines Workshop The University of Southern California Center for Cognitive Technology & Institute for Creative Technology Dr. Richard E. Clark, Rich DiNinni, Kenneth Yates Lesson 3

2. 2 Lesson Goals You will learn to use the checklist to determine whether DL designs contain adequate: 1.Practice exercises to help trainees apply what they learn 2.Feedback to help them adjust their learning so that it is accurate 3.Assess learning to insure that ELO’s and TLO’s are achieved

3. 3 Reasons Practice is seldom provided, which causes problems with learning and effective transfer to the mission environment Designers often give very difficult problems for practice, which damages trainee confidence Practice seldom reflects objectives, assessment or mission environment

4. 4 Reasons Feedback often focuses trainees on the fact that they made a mistake, or were “wrong” – this slows and decreases learning Assessment of training is most often a “memory test” – NOT adequate for use/apply goals Objectives, knowledge type/use, practice AND assessment must be aligned

5. 5 Practice Part task to whole task practice  After four or five small (part-task) practices, present a big problem that requires the use of all previously practiced tasks (whole-task) Set practice in mission environment Simple tasks, then more complex & novel Practice must reflect learning objectives Practice until they succeed or exceed limit

6. 6 Feedback Give frequent feedback during practice so mistakes do not accumulate Feedback should:  Acknowledge success “great job up to this point”  Do not say “you made an error” instead say “Your strategy did not work on this section/part…”  Give opportunity to revise their strategy, and to review relevant parts of module  Align practice with learning objective, mission environment and assessment of performance

7. 7 Assessment Do assessments ask trainees to apply what they have learned (Tell and/or Use)? Are the assessments aligned with the ELO’s and TLO’s and the exercises? Do assessments reflect the mission environment (transfer) as much as possible?

8. 8 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1 Do all lessons include practice ?  On page 33 – “Practice and tests will occur in several places throughout each module and lesson, to enhance and reinforce learning but also to prepare students for the testing process. Types of quiz questions and type of feedback will vary by which type of quiz is being developed as follows: Drill/practice or ‘Obstacles’ – These will be checks on learning to test both knowledge acquisition and to provide early practice for application of that knowledge to scenarios based on the Irvington terrain…. “ (See Notes Below)

9. 9 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1 Do all lessons include practice ? X  It appears that all lessons will be practiced.

10. 10 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.1. Is part-task practice followed with whole-task practice?  After describing how to fell trees and remove stumps with explosives, the course covers the construction of an abatis using trees: “An abatis can be constructed rapidly using demolition to fell the trees. The trees should remain attached to the stumps to make the obstacle more effective and difficult to clear.” Is the abatis practice also a larger practice of the previous training on how to feel trees and uproot stumps?

11. 11 Abatis

12. 12 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.1. Is part-task practice followed with whole-task practice? X  We believe that this design offers the opportunity for constant practice of larger and larger chunks of learning. However, it might be a good idea to bring this up with the contractor to insure that they provide a “whole” practice at the end of the course where trainees have to remove a number of obstacles in a real or simulated field setting.

13. 13 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.2. Does whole-task practice mirror the mission environment?  From the design on page 33: “Skills are trained for implementation in the geographic region nearest to the site of practical exercise implementation. …provide early practice for application of that knowledge to scenarios based on the Irvington, KY terrain…” Is this statement an attempt to mirror the mission environment?

14. 14 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.2. Does whole-task practice mirror the mission environment? X  Skills are trained for implementation in the geographic region nearest to the site of practical exercise implementation. While this may not mirror the “mission environment”, one could conclude that the designers thought carefully and practically about the transfer environment they could anticipate.

15. 15 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.3. Does practice begin with simple problems and progress to more complex problems?  Does this appear to be an attempt to sequence by difficulty level? In the design document, on pages 6-7 KNK01410 - Calculate, Designate, and Employ Demolitions 052-193-2015 Place Timber-Cutting Charges 052-193-2016 Place Steel-Cutting Charges 052-193-2017 Place Breaching Charges 052-193-2018 Place Cratering Charges 052-193-3022 Calculate Timber-Cutting Charges 052-193-3023 Calculate Steel-Cutting Charges 052-193-3024 Calculate Breaching Charges

16. 16 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.3. Does practice begin with simple problems and progress to more complex problems?  X The designers did not state that they sequenced tasks by difficulty levels.

17. 17 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.4. Does practice require trainees to solve increasingly novel problems?  Will sequencing by difficult level also lead to practice of Increasingly novel problems? In the design document, on pages 6-7 KNK01410 - Calculate, Designate, and Employ Demolitions 052-193-2015 Place Timber-Cutting Charges 052-193-2016 Place Steel-Cutting Charges 052-193-2017 Place Breaching Charges 052-193-2018 Place Cratering Charges 052-193-3022 Calculate Timber-Cutting Charges 052-193-3023 Calculate Steel-Cutting Charges 052-193-3024 Calculate Breaching Charges

18. 18 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.4. Does practice require trainees to solve increasingly novel problems?  X Since the contractor did not state that more familiar practice exercises would be placed before less familiar exercises, this item receives a NO-GO.

19. 19 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.5. Is there consistency between the practice exercises and the learning objectives?  Does this statement indicate an attempt to connect learning objectives with practice exercises? Page 33, 34 in the course design states: “End of ELO tests or quizzes – These will cover at least one question per learning objective. Questions will be performance/scenario based. Feedback will again be remedial but may not be as detailed (i.e. each answer does not need a feedback statement), and should at least refer the student back to the reference relating to the lesson.”

20. 20 Evaluating Practice, Feedback, and Assessment GuidelineMeets Guideline? GONO-GO 3.1.5. Is there consistency between the practice exercises and the learning objectives? X  We believe that the designers will attempt to connect learning objectives with practice exercises.

21. 21 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.6. Are all types and uses of knowledge taught in the course practiced?  Does this statement describe a comprehensive strategy for practice of ALL types and uses of knowledge taught in the course? From page 33: “General Practice Strategies: Practice and tests will occur in several places throughout each module and lesson, to enhance and reinforce learning but also to prepare students for the testing process.”

22. 22 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.6. Are there an adequate number of practice exercises to master each skill? OX The contractor describes frequent practice, but not of all types and uses of knowledge taught in the course.

23. 23 Feedback Give frequent feedback during practice so mistakes do not accumulate Feedback should:  Acknowledge success “great job up to this point”  Do not say “you made an error” instead say “Your strategy did not work on this section/part…”  Give opportunity to revise their strategy, and to review relevant parts of module  Align practice with learning objective, mission environment and assessment of performance

24. 24 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.2. During practice, is feedback provided?  Page 16 in design: “Feedback …will be used throughout this program/course. Each lesson will contain feedback tools to assist the student in his retention of training materials. Feedback will utilize variety, humor, and interactivity to enhance student motivation and shape an online environment conducive to learning …” Does this statement indicate that feedback is provided during practice?

25. 25 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.2. During practice, is feedback provided? x  We believe that the contractors design clearly indicates that feedback will be provided during practice

26. 26 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.2.1. Is feedback frequent, so that errors do not accumulate?  Did the preceding notes section describe feedback that is frequent enough to correct errors?

27. 27 Feedback Example #1

28. 28 Feedback Example #2

29. 29 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.2.1. Is feedback frequent, so that errors do not accumulate? X  We believe that feedback will be frequent enough so that errors do not accumulate

30. 30 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? 3.2.2. When trainees make mistakes, does feedback: GONO-GO 3.2.2.1. Acknowledges correct use of strategies or steps and does not focus on their mistake?  Look at example on next slide

31. 31 Does this focus on errors or strategy?

32. 32 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? 3.2.2. When trainees make mistakes, does feedback: GONO-GO 3.2.2.1. Acknowledges correct use of strategies or steps and does not focus on their mistake?  X Feedback Page 33; feedback uses a GO / NO-GO; student strategy use is not discussed in design document. Change feedback from “mistake, you are a no-go” to “ the strategy you are using needs to be modified”. Consider the following hint: “Recompute the critical dimension of the target.”

33. 33 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? 3.2.2. When trainees make mistakes, does feedback: GONO-GO 3.2.2.2. Provides trainees with the opportunity to correct their strategy?  Page 33 “…in many instances students will be required to do the exercise over until they perform correctly… Where appropriate, students will be allowed two tries at the correct answer and then if incorrect will receive remediation.” Does these statements indicate that students will have the opportunity to correct their strategy?

34. 34 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? 3.2.2. When trainees make mistakes, does feedback: GONO-GO 3.2.2.2. Provides trainees with the opportunity to correct their strategy? X  It appears that students will have the opportunity to correct their strategy

35. 35 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? 3.2.2. When trainees make mistakes, does feedback: GONO-GO 3.2.2.3. Provide trainees with the opportunity to review the relevant parts of the demonstration?  Review the description of the exercises below to see if The designers intended to allow the trainees to review the demonstrations.

36. 36 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? 3.2.2. When trainees make mistakes, does feedback: GONO-GO 3.2.2.3. Provide trainees with the opportunity to review the relevant parts of the demonstration? X  It appears that trainees are able to “go back” within modules at any time, in order to review relevant portions of the demonstration. Additionally, they may repeat any section or module of the course, both before and after module tests.

37. 37 Assessment Do assessments ask trainees to apply what they have learned (Tell and/or Use)? Are the assessments aligned with the ELO’s and TLO’s and the exercises? Do assessments reflect the mission environment (transfer) as much as possible?

38. 38 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.3. Does the lesson provide a method of assessing the trainees’ learning by asking them to apply what was learned?  “Four specific tests for record will occur in the DL phase, testing the OPFOR/IPB content, the ERO content, the CADD content, and the R&S content. Each of these four tests for record will contain some computer-graded knowledge portion as well as application based products that are to be sent to the SGI for grading. All portions of the tests for record will be performance-oriented so that even the knowledge portion will use graphics and scenarios as well as standard computer-graded question formats.”

39. 39 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.3. Does the lesson provide a method of assessing the trainees’ learning by asking them to apply what was learned? X  We believe that the lesson uses application exercises to assess the trainees’ learning.

40. 40 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.3.1 Are the assessments aligned with the learning objectives and practice exercises?  We determined earlier that objectives were aligned with practice exercises – this asks if both practice and objectives are aligned with assessments (tests, scored practice). What is your decision after reading the description in the Notes Section below (taken from page 33-35 in the design).

41. 41 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.3.1 Are the assessments aligned with the learning objectives and practice exercises? X  The design promises that: “The learner will complete an MPA at the end of each module. They will be tested to determine if they acquired the knowledge presented within the associated module. If they passed the MPA, they are granted credit for the module. However, if they fail, they will need to retake the MPA until they pass. They will receive feedback as to which topics they failed and will receive a recommendation to retake that portion of the Module.”

42. 42 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.3.2. Does the assessment reflect the performance of the learning objective(s) in the mission environment as closely as the media will permit?  Check the description from the design in the Notes section below for the “Course Practical Exercise” Does it meet this checklist requirement?

43. 43 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.3.2. Does the assessment reflect the performance of the learning objective(s) in the mission environment as closely as the media will permit? X  The design states: “After completing all recommended modules, the learner will take the Course Practical Exercise (and) will be placed in a simulated environment where they will apply all of the skills and knowledge acquired in the course…(and) Skills are trained for implementation in the geographic region nearest to the site of practical exercise implementation (and) …provide early practice for application of that knowledge to scenarios based on the Irvington, KY terrain.” This seems to fit the checklist requirement

44. 44 Team Exercise #3 Evaluating Practice, Feedback and Assessment Each team selects:  A Team Leader (coordinator, timekeeper)  A Reporter (summarizes and presents teamwork to rest of us) Assign responsibility to find examples of each introduction item (3.1 to 3.3.2) in the design for the Armor Captains Career Course Report those examples to the rest of the workshop when it is your turn

45. 45 3.1. Do all lessons include practice? 3.1.1. Is part-task practice followed with whole-task practice? 3.1.2. Does whole-task practice mirror the mission environment? 3.1.3. Does practice begin with simple problems and progress to more complex problems? 3.1.4. Does practice require trainees to solve increasingly novel problems? 3.1.5. Is there consistency between practice exercises and the learning objectives? 3.1.6. Are there an adequate number of practice exercises to master each skill? 3.2. During practice, is feedback provided? 3.2.1. Is feedback frequent, so that errors do not accumulate? 3.2.2. When trainees make mistakes does feedback: 3.2.2.1. Acknowledge correct use of strategies or steps and not focus on their mistakes? 3.2.2.2. Provide trainees the opportunity to correct their strategy? 3.2.2.3. Provide the opportunity to review the relevant parts of the demonstration? 3.3. Does the lessons provide a method of assessing the trainees’ learning by asking them to apply what was learned? 3.3.1. Are assessments aligned with the learning objectives and practical exercises? 3.3.2. Do the assessments reflect the performance of the learning objectives in the mission environment as closely as media will permit? Practice, Feedback, and Assessment Design

46. 46 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1 Do all lessons include practice ? 

47. 47 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.1. Is part-task practice followed with whole-task practice? 

48. 48 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.2. Does whole-task practice mirror the mission environment? 

49. 49 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.3. Does practice begin with simple problems and progress to more complex problems? 

50. 50 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.4. Does practice require trainees to solve increasingly novel problems? 

51. 51 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.5. Is there consistency between the practice exercises and the learning objectives? 

52. 52 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.1.6. Are there an adequate number of practice exercises to master each skill? 

53. 53 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.2. During practice, is feedback provided? 

54. 54 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.2.1. Is feedback frequent, so that errors do not accumulate? 

55. 55 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? 3.2.2. When trainees make mistakes, does feedback: GONO-GO 3.2.2.1. Acknowledges correct use of strategies or steps and does not focus on their mistake? 

56. 56 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? 3.2.2. When trainees make mistakes, does feedback: GONO-GO 3.2.2.2. Provides trainees with the opportunity to correct their strategy? 

57. 57 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? 3.2.2. When trainees make mistakes, does feedback: GONO-GO 3.2.2.3. Provide trainees with the opportunity to review the relevant parts of the demonstration? 

58. 58 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.3. Does the lesson provide a method of assessing the trainees’ learning by asking them to apply what was learned? 

59. 59 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.3.1 Are the assessments aligned with the learning objectives and practice exercises? 

60. 60 Practice, Feedback, and Assessment Design GuidelineMeets Guideline? GONO-GO 3.3.2. Does the assessment reflect the performance of the learning objective(s) in the mission environment as closely as the media will permit? 