1 System for Administration, Training, and Educational Resources for NASA Training Evaluations

2 Agenda  Lesson 1: Overview for Training Evaluation  Lesson 2: Research Item Evaluation Surveys  Lesson 3: Gather and Analyze Survey Results  Lesson 4: Evaluate Learning Using Pre- and Post-Test

3 Lesson 1 Overview for Training Evaluations

4 Lesson Objectives  Identify all levels of evaluation and explain which evaluation tools you will use in SATERN.  Understand evaluation terms and definitions  Explain how evaluations are implemented at NASA  Identify SATERN configurations needed to support evaluations

5 Evaluating Training Results Using SATERN Levels of Evaluation

6 How Evaluations are Implemented at NASA  A questionnaire survey will automatically be assigned to leaners after successful completion of some of the instructor-led course or an online module.  SATERN administrators can use their own discretion and decide whether to use Level 2 evaluations pre- and post-tests.  NASA will continue to use Metrics that Matter surveys. Administrators will not create or track follow-up evaluations in SATERN.

7 Configuration to Support Evaluations  Completion Status Triggers Survey Assignment  Rating Scale Used for Surveys Strongly Disagree Neither Agree nor Disagree AgreeStrongly Agree 12345

8 Lesson Wrap-Up  Identified all levels of evaluation and explained which evaluation tools you will use in SATERN.  Understand evaluation terms and definitions  Explained how evaluations are implemented at NASA  Identified SATERN configurations needed to support evaluations

9 Lesson Check

10 Lesson 2 Research Item Evaluation Surveys

11 Lesson Objectives  Identify the purpose of an item evaluation  Identify possible question types for a item evaluation survey  Research and preview item evaluation surveys used in SATERN  Determine configuration options for item evaluation surveys  Associate an item evaluation survey with an item

12 Item Evaluations Collect Data on Learner Satisfaction  Assesses a learner’s reaction to a training event  Most effective if delivered immediately after the training event ends  Sometimes referred to as a ‘smile sheet’  Measures how well learners liked the training

13 Question Types for Item Evaluations  Rating Scale  One Choice  Multiple Choice  Open Ended

14 Lab 1 Research and Preview a Published SATERN Survey

15 Configuration Options for Level 1 Surveys

16 How to Associate a Survey with an Item  There are two ways to make an association between an item and a survey: Item Usage tab of questionnaire survey Evaluations tab of item record

17 Lab 2 Method 1: Associate an Item with a Survey

18 Lab 3 Method 2: Associate a Survey within an Item

19 Lesson Wrap-Up  Identified the purpose of an item evaluation  Identified possible question types for a item evaluation survey  Researched and previewed item evaluation surveys used in SATERN  Determined configuration options for item evaluation surveys  Associated an item evaluation survey with an item

20 Lesson Check

21 Lesson 3 Gather and Analyze Survey Results

22 Lesson Objectives  Launch and complete assigned surveys from the learner side of SATERN  Check learners’ progress from a survey record  Run reports from the survey record  Run item evaluation reports from the reports tab  Remove item evaluations from learner records

23 How Learners Complete Surveys  An alert displays on their SATERN home page  A learner is also sent a notification  If an answer is missed, a warning message displays to inform the learner to complete the question  Learners can save the survey and return later to continue editing

24 Lab 4 Launch and Complete an Assigned Survey

25 Check Progress from the Survey Record

26 Reports Related to Evaluations  Item Evaluation Report  Item Evaluation by Individual Response Report  Item Evaluation by Instructor Report

27 Lab 5 Run an Item Evaluation Report

28 Removing Outdated Surveys  The best way to capture accurate results is to have learners complete the survey soon after training.  If a considerable amount of time has passed without survey completion, you may want to remove the survey from the learner record.

29 Lab 6 Method 1: Remove Surveys via Learner Needs Management

30 Lab 7 Method 2: Remove Surveys Directly from the Learner Record

31 Lesson Wrap-Up  Launched and completed assigned surveys from the learner side of SATERN  Checked learners’ progress from a survey record  Ran reports from the survey record  Ran item evaluation reports from the Reports tab  Removed item evaluations from a learner record

32 Lesson Check

33 Lesson 4 Evaluate Learning Using Pre- and Post-Tests

34 Lesson Objectives  Explain the purpose of Level 2 evaluations  Associate a pre-test and post-test as Level 2 evaluations

35 Level 2 Learning Evaluations  Purpose of a pre- and post-exam is to assess the learner population’s knowledge of the content prior to the training, and then just after the training.  If the training program is effective, there will be an increase in knowledge when the two exam results are compared

36 Lab 8 Associate Pre- and Post-Exams with an Item

37 Lesson Wrap-Up  Explained the purpose of Level 2 evaluations  Associated a pre-test and post-test as Level 2 evaluations

38 Lesson Check

39 Course Summary  Identified all levels of evaluation and explained which evaluation tools are used in SATERN  Understand evaluation terms and definitions  Explained how evaluations are implemented at NASA  Identified SATERN configurations needed to support evaluations