1. 1 Internal Testing Snapshot Jacob Hodges 8 August 2014 This briefing is provided for informational and lessons learned purposes only – all of the data collected during the Internal Evaluation will be destroyed

2. 2 Introduction Timeframe: 20 July – 8 August 2014 Number of emails sent: 20 Participants Update: ActivityNumber Registered10 Finished Pre-Test10 Completed Evaluation7

3. 3 Demographics Organizations involved: DISAM, DISAM Contractors Lockheed Martin International Training Team 711 th HPW/RHAS ONR RC SSC Pacific

4. Content Evaluation 4 Content TypeNumber of Reviews Video7 Web-based7 E-pub (Android/Apple)3 E-pub (Kindle)1 DISAM App4

5. Non-Testable Content Reviewed 5 Content MaterialNumber of Reviews Informality6 Egalitairism6 Short-Term Mentality6 Daily Life in America7

6. What did you like most about the learning experience? 6 Although I was unable to run this from a handheld Android device the desktop worked fine The video and web-based content was very engaging Fast, fun, well-constructed I noted the.flv movies are not working properly but the link at the very end of the last page (Daily Life in America) did not show up and hyperlinks was not accessible The video was very informative but too long so I liked that best. It was good to see and hear the speaker throughout the video. The graphics used nicely illustrated the ideas being discussed. Comparing the same content presented in multiple formats

7. What did you like least about the learning experience? 7 The inability to launch from an Android device Could not link to the mobile app – need a QR Code Really can’t do that – I wrote and produced it Thought it was not a real big deal, I noticed that I did not enjoy the web-based format as much because it required my interaction to click to the next module. The Daily Life in America module didn’t provide a button to click on to complete the module. The videos were not available in the web section at the time of my evaluation

8. List five words that express your experience 8

9. Overall, I was satisfied with this course 9

10. Problems Solved During Internal Evaluation 10 1)A sentence was added to the Learning Formats page informing participants when they would receive the MLS Final Instructions 2)The QR Code was added so that mobile devices can connect directly to the DISAM Mobile App

11. 11 3)Modified the last question on the Post-Test Questionnaire to read “Which word refers to...” to read versus “Which work refers to...” 4)Created shorter PINs for users accessing the MLS Data Portal on mobile devices 5)Streamlined the creation of PINS so that no human interface is required; thereby eliminating any potential violation of the research protocol requirements.

12. Final Thoughts.. 12 Two Learning Format issues (video and web-based) raised by DISAM will be addressed when Björn returns from vacation. The learning options are somewhat limited for Apple iPad/iPhone users (i.e., cannot access the DISAM App or Web-based formats) Issues noted in comments will be further reviewed to ensure resolution, if possible (i.e.,.flv movies are not working properly but the link at the very end of the last page (Daily Life in America) did not show up and hyperlinks was not accessible) Interim provided a some good insight and testing of the evaluation process – more streamlined approach What user liked most/least will be reviewed to see where improvements can be made Good to see the humor in some comments

13. 13 Questions/Comments