J.S. Bresee Conor McKenna iFest – 08/01/17 Benefits of Using Machine-generated Human Factors Performance Data for Training and Assessment J.S. Bresee Conor McKenna iFest – 08/01/17

Bottom Line Up Front Adaptive training depends on continuous assessment Typical assessment methods are intrusive and not suitable for team tasks Bio-signal data enables data gathering that is continuous and unobtrusive Start saving audio: to make this useful, it’s time to start collecting, baselining and validating these data.

Customized Learning: Unrealized Potential Historically: Customized learning = 1 teacher + 1 student Practically: Customized learning depends upon automation

Two key requirements: Continuous assessment Selecting an alternative learning option requires knowledge of student progress Alternative learning pathways and events Alternate paths designed in advance and pre-loaded Alternate paths generated by gaming structures and logic.

Challenges Anticipating (or generating) - the number and type of alternate learning pathways that will be needed. Dealing with both the creative and financial challenges of providing and selecting alternate pathways. Accurately assessing student progress across a range of dimensions. Performance Engagement Collaboration Communication

Historic assessment methods Verbal information and cognitive tasks: Periodic insertion of “test questions” Easily designed Inexpensively implemented Obtrusive; interrupts the flow of learning Not suitable for assessing continuous performance

Historic assessment methods Continuous performance/team tasks: Instructor observation Record/replay and critique After-the-fact evaluation forms and questionnaires Requires either recall of performance or informed interpretation Resulting data is usually non-numeric and based on informed opinion (subjective).

Assessment with Bio-signal data Unobtrusive Continuously collected Objective numeric ratio-scale data Useful for individual feedback or aggregate analysis Available technology

Eye-tracking Used in cockpits for decades for weapons control Common human factors research variable Used in simulation visual systems

Facial expression recognition Spin-off of facial recognition technology Widely used in market research Demonstrated use in gathering learner feedback Body of research findings is substantial and expanding e.g., Dr. Sidney D’Mello, Notre Dame

Speech Signal analysis Based on well-established speech science Prosodic accommodation research is well established e.g.; Celine De Loose, Brian Vaughan, Trinity College, Dublin Non-verbal prosodic analysis works with any language

Sample use case for speech signal analysis

Applications: Any team performance task: Crew performance in flight simulation. Bio-signal data correlated with digital flight data recording and playback Applicable to multiple platforms (formations), crew-served platforms, remotely-piloted platform teams. Medical/first responder team performance Command center/CIC team training Consultation/negotiation training

The Path Ahead DATA Machine learning (AI) Adaptive Personalised Evidenced Develop individual performance and behavioural data to learners. Gather bio-signal data and associated meta-data in significant quantities Apply Big Data techniques to identify useful trends for adaptive learning Uncover useful adaptive learning paradigms for use with LMS or game-driven syllabus control

Company Overview >> Vocavio provides speech signal processing software to augment simulation, training and operations in aerospace & defense. Technology applications include; Monitoring and assessing crew communications in simulation environments i.e. integrating our middleware into LVC (live, virtual, constructive) training environments to measure and validate mission critical skills. Our software uses speech signal processing (processing non-verbal aspects of speech) to output a range of performance data relating to communication dynamics [e.g. communication effectiveness, leadership, cohesiveness, intensity etc.. ] We are a campus company of Trinity College Dublin. (incorporated in 2014). Patented technology (US, EUR, CAN) that was developed and scientifically validated at Trinity College Dublin (with assistance of military and commercial pilots). Industry achievements Modsim entrepreneur award 2016 by NDIA (Virginia Beach, USA) Top 20 cloud technology company 2016 (Clearwater M&A, UK) Jerry@vocavio.com