IMPRINT Pro Usability Assessment Pratik Jha, Julie Naga and Dr. Raja Parasuraman George Mason University

2 Introduction A “professional” (Pro) version of IMPRINT is currently under development + More Modular + Added Functionality + Greater Flexibility This has also resulted in series of User Interface changes + Multiple windows to simultaneously display information + Tree structure for navigation + Pallete to show common used functionality

3 Project Goals To Uncover Usability Issues of IMPRINT Pro –Does the new interface allow users to conduct their analyses as effectively as with older versions of IMPRINT? –Is the added functionality and flexibility understandable and usable? –Is the new system easy to learn for a user trained on earlier versions? –Does the IMPRINT Pro interface allow users to navigate through the space of modeling options in an effective and timely manner?

4 Method 2 step approach –Heuristic evaluation (first cut) –Usability walkthrough (confirmatory)

5 Heuristic Evaluation Methodology Familiarization phase –Team read product manuals and other background material –Conducted high level task analysis –Gathered information on user characteristics –Attended IMPRINT workshop and tutorial Review Phase –3 reviewers conducted independent usability evaluation –7 well established heuristics were used for evaluation –Several group meetings were held to discuss reviews and consolidate evaluation

6 Heuristics

7 IMPRINT Pro Areas Evaluated 1.Overall look and feel (windows, menu structure, legibility of text and icons etc.) 2.Analysis tree 3.Navigational tab 4.Properties window 5.Adding a mission from library 6.Adding warfighter 7.Adding mission 8.Adding equipment

8 Tasks 1.Add new analysis a. Add name of analysis b. Add brief description and purpose of the analysis 2.Add a warfighter a.Add name b.Select MOS c.Select workload strategy 3.Add a mission 3.Add mission name and description 4.Add task network (functions and task) 5.Add task parameters or change properties 4.Add a equipment a.Add equipment name and description b.Add or change parameters 5.Save an analysis 6.Add analysis from the library

9 Results 37 usability issues were uncovered from heuristic analysis Results suggest some major usability problems in IMPRINT PRO software that could lead to –Poor user experience –Increased training time and poor learnability –Negative training transfer effect because the software is inconsistent with the previous versions Majority of the usability issues are severe or moderate in nature

10 Results – Usability Issues by Severity

11 Results – Usability Issues by Heuristic Criteria

12 Results Ease of Navigation –Software does not lead the user to the next step –User options are not visible –Number of user clicks are not optimized –Analysis tree and navigational bread crump gets cluttered fairly soon Look and Feel –Display clutter due to multiple windows –Poor contrast between background and text

13 Results Functionality –Pane seems to be of limited use and adds to display clutter –Windows functionality in the view menu is not enabled –Parameters cannot be changed in the properties window Enabling user options –A new version is added instead of new analysis –Cant drag and drop missions from other analysis

14 Results Maximize Recognition –Excessive use of Acronyms –MOS categories and workload strategies are not defined Preventing User Error –Incorrect user input sort (for example user is asked to input probability in percentage)

15 Results Providing User Feedback –Navigational tab and analysis tree are not synchronized –No feedback provided to the user if the analysis is saved or not –Property windows are not labeled

16 Usability Walkthrough We conducted a usability walkthrough to validate results of heuristic analysis Methodology –Tested 4 participants on Pro (2 experts and 2 novice) –Participants performed a series of task –Participants were asked to think aloud –Session was video taped for future analysis

17 Task 1.Add new analysis –Name it NCops –Add brief description about the purpose of the analysis for reference 2.Add new Warfighter –Name the warfighter as soldier 1 –Select appropriate MOS category –Select appropriate workload management strategy 3.Add new mission –Name the mission Network Centric Warfighter –Add a brief description 4.Add task network –Add function and name it Gather Information –Add a task to above function and name it Situation Monitoring (task 1) –Add another task to above function and name it Scan Head Mounted Display (task 2) –Change the priority of task 1 to 1 (Priority can be changed from the Effects screen of task properties) –Change the priority of task 2 to 2 –Review the task properties in snapshot view 5.Save your analysis and exit

18 Data Collected Background questionnaire –IMPRINT usage –Experience –Others Subjective ratings –Overall reaction to the application –Screen –Terminology and system operations –Learning –System Think aloud data Comments

19 Results – Overall Reaction to Application

20 Results

21 Results – User Comments Feedback " It is very different than 7 and will require old IMPRINT users to have new training because it is vastly different“ “ There is no feedback on saving an analysis."

22 Recommendations 1.A detailed task analysis should be conducted to improve the navigation and refine the functionality of IMPRINT Pro 2.An analysis should be performed to investigate if the radically different interface of IMPRINT Pro from the previous versions could cause a negative transfer training effect 3.Functionality aimed at minimizing training time or to provide users help in conducting the analysis should be explored. For example, the efficacy of “wizards” that help users in building an analysis should be evaluated.

23