1. J. Fittipaldi / EO20 Crew Ops 256.961.0230 Displays Overview for Ops Leads Crew Ops 202 30 September 2015

2. J. Fittipaldi / EO20 Crew Ops 256.961.0230 2 Displays Overview Course Goal Provide EO20 Ops Leads with an overview of PDRT roles & responsibilities in ISS ops product development and the usability assurance process.

3. J. Fittipaldi / EO20 Crew Ops 256.961.0230 3 Displays Overview Course Objectives At completion of this course the student will have learned: –About the role and responsibilities of the PDRT; –About the payload crew display development process, from initial submittal to final product baseline; –What Usability is, what it isn’t, and why usability verification is a key component of the POIF Crew Support function;

4. J. Fittipaldi / EO20 Crew Ops 256.961.0230 Displays Overview Course 4 #Learning PointsSlide # 1Roles and Responsibilities5 2Usability: What It Is, What It Isn’t and Why It’s Important 14 3Display Review23 4Usability Verification38 5Display Baseline51

5. J. Fittipaldi / EO20 Crew Ops 256.961.0230 5 Learning Point# 1 Roles & Responsibilities PDRT - History ISS Mission Statement (draft 1996): –Charter for Payload Displays Review Panel (PDRP) 1.0 Purpose … The PDRP is an implementation review panel responsible for ensuring that United States (US) payload displays… …in the US elements possess a high degree of usability and conform to program standards… 2.0 Policy…The PDRP advises US payload display developers on standards and operability guidelines and provides final approval of US payload displays prior to crew training and flight…. 2.3 Products/Review Items…The PDRP will manage US payload display development review processes to ensure a high degree of payload usability and adherence to the DGCS document….

6. J. Fittipaldi / EO20 Crew Ops 256.961.0230 6 Learning Point# 1 Roles & Responsibilities Today –Scope of PDRT activities is unchanged: All crew displays supporting U.S. payloads and payloads operating within the USOS and its facilities. –Other: Crew procedures usability review (directive issued prior to POIWG #14) Display development training  PDRT open work and display development resources (for PDs) are maintained in the PDRT SharePoint Page: https://poifsps.ndc.nasa.gov/branches/EO20/PDRT/PDRT/PDRT.aspx

7. J. Fittipaldi / EO20 Crew Ops 256.961.0230 Learning Point# 1 Roles & Responsibilities PDRT is organized under the IDAGS Panel 7 MIOCB IDAGS PANEL IDAGS TEAM PDRT ODFCB

8. J. Fittipaldi / EO20 Crew Ops 256.961.0230 8 Learning Point# 1 Roles & Responsibilities – Working w/ IDAGS Integrated Display and Graphics Standards (IDAGS) Panel The IDAGS Multilateral Panel is a program-level panel that establishes a baseline for and controls changes to the ISS Display and Graphics Commonality Standard (DGCS) for all ISS International Partners/Participants (IP/Ps). The IDAGS Panel is comprised of all the IPs, Astronaut Office, Avionics & Software, Safety, JSC Human Factors, and PDRT [DGCS 1.3.1]: U.S. Payload displays shall be reviewed per POIF-1004, POIC Payload Operations Handbook, Vol. 1… [DGCS H1.4]: For U.S. payloads and payloads flying under a U.S. Science Program, the Payload Display Review Team (PDRT) acts as a mentor to payload developers, assisting them through the payload display development process.

9. J. Fittipaldi / EO20 Crew Ops 256.961.0230 Learning Point# 1 Roles & Responsibilities – Working w/ IDAGS 9 PDRT and IDAGS share responsibility for display on-orbit crew use acceptability, but pursue this goal with a different emphasis.

10. J. Fittipaldi / EO20 Crew Ops 256.961.0230 10 Learning Point# 1 Roles & Responsibilities – Interaction w/ Ops Leads PDRT & Ops Lead – Interaction –Payload Ops TIM / PDR / CDR support –Procedure Reviews (TIR & ECR): Review for interoperability Display OpNom –Usability Verification Planning Readiness review Test conduct / reporting Issues tracking Display product baseline

11. J. Fittipaldi / EO20 Crew Ops 256.961.0230 11 Learning Point# 1 Roles & Responsibilities - Documentation Display and Graphics Commonality Standard (DGCS) Main Volume and Appendices SSP 50313 (E) Operations Nomenclature (OpNom) SSP 50254 US PODF Management Plan POIF-2001 Pressurized Payloads Interface Requirements Document (IRD), Appendix C SSP 57000 Payload Data Sets Blank Book (PDSBB) SSP 52000-PDS POIC Payload Operations Handbook (POH), Volume 1 POIF-1004

12. J. Fittipaldi / EO20 Crew Ops 256.961.0230 12 Learning Point# 1 Roles & Responsibilities – The DGCS PDRT must assure display compliance with SSP 50313, the Display & Graphics Commonality Standards (DGCS) –DGCS is the body of requirements for presentation, interaction, and layout of ISS displays (DGCS 3.3) –DGCS Main Volume Addresses display requirements for all ISS Displays –DGCS appendices: Colors, graphic symbols, robotics, acronyms, changes requests Appendix H covers p/l display design standards and guidelines Appendix K (Mobile Platform Displays); and Appendix L (Hardware Displays) (new)

13. J. Fittipaldi / EO20 Crew Ops 256.961.0230 Learning Point# 1 Roles & Responsibilities – Other Design Standards  Non-Laptop displays may be derived by PDs using other design guidelines: Apple iOS Human Interface Guidelines: https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/MobileHIG/index.html Google Android User Interface Guidelines: https://developer.android.com/guide/topics/ui/index.html Microsoft Windows Desktop App User Interface Guidelines: https://msdn.microsoft.com/en-us/library/windows/desktop/ff657751(v=vs.85).aspx  PDRT and IDAGS work with PDs to merge these displays with the DGCS standards. Additional Resources: Designing Interfaces (O’Reilly), by Jenifer Tidwell © 2011 Designing Mobile Interfaces (O’Reilly), by Steven Hoober & Eric Berkman © 2012 MIL-STD-1472G Human Engineering Design Criteria for Military Systems, Equipment, and Facilities 13

14. J. Fittipaldi / EO20 Crew Ops 256.961.0230 14 Learning Point #2: Usability: What It Is, What It’s Not and Why It’s Important What is Usability? What Usability Is Not Usability vs. Utility Why is P/L Usability Assurance Important?

15. J. Fittipaldi / EO20 Crew Ops 256.961.0230 15 Learning Point #2: Usability: What It Is, What It’s Not and Why It’s Important USABILITY – GENERAL DEFINITIONS “… design features that enable something to be user-friendly. The term, and the concepts it encompasses, can be applied to anything from a paring knife to a space station,…” www.congressonlineproject.org/glossary.html “…the ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a system or component” [IEEE 90] “…whether a system is easy to learn, pleasant to use, error-free and error-forgiving, easy to remember and efficient.” www.eurocontrol.int/eatmp/glossary/terms-21.htm

16. J. Fittipaldi / EO20 Crew Ops 256.961.0230 16 Learning Point #2: Usability: What It Is, What It’s Not and Why It’s Important USABILITY: The Definition We Use: (ISO 9241-11): …the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. Effectiveness: accuracy and completeness Efficiency: resources expended (in relation to accuracy and completeness) to achieve goals Satisfaction: acceptability, lack of frustration, inspires confidence

17. J. Fittipaldi / EO20 Crew Ops 256.961.0230 17 Learning Point #2: Usability: What It Is, What It’s Not and Why It’s Important What Usability is Not: Usability vs. Utility http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnwui/html/uidesign.asp –Utility refers to the product’s ability to perform the task or tasks. The more tasks a product is designed to perform, the more utility it has The developer should ask: Does the product do what users need/want it to do? –Usability and utility are equally important. It matters little that something is easy if it's not what you want. Examples: 1980’s-era Microsoft MS DOS Word Processor = high utility / low usability Clamshell-type cell phone: low utility / high usability –At a minimum, the utility of an ops product should be assured prior to usability verification Procedure validation

18. J. Fittipaldi / EO20 Crew Ops 256.961.0230 18 Utility vs. Usability http://blog.druvaa.com/2009/02/22/six-common-usability-mistakes-in-software-product- design/ Often, with increased utility comes increased complexity. Learning Point #2: Usability: What It Is, What It’s Not and Why It’s Important

19. J. Fittipaldi / EO20 Crew Ops 256.961.0230 19 UtilityUsable Learning Point #2: Usability: What It Is, What It’s Not and Why It’s Important

20. J. Fittipaldi / EO20 Crew Ops 256.961.0230 20 What Payload Display Usability Verification is Not: –Procedure Validation –HFIT –PSIV –The TIR or JIT –PTDR –Crew Training Learning Point #2: Usability: What It Is, What It’s Not and Why It’s Important

21. J. Fittipaldi / EO20 Crew Ops 256.961.0230 21 Why do we need to assure display usability? Living and working in space imposes unique capabilities and limitations on the human operator: –Crew rotation-handoff, continuity –Cross-cultural differences/English as a second language –Crew experience with multiple types of software –Crew payload training - minimal pre-flight training/familiarization –Extended time between training and use (extinction) –Long on-orbit duration; crew endurance –Payload operations schedule onboard –Crew availability to service payload Learning Point #2: Usability: What It Is, What It’s Not and Why It’s Important

22. J. Fittipaldi / EO20 Crew Ops 256.961.0230 22 Why do we need to assure displays usability? (cont.) Historical Case-in-point: Spacelab IML-1: Crew “…challenged by a large variety of different displays and experiment-unique command syntax....forced to bear an excessive training burden …(decreased efficiency)” ISS displays are hosted on multiple types of platforms & operating systems: ISS system displays all run on Solaris (UNIX environment) Payload displays run on various software and hardware platforms International Procedure Viewer (iPV) is web-based; Uses Internet Explorer Learning Point #2: Usability: What It Is, What It’s Not and Why It’s Important

23. J. Fittipaldi / EO20 Crew Ops 256.961.0230 23 Learning Point #3 How To Review Displays Now that we know what usability is, how are payload displays reviewed?

24. J. Fittipaldi / EO20 Crew Ops 256.961.0230 24 PDRT Process Flow

25. J. Fittipaldi / EO20 Crew Ops 256.961.0230 25 Learning Point #3 How To Review Displays PDRT looks for several things when reviewing displays: Has the PD provided a simple, sufficient display architecture? Does the GUI allow the crew to have constant situation awareness of the payload operational state? Do the displays automate as many payload functions as possible? Does the software minimize the amount of data the crew must input manually via use of pre-defined tables of values; command lists, etc.? Do the displays reduce both the time required and the possibility of input errors? Do they allow for error recovery? If the GUI comprises multiple screens, is the navigation path rendered clear and efficient; i.e. through use of nav buttons, titles, headers, GLIs, tabs, etc.? Do the screens arrange functions appropriately by hierarchy, ops sequence; frequency of use; etc.? Are display actions accompanied by appropriate visual feedback; incl. confirmation, processing delays; etc.? Display density: Goal < 50%

26. J. Fittipaldi / EO20 Crew Ops 256.961.0230 26 Learning Point #3 How To Review Displays Yikes!

27. J. Fittipaldi / EO20 Crew Ops 256.961.0230 27 Learning Point #3 How To Review Displays You’re kidding, right?

28. J. Fittipaldi / EO20 Crew Ops 256.961.0230 28 Oh, H*ll No! Learning Point #3 How To Review Displays

29. J. Fittipaldi / EO20 Crew Ops 256.961.0230 29 Learning Point #3 How To Review Displays Why make the crew jump through the programmer’s hurdles?… …If you can automate those functions instead?

30. J. Fittipaldi / EO20 Crew Ops 256.961.0230 30 Learning Point #3 How To Review Displays What happened? Who knows?!

31. J. Fittipaldi / EO20 Crew Ops 256.961.0230 31 Learning Point #3 How To Review Displays Why present the crew with this……When it may be feasible to do this 34

32. J. Fittipaldi / EO20 Crew Ops 256.961.0230 32 Learning Point #3 How To Review Displays When reviewing displays (cont.): Consider the display spatial layout –Physical similarity Reasonable correspondence of control and display physical features (e.g., schematic representation of rack, hardware) –Physical arrangement Controls/displays arranged in corresponding patterns Proximity of controls and displays Consistent arrangement of sets of similar devices (e.g., phone keypads)

33. J. Fittipaldi / EO20 Crew Ops 256.961.0230 33 Learning Point #3 How To Review Displays DGCS Section H3.2 STANDARD PAYLOAD LAYOUT

34. J. Fittipaldi / EO20 Crew Ops 256.961.0230 34 Learning Point #3 How To Review Displays DGCS H3.2.1 Required Components Some components in the Standard Payload Template are required on all payload displays. These are: – Title bar – Clock data field – Primary button panel – Exit button – Workspace – Access to the display software version number

35. J. Fittipaldi / EO20 Crew Ops 256.961.0230 35 Learning Point #3 How To Review Displays DGCS Section H3.2.1 – Required Components (cont.)

36. J. Fittipaldi / EO20 Crew Ops 256.961.0230 36 Learning Point #3 How To Review Displays DGCS H3.2.2 Optional Components –Alert bar –Payload Status area –Menu bar –Optional primary buttons –Status bar

37. J. Fittipaldi / EO20 Crew Ops 256.961.0230 37 Learning Point #3 How To Review Displays DGCS Section H3.2.2 – Optional Components (cont.)

38. J. Fittipaldi / EO20 Crew Ops 256.961.0230 38 Learning Point #4 Usability Verification Usability Verification –Verification Methods –Test or Trial? –The Usability Test Plan –The Usability Test Report

39. J. Fittipaldi / EO20 Crew Ops 256.961.0230 39 Learning Point #4 Usability Verification Usability Verification Methods – 3 General Types; several sub-alternatives: –Inquiry (e.g. Interviews, Focus Groups, Self-Report Logs, etc.) –Inspection (e.g. Standards Inspection, Guideline Checklists, Cognitive Walkthroughs, etc.) –Testing (e.g. Thinking Aloud Protocol, Question Asking Protocol, Performance Measurement, Eye Tracking, etc.) PDRT methods –Usability Test Thinking Aloud protocol –Usability Trial Thinking Aloud protocol / Cognitive Walkthrough

40. J. Fittipaldi / EO20 Crew Ops 256.961.0230 40 Learning Point #4 Usability Verification SSP 58700, PODF Management Plan Section 4.2.1 - Usability Verification The usability (verification event) is a POIF Crew Ops milestone. It is: …an evaluation to determine the readiness of a payload’s onboard displays and procedures for flight. It is conducted to ensure operability for payloads, regardless of the type of Human Computer Interface (HCI). The onboard payload HCI is evaluated for: -- commonality within a payload and across payloads (look and feel) -- compatibility of displays with procedures -- usability of displays and procedures

41. J. Fittipaldi / EO20 Crew Ops 256.961.0230 41 Learning Point #4 Usability Verification Usability Test - Think(ing)-Aloud Method -A technique in user testing whereby users are asked to speak their thoughts, feelings and opinions as they perform a task or scenario. -Can be performed during any stage of development -While the focus in user testing is observing how effectively/efficiently a user performs the required tasks (and not on how users believe they are performing), the user’s subjective comments are also useful: in understanding the user’s mental model/interaction with the product In cataloging the user’s expectancies in identifying mistakes and getting ideas about the causes for understanding how the interface could be improved to avoid problems

42. J. Fittipaldi / EO20 Crew Ops 256.961.0230 42 Learning Point #4 Usability Verification Test or Trial? Criteria: A usability test is performed when a payload has: –Facility-class or facility-integrated payloads –Complex installation requirements; unique crew interface –Multiple and/or complicated crew procedures –Displays that require crew decision-making or process control inputs; transactions –Interaction with other ISS facilities Test Examples: MSG (subrack p/l’s); FCF; WORF; EXPRESS (subrack p/l’s) Excluded from Usability Testing: TV/Photo Ops and Express software loads procedures; Mals/Correctives Assumed core skill Ops Lead ensures procedure validation; Subrack Ops Lead ensures delivery to PSIV (Express procedures)

43. J. Fittipaldi / EO20 Crew Ops 256.961.0230 43 Learning Point #4 Usability Verification Test or Trial? (cont.) –The usability test is structured to emulate the crew on-orbit operational setting to the extent practical. Assumptions: –Crew displays are mature, candidate flight type –Crew procedures have completed Tabletop – Availability/access to flight-like payload hardware –For displays-only usability testing, only the displays being assessed need be flight-like.

44. J. Fittipaldi / EO20 Crew Ops 256.961.0230 44 Learning Point #4 Usability Verification Test or Trial? (cont.) Criteria: A usability trial is primarily a cognitive walkthrough, often conducted without access to flight-like h/w or dynamic displays. Trials can be performed when a payload incorporates: –simple or no installation requirements; minimal crew interface –straightforward, easily-understood crew procedures –displays that require minimal, uncomplicated crew decision- making or control input transactions –little or no interaction with other ISS facilities Trial Examples: SEM Satchel; EMMYS; Yeast-GAP; Iris

45. J. Fittipaldi / EO20 Crew Ops 256.961.0230 45 Learning Point #4 Usability Verification Usability Trial Payload hardware and operational complexity are diverse; usability trial methods should be tailored to the unique attributes of the payload –PD resource constraints should be factored in –Flight drawings and/or flight article still photos in lieu of hands-on hardware –Displays can range from static drawings to flight-fidelity interactive displays (e.g. exe files) –What is important is that the method and materials used be appropriate and sufficient to yield accurate results

46. J. Fittipaldi / EO20 Crew Ops 256.961.0230 46 Learning Point #4 Usability Verification The Usability Verification Plan, Readiness Review, Event and Report

47. J. Fittipaldi / EO20 Crew Ops 256.961.0230 47 Learning Point #4 Usability Verification Verification Event Planning Once the verification method assessment is approved, the Usability Test Plan will be distributed to the usability team. It will include the: –Time, date and location of the test/trial –Event objectives –Test setup (hardware, software, unique configurations, environment, conditions, etc.) –Methodology (including a list of the procedures to be verified), and –Instructions for collecting data and reporting results, recommendations, and action items –Mandatory test team participants: PDRT POC, POIF Ops Lead, Astronaut Office Rep, Astronaut Office Test Participant, PD Team POC; Other SMEs (as req’d); Event observers.

48. J. Fittipaldi / EO20 Crew Ops 256.961.0230 48 Learning Point #4 Usability Verification Verification Event Planning (cont.) Astronaut Office Support –Prior to the verification event, PDRT will notify JSC-CB of initial planning and request astronaut office participation Usability Testing at JSC: –Usability Tests at JSC Building 9/PDL-I; PDL-ll; and Facility training units (e.g. WORF); Building 5/ Rack Buildup Area; SSTF; SSC Lab (Sonny Carter)/Test Bed 1C Usability Testing at MSFC: –Payload development and training facilities frequently used Building 4493 / SSITF / MSG EU/GU; other facilities Building 4663 / Lab Training Complex (LTC) Usability Testing at KSC: SSPF / Offline Lab Usbility Testing at PD Site

49. J. Fittipaldi / EO20 Crew Ops 256.961.0230 49 Learning Point #4 Usability Verification Readiness Review Completed prior to the test / trial; PDRT POC will show the following criteria have been fulfilled:  Test location – reserved  Validated crew procedures - complete  Test hardware - complete  Flight candidate crew displays - complete  Usability test/trial verification plan – concurred by test team  Qualified test participant - confirmed  Additional test support participation – confirmed

50. J. Fittipaldi / EO20 Crew Ops 256.961.0230 50 Learning Point #4 Usability Verification Usability Verification Event Report The PDRT POC submits a Usability Verification Report to the mandatory participants within five working days of the completion of the test. The report shall include: –Time, date and location of the test –Contact information for the test participants –Description of the test setup; anomalies (if any) –Summary of test results –Test comments/status matrix incl. actions/recommendations/questions –Timetable and instructions for disposition of all actions Within 10 days of its release, the PD POC/Ops Lead will respond to the Report action item (“Will Implement” / “Will Not Implement” + Rationale).

51. J. Fittipaldi / EO20 Crew Ops 256.961.0230 51 Learning Point #5 Display Baselining Final Display Baseline

52. J. Fittipaldi / EO20 Crew Ops 256.961.0230 52 Learning Point #5 Display Baselining POIF-2001, PODF Management Plan Section 4: Displays shall be submitted for baseline by ECR submission to the US PODFCB POIF-1007, POIF Configuration Management Plan US PODF Payload Displays are approved by the EO20 PODFCB Board Chair Final crew display are included as Cadre Operation Products & Tools POH Vol. 1, SOP 9.4.4 describes the display baseline process: Groundrules: –All action items identified at the usability event have been implemented or resolved –Payload displays are submitted by a negotiated due date (I-Minus 10, approx.) –The display baseline process locks down only the crew GUI; not the software code –PD will submit a final, complete, clean set of displays (static screen shots) ECR Form –An ECR number may be obtained from the ECR Log Book in the EO20 Office. Form can be found at: https://poif.msfc.nasa.gov/station/cm/cm_index.htmlhttps://poif.msfc.nasa.gov/station/cm/cm_index.html

53. J. Fittipaldi / EO20 Crew Ops 256.961.0230 53 Learning Point #5 Display Baselining Prior to final baseline ECR submittal crew displays, including DGCS deviations (if any) and display development history charts are submitted in a CR in WorkFlow for IDAGS Concurrence. On CR approval (ODFCB Chair), final displays are submitted in an ECR to POIF-CM for final baseline approval. No TCM is required to baseline displays; the usability verification stands in lieu of the TCM The ECR package is signed out of board (PODFCB Chair) Upon approval, CM will enter the final displays in the POIF CM Master List.

54. J. Fittipaldi / EO20 Crew Ops 256.961.0230 54 Summary Per ISO 9241-11, what are the 3 key characteristics that define payload usability? True or False: The PDRT is responsible for ensuring the usability of only payload displays. Are payload usability and utility equally important? Why? Name two things that payload usability is not. True or false: Crew time is the most critical resource aboard ISS True of False: ‘Commonality’ means that payload displays must look identical. True or False: Compliance with the DGCS program document is more important than assuring usability. As a goal, display screen densities should not exceed _____%. The Exit primary navigation button (circle one) is / is not a required display component. The colors ____ and ____ are reserved and not for use in payload displays. True or False: Color coding by itself is acceptable A usability trial may be performed when (specify at least one criterion). True or False: A usability trial requires the PD to provide a set of dynamic displays. True/False: In the Usability Verification Report all action items and recommendations must be resolved / addressed prior to closing the report.

55. J. Fittipaldi / EO20 Crew Ops 256.961.0230 PDRT SharePoint: Payload Displays 101 Site https://poifsps.ndc.nasa.gov/branches/EO20/PDRT/PDRT/Payload%20Displays%20101.aspx One stop shop for payload display process, standards, & resources For access to the POIF SharePoint, contact Leigh Weston at leigh.weston@nasa.govleigh.weston@nasa.gov Overview & Process Standards Guidelines Resources Deliverables POCs

56. J. Fittipaldi / EO20 Crew Ops 256.961.0230 56 Displays Overview Course Questions?

57. J. Fittipaldi / EO20 Crew Ops 256.961.0230 57 Abbreviations & Acronyms CCBD COTS DGCS DRAG ECR.exe HCI HFIT IDAGS IML-1 IP IP/P iPV ISO ISS Change Control Board Directive Commercial Off-The-Shelf Display & Graphics Commonality Standard Display Requirements Applicability Guideline Engineering Change Request Executable file Human-Computer Interface Human Factors Implementation Team International Display & Graphics Commonality Standard International Microgravity Laboratory -1 International Partner International Partner/Participant International Procedures Viewer International Organization for Standardization International Space Station

58. J. Fittipaldi / EO20 Crew Ops 256.961.0230 58 Abbreviations & Acronyms OpNom PCB PD PDRP PDRT PIMS p/l PODF PODFCB POIF PSIV PTDR PTP SE Operations Nomenclature Payload Control Board Payload Developer Payload Display Review Panel Payload Display Review Team Payload Information Management System Payload Payload Operations Data File Payload Operations Data File Change Board Payload Operations Integration Function Payload Software Integration Verification Payload Training Dry Run Payload Training Panel Simulation Engineer

59. J. Fittipaldi / EO20 Crew Ops 256.961.0230 59 Abbreviations & Acronyms SSITF TCM TIM TST XML Space Station Integrated Test Facility Technical Change Memo Technical Interchange Meeting Training Strategy Team XMetaL