Usability Evaluation of e-Navigation MSP 2(NAS) 2016. 3. 14. Yung Ho, YU Professor of Korea Maritime and Ocean University http://itship.hhu.ac.kr , yungyu@hhu.ac.kr IlSik, Shin Researcher of RIMS(Research Institute of Medium and Small Ship) http://www.rims.re.kr, issin@rims.re.kr

MSP : e-Navigation IT Service Contents Introduction MSP : e-Navigation IT Service What is the HCD IMO HCD Assessment-A NAS Tool for HCD Assessment 요것도 이쁜걸루 수정합세다 HCD Assessment utilizing Tools Result and Proposal

Introduction e-Navigation Systems MSP Services Evaluation Tool Usability evaluation of e-Navigation systems and MSP required Usability of MSP should be verified before adopted in real service Requirement of each MSP should be defined in standards Usability evaluation scale =? Efficiency Effectiveness Satisfaction Study progress on usability evaluation for verification of MSP Usability evaluation using ship simulator Usability evaluation using evaluation tool MSP2 Service(NAS) Verification of NAS

MSP : e-Navigation IT Service Service description MSP1 VTS Information Service (INS) MSP2 Navigation Assistance Service(NAS) MSP3 Traffic Organization Service(TOS) MSP4 Local Port Service (LPS) MSP5 Maritime Safety Information (MSI) service MSP6 Pilotage service MSP7 Tugs service MSP8 Vessel shore reporting MSP9 Telemedical Maritime Assistance Service(TMAS) MSP10 Maritime Assistance Service (MAS) MSP11 Nautical Chart Service MSP12 Nautical publications service MSP13 Ice navigation service MSP14 Meteorological information service MSP15 Real-time hydrographic and environmental information services MSP16 Search and Rescue(SAR) Service Service description Particulars IT service description MSP2 Navigation Assistance Service(NAS) Route Plan Exchange Service Evaluate usability of route exchange service using IMO HCD criteria

Evaluation of Route Plan Exchange Service Based on HCD What is the HCD HCD (Human Centered Design) HCD is the set of activities to achieve usability goals in e-Navigation systems and IT services U-TEA is used to ensure good usability in HCD HCD, SQA, U-TEA guidelines are considered concurrently SQA helps ensure S/W reliability in e-navigation Usability evaluation process based on U-TEA activities Results of usability evaluation Effectiveness : The rate of mission success (Normally 80%) Efficiency : The planned time on mission accomplishment Satisfaction : Subjective evaluation through surveys and interviews with participants 1. Planning Timing 2. Preparation Setting of tasks Selection of test participants (if required) Preparation for tests and selection of scenarios 3. Implementation Implementation and record 4. Analysis Evaluation and analysis Report of test results 5. Feedback Iterate if required (whether formative or summative approach is used) Evaluation of Route Plan Exchange Service Based on HCD

Route schedule node information IMO HCD Evaluation – Route Plan Exchange Route Plan Exchange Using NAS NAS uses Route exchange to recommend and optimize a ship’s route IEC 61174 Electronic chart and information system(ECIDS) Example of distance of waypoint 2 and waypoint 3 Example of parameters that belong to waypoint 3 section XML Compressed File The extension .rtz Route Plan Exchange format description(IEC 61174) 1 2 3 Route node information Waypoint node information Route schedule node information IEC 61162-450 transmission

IMO HCD Evaluation – Route Plan Exchange Route Plan Exchange IT Service Usability evaluation for verification of NAS Full Mission Ship-Heading Simulator used Proposed simulation process Navigation Route Plan Vessel Traffic Center(VTS) Using AIS or VHF Equipment Route information distribution When route plan information of other ships is not provided When Route plan information of other ships is provided ECDIS or INS Display

IMO HCD Evaluation – Route Plan Exchange Information of simulation navigators Company Embarkation career 1st grade license-holder A Panstar-line 11 year 1st grade license-holder B HYUNDAI Shipping 7.5 year 3rd grade license-holder C HANJIN Shipping 6 month 3rd grade license-holder D Sangji Shipping 9 month Simulation Ports Container ship 52,000t LOA : 222.2m Breadth : 30.m Busan port Osaka port effects of wind and tide Wind Tide Direction Speed Busan port 280° 10 knots 100° 0.5 knot Osaka port 053° 233°

IMO HCD Evaluation – Route Plan Exchange Result of simulation Busan port Osaka port 1st grade license-holder A Without NAS Collision (O) With NAS No collision (X) 1st grade license-holder B 3rd grade license-holder C 3rd grade license-holder D Total number 4th Usability evaluation scale HCD Effectiveness Proximity distance evaluation to dangerous line Ruder and engine controllability evaluation Usability Evaluation Efficiency X Satisfaction Likert –Type subjective evaluation

IMO HCD Evaluation – Route Plan Exchange The proximity distance evaluation of Busan Port The proximity distance evaluation of Osaka Port Result of proximity distance Result of proximity distance A D B C B C A D Provided IT service Not provided IT service Provided IT service Not provided IT service

NAS doesn’t improve Control IMO HCD Evaluation – Route Plan Exchange The controllability evaluation of Busan Port Rudder Angle Engine RPM Average(degree) Allowance(%) Average(rpm) Captain A 4.8689 86.09 34.4754 65.52 Captain B 4.2131 87.96 50.1148 49.89 3rd Mate C 8.2951 76.30 29.5082 70.49 3rd Mate D 5.0492 85.57 49.9672 50.03 With NAS, Without NAS The controllability evaluation of Osaka Port Rudder Angle Engine RPM Average(degree) Allowance(%) Average(rpm) Captain A 7.0984 79.72 52.3607 47.64 Captain B 6.3559 81.84 57.5082 42.49 3rd Mate C 6.2143 82.24 45.6071 54.39 3rd Mate D 11.6393 66.74 54.1639 45.84 NAS doesn’t improve Control

IMO HCD Assessment – Route Plan Exchange Subjective Assessment Item Item 3 2 1 -1 -2 -3 Control Risk Very Safe Somewhat Not sure Unsafe Psychological Burden Comfortable Uncomfortable Subjective assessment of IT service Officer comments Route Plan Exchange information help to reduce psychological burden Notice of relatively dangerous vessels is helpful to improve safety of navigation Too much dependence on NAS information makes it difficult to cope with unexpected situations Not helpful during emergencies situations due to embarrassment of Mariner Control Risk Psychological Burden Not IT service provided IT service provided

Need to develop assessment tools for IT service Tool for HCD Assessment User requirements of assessment tool for IT Service Ship Handling Simulator Easy to edit and operate IT service scenario Possible to control own ship Easy to change navigational environment Allows recording, stopping & replaying Play with smooth movement and realistic navigational environment on the PC Vital equipment onboard should be implemented with inter relational operation Full Mission Navigational Simulator Understanding and education for complex dynamics of vessel Expensive equipment Operation requires a lot of manpower Test preparation takes a long time Need to develop assessment tools for IT service

Simulation Tool for HCD Assessment Simulator Configuration for assessment of NAS 3D Navigation visualization Radar ECDIS displaying Route Plan Exchange for NAS

HCD Assessment utilizing Developed Simulator Route Plan Exchange Carried out additional simulation which is not easy with FMSS What happened when route changed suddenly different from informed planned route? What should be done to increase efficiency of Route Plan Information Exchange? Assessment Tool System The usability test for Route Plan Exchange Scenario Class Information Simulation No. 1 Radar, Route Plan Exchange Route of the ship with the risk of collision not changed Simulation No. 2 Route of ship with the risk of collision changed without correcting information External Environment Sea Status 3, clear day Busan port entry conditions (initial speed of 10 knots ) Effectiveness , Efficiency and Satisfaction Evaluation

HCD Assessment utilizing Developed Simulator Object of experiment License class Boarded Career Position Respondent 1 1st 12 years Master 7 3rd 6 month Trainee 2 8 years 8 3 7 years Chief Mate 9 4 2nd 6.5 years Second Mate 10 5 3 years 11 6 12

HCD Assessment utilizing Developed Simulator Scenario No.1 Nine ships participate in simulation One of the nine ships is scheduled to be collided with own ship No change ship’s route with the risk of collision Route (Ship with risk of collision) Planned Own ship route Expected collision point ECDIS displaying Route Plan Exchange

HCD Assessment utilizing Developed Simulator Scenario No.2 Nine ships participate in simulation One of the nine ships is scheduled to be collided with own ship Change ship’s route with the risk of collision Route(own ship) Route (Ship with risk of collision) Expected collision point ECDIS displaying Route Plan Exchange

HCD Assessment utilizing Developed Simulator Criterion of usability test Test Item Test Method Effectiveness Mission achievement Achievement rate of avoiding risk of collision Distance from the ship with risk Measure the nearest distance between own ship and ship at risk for collision Controllability during a specific time Control ratio of ship handling during 7-13 minutes Satisfiability Satisfaction of the NAS provided Questionnaire Survey about satisfaction of service provided (Likert Type) Time measurement for mission accomplishment(scheduled time) is excluded

HCD Assessment utilizing Developed Simulator Effectiveness Simulation No.1 Simulation No.2 Mission achievement 58% 67% Distance from the ship with risk (avg.) 166m 159m Number of Control during Simulation No.1(No route change) Simulation No.2(Route change) The number of total uses average rate the number of total uses ± 3 minutes from the time to collision (7 ~ 13 minutes) 75 12.5 60% 72 12 63% Exclusion of ± 3 minutes 50 4.3 40% 44 4.4 37% Total period 125 7.2 100% 116 7.8 Simulation No.1 Simulation No.2 -3 +3 (minutes) -3 +3 (minutes) Simulation start time Estimated time to collision Simulation start time Estimated time to collision

HCD Assessment utilizing Developed Simulator Likert-Type Assessment Scale for the Service satisfaction Scale of helpfulness for safe navigation Strong Agree Don’t know Disagree 5 4 3 2 1 Satisfaction Result Navigator Utilization of Service Helpfulness for the safe navigation Answer Level Navigator 1 X Agree 4 Navigator 2 ○ Strongly Agree 5 Navigator 3 Disagree 2 Navigator 4 Navigator 5 Navigator 6 Navigator 7 Navigator 8 Navigator 9 Navigator 10 Don’t know 3 Navigator 11 Navigator 12 Average 75% 2.67 – NAS does not improve safe navigation

Result of HCD Assessment Low mission success rate (each 58%, 67%) Too close from the risk (average distance 160m) In case of Simulation No. 2(route change without correcting route information) rudder command commenced faster rather than Simulation No. 1 When collision is imminent, rudder command commenced. Too low service satisfaction in questionnaire survey (disagree 2.67) Proposal Provision of route planning exchange should be done in real time. Correction Route change should be planned before making the change Only display route of ship with collision risk to avoid screen clutter. Mariner should not depend on route exchange. Route should be shared between large and medium-small ships. Based on the evaluation results various studies, experiments, education and training of mariner for practical application should be done.

The OCEAN IS PAST, PRESENT AND FUTURE, ALWAYS BLUE OCEAN. . Thank you for Listening.