1. www.CrewAssistant.com 1 The MECA Project Reasoning Agents on Mars Leo Breebaart (S&T) 12 October 2006

2. www.CrewAssistant.com 2 The MECA Consortium TNO Human Factors Human behaviour and performance in technical high-demand environments; methods to attune the environment to (momentary) human capacities. S&T bv Software development company with specific expertise on creating system health management applications. OK Systems Software development company focusing on technology areas of AI, user interfaces, databases and web-based systems (specially scheduling systems). EADS-ST Technologies, development, production and utilization of manned and unmanned space missions, including experiments, space transportation systems, propulsion systems and support of these systems concerning operations, maintenance and mission handling.

3. www.CrewAssistant.com 3 Project Goal The MECA Project aims to: Provide support to and increase autonomy of astronauts while: –Executing complex tasks –In a hostile and large unknown environment –Possibly disconnected from Mission Control

4. www.CrewAssistant.com 4 Objective & Vision Objective: support mission goals (without injury or loss of life) by empowering the cognitive capacities of human-machine teams during planetary exploration missions. in order to cope autonomously with unexpected, complex and potentially hazardous situations. Vision: crew support that acts in a ubiquitous computing environment as electronic partner, helping the crew –to assess the situation, –to determine a suitable course of actions to solve a problem, –to safeguard the astronaut from failures.

5. www.CrewAssistant.com 5 MECA and Agents Where does agent technology fit in? A MECA System can be considered as an instance of a group of software agents. The MECA Architecture will make use of the outcome of agent technology research.

6. www.CrewAssistant.com 6 This presentation Phase 1 (2005-2006) –MECA 2017 –RB: Requirements Baseline Phase 2 (2006-2007) –MECA 2007 –Demonstrator Prototype –Refined RB

7. www.CrewAssistant.com 7 Why the need for support? Current astronaut support rather antiquated. Long-distance manned explorations impose new challenges.

8. www.CrewAssistant.com 8 Background

9. www.CrewAssistant.com 9 State of the Art LAPAPSCOPE Our legacy:… But MECA should extend this…. Operational Procedures Hardcopy +

10. www.CrewAssistant.com 10 MECA design process Background Operability-based design Anticipate new Intelligent Interfaces Anticipate new HFE standard Approach Human-Operation centered Enabling Technology focus Iterative process (specify-test-refine cycles) From abstract to detailed specifications Sound theoretical and empirical foundation 10 -- 25 yrs

11. www.CrewAssistant.com 11 Example Scenario Human and System Health Management

12. www.CrewAssistant.com 12 EVA 2 astronauts 2km from habitat, sample collection for scientific experiment. Habitat = MECA unit

13. www.CrewAssistant.com 13 MECA indicates a problem with the temperature regulation of one of the space suits. Habitat

14. www.CrewAssistant.com 14 Astronaut, MECA and spacesuit collaborate on finding the cause of the failure. Habitat MECA interrogates spacesuit to obtain more parameters for diagnosis.

15. www.CrewAssistant.com 15 The heater of the space suit is damaged and cannot be repaired locally. Habitat

16. www.CrewAssistant.com 16 MECA simulates the consequences of the broken heater. Habitat MECA predicts that astronaut has to be brought to habitat and has a risk of fainting.

17. www.CrewAssistant.com 17 Astronaut's MECA Unit and Habitat MECA Unit reschedule and plan safe return. Habitat Habitat is preparing for treatment of hypothermic astronaut (preparing and activating resources) Astronaut MECA asks for help of transporting astronaut (since he is likely to faint)

18. www.CrewAssistant.com 18 MECA Rovers respond to call for help, MECA habitat chooses rover Habitat MECA habitat chooses rover that has enough power/resources to pick astronaut up and transport to habitat (fuel, location, speed, pressurized or unpressurized rover etc.).

19. www.CrewAssistant.com 19 MECA communicates adjusted schedule to astronaut(s) in the right manner (keeping in mind cognitive task load). Habitat Hypothermic astronaut has a high task load due to high stress levels, MECA Unit shall adapt communication according to task load and affective state. The astronaut accompanying him can be given information in a different manner.

20. www.CrewAssistant.com 20 Astronaut faints before rover arrives. MECA communicates fainting of astronaut. Habitat

21. www.CrewAssistant.com 21 Astronaut has fainted earlier than predicted, MECA rover has to find a way to pick up astronaut (sensemaking). Habitat

22. www.CrewAssistant.com 22 MECA of fainted astronaut and other astronaut collaborate to get fainted astronaut in rover Habitat

23. www.CrewAssistant.com 23 Hypothermic astronaut is transported to habitat by rover. Habitat

24. www.CrewAssistant.com 24 Hypothermic astronaut is in habitat being treated. Habitat

25. www.CrewAssistant.com 25 Iterative Requirements Analysis

26. www.CrewAssistant.com 26 Operational Demands MECA shall take account of: the high-level operation goals –e.g., safe return to earth the environment –e.g., radiation and social monotony task performance –e.g., people will get seriously ill

27. www.CrewAssistant.com 27 Human Factors Demands Cognitive Task Load Situation Awareness and Sense Making Diversity of Cognitive Capacities Trust and Emotion Collaboration Crew Resource Management Decision Making

28. www.CrewAssistant.com 28 Envisioned Technology (1) An infrastructure will be available for automatic distribution of data, software and reference documents.

29. www.CrewAssistant.com 29 Envisioned Technology (2) MECA shall make use of this infrastructure and can cope with possible failures Continuous analysis and extrapolation of emerging technologies, e.g. –multi-agent systems –automatic planning and scheduling –model-based health management Technical requirements, such as –maturity –graceful degradation –maintainability –fault tolerance

30. www.CrewAssistant.com 30 Requirements Baseline Generic task level requirements Implement key Human Factor knowledge. Enhance autonomy of individual actors and groups of actors. Support collaboration among the different actors. Hardware and software systems must be highly reliable. Manage environmental information.

31. www.CrewAssistant.com 31 Current Requirements Different types of requirements: –task level, –functional, –user interface, –technical interface, –operational and –technical requirements. All requirements are linked to use cases.

32. www.CrewAssistant.com 32 Use Case Template ID78 TitleHypothermic astronaut LevelLevel 0 GoalTreat hypothermic astronaut that is on EVA… ActorPersonal MECA, MECA habitat, astronaut in habitat, rover.. Pre-conditionAstronaut on EVA is hypothermic Post-conditionHypothermic astronaut is in medical facility being treated FrequencyNot frequent Main success scenario Personal MECA detects hypothermia and communicates to hypothermic astronaut … Alternative scenario.., Doctor is not in habitat, MECA will ask astronaut_1 to prepare… CommentDerived from RefDoc3 …

33. www.CrewAssistant.com 33 Outline of Functional Requirements ProcessMECA function Information Gatheringdetect needs for operations and training Goal Settingselect and prioritize goals for operations and training Plan Generation or Selectiongenerate plans, or select pre-generated plans and procedures, for operations and training Plan Evaluationevaluate operational and training plans Prepare for Executionprepare the resources for executing operational and training plans Executionexecute operational and training plans Processing Evaluation of Results evaluate execution results for operational and/or training purposes

34. www.CrewAssistant.com 34 Incremental and Iterative Prototyping Human-, task- and context-driven design and evaluation. Both MECA and the humans will show mutual adaptive behaviour, which effects should be well tested with realistic scenarios. Prototyping, simulation and testing is therefore essential to establish a sound and coherent set of requirements. A game-based simulation environment can provide an effective platform for testing the human-machine collaboration (e.g. the Unreal Tournament game- engine) in combination with other simulators.

35. www.CrewAssistant.com 35 Evaluation Criteria Long-term human in the loop effects Standard usability measures –effectiveness –efficiency –satisfaction –learnability Human experience measures, such as –situation awareness (perception, comprehension and projection) –trust (persistence and behavioural competence, servitude, and the understanding of the machine) –emotion (arousal and valence)

36. www.CrewAssistant.com 36 Break? Break!

37. www.CrewAssistant.com 37 Background Reasoning will involve complex system behavior and scarce resources

38. www.CrewAssistant.com 38 Background Time is a scarce resource Control of a wide array of tasks and experiments Control of each task and experiment is complex –Nominal -- Correct sequence of steps –Off-nominal -- Detect, isolate, and compensate failures Summary: –Optimize crew time by supporting control activities Check plan execution Support control of complex equipment especially in the case of malfunctions Resource usage

39. www.CrewAssistant.com 39 Human decision making Process Under Control Understand Act Situational awareness, e.g.: Warning: At current rates, resources will be depleted within two days… MECA Unit Must reduce load by 10% but must also generate more oxygen… I lowered the consumption, but I still dont see any change.. Complex, uncertain, and dangerous world plan

40. www.CrewAssistant.com 40 Traditional support Operational procedure: verify action Next action okay Not okay off-nominal procedures guided by mission control Based on operational procedure and Mission Control

41. www.CrewAssistant.com 41 Operational procedures Nominal control Off-nominal: Fault detection, isolation, repair Operational procedures: Huge pile of papers No feedback of payload

42. www.CrewAssistant.com 42 The 2017 MECA system The 2017 MECA system helps the astronaut to make the right decisions in situations that : –Are novel, or near-novel, e.g., because equipment is failing –Are complex, e.g., effects of the decision are hard to predict Intricate interactions between processes, systems, components, … The effects are noticeable only late in time –Require human--human and human--machine cooperation

43. www.CrewAssistant.com 43 Automated support Process Under Control Understand Act MECA Unit plan Improve situational awareness by interpretation measurement data, in particular: fault diagnosis Determine alternative plan (e.g., repair, reconfiguration) High level plan --> control action Monitor plan execution

44. www.CrewAssistant.com 44 Automated support Validate success of plan step Automatic fault detection and isolation Adequate repair procedures presentation of background info Automatic (re)plan

45. www.CrewAssistant.com 45 Automated support Operational procedure is generalized by Actions of a Plan –Plan describes pre- and postconditions –Postconditions used for verification plan step and diagnosis Automated diagnosis Automated reconfiguration (repair, or redundancy management) prepostprepostprepost Supply pressure to fuel and oxidizer Pb = high

46. www.CrewAssistant.com 46 Layered Reasoning

47. www.CrewAssistant.com 47 2017 MECA: Collaboration Process Under Control UnderstandAct PUC UnderstandAct I need your capabilities to repair the equipment Resource level from team mate is enough to complete task…

48. www.CrewAssistant.com 48 Plan-based Architecture

49. www.CrewAssistant.com 49 MECA Unit Functional Decomposition MMI procedure executor (PE) what-if simulation & rehearsal (PR) execution monitoring (SM) sense- making reconfiguration (CO) physical equipment/facility interface (DA) planning & scheduling (PS) derive capabilities (SA) health & status monitoring (FDIR) inter- face to other MECA units CTL resource manager health monitoring plan model status + history collaboration(simulated) PUC other MECA units other MECA units use cases

50. www.CrewAssistant.com 50 Software Architecture

51. www.CrewAssistant.com 51 MECA 2017 is not MECA 2007! For exampleMECA 2017 (Real MECA)MECA 2007 (Demonstrator) Software TechnologiesMake generic choices: ontologies, agents, intelligent reasoning support. Choose specific instantiations: OWL/RDF, Jade, Uptime. Hardware PlatformsWearable / ubiquitous / brain-jack interface... Tablet PC / Laptop. Autonomy of components in Process under Control Components can be independently autonomous (i.e. non-MECA intelligence). No (interface to) 3 rd -party intelligence, all autonomy implemented / controlled by MECA. ReasoningOptimal combination of proactive and reactive. Combination of proactive and reactive. Unit complexityCompletely hierarchical systems-of- systems nature of MECA Units. Only simple aggregation and limited levels of containment for MECA Units. CooperationAdaptable to dynamic leadership, based on model on teamwork. Limited models of teamwork. Cognitive task loadAware of task load based on physiological sensors, imposed loads, and actual astronaut reactions. Limit awareness of task load based on limited indicators.

52. www.CrewAssistant.com 52 Challenge Areas Data Architecture: how to ensure worldviews of different levels of MECA Units remain compatible? How is the worldview represented? (Unit) Collaboration: how to synchronize, authenticate, de-conflict, negotiate work share, re-plan when two MECA Units meet up? Good interfaces will be key!

53. www.CrewAssistant.com 53 Ontologies and reasoning Use ontologies to describe and reason about: –Design of MECA –Capabilities and Domains Equipment (PUC, Resources) Environment Tasks Time Communications etc... Formal ontology representation allows: –automated validation of instance data –generation of documentation, templates, code –fit in with future semantic web approaches

54. www.CrewAssistant.com 54 Ontology example @prefix xsd:. @prefix meca:. # OWL Classes: meca:Unit a owl:Class ; # Owl Properties: meca:isPersonalUnit a owl:DatatypeProperty; rdfs:domain meca:Unit; rdfs:range xsd:boolean. meca:autonomyMode a owl:ObjectProperty; rdfs:domain meca:Unit. meca:synchronisationPolicy a owl:ObjectProperty; rdfs:domain meca:Unit.

55. www.CrewAssistant.com 55 Planning & Scheduling - Design Concepts Mission objectives are predefined, and default general plans are arranged before mission begins. Short term goals and plans will be decided and controlled autonomously by crew during the mission, adapting to changing circumstances. Continuous planning, scheduling and rescheduling of the tasks assigned to a heterogeneous team is a complex process that consumes time and determines the chances of success. MECA should assist in the generation of plans, reduce the overload of crewmembers, check constraints and conditions, and optimize the usage of time and resources. Crewmembers should have quick and easy access to all information related with plans, be able to evaluate alternatives and make changes.

56. www.CrewAssistant.com 56 Planning & Scheduling - External Interfaces

57. www.CrewAssistant.com 57 Planning & Scheduling - Internal Components Rule system: editable by users, enabling explanation of decisions Inference Engine: using Rules to generate and optimize Plans and Schedules Representation: in a format that facilitates distribution & collaboration

58. www.CrewAssistant.com 58 Demonstrator – First Design Iteration Decompose scenario into Sequence Diagrams (bring time dimension into play, identify actors) Decompose scenario into Activity / State Diagrams Illustrate actor timelines with storyboards Identify order and depth to which storyboards are to be implemented

59. www.CrewAssistant.com 59 Demonstrator – Software Proposals Application Framework: Use EADS Java Framework for personal MECA Unit GUIs. Use Jade for implementing agent-aspects of Units themselves. Use Uptime for Model-Based Programming: autonomous planning, reconfiguration and fault diagnosis. Use RDF/XML+OWL for Knowledgebase. Use Students models for PuC Components...

60. www.CrewAssistant.com 60 Questions? Photo: ESA/Aurora