Bio-Cybernetic Control for WBI: a bird’s eye view Prof. A Bio-Cybernetic Control for WBI: a bird’s eye view Prof. A. Taleb-Bendiab School of Computing Liverpool John Moores University email: a.talebbendiab@livjm.ac.uk http://www.cms.livjm.ac.uk/taleb Bio-cybernetic Control of Whole Body Interaction In the context of the Whole Body Interaction (WBI), this talk will focus on the technical challenges, requirements and conceptual frameworks for bio-cybernetics control. The talk will start with a brief overview of emerging requirements for WBI, which will be followed by a review of the state-of-the-art relevant to bio-cybernetics control for WBI including: ambient intelligence, body area sensor and actuator networks, cognitive systems and middleware. This will be followed by a presentation of the underpinning technologies, and a detailed description of a conceptual framework for bio-cybernetics control. Real-life examples will be used to illustrate the contents of this presentation. The talk will conclude with a discussion on remaining research challenges relevant to WBI. Prof. A. Taleb-Bendiab, talk: Whole Body Interaction Workshop’07, Date: 01/04/2017, Slide: 1

Outline In the context of the Whole Body Interaction (WBI) Definitions Biocybernetics, control, Augmented Reality, WBI Drivers for WBI Overview of emerging requirements for WBI, Technical challenges, requirements State-of-the-art relevant to bio-cybernetics control for WBI including: Bio and socio-inspired computational models ambient intelligence, Autonomic Computing Middleware body area sensor and actuator networks, cognitive systems Underpinning technologies, Conceptual frameworks for bio-cybernetics control Conclusions Open questions research challenges relevant to WBI Q&A Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 2

Some Definitions #1 Biological Cybernetics (Bio-Cybernetics) … is the application of cybernetics theory and systemics principles to further our understanding of systems biology integrate different levels of information and information processing to understand biological systems function and development of cellular, multicellular systems, organisms, complex living systems and ecosystems. [http://www.biological-cybernetics.de/wiki/index.php/Main_Page] [Source: Wikipedia] Cybernetics the study of control and communication in the animal and the machine including: organization, information flows, control Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 3

Some Definitions #2 Augmented Reality: …. combination of real world and computer generated data uses of live video imagery which is digitally processed and "augmented" by the addition of computer generated graphics. Advanced research includes the use of motion tracking data, etc.., and the construction of controlled environments containing any number of sensors and actuators. [http://en.wikipedia.org/wiki/Augmented_reality] Mixed reality: refers to the merging of real and virtual worlds to produce new environments and visualisations where physical and digital objects co-exist and interact in real time. [http://en.wikipedia.org/wiki/Mixed_reality] Virtual Worlds: “… simulated environment intended for its users to inhabit and interact via avatars …” [http://en.wikipedia.org/wiki/virtual_world] Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 4

Socially Augmented Intelligent PICT Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 5

Convergence of Real and Virtual Worlds: Second Life hand-held and wearable systems that act as gateways between the real and virtual worlds Eyewear, display badges and speakers worn about the neck will allow us to live more fully through our avatars a wearable box that creates a 3D sound field that allows the wearer to hear voices from the virtual world without completely shutting out the real people around him Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 6

Vodafone in Second Life More recently Vodafone now allows you to interact with your real phone via a Second Life avatar of the phone The real phone can be answered from within SL Text messages can be sent and received between real-world and SL Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 7

Sensors that Control Second Life Objects Sending external data into virtual worlds such as Second Life Using Potentiometer the cube can be controlled Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 8

Importance of WBI WBI Technical Requirements Interface Transparency and virtualisation Systems and Processes Interaction Behaviour Structure/organisation Affective Sensing and Actuation Co-evolve Technical Requirements Protocols for WBI Standards and legacy systems Programming, Interaction and Control Models Usability Assurance QoS Safety and Security, Etc. Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 9

Example: Bio-Cybernetics Control Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 10

Virtual Lab Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 11

IBM Sensors Link Real and Virtual Sensor data is visualised in real-time Blue balls with white designs represent active Bluetooth devices Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 12

The Story so Far #1 State-of-the-art in Situated AmI including; autonomic software models and architecture, standards tools and techniques to support the design, modelling, analysis and evolution of autonomic software Define models for their programming, control interaction models with human and/or other non-AC systems (legacy). Delegation of authority and its adjustment Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 13

The Story so Far #2 Has been informed by a set of design paradigms Model-based vs self-organising systems design models Top-down vs bottom-up Applying and/or revisiting: cybernetic principles control systems theory, regulation, reward and sanctions Decision theory, DAI and CI dynamic planning, deliberative models, ML Middleware support self-awareness, reflection and deliberation etc. Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 14

Research Challenges #1 Social and other requirements Conceptual Design and Management models “… how do we design, build, and evolve such systems so that they can meet given—and evolving—requirements ...” Incremental deployment of SAmI capabilities in legacy systems. Hot-swapping, Dynamic AOP-based evolution, Interoperation Programming, control and Interaction Models Usability and Interface Balancing and adjusting governance and autonomy Social and other requirements Not covered here Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 15

Cybernetics: The Viable System Model Beer’s VSM implements a control & communication structure via hierarchies of homeostats (feedback loops) (1950) Defines 6 major systems ensure ‘viability’ of the system Implementation S1 Monitoring S2 Audit S3* Control S3 Intelligence S4 Policy S5 Offers an extensible, recursive, model-based architecture, devolving autonomy to sub-systems Autonomic Systems Cognitive Systems Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 16

Situated AmI Capabilities A software system is autonomic, if it possesses the following capabilities: Self-configuring — choosing a suitable behaviour, based on user preferences, context, … Self-tuning — choosing behaviours that optimize certain qualities (performance, year-end profits, …) Self-repairing — shifting execution to another behaviour, given that the current one is failing Self-protecting — choosing a behaviour that minimizes risks (attacks, viruses, …) Self-knowledge - Detailed knowledge of constituent components, current status, etc. Self-configuration/re-configuration - Adjustments to a changing environment Self-optimizing - Monitor constituent parts and optimize accordingly Self-healing - The ability to recover from malfunction Self-protecting - Detect, identify and protect itself from attack Environmentally aware – know its environment, the context surrounding its activity and act accordingly Co-operative – Interact with other systems in a heterogeneous world – open standards Anticipatory – anticipate and transparently implement the resources required to meet user goals Self configuring: Automated configuration of components and systems that follow high-level policies. The rest of the system adjusts automatically and seamlessly. Self-healing: The System automatically detects, diagnoses, and repairs localized software and hardware problems. Sometimes also reintegrating the repaired resource back into itself. Self-optimization: Components and systems continually seek to improve their own performance and efficiency. Self-protection: The system automatically defends against malicious attacks and cascading failures. It anticipates and prevents system-wide failures. Self monitoring: The system continually measures its current state to be compared with the desired state. Environment aware: The system periodically monitors and measures its related environment to have a suitable action against threats or different circumstances. Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 17

Our Approaches #1 Related Work: Our model – J Reference IBM blueprint (www-03.ibm.com/autonomic/pdfs/ACBP2_2004-10-04.pdf) An autonomic manager contains a continuous control loop that monitors activities and takes actions to adjust the system to meet business objectives Autonomic managers learn from past experience to build action plans Elements need to be instrumented consistently, based on open standards Our model – J Reference Gauges for dynamic assembly and adaptation Requirements / Event-based architectural models (Before) Run-time performance adaptation (After) Model -- Implementation mapping / Transformation (During) Ontology-based gauges Run-time Configuration Open Points Connectivity Constraint Gauges Reconfiguration Cost Early Warning Dependability (for Security) Runtime Event Monitoring Architecture Transformation Technology Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 18 Evolution and Integration Command Center Performance-based Architectural Adaptation Architecture-driven Dynamism

Projects in the Networked Appliances Lab. Composition of an Internet of Things Infinitely Adaptable Gaming Bridging between Physical and Virtual (Demo) Managing 3D Multimedia Streams Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 19

What It Looks Like Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 20

Prototype Implementation Paul Fergus, David Llewellyn-Jones, Amjad Shaheed, Madjid Merabti, Abdennour El Rhalibi, Networked Appliances: Manipulation of 3D Multimedia Streams, Accepted for publication in the 5th IEEE International Consumer Communications and Networking Conference: Workshop on Networking Issues in Multimedia Entertainment, 2008, Las Vegas, USA Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 21

Augmenting Frames Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 22

Modelling Autonomy Our model Evolving and Adjustable Autonomy Gauges for dynamic assembly and adaptation Our model Using SSC used to formalizes the behaviour of dynamically changing systems FOL (McCarthy, 1963).. Support concurrent actions and timing constraints. Each situation can be viewed as a history of previous actions. Action, guards and time can be modelled at deliberation points in an autonomic setting. M. Randles, A. Taleb-Bendiab, Philip Miseldine, Andy Laws, "Adjustable Deliberation of Self-Managing Systems", ECBS 2005: 449-456. [ppt] Evolving and Adjustable Autonomy Software governance (Control) via Formal modelling of norms, policies Enactment support – from spec. to code using Neptune language Requirements / Event-based architectural models (Before) Run-time performance adaptation (After) Model -- Implementation mapping / Transformation (During) Ontology-based gauges Run-time Configuration Open Points Connectivity Constraint Gauges Reconfiguration Cost Early Warning Dependability (for Security) Runtime Event Monitoring Architecture Transformation Technology Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 23 Evolution and Integration Command Center Performance-based Architectural Adaptation Architecture-driven Dynamism

Programming Autonomic Systems Neptune Meta-Language Integrated Development Environment: Miseldine, P., Taleb-Bendiab A. A Programmatic Approach to Applying Sympathetic and Parasympathetic Autonomic Systems to Software Design. to appear in the 2005 International Conference on Self-Organization and Adaptation of Multi-agent and Grid Systems (SOAS’2005). Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 24

Conclusions and Future Work Understanding WBI Interface Transparency and virtualisation Systems and Processes Interaction Behaviour Structure/organisation Affective Sensing and Actuation Co-evolve Address the challenges including: Protocols for WBI Standards and legacy systems Programming, Interaction and Control Models Usability Assurance QoS Safety and Security, Etc. Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 25

Acknowledgements Acknowledgements My thanks to All CMS Research Team  Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 26

That’s the end – so I’m off ! Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 27

Extra Slides … Prof. A. Taleb-Bendiab, talk: Whole Body Interaction’07, Date: 01/04/2017, Slide: 28