1. © 2003 Gina Joue & Brian Duffy Dr. Brian Duffy brd@media.mit.edu

2. © 2003 Gina Joue & Brian Duffy A Flexible Framework for All Socially engage each other on an equal footing by encouraging and facilitating minority groups to seamlessly integrate with everyone through universal design and a modular peripherals approach The Multimodal Approach Flexible integration between sensor modalities: Fusion Flexible integration of actuator modalities: Fission

3. © 2003 Gina Joue & Brian Duffy Multimodal Fusion Typing Handwriting Pushing and clicking Gloved hand gestures Speaking Body movement Head movement Free hand gestures Facial expression Eye movement Hand pressure Muscle activity Body stress levels Brain activity Position / Motion Audio Video Tactile / Force Skin Conductance Neural Voltage keyboard tablet mouse glove microphone camera haptic device EEG EMG GSR

4. © 2003 Gina Joue & Brian Duffy Plug′n PLAY Flexible Hardware :: Social Interfaces :: Play with Reality :: Active Chair Robotics Virtual & Augmented Reality

5. © 2003 Gina Joue & Brian Duffy The Active Chair :: Design Air System 3 pneumatic cylinders Variable air speed controllers Pressure controllers Solenoid valves Control System USB system using a PIC controller (PIC16F87x) with Serial Communications Interface in full duplex mode (asynchronous) & a USB interface circuit Design Holds the occupant in a minimal-stress posture between the neutral body posture of zero gravitation and a normal posture of relaxation on a flat, level surface (based on NASA experimentation).

6. © 2003 Gina Joue & Brian Duffy The Active Chair :: Control

7. © 2003 Gina Joue & Brian Duffy The Active Chair :: Vicarious Adrenaline Can you really know how I feel? … where the OBSERVER becomes the PARTICIPANT OBJECTIVE: the transference of adrenaline- based motor sport experiences to an observer through a complex inter-modal interaction.

8. © 2003 Gina Joue & Brian Duffy The Active Chair :: Vicarious Adrenaline “I used the term ‘rodeo-chair’. It is embedded in therapeutic connotations that are attached to the medical viewpoint of disability. However, the haptic chair uses a reality that involves a playful atmosphere. This maze allows the mind to be in tune with an exterior force. It can be set in a virtual environment or ultimately in a social context. If the body is paralysed, it does not mean it has to curl up. It needs exercise. The mind is in the driving seat.” JAMES BROSNAN

9. © 2003 Gina Joue & Brian Duffy Immersive Play for All :: TRUST Installed at the KK Women & Children’s Hospital in Singapore in 2005

10. © 2003 Gina Joue & Brian Duffy Social Interfaces :: JoeRobot OBJECTIVE: To develop a robust modular robot for explicit social interaction between a robot and one or more other robots, virtual avatars or humans The ideal is a balance between developing a system that can realistically meet the expectations of the people it interacts with while still embracing its machine-like qualities that make it ultimately useful

11. © 2003 Gina Joue & Brian Duffy Social Interfaces :: JoeRobot Articulation When it moves it comes “alive” Perception Vision: Colour tracking, feature recognition Localisation Biosensors Communication Speech recognition & synthesis Gestures Control Modular code structure Behaviour libraries Vision algorithms Communication protocols Software interfaces (presentation) Embedded systems (e.g. speech)

12. © 2003 Gina Joue & Brian Duffy Play with Reality :: Seamless Worlds Seamless integration of realities Judicious synthesis between the real & the virtual Manage how Virtual & Augmented Reality can become a powerful medium in our social play fuse the REALGAME

13. © 2003 Gina Joue & Brian Duffy Make it real … A strong need to develop flexible modular designs to promote real interaction regardless of abilities, age, or situation. SENSE – FEEL –EXPERIENCE The integration of multi-sensory stimulation creates a highly immersive environment that helps facilitate motor control rehabilitation. “… allows the mind to be in tune with an exterior force.” James Brosnan