May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Evaluation of a robot as embodied interface for Brain Computer Interface systems L. Tonin, F. Piccione, S. Silvoni, K. Priftis, Emanuele Menegatti K. Priftis Department of General Psychology University of Padua, Italy IRCCS San Camillo, Venice, Italy Dept. Of General Psychology University of Padua Intelligent Autonomous Systems Lab University of Padua

2 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems What is BCI? There are four general steps in a BCI system: 1.Signal acquisition 2.Classification and recognition 3.Interaction with device 4.Feedback to Subject According to Birbaumer et all.(1999): “Brain Computer interfaces are systems that allow to translate in real- time the electrical activity of the brain, in commands to control devices“ Our target for the use of BCI system are patients severely paralyzed (locked-in) due to injury or disease

3 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Original P300-based BCI of IRCCS San Camillo This BCI system is based on P300 wave detection P300 is event-related potential Visual interface gives to patient stimulus to activate P300 signal The P300 detection will control the movement of cursor Directional cues are represented by the single flash of one out four target-direction arrows Participants focus their attention on the flashing arrow corresponding to the desired direction of motion Validated on 5 tetraplegic patients and 7 healthy patients [Piccione et al. Clinical Neurophysiology 2006]

4 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems P300-based BCI of IRCCS San Camillo Focusing own attention, patient will generate P300 signal:

5 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Goals of our project In this project we aim at investigating two main issues: Giving to locked-in patients the control of a physical actuactor to improve their rehabilitation experience Developping telepresence of the patient with the body of the robot as medium The questions are: Can the robot lead patient to full and better control of BCI via a higher patient engagement in the task? Might patients identify them-selves into the robot and think of it as an extension of their perceptions and actions (i.e. their body)?

6 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Robot as a physical actuator To be able to move a body again, might give new strength and will to the locked-in patients 1.Custom built holonomous robot with hexagonal structure and three omnidirectional wheels 2.The holonomous nature of the robot is very important to replicate the motion and the apparence of the interface used by our BCI The robot is fitted with: Omnidirectional camera Audio board WiFi connectivity

7 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems BCI and robot connection BCI system SERVER Robot Client Command Action result Interaction between robot and BCI is based on client-server system with TCP-IP protocol via IEEE Security about connection between BCI and robot is guaranteed by: System protocol and network protocol check data received; Virtual Private Network between BCI and robot with autentication and encryption

8 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Test and Demo We carried out many tests to integrate BCI and robot. These tests accomplished at IRCCS San Camillo in Venice, and in IAS Laboratory of University of Padua: Robot and BCI computer connected with wired network Testing healthy subject performances to move robot BCI and robot connected in larger network with VPN connection (robot was in Padova, and BCI system in Venice)

9 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Test and Demo During the scientific festival Discovery on Film in Rovereto (May 2008). A healthy subject at San Camillo in Venice was able to move the robot using the BCI at museum of Rovereto (about 200 Km from Venice). The test was successful and subject was able to reach the target. You can see it at: the web TV of the Museo Civico of Rovereto

10 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Robot as actuator and generator of interface Robot is able to send back to patient images and sounds about environment. The images token by omnidirectional camera can be integrated in BCI interface to allow patient to see through robot’s camera. A new integrated interface for BCI is under clinical validitation by prof. Priftis of Psychology Department of University of Padua

11 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Robot as actuator and generator of interface Figure 6: classification accuracy (%) of the 5 healthy subjects who performed Experiment 1, and classification accuracy (%) of one subject who performed Experiment 2.

12 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Robot senses

13 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Robot as a Telepresence medium The patient is able to see and listen with the senses of the robot and so he is able both to extend own social life and communicate own needs Telepresence for museum visits: more autonomous navigation,obstacle avoidance, new graphical user interface Telepresence for household rehabilitation: consumer robot Rovio holonomous robot with frontal camera for remote perception.

14 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Robot as a Telepresence for museum visits Robot as a Telepresence for museum visits TCP/IP The BCI Server Internet Patient in Venice, robot in Agrigento, Sicily, 1467 km

The Client robots developed at University of Palermo, Sicily, ITALY Pioneer3-AT (Outdoor Robot) PeopleBot (Indoor Robot) ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

STEP 1: Choose of the real robot The Graphical Interface ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

STEP 2: The Session Start and Sensors and Actuators are Activated The Graphical Interface ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

STEP 2.1: The Applet for the visualization of the navigation environment through the robot camera The Graphical Interface ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

STEP 3: Robot Movements Activation through the use of the 4 directional arrows The Graphical Interface ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

The Graphical Interface ARES – CISIS 2009 Fukuoka Institute of Technology, Fukuoka 16 March 2009

21 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Next step: fully autonomous robot, robot has a map of the environment, robot presents four topological choices (e.g. the four closest masterpieces), subject selects destinations Robot as a Telepresence for museum visits Robot as a Telepresence for museum visits

22 May 12, 2009BCI Workshop ICRA 2009, Kobe Evaluation of a robot as embodied interface for Brain Computer Interface systems Next step: fully autonomous robot, robot has a map of the environment, robot presents four topological choices (e.g. the four closest masterpieces), subject selects destinations Robot as a Telepresence for museum visits Robot as a Telepresence for museum visits

The Experimental Results: Simulation The BCI-Robot moving in the DINFO Department. Time T1 : Command Go Forward BCI- Web_InterfaceSnapshot of Robot Position BCI- Web_Interface Snapshot of Robot Position Time T10 : Command Stop

Many Thanks to You and to: Dept. of Information Engineering, University of Palermo, Palermo, Italy Rosario Sorbello : Antonio Chella : Salvatore Anzalone : Francesco Cinquegrani : Dept. of Information Engineering, University of Padova, Padova, Italy Emanuele Menegatti : Dept. of Neuro-rehabilitation, S.Camillo Hospital, Venice, Italy Laboratory: Dept. of General Psychology, University of Padova, Padova, Italy Konstantinos Prifitis :