MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra, Jürgen Schmidhuber IDSIA - Lugano - Switzerland

MindRACES, First Review Meeting, Lund, 11/01/ Overview Real world scenarios Fovea Software Framework Robertino Robot Simulated 3D scenarios Integration and Future Work

MindRACES, First Review Meeting, Lund, 11/01/ Instances of scenario 1: find an object Still Image Robot Lab Office Room Movie (LUCS) 2D Environments 3D Environments

MindRACES, First Review Meeting, Lund, 11/01/ Fovea Centralis Simulation Three regions with increasing resolutions

MindRACES, First Review Meeting, Lund, 11/01/ Fovea Centralis Simulation Three regions with increasing resolutions Subsample each region to 13x11 pixels

MindRACES, First Review Meeting, Lund, 11/01/ Fovea Centralis Simulation Three regions with increasing resolutions Subsample each region to 13x11 pixels Combine the subsampled image

MindRACES, First Review Meeting, Lund, 11/01/ ??? ???? ????? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley BinsObjects 1-D world Transformation Fovea Memory State 0

MindRACES, First Review Meeting, Lund, 11/01/ ??? ???? ????? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 1 BinsObjects 1-D world Transformation Fovea Memory

MindRACES, First Review Meeting, Lund, 11/01/ ??? ???? ????? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 2 BinsObjects 1-D world Transformation Fovea Memory

MindRACES, First Review Meeting, Lund, 11/01/ ??? ???? ????? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 3 BinsObjects 1-D world Transformation Fovea Memory

MindRACES, First Review Meeting, Lund, 11/01/ ???? ?? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 4 BinsObjects 1-D world Transformation Fovea Memory

MindRACES, First Review Meeting, Lund, 11/01/ ???? ?? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 5 BinsObjects 1-D world Transformation Fovea Memory

MindRACES, First Review Meeting, Lund, 11/01/ ??? ?? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 6 BinsObjects 1-D world Transformation Fovea Memory

MindRACES, First Review Meeting, Lund, 11/01/ ??? ?? Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 7 BinsObjects 1-D world Fovea Memory Transformation

MindRACES, First Review Meeting, Lund, 11/01/ Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 8 BinsObjects 1-D world Fovea Memory Transformation

MindRACES, First Review Meeting, Lund, 11/01/ Simulated Vision: 1-D Fovea Simulation Task: Find the sad smiley State 9 BinsObjects 1-D world Fovea Memory Transformation

MindRACES, First Review Meeting, Lund, 11/01/ Simulated Vision: abstract 2-D Simulation Extended 1-D Simulation Only the centered sensor can differentiate between objects

MindRACES, First Review Meeting, Lund, 11/01/ Simulated Vision: 2-D Simulation Original image with 2 objects

MindRACES, First Review Meeting, Lund, 11/01/ Simulated Vision: 2-D Simulation Original image with 2 objects Simulated fovea images Center of the fovea Step 1

MindRACES, First Review Meeting, Lund, 11/01/ Simulated Vision: 2-D Simulation Original image with 2 objects Simulated fovea images Center of the fovea Detected!!! Step 1Step n

MindRACES, First Review Meeting, Lund, 11/01/ Software Framework Server Robertino Debian/GNU Linux operating system CAN bus interface FireWire (IEEE 1394) interface Video4Linux library Fovea simulation Client Robosim Linux or Windows Ogre framework Fovea simulation Simple physical and collision detection system Same interface as the real robot Robomon Linux or Windows Experiment management Interface for learning algorithms Full remote control of the robot/simulation TCP/IP Previously recorded data

MindRACES, First Review Meeting, Lund, 11/01/ Robertino - Software Framework Linux CANV2L Local API (network, sensors, actuators) Remote control servers Local Application Remote control client API Remote Application Fovea implementation

MindRACES, First Review Meeting, Lund, 11/01/ Robosim - Software Framework Ogre 3D simulation framework on various operation systems Robot simulation with specific environment Remote control servers Local Application Remote control client API Remote Application Fovea implementation

MindRACES, First Review Meeting, Lund, 11/01/ Robertino – Client Software Manual control of the robot Learning control interface Fovea image

MindRACES, First Review Meeting, Lund, 11/01/ Robertino - Overview Diameter: 40 cm Height: 43 cm Weight: 6.5 kg. Holonomic three wheeled drive PC-103 (industry standard) with a 500MHz Intel Mobile- Pentium II processor on-board WLAN (IEEE a) 2 cameras, used to simulate the fovea Actuators: the three wheels and the simulated fovea

MindRACES, First Review Meeting, Lund, 11/01/ Omnidirectional Camera Web cam Omnidirectional mirror Image of an office environment, as seen by the robot

MindRACES, First Review Meeting, Lund, 11/01/ Image Transformation Used only for human convenience for navigation Simple transformation algorithm

MindRACES, First Review Meeting, Lund, 11/01/ High Quality Camera Imaging Source DFK21 AF04Image of an office environment, as seen by the robot

MindRACES, First Review Meeting, Lund, 11/01/ Robot Lab Environment Robot lab 550 cm x 280 cm Robot can navigate through the world and move the fovea Robot can be observed from a top view camera and local camera

MindRACES, First Review Meeting, Lund, 11/01/ Robot Lab Environment Camera image Fovea dataComposition No video for his presentation!

MindRACES, First Review Meeting, Lund, 11/01/ D Simulation of the Robot Environment design in 3D Studio MAX or G-MAX. Simulation in Ogre3D Same objects as in the real world Shadows (optional)

MindRACES, First Review Meeting, Lund, 11/01/ Simulated vs. Real Vision (Fovea) Simulation Real Robot Fovea transformation brings both world more together Artificial noise simulates camera noise (optional)

MindRACES, First Review Meeting, Lund, 11/01/ Integration and Future Work Develop and evaluate anticipatory learning algorithms in the simulated environments (UW, ÖFAI); compare them also to non- anticipatory ones Share scenarios for attentive vision (LUCS, NBU) Real robot with simulation as anticipation for surprise studies (CNR) Simulation-based learning of anticipation (UW) Share fovea simulation (IST) Evaluate transfer of learned behavior to the real robot Systematically increase the complexity of simulated and real environments Possibly use a movable camera and zoom lens mounted on the robot

MindRACES, First Review Meeting, Lund, 11/01/ Conclusion Learn to use a fovea for searching objects Objects possibly occluded or partially invisible Several real-world and simulated scenarios developed for training First experiments with simulated 1D and 2D worlds successful Starting with 3D experiments

MindRACES, First Review Meeting, Lund, 11/01/ Questions?