Robotic Perception and Action - Project Augmented Reality for the “Man in the Loop”

NOTE: Introduci l’animazione e AR in industria 4.0 Chiedi a Kato e De Paolis materiale ppt se lo vogliono condividere

AR interaction technologies ToF cameras (HW) UNITY basics C# Introduction to AR AR interaction technologies ToF cameras (HW) UNITY basics C# State machine with UNITY How to display an image on the projector with UNITY A software based on a state machine structure able to interact with the user with UNITY Identification and localization module: Superquadrics Optimization for fitting RANSAC Exercitation with matlab PCL C++ Calibration of Tof camera, traditional cameras and projectors on a plane Project – prerequisites

A projector to display on a table indications to the user (HW) A table with retro-reflective sheets? (HW) A Kinect v2 ToF camera (HW) A software based on a state machine structure able to interact with the user (sw) An identification and localization module (sw) Calibration of Tof camera with the projector and with the table (table and projector could simply fit by moving properly the projector) Project – components

An example of setup from literature Jaakko Hyry, Max Krichenbauer, Goshiro Yamamoto, Takafumi Taketomi, Christian Sandor, Hirokazu Kato, Petri Pulli, Design of Assistive Tabletop Projector-Camera System for the Elderly with Cognitive and Motor Skill Impairments An example of setup from literature

The user quest for help (voice, gesture or virtual button on the table) The system: Item := 1; Detect the item to manipulate Highlight it to the user Animate the action to be performed Verify that the item was moved Ask if the first step was performed correctly Item = item + 1 If not end-procedure, goto point 2 Project – interaction sequence

Project – modules to develop Description Software to use Groups State Machine > Main software that manages the HMI interaction. > Main projected interface. HW: virtual buttons with retroreflective material C# - UNITY Objects Localization with AR Toolkit > Markers placed on the objects extraction (position and location on the projector space). > Calibration of camera/projector/furnitures HW: kitchen simulated furnitures Skeleton acquisition > Skeleton acquisition with ToF (kinect) > Control of task execution UNITY Animations Development of proper anmations (for example with Blender) UNITY - Blender Objects Localization with ToF camera > Point cloud extraction and its use for objects recognition and localization C++ (PCL) Sensor Fusion > Sensor Fusion of Objects locations estimated with Ar Toolkit and the ToF camera Project – modules to develop

1. State Machine

Objects Localization with AR Toolkit

Skeleton acquisition

Link to some examples: https://www. youtube. com/watch 2D: https://www.youtube.com/watch?v=RKEuxDXpcFA Animations

Objects Localization with ToF camera Link to our project: http://www.robosense.it/index.php/component/allvideoshare/video/tof-pallet-identification-mocup?Itemid=172 Objects Localization with ToF camera

Sensor Fusion