1. Robotic Perception and Action - Project
Augmented Reality for the “Man in the Loop”

2. NOTE:
Introduci l’animazione e AR in industria 4.0
Chiedi a Kato e De Paolis materiale ppt se lo vogliono condividere

3. AR interaction technologies ToF cameras (HW) UNITY basics C#
Introduction to AR
AR interaction technologies
ToF cameras (HW)
UNITY basics C#
State machine with UNITY
How to display an image on the projector with UNITY
A software based on a state machine structure able to interact with the user with UNITY
Identification and localization module:
Superquadrics
Optimization for fitting
RANSAC
Exercitation with matlab
PCL C++
Calibration of Tof camera, traditional cameras and projectors on a plane
Project – prerequisites

4. A projector to display on a table indications to the user (HW)
A table with retro-reflective sheets? (HW)
A Kinect v2 ToF camera (HW)
A software based on a state machine structure able to interact with the user (sw)
An identification and localization module (sw)
Calibration of Tof camera with the projector and with the table (table and projector could simply fit by moving properly the projector)
Project – components

5. An example of setup from literature
Jaakko Hyry, Max Krichenbauer, Goshiro Yamamoto, Takafumi Taketomi, Christian Sandor, Hirokazu Kato, Petri Pulli, Design of Assistive Tabletop Projector-Camera System for the Elderly with Cognitive and Motor Skill Impairments
An example of setup from literature

6. The user quest for help (voice, gesture or virtual button on the table)
The system:
Item := 1;
Detect the item to manipulate
Highlight it to the user
Animate the action to be performed
Verify that the item was moved
Ask if the first step was performed correctly
Item = item + 1
If not end-procedure, goto point 2
Project – interaction sequence

7. Project – modules to develop
Description
Software to use
Groups
State Machine
> Main software that manages the HMI interaction. > Main projected interface. HW: virtual buttons with retroreflective material
C# - UNITY
Objects Localization with AR Toolkit
> Markers placed on the objects extraction (position and location on the projector space). > Calibration of camera/projector/furnitures HW: kitchen simulated furnitures
Skeleton acquisition
> Skeleton acquisition with ToF (kinect) > Control of task execution
UNITY
Animations
Development of proper anmations (for example with Blender)
UNITY - Blender
Objects Localization with ToF camera
> Point cloud extraction and its use for objects recognition and localization
C++ (PCL)
Sensor Fusion
> Sensor Fusion of Objects locations estimated with Ar Toolkit and the ToF camera
Project – modules to develop

8. 1. State Machine

9. Objects Localization with AR Toolkit

10. Skeleton acquisition

11. Link to some examples: https://www. youtube. com/watch
2D:
Animations

12. Objects Localization with ToF camera
Link to our project:
Objects Localization with ToF camera

13. Sensor Fusion