1. The research project: FP7-People-Marie Curie-COFUND-Trentino-post-doc 2009 - Incoming
Laser-Camera systems and algorithms to sense the environment for robotic applications
Presenter: Dr. Ilya Afanasyev
Lab. Mechatronics, University of Trento, Italy
PAT mid-term review, Trento, 2011
18/10/2011

2. About the grant holder AREAS OF EXPERTISE
1. Robotics. Computer vision. Data Treatment and Analysis.
2. Experimental Physics. Optical sensors (UV, Visible, and IR).
QUALIFICATIONS AND EDUCATION
Ph.D. (in Technical Sciences), Vavilov State Optical Institute, St. Petersburg, Russia (
Specialization: Optical-electronic instruments
full-time student at the Baltic State Technical University, Russia (
Qualification: Engineer
Specialization: Automatic Control Systems for flying apparatus
JOB EXPERIENCE
2010 – present Marie Curie-COFUND post-doc, Mechatronics Dep., Faculty of Engineering, University of Trento, Italy
2007 – 2010 Senior engineer, Sensor Department, Development Center Russia, Giesecke & Devrient GmbH (G&D,
2006 – 2007 Senior engineer, Aerospace Optical-Electronic Systems Dep., Vavilov State Optical Institute, Russia (
2000 – Researcher/PhD student, Aerospace Physical Optics Lab., Vavilov State Optical Institute, Russia (
18/10/2011

3. Description of the Project
Status of the Project
The 3-years project in Mechatronics has been doing since May 2010.
Description of the Project
The current project proposes using a Laser-Camera System (LCS) as an instrument to get 3D information of the world environment. LCS includes a scanning laser rangefinder (LRF) and RGB camera. The rotated LRF captures an object as unstructured “clouds of points” of 3D data, whereas a camera gives a 2D image of the object. The fusion of 2D and 3D sensor’s data gives full and detailed information about the object and its environment.
18/10/2011

4. The calibration of laser-camera system
2D and 3D sensor fusion should be a result of a mutual sensor calibration. The calibration is important for:
a compatibility of sensors measurements, and
having the metrical coordinate system with good accuracy.
We are using the standard camera calibration technique (Jean-Yves Bouget method).
18/10/2011

5. 3D data processing Then joint 3D and 2D data processing is used:
to recognize an object,
to estimate its orientation in 3D space, and
to improve LCS calibration by mathematical modeling.
3D object visualization / reconstruction can be executed by math. modeling with SuperQuadrics (SQ).
The explicit equation of superquadrics is
The implicit equation of superquadrics is
where x,y,z - superquadric system coordinates; η, ω – spherical coordinates, where
a1, a2, a3 – parameters of object scaling; a4, a5 – parameters of object shape.
18/10/2011

6. Object recognition algorithm
The object recognition algorithm is based on using RANSAC (RANdom SAmple Consensus) method of SuperQuadrics model fitting to objects presented in 3D “cloud of points”. The solution is verified by evaluating the matching score between the SQ object model and 3D real data by a robust least square fitting.
18/10/2011

7. Mid-term results & achievements
- Building a prototype of a robotic vehicle with installed laser-camera system
The laser and the camera oriented and calibrated together have been formed the Laser-Camera system
18/10/2011

8. Mid-term results & achievements
- Testing the sensitivity (laser remission effectiveness) of the Laser-Camera System to registration of the objects from different materials (metals, plastic, wood, paper, polyethylene, etc.) at the different illumination conditions.
The polyethylene is transparent for IR rangefinder, but multilayer polyethylene decreases the depth of recognition significantly.
18/10/2011

9. Mid-term results & achievements
- Capturing 3D data and development of 3D object reconstruction / recognition algorithms (for human body, etc. achieved at
3D Human Body Pose Estimation: photo, “cloud of points”, fitting a math. model to 3D data, final math. model of pose.
18/10/2011

10. Future project work
According to the work plan, the future project work should be dealt with:
installation of the developed Laser-Camera System on a robotic vehicle.
research & control of robot vehicle motion.
improving Laser-Camera System calibration technique.
improving object recognition algorithms.
implementation algorithms to a robotic vehicle.
developing algorithms of relative position estimation (robot - object).
verification of the algorithms with hardware facilities.
18/10/2011

11. Possible impacts
3D Laser-Camera System can provide 2D/3D measurements capturing 2D/3D data of an object (with distance and color information) for indoor applications.
Object recognition and pose estimation algorithms can be applied for:
detection and reconstruction by math. models of different real objects (including complex objects, like human body).
detection, classification and selection a definite thing in a heap of different objects for robotic manufacturing (such as a search an object in mechanical details laying in mess).
different scientific projects (with a possibility of a collaboration).
18/10/2011

12. Training experiences During the project I participated:
at 21th Summer School in Jyväskylä University, Finland, 8-19 August 2011 (
in preparation of 3 conference papers:
to 21th Int. Conference: GraphiCon'2011 (Moscow, 2011,
to RGB-D Workshop on 3D Perception in Robotics (Sweden,
to Int. Conference VISAPP-2012 (Rome, 2012, February 2012,
Intensive course of Italian as a second language (Corso intensivo di lingua italiana per stranieri), CIAL, University of Trento, test (Livello A2), February 2011.
3D Modelling in 3ds Max. Training at St. Petersburg State Polytechnical University, Certificate #1KQYHQDCK1.
18/10/2011

13. Training experiences During the project I have been engaging in:
Leading a Mechatronics group (2 Master students) on measurements of remission from different materials (wood & aluminum foil & polyethylene) by IR rangefinder.
Giving a lecture on “Object Detection with Superquadrics” in “Robotics and Sensor Fusion for Mechatronics Systems” (course of Assoc. Professor Mariolino De Cecco, University of Trento, Italy). Povo, 05 April 2011.
Close collaboration with Italian colleagues for experimental and theoretical work, making seminars with presentation:
Creating, discussing and sharing some algorithms in MATLAB:
18/10/2011

14. Impacts on career development
The project helps in:
developing the professional skills in computer vision and robotics,
getting an experience in resource, time and money managing, and
strengthening the researcher reputation,
that must be valuable for future work at a research center or an industrial company.
18/10/2011

15. Acknowledgements
This work of Ilya Afanasyev on laser-camera system creation and algorithms of 3D object recognition, localization and reconstruction has been supported by the grant of EU\FP7- Marie Curie-COFUND - Trentino post-doc program,
He is very grateful to colleagues from Mechatronics Dep. of UniTN, especially supervisor - Professor Mariolino De Cecco.
Grazie!!
18/10/2011