A Versatile Depalletizer of Boxes Based on Range Imagery Dimitrios Katsoulas*, Lothar Bergen*, Lambis Tassakos** *University of Freiburg **Inos Automation-software GmbH

Introduction  Definition of Depalletizing problem.  Our research focuses on depalletizing of objects in distribution centers (boxes, box-like objects, sacks).  This contribution concerns depalletizing of boxes.  Our system deals with pallets containing piled boxes of various dimensions.  Our system is model based.

Related Work (1)  Camera-based systems.  Range Imagery based: Katsoulas et.al 2002 Kristensen et.al Baerveldt 1993 Vayda et.al. 1990

Related Work (2)  Chen et. Al. 1989: Model based. Hypothesis generation and verification framework. Idea: object vertices incorporate necessary constraints for unique determination of pose transform. Vertices are represented via surface normals and the vertex point. Vertex point determined via intersection of surfaces. 3 surfaces need to be exposed for accurate computation of vertex point.

Our Approach.  Data Acquisition via a time of flight laser sensor mounted on the hand of the robot.  Accurate detection of box edges.  Vertex representation via box edges.  Hypothesis generation and verification framework triggered by detected vertices.

The System in Operation

Edge Map Creation via Scan Line Approximation  Edge points are detected by approximating the rows and columns of the range image with linear segments.  The algorithm uses infomation from long line segments to compute the edge points.  It is better than local approaches in terms of accuracy.  It is fast.

Vertex Detection(1)  Accurate detection of 3D lines corresponding to edges of boxes.  Grouping of compatible box edges to yield Vertices. Representation of a Vertex as a triplet: (feature set).  Two-step, 3D line Detection (Inspired from the dynamic generalized Hough transform): Get rough approximation of the position of the lines in space. Exploit the sparsity of lines in the 3D space to refine the roughly computed line parameters.

Vertex Detection(2) – 3D Line Detection 2D connected components (CCs) are determined. 3D line segments are fitted to the points defined by the CCs. Points in the vicinity of the segments are considered. Robust Calculation of the line‘s direction vector. Robust Calculation of the line‘s starting point.

Vertex Detection(3) – 3D Line detection  Calculation of the direction vector: Selection of random pairs of points and calculation of difference vectors. Accumulation of the coordinates of the difference vectors in 3 1D accumulators. The maxima of the accumulators are the desired parameters provided that the standard deviation of the accumulators is below a threshold.  Similar technique for calculating the starting point.

Object Recognition(1)  Based on Hypotheses generation and verification.  A scene vertex is aligned to a model box vertex.  This alignment produces a transform which brings the model to the scene. This is equivalent to hypothesis creation.  Verification of the hypothesis is performed via examination of the position of neighbouring vertices.

Object Recognition (2)- Model Database  The model database contains triplets of the form (model local feature set) which express model vertices.  6 feature sets per model are stored in the model DB.

Object Recognition(3)

Hypothesis Generation  If a scene vertex and a model vertex, a box location hypothesis is generated as follows: Rotation Matrix: Translation Vector:

Hypothesis Verification (1)  For every adjacent scene vertex, its position in the model coordinate system is determined:  If one of the vertices of the model which created the hypothesis is close to the back- projected adjacent vertex, it is considered compatible to the scene vertex which created the hypothesis.  Verification depends on the number of compatible scene vertices found.

Hypothesis Verification (2)

Hypothesis Verification (3)  Verification criterio when only one surface is exposed: detection of one compatible vertex which shares no common edge with the hypothesis generating vertex.  This safely verifies the occurence of a box side (surface).  Not a problem, since we are looking for graspable surfaces.  However, the set of necessary constraints for verification needs to be further reduced.

Hypothesis Verification (4)

Hypothesis Verification (5)

Hypothesis Verification (6)

Plane Fitting Test  Idea: The border points of the hypothesised surface can be accurately computed.  3D range points lying in the hypothesised surface segment are extracted.  A plane is fitted to the 3D points.  If the fitting error is below a threshold the hypothesis is considered verified.

Hypothesis Verification (6)

Discussion on the Plane Fitting Test  It is implemented via polygon rasterization.  It reduces the number of scene vertices that need to be detected for safe verification.  It could be used as a verification method when no compatible scene vertices are detected.  It ensures that a detected box surface is graspable.  Helps for an even more accurate grasping.

Experimental Results (1) – Intensity Image

Experimental Results (1) – Edge Map

Experimental Results (1) – Robust Lines

Experimental Results (1) – Vertices

Experimental Results (1) – Detected Boxes

Experimental Results (2) – Intensity Image

Experimental Results (2) – Edge Map

Experimental Results (2) – Robust Lines

Experimental Results (2) – Vertices

Experimental Results (2) – Detected Boxes

System Advantages  Computational efficiency: fast vertex Detection, fast hypothesis generation and verification. Detection of more than one boxes per scan are detected. <15 secs per box.  Accuracy: Accurate hypothesis generation due to robust detection of vertices. 4 degrees in orientation, 2cm in translation.  Robustness: robust verification criteria. No false identifications.

System Advantages (2)  Independence from lighting conditions: Employment of time of flight laser sensor.  Versatility: Deals with both layered and jumbled configurations.  Ease of Installation: Sensor on the hand of the robot.  Low Cost: Cost of the sensor ~ 3000 Euro.  Simplicity.

Problems  System fails when the objects are placed very close to each other in distinct layers, because no edges points or vertices can be detected.  Solution: additional sensor (e.g. intensity camera) to resolve the correct orientation.

Future Work  Exploit vertices detected in previous scans to make the recognition process faster.  Depalletizing of non rigid box-like objects and piled sacks.