Robot Vision: Multi-sensor Reconnaissance

Overview An individual robot can develop an interpretation about its environment. Groups of robots can combine their resources to develop an improved, collective interpretation. Such a “swarm” is reliant on centralized processing to refine the collective interpretation. The swarm has more insight than solitary individuals.

Single-sensor Vision Our setup involves numerous robots, each with a single video camera. What can a single-camera robot see? A background image may be used to establish a baseline of irrelevant imagery. Anything that differs from the background is considered relevant – background subtraction. minusequals

Making Sense of Pixel Data Background subtraction and classical image processing algorithms can mark every pixel in every frame as foreground (interesting) or background (uninteresting). Conventional techniques can clean noise. Regions of contiguous foreground pixels are likely to constitute a single object: a “blob.” This assumption is false when one object occludes another. Later we will see a way around this.

Developing the Individual’s Interpretation Pixel-space blobs have measurable properties: color, shape, and location. Geometric properties (location, size) are relative to the camera’s perspective. A single robot can provide a list of objects with a bearing to each, but has no depth perception. Each robot’s interpretation may be combined to form a much stronger interpretation for the entire swarm.

Developing the Swarm’s Interpretation Each robot has a list of bearings to potential objects. This information can be visualized as rays originating from the robot’s location. The intersections of these rays represent potential objects in 3D space. Many intersections are bogus. Many intersections conflict with others – each ray can only correspond to one object.

Culling False Objects Algorithm: group compatible locations together. This yields disjoint sets of intersections that can coexist. The set with the most supporting evidence wins.

Combining the Individual and Swarm Interpretations At this point we have a set of objects with 3D locations. Individual robots can provide silhouettes of the objects. This information may be combined to create a 3D shape. Incorporating past history can strengthen our conclusions.

3D Hulls Each camera contributes a silhouette of an object, and a ray on which the silhouette lies. Projecting the silhouette along the ray forms a “cone”. The intersection of these cones carves a 3D solid; imagine pushing cookie cutters through space. The solid is guaranteed to enclose the true shape, but will be convex, i.e., ignores indentations. Such an upper bound is termed a hull. Hulls are typically represented as a mesh of triangles.

Applications of 3D Hulls The 2D silhouettes can be “painted” on the mesh. The solid can be rendered from any angle. 3D shape may be used to classify objects – threat assessment, for example. Meshes may be recorded for future use…

Establishing Object Tracks Object records from successive frames can be combined to establish a log of known objects. These “object tracks” can aid future processing, establishing a positive feedback loop: –Distinguishing between one large object, and two objects close to one another. –Past motion can predict where objects will be located, minimizing the occlusion problem mentioned earlier. –Past hulls can predict how an object’s silhouette will appear in each camera.

Summary Swarms of camera-equipped robots can collaborate to track and model objects in space. The swarm’s results are more concrete than any individual’s observations. Observation is passive and uses relatively few resources (weight, energy, money).