Abstraction + Geometry = Realization (Views of Illuminated Surfaces) A (potential) application of singularity theory to vision BIRS, Banff, 30 August 2011 James Damon, UNC Chapel Hill Peter Giblin and Gareth Haslinger, University of Liverpool

Abstraction + Geometry = Realization (Views of Illuminated Surfaces) A (potential) application of singularity theory to vision BIRS, Banff, 30 August 2011 James Damon, UNC Chapel Hill Peter Giblin and Gareth Haslinger, University of Liverpool me

1. Occluding contour = profile = apparent contour = silhouette 2. Surface discontinuity = crease, one ‘sheet’ invisible 2 ’. Edge, or surface boundary 3. Crease, both sheets visible 4. Surface marking 5. Change of texture 6. Cast shadow 7. Shade curve = terminator, light grazing the surface 8. Specularity 2’ Some typical edges which occur in images

Corners of various types, where 3 surfaces, not necessarily flat, and 3 creases, meet in a point.

Of course there are many kinds of junctions between different types of edge Cast shadow and 3 creases at a corner (need not be flat) Apparent contour and crease Where the cast shadow from a crease (ridge) meets the crease, the light direction grazes (lies in the tangent plane to) the shadowed surface ‘Multilocal’ crossing (depth discontinuity)

And when we allow changes of viewpoint then we expect to see some ‘interactions’ between junctions, as here where the apparent contour sweeps across the corner. (A ‘notch’ corner.)

and there may be different interactions when the viewpoint swings through a full circle

Of course, movement can resolve ambiguities in the shape of an object

Putting together a ‘classification’ of the possible interactions involves several steps. The same would go for classifications of other visual phenomena. 1.Make abstract models of the various situations, using manifolds and maps between them. (Note that in our work we assume a ‘stable illumination’, and a single predominant light source.) 2.Decide which abstract models will arise in the course of the particular classification you seek. 3.Carry out an abstract classification to see what is in principle possible. 4.Bring in the geometry of the particular problem you are studying and see which among the abstract classification can actually be realized by physical objects.

For example, consider creases (two surfaces meeting along a curve Cr) where there is a shade curve (S, light grazing the surface) on one sheet, S casting a shadow CS on the other sheet, and an apparent contour C on the sheet with the cast shadow The model is two half-planes, a line in one, representing the shade curve, and a curve in the other, tangent to the crease, representing the cast shadow. S CS Shade curve S here artificially indicated by a white line

Deciding which abstract models need to be considered is a matter of considering what are the ‘generic’ situations. For example with a crease we will only have shade curves on both sheets meeting on the crease when the light is pointing directly along the tangent to the crease curve itself. Shade curve on a smooth surface: light grazes the surface so is in the tangent plane; for two surfaces meeting at an angle light would have to be in both tangent planes, hence along the tangent line to the crease curve. We would not expect to meet this in a ‘generic’ movement of the viewer past a surface.

This actually looks like a cast shadow (right) and a shade curve (left) meeting on the crease but in fact this requires the same condition for the light. A complete list of ‘visual codimension 1’ events can be drawn up, that is interactions between contours, shade curves, marking curves, cast shadows and creases, which will generically be seen by a moving observer.

For an actual surface we would view it along some direction, which amounts to mapping the surface to the viewplane by a linear projection: this surface is projected to the plane of the screen. ‘The surface is warped but the view is straight’ For our model surface we allow ‘projections’ which are just smooth maps to a plane. ‘The model is straight but the view is warped’

S CS We then look at mappings from this configuration to the plane, which have a critical set (modelling an apparent contour in the image) on the surface x = 0. This part may give a lot of trouble since we are really interested in the ‘visual nature’ of the image: the changes of coordinates which best express this might not be differentiable.

Change of viewpoint There are four cases and this is what the images ‘look like’ with no account taken of occlusion, nor of the physicality of the cast shadow

Then we have to return to the real world and seek actual surfaces which give rise to these abstract models. This is where the geometry of the surfaces plays a crucial role. In fact it may turn out that there is no actual surface which realizes a particular abstract model. This happens for types 1 and 4 in this example.

In other examples there are subtle geometrical restrictions which prevent all occurrences of a particular abstract situation. For example surface markings, boundary edges and creases are ‘arbitrary’ curves on surfaces, but shade curves and cast shadows have to obey certain geometrical rules.

The most basic rule is that a shade curve, where light grazes the surface, is a ‘contour of the surface from the direction of illumination’. Its tangent is conjugate to the illumination direction. But for a marking curve the tangent could be anything. illumination tangent to the shade curve shade curve

This implies among other things that is the shade curve and the contour generator are tangent on a surface then the point of tangency is parabolic. You can see they are tangent here by tilting the surface slightly (and leaving both curves in the same place).

This is the principle: Abstract models Which ones are relevant? Abstract classification, using the appropriate equivalence: this may involve serious applications of ‘singularity theory’ to carry out the classification. Realization, bringing in the geometry. From the classification we can list all the possible interactions between surface ‘features’, apparent contours and shade/shadow curves. Or: when moving along an edge of a particular type, which junctions are possible?

Jim Damon and I would be interested to know the state-of-the- art in edge extraction for the various edges which we consider.

Published versions of this work: Part I: Int. J. Computer Vision 2009; Part II: SIAM J. of Imaging Sciences 2011; Part III to appear (eventually......)

Some more corners, and in case you ever wondered where the rainbow ends, it ends at my office where it sometimes rains... Thank you for your attention