Feature Point Detection and Curve Approximation for Early Processing of Free-Hand Sketches Tevfik Metin Sezgin and Randall Davis MIT AI Laboratory

Metin Sezgin Outline Motivation and problem statement Feature point detection -Curvature and speed as information sources -Average based filtering; Scale space filtering -Hybrid fit generation -Handling strokes with curves Evaluation Related work Future work

Metin Sezgin Terminology Sketch: Informal, messy diagrams consisting of several strokes Stroke: Array of timestamped points Feature Point: Corners of a stroke (vertices)

Metin Sezgin Task Given a freehand stroke, generate a description of the stroke in terms of geometric primitives such as circles, lines, and curves

Metin Sezgin Lifecycle of a stroke Data Collection Data Collection Vertex Detection Vertex Detection Polyline Fit Polyline Fit Oval Fit Curve Detection Curve Detection Line Fit Line Fit Complex Fit (lines and curves) Complex Fit (lines and curves) Best Fit Selection Best Fit Selection Output

Metin Sezgin Lifecycle of a stroke Data Collection Data Collection Vertex Detection Vertex Detection Polyline Fit Polyline Fit Oval Fit Curve Detection Curve Detection Line Fit Line Fit Complex Fit (lines and curves) Complex Fit (lines and curves) Best Fit Selection Best Fit Selection Output

Metin Sezgin Collecting Strokes

Metin Sezgin Vertex detection

Metin Sezgin Vertex detection Curvature

Metin Sezgin Vertex detection Curvature Speed

Metin Sezgin Problems with local extrema Input is noisy -Digitization noise -Hand jittering (tremor) Picking all local extrema introduces many false positives Picking the global extremum gives us a single point Picking the n-best extrema doesn’t help either

Metin Sezgin Solutions Average-based filtering -Compute threshold based on data average -Partition using this threshold -Select global extremum in each partition

Metin Sezgin Average-based filtering Curvature Speed

Metin Sezgin Average-based filtering Pros -Reliable (when data isn’t too noisy) -Fast: O(n) -False positives easy to filter out Cons -Still false positives -Need pre-set constant for scaling

Metin Sezgin Scale-space filtering Real world data has features at many scales Represent the data multiple different scales Selecting scale(s) at which data is meaningful

Metin Sezgin Deriving the scale space Convolve the original signal with Gaussian signals of increasing width Feature points in the curvature scale space

Metin Sezgin Selecting a scale

Metin Sezgin Selecting a scale

Metin Sezgin Scale selected by the algorithm The original stroke Output generated at the scale chosen by our algorithm

Metin Sezgin Handling extremely noisy strokes (cont.) Curvature fit using SSF, 9 fpsSpeed fit using SSF, 7 fps

Metin Sezgin Combining information sources Individual detection methods may miss points Speed fit missing a point Curvature fit missing a point Need a way to combine information provided by each method Generate a series of hybrid fits and pick the best

Metin Sezgin Hybrid fit generation Given N fits generated using N different methods -Within each fit, compute vertex certainties and sort by certainty Initial fit contains points common to all methods Generate N new fits by adding best candidate from each method. Select the one of these with the least error Terminate when error smaller than preset error bound

Metin Sezgin Data Collection Data Collection Vertex Detection Vertex Detection Polyline Fit Polyline Fit Oval Fit Curve Detection Curve Detection Line Fit Line Fit Complex Fit (lines and curves) Complex Fit (lines and curves) Best Fit Selection Best Fit Selection Output

Metin Sezgin Curve approximation c2c2 c1c1 u v

Metin Sezgin Results

Metin Sezgin Results

Metin Sezgin User study results All users except one perceived the system as more natural. Overall it took users much less time to specify geometries. All subjects except one indicated they would prefer an interface that feels like our system for specifying shapes in a sketchy fashion. Strokes classified correctly 96% of the time

Metin Sezgin Related work Existing systems lack -Support for arbitrary shape creation with a single stroke -Automatic feature point detection -Modeless drawing -Natural feel

Metin Sezgin Future work Integration with the new sketch recognition architecture Support for over-tracing Further investigation of sketching behavior to enhance/aid feature point detection and stroke classification