Learn how to make your drawings come alive…  COURSE: SKETCH RECOGNITION Analysis, implementation, and comparison of sketch recognition algorithms, including feature-based, vision-based, geometry-based, and timing-based recognition algorithms; examination of methods to combine results from various algorithms to improve recognition using AI techniques, such as graphical models.

Sezgin Shape Fitting: Circles and Ellipses Least squares fit Make a bounding box of stroke and form Oval is formed from that bounding box. Circle is easy: –Find out how far the circle is from the center = d, return (d-r)^2 –Ellipse more complicated

Sezgin Bezier Curve Fitting Want to replace with a Bezier curve –Bezier Demo 2 endpoints and 2 control points

Sezgin Bezier Curve Fitting Control Point Approximation –c1 = k*t1 + v –c2 = k*t2 + u Notation: –Endpoints: u = p1, v = p2 –T1= tangent vector – initial direction of stroke at point p1 (u) –T2 = tangent vector of p2 (v) – initial direction of stroke at point p2 K = stroke length / 3 –3 is common approximation

To test curve error: Breaks into linear curves. If error to great, splits curve in half (by stroke point – not distance)

Bezier curve equation mlhttp:// ml P0 (x0,y0) = p1, P1 = c1, P2 = c2, P3 = p2 x(t) = axt 3 + bxt 2 + cxt + x0 y(t) = ayt 3 + byt 2 + cyt + y0 cx = 3 (x1 - x0) bx = 3 (x2 - x1) - cx ax = x3 - x0 - cx - bx cy = 3 (y1 - y0) by = 3 (y2 - y1) - cy ay = y3 - y0 - cy - by

Matlab Curve Fitting function [estimates, model] = fitcurvedemo(xdata, ydata) % Call fminsearch with a random starting point. start_point = rand(1, 2); model estimates = fminsearch(model, start_point); % expfun accepts curve parameters as inputs, and outputs sse, % the sum of squares error for A * exp(-lambda * xdata) - ydata, % and the FittedCurve. FMINSEARCH only needs sse, but we want to % plot the FittedCurve at the end. function [sse, FittedCurve] = expfun(params) A = params(1); lambda = params(2); FittedCurve = A.* exp(-lambda * xdata); ErrorVector = FittedCurve - ydata; sse = sum(ErrorVector.^ 2); end

Yu Overview Recursive corner finding method –Takes stroke and tries to classify it as Line, Arc, Circle, Ellipse, Spiral –If error is too great, splits stroke at highest point of curvature and tries again.

Line Recognizer Feature Area

Arc Recognizer Feature Area: Note must have best fit arc

Circle Recognition: Fit Direction Graph to Line

Circle Fit Use feature area to circle center

Ellipses Handled similarly Area should by symmetrical about the major and minor axes

Overtraced Circle Break direction graph every 2 pi If all circles have about same radius and center point – is circle

Spiral If overtraced circles have ascending or descending radii, then is a spiral (they incorrectly call this a helix in the paper)

Foundation Knowledge Identify The Problem Explore Interpretations & Connections Prioritize Alternatives Envision Strategic Innovation Confused Fact Finder Biased Jumper Perpetual Analyzer Pragmatic Performer Strategic Re-visioner Distinguish relevant & irrelevant Information Read conflicting opinions Relate assumptions & biases Analyze pros & cons Prioritize issues and information Justify assumptions Articulate vision Reinterpret information Steps for Better Thinking Performance Patterns, Developmental Framework for Critical Thinking ObservationInterpretation Judgment Planning Steps in Critical Thinking Performance Patterns Interventions Step 1Step 2Step 3Step 4

Computer – Sketch Recognition Foundation Knowledge: Stroke, Rubine Method, Sezgin Method, Yu Method Which primitive (or group) or gesture is this stroke? Line, Arc, Ellipse, RGesture1, RGesture2 Precedence, Error, Tolerance What about combination of strokes? May change lower level interpretations… geometric context Foundation Knowledge Identify The Problem Explore Interpretations & Connections Prioritize Alternatives Envision Strategic Innovation ObservationInterpretation Judgment Planning

Recognizing a Line Foundation Knowledge: Stroke, Sezgin Method, Yu Method, Geometry How do we find *line* tolerance & error Options: –Least-squares error from endpoints: + Uses endpoints - Endpoint tails not removed - Error may be larger than true error –Least-square error with best fit line: + Find best-fitting line - Doesn’t necessarily use perceptually important start & end points + Can remove non-perceptually important tails –feature area: + In theory, can be compared to other shapes - Confusing - Value not apparent ? Smaller range –ratio: euclidean length/stroke length: + Easy to calculate + Uses perceptually important start & end point - Endpoint tails not removed Doesn’t differentiate between one point being far away and several points being near - Bigger range so harder to figure out a good threshold –Least-squares error using best fit, but then use endpoints - Error not same as what is chosen + Error is more representative of line + Perceptually important endpoints

Recognizing an Arc Foundation Knowledge: –Method to find sample arc as part of a circle: Connect endpoints, find perpendicular bisector of that line Find where that line intersects stroke Make two lines connecting center stroke point and endpoints Find perpendicular bisector of each line Intersecting point is circle center Find feature area –A curve of order 2 Options: –Least-squares error from endpoints with a curve of order 2: + Uses endpoints + Easy to compute - Not actually an arc –Feature area + Uses real arc + Faster ? - Need the line of the arc, because takes the feature area - Difficult – polygons could be above or below Idea: Add threshold to radius – for comparing against line –Least squared error with arc itself + Uses real arc - Harder to compute + Faster? Compute distance of each point to the center – subtract from radius

Recognizing a Circle Foundation Knowledge: –Method to find direction graph slope Direction graph: Find direction of each point –Direction vs. time since start »Depends on time since start –Direction vs. point number »Depends on sampling rate –Direction vs. stroke length »More time computationally Find slope Fit a line to direction graph – use same least square method –Splitting: Spilt it when change in direction (every) 2pi –Circle center: center of bounding box or average of all points –Circle radius: Bounding box / 2 Average distance from center Options: –Slope of the direction graph == 2pi/n Doesn’t handle tails Overtracing difficult because have to split –Direction graph is linear –Circle least squares –Circle feature area

Recognizing an Ellipse Foundation Knowledge: –Method to find direction graph slope Use endpoints Find best fit line of direction graph –Major axis and minor axis not equal –To find major axis Two points w/ greatest distance is the major axis –Perpendicular bisector is minor axis (where it intersects stroke) –Should points also intersect a calculated center point? Fit a line to the ellipse –To find center point Center of bounding box Center of longest line Center of mass –Area of Ellipse PI * (length of major axis/2) * (length of minor axis/2) –Definition of Ellipse Sum of the distance from focus 1 and focus 2 is constant X^2 / a^2 + y^2 / b^2 = 1 –A = ½ major axis, b = ½ minor axis Focal point is the point on the major axis that is distance ‘A’ from where minor axis intersects ellipse Options: –Slope of the direction graph ~2pi/n –Ellipse least squares – need foci –Ellipse feature area Small triangles to center vs actual ellipse area

Recognizing an Ellipse, Part 2 Length of Major Axis –Fit a line –Longest distance Length of Minor Axis –Rotate Ellipse to find height of BB –Average distance of stroke points that intersect minor axis from the major axis –Calculate from perimeter formula (p = strokelength = pi * sqrt(2*(s^2 + b^2) – (a-b)^2/2) –Calculate distance from every point to the major axis: Minor axis = average distance * pi / 2 –Eigenvector method –Closest point to center

Helix Recognition Find major, minor axis (rotated b b) Find number of rotations (direction graph from circle) Combine n helix components Rotate, scale and translate X = cos(t) + change in x at t Y = sin(t) + change in y at t

Kim and Kim 8 possible directions in pixel space Resample Data of distance 5 pixels Compute best fit lines of 10 pixels long. Sharp angles smoothed out.

Curvature: Local Convextity When computing curvature – if curvature sign is the same (moving in same curved direction) add that to total curvature. Fig. 5. Algorithm 1: Curvature estimation only with local convexity

Area of Examination Area continues as long as it is convex. Fig. 6. The effect of the adaptively changing support with k=1. The actual support is denoted by the dotted rectangle. Without convexity criterion, however, actual support is set to the solid-line rectangle.

Local Monotonicity Check if the current direction is < min direction – to find when it changes its sharpness Fig. 8. Algorithm 2: estimate curvature at point p i.

Perceptual Recognition Follows the line – smoothing points in order perceptual importance.

Multipass Passes through the algorithm 5 times Each time the strictness threshold gets looser. Strictness = iteration number (1-5)

Examine each line segment Each line segment to see if want to join with line on either side. Line segments are sorted by length, so largest lines are processed first –Idea – want to join with a small

Threshold setting Messythreshold =.05 * strictness –Used to merge lines together Ratio = min(length1, length2) of line lengths

Research Projects Sketch-based Interfaces and Modeling: SBIM’08: Annecy, France. Deadline in February, 2008 Free trip for any accepted papers

Project Idea: Cusp/Corner Detection in Complex Shapes Develop and implement a new corner detection algorithm that has improved recognition accuracy for complex fits. It must be fast and have greater accuracy than current methods.

Multi-stroke primitive recognition Extend the low level recognizers to be multi-stroke recognizers. Detail how things change. – How are they combined? How do you know when strokes should be combined?

Compare Drawing Surfaces Perform a user study on several drawing surfaces. See how the primitives are drawn differently on the different surfaces, and recognize them appropriately. Write up the differences and how one could shift

Project Suggestions Build a finite state machine recognizer for the computability class to easily draw and hand in their diagrams. Build a physics drawing program that attaches to a design simulator (we have interactive physics 2005) Build a fashion drawing program. You draw clothes on a person, and it puts them one the person.

Project Ideas Build a robot drawing and simulation program. You draw the robot and have a number of gestures to have it do different things Gesture Tetris

Project Suggestions Use both rubine and geometrical methods in recognition Develop new ways for editing. Build new low level recognizers

Projects! 2 types: –Cool application Sketch front end to your own research system Fun application to go on smart board/vis contest –Gesture Tetris –TAMU gesture-based map/directory info –Computability/Physics/EE/MechEng simulator –New recognition algorithm Significant change to old techniques to make a new application

Final Project Handin Implementation… Build it… In class Demonstration (5-10 minutes) Previous work –Find at least 3 relevant papers (not read inc class) –Assign one for class to read, you lead short discussion Test –Run your recognition system on data. –Find out what data you need (e.g., UML class diagrams) –Each student in class will supply others data –+ Find 6 more people outside (to give 15 different people) Paper –Introduction (why important) –Previous Work –Implementation –Results –Conclusion

Improvement Ideas Instead of testing the error, we test the improvement, if it is great enough, we add the vertex