1. Learn how to make your drawings come alive…  COURSE: SKETCH RECOGNITION Analysis, implementation, and comparison of sketch recognition algorithms, including feature-based, vision-based, geometry-based, and timing-based recognition algorithms; examination of methods to combine results from various algorithms to improve recognition using AI techniques, such as graphical models.

2. Gesture Recognizers?

3. Rubine Long Discussion

4. Paulson Review

5. Sezgin Corner Finding Algorithm Finds corners in a polygon or in a complex shape.

6. Implications By finding corners, –You can build objects geometrically –Users can use multiple strokes

7. Claim to Fame This algorithm was the first to notice that corners were identifiable not only be curvature, but also by speed. The pen slows down when you ‘intend’ to make a corner.

8. Algorithm Overview Find corners based on curvature alone Find corners based on speed alone Find intersection Add one extra corner at a time until error is below threshold.

9. Direction Graph Direction of each stroke segment = arctan2(dy,dx) Add check to make sure graph continuous (e.g., add 2pi)

10. Curvature Graph Change in direction for each segment

11. Speed Graph Speed of each segment (already computed in Rubine)

12. Curvature Threshold The threshold is set to the mean curvature

13. Speed Threshold The threshold is set to 90% of the mean.

14. Select Curvature Vertices Max of all sections above threshold Fd = curvature points

15. Select Speed Vertices Max of all sections below threshold Fs = speed points

16. Initial Fit You now have a list of curvature corners: Fd And a list of speed corners: Fs The initial fit is the Intersection of Fd and Fs - (and of course the endpoints) (Note that now the candidate corners are limited to Fs union Fd)

17. Improving Fit We check the error, if it is small enough, we stop, but if not, we want to try to add a vertex Want to add one vertex at a time Which do we add? We try one curvature vertex We try one speed vertex

18. Picking the Curvature Vertex to Add Pick the vertex with the highest curvature CCM(vi) = Curvature Certainty Matrix at vertex I Compute CCM for each vertex candidates in Fd Average-based filtering Find curvature at vertex i using a neighborhood –CCM(v i ) = d i-k – d i+k /l –d i = curvature at vertex I –l = length of substroke from v i-k to v i+k Note: thie is not Euclidean distance – even though variable in paper is used to mean this later.

19. What is k? K is neighborhood size = number of stroke points on either side to search Paper does not specify. Suggestions: –Set k = 3 … AND / OR –Set k to the minimum value such that l i is greater than 6 pixels

20. Picking the Speed Vertex to Add Pick the slowest vertex SCM(v i ) = Speed Certainty Metric at stroke point i. SCM(v i ) = 1 – speed i / speed max Compute SCM for all speed vertex candidates (Fs) Note: paper uses v for speed and vertex

21. Vertex Possibilities: Fd: Possibilities based on curvature Fs: Possibilities based on speed CCM(Fd) : Curvature Certainty Metric –Used to rank Fd candidates SCM(Fd) : Speed Certainty Metric –Used to rank Fs candidates Remember: –H 0 = Initial Hybrid Fit = intersection of Fd and Fs (and endpoints)

22. Improving Fit Compute Error for H 0 If error below threshold stop… else Add vertex with largest SCM (speed) not already chosen –Hi’ = H 0 + max(SCM(Fs – H 0 ) ) Add vertex with largest CCM (curvature) not already chosen –Hi’’ = H 0 + max(CCM(Fd – H 0 )) Compute error for Hi’ and Hi’’ Add the vertex that created the smaller error –You have only added one vertex Repeat

23. Create Line Segments for Fit Approximation For each vertex v i in Hi – create a list of line segments between v i and v i+1 If there are n points in the vertex selection, there will be n-1 line segments

24. Measuring Fit Error S = total stroke length (not Euclidean distance) ODSQ = orthogonal distance squared ODSQ(s,Hi) = distance of stroke point s to appropriate line segment (previous slide)

25. Distance from point to line segment if isPointOnLine(point, line) theDistance = 0; else array1 = getLineAxByC(line); array2 = getAxByCPerpendicularLine(line, point); A = [array1(1), array1(2); array2(1), array2(2)]; b = [array1(3); array2(3)]; intersectsPoint = linsolve(A, b)'; if isPointOnLine(intersectsPoint, line) theDistance = getLineLength([intersectsPoint; point]); else dist(1) = getLineLength([line(1,:); point]); dist(2) = getLineLength([line(2,:); point]); theDistance = min(dist); end

26. What is the Error Threshold? Stop when error is below the threshold. … what is threshold? Not in paper Your choice: Suggestion: –Compute the error for H 0 = e 0 –Compute the error for H all (all the chosen v) = e all – We want something in the middle: close to e all –.1*(e 0 -e all ) + e all –You will try other thresholds in your implementation

27. Final Fit Final fit is a polyline fit. We have not yet created the complex fit.

28. Creating the Complex Fit Stroke between corners can be curve or line Compute euclidean distance (l1) / substroke length (l2) L1/l2 –Is <= 1 (because l1 < l2) –If is close to 1, it is a line, else a curve Paper leaves threshold to you: –Suggestion:.9 < l2/l1

29. Homework Find corners using Sezgin method Identify if substrokes between corners are lines or curves/arcs using distance metric

30. Sezgin Bezier Curve Fitting Want to replace with a Bezier curve http://www.math.ubc.ca/people/faculty/cass/gfx/bezier.html –Bezier Demo 2 endpoints and 2 control points

31. Sezgin Bezier Curve Fitting Control Point Approximation –c1 = k*t1 + v –c2 = k*t2 + u Notation: –Endpoints: u = p1, v = p2 –T1= tangent vector – initial direction of stroke at point p1 (u) –T2 = tangent vector of p2 (v) – initial direction of stroke at point p2 K = stroke length / 3 –3 is common approximation

32. To test curve error: Breaks into linear curves. If error to great, splits curve in half (by stroke point – not distance)

33. Bezier curve equation http://www.cl.cam.ac.uk/Teaching/2000/AGraphHCI/SMEG/node3.ht mlhttp://www.cl.cam.ac.uk/Teaching/2000/AGraphHCI/SMEG/node3.ht ml http://www.moshplant.com/direct-or/bezier/math.html P0 (x0,y0) = p1, P1 = c1, P2 = c2, P3 = p2 x(t) = axt 3 + bxt 2 + cxt + x0 y(t) = ayt 3 + byt 2 + cyt + y0 cx = 3 (x1 - x0) bx = 3 (x2 - x1) - cx ax = x3 - x0 - cx - bx cy = 3 (y1 - y0) by = 3 (y2 - y1) - cy ay = y3 - y0 - cy - by

34. Circles and Ellipses Least squares fit Make a bounding box of stroke and form Oval is formed from that bounding box. Circle is easy: –Find out how far the circle is from the center = d, return (d-r)^2 –Ellipse more complicated

35. Improvement Ideas Instead of testing the error, we test the improvement, if it is great enough, we add the vertex