1. Multi-Touch Navigation Engine Presented by Team Extra Touch: Chris Jones Shuopeng Yuan Nathan Wiedeback Detailed Design Review 1

2. THE PLAN 2

3. Make it interesting… Challenge Phantom touches “[40K] ought to be enough for anybody!” Need “tap points” for intermediate data We’re not really programmers… Solution Compare to previous frame Low resolution (240 x 180) for proof-of-concept Make each stage’s data available at the output Get help! 3

4. Serial Input 4

5. Serial Stream Parser 74 byte “packets” from the screen Interrupt-driven I/O on ARM -- until an entire packet received Then process it to get 32 sensor values, average, etc. Normalize by subtracting the lowest value from each Variations of ~100-200 5 (Little-endian) Average: 0x488B = 18,571 First sensor: 0x4886 = 18,566

6. Interpolation 6

7. Interpolator Input: 32 x 2-byte sensor levels Output: “Image” frame -- n x m x 1 bit pixel array Proof-of-concept: 240 x 180 (memory constraints) Find intersection points of sensors (16 x 16) Interpolate linearly based on distance – first in X, then in Y (bilinear) Set a threshold and discretize at the end Each pixel ends up as a 1 or a 0 7 Image source: Wikipedia

8. Phantom Filtering 8

9. Phantom Filter Input: Frame Output: Filtered frame ( 240 x 180 x 1 bit) Intersections method -> phantom touches Remember what was touched first – i.e., save and compare to the last frame Proof of concept -- only the simple case covered 9

10. Pattern Recognition 10

11. Pattern Recognizer: Flowchart 11

12. Input: Whole frame (60*20) 12

13. First step: Blob classification 13

14. (Visio files provided separately) 14

15. Output: the outline of each single blob 15

16. The X-Y Coordinates in separate arrays xy 73 123 64 1445 1556 156 67 157 68 148 69 149 610 1410 611 1411 712 1312 713 1413 714 1314 715 1315 716 1316 817 1317 918 1218 xy 323 343 314 354 315 345 xy 3911 4011 3812 4112 3913 4113 16

17. Voronoi Skeleton Algorithm 1 17

18. The block diagram 18

19. Voronoi Skeleton Algorithm 2 (The ideal case) 19

20. Voronoi Skeleton Algorithm (The actual result) 6 5 4.5 6.5 2.5 7.5 -0.5 8.5 -4.5 9.5 -9.5 10.5 8 5 9.5 5.5 10 6.5 10.1053 6.65789 10 7.5 9.5 9.5 7.5 10.5 5.5 12.5 3.5 13.5 0.5 14.5 -3.5 15.5 10 8.5 10 12.5 10.0833 13 10 14.5 8.5 15.5 10 13.5 10 9.33333 10.3 15.5 10 14.5 10 14.75 10.3333 15.6667 10.5 16 11.5 16.5 11.75 6 10 8.5 10 9.33333 10.5 9.5 12.5 10.5 14 12 13 13 14.5 14.5 16.5 15.5 19.5 16.5 13.5 5.5 12.5 6 13.5 6.5 15.5 8.5 17.5 9.5 20.5 10.5 26.5 12 20

21. Shape recognition 21

22. Slope algorithm If a skeleton is linear, according to geometry, angle a1 = a2 = a3 = a4 Therefore, only linear shaped skeleton can pass this filter. To the touch panel application, it means only the side of a palm instead of a palm shape can pass. 22

23. Channel Assignment 23

24. Channel Assignment Data In: Center coordinates (X,Y) Processing: Data gets transformed into a 2d array, and the compared against each channel frame to determine its channel. Channel frames are defined before compile. Data Out: (X,Y) coordinates with channel data attached. Details: Aiming for 2 channels initially Should be simple to add channels in the final iteration. 24

25. Channel Assignment – Flow Chart 25

26. Channel Assignment - Visual Representation 00000 00110 11010 11000 11000 Channel 0 Frame 00111 00001 00001 00111 00111 Channel 1 Frame Channel 2 Frame 11000 11000 00100 00000 00000 Output to PC: Coordinates 3,3 Channel 2 Input: Coordinate Data (3,3) touched We have a match at Channel 2! 26

27. Questions? 27