Multi-Touch Navigation Engine Presented by Team Extra Touch: Chris Jones Shuopeng Yuan Nathan Wiedeback

Multi-Touch Navigation Engine Background System Design Accomplishments 2

Background AIS offers single-touch Multi-touch is the “cool new thing” –iPhone, MacBook –MS Surface Mitch Butzer had good ideas… 3

Project Specifications 4 Microcontroller-based Written in C independent control channels Defined at compile time Recognize defined patterns e.g., finger but not palm

5 System Flow

Serial Input 6

Serial Stream Parser 74 byte “packets” from the screen Processed to get 32 sensor values, etc. Subtract the lowest value from each 7 (Little-endian) Average: 0x488B = 18,571 First sensor: 0x4886 = 18,566

Interpolator 8

Red dots: intersections of sensors (16 x 16) Green dot `P`: interpolated point Set a threshold and discretize at the end Each pixel -> 1 or 0 9 Image source: Wikipedia

Phantom Filter 10

Phantom Filter 11 One touchTwo touches Phantom touches (unfiltered) Intersections -> phantom touches Compare touches to last frame for filtering

Pattern Recognizer 12

Pattern Recognizer: Flowchart 13 Data In : Frame of Data Resolution: 240 * 180

Blob Classification and Checking Engine 14 Input Example: Resolution 60*20

Blob edge finding algorithm First edgeLast edge

16 Blob Classification and Checking Engine Input Example: Resolution 60*20 -- two blobs found

Voronoi Skeleton Algorithm 17 Ideal caseActual case

18 Voronoi Skeleton with Test Data

Slope Algorithm Linear skeleton should have the same angle at each point. (a1 = a2 = a3 = a4) 19 Linear -> accept and find centerNonlinear -> reject

Channel Assignment 20

21 Channel Frames – Overall Look Current FormPotential Form

22 Current Setup Channel Frames Channel 1 FrameChannel 2 Frame

Channel Assignment – How it Works Channel 1 Frame Channel 2 Frame Channel 3 Frame Output to One Touch per Channel Limiter: Coordinates 3,3 Channel 3 Input: Coordinate Data (3,3) touched We have a match at Channel 3! 23

24 One Touch per Channel Limiting Channel A – 3 touches Channel A -Phase 1 Reduction Channel A - Phase 2 Reduction

25 Output Data X0 Y0 X1 Y1 X2 Y2 X3 Y3 C0 C1 C2 C3 Altered Coordinate Array Channel Array

Accomplishments Translate sensor wires into 16x16 matrix Interpolate Recognize finger touches, reject other shapes Assign to either of two channels 26

Challenges Challenge Phantom touches “[40K] ought to be enough for anybody!” Need “tap points” for intermediate data Solution Compare to previous frame Low resolution (240 x 180 ?) for proof-of-concept Make each stage’s data available at the output 27

Future Serial interrupt issues Phantom filtering implementation –Would also enable more shapes Different touchscreen technology? 28

Questions?

Channel Assignment – Text Overview Data In: Center coordinates (X,Y) Processing: Data gets transformed into a 2d array, and the compared against each channel frame to determine its channel. Channel frames are defined before compile. Data Out: (X,Y) coordinates with channel data attached. Details: Aiming for 2 channels initially Should be simple to add channels in the final iteration.

Channel Assignment: Flow Chart 31

Summary Major design: “Proof-of-concept” –240 x 180 x 1-bit “color” –Simple phantom filtering –Accept a single finger per channel, reject other cases –Two channels 32