2. Reinventing Printed Document Keeping digital as digital with uPen Jian Wang Senior Researcher Microsoft Research Asia July 19, 2005 Jian Wang Senior Researcher Microsoft Research Asia July 19, 2005

3. Outline Challenges and Approach Technologies Embedded Interaction Code(EIC) system Computer vision system HW/SW architecture Hardware prototype design EIC SDK: system design and APIs Demos Challenges and Approach Technologies Embedded Interaction Code(EIC) system Computer vision system HW/SW architecture Hardware prototype design EIC SDK: system design and APIs Demos

4. Challenges ``non-digital way of doing things'' like paper and pencils PCs still create non-digital document Such as printed document Keyboard/mouse confine people to desktop Mobile interaction device ``non-digital way of doing things'' like paper and pencils PCs still create non-digital document Such as printed document Keyboard/mouse confine people to desktop Mobile interaction device

5. Goal Digital original Keep digital as digital Makes every printed document digital Brings digital computing to places where things are still being done in non-digital ways Digital original Keep digital as digital Makes every printed document digital Brings digital computing to places where things are still being done in non-digital ways

6. Approach: EIC System High data-embedding capacity to provide a gigantic address space up to 2 192 bits. A family of EIC visual patterns to adapt to various hardware implementations, applications and surfaces, and to meet the needs for usability and readability. Multiple data-embedding channels to embed x and y coordinates, and both global (such as document ID) and local metadata in a digital document. High data-embedding capacity to provide a gigantic address space up to 2 192 bits. A family of EIC visual patterns to adapt to various hardware implementations, applications and surfaces, and to meet the needs for usability and readability. Multiple data-embedding channels to embed x and y coordinates, and both global (such as document ID) and local metadata in a digital document.

7. 1 0 0 1 0 1 1 0 0 0 1 1 1 0 0 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 EIC Pattern EIC: Absolute X, Y Coordinates

8. A pseudo-random 1D array of 0 ’ s and 1 ’ s with “ window ” property An example with primitive polynomial of order 6: 000001000011000101001111010001110010010110111011001101010111111 h(x)=x 6 +x+1, code length = 2 6 - 1=63 The window property ensures the unique decodability Window Property of m-sequence

9. 0 m1 = 9 m2=7 Construction of m-array 0 0 1 0 1 1 1 1 0 0 1 0 1 0 0 1 1 0 0 1 1 1 0 1 0 0 1 1 1 1 1 0 1 0 0 0 0 1 1 0 1 0 0 1 0 1 1 0 0 0 0 0 0 1 0 1 1 1 1 0 1 1 000001000011000101001111010001110010010110111011001101010111111 Create a 2D array with size of m1*m2, where gcd(m1,m2) =1 Folding the sequence The 2D array retains the window property. If L is the bit location in m- sequence, then its location in m-array: x= L mod m1 y= L mod m2 Create a 2D array with size of m1*m2, where gcd(m1,m2) =1 Folding the sequence The 2D array retains the window property. If L is the bit location in m- sequence, then its location in m-array: x= L mod m1 y= L mod m2

10. Data-embedding

11. EIC pattern Decodable area EIC + Encoded content Non- decodable area Bounding box Action Biding EIC with Document

12. Hardware Architecture Mechanical Size: 14x17x160mm Ink cartage Pressure sensor Optical Camera Lens IR LED Electrical Processor/Memory Bluetooth Battery Mechanical Size: 14x17x160mm Ink cartage Pressure sensor Optical Camera Lens IR LED Electrical Processor/Memory Bluetooth Battery

13. Computer Vision System To obtain high resolution (> 400 dpi), accurate x and y positions and metadata on printed digital documents (papers). To obtain actual x and y position of the pen tip regardless of pen posture (tilting and rotating). Calibration error < 0.15mm. To obtain sufficient spatial and temporal resolution (> 400 dpi x 120 Hz) of uPen digital ink (so that it’s compatible with the resolution of Tablet PC ink). To obtain high resolution (> 400 dpi), accurate x and y positions and metadata on printed digital documents (papers). To obtain actual x and y position of the pen tip regardless of pen posture (tilting and rotating). Calibration error < 0.15mm. To obtain sufficient spatial and temporal resolution (> 400 dpi x 120 Hz) of uPen digital ink (so that it’s compatible with the resolution of Tablet PC ink).

14. Document/EIC pattern segmentation T : document content A: EIC pattern Z : background area adaptive threshold of EIC pattern at (i, j) adaptive threshold of document content at (i, j)

15.                    nYXSW nYXSB kl i il j jk klji kl i il j jk klji /)),max(( /)),min((, 1 1 1 1,,, 1 1 1 1,, EIC Pattern Processing  kkkkk ijj n i kkkiik YXXXb bxyjSibxyJ k k k kk ' 1 ' 1 2 }|||{,)( 1        Bits in camera FOV: 98.5bits in average Extracted bits: 93% in average Determine EIC pattern grid Determine orientation of pattern Determine the value of each bits

16. Efficient Image Matching :search radius. :Maximum grid alignment error. : transformation matrix from camera coordinate to document image coordinate

17. EIC SDK Architecture uPen prototype/third-party hardware Applications EIC Embedded SDK EIC SDK uPen Firmware EIC Pattern Metadata EIC Document.Net Compact Framework

18. Demos

19. Thank you