1. UIST 2004Kim, Seitz, Agrawala Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops Jiwon KimSteven M. SeitzManeesh Agrawala University of WashingtonMicrosoft Research

2. UIST 2004Kim, Seitz, Agrawala Motivation

3. UIST 2004Kim, Seitz, Agrawala Unifying physical and electronic desktops Record video of paper on physical desktop Video camera Desktop

4. UIST 2004Kim, Seitz, Agrawala Unifying physical and electronic desktops Record video of paper on physical desktop –Tracking Video camera Desktop

5. UIST 2004Kim, Seitz, Agrawala Unifying physical and electronic desktops Record video of paper on physical desktop –Tracking –Recognition Video camera Desktop

6. UIST 2004Kim, Seitz, Agrawala Unifying physical and electronic desktops Record video of paper on physical desktop –Tracking –Recognition –Linking Video camera Desktop

7. UIST 2004Kim, Seitz, Agrawala Applications Video camera Desktop

8. UIST 2004Kim, Seitz, Agrawala Applications Find lost document Video camera Desktop

9. UIST 2004Kim, Seitz, Agrawala Applications Find lost document Browse remote desk Video camera Desktop

10. UIST 2004Kim, Seitz, Agrawala Applications Find lost document Browse remote desk Find electronic version Video camera Desktop

11. UIST 2004Kim, Seitz, Agrawala Applications Find lost document Browse remote desk Find electronic version History-based queries Video camera Desktop

12. UIST 2004Kim, Seitz, Agrawala Example Input Video

13. UIST 2004Kim, Seitz, Agrawala Demo – Remote Desktop

14. UIST 2004Kim, Seitz, Agrawala Related Work Interactive desktops DigitalDesk [Wellner 93]

15. UIST 2004Kim, Seitz, Agrawala Related Work Interactive desktops Self-Organizing Desk [Rus et al. 97] DigitalDesk [Wellner 93]

16. UIST 2004Kim, Seitz, Agrawala Related Work Interactive desktops Augmented paper PADD [Guimbretière 03]

17. UIST 2004Kim, Seitz, Agrawala Related Work Interactive desktops Augmented paper PADD [Guimbretière 03] CyberCode [Rekimoto et al. 00]

18. UIST 2004Kim, Seitz, Agrawala Related Work Interactive desktops Alternative media Object tracking & recognition SIFT [Lowe 04]

19. UIST 2004Kim, Seitz, Agrawala System Overview Video camera Desk User Computer

20. UIST 2004Kim, Seitz, Agrawala System Overview Video of desk

21. UIST 2004Kim, Seitz, Agrawala System Overview Video of desk Images from PDF

22. UIST 2004Kim, Seitz, Agrawala System Overview Video of desk Images from PDF Track & recognize

23. UIST 2004Kim, Seitz, Agrawala System Overview Video of desk Images from PDF Track & recognize TT+1 Desk Internal representation

24. UIST 2004Kim, Seitz, Agrawala System Overview Video of desk Images from PDF Track & recognize TT+1 Desk Internal representation Scene Graph

25. UIST 2004Kim, Seitz, Agrawala System Overview Video of desk Images from PDF Track & recognize TT+1 Desk Internal representation Where is my W-2?

26. UIST 2004Kim, Seitz, Agrawala System Overview Video of desk Images from PDF Track & recognize TT+1 Desk Internal representation Where is my W-2? Answer

27. UIST 2004Kim, Seitz, Agrawala System Overview Video of desk Images from PDF Track & recognize TT+1 Desk Internal representation Where is my W-2?

28. UIST 2004Kim, Seitz, Agrawala Tracking & Recognition … …

29. UIST 2004Kim, Seitz, Agrawala Tracking & Recognition Event … …

30. UIST 2004Kim, Seitz, Agrawala Event Types Move beforeafter

31. UIST 2004Kim, Seitz, Agrawala Event Types Move Entry beforeafter

32. UIST 2004Kim, Seitz, Agrawala Event Types Move Entry Exit beforeafter

33. UIST 2004Kim, Seitz, Agrawala Tracking & Recognition … Event … … Desk

34. UIST 2004Kim, Seitz, Agrawala Tracking & Recognition … … Event … … Desk

35. UIST 2004Kim, Seitz, Agrawala Tracking & Recognition … … … … Desk Event tut-article.pdf sanders01.pdf objectspaces.pdfkidd94.pdf lowe04sift.pdf

36. UIST 2004Kim, Seitz, Agrawala Assumptions Document –Corresponding electronic copy exists –No duplicates of same document

37. UIST 2004Kim, Seitz, Agrawala Assumptions Document –Corresponding electronic copy exists –No duplicates of same document Motion –3 event types: move/entry/exit –One document at a time –Only topmost document can move

38. UIST 2004Kim, Seitz, Agrawala Non-Assumptions Desk need not be initially empty

39. UIST 2004Kim, Seitz, Agrawala Non-Assumptions Desk need not be initially empty Stacks may overlap

40. UIST 2004Kim, Seitz, Agrawala Algorithm Overview Input Frames … …

41. UIST 2004Kim, Seitz, Agrawala Algorithm Overview Input Frames … … Event Detection beforeafter

42. UIST 2004Kim, Seitz, Agrawala Algorithm Overview Input Frames … … Event Detection Event Interpretation “A document moved from (x 1,y 1 ) to (x 2,y 2 )” beforeafter

43. UIST 2004Kim, Seitz, Agrawala Algorithm Overview Input Frames … … Event Detection Event Interpretation “A document moved from (x 1,y 1 ) to (x 2,y 2 )” Document Recognition beforeafter File1.pdf File2.pdf File3.pdf

44. UIST 2004Kim, Seitz, Agrawala Algorithm Overview Input Frames … … Event Detection Event Interpretation “A document moved from (x 1,y 1 ) to (x 2,y 2 )” Document Recognition beforeafter File1.pdf File2.pdf File3.pdf Scene Graph Update Desk

45. UIST 2004Kim, Seitz, Agrawala Algorithm Overview Input Frames … … Event Detection Event Interpretation “A document moved from (x 1,y 1 ) to (x 2,y 2 )” Document Recognition beforeafter File1.pdf File2.pdf File3.pdf Scene Graph Update Desk

46. UIST 2004Kim, Seitz, Agrawala Event Detection … …

47. UIST 2004Kim, Seitz, Agrawala Event Detection time Frame differences … …

48. UIST 2004Kim, Seitz, Agrawala Event Detection time Frame differences … …

49. UIST 2004Kim, Seitz, Agrawala Event Detection time Frame differences … …

50. UIST 2004Kim, Seitz, Agrawala Event Detection time Frame differences … …

51. UIST 2004Kim, Seitz, Agrawala Event Detection time Frame differences … …

52. UIST 2004Kim, Seitz, Agrawala Event Detection time Threshold Event Frames time … … Motion Frames Image motion

53. UIST 2004Kim, Seitz, Agrawala Event Detection beforeafter Motion Frames … …

54. UIST 2004Kim, Seitz, Agrawala Algorithm Overview Input Frames … … Event Detection Event Interpretation “A document moved from (x 1,y 1 ) to (x 2,y 2 )” Document Recognition beforeafter File1.pdf File2.pdf File3.pdf Scene Graph Update Desk

55. UIST 2004Kim, Seitz, Agrawala Event Interpretation Move Entry Exit beforeafter

56. UIST 2004Kim, Seitz, Agrawala Event Interpretation Move Entry Exit 1. Move vs. Entry/Exit beforeafter

57. UIST 2004Kim, Seitz, Agrawala Event Interpretation Move Entry Exit 2. Entry vs. Exit beforeafter

58. UIST 2004Kim, Seitz, Agrawala Event Interpretation Move Entry Exit 1. Move vs. Entry/Exit beforeafter

59. UIST 2004Kim, Seitz, Agrawala Move vs. Entry/Exit before after

60. UIST 2004Kim, Seitz, Agrawala Move vs. Entry/Exit before after

61. UIST 2004Kim, Seitz, Agrawala Move vs. Entry/Exit before after

62. UIST 2004Kim, Seitz, Agrawala Move vs. Entry/Exit before after

63. UIST 2004Kim, Seitz, Agrawala Event Interpretation Use SIFT [Lowe 04] –Rotation- and scale-invariant –Highly distinctive (128-bit vector)

64. UIST 2004Kim, Seitz, Agrawala Move vs. Entry/Exit before after

65. UIST 2004Kim, Seitz, Agrawala Move vs. Entry/Exit before after

66. UIST 2004Kim, Seitz, Agrawala Move vs. Entry/Exit before after

67. UIST 2004Kim, Seitz, Agrawala Move vs. Entry/Exit before after

68. UIST 2004Kim, Seitz, Agrawala Move vs. Entry/Exit before after

69. UIST 2004Kim, Seitz, Agrawala Move vs. Entry/Exit before after

70. UIST 2004Kim, Seitz, Agrawala Move vs. Entry/Exit before after Motion: (x,y,θ)

71. UIST 2004Kim, Seitz, Agrawala Algorithm Overview Input Frames … … Event Detection Event Interpretation “A document moved from (x 1,y 1 ) to (x 2,y 2 )” Document Recognition beforeafter File1.pdf File2.pdf File3.pdf Scene Graph Update Desk

72. UIST 2004Kim, Seitz, Agrawala Document Recognition … File1.pdf File2.pdfFile3.pdfFile4.pdfFile5.pdfFile6.pdf Match against PDF image database …

73. UIST 2004Kim, Seitz, Agrawala Document Recognition Performance analysis –Tested 20 pages against database of 162 pages

74. UIST 2004Kim, Seitz, Agrawala Document Recognition Performance analysis –Tested 20 pages against database of 162 pages –~200x300 pixels per document for reliable match Document Resolution Recognition Rate

75. UIST 2004Kim, Seitz, Agrawala Document Recognition Performance analysis –Tested 20 pages against database of 162 pages –~200x300 pixels per document for reliable match Document Resolution Recognition Rate 300 0.9

76. UIST 2004Kim, Seitz, Agrawala Algorithm Overview Input Frames … … Event Detection Event Interpretation “A document moved from (x 1,y 1 ) to (x 2,y 2 )” Document Recognition beforeafter File1.pdf File2.pdf File3.pdf Scene Graph Update Desk

77. UIST 2004Kim, Seitz, Agrawala Scene Graph Update before after Motion: (x,y,θ) Desk

78. UIST 2004Kim, Seitz, Agrawala Scene Graph Update before after Motion: (x,y,θ) Desk

79. UIST 2004Kim, Seitz, Agrawala Scene Graph Update before after Motion: (x,y,θ) Desk

80. UIST 2004Kim, Seitz, Agrawala Results Input video –~40 minutes –1024x768 @ 15 fps –22 documents, 49 events Running time –Video processed offline –No optimization –A few hours for entire video

81. UIST 2004Kim, Seitz, Agrawala Demo – Paper tracking

82. UIST 2004Kim, Seitz, Agrawala Photo Sorting Example

83. UIST 2004Kim, Seitz, Agrawala Photo Sorting Example

84. UIST 2004Kim, Seitz, Agrawala Demo – Photo Sorting

85. UIST 2004Kim, Seitz, Agrawala Future Work Enhance realism More applications

86. UIST 2004Kim, Seitz, Agrawala Future Work Enhance realism –Handle more realistic desktops

87. UIST 2004Kim, Seitz, Agrawala Moving a stack of documents

88. UIST 2004Kim, Seitz, Agrawala Documents with no electronic versions

89. UIST 2004Kim, Seitz, Agrawala Future Work Enhance realism –Handle more realistic desktops –Real-time performance

90. UIST 2004Kim, Seitz, Agrawala Future Work More applications –Support other document tasks E.g., attach reminder, cluster documents

91. UIST 2004Kim, Seitz, Agrawala Future Work More applications –Support other document tasks E.g., attach reminder, cluster documents –Beyond documents

92. UIST 2004Kim, Seitz, Agrawala Future Work More applications –Support other document tasks E.g., attach reminder, cluster documents –Beyond documents

93. UIST 2004Kim, Seitz, Agrawala Future Work More applications –Support other document tasks E.g., attach reminder, cluster documents –Beyond documents

94. UIST 2004Kim, Seitz, Agrawala Future Work More applications –Support other document tasks E.g., attach reminder, cluster documents –Beyond documents

95. UIST 2004Kim, Seitz, Agrawala Acknowledgments NSF Intel Corp. Li Zhang