1 Toolkits for Physical Objects (TUIs) Jack Li Advanced User Interface Software November 30 Fall 2004

2 Outline Motivation for TUIs Input Technologies TUI Toolkits

3 Current Limitations of UIs Desktops and laptops suck in our entire attention As a result we live in two worlds: the real physical world and the virtual cyber world Constantly “wired” in order to be in both worlds Loss of rich interactions found in everyday things (particularly haptic) MOTIVATION FOR TUIS

4 Motivation Driven by ubiquitous computing Both people and technology moving away from the desktop into the environment Input: different ways of interacting Output: different ways of displaying MOTIVATION FOR TUIS

5 Tangible User Interfaces TUIs augment the physical world by integrating digital information with everyday physical objects Goal: to bridge the gap between bits and atoms 1. Interactive surfaces 2. Coupling information with physical objects 3. Ambient media for background awareness MOTIVATION FOR TUIS

6 Outline Motivation for TUIs Input Technologies TUI Toolkits

7 Physical Objects as Input Pretty much anything … walls, papers, utensils, clothes hangers! Inspiration sci-fi novels, own imagination everyday life, current practices INPUT TECHNOLOGIES

8 Devices Cameras Electronic tags (RFID technology) Sensors Light Motion Heat Force Motors Orbs INPUT TECHNOLOGIES

9 Framework for the GUI (a)model-view-controller INPUT TECHNOLOGIES

10 Framework for the TUI (b)MCRpd: model-control- representation(physical and digital) INPUT TECHNOLOGIES

11 Outline Motivation for TUIs Input Technologies TUI Toolkits Phidgets (2001) Calder (2004) Papier-Mâché (2004)

12 Toolkit Goals Same as when GUIs were built Make easy for non-specialized experts to create (should not need a vision or hardware guru on the team) Hide dirty implementation details Abstract proper functionality with well- defined API

13 Overview First toolkit for physical objects Reusable physical devices with proper abstraction for creating a wide variety of applications An API that had the right amount of abstraction PHIDGETS

14 GUI Differences Connection manager Way of detecting when devices go on/off Identification Every physical object should have a virtual counterpart Simulation A way to test code even if phidget not present PHIDGETS

15 Architecture PHIDGETS

16 The Physical Device PHIDGETS Primitive device components Sensors Motors Switches Circuit board CY7C63000 USB micro-controller Communications layer USB communication standard

17 Wire Protocol PHIDGETS Low-level software in Windows and microcontroller for communication that’s hidden from end programmers Commands from phidgets Unique Id numbers Device type (servo, powerbar) Specific events (tag for RFID reader) Commands from Windows Set device state (output like servo)

18 Computer Software PHIDGETS PhidgetManager Connection manager Event onAttach() Event onDetach() Int count Enum item Phidget-specific devices All have: String deviceType Boolean isAttached Long serialNumber RFID onTag() Servo onPositionChanged(), motorPosition(index)

19 Simple Application PHIDGETS Phidget Manager Phidget Servo

20 Outline Motivation for TUIs Input Technologies TUI Toolkits Phidgets (2001) Calder (2004) Papier-Mâché (2004)

21 Main goals Primarily geared towards interaction/product designers Support for more refined products Support on the hardware side CALDER

22 Supported Devices CALDER Small size is big win!

23 Fluid prototyping CALDER Objects easily embedded!

24 Communication Wired PIC16C745  C with USB interface engine Same as Phidgets (and iStuff) Wireless Tradeoff between size and power (perhaps Moore’s law will remedy) Small batteries, antenna, and pulse-width modulation CALDER

25 Software Surrogates (GUI widgets) to manipulate physical object output CALDER

26 Outline Motivation for TUIs Input Technologies TUI Toolkits Phidgets (2001) Calder (2004) Papier-Mâché (2004)

27 Overview Emphasis on software side side Input model that provides common programming abstractions Focus on objects being lightweight PAPIER-MACHE

28 Software Abstractions Phob—encapsulates information related to this physical object All—source producer, time RFIDPhob—tag VisionPhob—mean color, bounding box, rotations PhobListener—listens for Phobs produced phobAdded(), phobUpdated(), phobRemoved() PhobProducer—actual devices that generates events PhobManager—maintains PhobProducers PAPIER-MACHE

29 A Common Event Model PhobListener is the interface among all devices Methods are the same phobAdded(), phobUpdated(), phobRemoved() Vision phobAdded() when new object detected, phobUpdated() when old object changed (size, color), phobRemoved() when old object disappears RFID (different from what the paper says) phobAdded() when tag detected, no phobUpdated(), phobRemoved() when tag disappears Facilitates easier retargetting and substitutions (allows for more exploration and faster, prototyping) PAPIER-MACHE

30 Marble Answering Example 1. Get the manager that keeps track of all input devices (PhobProducers) RFIDManager manager = RFIDManager.getManager(); 2. Set up a map associating all objects with an audio clip AssociationFactory factory = new AudioClipFactory(); AssociationTreeTable assocMap = new AssociationTreeTable(factory); 3. Attach this map so new objects are assigned new clips and old ones get played manager.addPhobListenerForAllProducers(assocMap); PAPIER-MACHE

31 A Behavior: Associations A common behavior we noticed New objects get stored/assigned, old objects get retrieved a database of sorts An extension of PhobListener PAPIER-MACHE

32 A Behavior: Associations AssociationTreeTable addClassifierFactoryPair(Classifier, Factory) ObjectClassifier isMemberOfClass(Phob phobToCompare) AssociationFactory createAssociationEltForPhob(Phob phob) PAPIER-MACHE

33 Monitoring Window PAPIER-MACHE

34 Device state—connection mgr PAPIER-MACHE

35 Camera Feedback PAPIER-MACHE

36 Association Map PAPIER-MACHE

37 Extras PAPIER-MACHE

38 Summary

39 Major Toolkit differences Phidgets Complete, balanced, varied assortment of physical objects easy to program with Calder Small size allows for more fluid prototyping at the PHYSICAL level Papier-Mâché Unified event model allows easier development and retargetting at the SOFTWARE level

40 Conclusion TUIs seek to augment the physical world with digital information Toolkit support allows non-experts to develop TUI application, like GUI toolkits Successful toolkits must manage physical connections, gather/alter relevant physical state information, and provide good feedback for these added complexities

41 Future work Input adaptation M to N mapping of input devices Input models What other kinds of behaviors can be abstracted Will be more clear with the development of more applications

42 References Hiroshi Ishii, Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p , March 22-27, 1997, Atlanta, Georgia, United States Ullmer, B. and H. Ishii, Emerging Frameworks for Tangible User Interfaces, in Human-Computer Interaction in the New Millennium, Addison-Wesley. pp , Saul Greenberg, Chester Fitchett. "Phidgets: easy development of physical interfaces through physical widgets," Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida Scott R. Klemmer, Jack Li, James Lin, James A. Landay. "Papier-Mache: toolkit support for tangible input," Proceedings of the 2004 conference on Human factors in computing systems, p , April 24-29, 2004, Vienna, Austria Lee, J.C.; Avrahami, D.; Hudson, S.E.; Forlizzi, J.; Dietz, P.H.; Leigh, D.L., "The Calder Toolkit: Wired and Wireless Components for Rapidly Prototyping Interactive Devices", Designing Interactive Systems (DIS), ISBN: , pp , August 2004