1. Lecture 19: Physical Gadgets and their Interaction Techniques
Brad Myers
: Interaction Techniques
Spring, 2019
© Brad Myers

2. Definitions and Synonyms
Ubiquitous computing (UbiComp) - computing everywhere and anywhere
Pervasive computing – (no separate definition)
Ambient intelligence (mostly used in Europe) – environment is instrumented so it is sensitive and responsive to people
Information appliances – Smartphone or PDA
Context-aware computing – mobile device that knows its surroundings, such as location, light, sound, etc.
Tangible user interfaces (TUIs) -- person interacts with digital information through the physical environment
Formerly “graspable UIs”
Has its own conference series:
TEI’19: 13th International Conference on Tangible, Embedded and Embodied Interaction, The Netherlands, March 17-20, 2019 in Tempe, Arizona
Internet of Things (IoT)
© Brad Myers

3. Definitions, cont. Physical Gadgets
Are to physical (tangible) user interfaces what interaction techniques are to graphical user interfaces
Adapted from [Greenberg’01]
An interaction technique embodied in a physical entity
Must be reusable
Many other TUIs are tabletop interactions with physical objects sensed on a table with a projector
ACM International Conference on Interactive Surfaces and Spaces (formerly ACM ITS, International Conference on Interactive Tabletops and Surfaces
© Brad Myers

4. Scope
There are lots of interesting, cute, even useful applications of tangible and ubiquitous user interfaces
Most are not interaction techniques
E.g., Ambient displays – no interaction
Mankoff’s BusMobile
E.g., Tangible applications – not a reusable widget
Bottles that play sounds when opened
© Brad Myers

5. Logo “Turtle”
From 1967 by Daniel G. Bobrow, Wally Feurzeig, Seymour Papert and Cynthia Solomon at MIT
Simplified programming for children
Originally drove a physical turtle on the floor with a physical pen
Pen up / down
Walk forward / turn
© Brad Myers

6. Lego Mindstorms Introduced in 1998
Named after Seymour Papert’s book
Original kit contained light sensors, buttons, touch sensors, motors, etc.
1st version programmed using “RCX code”
Blocks language implemented in Macromedia Director
Could also be downloaded from other languages
Slow processor, low-quality sensors and actuators
© Brad Myers

7. Phidgets “Physical widgets”
Saul Greenberg and Chester Fitchett Phidgets: easy development of physical interfaces through physical widgets. In Proceedings of the 14th annual ACM symposium on User interface software and technology (UIST '01). ACM, pp
“Physical widgets”
Previously was very difficult to build TUIs
Had to build custom hardware and microprocessors
Soldering, circuit design (EE), assembly-language programming, etc.
Lots of new sensors
Encapsulated complexities of using physical objects
Lights, motors, sensors, cameras, switches, etc.
Mostly USB
Interactive since sensors for motion, light, sound, etc.
Sensor+control counts as interaction technique, not just a moving flower
Programmed (originally) in Visual Basic
Simulation mode to help create the software
Formed a company to market his phidgets
Video, 6:10 (2001)
© Brad Myers

8. Controllers Started about 2005 with Arduino http://arduino.cc/
Single-board microcontroller
Open source electronics prototyping platform
Now about $9 to $30 each
Easy to program and attach devices to
Still a solid choice
Current alternatives (from Chris Harrison)
- cheap and super popular
- ODroid - more powerful single board computer
- faster/smaller version of Arduino
If you want to start from a phone, there is the IOIO-OTG board for android:
© Brad Myers

9. Toys and Robots Many toys Many robots
Allison Druin NOOBIE: the animal design playstation. SIGCHI Bull. 20, 1 (July 1988),
Giant stuffed animal with sensors and a screen
Many toys
E.g., Furby from 1998 by Tiger Electronics
Many robots
Some programmed by example
Rethink Robotics
Nao
Boston Dynamics
© Brad Myers

10. Research: Hiroshi Ishii
Tangible Media Group:
He will be awarded the Lifetime Achievement award at CHI’2019
Dozens of projects dating back to 1990
But most are not “interaction techniques”
One that is:
John Underkoffler and Hiroshi Ishii Urp: a luminous-tangible workbench for urban planning and design. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (CHI '99). ACM, pp
Physical tools for measuring, changing building material, turning on wind, changing light paths, etc.
© Brad Myers

11. Ishii, cont.
Hiroshi Ishii, Dávid Lakatos, Leonardo Bonanni, and Jean-Baptiste Labrune Radical atoms: beyond tangible bits, toward transformable materials. interactions 19, 1 (January 2012),
Includes a survey of tangible Uis
Lists lots of toolkits to create TUIs
“Tangible design seeks an amalgam of thoughtfully designed interfaces embodied in different materials and forms in the physical world—soft and hard, robust and fragile, wearable and architectural, transient and enduring.”
Future: physical-digital “atoms” that can transform, conform and inform
E.g., “clay” that changes its own shape based on rules, user commands, & constraints
© Brad Myers

12. More Research: Skweezee
Karen Vanderloock, Vero Vanden Abeele, Johan A.K. Suykens, and Luc Geurts The skweezee system: enabling the design and the programming of squeeze interactions. InProceedings of the 26th annual ACM symposium on User interface software and technology(UIST '13). ACM, pp
Soft tangible objects, filled with conductive padding and embedded sensors (eight electrodes)
Toolkit for defining squeeze gestures by example
Learns from a single example
© Brad Myers

13. “Wearable” technology
Fitbits, Apple Watch, etc.
Project Jacquard Levi's Jean Jacket with the touch sensitive cuff:
“Jacquard™ by Google woven in”
Video:
$350
© Brad Myers

14. 1997 Portable UbiComp
S. Feiner, B. MacIntyre, T. Höllerer, and T. Webster, A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Proc. ISWC '97 (First IEEE Int. Symp. on Wearable Computers), October 13-14, 1997, Cambridge, MA. Also in Personal Technologies, 1(4), 1997, pp
head-tracked, see-through, headworn, 3D display, and an untracked, opaque, handheld, 2D display with stylus and trackpad, GPS
Compare to modern smartphone!
© Brad Myers

15. Research: Hand-Held Projectors
Xiang Cao, Clifton Forlines, and Ravin Balakrishnan Multi-user interaction using handheld projectors. UIST '07. ACM,
Each person holds a projector with integrated camera
Interact by moving projector or items in the scene
Extra buttons on projector or externally
Video (6:08)
Current projectors are the size of a smartphone
Sony Portable HD Mobile Projector (MPCL1)
© Brad Myers

16. Hybrid mobile device and physical
Sensors on smartphones – physical manipulate the phone itself
Also mobile + environment
Early example: Jun Rekimoto and Masanori Saitoh Augmented surfaces: a spatially continuous work space for hybrid computing environments. CHI '99, video 3:14
Andy Wilson at Microsoft research has lots, e.g.: Andrew D. Wilson PlayAnywhere: a compact interactive tabletop projection-vision system. In UIST '05. ACM, videos
© Brad Myers

17. 3D printed physical gadgets
Marynel Vázquez, Eric Brockmeyer, Ruta Desai, Chris Harrison, and Scott E. Hudson D Printing Pneumatic Device Controls with Variable Activation Force Capabilities. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15),
Pneumatic actuation
Still need a hefty air pump (not shown)
© Brad Myers

18. Scott Hudson’s class
05-833: “Applied Gadgets, Sensors and Activity Recognition in HCI”
Not this year (Scott is on sabbatical)
Ink and slide – Adobe, Adonit
© Brad Myers