Ubiquitous Computing Lecture 28

Topics we shall Cover today Introduction to Ubiquitous Computing History Definition Need Phases Challenges and Researches in Ubiquitous Computing.

The Trends in Computing Technology Late 1990s Now and Tomorrow ?

Pervasive Computing Era

Computing Evolution

The Major Trends in Computing 4/21/2017 The Major Trends in Computing Mainframe (Past) 1:N one computer shared by many people  Personal Computer (Present) 1:1 one computer, one person   Ubiquitous Nk:1 Computing N:1 *Internet - Widespread Distributed Computing* In the past fifty years of computation there have been two great trends in this relationship: the mainframe relationship, and the PC relationship. Today the Internet is carrying us through an era of widespread distributed computing towards the relationship of ubiquitous computing, characterized by deeply imbedding computation in the world.

Phase I - The Mainframe Era 4/21/2017 Phase I - The Mainframe Era Computers were a scarce resource run by experts behind closed doors. The first era we call "mainframe", to recall the relationship people had with computers that were mostly run by experts behind closed doors. Anytime a computer is a scarce resource, and must be negotiated and shared with others, our relationship is that of the mainframe era.

4/21/2017 Phase II - The PC Era In 1984 the number of people using PCs surpassed that of people using mainframe computers. PC Era: You have your computer, it contains your stuff, and you interact directly and deeply with it. The PC is most analogous to the automobile. Now we are in the personal computing era, person and machine staring uneasily at each other across the desktop. The second great trend is that of the personal computer. In 1984 the number of people using personal computers surpassed the number of people using shared computers. The personal computer is most analogous to the automobile - a special, relatively expensive item, that while it may "take you where you want to go", requires considerable attention to operate. Some people name their PC - many people curse or complain to their PC. Any computer with which you have a special relationship, or that fully engages or occupies you when you use it, is a personal computer.

Transition Phase - The Internet 4/21/2017 Transition Phase - The Internet The Internet brings together elements of the mainframe era and the PC era.  Client = PC Server = Mainframe It is client-server computing on a massive scale, with web clients the PCs and web servers the mainframes.

4/21/2017 Phase III - The UC Era The UC era will have lots of computers shared by each one of us. UC is fundamentally characterized by the connection of things in the world with computation. Frequently used related terms: Pervasive computing, Wearable computers, Intelligent environment, Things That Think (T³), Wearware, Personal Area Networking (PAN). Some of these computers will be the hundreds we may access in the course of a few minutes of Internet browsing. Others will be imbedded in walls, chairs, clothing, light switches, cars - in everything. All kinds of jargon. . Wireless internet. Peripheral computing. Self-configuring, adaptively coordinated Embedded Nets.. Locator Tags. JINI.

Ubiquitous Computing Mark Weiser, Xerox PARC 1988 “Ubiquitous computing enhances computer use by making many computers available throughout the physical environment, but making them effectively invisible to the user.” Source: Weiser, 1993a

Pervasive (Ubiquitous) Computing Vision “In the 21st century the technology revolution will move into the everyday, the small and the invisible…” “The most profound technologies are those that disappear. They weave themselves into the fabrics of everyday life until they are indistinguishable from it.” Mark Weiser (1952 –1999), XEROX PARC Small, cheap, mobile processors and sensors in almost all everyday objects on your body (“wearable computing”) embedded in environment (“ambient intelligence”)

Related Topics Several terms that share a common vision Pervasive Computing Sentient computing Ubiquitous Computing Ambient Intelligence Wearable Computing Context Awareness ...

What is Ubiquitous Computing? Ubiquitous computing (ubicomp) integrates computation into the environment, rather than having computers which are distinct objects. The idea of ubicomp enable people to interact with information-processing devices more naturally and casually, and in ways that suit whatever location or context they find themselves in. ~from Wiki

Goals of Pervasive (Ubiquitous) Computing Ultimate goal: Invisible technology Integration of virtual and physical worlds Throughout desks, rooms, buildings, and life Take the data out of environment, leaving behind just an enhanced ability to act

4/21/2017 What UC is NOT It is not science fiction (SF), though it relies a great deal on it. It is not impossible. It is not Virtual Reality (VR). It is not a Personal Digital Assistant (PDA). It is not a personal agent (PA). It cannot be mistaken for VR, ever They are not “multimedia computers” Ubiquitous computing is roughly the opposite of virtual reality It is not a Personal Digital Assistant (PDA) such as Apple's Newton. It is not a personal or intimate computer (IA)with agents doing your bidding. UC vs. VR Virtual reality puts people inside a computer-generated world. Ubiquitous computing forces the computer to live out here in the world with people. Virtual reality is primarily a horse power problem Ubiquitous computing is a very difficult integration of human factors, computer science, engineering, and social sciences. Unlike virtual reality, ubiquitous computing endeavers to integrate information displays into the everyday physical world. It considers the nuances of the real world to be wonderful, and aims only to augment them. Diametrically opposed to our vision is the notion of "virtual reality," which attempts to make a world inside the computer. Users don special goggles that project an artificial scene on their eyes; they wear gloves or even body suits that sense their motions and gestures so that they can move about and manipulate virtual objects. Although it may have its purpose in allowing people to explore realms otherwise inaccessible -- the insides of cells, the surfaces of distant planets, the information web of complex databases -- virtual reality is only a map, not a territory. The opposition between the notion of virtual reality and ubiquitous, invisible computing is so strong that some of us use the term "embodied virtuality" to refer to the process of drawing computers out of their electronic shells. UC vs. PDA Unlike PDA's, ubiquitious computing envisions a world of fully connected devices, with cheap wireless networks everywhere Unlike PDA's, it postulates that you need not carry anything with you, since information will be accessable everywhere. UC vs. IA Unlike the intimate agent computer that responds to one's voice and is a personal friend and assistant, ubiquitous computing envisions computation primarily in the background where it may not even be noticed. Whereas the intimate computer does your bidding, the ubiquitous computer leaves you feeling as though you did it yourself. World Is Not a Desktop: Take intelligent agents. The idea, as near as I can tell, is that the ideal computer should be like a human being, only more obedient. Anything so insidiously appealing should immediately give pause. Why should a computer be anything like a human being? Are airplanes like birds, typewriters like pens, alphabets like mouths, cars like horses? Take magic. The idea, as near as I can tell, is to grant wishes: I wish my computer would only show me what I am interested in. Take virtual reality. The idea, as near as I can tell, is that by moving to full-body-sensing and interaction we'll solve the user interface problem by maximally utilizing all of our body's input and output channels. Take voice input. The idea, as near as I can tell, is that if I could just talk to my computer it would finally understand me. The problem is, if I could talk to my computer today, I'd have to talk in C or Fortran or CSH, because that is what they understand. I do think that research on agents, speech recognition, and so on is important; the problem is that they are all in the domain of the conscious interaction.

Phases of Ubiquitous Computing

Pervasive Computing Phase I Smart, ubiquitous I/O devices: tabs, pads, and boards Hundreds of computers per person, but casual, low-intensity use Many, many “displays”: audio, visual, environmental Wireless networks Location-based, context-aware services Using a computer should be as refreshing as a walk in the woods

Smart Objects Real world objects are enriched with information processing capabilities Embedded processors in everyday objects small, cheap, lightweight Communication capability wired or wireless spontaneous networking and interaction Sensors and actuators

Smart Objects (cont.) Can remember pertinent events They have a memory Show context-sensitive behavior They may have sensors Location/situation/context awareness Are responsive/proactive Communicate with environment Networked with other smart objects

Smart Objects (cont.)

Pervasive Computing Enablers Moore’s Law of IC Technologies Communication Technologies Material Technologies Sensors/Actuators

Moore’s Law Computing power (or number of transistors in an integrated circuit) doubles every 18 months

Moore’s Law 1965 Computing power (or number of transistors in an integrated circuit) doubles every 18 months

Generalized Moore’s Law Most important technology parameters double every 1–3 years: computation cycles memory, magnetic disks bandwidth Consequence: scaling down Problems: • increasing cost • energy

2nd Enabler: Communication Bandwidth of single fibers ~10 Gb/s 2002: ~20 Tb/s with wavelength multiplex Powerline coffee maker “automatically” connected to the Internet Wireless mobile phone: GSM, GPRS, 3G wireless LAN (> 10 Mb/s) PAN (Bluetooth), BAN

Body Area Networks Very low current (some nA), some kb/s through the human body Possible applications: Car recognize driver Pay when touching the door of a bus Phone configures itself when it is touched nA= nanoampere

Spontaneous Networking Objects in an open, distributed, dynamic world find each other and form a transitory community Devices recognize that they “belong together”

3rd Enabler: New Materials Important: whole eras named after materials e.g., “Stone Age”, “Iron Age”, “Pottery Age”, etc. Recent: semiconductors, fibers information and communication technologies Organic semiconductors change the external appearance of computers “Plastic” laser Flexible displays,… Plastic laser – means technology being so advanced that visual items are flexible or foldable.

Interactive Map Foldable and rollable You are here!

Foldable Cell Phone

Smart Clothing Conductive textiles and inks print electrically active patterns directly onto fabrics Sensors based on fabric e.g., monitor pulse, blood pressure, body temperature Invisible collar microphones Kidswear game console on the sleeve? integrated GPS-driven locators? integrated small cameras (to keep the parents calm)?

Solar Coat – Smart Clothings

Smart Glasses By 2009, computers will disappear. Visual information will be written directly onto our retinas by devices in our eyeglasses and contact lenses -- Raymond Kurzweil

Google Glass

4th Enabler: Sensors/Actuators Miniaturized cameras, microphones,... Fingerprint sensor Radio sensors RFID Infrared Location sensors e.g., GPS ...

Example: Radio Sensors No external power supply energy from the actuation process piezoelectric and pyroelectric materials transform changes in pressure or temperature into energy RF signal is transmitted via an antenna (20 m distance) Applications: temperature surveillance, remote control (e.g., wireless light switch),...

RFIDs (“Smart Labels”) Identify objects from distance small IC with RF-transponder Wireless energy supply ~1m magnetic field (induction) ROM or EEPROM (writeable) ~100 Byte Cost ~$0.1 ... $1 consumable and disposable Flexible tags laminated with paper

Past, Present and Future Researches of Ubiquitous Computing

Computing with natural interfaces Ubicomp inspires “off-the-desktop” applications Needs “off-the-desktop” means of interaction Speech, gestures, writing More accessible Easier to use???

Computing with natural interfaces Error prone interaction Permit new and numerous mistakes People do not have perfect recognition As low as 54%; cursive handwriting 88%; printed handwriting 96.8% Recognition accuracy == user satisfaction?? Not really: complexity of error recovery dialogues and value-added benefit of any given efforts Entering a command vs. writing journal entries Several research areas Error reduction (about 5-10%) Error detection Reusable toolkit for error handling

Context aware computing Current Systems Generally using position and identification of objects Still do not provide a complete context Definition of context is limited Research areas Context toolkits Toolkit for sensing environment Explicit use of sensed information is up to program What is context? How is context represented?

What is context? Who What Where When Why Currently generally tailored to one user How important are others in determining our behavior How could this be captured? What Attempt to figure out what is currently happening Sense environment, use calendar software etc. Where Location based information, e.g., GPS Most explored context information When Easily obtained information -- Computer is good at remembering time Although determining when one event stops and another begins is not easy Why Even harder than the “what” question, biometric sensors might help (e.g., body temperature, heart rate, etc)

Toward context aware computing Context representation Requires universal context schemes or toolkits with standard context representations Context sensing and fusion How to make context-aware computing “ubiquitous”? In practice, there are few truly ubiquitous, single-source context services E.g., GPS does not work indoors; different indoor localization schemes have different characteristics (e.g., cost, range) Like sensor fusion, context fusion handles seamless handling of sensing responsibility between boundaries of different context services Combining multiple context sources can increase the accuracy of context information

Automated capture and access Recording information and data as it occurs Computers are inherently good at recording, people are not People freed up to summarize and understand Most work in academic/ classroom settings Time stamping lectures, digital whiteboards Challenges in “capture and access” Sometime we don’t know we want to capture something until after its already happened How could the computer know that? If it captures everything then we need a system of sorting and filtering (access) Access is a problem because capturing of raw data can be burdensome for sifting through; systems need to recognize important events facilitate access

Everyday computing Continuous interactions (i.e., no clear beginning or end) Both fundamental activities like communication and long-term endeavors do not have predefined starts and ends; information from past can be recycled Very different traditional HCI design which assumes “closure” with clear goals like spell checking, dialogue, etc. Interruption is expected: People are constantly interrupted Computer systems must recognize interruption and change state Also computers must appropriately inform users Multiple activities operate concurrently: People multitask and rapidly switch task based on external unpredictable environment Systems need to adapt to this opportunistic behavior and change accordingly

Toward everyday computing Develop continuously present interface No current model of continuously present interfaces, even people are not continuously present Create an interface that doesn’t get annoying (e.g., wearable devices) Determine what information should require my attention and what should be display peripherally Connect events in the physical and virtual worlds (e.g., face to face vs. email, document, webs) Modify/fuse existing HCI schemes to efficiently support everyday computing (but evaluation is challenging and laborious)

System evaluation challenges Hard to evaluate Ubicomp Systems Little publish on ubicomp evaluation Systems often required to be fully connected leading to systems that are hard to build Lack of development toolkits make system creation difficult Systems often need to be integrated into peoples lives which using big clunky prototypes does not lead itself well too Task/Goal centric approaches don’t work in ubicomp

Example Projects Pervasive computing projects have emerged at major universities and in industry: Project Aura (Carnegie Mellon University) Oxygen (Massachusetts Institute of Technology) Portalano (University of Washington) Endeavour (University of California at Berkeley) Place Lab (Intel Research Laboratory at Seattle) For illustration let us look at Project Aura

Example Projects : Project Aura (1) Aura (Carnegie Mellon University) Distraction-free (Invisible) Ubiquitous Computing. The most precious resource in a computer system is no longer its processor, memory, disk or network. Rather, it is a resource not subject to Moore's law: User Attention. Today's systems distract a user in many explicit and implicit ways, thereby reducing his effectiveness. Project Aura will fundamentally rethink system design to address this problem. Aura's goal is to provide each user with an invisible halo of computing and information services that persists regardless of location. Meeting this goal will require effort at every level: from the hardware and network layers, through the operating system and middleware, to the user interface and applications. Project Aura will design, implement, deploy, and evaluate a large-scale system demonstrating the concept of a “personal information aura” that spans wearable, handheld, desktop and infrastructure computers. PROJECT WEBSITE: http://www.cs.cmu.edu/~aura/

Example Projects : Project Aura (2) Moore’s Law Reigns Supreme Processor density Processor speed Memory capacity Disk capacity Memory cost ... Glaring Exception Human Attention Human Attention The most precious resource in a computer system is no longer its processor, memory, disk or network. Rather, it is a resource not subject to Moore's law: User Attention. Today's systems distract a user in many explicit and implicit ways, thereby reducing his effectiveness. Project Aura will fundamentally rethink system design to address this problem. Aura's goal is to provide each user with an invisible halo of computing and information services that persists regardless of location. Meeting this goal will require effort at every level: from the hardware and network layers, through the operating system and middleware, to the user interface and applications. Project Aura will design, implement, deploy, and evaluate a large-scale system demonstrating the concept of a “personal information aura” that spans wearable, handheld, desktop and infrastructure computers. PROJECT WEBSITE: http://www.cs.cmu.edu/~aura/ Adam & Eve 2000 AD

Example Projects : Project Aura (3) Aura Thesis: The most precious resource in computing is human attention. Aura Goals: Reduce user distraction. Trade-off plentiful resources of Moore’s law for human attention. Achieve this scalably for mobile users in a failure-prone, variable-resource environment. The most precious resource in a computer system is no longer its processor, memory, disk or network. Rather, it is a resource not subject to Moore's law: User Attention. Today's systems distract a user in many explicit and implicit ways, thereby reducing his effectiveness. Project Aura will fundamentally rethink system design to address this problem. Aura's goal is to provide each user with an invisible halo of computing and information services that persists regardless of location. Meeting this goal will require effort at every level: from the hardware and network layers, through the operating system and middleware, to the user interface and applications. Project Aura will design, implement, deploy, and evaluate a large-scale system demonstrating the concept of a “personal information aura” that spans wearable, handheld, desktop and infrastructure computers. PROJECT WEBSITE: http://www.cs.cmu.edu/~aura/

Example Projects : Project Aura (4) The Airport Scenario Jane wants to send e-mail from the airport before her flight leaves. She has several large enclosures She is using a wireless interface She has many options. Simply send the e-mail Is there enough bandwidth? Compress the data first Will that help enough? Pay extra to get reserved bandwidth Are reservations available? Send the “diff” relative to older file Are the old versions around? Walk to a gate with more bandwidth Where is there enough bandwidth? How do we choose automatically? The most precious resource in a computer system is no longer its processor, memory, disk or network. Rather, it is a resource not subject to Moore's law: User Attention. Today's systems distract a user in many explicit and implicit ways, thereby reducing his effectiveness. Project Aura will fundamentally rethink system design to address this problem. Aura's goal is to provide each user with an invisible halo of computing and information services that persists regardless of location. Meeting this goal will require effort at every level: from the hardware and network layers, through the operating system and middleware, to the user interface and applications. Project Aura will design, implement, deploy, and evaluate a large-scale system demonstrating the concept of a “personal information aura” that spans wearable, handheld, desktop and infrastructure computers. PROJECT WEBSITE: http://www.cs.cmu.edu/~aura/

Example Projects : Project Aura (5) The Mobile Task Scenario Aura saves Scott’s task. Scott enters office and gets strong authentication and secure access. Aura restores Scott’s task on desktop machine and uses a large display. Scott controls application by voice. Bradley enters room. Bradley gets weak authentication, Scott’s access changes to insecure. Aura denies voice access to sensitive email application. Scott has multi-modal control of PowerPoint application. Aura logs Scott out when he leaves the room. The most precious resource in a computer system is no longer its processor, memory, disk or network. Rather, it is a resource not subject to Moore's law: User Attention. Today's systems distract a user in many explicit and implicit ways, thereby reducing his effectiveness. Project Aura will fundamentally rethink system design to address this problem. Aura's goal is to provide each user with an invisible halo of computing and information services that persists regardless of location. Meeting this goal will require effort at every level: from the hardware and network layers, through the operating system and middleware, to the user interface and applications. Project Aura will design, implement, deploy, and evaluate a large-scale system demonstrating the concept of a “personal information aura” that spans wearable, handheld, desktop and infrastructure computers. PROJECT WEBSITE: http://www.cs.cmu.edu/~aura/

Other Scenarios of Ubiquitous Computing Buy drinks by Friday (1) Take out the last can of soda Swipe the can’s UPC label, which adds soda to your shopping list Make a note that you need soda for the guests you are having over this weekend

Other Scenarios Buy drinks by Friday (2) Approach a local supermarket AutoPC informs you that you are near a supermarket Opportunistic reminder: “If it is convenient, stop by to buy drinks.”

Other Scenarios Buy drinks by Friday (3) Friday rolls around and you have not bought drinks Deadline-based reminder sent to your pager

Other Scenarios Screen Fridge Provides: Email Video messages Web surfing Food management TV Radio Virtual keyboard Digital cook book Surveillance camera

Other Scenarios The Active Badge This harbinger of inch-scale computers contains a small microprocessor and an infrared transmitter. The badge broadcasts the identity of its wearer and so can trigger automatic doors, automatic telephone forwarding and computer displays customized to each person reading them. The active badge and other networked tiny computers are called tabs.

Other Scenarios The Active Badge

Other Scenarios Edible computers: The pill-cam Miniature camera Diagnostic device It is swallowed Try this with an ENIAC computer!

Other Scenarios Artificial Retina Direct interface with nervous system Whole new computational paradigm (who’s the computer?)

Other Scenarios Smart Dust Nano computers that couple: Sensors Computing Communication Grids of motes (“nano computers”)

Summary Moving our focus of interaction away from the traditional two-dimensional graphical user interface on the desktop presents many exciting and new challenges to the field of HCI. Weiser’s vision of ubiquitous computing was human-centered, and many years later, it still presents a grand challenge for those who wish to address this new interaction paradigm.