PERVASIVE COMPUTING AT NIST Martin Herman, Chief INFORMATION ACCESS & USER INTERFACES DIVISION INFORMATION TECHNOLOGY LABORATORY NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

VISION Small inexpensive computers and sensors in every device, appliance and piece of equipment in buildings, offices, classrooms, homes, cars, factories people carrying portable and wearable computers networked to each other and the Internet “A trillion devices connected to the Net!" sensing and reacting intelligently to the environment Information instantly accessible anywhere and anytime An emerging new paradigm for IT Enormous new global market transforming the way U.S. industry does business transforming the way people live and learn requiring the development of a measurement and standards infrastructure ensuring that U.S. manufacturers are not at a disadvantage

NIST’s Role n Pervasive computing is in infancy n Many required technologies are immature & have high risk n NIST’s unique role –develop metrics, tests and standards to push forward underlying technologies and help pervasive computing mature and grow –develop test data and test scenarios/tasks –provide impartial, large-scale evaluations across many different systems and companies –develop interoperability specifications and test methods –perform & provide tools for conformance testing

Current Industry Efforts n Information appliances –Web TVs, screen phones, net-enabled smart handheld computers, internet gaming consoles, internet automotive systems, electronic books n Integration into Internet –wireline high-speed access (high-speed data over phone lines; cable) –wireless high-speed access (terrestrial/LMDS; satellite services) –short-range wireless LANs (HomeRF, Bluetooth) –networking/communication protocols for information appliances (Jini, Universal Plug & Play, Inferno, Odyssey, T spaces, JetSend) n Applications –networked household (TV set-top boxes, handheld communicators, PDAs, PCs, laptops, printers/scanners, digital cameras) –networked automobile (GPS, PDAs, cell phones, video games, e- mail, voice mail, web surfing, navigation assistance) –BodyLANs (e.g., FedEx integration of handheld computers, scanners & printers to determine status of packages)

Proposed NIST Efforts n Measurements and standards for long-term as well as short-term needs. n Rich, natural human-computer interaction n Extensive information presentation & visualization n Dynamic networking n Sophisticated multi-media information extraction n Collaborative working environments n Security & reliability n System integration and interoperability n Tight integration of IT with electronic, semiconductor, optical, manufacturing, chemical, materials, building and fire technologies

Worldwide Information Appliance Forecast Unit Shipments (000)

Appliances vs PCs US Shipments, Consumer Devices, In Millions

Applications & Scenarios Personal Assistants Extension of today’s pocket organizers, smart cards, smart mobile phones New functions: electronic money clip, locator, visual communicator Provide access to personal data, monitor offices & homes, transfer funds, make secure transactions Increase effectiveness of people’s time; extend the range of their actions Personal assistant for youth/elderly Provide ID information Vital data GPS-based location Remote transmission of location “Help me” communication function Access to medical information Funds accessible for emergencies Voice I/O Embedded camera for visual interaction Personal assistant for adult Secure access to information Embedded biometric sensors Coordinate user’s schedule Retrieve & store phone, fax, Personal e-business Electronic purchasing of tickets Electronic toll payments Electronic banking Integration with office computer, printer, in-car digital maps, WWW Voice I/O, voice-to-text, text-to-voice Embedded camera for visual interaction

Applications & Scenarios Smart Spaces Smart Spaces are work spaces embedded with computers, information appliances, and multi-modal sensors that allow people to work efficiently, together or individually, through unprecedented access to information and help from computers. Smart Space Examples Business meeting room Medical consultation room Training/education facility Military command center Crisis management center Identify and perceive users and their actions and goals Speech, natural language, computer vision input Interaction with information-rich sources Provide for the use of mobile devices and receivers Provide extensive information-presentation capabilities Understand and anticipate user needs during task performance Distributed and local collaboration, including collaboration with field personnel & mobile workers Provide improved memory and summaries of activities and deliberations for later use

Technologies for ITL Involvement Rich, natural HCI Speech & natural language understanding Interactive dialogue Text-to-speech generation Detecting, localizing, & tracking people Recognizing people Activity & gesture recognition Situation awareness Summarizing activities & events recorded by sensors Computer vision processing Distributed multi-sensor fusion & integration Integrated information presentation & visualization Dynamic networking Mobile/wireless communications Real-time system & communications protocol Data compression Distributed databases Multi-media information extraction Document retrieval Question answering Collaborative working environments Distributed & local collaboration Security Computer/Network Security Biometric authentication System integration technologies & interoperability

Perceptive Interface Overview VideoDSP WorldModelPatternClassifiers AcousticDSP DisplayServices

Smart Space Test Bed Camera Elements Large Screen Display Microphone Phased Array Array Beams Equipment Room n 64 ADCs/Microphones n TI 320C32 60MHz with 512k of RAM n 3 cameras/video capture n Gigabit ethernet n Servers for data n Pervasive devices –Palm tops –Tablets –Wireless LAN

One-dimensional Microphone Array Geometry Direction of propagation k Sensor Locations Z m -k IAUI Prototype Analog Arrays 59 Microphones at 15 mm Inter-sensor distances 45 Microphones at 20 mm Inter-sensordistances

Frontal Face Detection Skin pixel extraction Face localization Template matching Largest cluster of skin pixel coordinate Template

Cropping Edge detector Hough Transform Face feature extraction n Crop largest skin pixel cluster from frame n Apply: –Edge detector to emphasize circular shapes –Hough Transform for eye-extraction –Face Normalization to gallery conditions