SMART SPACES TESTBED Marty Herman, Chief Vince Stanford, Computer Specialist INFORMATION ACCESS AND USER INTERFACES DIVISION NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

Smart Space Embedded with computers, information appliances, multi-modal sensors Allows people to work efficiently through unprecedented access to information and help from computers Work can be done collaboratively or individually

Characteristics of Smart Spaces Situation Awareness Identify and perceive users and their actions and goals Understand and anticipate user needs Uses speech and natural language understanding, computer vision, multi- modal fusion Information Access Facilitate interaction with information-rich sources Allow rich, natural forms of interaction between humans and computers Provide multi-media (e.g., text, voice, images, video) information extraction Information presentation Provide extensive capabilities for presenting information in an optimal and integrated manner Provide immersive presentation capabilities

Characteristics of Smart Spaces (cont.) Collaboration Provide for collaborative working environment Increase the rate of information interchange among users Permit mobile workers to interact and participate in smart space activities Mobile computing Provide for use of mobile devices and receivers, allowing ubiquitous “anywhere, anytime” access Provide for discovery and integration of mobile devices with smart space infrastructure Record keeping Provide improved memory of activities and deliberations for later use Record and summarize dialogue, activities, and events

Create Smart-Space Integration Testbed at NIST Showcase future of Smart Space technologies Integrate component technologies for demos Develop and apply measurement and testing approaches Develop and test prototype standards (e.g., interface standards) Develop test collections to support evaluations Infrastructure for industry and academia to work hand-in-hand with NIST Facilitate adoption by industry

NIST Smart Spaces Testbed Scenario: Smart Meeting Room Room understands and guides meeting participants Senses who is talking to whom and where they are located Realizes when information is requested and outputs information that seems to be useful Engages in dialogue with participants to get more information Room connected to Internet and digital libraries/databases Personal information from palm/pocket computers are integrated into room’s information infrastructure Information displayed on information appliances and wall displays Room aids in collaboration both within the room and with field personnel

Situation awareness Speech, natural language input Computer vision input Integration of handheld computers into room Retrieval & visualization of information Distributed collaboration

Smart Meeting Room Applications: Business meeting room Medical consultation meeting room Training and education facility Military command center Crisis management command center

Role of NIST Smart Spaces is an emerging research area. Many of its component technologies require research to advance the state of the art. NIST can provide the following: measurement methods testing and evaluation approaches benchmark tests reference materials (test data and test protocols)

Role of NIST (cont.) Many technology suppliers will be required to make the vision of Smart Spaces a commercial reality. To encourage commercial innovation, NIST can work with industry to provide open interface standards for interoperability.

Near Term Smart Space Experimental Collaborative interface using: Microphone array, with beam fan, for acoustic signal acquisition Acoustic source location and tracking Continuous speech recognition (CSR) using single channel reduced from array Speaker identification combined with CSR for collaborative groups Shared high resolution visual interface with information visualization While Situation Awareness focuses on a central Smart Space for command, management, and collaborative design, some situations will require mobile elements acting in concert over larger spaces, such as in the FEMA scenario presented in the introduction. Wireless, dynamic wide area networks coupled with fielded sensors will be needed to create wide area smart spaces. As speech transcription, understanding and task modeling improve, it should be possible for a Smart Space to anticipate the data needs of a working group pre-query and queue information before it is actually needed to reduce latency for working groups. Objects will be used to encapsulate audiovisual recordings along with annotated transcripts, and with display and transport methods. There are significant opportunities to develop new generation collaborative interfaces that are not limited to the standard Windows Icons Menus and Pointers (WIMP) interfaces of twenty years ago.

Near Term Smart Space (cont.) Experimental Collaborative interface using: Video camera array to identify, locate and track individuals in the smart space Video teleconferencing Initial test case will use several functions: Broadcast news transcription Real time spoken query processing Information retrieval using BN transcripts While Situation Awareness focuses on a central Smart Space for command, management, and collaborative design, some situations will require mobile elements acting in concert over larger spaces, such as in the FEMA scenario presented in the introduction. Wireless, dynamic wide area networks coupled with fielded sensors will be needed to create wide area smart spaces. As speech transcription, understanding and task modeling improve, it should be possible for a Smart Space to anticipate the data needs of a working group pre-query and queue information before it is actually needed to reduce latency for working groups. Objects will be used to encapsulate audiovisual recordings along with annotated transcripts, and with display and transport methods. There are significant opportunities to develop new generation collaborative interfaces that are not limited to the standard Windows Icons Menus and Pointers (WIMP) interfaces of twenty years ago.

Current Status... Lab space allocated; construction begins: Initial smart space November 23 Acoustically conditioned space 1Q99 Computing, display, sensing infrastructure: Dual Processor SGI Octane 640Mb Ram thirty-two channels A/D Dual camera inputs 1280x1024 large screen projector Parallel Linux cluster: 112Gb disk array Seven 400Mhz P-II compute nodes Gigabit Ethernet switch in room Microphones: Desktop Wall mounted array(s) Cameras: Three computer controlled One digital fixed mount 384Kb video teleconferencing Wireless connection/Palm Pilot External cooperative research efforts: Microphone array/acoustic source locator from Rutgers CAIP; Jim Flanagan P.I. BBN and IBM offering research and real time speech recognition Investigating possible use of MIT speaker identification Investigating possible use of U of Md visual localization and gesture recognition Internal cooperative efforts: Information retrieval (PRISE) Spoken document retrieval with spoken query (SPIDERS) VRML smart space representation Information visualization Face recognition technology

Desired Long Term Capabilities/Technologies Accurate, robust speech recognition for transcription and command Anticipatory Web and database query based on speech Speech understanding: Spoken dialog abstraction Selective recording of meeting minutes Data stream segmentation/annotation Face/expression recognition Persistence of meeting memory, automatic links to related meetings Sensor fusion (acoustic/visual/other) Smart white boards Image understanding and person recognition Participant sensing, task identification, and adaptive response Several new, or substantially improved, capabilities are required. We will need better acoustic signal acquisition which reduces multi-source interference, room reverberation, and background noise. Incremental improvements in speech recognition will be required. Current accuracy rates of about 85 percent have been obtained for the best broadcast news processing systems; accuracy levels not adequate for mission critical spoken interfaces. As accuracy levels increase we will want to support real time processing of recognized speech to anticipate the needs of the Smart Space users as tasks are performed. Mission critical application will require very high accuracy levels. Good robustness in terms of non-voice, and multi-voice rejection is needed. The speech and video channels can be annotated using the recognized speech and gestures, as well as speech understanding and dialog summarization. Archiving of sensor data streams gathered during critical events could allow for more complete postmortem analyses. Automatic content abstracting could be used to generate links into long-term event databases. Sensor fusion will allow correlation among data channels for annotation such as transcripts combining what was said with who said it, or what was written or typed with who did the writing or typing. Smart Whiteboards capturing drawing and writing could be a source of additional documentation of meetings. Image understanding for interfaces including visual gesture recognition , and person tracking for use in fusion with other sensors offers opportunity high performance information processing spaces.

Metrics and Metrics Research Needed End-to-end metrics needed for multi-stage processes, e.g.: Full SDR Sensing Recognition Understanding Response Training and test reference data sets needed for: Component recognition tasks, e.g.: CSR, speaker ID, gesture recognition Data reduction, e.g.: array processing and source location Information extraction and summary Significant research is required in order to construct measurement protocols for mixed initiative systems that allow multiple actions and paths. Experience in the speech recognition community has shown that well drawn measurement programs can be very important to ongoing technical improvements in a new technology. Reference data sets will be needed for the several recognition tasks required in Smart Spaces. Some of these are already under development elsewhere in the DARPA community, but Smart Spaces can provide a test bed for integrated functionality. For example a topic detection and summarization task would use speech recognition, possibly speaker identification, and natural language parsing and discourse analysis. Measurements tasks requiring the use of multiple cascaded and parallel technologies will have to be designed. An initial Smart Space test bed will be required to record and render some of the reference materials. That is: an initial Smart Space will be needed to allow further progress to be made in the art and science of Smart Spaces; development will be iterative. It was noted by the chairman that the military mission planning community has tests and doctrine that could be incorporated into benchmarks and scoring procedures. These offer examples of how to score command team activities in terms of solution time, quality, time to solution, labor costs and other measurements.

Metrics and Metrics Research Needed (cont.) Initial Smart Space is needed as test bed to bootstrap future improvements Command/Crisis Management may offer measurable, reproducible tasks Need well defined tasks with solution quality, time, and labor level measurements Significant research is required in order to construct measurement protocols for mixed initiative systems that allow multiple actions and paths. Experience in the speech recognition community has shown that well drawn measurement programs can be very important to ongoing technical improvements in a new technology. Reference data sets will be needed for the several recognition tasks required in Smart Spaces. Some of these are already under development elsewhere in the DARPA community, but Smart Spaces can provide a test bed for integrated functionality. For example a topic detection and summarization task would use speech recognition, possibly speaker identification, and natural language parsing and discourse analysis. Measurements tasks requiring the use of multiple cascaded and parallel technologies will have to be designed. An initial Smart Space test bed will be required to record and render some of the reference materials. That is: an initial Smart Space will be needed to allow further progress to be made in the art and science of Smart Spaces; development will be iterative. It was noted by the chairman that the military mission planning community has tests and doctrine that could be incorporated into benchmarks and scoring procedures. These offer examples of how to score command team activities in terms of solution time, quality, time to solution, labor costs and other measurements.

Interoperability Issues are Severe Rapid handheld appliance discovery and connection to IP infrastructure, e.g.: IP masquerading or NAT Dense wireless device deployment: IR RF Multi-resolution devices and displays Device protocol discovery, automatic translation among heterogeneous protocols used by multiple devices Multimedia output into the space and acoustic input Security and accessibility to and by infrastructure The group noted that there would be interoperability issues, especially in the areas of wireless IR/RF LANs. Another issue raised was that of devices acting as communication conduits for each other to establish links between incompatible devices. Protocols for network routing will also need to be investigated.