1. SMART SPACES TESTBED Marty Herman, Chief
Vince Stanford, Computer Specialist
INFORMATION ACCESS AND USER INTERFACES DIVISION
NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

2. Smart Space
Embedded with computers, information appliances, multi-modal sensors
Allows people to work efficiently through unprecedented access to information and help from computers
Work can be done collaboratively or individually

3. Characteristics of Smart Spaces
Situation Awareness
Identify and perceive users and their actions and goals
Understand and anticipate user needs
Uses speech and natural language understanding, computer vision, multi-
modal fusion
Information Access
Facilitate interaction with information-rich sources
Allow rich, natural forms of interaction between humans and computers
Provide multi-media (e.g., text, voice, images, video) information extraction
Information presentation
Provide extensive capabilities for presenting information in an optimal and integrated manner
Provide immersive presentation capabilities

4. Characteristics of Smart Spaces (cont.)
Collaboration
Provide for collaborative working environment
Increase the rate of information interchange among users
Permit mobile workers to interact and participate in smart space activities
Mobile computing
Provide for use of mobile devices and receivers, allowing ubiquitous “anywhere, anytime” access
Provide for discovery and integration of mobile devices with smart space infrastructure
Record keeping
Provide improved memory of activities and deliberations for later use
Record and summarize dialogue, activities, and events

5. Create Smart-Space Integration Testbed at NIST
Showcase future of Smart Space technologies
Integrate component technologies for demos
Develop and apply measurement and testing approaches
Develop and test prototype standards (e.g., interface standards)
Develop test collections to support evaluations
Infrastructure for industry and academia to work hand-in-hand with NIST
Facilitate adoption by industry

6. NIST Smart Spaces Testbed Scenario: Smart Meeting Room
Room understands and guides meeting participants
Senses who is talking to whom and where they are located
Realizes when information is requested and outputs information that seems to be useful
Engages in dialogue with participants to get more information
Room connected to Internet and digital libraries/databases
Personal information from palm/pocket computers are integrated into room’s information infrastructure
Information displayed on information appliances and wall displays
Room aids in collaboration both within the room and with field personnel

7. Situation awareness
Speech, natural language input
Computer vision input
Integration of handheld computers into room
Retrieval & visualization of information
Distributed collaboration

8. Smart Meeting Room Applications: Business meeting room
Medical consultation meeting room
Training and education facility
Military command center
Crisis management command center

9. Role of NIST
Smart Spaces is an emerging research area. Many of its component technologies require research to advance the state of the art. NIST can provide the following:
measurement methods
testing and evaluation approaches
benchmark tests
reference materials (test data and test protocols)

10. Role of NIST (cont.)
Many technology suppliers will be required to make the vision of Smart Spaces a commercial reality.
To encourage commercial innovation, NIST can work with industry to provide open interface standards for interoperability.

11. Near Term Smart Space Experimental Collaborative interface using:
Microphone array, with beam fan, for acoustic signal acquisition
Acoustic source location and tracking
Continuous speech recognition (CSR) using single channel reduced from array
Speaker identification combined with CSR for collaborative groups
Shared high resolution visual interface with information visualization
While Situation Awareness focuses on a central Smart Space for command, management, and collaborative design, some situations will require mobile elements acting in concert over larger spaces, such as in the FEMA scenario presented in the introduction. Wireless, dynamic wide area networks coupled with fielded sensors will be needed to create wide area smart spaces.
As speech transcription, understanding and task modeling improve, it should be possible for a Smart Space to anticipate the data needs of a working group pre-query and queue information before it is actually needed to reduce latency for working groups.
Objects will be used to encapsulate audiovisual recordings along with annotated transcripts, and with display and transport methods.
There are significant opportunities to develop new generation collaborative interfaces that are not limited to the standard Windows Icons Menus and Pointers (WIMP) interfaces of twenty years ago.

12. Near Term Smart Space (cont.)
Experimental Collaborative interface using:
Video camera array to identify, locate and track individuals in the smart space
Video teleconferencing
Initial test case will use several functions:
Broadcast news transcription
Real time spoken query processing
Information retrieval using BN transcripts
While Situation Awareness focuses on a central Smart Space for command, management, and collaborative design, some situations will require mobile elements acting in concert over larger spaces, such as in the FEMA scenario presented in the introduction. Wireless, dynamic wide area networks coupled with fielded sensors will be needed to create wide area smart spaces.
As speech transcription, understanding and task modeling improve, it should be possible for a Smart Space to anticipate the data needs of a working group pre-query and queue information before it is actually needed to reduce latency for working groups.
Objects will be used to encapsulate audiovisual recordings along with annotated transcripts, and with display and transport methods.
There are significant opportunities to develop new generation collaborative interfaces that are not limited to the standard Windows Icons Menus and Pointers (WIMP) interfaces of twenty years ago.

14. Current Status... Lab space allocated; construction begins:
Initial smart space November 23
Acoustically conditioned space 1Q99
Computing, display, sensing infrastructure:
Dual Processor SGI Octane
640Mb Ram
thirty-two channels A/D
Dual camera inputs
1280x1024 large screen projector
Parallel Linux cluster:
112Gb disk array
Seven 400Mhz P-II compute nodes
Gigabit Ethernet switch in room
Microphones:
Desktop
Wall mounted array(s)
Cameras:
Three computer controlled
One digital fixed mount
384Kb video teleconferencing
Wireless connection/Palm Pilot
External cooperative research efforts:
Microphone array/acoustic source locator from Rutgers CAIP; Jim Flanagan P.I.
BBN and IBM offering research and real time speech recognition
Investigating possible use of MIT speaker identification
Investigating possible use of U of Md visual localization and gesture recognition
Internal cooperative efforts:
Information retrieval (PRISE)
Spoken document retrieval with spoken query (SPIDERS)
VRML smart space representation
Information visualization
Face recognition technology

15. Desired Long Term Capabilities/Technologies
Accurate, robust speech recognition for transcription and command
Anticipatory Web and database query based on speech
Speech understanding:
Spoken dialog abstraction
Selective recording of meeting minutes
Data stream segmentation/annotation
Face/expression recognition
Persistence of meeting memory, automatic links to related meetings
Sensor fusion (acoustic/visual/other)
Smart white boards
Image understanding and person recognition
Participant sensing, task identification, and adaptive response
Several new, or substantially improved, capabilities are required. We will need better acoustic signal acquisition which reduces multi-source interference, room reverberation, and background noise. Incremental improvements in speech recognition will be required. Current accuracy rates of about 85 percent have been obtained for the best broadcast news processing systems; accuracy levels not adequate for mission critical spoken interfaces. As accuracy levels increase we will want to support real time processing of recognized speech to anticipate the needs of the Smart Space users as tasks are performed. Mission critical application will require very high accuracy levels. Good robustness in terms of non-voice, and multi-voice rejection is needed. The speech and video channels can be annotated using the recognized speech and gestures, as well as speech understanding and dialog summarization. Archiving of sensor data streams gathered during critical events could allow for more complete postmortem analyses. Automatic content abstracting could be used to generate links into long-term event databases. Sensor fusion will allow correlation among data channels for annotation such as transcripts combining what was said with who said it, or what was written or typed with who did the writing or typing. Smart Whiteboards capturing drawing and writing could be a source of additional documentation of meetings. Image understanding for interfaces including visual gesture recognition , and person tracking for use in fusion with other sensors offers opportunity high performance information processing spaces.

16. Metrics and Metrics Research Needed
End-to-end metrics needed for multi-stage processes, e.g.: Full SDR
Sensing
Recognition
Understanding
Response
Training and test reference data sets needed for:
Component recognition tasks, e.g.: CSR, speaker ID, gesture recognition
Data reduction, e.g.: array processing and source location
Information extraction and summary
Significant research is required in order to construct measurement protocols for mixed initiative systems that allow multiple actions and paths. Experience in the speech recognition community has shown that well drawn measurement programs can be very important to ongoing technical improvements in a new technology. Reference data sets will be needed for the several recognition tasks required in Smart Spaces. Some of these are already under development elsewhere in the DARPA community, but Smart Spaces can provide a test bed for integrated functionality. For example a topic detection and summarization task would use speech recognition, possibly speaker identification, and natural language parsing and discourse analysis. Measurements tasks requiring the use of multiple cascaded and parallel technologies will have to be designed. An initial Smart Space test bed will be required to record and render some of the reference materials. That is: an initial Smart Space will be needed to allow further progress to be made in the art and science of Smart Spaces; development will be iterative. It was noted by the chairman that the military mission planning community has tests and doctrine that could be incorporated into benchmarks and scoring procedures. These offer examples of how to score command team activities in terms of solution time, quality, time to solution, labor costs and other measurements.

17. Metrics and Metrics Research Needed (cont.)
Initial Smart Space is needed as test bed to bootstrap future improvements
Command/Crisis Management may offer measurable, reproducible tasks
Need well defined tasks with solution quality, time, and labor level measurements
Significant research is required in order to construct measurement protocols for mixed initiative systems that allow multiple actions and paths. Experience in the speech recognition community has shown that well drawn measurement programs can be very important to ongoing technical improvements in a new technology. Reference data sets will be needed for the several recognition tasks required in Smart Spaces. Some of these are already under development elsewhere in the DARPA community, but Smart Spaces can provide a test bed for integrated functionality. For example a topic detection and summarization task would use speech recognition, possibly speaker identification, and natural language parsing and discourse analysis. Measurements tasks requiring the use of multiple cascaded and parallel technologies will have to be designed. An initial Smart Space test bed will be required to record and render some of the reference materials. That is: an initial Smart Space will be needed to allow further progress to be made in the art and science of Smart Spaces; development will be iterative. It was noted by the chairman that the military mission planning community has tests and doctrine that could be incorporated into benchmarks and scoring procedures. These offer examples of how to score command team activities in terms of solution time, quality, time to solution, labor costs and other measurements.

18. Interoperability Issues are Severe
Rapid handheld appliance discovery and connection to IP infrastructure, e.g.: IP masquerading or NAT
Dense wireless device deployment: IR RF
Multi-resolution devices and displays
Device protocol discovery, automatic translation among heterogeneous protocols used by multiple devices
Multimedia output into the space and acoustic input
Security and accessibility to and by infrastructure
The group noted that there would be interoperability issues, especially in the areas of wireless IR/RF LANs. Another issue raised was that of devices acting as communication conduits for each other to establish links between incompatible devices. Protocols for network routing will also need to be investigated.