Smart Meeting Systems Josh Reilly

Why are Smart Meeting Systems worth studying?

Objectives of a Smart Meeting System Improves the productivity of a team by automating the: –Capture of the meeting –Processing of the meeting for valuable information –Displaying of that information accurately and effectively to the end user through a client application

Organization of Smart Meeting System Processes A smart meeting system can be decomposed into three sets of processes  Meeting Capture  Meeting Recognition  Semantic Processing

Organization of Smart Meeting System Processes

Meeting Capture Gathering raw inputs from the meeting  Video Capture  Audio Capture  Other Context

Video Capture Video feeds from:  Cameras for the attendees Could use a single static camera Could use a single camera with pan, tilt, zoom (PTZ) capabilities Recommend camera view of every contributor's face  Visual Aids Separate camera Digital feed from device

Microsoft Distributed Meetings Project Camera Placement

Microsoft Distributed Meetings Project Video Capture RingCam Array of 90 º Cameras 360 º Panoramic view

Audio Capture Use an array of microphones  Placed on the table  Placed on the ceiling  Worn on the person Levels need to be controlled so that they are similar levels for each contributor

Microsoft Distributed Meetings Project Audio Capture RingCam  Has an array of microphones on its base.

Other Context Capture RFID to track attendees  Attendees swipe their RFID cards when they enter the meeting to add their ID to the list of people attending this meeting Motion Detectors  to track the locations of attendees within the room

Organization of Smart Meeting System Processes

Meeting Recognition The processing of the raw capture before it is organized into something useful Steps:  Person Identification  Attention Detection  Activity Recognition  Hot Spot Recognition  Summarization

Person Identification is associating sections of video, audio, and the visual aids that were captured from the meeting with the attendee(s) that they belong to Face Recognition Face Tracking Speech Recognition SSL Beamforming

Person Identification Face Recognition Facial Recognition Identify the person speaking from a list of attendees Eigenface Approach Challenges Poor Quality Images Poor Room Lighting Continuously changing facial expressions Occlusion

Face Recognition The Eigenface Approach All faces are assumed to be made up of different percentages of different eigenfaces A set of eigenfaces is a set of very generalized pictures of faces that were generated so that each has a basic ingredient that can be used to make a face Eigenfaces from AT&T Laboratories Cambridge

Person Identification Speech Recognition Speech Recognition Match the voice of the person speaking to someone on the list of attendees Using Voice recognition in conjunction with face recognition allows for an accurate identification of the speaker Sound Source Localization (SSL) Used to determine which camera is pointed at the speaker Could be used to point PTZ camera Beamforming

Person Identification Writer Recognition Writer Recognition When someone writes on the whiteboard, they may not be in clear view of the cameras Writing recognition algorithms can be used to identify who wrote what during a meeting

Attention Detection Attempt to determine who is looking at whom during a meeting. Provides information used for activity recognition and hot spot recognition Done using: Hidden Markov Models (HMM) Sound Source Localization (SSL) Known layout of room

Activity Recognition Determine what is happening during the meeting Step 1: Determine what each individual is doing at each point during the meeting Person Identification, Attention Detection, SSL, Gesture Recognition Step 2: Take that information to determine what activity the entire group is engaging in at each point during the meeting

Hot Spot Recognition Find the important parts of the meeting Using sound queues Ex: Changes in pitch Using activity recognition When people are nodding When their focus changes

Summarization Takes all of the information that the smart meeting system has learned about the meeting and creates a quick overview of the events that took place during that meeting. This information will be used in the semantic processing stage

Organization of Smart Meeting System Processes

Semantic Processing Takes the information from the meeting recognition step and makes it usable by the end user. Meeting Annotation Meeting Indexing Meeting Browsing

Meeting Annotation Describe the raw data from the meeting from each viewpoint Attempt to label all meeting segments Implicitly Automatically Explicitly By Hand

Meeting Annotation Implicit Automated Annotation Assumes that the meeting recognition processes performed with relatively high efficiency Tags every person in the video Narrates what was happening during the meeting Has not been achieved

Meeting Annotation Explicit Annotation By Hand When the recognition processes fail to gather sufficient correct information about the raw data Users will have to go through the meeting and tag the people attending as well as indicate what events are happening all through the meeting

Meeting Indexing Indexing is done at all levels of data from a raw audio feed to the annotations The best form of indexing to use is the event-based indexing An index is created every time an event occurs This is the best way for users to find a specific spot in the meeting when performing a query

Meeting Browsing The interface that the end user uses to retrieve information from the meetings Functions: Can browse/search a list of all meetings for a specific meeting Can browse/search the contents of the chosen meeting Aided by tools like bookmarks, a meeting outline, and queries (content, people, camera angles, visual aids, etc...)

Meeting Browsing Microsoft Distributed Meetings

Remote Attendee Use the smart meeting system as the attendee's eyes and ears Microsoft's PING project Uses a monitor and speaker to display the remote attendee's voice and audio during the meeting However, the remote attendee is often ignored

Carnegie Mellon University’s Meeting System Architecture Lacks Activity Recognition Hot Spot Recognition Annotations

University of California, San Diego AVIARY System Architecture 2 PCs 4 Static Cameras 4 PTZ Cameras No SSL

Ricoh Portable Meeting Recorder

Ricoh Portable Meeting Recorder Doughnut Camera

Ricoh Portable Meeting Recorder Meeting Browser

Technology Limitations Speech recognition and facial recognition algorithms are not yet as efficient as they should be in order for a smart meeting system to perform accurately

Workspace Limitations Cameras and microphones can block view, distract, or intimidate attendees during the meeting Security and Privacy needs to be addressed

References [1] Zhiwen Yu and Yuichi Nakamura Smart meeting systems: A survey of state-of-the-art and open issues. ACM Comput. Surv. 42, 2, Article 8 (March 2010), 20 pages. DOI= / [2]Ross Cutler, Yong Rui, Anoop Gupta, Jj Cadiz, Ivan Tashev, Li-wei He, Alex Colburn, Zhengyou Zhang, Zicheng Liu, Steve Silverberg. (2002). Distributed Meetings. A Meeting Capture and Broadcasting System. 10 pages. [3]Harold Fox The eFacilitator: A Meeting Capture Application and Infrastructure. 89 pages. [4]Yong Rui, Eric Rudolph, Li-wei He, Rico Malvar, Michael Cohen, Ivan Tashev Ping: A Group-To- Individual Distributed meeting System. 4 pages. [5]Dar-Shyang Lee, Berna Erol, Jamey Graham, Jonathan Hull, Norihiko Murata Portable Meeting Recorder. 10 pages.