1. Department of Computer Science and Engineering, CUHK 1 Final Year Project 2003/2004 LYU0302 PVCAIS – Personal VideoConference Archives Indexing System Supervisor: Prof Michael R. Lyu Presented by: Lewis Ng, Philip Chan 20 April 2004

2. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 2 Outline Introduction Motivation Architecture of PVCAIS - Media Acquisition Module - Archive Indexing Module - Videoconference Accessing Module Implementation Conclusion

3. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 3 Introduction PVCAIS stands for Personal VideoConference Archives Indexing System A system that provides convenient searching and browsing support for videoconferencing users on past videoconference archives

4. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 4 Introduction What is videoconference? A real-time communication technology which combines different media: audio, video, text chat, file transfer, whiteboard and shared applications - More precisely is “multimedia conference” - Standard of video conferencing : ITU-T H.323

5. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 5 Motivation – Videoconferencing is becoming popular in education, business and personal communication – Participants wish to keep videoconference archives for later references – Normal video and audio files are neither searchable nor helpful to recall their contents – Indexing of videoconference archives has not been investigated till now

6. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 6 Architecture of PVCAIS Consists of 3 modules: - Media Acquisition Module - Archive Indexing Module - Videoconference Accessing Module

7. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 7 Architecture of PVCAIS

8. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 8 Extracts channel data and forms media files Videoconferencing physically contains 4 types of channels: Audio, Video, Data and Control Audio and Video channels: transmit incoming/ outgoing audio and video information Data channel: carries information for user application such as Text Chat, Whiteboard and File Transfer Control channel: transmits system control information such as Member Information Architecture Media Acquisition Module

9. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 9 Architecture Media Acquisition Module Video-in and Video-out channel – Reduce redundancy : just store key-frames – Detect scene change in real time – Each key frame picture is stored with a timestamp

10. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 10 Architecture Media Acquisition Module Audio-in and Audio-out channel – Mixed into one stream after videoconference – Will be used for Speech Recognition

11. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 11 Architecture Media Acquisition Module Text Chat channel – sender / receiver – message – store with timestamp

12. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 12 Architecture Media Acquisition Module Whiteboard channel – Consists of a text-based index file and a number of snapshot pictures – Index file records timestamp for each whiteboard update event and the path of the corresponding snapshot picture – Update of this channel happens in a period of time -> need to detect when update begins and ends by monitoring data transfer in this channel

13. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 13 Architecture Media Acquisition Module File Transfer channel – Make a copy of the sent/received files to the directory of archive – Index file includes sender’s / recipient’s user names and the path of the files

14. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 14 Architecture Media Acquisition Module Control channel – Contains timestamp and information of each event such as member joined and member left

15. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 15 Architecture Media Acquisition Module Paradigm of storing the videoconference archives. Video_in Video_out Audio_in Audio_out Text_chat Whiteboard File_in File_ out Control Time 0:00:00 One line Two lines One line Two lines Three lines One line Two lines Three lines Four lines One line Two lines Three lines Four lines Five lines i iii Video_in archive Audio archive Text chat archive Whiteboard archive Document archive Control archive Video_out archive

16. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 16 Architecture Archive Indexing Module Raw files are extracted in Media Acquisition Module Need to implement some multimedia indexing functions to retrieve more information These includes: Face Detection, Face Recognition, Speech Recognition, Time-based Text Merging

17. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 17 Architecture Archive Indexing Module Face Detection - If face is detected, find out the face region Face

18. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 18 Architecture Archive Indexing Module Face Recognition - Associate human faces in Video-in with name - Need to keep a face base - If no match in the face base, ask remote user to enter the name

19. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 19 Architecture Archive Indexing Module Speech Recognition - Generate speech script from audio archive - Speech of a videoconference contains important information - Can use commercial libraries: Microsoft SAPI, IBM Via Voice Time-based Text Merging - Merge the Speech script, Chat messages and Whiteboard text into the Text Source according to their timestamps

20. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 20 Architecture Videoconference Accessing Module Provides an interface for user to manage, search and review all indexed conference archives. Allows user to search for a conference by different criteria, such as meeting date, member name and title. Allows user to review a conference by playing back different media in a synchronized way.

21. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 21 Implementation NetMeeting 3.0 – A Windows feature that provides Internet conferencing function. – Supports video, audio and data conferencing including application sharing, chat, whiteboard and file transfer. NetMeeting 3.0 SDK – An extension of NetMeeting, provides an interface for programmers and Web developers to integrate conferencing capabilities into their applications. – API is in the form of COM interfaces and functions..

22. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 22 Implementation Videoconferencing Client A videoconferencing program built on top of the NetMeeting 3.0 SDK. Support: – Video Streaming – Audio Streaming – Text Chat – File Transfer – Whiteboard

24. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 24 Implementation Videoconferencing Client Face Verification Login Feature – Face Training: 8 different face images of each user are needed for training and their face images are saved in face base – Face Verification: EigenFace algorithm is implemented to check the face of a user against his/her user ID. If the verification is successful, user can then join a videoconference

26. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 26 Implementation Media Acquisition Module By directly using the functions of the API, the following raw data can be obtained: – member information – file transfer record – text message record Video, audio and whiteboard data cannot be directly obtained.

27. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 27 Implementation Media Acquisition Module Video – create a thread to check the display of the video windows – if scene change is detected, the video will be captured and stored as a still image. – the stored images are key frames of the conference.

28. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 28 Implementation Media Acquisition Module Audio – create a thread to record the local audio from the microphone. – members of the conference will continuously exchange the audio data. – all the received audio files and locally recorded audio files will be combined to generate a single audio file.

29. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 29 Whiteboard – cannot capture the NetMeeting whiteboard information because the format of the data is not stated in the API. – solution: create our own whiteboard function and data format. Implementation Media Acquisition Module

31. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 31 Implementation Media Acquisition Module - The whiteboard update flow:

32. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 32 Implementation Archive Indexing Module The stored key-frames will be used for face detection and recognition after the conference. The final audio file will be used for voice recognition, the voice engine used is Microsoft SAPI.

33. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 33 Implementation Videoconference Accessing Module Consists of: - Searching Interface: search conference by title, date, participants, text, file transferred, whiteboard content. - Playback Interface: review conference by playing back the content of the conference by using SMIL

35. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 35 Implementation Videoconference Accessing Module SMIL – stands for Synchronized Multimedia Integration Language – HTML-like language – can integrate streaming audio and video with images, text, or any other media type into one presentation

36. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 36 Implementation Videoconference Accessing Module SMIL document generation process :

38. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 38 Conclusion We have developed the PVCAIS which supports: – Videoconferencing functions – Acquisition of all videoconference content – Archive indexing – Searching and synchronized playback of videoconference archives

39. LYU0302 : PVCAIS – Personal VideoConference Archives Indexing System 39 Q & A Session