2. Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht

3. Projects Content-Based Image Retrieval Using Relevance Feedback (Oge Marques) IP Simulcast - An Innovative Video and Audio Broadcasting Technique Over the Internet (Ray Westwater and Jeff Ice) XYZ Video Encoding Technique (Ray Westwater & Joshua Greenberg) An Innovative Motion Estimation Algorithm for MPEG Codec (Joshua Greenberg) A Fast Content-Based Multimedia Retrieval Technique (P. Saksobhavivat) Interactive Progressive Encoding System (Joe Celi)

4. Internet Broadcasting or Webcasting Broadcasting multimedia data (audio and video) over the Internet - from a server (sender) to a large number of clients (receivers) Applications include: radio and television broadcasting real-time broadcasting of critical data distance learning videoconferencing database replication electronic software distribution

5. Broadcast Pyramid Applied in IP Simulcast

6. IP Simulcast - An Innovative Technique for Internet Broadcasting IP Simulcast reduces the server (or sender) overhead by distributing the load to each client (or receiver) Each receiver becomes a repeater, which rebroadcasts its received content to two child receivers The needed network bandwidth for the server is significantly reduced

7. Characteristics of IP Simulcast It is a radically different model of digital broadcast, referred to as repeater-server model The server manages and controls the interconnection of repeaters Each repeater not only plays back the data stream, but also transmits the data to two other repeaters IP Simulcast provides guaranteed delivery of packets, which is not the case with IP Multicast

8. Product: AllCast www.allcast.com

9. Broadcasting Tree Once the AllCast Broadcaster is configured, many users can connect to hear/view content. The bandwidth usage is distributed across the participants, as illustrated by the dynamic, self-healing dissemination tree shown in the AllCast main window.

10. Microsoft Media Player with AllCast Plug-in Users can connect to a broadcast using the Microsoft Windows Media Player together with a small, seamlessly integrated AllCast Plug-in. The plug-in enables the Windows Media Player to participate in peer-to-peer broadcasts.

11. XYZ - New Video Compression Technique The XYZ video compression algorithm is based on 3D Discrete Cosine Transform (DCT) It provides very high compression ratios and excellent video quality It is very suitable for real-time video compression

12. Forming Video Cube for XYZ Compression

13. Block Diagram of the XYZ Compression

14. Key Encoding Equations Both encoder and decoder are symmetrical, which makes the algorithm suitable for VLSI implementation

15. XYZ Versus MPEG

16. Complexity of Video Compression Techniques

17. XYZ Versus MPEG Original MPEG Cr=11,NRMSE=0.08 MPEG, Motion Est. only Cr=27, NRMSE=0.14 XYZ Cr=45, NRMSE=0.079

18. Examples of XYZ Compression Original XYZ-compressed Cr=51

19. Examples of XYZ Compression Original XYZ-compressed Cr=110

20. Sensitivity of the XYZ Algorithm to Various Video Effects

21. Characteristics of XYZ Video Compression XYZ gives significantly better compression ratios than MPEG for the same quality of video For similar compression ratios, XYZ gives much better quality than MPEG XYZ is faster than MPEG (lower complexity) XYZ is simple for implementation

22. Applications of the XYZ Interactive TV and TV set-top boxes TV phone Video broadcasting on the Internet Video-on-demand applications Videoconferencing Wireless video

23. TV Phone Videophone is a box on the top of TV with a small camera, modem, and video/audio codec. Videophone is a box on the top of TV with a small camera, modem, and video/audio codec.

24. Design of the TV Phone

25. A Fast Content-Based Multimedia Retrieval Technique Two main approaches in indexing and retrieval of images and videos Keyword-based indexing and retrieval Content-based indexing and retrieval

26. Keyword-Based Retrieval and Indexing Uses keywords or descriptive text, which is stored together with images and videos in the database Retrieval is performed by matching the query, given in the form of keywords, with the stored keywords This approach is not satisfactory - the text-based description is incomplete, imprecise, and inconsistent in specifying visual information

27. New Algorithm for Similarity- Based Retrieval of Images Images in the database are stored as JPEG-compressed images The user submits a request for search-by- similarity by presenting the desired image. The algorithm calculates the DC coefficients of this image and creates the histogram of DC coefficients. The algorithm compares the DC histogram of the submitted image with the DC histograms of the stored images.

28. Histogram of DC Coefficients for the Image “Elephant”

29. Comparison of Histograms of DC Coefficients

30. Example of Similarity-Based Retrieval Using the DC Histograms

31. Similarity-Based Retrieval of Compressed Video Partitioning video into clips - video segmentation Key frame extraction Indexing and retrieval of key frames

32. DC Histogram Technique Applied for Video Partitioning

33. Example of Similarity-Based Retrieval of Key Frames Using DC Histograms

34. Interactive Progressive Encoding System Users submit requests for imagery to the image database via a graphical user interface Upon an initial request, a DCT image (version of the image based on DC coefficients only) is transmitted and reconstructed at the user site. The user can then isolate specific regions of interests within the image and request additional levels of details.

35. Band Transmission in Interactive JPEG System Based on Spectral Selection

36. Prototype System - IPES and Experimental Results Original image “Airport”

37. Interactive Progressive Transmission in Four Scans

38. Selection of Two Regions

39. Cumulative Number of Transmitted Bits

40. Extracted Images From a Group of Images

41. Applications Retrieval and transmission of complex images over low bandwidth communication channels (image transmission over the Internet, real-time transmission of medical images) Archiving and browsing visually lossless image databases (medical imaginary, space exploration and military applications)

42. Content-Based Retrieval Large, complex, and ever growing, distributed, mostly unstructured, multimedia repositories Three ways of retrieving multimedia information: –Free browsing (inefficient, time-consuming, doesn’t scale well) –Text-based retrieval (relies on metadata, time- consuming, subjective) –Content-based retrieval (requires intelligent interpretation of the contents)

43. Design of MUSE System Image Analysis Image Feature Extraction -Color -Shape -Texture Image Representation & Feature Organization Image Archive User GUI -Image selection -Result viewing Probability recalculation & candidate ranking Feature Extraction Similarity comparison Interactive learning & Display update Off-lineOnline

44. Query By Example Example image Best result Similarity Score [0,1]

45. Relevance Feedback Good Bad Neither

46. Relevance Feedback - Next

47. Technology Behind the MUSE System Feature extraction Extraction of relevant image features impacts the overall performance of the system. MUSE uses: –color-related features (color histograms, color space partitioning and/or quantization, color moments, color coherence vectors) –texture-related features (Multiresolution Simultaneous Autoregressive Model - MSAR) –frequency-related features (DFT, DCT)

48. Technology Behind the MUSE System Bayesian formulation MUSE is based on a Bayesian framework for relevance feedback. During each iteration of a MUSE session, the system displays a subset of images from its database, and the user takes an action in response, which the system observes. Based on the user’s actions, the probability distribution over possible targets is refined. (Most systems refine the user’s query) The best candidates are then displayed back.