1. Event-Based Fusion of Distributed Multimedia Data Sources Vincent Oria Department of Computer Science New Jersey Institute of Technology Newark, NJ 07102

2. 2 Outline Classical Data Integration Problem Multimedia Data An Architectural approach to Multimedia Data Integration Event-Based Integration of Data Sources Conclusion

3. 3 Classical Data Integration* * Borrowed from M. Lenzerini

4. 4 Classical Data Integration Issues How to construct the global schema? (Automatic) source wrapping How to discover mappings between the sources and the global schema? Limitations in the mechanisms for accessing the sources Data extraction, cleaning and reconciliation How to process updates expressed on the global schema, and updates expressed on the sources? The modeling problem: How to model the mappings between the sources and the global schema? The querying problem: How to answer queries expressed on the global schema? Query optimization

5. 5 Multimedia Data Multimedia data management is more than physical server design Logical data modeling is important Multimedia data management is more than similarity search “Show me all the images that are similar to this one [in terms of color, texture, shape].” Querying is much more complicated Give me all the news items on Baghdad over the last 2 weeks

6. 6 Multimedia Data … Multimedia data is heterogeneous in both format and in access primitives and this has to be accommodated You cannot store all the data in a single DBMS; the system has to be open Query-based access to multimedia data is important as well as browsing and some transactional access Some DBMS-like interface and control over multimedia data should be provided

7. 7 Multimedia Data … Multimedia data management is not data model independent The complexity of the primitive data types and the required extensibility necessitate certain functionality It does not make sense to completely ignore standardization or to be slave to them Follow, and perhaps extend, standards (e.g., XML, MPEG, …)

8. 8 Multimedia Database Processing Multimedia Data Preprocessing System Database Processing MM Data Pre- processor Additional Information..... Multimedia DBMS Users Query Interface MM Data Instance..... Recognized components MM Data Instance MM Data Meta-Data

9. 9 Document Database Architecture Document Processing System Database Processing DTD or XML Schema files Schema Parser..... DTD/ XML Manager Type Generator Document DBMS Users Query Interface DTD/ XML Schema Document content Documents XML or SGML Document Instance Parse Tree.......... DTD/ XML Schema Types Objects Document Parser Instance Generator

10. 10 Image Database Architecture Image Processing System Database Processing Image Analysis and Pattern Recognition Image Annotation Image DBMS Users Query Interface Image Content Description Image Syntactic Objects Semantic Objects..... Meta-Data

11. 11 Video Database Architecture Video Processing System Database Processing Video Analysis and Pattern Recognition Video Annotation Video DBMS Users Query Interface Video Key Frames Meta-Data..... Video Content Description Video

12. 12 Multimedia Data Integration: An Architectural Perspective Simple Client-Server Integrated Server Database Server Middleware and Mediation

13. 13 Simple Client-Server Heavy-duty client  Synchronization, user interface, QoS, … Client has to access each server Scalability problems  client code has to be updated when new servers come on-line Meta- data Database Server Database Server Text Server Text Server Text Image Server Image Server Images CM Server CM Server Video/ Audio Client

14. 14 “Integrated” Server Heavy-duty server DBMS should be able to handle multiple storage systems Real-time constraints on CM Meta- data Video/ Audio Images Text Image Server CM Server Object Storage Server DBMS Functions Client

15. 15 Database Server Lighter client Client has to access only one server Scalability problems  server may become a bottleneck - distribute and interoperate Meta- data CM Server CM Server Video/ Audio Image Server Image Server Images Text Server Text Server Database Server Database Server Client

16. 16 Document Server Document-centric view  Multimedia objects are parts of documents Might be suitable for, e.g., e-commerce catalogs Video/ Audio Images Text Structured Document DBMS Image DBMS CM DBMS Client

17. 17 Interoperable System Middleware Wrapper Text DBMS Text DBMS Video DBMS Image DBMS Image DBMS Mediator Client

18. 18 Event-Based Multimedia Data Integration An event aims at modeling any happening  Facts, context An event has 3 components  Time  Space (location)  Objects

19. 19 Events: Temporal Dimension Time Line and Temporal relationships Event1 ImageVideo Text Event2 ImageVideo Event3 ImageVideo Image Time Line

20. 20 Events: Spatial Dimension GIS (Location and Spatial Relationships) Event1 Event2 Event3 Directional and Topological relationships

21. 21 Events: Object Dimension Which real world objects are involved in the event?  Object Recognition  Classical Data Integration

22. 22 Event: Spatio-Temporal Dimension Moving Objects and their Trajectories  Raw representation: The trajectory T of a moving object is defined as a sequence of vectors T=[t 1, …, t n ] Each r i show the successive positions of the moving object over a period of time.  Movement sequence: The trajectory of a moving object is represented by a sequence of (movement direction, distance ratio) pairs. This representation is not affected by rotation, shifting or scaling. M=[m 1, …, m n-1 ] Each m i is a pair of (movement direction, distance ratio).

23. 23 Event Model Events model interpretation context Example: KIMCOE 2006 is an event Participants are objects Location: Hilton Garden Inn, Suffolk, Virginia Date/Time: October 24 - 27, 2006 Has sub-events like sessions or visit of Lockheed Martin's Center For Innovation Event Properties Discrete or continuous Local or distributed Simple or composite Descriptors: Data (classical and multimedia)

24. 24 Event Querying Objects: RDBM, XML Time Space: GIS

25. 25 Event Querying Objects: RDBM, XML Time Space: GIS

26. 26 Event Querying Objects: RDBM, XML Time Space: GIS

27. 27 Event Operators Temporal Operator Spatial Operators Spatio-Temporal Operator Aggregation

28. 28 Aggregation and Concept Hierarchy Dimensions are hierarchical by nature: total orders or partial orders Example: Location(continent  country  province  city) Time(year  quarter  (month,week)  day) Industry Country Year Category Region Quarter Product City Month Week Office Day

29. 29 Aggregation and Concept Hierarchy: Operators  roll-up (increase the level of abstraction)  drill-down (decrease the level of abstraction)  slice and dice (selection and projection)  pivot (re-orient the multi-dimensional view)  drill-through (links to the raw data)

30. 30 Aggregation and Concept Hierarchy: Roll-up Use of aggregation to summarize at different levels of a dimension hierarchy  Ex: if we are given total sales per city we can aggregate on the market to obtain sales per state Dayton Q1 Q2Q4 Drama Horror Sci. Fi.. Comedy Time (Quarters) Market (city, state) Q3 Newark S. Orange N. York Category Roll-up on Market Ohio Q1 Q2Q4 Drama Horror Sci. Fi.. Comedy Time (Quarters) Market (States,, USA) Category Q3 New Jersey New York Arizona

31. 31 Aggregation and Concept Hierarchy: Drill-down Inverse of roll-up  Given a total sales by state, we can ask for more detailed presentation by drilling down on market Dayton Q1 Q2Q4 Drama Horror Sci. Fi.. Comedy Market (city, state) Q3 Newark S. Orange N. York Category Drill-down on Market Ohio Q1 Q2Q4 Drama Horror Sci. Fi.. Comedy Time (Quarters) Market (States,, USA) Category Q3 New Jersey New York Arizona

32. 32 Aggregation and Concept Hierarchy: Dice and Slice January Slice on January Newark Electronics January Dice on Electronics and Newark

33. 33 Conclusion Event model: A data Integration model This is a work in progress: We need to fully define the event model We want to build on existing Technology (RDBMS, XML, GIS,..)