1. Dec 9-11, 2003 ICADL 2003 1 A Case Study of a Stream-Based Digital Library: Medical Data Mohamed Kholief Kurt Maly Stewart Shen

2. 2Dec 9-11, 2003ICADL 2003 Overview IntroductionApproachArchitecture Implementation of the case study Conclusions and future work

3. 3Dec 9-11, 2003ICADL 2003 Introduction Data streams are important sources of information for applications such as: News-on-demand News-on-demand Weather services Weather services Scientific research Scientific research … A data stream is a sequence of data units produced over a period of time Examples: video, audio, sensor readings, …

4. 4Dec 9-11, 2003ICADL 2003 Introduction Saving data streams in digital libraries is advantageous: Archival Archival Preservation Preservation Administration Administration Access control Access control …

5. 5Dec 9-11, 2003ICADL 2003 Introduction Most common method of retrieval from digital libraries is by using bibliographic information Content based retrieval is a hot research area from multimedia digital libraries We introduce event-based retrieval An event is a noteworthy occurrence that occurred during the stream There has been psychological evidence that events are easier to remember than specific time instances at which they occurred. Searching data streams in a digital library by events gives users an extra flexibility and allows for faster and more precise retrieval

6. 6Dec 9-11, 2003ICADL 2003 Approach Considering some potential applications: digital libraries for: stock market, news streams, census bureau statistics, weather, sports games, and the educational environment digital libraries for: stock market, news streams, census bureau statistics, weather, sports games, and the educational environment Forming categories of possible users and their basic requirements Identifying a list of design considerations Implementing a medical digital library to illustrate and validate our approach

7. 7Dec 9-11, 2003ICADL 2003 Digital Library Architecture Data streams organization Events organization Retrieval architecture All to be presented in the context of the case study.

8. 8Dec 9-11, 2003ICADL 2003 Case Study: A Medical Digital Library Actual CT scan streams, sample text and audio streams Domain expert – radiologist: Provided the CT scans Provided the CT scans Generated sample events Generated sample events Specified the specific metadata fields Specified the specific metadata fields Described the mapping between the raw metadata format and the bibliographic metadata Described the mapping between the raw metadata format and the bibliographic metadata

9. 9Dec 9-11, 2003ICADL 2003 Structure of the Stream Object Stream Object Folder Data FolderMetadata Folder A Stream Data File Temporal Metadata Bib Metadata Specific Metadata

10. 10Dec 9-11, 2003ICADL 2003 Stream Metadata Standard Bib: ID, ENTRY, TITLE, TYPE, CREATOR, DESCRIPTION, FORMAT, SOURCE, START_DATE, END_DATE Type Specific Bib: patient name, birth date, age, sex, history, slice thickness, contrast, and reason Temporal: timing information of stream frames

11. 11Dec 9-11, 2003ICADL 2003 Structure of the Digital Library Repository Archive 1Archive 2Archive n Digital Library Repository Stream Objects

12. 12Dec 9-11, 2003ICADL 2003 The ER Diagram of the Database

13. 13Dec 9-11, 2003ICADL 2003 Publishing Mass publishing using PERL scripts Individual publishing manually In a future extension: a publishing tool Publishing Tool Indexing Tool Stream Templates Pool Streams Repository Publisher Stream Type Stream Template Stream Object Stream Data and metadata

14. 14Dec 9-11, 2003ICADL 2003 Events The following events were used: Calcification Calcification Tumor started to appear Tumor started to appear Maximum diameter of the tumor Maximum diameter of the tumor Necrosis appearing Necrosis appearing Maximum necrosis diameter Maximum necrosis diameter Event information and time instances were manually inserted in the database

15. 15Dec 9-11, 2003ICADL 2003 Bib. Metadata Time Instances List Related Streams List Event Criteria Event Display Method Structure of the Event Object

16. 16Dec 9-11, 2003ICADL 2003 Structure of the Event-Class Object Bibliographic Metadata (e.g. id, type, description, terms and conditions) Event-class display method List of events of that type

17. 17Dec 9-11, 2003ICADL 2003 The Event Generation Process Related Stream Objects Event Template Specifications: Bib metadata Event criteria Related streams Domain Experts Event Generation Tool Criteria Modul e Display Module Bibliographic Fields Related Streams List Digital library administrators Event Object Bibliographic metadata Related streams list Time instances list Criteria Module Display Module Event Templates Repository Event Editing Tool

18. 18Dec 9-11, 2003ICADL 2003 Related Streams Represented in database tables Generated in two ways: Automated Automated PERL scripts used to relate streams using simple criteria (e.g. same patient) Manual Manual Insert directly into database

19. 19Dec 9-11, 2003ICADL 2003 Retrieval Architecture Search Interfac e Search Results Object’s Display Interface Contro l Applet Player Applet s Metadata Objects Repository Web Client Web ServerStorage Search Script Displa y Script Contro l Script Streaming Server Resource Discovery user Database Server File System

20. 20Dec 9-11, 2003ICADL 2003 The User Interface Search: event-based, simple, browse Search results Stream display Playback interface Demo

21. 21Dec 9-11, 2003ICADL 2003 The Simple Search Form

22. 22Dec 9-11, 2003ICADL 2003 The Event Based Search Form

23. 23Dec 9-11, 2003ICADL 2003 The Events Menu

24. 24Dec 9-11, 2003ICADL 2003 Search Results

25. 25Dec 9-11, 2003ICADL 2003 Browsing Results

26. 26Dec 9-11, 2003ICADL 2003 Stream Display Interface

27. 27Dec 9-11, 2003ICADL 2003 Sample Playback Interface

28. 28Dec 9-11, 2003ICADL 2003 Playing Synchronous Streams

29. 29Dec 9-11, 2003ICADL 2003 Conclusions Introduced a new approach of retrieval of data streams from digital libraries that is natural and powerful Showed the feasibility of using events for retrieval from digital libraries Showed the feasibility of using events for retrieval from digital libraries Provided a new paradigm for medical digital libraries Provided a new paradigm for medical digital libraries Accessing medical streams more efficiently Key to the specification of events and what metadata to use were the domain experts (radiologists) The library prototype can be adapted to easily support more stream types The library prototype can be adapted to easily support more stream types The system is efficient and scales to support a production DL The system is efficient and scales to support a production DL

30. 30Dec 9-11, 2003ICADL 2003 Future Work Event generation tool Automatic event generation Dynamic event generation Advanced search interface OAI compliance Manipulating a greater number of events Event-related streams Further automating the process of relating streams Publishing tool More applications

31. 31Dec 9-11, 2003ICADL 2003

32. 32Dec 9-11, 2003ICADL 2003 Introduction Digital libraries are storehouses of information available through the Internet that provide ways to collect, store, and organize data and make it accessible for search, retrieval, and processing Digital library collections are not limited to document resources: they extend to digital artifacts that cannot be represented or distributed in printed formats. A digital library may contain text, images, sound and video in an integrated repository

33. 33Dec 9-11, 2003ICADL 2003 Motivation First, some realizations from the (old) IRI system. Many streams are used : Many streams are used : video, audio, button clicks, slides, … Playback of recorded sessions required much knowledge Playback of recorded sessions required much knowledge session title, date, and time. use of a slider for random access. Using events seemed more intuitive. Playback could only be done through IRI. Playback could only be done through IRI. Using the web seemed more flexible. Only students could access the recordings. Only students could access the recordings. Using the web would increase the audience. Using DL’s would still provide access control. Then, generalization to any stream and for other applications such as weather services.

34. 34Dec 9-11, 2003ICADL 2003 Objective To study the issues involved in building a digital library that: contains data streams contains data streams allows event-based retrieval allows event-based retrieval allows other traditional retrieval approaches. allows other traditional retrieval approaches.

35. 35Dec 9-11, 2003ICADL 2003 Objective Details Building a streams-based digital library Supporting event-based retrieval Developing a prototype for the digital library Developing a digital library application of the proposed prototype

36. 36Dec 9-11, 2003ICADL 2003 Issues of Building a Streams- Based Digital Library Data organization Required metadata Services provided to users Other retrieval approaches

37. 37Dec 9-11, 2003ICADL 2003 Issues of Supporting Event-Based Retrieval Creating events and inserting them The physical representation Event metadata Using events for retrieval Representing events that depend on many streams Using events to synchronize the playback of multiple streams

38. 38Dec 9-11, 2003ICADL 2003 Issues of Developing a Prototype Process of deciding on important streams Process of determining events to be considered Process of applying the prototype for an application Deciding on software components to be included in the prototype Supporting extensibility in the prototype

39. 39Dec 9-11, 2003ICADL 2003 Developing an Application A repository A few hundred data streams of text, images, and audio formats A few hundred data streams of text, images, and audio formats Search interfaces employ event-based retrieval employ event-based retrieval retrieval based on bibliographic information retrieval based on bibliographic information A player application for each supported stream type and the corresponding streaming servers A control module A process for extending the digital library to support any new stream type or format

40. 40Dec 9-11, 2003ICADL 2003 System Model A frame: f = (t,d); or f(t)=d A stream: s = {f(t 0 ), f(t 1 ), f(t 2 )…} An event: E = (name, T), where T = {t i, t j …} Criteria of an event E: C E T = {t i | C E is not satisfied for f(t i-1 ) and is satisfied for f(t i )

41. 41Dec 9-11, 2003ICADL 2003 Generalization All streams in the library: S = {s 1, s 2, s 3 …} Related streams to E: S E  S New event: E = (name, T, S E ) New criteria: C E (S E ) F SE (t) is the set of frames from all the related streams that happen at time t

42. 42Dec 9-11, 2003ICADL 2003 Generalization If f(t) exists, then f(t m ) = f(t), else if f(t  t) exists (  t is a tolerance) then f(t m ) = f(t  t) else f(t m ) = NULL. T = {t | C E (S E ) is not satisfied for F SE (t m-1 ) and satisfied for F SE (t m )}

43. 43Dec 9-11, 2003ICADL 2003 Applications An educational digital library A weather-related digital library A stock market digital library A digital library containing news-streams A digital library containing census bureau statistics A digital library containing sports games multimedia streams A medical digital library

44. 44Dec 9-11, 2003ICADL 2003 An Educational Digital Library Example: IRI Streams: teacher audio and video teacher audio and video students audio and video students audio and video shared room view shared room view button clicks button clicksEvents: teacher started tool teacher started tool students joined session students joined session somebody asked a question, etc somebody asked a question, etc Potential users: teachers, students, general public, education researchers teachers, students, general public, education researchers

45. 45Dec 9-11, 2003ICADL 2003 An Educational Digital Library Query: When the teacher started Netscape, what did he say? Query: Return all sessions in which Dr. Smith is the teacher Query: When any student started speaking, what was the running application, who was the teacher, what is that student’s name, etc? Query: Return all streams in which Netscape was executing and the teacher opened an XTERM

46. 46Dec 9-11, 2003ICADL 2003 A Weather-Related Digital Library Streams: temperatures, wind, humidity, radar maps, and satellite images temperatures, wind, humidity, radar maps, and satellite imagesEvents: wind, temp., rain, etc exceeding certain limit, wind, temp., rain, etc exceeding certain limit, hurricane upgrade, downgrade, etc. hurricane upgrade, downgrade, etc. Potential users: Meteorologists, public, weather researchers, … Meteorologists, public, weather researchers, …Queries: Display the wind stream starting from the time at which a hurricane upgrade was reported over this area Display the wind stream starting from the time at which a hurricane upgrade was reported over this area Display the radar maps and satellite images during a snowstorm Display the radar maps and satellite images during a snowstorm Display the TV weather reports that relate to a specific event Display the TV weather reports that relate to a specific event

47. 47Dec 9-11, 2003ICADL 2003 Design Considerations Multiple modes for retrieval Event instances an event usually occurs more than once during a data stream Some events occur only once (esp. media events) There maybe a very large number of events to browse through Atomic and composite events Composite events have durations Issues with event generation Who? How? Manual or Automated? Tools? Domain or field experts are required to define and specify event instances

48. 48Dec 9-11, 2003ICADL 2003 Design Considerations Accommodation for related streams streams the user might want to retrieve whenever she retrieves a stream Stream-related e.g.: audio and video streams of a class session Event-related e.g.: audio stream of a description of that event Instance-related e.g.: the CNN broadcast when an upgrade occurs in a specific hurricane Concurrent streams can be synchronized while they are played back, e.g. the video and audio streams of the same class session can be synchronized while they are played back, e.g. the video and audio streams of the same class sessionNon-concurrent, e.g.: news broadcast about a hurricane upgrade e.g.: news broadcast about a hurricane upgrade Domain experts to determine the relationship criteria

49. 49Dec 9-11, 2003ICADL 2003 Design Considerations Multiple stream formats Multiple stream types A set of streams that share the same characteristics are of the same stream type: metadata, players, data access methods A set of streams that share the same characteristics are of the same stream type: metadata, players, data access methods Type is not format (Teacher audio is not the same type as TV audio) Type is not format (Teacher audio is not the same type as TV audio) Different bibliographic metadata fields for different stream types “Specific metadata” and “Standard metadata” “Specific metadata” and “Standard metadata” Certain software modules needed for every stream type To display stream information, to play back, and to access data To display stream information, to play back, and to access data Various roles for DL users domain experts, publishers, resource discovery users domain experts, publishers, resource discovery users

50. 50Dec 9-11, 2003ICADL 2003 Type Extension Add new record in the stream types table in the database Implement the Player Applet for streams of this new type: It has to implement a pre-defined JAVA interface It has to implement a pre-defined JAVA interface An existing player could be used as a template An existing player could be used as a template

51. 51Dec 9-11, 2003ICADL 2003 Objective To study the issues involved in building a digital library that: Contains data streams Contains data streams Allows event-based retrieval Allows event-based retrieval Allows other traditional retrieval approaches Allows other traditional retrieval approachesComponents: Building a streams-based digital library Building a streams-based digital library Supporting event-based retrieval Supporting event-based retrieval Developing a prototype for the digital library Developing a prototype for the digital library Developing a digital library application of the proposed prototype Developing a digital library application of the proposed prototype