Information Integration Across Heterogeneous Sources: Where Do We Stand and How to Proceed? Aditya Telang Sharma Chakravarthy, Yan Huang

Motivation “Retrieve castles near London that are reachable by train in less than 2 hours” “Find 3-bedroom houses in Houston within 2 miles of a school and within 5 miles of a highway and priced under 250,000$” “Retrieve French restaurants within 1 mile of IMAX Theater in Dallas, Texas” …

Motivation Search engines Meta-search engines Faceted search engines Domain-specific portals

Current Scenario “Retrieve castles near London that are reachable by train in less than 2 hours” London Train schedules Trains from London Castles Near London - Decision Making Process - Manually Combine Results to arrive at a decision - Decision Making Process - Manually Combine Results to arrive at a decision

Ideal Scenario Information Integration System Intent: Retrieve castles near London that are reachable by train in less than 2 hours Actual Results for the intent

Focus of the Paper Identify the salient challenges needed to be encountered to address this problem Survey existing work to identify the challenges for which acceptable solutions are available Propose a framework that could provide potential solutions towards the problem

Broader Challenges Intent specification and formulation Query processing and optimization Discover of sources, their schemas and characteristics Data Extraction, Integration and Ranking Result Visualization Issues with inconsistency, security, privacy, …

Intent Specification “Retrieve castles near London that are reachable by train in less than 2 hours” – Keyword-based (e.g., search engine query)? – Structured (e.g., SQL) ? – Unstructured (e.g., natural language) ? – Template/Form/Menu-based (deep Web query) ?

Query Processing The number of sources to be integrated are much larger than in a normal database environment. Heterogeneous sources (RDBMS, websites, web services, etc.) do not provide the same processing capabilities found in a typical database system (such as the ability to perform joins). Unlike relational databases, there might be restrictions on how a source can be accessed.

Query Processing In contrast to query optimization in DBMS, the query optimizer in information integration has little information about the data since it resides in remote autonomous sources Web data sources are not necessarily database systems and may have different processing capabilities. Hence, the query optimizer must consider the possibility of exploiting a data source’s query-processing capabilities.

Discovery Source discovery – Given the domain of travel, determine all possible source providing airfare information – Not a simple crawling process since categorization is necessary after crawling [Gal:VLDB’06] – Use of search engines ? web directories ?

Discovery Discovery of source schema and characteristics – Understanding source schema – Understanding query mechanism (for deep Web sources) – Understanding characteristics of sources

Data Extraction How to extract data for individual sub- queries? – APIs, Web services for deep Web? – Data extractors (e.g., Lixto, Florid) for surface Web? Temporary storage of extracted data (becomes a critical issue when data can be large in size such as spatial data)

Data Integration Schema integration a complex challenge across domains [Gal:VLDB’06] Additional challenges while integrating data – Inefficient execution of recursive integration plans – No support to dynamic service composition – Lack of operators to support GeoSpatial data types – No support for record linkage and object consolidation in the mediator can incorporate the source into a new or existing workflow

Ranking In context of integration, ranking has not been addressed as a significant challenge [Telang:ICDE’07] When to rank? – Before integrating sub-query results? – After integrating sub-query results? Source-independent ranking possible?

Other Challenges Visualization of results Handling inconsistencies Ensuring no breach of privacy and security ….

The Current Big Players Industry-level – Google (Google Base) [Madhavan:CIDR’07] – IBM (Web Sphere) – Yahoo (Trip Planner) Academia-level – Havasu [Kambhampatti:ICDE’05] – MetaQuerier [Chang et. al: VLDB’05, CIDR’07] – Ariadne [Knoblock: VLDB’02,03] – …

The InfoMosaic Approach

Knowledge-Base Identify different types of information needed for the domains and sources to answer a query. Domain Knowledge – – Necessary information/knowledge required for elaborating and refining the query based on the domains and keywords provided by the user Source Semantics – – Information store for modeling and maintaining all the necessary information for each source within a given domain Knowledge Base Domain Knowledge Source Semantics MetadataOntology VocabularyOperators Attributes Statistics Schemas

User Intent Specification Specify intent that is more precise than a “search” but less rigid than a “SQL-query” Ability to resolve concepts and their attributes elegantly with minimal user interaction Effectiveness depends on user feedback and past query statistics [Telang:COMAD’08] Feedback-centric Query Specification User Intent Feedback Knowledge-Base Refined Query

Multi-level Query Planning Evaluation is made at each stage to prune plans using relevant cost metrics. Some of the additional cost metrics – – volume of data retrieved from each source – number of calls made to and amount of data sent by each source – quantity of data processed – the number of integration queries executed Refined Query Domain-Level Source- Level SP-1 Domain Level Plan Source- Level … SP-2 … Query Planner & Optimizer Knowledge-Base

Query Execution & Data Extraction Checking availability of sources, identifying attributes to be extracted (using the source semantics) and extracting data Determining the output in XML and spatial data formats for storage and further querying Reuse of previously retrieved results is an integral part of this task Query Executor & Data Extractor Internet Results Query Extracted Results Data Store XML Data Repository Spatial Data Repository Query Plan Knowledge-Base

Integration of Results Generation of XQueries for combining extracted data Develop external functions for XQueries to access spatial data The result of the query will be transformed into a homogeneous schema for understanding and analyzing the results. Data Store XML Data Repository Spatial Data Repository Integrator Results Query Domain Level Plan Knowledge-Base Result Set

Ranking Two approaches to ranking [Telang:DBRank’07]– – Rank Before Integration: Applicable when user-specified metrics can be decomposed and applied to individual sub- queries – Rank After Integration: Applicable when user-specified metrics CANNOT be decomposed and applied to individual sub-queries Ranking Query Executor & Data Extractor Integrator

To Conclude Ideally, an information integration system should allow users to specify what information is needed without having to provide detailed instructions on how or from where to obtain the information. A number of challenges need to be addressed by different research communities (AI, DB, IR, NLP, Semantic Web, …) Existing work suggests we are on the right track Our proposed framework (InfoMosaic) could be a further step in this direction

Thank You !