Building Data Integration Systems for the Web Alon Halevy Google NSF Information Integration Workshop April 22, 2010

Without (too much) Loss of Generality Web Enterprise, Science projects, … Information integration ≅ data management

A Few Principles Data management “in situ” –Data meaning is derived from its context –Manipulate data in its natural location Pay-as-you-go data management –Provide services before modeling is done –Data can be about any domain Collaboration should be built in –Query answering is only step the first step

Alex Facebook

Structured Data & The Web

Discover Manage, Analyze, Combine ExtractPublish Hard to query, visualize, combine data across organizations Requires infrastructure, concerns about losing control Hard to find structured data via search engines Data is embedded in web page, behind forms

Outline Surfacing the Deep Web Searching tables on the surface Web Fusion Tables: a platform for data management on the Web.

What is the Deep Web? store locations used cars radio stations patents recipes Deep = not accessible through general purpose search engines –Major gap in the coverage of search engines.

Tree Search Amish quilts Parking tickets in India Horses

Solution Constraints Can’t design a solution that requires domain engineering –(unless you can make money in that domain!) Boundaries between domains are fuzzy Solution needs to be integrated into general web search –Can’t assume special query syntax

Surfacing the Deep Web [Madhavan et al. VLDB 2008] Surfacing: –Find high-quality forms –Guess good queries to submit –Put the resulting HTML pages in the index ~3M sites, 50 languages, 700 domains queries per-second get results from the deep web. 400K forms served per day, 800K per week Impact mostly on the long and heavy tail of queries

Deep Web: The Future Still an opportunity to go deeper into the deep web: –E.g., map the user query into a form submission. Key challenge: given a keyword query, map it to forms in any domain Understanding the meaning of forms is still hard (e.g. content, geo constraints).

Outline Surfacing the Deep Web  Searching tables on the surface Web Fusion Tables: a platform for data management on the Web.

Bad table

Vertical Tables

Sub-Header Rows

Winners of the Boston Marathon (but that’s nowhere in the table)

Schema Ok, but context is subtle (year = 2006)

WebTables: Exploring the Relational Web [Cafarella et al., VLDB 2008, WebDB 08] In corpus of 14B raw tables, we estimate 154M are “good” relations –Single-table databases; Schema = attr labels + types –Largest corpus of databases & schemas we know of The Webtables system: –Recovers good relations from crawl and enables search –Builds novel apps on the recovered data

(Web-scale) Schema Collection name | , phone|telephone, _address| _address, date|last_modified instructorcourse-title|title, day|days, course|course-#, course-name|course-title electedcandidate|name, presiding-officer|speaker abk|so, h|hits, avg|ba, name|player sqftbath|baths, list|list-price, bed|beds, price|rent With 2.6 million schemas you can do some very interesting things. Synonym discovery

“KR”-Based Table Search [Wu, Madhavan, Miao, Pasca, Shen] Ideally, we describe every table: –Class of entities it contains –Properties being modeled –Context, quality, … Use Web-extracted knowledge bases –Extract isa-hierarchy using patterns: –“cities such as Paris and London” –“chemical elements including hydrogen and oxygen”

Step 1: Find “Subject” of Table Not always the left (or first non-number column)

Step 2: associate classes with subject Chemical elements Most of the time, the class labels are not in attribute name

Leveraging Web-extracted Ontologies Given a query, e.g., (country, GDP) –Rank tables about countries that have GDP somewhere in the schema. –Very high precision (~90%) Next challenge: understand binary properties and binary relationships. Domain specialization: –System should improve if given ontologies in a particular domain.

25 Combine Search, Extraction, Cleaning and Integration [Cafarella, Koussainova, H., VLDB 2009], Try to create a database of all “VLDB program committee members”

Outline Surfacing the Deep Web Searching tables on the surface Web  Fusion Tables: a platform for data management on the Web.

Data Management for the Web Era Integrate seamlessly with the Web: –Search, maps, … Easy to use: –Much broader user base, pay-as-you-go –Very simple data integration Provide incentives for sharing data Facilitate collaboration Fusion Tables – our current attempt [Madhavan, Gonzalez, Langen, Shapley, Shen]

We store and leverage a large collection of tables. Incentive

Incentive, Pay-..-Go

Coffee Production

Coffee Consumption

Seamless integration with other web tools

Toilet heat map…

Database functionality on map

Collaboration Table Search

Show up in search results!

Data Integration

Merged Table Carries attribution from both base tables. Owners maintain control of their own data.

Fine Grained Discussions

Example Uses of Fusion Tables Tracking potholes in Spain Displaying bike routes (MTBGuru) State of California statistics Government data from data.gov Data about voting locations in the USA Brazilian beaches Chicago homicides Most requested pop songs by year

Conclusions Information integration “in situ” –Blur the boundary between structured and unstructured data Combine search, extraction, cleaning and integration into a single experience Pay-as-you-go: introduce complexity as needed –Serve enterprises without IT depth OpenII – an open-source platform for information integration.

References Fusion Tables: –tables.googlelabs.com –SIGMOD, SOCC, 2010 Deep-web crawling: –[Madhavan et al., VLDB 08] WebTables: –[Cafarella et al., VLDB 08] Octopus: –[Cafarella et al., VLDB 09], –[Elmeleegy et al, VLDB 09]