1. Deep-Web Crawling “Enlightening the dark side of the web”
Daniele Alfarone ○ Erasmus student ○ Milan (Italy)
Deep-Web Crawling “Enlightening the dark side of the web”

2. Structure Introduction Google’s Approach Improvements Conclusions
What is the Deep-Web
How to crawl it
Google’s Approach
Problem statement
Main algorithms
Performance evaluation
Improvements
Main limitations
Some ideas to improve
Conclusions

3. Deep-Web is the content “hidden” behind HTML forms
What is the Deep-Web?
Deep-Web is the content “hidden” behind HTML forms
Introduction
Google’s approach
Improvements
●○○○○
○○○○○○○○○○○○○○
○○○○○○○○○○

4. Hidden content
This content cannot be reached by traditional crawlers Deep-Web has 10 times more data than the currently searchable content
Introduction
Google’s approach
Improvements
●●○○○
○○○○○○○○○○○○○○
○○○○○○○○○○

5. How do webmasters deal with it?
Not only the search engines are interested: the websites want to be more accessible to the crawlers
The websites publish pages with long lists of static links to let traditional crawlers index them
Introduction
Google’s approach
Improvements
●●●○○
○○○○○○○○○○○○○○
○○○○○○○○○○

6. How can search engines crawl the Deep-Web?
But search engines cannot pretend that every website does the same…
How can search engines crawl the Deep-Web?
Developing vertical search engines
focused on a specific topic
flights jobs
But…
Limited to the number of topics for which a vertical search engine has been built
Difficult to keep semantic maps between individual data sources and a common DB
Boundaries between different domains are fuzzy
Introduction
Google’s approach
Improvements
●●●●○
○○○○○○○○○○○○○○
○○○○○○○○○○

7. Are there smarter approaches?
Currently the Web contains more than 10 millions “high-quality” HTML forms and it is still growing exponentially
Number of websites since 1990 (7% has an high-quality form)
Any approach which involves human effort can't scale: we need a fully-automatic approach without site-specific coding
Solution: the surfacing approach
Choose a set of queries to submit to the web form
Store the URL of the page obtained
Pass all the URLs to the crawler
Introduction
Google’s approach
Improvements
●●●●●
○○○○○○○○○○○○○○
○○○○○○○○○○

8. Part 2 Google’s approach
Problem statement
Main algorithms
Performance evaluation

9. Solving the surfacing problem: Google’s approach
The problem is divided in two sub-problems 1 2
Decide which form inputs to fill
Find appropriate values to fill-in these inputs
Introduction
Google’s approach
Improvements
●●●●●
●○○○○○○○○○○○○○
○○○○○○○○○○

10. HTML form example Free-text inputs Choice inputs ●●●●● ●●○○○○○○○○○○○○
Introduction
Google’s approach
Improvements
●●●●●
●●○○○○○○○○○○○○
○○○○○○○○○○

11. HTML form example Presentation inputs Selection inputs
Introduction
Google’s approach
Improvements
●●●●●
●●●○○○○○○○○○○○
○○○○○○○○○○

12. Which form inputs to fill: Query templates
Defined by Google as: the list of input types to be filled to create a set of queries
Query Template #1
Introduction
Google’s approach
Improvements
●●●●●
●●●●○○○○○○○○○○
○○○○○○○○○○

13. Which form inputs to fill: Query templates
Defined by Google as: the list of input types to be filled to create a set of queries
Query Template #2
Introduction
Google’s approach
Improvements
●●●●●
●●●●●○○○○○○○○○
○○○○○○○○○○

14. How to create informative query templates
discard presentation inputs
currently a big challenge
choose the optimal dimension for the template
too big: increase crawling traffic and produce pages without results
too small: every submission will get a large numbers of results and the website site may:
limit the number of results
allow to browse results through pagination (which is not always easy to follow)
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●○○○○○○○○
○○○○○○○○○○

15. Informativeness tester
How Google evaluates if a template is informative?
Query templates are evaluated upon the distinctness of the web pages resulting from the form submissions generated
To estimate the number of distinct web pages, the results are clustered based on the similarity of their content
# distinct pages
# pages
> 25%
A template is informative if…
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●○○○○○○○
○○○○○○○○○○

16. How to scale to big web forms?
Given a form with N inputs, the possible templates are
2N – 1
To avoid running the informativeness tester on all possible templates, Google developed an algorithm called
Incremental Search for Informative Query Templates
I.S.I.T.
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●○○○○○○
○○○○○○○○○○

17. √ √ X X ISIT example ●●●●● ●●●●●●●●●○○○○○ ○○○○○○○○○○ Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●○○○○○
○○○○○○○○○○

18. Generating input values
To assign values to a select menu is as easy as select all the possible values
To generate meaningful values for text boxes is a big challenge
Text boxes are used in different ways in web forms:
Generic text boxes: to retrieve all documents in a database that match the words typed (e.g. title or author of a book)
Typed text boxes: as a selection predicate on a specific attribute in the where clause of a SQL query (e.g. zip codes, US states, prices)
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●○○○○
○○○○○○○○○○

19. Values for generic text boxes2 1
Initial seed keywords are extracted from the form page
A query template with only the generic text box is submitted 4
Discard keywords not representative for the page (TF-IDF rank)
Additional keywords are extracted from the resulting page 3
Runs until a sufficient number of keywords has been extracted
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●○○○
○○○○○○○○○○

20. Values for typed text boxes
The number of types which can appear in HTML forms of different domains are limited (e.g.: city, date, price, zip)
Forms with typed text boxes will produce reasonable result pages only with type-appropriate values
To recognize the correct type, the form is submitted with known values of different types and the one with highest distinctness fraction is considered to be the correct type
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●○○
○○○○○○○○○○

21. Performance evaluation query templates with only select menus
As the number of inputs increase, the number of possible templates increases exponentially, but the number tested only increases linearly, as does the number found to be informative
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●○
○○○○○○○○○○

22. Performance evaluation mixed query templates
Testing on 1 million HTML forms, the URLs were generated using a template which had:
only one text box (57%)
one or more select menus (37%)
one text box and one or more select menus (6%)
Today on Google.com one query out of 10 contains "surfaced" results
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●●
○○○○○○○○○○

23. Part 3 Improvements
Main limitations
Some ideas to improve

24. 1. POST forms are discarded
The output of the whole Deep-Web crawling by Google is a list of URLs for each form considered.
The result pages from a form submitted with method=“POST” don’t have a unique URL
Google bypasses these forms relying on the fact the RFC specifications recommend POST forms only for operations that write on the website database (e.g.: comments in a forum, sign-up to a website)
But …
In reality websites make massive use of POST forms, for:
URL Shortening
Maintaining the state of a form after its submission
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●●
●○○○○○○○○○

25. How can we crawl POST forms?
Two approaches can drop the limitation put by Google:
POST forms can be crawled sending to the server a complete HTTP request, rather than just an URL. The problem becomes how to link (in the SERP) the page obtained submitting the POST form.
An approach which would solve all the problems stated is to simply convert the POST form to its GET equivalent. An analysis is required to assess which percentage of websites accept also GET parameters for POST forms.
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●●
●●○○○○○○○○

26. 2. Select menus with bad default values
When instantiating a query template, for select menus not included in the template, the default value of the menu is assigned, making the assumption that it's a wild card value like "Any" or “All”.
This assumption is probably too strong: in several select menus the default option is simply the first one of the list.
e.g. for a select menu of U.S. cities we would expect “All”, but we can find “Alabama”. If a bad option like Alabama is selected, a high percentage of the database will remain undiscovered.
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●●
●●●○○○○○○○

27. How can we recognize a bad default value?
Idea: to submit the form with all possible values and count the results
… if the number of results with the (potentially) default value is close to the sum of all the other results, probably it is a “real” default value.
Once we recognize a bad default value, we force the inclusion of the select menu in every template for the given form.
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●●
●●●●○○○○○○

28. 3. Managing mandatory inputs
Often the HTML forms indicate to the user which inputs are mandatory (e.g.: with asterisks or red borders).
To recognize the mandatory inputs can offer some benefits:
Reduce the number of URLs generated by ISIT only the templates which contain all the mandatory fields will be passed to the informativeness tester
Avoid to instantiate the default value (not always correct) to inputs that can just be discarded because they are not mandatory
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●●
●●●●●○○○○○

29. 4. Filling text boxes exploiting Javascript suggestions
An alternative approach for filling text boxes can be to exploit whenever a website uses suggestions proposed via Javascript.
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●●
●●●●●●○○○○

30. Algorithm to extract the suggestions
Type in the text box all the possible first 3 letters (with the English alphabet: 263 = submissions)
For each combination of 3 letters, retrieve all the auto-completion suggestions using a Javascript simulator
All suggestions can be assumed as valid inputs, we don’t need to filter according to relevance
The relevance filter will be applied only if the website is not particularly interesting
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●●
●●●●●●●○○○

31. 5. Input correlations not taken into account
Google uses the same set of values to fill an input for all templates that contain that input.
Usually some inputs are correlated e.g.: the text box “US city" and select menu "US state" or two text boxes representing a range
Advantages of taking correlation into account:
More relevant keywords for text boxes e.g. in a correlation between a text box and a select menu, we can submit the form for different select menu values and extract relevant keywords for the associated text box
Less zero-results pages are generated, resulting in less load for the search engine crawler and the website servers
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●●
●●●●●●●●○○

32. How to recognize a correlation?
To detect correlations between any two input types we can:
Use the informativeness test assuming that values are correlated only if the query results are informative
Recognize particular types of correlations e.g. if we have 2 select menus, where filling the first one restricts the possible values of the second one (US state/city, car brand/model) we can use a Javascript simulator to manage the correlation
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●●
●●●●●●●●●○

33. Conclusions
Deep-Web Crawling is one the most interesting today’s challenges for search engines
Google already implemented the surfacing approach obtaining encouraging results
But …
There are still several limitations
Some ideas have been illustrated to solve them
Introduction
Google’s approach
Improvements
●●●●●
●●●●●●●●●●●●●●
●●●●●●●●●●

34. References
J. Madhavan et al. (2008) Google’s Deep-Web Crawl
J. Madhavan et al. (2009) Harnessing the deep web: Present and future
W3C, Hypertext Transfer Protocol - HTTP/1.1 GET and POST methods definition
E. Charalambous How postback works in ASP.NET

35. Thank you
for the attention :)
Questions?