1. Panos Ipeirotis Luis Gravano
When one Sample is not Enough: Improving Text Database Selection using Shrinkage
Panos Ipeirotis
Luis Gravano
Computer Science Department
Columbia University

2. “Regular” Web Pages and Text Databases
Link structure
Crawlable
Documents indexed by search engines
Text Databases (a.k.a. “Hidden Web”, “Deep Web”…)
Usually no link structure
Documents “hidden” in databases
Documents not indexed by search engines
Need to query each collection individually
1/16/2019
Panos Ipeirotis - Columbia University

3. Text Databases: Examples
Search on U.S. Patent and Trademark Office (USPTO) database:
[wireless network]  26,012 matches
(USPTO database is at
Search on Google restricted to USPTO database site:
[wireless network site:patft.uspto.gov]  0 matches
Database
Query
Database Matches
Site-Restricted Google Matches
USPTO
wireless network
26,012
Library of Congress
visa regulations
>10,000
PubMed
thrombopenia
27,960
172
as of June 10th, 2004
1/16/2019
Panos Ipeirotis - Columbia University

4. Metasearchers Provide Access to Distributed Text Databases
Database selection relies on simple content summaries: vocabulary, word frequencies
thrombopenia
PubMed (11,868,552 documents) …
aids ,491 cancer 1,562,477 heart ,360 hepatitis ,129 thrombopenia ,960
Metasearcher
...
thrombopenia 27,960
thrombopenia 0
thrombopenia 42   ?
PubMed
NYTimes Archives
USPTO
1/16/2019
Panos Ipeirotis - Columbia University

5. Extracting Content Summaries from Autonomous Text Databases
Send queries to databases
Retrieve top matching documents
If “stopping criterion met” (e.g., sample>300 docs)
then exit;
else go to Step 1
Content summary contains words in sample
and document frequency of each word
If you have several points, steps, or key ideas use multiple slides. Determine if your audience is to understand a new idea, learn a process, or receive greater depth to a familiar concept. Back up each point with adequate explanation. As appropriate, supplement your presentation with technical support data in hard copy or on disc, , or the Internet. Develop each point adequately to communicate with your audience.
Problem: Summaries from small samples are highly incomplete
1/16/2019
Panos Ipeirotis - Columbia University

6. Problem: Summaries Derived from Small Samples Fundamentally Incomplete
Log(Frequency)
Sample=300
107
106
10% most frequent words
in PubMed database
9,000 . .
………………………………………
endocarditis
~9,000 docs / ~0.1%
103
102
2·104
4·104
105
Rank
Many words appear in “relatively few” documents (Zipf’s law)
Low-frequency words are often important
Small document samples miss many low-frequency words
1/16/2019
Panos Ipeirotis - Columbia University

7. Improving Sample-based Content Summaries
Challenge:
Improve content summary quality
without increasing sample size
Main Idea: Database Classification Helps
Similar topics ↔ Similar content summaries
Extracted content summaries complement each other
Classification available from directories (e.g., Open Directory) or derived automatically (e.g., QProber)
1/16/2019
Panos Ipeirotis - Columbia University

8. Databases with Similar Topics
Cancerlit contains “metastasis”, not found during sampling
CancerBacup contains “metastasis”
Databases under same category have similar vocabularies, and can complement each other
1/16/2019
Panos Ipeirotis - Columbia University

9. Content Summaries for Categories
Databases under same category share similar vocabulary
Higher-level category content summaries provide additional useful estimates of “word probabilities”
Can use all estimates in category path
1/16/2019
Panos Ipeirotis - Columbia University

10. Enhancing Summaries Using “Shrinkage”
Word-probability estimates from database content summaries can be unreliable
Category content summaries are more reliable (based on larger samples) but less specific to database
By combining estimates from category and database content summaries we get better estimates
1/16/2019
Panos Ipeirotis - Columbia University

11. Shrinkage-based Estimations
Adjust probability estimates
Pr [metastasis | D] =
λ1 * λ2 * λ3 * 0.092 + λ4 * 0.000
Select λi weights to maximize
the probability that the summary of D
is from a database under all its parent categories 
1/16/2019
Panos Ipeirotis - Columbia University

12. Computing Shrinkage-based Summaries
Root
Health
Cancer D
Pr [metastasis | D] = λ1 * λ2 * λ3 * 0.092 + λ4 * 0.000
Pr [treatment | D] = λ1 * λ2 * λ3 * 0.179 + λ4 * 0.184 …
Automatic computation of λi weights using an EM algorithm
Computation performed offline  No query overhead
Avoids “sparse data” problem and decreases estimation risk
1/16/2019
Panos Ipeirotis - Columbia University

13. Shrinkage Weights and Summary
new estimates
old estimates
CANCERLIT Shrinkage-based
λroot=0.02
λhealth=0.13
λcancer=0.20
λcancerlit=0.65
metastasis
2.5%
0.2% 5%
9.2% 0%
aids
14.3%
0.8% 7% 2%
20%
football
0.17% 1% …
Shrinkage:
Increases estimations for underestimates (e.g., metastasis)
Decreases word-probability estimates for overestimates (e.g., aids)
…it also introduces (with small probabilities) spurious words (e.g., football)
1/16/2019
Panos Ipeirotis - Columbia University

14. Is Shrinkage Always Necessary?
Shrinkage used to reduce uncertainty (variance) of estimations
Small samples of large databases  high variance
In sample: 10 out of 100 documents contain metastasis
In database: ? out of 10,000,000 documents?
Small samples of small databases  small variance
In database: ? out of 200 documents?
Shrinkage less useful (or even harmful) when uncertainty is low
1/16/2019
Panos Ipeirotis - Columbia University

15. Adaptive Application of Shrinkage
Database selection algorithms assign scores to databases for each query
When word frequency estimates are uncertain, assigned score has high variance
shrinkage improves score estimates
When word frequency estimates are reliable, assigned score has small variance
shrinkage unnecessary
Unreliable Score Estimate: Use shrinkage
Probability 1
Database Score for a Query
Reliable Score Estimate: Shrinkage might hurt
Probability
Solution: Use shrinkage adaptively in
a query- and database-specific manner 1
Database Score for a Query
1/16/2019
Panos Ipeirotis - Columbia University

16. Panos Ipeirotis - Columbia University
Searching Algorithm
Extract document samples
Get database classification
Compute shrinkage-based summaries
One-time process
To process a query Q:
For each database D:
Use a regular database selection algorithm to compute query score for D using old, “unshrunk” summary
Analyze uncertainty of score
If uncertainty high, use new, shrinkage-based summary instead and compute new query score for D
Evaluate Q over top-k scoring databases
For every query
1/16/2019
Panos Ipeirotis - Columbia University

17. Panos Ipeirotis - Columbia University
Evaluation: Goals
Examine quality of shrinkage-based summaries
Examine effect of shrinkage on database selection
CANCERLIT Correct
CANCERLIT Shrinkage-based
CANCERLIT Unshrunk
metastasis
12%
2.5% 0%
aids 8%
14.3%
20%
football
0.17%
regression 1%
1/16/2019
Panos Ipeirotis - Columbia University

18. Panos Ipeirotis - Columbia University
Experimental Setup
Three data sets:
Two standard testbeds from TREC (“Text Retrieval Conference”):
200 databases
100 queries with associated human-assigned
document relevance judgments
315 real Web databases
Two sets of experiments:
Content summary quality
Database selection accuracy
1/16/2019
Panos Ipeirotis - Columbia University

19. Results: Content Summary Quality
Recall: How many words in database also in summary?
Shrinkage-based summaries include 10-90% more words than unshrunk summaries
Precision: How many words in the summary also in database?
Shrinkage-based summaries include 5%-15% words not in actual database
1/16/2019
Panos Ipeirotis - Columbia University

20. Results: Content Summary Quality
Rank correlation: Is word ranking in summary similar to ranking in database?
Shrinkage-based summaries demonstrate better word rankings than unshrunk summaries
Kullback-Leibler divergence: Is probability distribution in summary similar to distribution in database?
Shrinkage improves bad cases, making very good ones worse
 Motivates adaptive application of shrinkage!
1/16/2019
Panos Ipeirotis - Columbia University

21. Results: Database Selection
Metric: R(K) = Χ / Υ
X = # of relevant documents in the selected K databases
Y = # of relevant documents in the best K databases
For CORI (a state-of-the-art database selection algorithm) with stemming over one TREC testbed
1/16/2019
Panos Ipeirotis - Columbia University

22. Panos Ipeirotis - Columbia University
Other Experiments
Choice of database selection algorithm
(CORI, bGlOSS, Language Modeling)
Comparison with VLDB’02 hierarchical database selection algorithm
Universal vs. adaptive application of shrinkage
Effect of stemming
Effect of stop-word elimination
1/16/2019
Panos Ipeirotis - Columbia University

23. Panos Ipeirotis - Columbia University
Conclusions
Developed strategy to automatically summarize contents of hidden-web text databases
Content summaries are critical for efficient metasearching
Strategy assumes no cooperation from databases
Shrinkage improves content summary quality by exploiting topical similarity
Shrinkage is efficient: no increase in document sample size required
Developed adaptive database selection strategy that decides whether to apply shrinkage on a database- and query-specific way
1/16/2019
Panos Ipeirotis - Columbia University

24. Thank you! Questions? http://sdarts.cs.columbia.edu
Shrinkage-based content summary generation
implemented and available for download at:
Questions?
1/16/2019
Panos Ipeirotis - Columbia University