1. Clustering and Load Balancing Optimization for Redundant Content Removal Shanzhong Zhu (Ask.com) Alexandra Potapova, Maha Alabduljalil (Univ. of California at Santa Barbara) Xin Liu (Amazon.com) Tao Yang (Univ. of California at Santa Barbara)

2. Redundant Content Removal in Search Engines Over 1/3 of Web pages crawled are near duplicates When to remove near duplicates?  Offline removal  Online removal with query-based duplicate removal Online index matching & result ranking Duplicate removal User query Final results Offline data processing Duplicate filtering Web Pages Online index

3. Tradeoff of online vs. offline removal Online-dominating approach Offline-dominating approach Impact to offlineHigh precision Low recall Remove fewer duplicates High precision High recall Remove most of duplicates Higher offline burden Impact to onlineMore burden to online deduplication Less burden to online deduplication Impact to overall cost Higher serving costLower serving cost

4. Challenges &issues in offline duplicate handling Achieve high-recall with high precision  All-to-all duplicate comparison for complex/deep pairwise analysis  Expensive  parallelism management & unnecessary computation elimination Maintain duplicate groups instead of duplicate pairs  Reduce storage requirement.  Aid winner selection for duplicate removal  Continuous group update is expensive.  Approximation.  Error handling

5. Optimization for faster offline duplicate handling Incremental duplicate clustering and group management  Approximated transitive relationship  Lazy update Avoid unnecessary computation while balancing computation among machines  Multi-dimensional partitioning  Faster many-to-all duplicate comparisons Page partition … Page partition Page partition Page partition

6. Two-tier Architecture for Incremental Duplicate Detection

7. Approximation in Incremental Duplicate Group Management Example of incremental group merging/splitting Approximation  Group is unchanged when updated pages are still similar to group signatures  Group splitting does not re-validate all relations Error of transitive relation after content update  A B, B C  A C  A C may not be true if content B is updated. Error prevention during duplicate filtering:  double check similarity threshold between a winner and a loser

8. Multi-dimensional page partitioning Objective  One page is mapped to one unique partition  Dissimilar pages are mapped to different partitions.  Reduce unnecessary cross-partition comparisons. Partitioning based on document length  Outperform signature-based mapping for higher recall rates. Multi-dimensional mapping  Improve load imbalance caused by skewed length distribution Pages …

9. Multi-dimensional page partitioning Dictionary Sub-dictionary A=(600) A=(280,320) 1D length space 2D length space

10. When does Page A compare with B? Page length vector A= (A 1, A 2 ), B=(B 1,B 2 ) Page A needs to be compared with B only if τ is the similarity threshold ρ is a fixed interval enlarging factor

11. Implementation and Evaluations Implemented in Ask.com offline platform with C++ for processing billions of documents Impact on relevancy  Continuously monitor top query results.  Error rate of false removal is tiny. Impact on cost.  Compare two approaches –A: Online dominating.  Offline removes 5% duplicates first.  Most of duplicates hosted in online tier-2 machines –B: Offline dominating.

12. Cost Saving with Offline Dominating Approach Fixed QPS target. Two-tier online index for 3-8 billion URLs. 8%-26% cost saving with offline dominating  Less tier-2 machines due to less duplicates hosted.  Online tier 1 machines can answer more queries  Online messages communicated contain less duplicates

13. Reduction of unnecessary inter-machine communiation & comparison Up to 87% saving when using up to 64 machines

14. Effectiveness of 3D mapping Load balance factor with upto 64 machines Speedup of processing throughput

15. Benefits of incremental computation Ratio of non-incremental duplicate detection time over incremental one for a 100 million dataset. Upto 24-fold speedup. During a crawling update, 30% of updated pages have signatures similar to group signatures

16. Accuracy of distributed clustering and duplicate group management Relative error in precision compared to a single- machine configuration Relative error in recall

17. Conclusion remarks Budget-conscious solution with offline dominating redundant removal  Up to 26% cost saving. Approximated incremental scheme for duplicate clustering with error handling  Upto 24-fold speedup  Undetected duplicates are handled online. 3D mapping still reduces unnecessary comparisons (upto 87%) while balancing load (3+ fold improvement)