1. Wujie Zheng 1, Hao Ma 2, Michael Lyu 1, Tao Xie 3, and Irwin King 1,4 1 CUHK, 2 Microsoft Research, 3 NCSU, 4 AT&T Labs Nov. 9, 2011 Mining Test Oracles of Web Search Engines

2. Problem How to check relevance of Web search results? 2

3. Problem Automation needed 3 Lots of search queriesExpected results change!

4. Related Work Multiple implementation testing –search engines often disagree with each other Mining specifications –often in implementation level The challenge –how to mine reliable test oracles from multiple search engines’ information 4

5. Our New Idea Mining input/output specifications across multiple systems 5

6. Key point: common input/output properties across different systems in different time Many ranking factors can also be used –http://www.vaughns-1-pagers.com/internet/google-ranking-factors.htm Example 6

7. The Approach Mine association rules between input/output properties as pseudo test oracles Detect violations in new search results 7

8. Evaluation Setup Google and Bing 3,432 queries 390,000 ranked lists of search results in 4 months (Top 10 results for each query in each day) 8

9. Example Mined Rules Rules 1-2: implications between Websites Rule 3: different opinions of search engines to certain Website Rule 4: the best top 1 search results 9

10. Example Violations April 1st, 2011, Bing violated the following rule: The top 1 search result of Bing was http://www.jcu.edu/index.php, from which one cannot find email entrance easily 10

11. Q&A 11 Thanks!