Download presentation
Presentation is loading. Please wait.
1
Introduction to Data Structures Vamshi Ambati vamshi@andrew.cmu.edu
2
Overview Java you need for the Project Search Engine and Data Structures THIS Code Structure On the Data Structure front Dictionaries (Dictionary Structures) Java Collections Linked List Queue [c] Vamshi Ambati2
3
Java you will need for the Project Core Programming + I/O and Files OOPS Inheritance Packages Encapsulation Java API Collections [c] Vamshi Ambati3
4
What is a Search Engine? A sophisticated tool for finding information on the web An Index for the World Wide Web Analogous to the Index on a textbook Just Imagine a world without Search Engine! [c] Vamshi Ambati4
5
Why Index in the first place? Which list is easier to search? sow fox pig eel yak hen ant cat dog hog ant cat dog eel fox hen hog pig sow yak A Sorted list always helps Permits binary search. About log2n probes into list log2(1 billion) ~ 3 [c] Vamshi Ambati5
6
How search engines work The search engines maintain data of web sites in its database. Use programs (often referred to as "spiders" or "robots") to collect information. The information is then indexed by the search engine. It allows users to look for the words or combination of words found in the index
8
Inverted Files A file is a list of words and this file contains words at various positions. Each entry of the word is associated with a position. [c] Vamshi Ambati8 POS 1 10 20 30 36 FILE a (1, 4, 24…) entry (17…) file (2, 10) contains(11,….) position (25…) positions (15…) word (20….) words (6,12..). INVERTED FILE
9
Inverted Files for Multiple Documents [c] Vamshi Ambati9 DOCID OCCUR POS 1 POS 2...... “jezebel” occurs 6 times in document 34, 3 times in document 44, 4 times in document 56... LEXICON WORD INDEX
10
A comprehensive form of Inverted Index [c] Vamshi Ambati10 SOURCE: http://www.searchtools.com/slides/bestsearch/bls-24.htmlhttp://www.searchtools.com/slides/bestsearch/bls-24.html
11
THIS Search engine for the website http://www.hinduonnet.com/ Website for the news paper The Hindu Not for the entire web Results are confined to only one web site [c] Vamshi Ambati11
12
Index Structure for our Project (THIS) http://www.hinduonnet.com/thehindu/thscrip/prin t.pl?file=2004091500081100.htm&date=2004/09 /15/&prd=blhttp://www.hinduonnet.com/thehindu/thscrip/prin t.pl?file=2004091500081100.htm&date=2004/09 /15/&prd=bl :: 4 http://www.hinduonnet.com/thehindu/thscrip/prin t.pl?file=2002102700140200.htm&date=2002/10 /27/&prd=maghttp://www.hinduonnet.com/thehindu/thscrip/prin t.pl?file=2002102700140200.htm&date=2002/10 /27/&prd=mag :: 7.. … http://www.hindu.com/2004/10/09/stories/2004100 904051900.htmhttp://www.hindu.com/2004/10/09/stories/2004100 904051900.htm :: 23 http://www.hindu.com/2004/10/09/stories/2004100 910970300.htmhttp://www.hindu.com/2004/10/09/stories/2004100 910970300.htm :: 3.. …. http://www.hinduonnet.com/thehindu/ gallery/0166/016606.htmhttp://www.hinduonnet.com/thehindu/ gallery/0166/016606.htm :: 2 http://www.hinduonnet.com/thehindu/ gallery/0048/004807.htmhttp://www.hinduonnet.com/thehindu/ gallery/0048/004807.htm :: 1.. … … … … … [c] Vamshi Ambati12 India ManMoh an Cricket Bollywo Sharukh Sachin … ….
13
Search Engines
14
Search Engine Differences Coverage (What part of the web do they really cover?) Crawling algorithms Frequency of crawl depth of visits http://www.msitprogram.net/ Depth -0 http://www.msitprogram.net/ http://www.msitprogram.net/admissions.html/ http://www.msitprogram.net/admissions.html/ Depth -1 Indexing policies Data Structures Representation Search interfaces Ranking [c] Vamshi Ambati14
15
[c] Vamshi Ambati15 Search Engine
16
Index [c] Vamshi Ambati16 Crawl Search
17
Index [c] Vamshi Ambati17 Query retrieve ResultSet FinalResult Sort by Rank ResultPage makePage TheWeb Spider Parser URLList crawl parse getNextUrl addUrls addPage Indexer store retrieve
18
Index [c] Vamshi Ambati18 Query retrieve ResultSet FinalResult Sort by Rank ResultPage makePage TheWeb Spider Parser URLList crawl parse getNextUrl addUrls addPage Indexer store retrieve Where are our data structures and algorithms lying? Queue Priority Queue Hashtable BinaryTree LinkedList MergeSort& InsertionSort
19
Code Structure (THIS) [c] Vamshi Ambati19 PageImgPageHref PageElement Spider WebSpider PageWord Queue SearchDriver PageLexer HttpTokenizerURLTextReader CrawlerDriver TreeDictionary Query addPage ListDictionary Indexer Index HashDictionary Index Save Restore Crawl Parse DictionaryInterface Inheritance Uses Calls DictionaryDriver
20
Dictionary Structures (Lexicon) A Dictionary is an unordered container that contains key- element pairs Ordered Dictionary has the elements in sorted order Keys are unique, but the values could be any [c] Vamshi Ambati20
21
Dictionary ADT size(): returns the number of items in D Output: Integer isEmpty(): Test whether D is empty. Output: Boolean elements(): Return the elements stored in D. Output: iterator of elements (objects) keys(): Return the keys stored in D. Output: iterator of keys (objects) findElement(k): if D contains an item with key == k, then return the element of that item, else return NO_SUCH_KEY. Output: Object findAllElements(k): Output: Iterator of elements with key k insertItem(k,e): Insert an Item with element e and key k into D. removeElement(k): Remove an item with key == k and return it. If no such element, return NO_SUCH_KEY Output: Object (element) removeAllElements(k): Remove from D the items with key == k. Output: iterator of elements [c] Vamshi Ambati21 Also see the Java Standard API for Dictionary http://java.sun.com/j2se/1.4.2/docs/api/java/util/Dictionary.html http://java.sun.com/j2se/1.4.2/docs/api/java/util/Dictionary.html
22
Dictionary ADT in THIS Project size(): returns the number of items in D Output: Integer isEmpty(): Test whether D is empty. Output: Boolean getKeys(): Return all the keys of the elements stored in D. Output: String array (Ideally it should be Vector!!) getValue(k): if D contains an item with key == k, then return the element of that item, else return NULL. Output: Object insertItem(k,e): Insert an Item with element e and key k into D. remove(k): Remove an Item with key k from D. We have customized the Dictionary a bit as we would be inserting only elements of the type !! [c] Vamshi Ambati22
23
Java Collections java.util.* (A quite helpful library) Has implementations for most of the Data Structures They make life really easy You can not use the data structures inbuilt unless specified (Eg:Task1 Tasklet-A) Use them for non-data structural purposes - Collections Eg: Arrays,Vectors, Iterators,Lists, Sets etc You would definitely be using “Iterator” atleast as you would be dealing with many Objects at a time! http://java.sun.com/j2se/1.4.2/docs/api/java/util/Iterat or.html. http://java.sun.com/j2se/1.4.2/docs/api/java/util/Iterat or.html [c] Vamshi Ambati23 See: http://java.sun.com/docs/books/tutorial/collections/http://java.sun.com/docs/books/tutorial/collections/
24
Other Data structures Queue LinkedList Beware! there are no Pointers in Java However there are “references” Learn more about References in Java Do not use the java.util package for DataStructures or Sorting Algorithms! You are expected to code them [c] Vamshi Ambati24
25
Summary Learn data structures by implementing THIS Mini version of a real search engine Frame work is provided More details in the next video [c] Vamshi Ambati25
26
THANK YOU [c] Vamshi Ambati26
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.