Introduction to Data Structures Vamshi Ambati

Overview  Java you need for the Project  Search Engine and Data Structures  THIS Code Structure  On the Data Structure front Dictionaries (Dictionary Structures) Java Collections Linked List Queue [c] Vamshi Ambati2

Java you will need for the Project  Core Programming + I/O and Files  OOPS Inheritance Packages Encapsulation  Java API Collections [c] Vamshi Ambati3

What is a Search Engine?  A sophisticated tool for finding information on the web  An Index for the World Wide Web Analogous to the Index on a textbook Just Imagine a world without Search Engine! [c] Vamshi Ambati4

Why Index in the first place?  Which list is easier to search?  sow fox pig eel yak hen ant cat dog hog  ant cat dog eel fox hen hog pig sow yak  A Sorted list always helps Permits binary search. About log2n probes into list  log2(1 billion) ~ 3 [c] Vamshi Ambati5

How search engines work  The search engines maintain data of web sites in its database.  Use programs (often referred to as "spiders" or "robots") to collect information.  The information is then indexed by the search engine.  It allows users to look for the words or combination of words found in the index

Inverted Files A file is a list of words and this file contains words at various positions. Each entry of the word is associated with a position. [c] Vamshi Ambati8 POS FILE a (1, 4, 24…) entry (17…) file (2, 10) contains(11,….) position (25…) positions (15…) word (20….) words (6,12..). INVERTED FILE

Inverted Files for Multiple Documents [c] Vamshi Ambati9 DOCID OCCUR POS 1 POS “jezebel” occurs 6 times in document 34, 3 times in document 44, 4 times in document LEXICON WORD INDEX

A comprehensive form of Inverted Index [c] Vamshi Ambati10 SOURCE:

THIS  Search engine for the website  Website for the news paper The Hindu  Not for the entire web  Results are confined to only one web site [c] Vamshi Ambati11

Index Structure for our Project (THIS) t.pl?file= htm&date=2004/09 /15/&prd=blhttp:// t.pl?file= htm&date=2004/09 /15/&prd=bl :: 4 t.pl?file= htm&date=2002/10 /27/&prd=maghttp:// t.pl?file= htm&date=2002/10 /27/&prd=mag :: 7.. … htmhttp:// htm :: htmhttp:// htm :: 3.. …. gallery/0166/ htmhttp:// gallery/0166/ htm :: 2 gallery/0048/ htmhttp:// gallery/0048/ htm :: 1.. … … … … … [c] Vamshi Ambati12 India ManMoh an Cricket Bollywo Sharukh Sachin … ….

Search Engines

Search Engine Differences  Coverage (What part of the web do they really cover?)  Crawling algorithms Frequency of crawl depth of visits  Depth -0   Depth -1  Indexing policies Data Structures Representation  Search interfaces  Ranking [c] Vamshi Ambati14

[c] Vamshi Ambati15 Search Engine

Index [c] Vamshi Ambati16 Crawl Search

Index [c] Vamshi Ambati17 Query retrieve ResultSet FinalResult Sort by Rank ResultPage makePage TheWeb Spider Parser URLList crawl parse getNextUrl addUrls addPage Indexer store retrieve

Index [c] Vamshi Ambati18 Query retrieve ResultSet FinalResult Sort by Rank ResultPage makePage TheWeb Spider Parser URLList crawl parse getNextUrl addUrls addPage Indexer store retrieve Where are our data structures and algorithms lying? Queue Priority Queue Hashtable BinaryTree LinkedList MergeSort& InsertionSort

Code Structure (THIS) [c] Vamshi Ambati19 PageImgPageHref PageElement Spider WebSpider PageWord Queue SearchDriver PageLexer HttpTokenizerURLTextReader CrawlerDriver TreeDictionary Query addPage ListDictionary Indexer Index HashDictionary Index Save Restore Crawl Parse DictionaryInterface Inheritance Uses Calls DictionaryDriver

Dictionary Structures (Lexicon)  A Dictionary is an unordered container that contains key- element pairs Ordered Dictionary has the elements in sorted order  Keys are unique, but the values could be any [c] Vamshi Ambati20

Dictionary ADT  size(): returns the number of items in D Output: Integer  isEmpty(): Test whether D is empty. Output: Boolean  elements(): Return the elements stored in D. Output: iterator of elements (objects)  keys(): Return the keys stored in D. Output: iterator of keys (objects)  findElement(k): if D contains an item with key == k, then return the element of that item, else return NO_SUCH_KEY. Output: Object  findAllElements(k): Output: Iterator of elements with key k  insertItem(k,e): Insert an Item with element e and key k into D.  removeElement(k): Remove an item with key == k and return it. If no such element, return NO_SUCH_KEY Output: Object (element)  removeAllElements(k): Remove from D the items with key == k. Output: iterator of elements [c] Vamshi Ambati21 Also see the Java Standard API for Dictionary

Dictionary ADT in THIS Project  size(): returns the number of items in D Output: Integer  isEmpty(): Test whether D is empty. Output: Boolean  getKeys(): Return all the keys of the elements stored in D. Output: String array (Ideally it should be Vector!!)  getValue(k): if D contains an item with key == k, then return the element of that item, else return NULL. Output: Object  insertItem(k,e): Insert an Item with element e and key k into D.  remove(k): Remove an Item with key k from D.  We have customized the Dictionary a bit as we would be inserting only elements of the type !! [c] Vamshi Ambati22

Java Collections  java.util.* (A quite helpful library) Has implementations for most of the Data Structures They make life really easy You can not use the data structures inbuilt unless specified (Eg:Task1 Tasklet-A)  Use them for non-data structural purposes - Collections Eg: Arrays,Vectors, Iterators,Lists, Sets etc You would definitely be using “Iterator” atleast as you would be dealing with many Objects at a time! or.html. or.html [c] Vamshi Ambati23 See:

Other Data structures  Queue  LinkedList Beware! there are no Pointers in Java However there are “references”  Learn more about References in Java  Do not use the java.util package for DataStructures or Sorting Algorithms! You are expected to code them [c] Vamshi Ambati24

Summary  Learn data structures by implementing THIS  Mini version of a real search engine  Frame work is provided  More details in the next video [c] Vamshi Ambati25

THANK YOU [c] Vamshi Ambati26