1. Richa Arora

2.  Tool Identified and Overview  Schema.xml  Tokenization, Stop words, and Synonym Handling  Indexing  Data Import Handler  Query format and Matching documents to query  Function Queries  Bibliography

4.  SOLR - Open Source enterprise search platform from Apache Lucene project  Purpose ◦ To implement a full text search functionality in a web application  Commercial Websites using SOLR ◦ www.digg.com www.digg.com ◦ http://www.whitehouse.gov/ - Uses SOLR via Drupal for site search w/highlighting & faceting http://www.whitehouse.gov/ ◦ http://beta.fcc.gov/ http://beta.fcc.gov/ ◦ http://www.netflix.com/ http://www.netflix.com/

5. Web serverDatabase server Web Application SOLR Document Database

6.  Features ◦ Full text search ◦ Rich document handling (including MS Word, PDF, RTF etc.) ◦ HTML administration interface ◦ Scalable  Technology ◦ Java programming language ◦ Lucene Java search library ◦ Runs as a search server within a servlet container such as Tomcat or Jetty

7. Documents Browser based web interface Solr Server Documents for indexing Search Queries Search Results Indexing Searching Index schema.xml solrconfig.x ml

8.  Documents form the basic unit of SOLR  Documents are composed of fields  Examples: ◦ Document for Person: Fields – name, height, age, etc. ◦ Document for Recipes: Fields – origin, ingredients, etc.  Documents are fed to SOLR  SOLR extracts the information from the fields in the documents and makes it searchable  Steps: ◦ Field Analysis ◦ Tokenization ◦ Filter application ◦ Indexing

10.  Governs how should SOLR build indexes from input documents  Defines field types and specific fields that the documents can contain  Describes how SOLR should handle the fields when adding documents to the index or when querying those fields

12.  These are used for examining the text of fields and to generate a token stream  Indexing Analyzers: The results of the analysis are added to an index and a set of terms like positions, sizes, etc for a field are defined  Querying Analyzers: The values being searched for are analyzed and the terms that result are matched against those that are stored in the field's index

14.  To splits a stream of text into tokens  Tokens are subsequences of the characters  A token contains various metadata in addition to its text value, such as the location at which the token occurs in the field  Example ◦ Standard Tokenizer: Treats whitespace and punctuation as delimiters  Input: “Email: johndoe@xyz.com”johndoe@xyz.com  Output: “Email:”, “johndoe@xyz.com”“johndoe@xyz.com ◦ N-Gram Tokenizer: Reads the field text and generates n-gram tokens of sizes in the given range (default minimum is 1 and maximum is 2)  Input: “hello world”  Output: “h”, “e”, “l”, “l”, “o”, “ “, “w”, “o”, “r”, “l”, “d”, “he ”, “el”, “ll”, “lo”, “o “, “wo”, “or”, “rl”, “ld”

15.  Filters take tokens as input from the Tokenizers and produce another stream of tokens as output  Multiple filters can be used one after the other  Example:

18.  Stop Filter: This filter is used to discard tokens that are on the given stop words list. A standard stop words list is included in the SOLR config directory, named stopwords.txt, for English language text  Example: Using the standard stopwords.txt Tokenizer Input : “welcome to the world of Solr” Tokenizer Output/Filter Input: “welcome”(1), “to”(2), “the”(3), “world”(4), “of”(5), “Solr”(6) Filter Output: “welcome”(1), “world”(2), “Solr”(3)

20.  Synonym Filter: This is used for finding synonyms at the time of indexing as well as while querying. Tokens are looked up in the list of synonyms and if a match is found, then the synonyms are put in place of the token  Example: We can define the synonyms in a file (test_synonyms.txt) and use it for comparing the tokens ◦ home, dwelling, house ◦ shop => workshop, store ◦ teh => the Tokenizer Input : “teh home shop” Tokenizer Output/Filter Input: “teh”(1), “home”(2), “shop”(3) Filter Output: “the”(1), “workshop”(2), “shop”(2), “home”(2), “dwelling”(3), “house”(3)

22.  Refers to adding the content to a SOLR index  To make the content searchable  Sources of data for indexing: ◦ XML ◦ CSV ◦ Rich text formats (PDF, MS Word, MS Excel, text etc.) ◦ Data extracted from tables in a database

23.  Uploading Data with SOLR Cell ◦ Using ExtractingRequestHandler ◦ With a POST ◦ With SOLR Cell and SOLRJ  Uploading Data with Index Handlers ◦ XMLUpdateRequestHandler for XML-formatted Data ◦ Using the CSVRequestHandler for CSV Content ◦ Indexing Using SOLRJ  Uploading Structure Data Store Data with the Data Import Handler  Content Streams

24.  curl posts and retrieves data over HTTP, FTP, and many other protocols  In the example below, the Extraction Request Handler is called, uploads the file tutorial.html and assigns it the unique ID doc1  curl “http://localhost:8983/solr/update/extract? literal.id=doc1&uprefix=attr_&fmap.content=attr_content&commit=true” -F "myfile=@tutorial.html"  literal.id provides a unique ID to the document uploaded to SOLR  commit=true makes the document searchable after indexing  The -F flag instructs curl to POST data using the Content-Type multipart/form-data and supports the uploading of binary files  The @ symbol instructs curl to upload the attached file  The argument myfile=@tutorial.html needs a valid file path

25. Order of operation: 1.Modify the schema.xml file to add the fields which may not be already existing in the schema.xml file, example: authors, dd, isbn, yearpub, publisher 2.Modify the schema.xml file to copy the newly created fields to text field to make the search results viewable 3.Run the curl utility with the command for adding XML document: curl http://localhost:8983/solr/update -H "Content-Type: text/xml" --data-binary " doc26 Patrick Eagar Sports 796.35 0002166313 1982 Collins "

26.  Often data is stored in relational databases  Data Import Handler (DIH) provides a mechanism to import data from database and to index it  DIH can also index content from RSS and ATOM feeds, e-mail repositories and structured XML

27.  Handler to be registered in the solrconfig.xml file ${solr.config.dir:./solr/conf}/dataimporthandler/data- config.xml  There can be multiple configuration files

28. 1.Create a database in SQL Server 2005 2.The tables and the relationships in the database are shown below

29. 3. Create an XML file called DIH_Test.xml for importing into SOLR 4. Modify solrconfig.xml file to instruct SOLR to import data as per the file DIH_Test.xml

30. 5. Do a full-import of the DIH from the browser using: http://localhost:8983/solr/dataimport?command=full -import

31. 7. Run queries on the newly indexed data from the database 8. Example: http://localhost:8983/solr/select?q= ipad2 The above query returns the result. Executing queries on the original database returns similar results

34. Request Handler Query Parser Index Response Writer qt: selects a Request Handler for a query using /select defType: selects a query parser for the query qf: selects which field to query in the index start: specifies an offset into the query results where the returned response should begin rows: specifies the number of rows to be displayed at run time fq: flters the query by applying an additional query to the initial query’s results; caches the results wt: selects a response writer for formatting the query response

35.  Advantage - Enables the user to specify very precise queries  Disadvantage – Is less tolerant of syntax errors than the DisMax query parser  Parameters Supported ◦ Terms – Use of wild card characters, Fuzzy Searches, Boosts and Ranges ◦ Fields – Identified by name followed by a colon ◦ Boolean Operators – AND, OR, NOT, &&, !, || ◦ Common query parameters – debugQuery, defType, explainOther, fl, fq, omitHeader, rows, sort, start, timeAllowed ◦ Functions – abs, constant, div, fieldValue, log, linear, max, etc. ◦ Faceting ◦ Highlighting ◦ MoreLikeThis (mlt)

36.  q – Defines a query using standard query syntax. This parameter is mandatory  q.op – Specifies the default operator for query expressions (this parameter’s value is defined in schema.xml). Possible values are “AND” or “OR”  df – Specifies a default field, overriding the definition of a default field in schema.xml Default parameter values are specified in solrconfig.xml

37.  Query http://localhost:8983/solr/select? q=id:6H500F0&popularity=6

38.  Fuzzy Searches - based on the Levenshtein Distance or Edit Distance  E.g. tight~ will match terms like flight, slight etc.  Additional parameter to specify degree of similarity – tight~0.8 will match sight. When set closer to 1, optional parameter causes only terms with higher similarity to be matched  If numerical parameter is omitted, the default value taken is 0.5

39.  Range Searches ◦ Specifies a range(with an upper and lower bound) of values for a field ◦ Can be inclusive or exclusive of the lower and upper bounds Query: http://localhost:8983/solr/select? q=popularity:{5 TO 7}

40. ParameterDescription defTypeQuery parser to be used (DisMax or Standard Query Parser) SortSorts the response to a query in asc or desc order based on response’s score or other characteristic StartOffset into the responses at which solr should begin displaying content RowsNumber of rows of responses displayed at a time fqFilter query for search results flLimits responses to a listed set of fields

41. ParameterDescription debugQueryInclude debugging information timeAllowedTime allowed for a query to be processed. If time elapses before response is complete are returned, partial information returned omitHeaderExcludes header information from returned results wtSpecifies the response writer

42.  Used to generate a relevancy score using the actual value of one or more numeric fields  Functions available for function queries ◦ abs – abs(x); abs(-5) ◦ constant - 1.5; _val_:1.5 ◦ div – div(1,y); div(sum(x,100), max(y,1)) ◦ linear – linear(x, m, c); linear(x, 2, 4) returns 2*x+4 ◦ log – log(x); log(sum(x,100)) ◦ …  Include function query in a SOLR query ◦ With a _val_keyword – e.g. _val_:myNumericField ◦ Parameter with an explicit type of FunctionQuery (DisMax query parser’s bf parameter)

43. http://localhost:8983/solr/select/?q=cat:electronics+_val_:”div(price,weight)”&fl=*,score

44.  Generated a formatted response of a search  wt parameter sets the response writer  Response writers supported ◦ Json ◦ Php ◦ Phps ◦ Python ◦ Ruby ◦ Xml ◦ xslt

45.  http://wiki.apache.org/solr/FrontPage http://wiki.apache.org/solr/FrontPage (link last accessed on 04/25/2011)  Lucid Works SOLR Reference Guide 1.4 http://www.lucidimagination.com/user_down load/certified/cdrg/lucidworks-solr- refguide-1.4.pdf http://www.lucidimagination.com/user_down load/certified/cdrg/lucidworks-solr- refguide-1.4.pdf (link last accessed on 04/25/2011)