1. Design a full-text search engine for a website based on Lucene
Chinese Students and Scholars Association
Presented by: Lijia Li, Yingyu Wu, Xiao Zhu

2. Outline Introduction Our goal System architecture
Conclusion and future work
Show demo

3. Introduction
With the development of the network, the amount of information on the Internet showed explosive growth, increased the difficulty of finding the target information, the search engine has brought great convenience to people looking for information, internet has become an indispensable tool.

4. Our goal
In this project, our goal is to implement a full-text retrieval engine based on Lucene.

5. Full-text retrieval engine
The full-text search engine based on the entire text retrieval technology for indexing and searching.
Features:
(1) The unstructured index file database
(2) Flexible retrieval methods
(3) Support nature language retrieval
(4) Retrieval efficiency
(1)索引文件能存储不同格式的数据文件
(2）支撑字符串检索等 多种检索方式
(4)不需要搜索整个文档，只需要索引
If we do full-text retrieval once time, we can use the result as many times as you want in a long peroid.

6. System Architecture
Search Engine is used to provide searching service to users. Our search engine has two main parts: online and offline.

7. Users website User Interface analyzer Search module Index File
Result sorting
Search module
Index File
Index module
Website database
crawler
Enter keyword
Online
Search
website
offline
Request
webpage

8. Lucene
Why
The index file format independent of the application platform
Inverted index
Object-oriented system architecture
Chinese parser (SmartchineseAnalyzer, IKAnalyzer)
Implement a set of powerful Query engine(RangeQuery, FuzzyQuery……)
Open Source

9. Web Crawler Analysis robots.txt Get robots.txt URL Analysis URL
Page database
Collection of start URL
URL Analysis
URL
Unprocessed URL queue
Page fetch module
Internet
Extract Links
Page analysis module
Extract the initial URL into unprocessed URL queue
Get a URL address from the head of the queue
Download pages according to their URL
Extract hyperlink from the download page
Extracted hyperlinks added to unprocessed URL queue
Check whether the unprocessed URL queue is null
if yes the program will be terminated
otherwise step 2 will be executed.
7. Loop
Architecture of web crawler

10. Work flow of web crawler
Extract the initial URL into unprocessed URL queue
Get a URL address from the head of the queue
Download pages according to their URL
Extract hyperlink from the download page
Extracted hyperlinks added to unprocessed URL queue
Check whether the unprocessed URL queue is null
if yes the program will be terminated
otherwise step 2 will be executed.
7. Loop

11. Index Whether Indexed? yes no Determine the type of document
Aset of documents to be index
Read and Analysis document
Whether Indexed?
Determine the type of document no
Date of index ealier than the creation data
yes
Whether exist same type
Parse document
Build index file
Call the corresponding document parser to parse document
Work flow

12. Document indexing steps
1. Creating a IndexWriter instance IndexWriter writer = new IndexWriter(indexPath, analyzer, boolean, maxFieldLength) 2. Creating a recode of Document Document doc = new Document() 3. Add Field Object in recode of Document doc.add(new Filed(string, tokenstream)) 4. Write recode of Document in Index writer.addDocument(doc); 5. Close Index Writer Object, end indexing writer.close()

13. Flow chart of searching
start
Example： User input: “ 大连理工 计算机”, “america ohio” After QueryParser: “大连理工” AND“计算机”, “america” AND “ohio”
Accept search string
from user
QueryParser analyze search string, output Query object
Set up Searcher
IndexSearcher object search related document in Index File
Output related document
end

14. Highlight search key word
Get position value of search key word
Get fragment of search key word, according position value of search key word
Use HTML and CSS attributes to highlight search key word

15. Conclusion and future work
What we learn through this project is how to use web crawler and Lucene to implement a full-text search engine.
Working on hadoop
Thank you!