1 Using the Lucene Search Engine

2 Team Phil Corcoran Project Leader 10 Years Software Telecoms, Finance, Manufacturing Reqs, Design, Test Derek O’ Keeffe Senior Developer 10 Years Software Java, ERP Joomla, Knowledge Tree Design, Code

3 Concepts

4 Lucene Full Text Search Cross Platform Lucene Document Inverted Index

5 Lucene Inserts index records Searches index Gets Lucene documents You Manage the indexing Select data files Parse files Manage querying Accept user’s query Display results to user Retrieve original documents

6 iViewXT

7 Search Improvements

8 Test Document Collections 1. UAT 2.

9 Super Mario

10 Implementation Derek

11 Performance

12 Lucene Implementation

13 Lucene Implementation: Indexing

14 Lucene Implementation: Indexing

15 Lucene Implementation: Indexing

16 Lucene Indexing

17 Lucene Indexing Step 1 of 5

18 Lucene Indexing Step 2 of 5

19 Lucene Indexing Step 3 of 5

20 Lucene Indexing Step 4 of 5

21 Lucene Indexing Step 5 of 5

22 Lucene Indexing

23 Text Extraction  Lucene not a complete application.  PDF files text extraction  Microsoft files text extraction

24 Lucene Implementation

25 Lucene Implementation

26 Searching:

27 Searching: Step 1 of 6

28 Searching: Step 2 of 6

29 Searching: Step 3 of 6

30 Searching: Step 4&5 of 6

31 Searching: Step 6 of 6

32 Searching:

33 Luke - Lucene Index Toolbox  Client application to link directly into your index.  Java-webstart app  Handy for testing searches and performance.

34 Some problems encountered  Max clause count exception: Take care automatically adding wildcards!!  Performance: Do the work while indexing, not while searching. Pagination: Get one page at a time from the Hits.  Our security model Stored collection of allowed containers in UserSession.  Visibility of indexing job. Added logging “Indexing document 426 of 204,532”

35 Resources (general)‏ An open source document management system in php with a java lucene search engine Handy ajax autocomplete component.

36 Resources (text extraction)‏ Text extractor for pdf files JXL Text extractor for excel files. Text extractor for word documents. API to access Microsoft format files. (xls/doc/ppt). I would recommend this one over jxl or text-mining above.

37 Summary Lucene querying is fast (take care what you do with the results) Indexing is slow (Make indexing job visible) Use Luke Add lots to the index (Do the work while indexing)

38 END