Networked Software Systems Laboratory Department of Electrical engineering, Technion Employing Web Search indexing for fast creation of filtered view of large text files Students : Agbaria Mostafa, Atamlh Ahmad. Superviser : Oved Itzhak Lab Engineer : Dr .Ilana David

Background As network bandwidth increase , network servers (e.g. Web, Mail etc) create exceedingly large log files . The problem of searching in such files resembles the Web Search problem were it is prohibitively long to search all the data simplistically. This project is continuing for VLTFV project (Very Large Text File Viewer), Application responsiveness is independent of input file size.

Motivation Log requires human inspection for analyzing incidents as well as getting insight into the server operation for tuning. Inspecting very large log files verbatim by humans is impractical . Simplistic filtering (a-la grep) requires going over the entire file for every filter. which is very time consuming.

Project Goals In this project we plan to implement a new type of Index to the VLTFV Application that supports fast creation of filtered view of large text files using a Web Search Indexing technique.  The implementation is in Microsoft .NET and C#. Creating a database using inverted indexing for pre-processing the data in the log files, by this providing the user with easy and fast way to search the log file .

The reverse-index data structure Dictionary of words Lists of containing lines . The dictionary store all the different words. Each one has a list containing lines . List of appearances : contains lines’ number for the word’ s appearance.

Architecture

Sequence Overview (initialization)

Multi-Threaded Indexing Problem Run-time is very important. Run-time is mainly due to I/O requests and not CPU processing. The indexer takes more time to build the database than expected. Solution : Multi-Threaded Indexing

Multi-Threaded Indexing We built the database using Multi-Threading, meaning that the indexing of the file made in parallel using specific number of threads, each indexing a different part of the file, for faster indexing. Each thread Creates new database for its section in the file Sends the database to Web Technique Searcher. After getting all the sub-databases, we merge them into a Main Database. The following figure shows the time for building the database using various number of threads (file size = 100Mb). Time[sec] Threads Number

Multi-Threaded V.S Single-Thread Single Thread running Single Thread Page Fault ,Disk access, CPU idle. Time In ideal world Thread 1 running Multi Threading Thread 2 running Thread 3 running Thread 4 running

Testing Using Multi-Threaded indexing originates necessity for cautious checking. Partitioning the file to different threads, based on size in bytes, presents several interesting cases, e.g. : The chunk ends at the middle of the line. Line ends on exactly the end of chunk. File size smaller than chuck size (1). Trying to use more threads than permitted. Storing the correct chunks’ total lines. Empty lines. Changing the original file with additional lines at the end of the file. Serialization & De-Serialization. (1) chunk size was set to be 1Mb and used for setting the number of threads running in the program. Note that there is limitation for the maximum threads number.

Web Technique Searching Plug-In * When starting the program, the user can choose the Web Technique Searching.

3.4. Conventional Scroll Bar User Interface Design 3.1. Open File 3.2. Go to Line 3.3. Search 3.4. Conventional Scroll Bar 3.5. Scroll Knob 3.6. Line Numbers 3.7. Search Results Pane 3.9. Progress Bar 3.8. File lines counter 3.10 Text view area

Questions