Analysis of Lucene Index on Hbase in an HPC Environment Prerna Shraff Anand Hegde

Concept BigTable Compressed, high performance database system built on GFS, Chubby Lock Service, SSTable etc. Hbase Hadoop database Open source distributed versioned column oriented Modeled after BigTable

Outline Data intensive computing requires storage solutions for huge amount of data. The requirement is to host very large tables on clusters of commodity hardware. HBase helps in fulfilling the above requirement. Hbase provides Bigtable like capabilities on top of Hadoop.

The Idea Current implementation in this field includes an experiment using Lucene Index on Hbase in an HPC Environment. (Xiaoming Gao, Vaibhav Nachankar, Judy Qiu) To expand the scope of the existing project. To evaluate the performance in terms of many other parameters.

Architecture

Implemented solution Use of inverted index using Lucene index. Index refers to doc1 -> “cloud” Inverted index refers to “cloud” -> doc1 Apache Lucene was used to implement inverted indices. Apache Lucene supports full-text search.

Implemented Design The existing design has separate tables for book images, book texts and Lucene indices.

System Implementation

Initial Analysis Experiment was performed on the Alamo HPC cluster of FutureGrid. Experiment was conducted with 5 Books. Total terms evaluated : 8263

Initial Data Analysis

Initial Data Analysis

Proposed Work To test across more number of data sets. To test across different clusters like India on FutureGrid. To test across different number of HDFS data nodes. To test across more number of client nodes with different number of client queries.

Obstacles we can face Hardware differs from cluster to cluster and performance will differ accordingly. Problems may occur with increase or decrease of data nodes. Important items to consider would be switching capacity of the device, number of systems connected and uplink capacity. Finding appropriate number of data sets.

References Hbase http://hbase.apache.org/book.html#ops_mgt BigTable http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/bigtable-osdi06.pdf Experiment using Lucene Index on Hbase in an HPC Environment. (Xiaoming Gao, Vaibhav Nachankar, Judy Qiu)

Thank You