Having Your Cake and Eating It Too With Apache OODT and Apache Solr Andrew F. Hart Paul M. Ramirez

About Myself… Software Engineer –NASA Jet Propulsion Laboratory –“Data Management” Committer: –OODT, SIS, Gora, Streams (Incubating) Mentor: Streams (Incubating)

What We’ll Cover Overview of OODT & Solr Projects Strategies for Combining OODT and Solr Detailed Deployment/Config. Example Where to Learn More & Participate

Apache OODT Object Oriented Data Technology Origin in NASA mission data systems Components for –Information integration –Data cataloging and archiving –Configurable workflow processing

Apache OODT Apache –Incubation: 2010, Graduation: 2011 –29 Committers –Latest Release: 0.5 (Dec. 26, 2012)

Apache OODT Karoo Array Telescope (KAT-7)

Apache OODT Virtual Pediatric Intensive Care Unit

Apache OODT Regional Climate Model Evaluation System

Apache OODT Commonalities between systems –Lots of data –Defined processing steps / algorithms Archives important (… search important)

Apache OODT Strengths of OODT for the above use cases –Loosely coupled components –Standard protocols, well-defined interfaces –Highly configurable –Vetted, reliable code

Apache Solr Search + Web Services –Powerful features –Flexible formats –Highly configurable

Apache Solr The White House

Apache Solr Netflix

Apache Solr NASA Planetary Data System

OODT & Solr Why use these projects together? Archives often need search capability Similarities / Compatibilities –XML-based configuration –Environment (Java, Tomcat)

Example Integration “Standard” Data Archive Pipeline

Example Integration “Standard” Data Archive Pipeline + Search

OODT Products Typically 1-1 with Files Each uniquely identifiable (GUID) Support for higher-level “ProductType” –A way to define collections

OODT Metadata Annotations for products Key:{Val|Multival} Common across all OODT components Two general classes: –System –User

OODT Metadata System Metadata –Added automatically by OODT Components –Used to track state –Used to encode relationships between data

OODT Metadata User Metadata –Specified as “policy” –Can be product-level, or productType-level –Used to extract & persist information from files as they are ingested (become products)

OODT Metadata Metadata (Policy) Example (external)

Solr Schema XML document Define what will be indexed (“Fields”) Provide high-level context hints –Data type, behavior, pre-processing Extremely flexible, extensible

Solr Schema Solr Schema Example (external)

Making the Connection SolrIndexer Tool –Part of the File Manager component tools –Map OODT Metadata to Solr Fields –Create Solr documents from OODT products –Note: only talking about metadata

SolrIndexer Tool Org.Apache.Oodt.Cas.Filemgr.Tools Available since 0.4 Release Recommend to use 0.5+ as some stability improvements were added Several modes of operation

SolrIndexer Tool

Invocation Examples: Ingest all products from the specified File Manager instance java -DSOLR_INDEXER_CONFIG=/path/to/indexer.properties \ -Djava.ext.dirs=/path/to/cas/filemgr/lib/ \ org.apache.oodt.cas.filemgr.tools.SolrIndexer \ --all \ --fmUrl \ --solrUrl

SolrIndexerTool Invocation Examples: Ingest all products from the specified ProductType(s) java -DSOLR_INDEXER_CONFIG=/path/to/indexer.properties \ -Djava.ext.dirs=/path/to/cas/filemgr/lib/ \ org.apache.oodt.cas.filemgr.tools.SolrIndexer \ --types urn:some:ProductType \ --fmUrl \ --solrUrl

SolrIndexerTool Invocation Examples: Ingest a single product by its unique product id java -DSOLR_INDEXER_CONFIG=/path/to/indexer.properties \ -Djava.ext.dirs=/path/to/cas/filemgr/lib/ \ org.apache.oodt.cas.filemgr.tools.SolrIndexer \ --product 19bcb4b e1-b581-8b d \ [--delete] \ --fmUrl \ --solrUrl

SolrIndexerTool Invocation Examples: Force optimization of the Solr index java -DSOLR_INDEXER_CONFIG=/path/to/indexer.properties \ -Djava.ext.dirs=/path/to/cas/filemgr/lib/ \ org.apache.oodt.cas.filemgr.tools.SolrIndexer \ --optimize --solrUrl

Indexer.properties Configuration file for the SolrIndexer Specify mapping between OODT product metadata and Solr fields Additional “pre-processing” features

Indexer.properties Example Indexer.properties file (external)

Use Case I Building a searchable data archive “Long-term” / “Lights-out” archive Products & metadata immutable Many NASA mission data systems use this model Want to make it easily searchable

Use Case I “Standard” Data Archive Pipeline + Search

Use Cases II Building an interactively editable, searchable data archive Data and metadata mutable Want to dynamically select product(s) to edit based on metadata

Use Case II Interactively Editable Data Archive Pipeline + Search

Use Case II Interactively Editable Data Archive Pipeline + Search Solr catalog out of sync!

Synchronization Two ways (at least) to solve this: A. Modify the OODT Curator Services B. Treat OODT Curator Services as “black box” and write “wrapper” service to invoke Curator Services AND update Solr (via scripted call to SolrIndexer, for example)

Modify Curator Services Services implemented in JAX-RS /curator/src/main/java/org/apache/oodt/cas/curation/service [curator_url]/services/metadata/update Options: –Utilize Solr Java API –Wrap call to OODT SolrIndexer tool

Use Case II-A Modified Curator Services to Simultaneously update Solr

Example Interactive event tagging

Wrap Curator Services Curator Service/API is “black box” Develop custom service that: –Issues POST request to Curator service –Updates Solr index via, e.g.: Utilize Solr Java API Wrap call to OODT SolrIndexer tool

Use Case II-B Wrapping OODT Curation Services with Custom UI & Services

Example

Lessons Solr compliments OODT File Manager RESTful interfaces (Solr + OODT Curator) allow for great flexibility in designing services and UI “Best” approach depends on situation

Next Steps Develop “SolrCatalog” for OODT File Manager? –Pros: Reduction in “moving parts” –Cons: Restrictive? Implement Use Case II-A as optional mode for Curator web service layer

Learning More Solr – OODT – Homehttps://cwiki.apache.org/confluence/display/OODT/ Home

Thanks! Questions?