LIMS SOLR Integration Jake Lin Shmulevich Lab

LIMS for Systems Genetics Systems Genetics - study of complex traits (phenotypes) resulting from multiple genotypes and environment interactions Systems Genetics LIMS web app- content and process management o Spring MVC with Addama components o Aid research and improve operations  Sample and experiment tracking  Annotations  Visualizing - relationships and results  Pipelines - bash + python + http  Data sharing

Resources and Content 8 Natural Variant Crossings o ~3000 progeny 67 Sequencing submissions o 46 Multiplexed ~48 degrees 400,000 progeny images ~10X more content

Robust Search Heart of Information Management Simple & Fast Accurate & Meaningful

Addama JCR JCR addama LIMS Web Search XPath Search Hierarchy o file directory RESTful - http/ajax Domain Drawbacks: Slow

SOLR + Lucene LIMS Web Search Wraps Lucene -.jar Doug Cutting Apache o Matured o Ported to C++/C#,Pyton,Perl,... o IBM, Apple,... High performance text search engine library o indexing o querying Simple Configuration Web admin REST/HTTP APIs solr.war

SOLR schema.xml $TOMCAT_HOME/webapps/ROOT/solr/conf/schema.xml... Field types determine tokenizing and indexing o impact 'fuzzy' and 'like' search o

SOLR HTTP Post #Update/Insert - CSV curl ' -H 'Content-type:text/plain; charset=utf-8' #Update - JSON curl ' -H 'Content-type:application/json' #Delete curl ' -H "Content-Type: text/xml" --data-binary ' ypgKey:testKey_001 '

Data Migration Update/Insert - CSV LIMS built in export results to CSV function Import from Database

SOLR HTTP Get //Find all progenies for YCR6 &fl=ypgKey,ypgBoxNumber,ypgCrossingRef,ypgMatingType,ypgGenotype,ypgParentA,ypgParentAlpha,yp gAlias,ypgTetrad,ypgStatus,ypgPosition,ypgComments,ypgDateFrozen,ypgSiblings QTime: 9 ms {"response":{numFound:400,"docs":[{"ypgKey:ypgX",...}, {"ypgKey:ypgX",...},...]}} &hl=true&hl.fl=ypgPosition,ypgStatus

More get examples //range //AND + OR

SOLR ExtJs AJAX Get function getYPGSolrUrl(searchTerm) { return "/solr/select/?" + "q=" + searchTerm + "&wt=json&rows=5000&" + "fl=ypgKey,ypgBoxNumber,ypgCrossingRef,ypgMatingType,ypgGenotype,ypgParentA,ypgParentAlpha," + "ypgAlias,ypgTetrad,ypgStatus,ypgPosition,ypgComments,ypgDateFrozen,ypgSiblings"; } function goSearch(index, ypgSearchInput, ypgSearchOption) { if (ypgSearchInput == '') { ypgSearchInput = 'YPG'; } ypgSearchInput = checkWildcard(ypgSearchInput); var searchWin = getSearchLoadingWindow("yprogeny-"); var searchUrl = getYPGSolrUrl(ypgSearchOption + ":" + ypgSearchInput); searchWin.on("show", function () { var sb = Ext.getCmp("yprogeny-search-statusbar"); sb.showBusy(); }); searchWin.show(); Ext.Ajax.request({ url: searchUrl, method: "GET", success: function(response) { var searchResultObj = Ext.util.JSON.decode(response.responseText); myYPGData = []; loadYPGSearchResult(index, searchResultObj.response, function() { Ext.getDom("sample-search-result-list").innerHTML = ""; Ext.getDom("yo-form").innerHTML = ""; searchWin.close(); renderYPGSearchResult(); }); }, failure: function() { eventManager.fireStatusMessageEvent({ text: "Search Results failed for url:" + searchUrl, level: "error" }); } }); } //Post function postSolrUpdates(jsonObj, callback) { var docsol = {}; docsol["doc"] = jsonObj; var add = {}; add["add"] = docsol; Ext.Ajax.request({ url: "/solr/update/json?commit=true", method: "POST", jsonData: { add: docsol }, success: function() { callback(); }, failure: function() { Ext.Msg.alert("Error", "Failed updating/adding record - please let Jake know:" + jsonObj); } }); }

SOLR ExtJs AJAX Post /* jsonObj contains new and existing annotation values from form */ function postSolrUpdates(jsonObj, callback) { var docsol = {}; docsol["doc"] = jsonObj; var add = {}; add["add"] = docsol; Ext.Ajax.request({ url: "/solr/update/json?commit=true", method: "POST", jsonData: { add: docsol }, success: function() { callback(); }, failure: function() { Ext.Msg.alert("Error", "Failed updating/adding record - please contact Infocore with this info:" + jsonObj); } }); }

SOLR Java HttpClient public void testPost(String url, JSONObject jsonObject) { try { HttpClient client = new HttpClient(); PostMethod post = new PostMethod(url); post.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new DefaultHttpMethodRetryHandler(3, false)); JSONObject postObject = new JSONObject(); postObject.put("doc", jsonObject); JSONObject addObject = new JSONObject(); addObject.put("add", postObject); //"docs":[{"limsadminKey":"dudley_limsadminkey","limsKey":"dudley_limsadminkey","limsadminYoCount":791, // "limsadminYoMaxNum":512,"limsadminYoBoxNum":7,"limsadminYoPosition":"G3", // "allFields":["dudley_limsadminkey","791","512","7","G3","420","420","6","B5","1768","1768","20","A1","367","367","531","531","240"," 240","344","344"]}]}} post.setParameter("jsonData", "application/json"); post.setRequestEntity(new StringRequestEntity(addObject.toString(), "application/json", null)); post.setRequestHeader("Content-Type", "application/json"); int statusCode = client.executeMethod(post); System.out.println("Post " + url + "\nStatus code:" + statusCode); System.out.println(IOUtils.toString(post.getResponseBodyAsStream(), "UTF-8")); post.releaseConnection(); assertEquals(0,0); } catch (IOException e) { e.printStackTrace(); assertEquals(0,1); } catch (JSONException ej) { ej.printStackTrace(); assertEquals(0,1); }

update act as inserts, delete existing doc must use lowercase for wild card (*) search keys must be primitive type index corruption with java 1.7 start/stop tomcat o ROOT]$../../bin/shutdown.sh o ROOT]$../../bin/startup.sh Notes and observations

References Lucene in Action - Manning Press o o o o Science Perspective Science Perspective In progress o

Thanks Shmulevich Lab o Andrea Eakin o Hector Rovira o John Boyle o Ilya Shmulevich Dudley Lab o Gareth Cromie o Cathy Ludlow o Patrick May o Adrian Scott o Aimee Dudley