Empowering EPrints Search with Xapian EPrints for Administrators Training @ University of Southampton 28th September 2011 Empowering EPrints Search with Xapian Sébastien François, EPrints Lead Developer EPrints Developer Powwow, ULCC EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Summary Review of EPrints Internal Search Indexing Searching Extras TO-DO’s Using & contributing Demo(s) EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 EPrints “Internal” Search - Overview Search DataSet List 1 1..n 1..n MetaField Field Condition 1..n EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 EPrints “Internal” Search – Overview (2) match = “EX” queries the main & auxilliary dataset tables match = “IN” queries the __rindex dataset table ordering is done via the __ordervalues_$langid dataset table EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 EPrints “Internal” Search – Downsides Simple search is not scalable Lots of derived data in the DB (backup?) No relevance matching -> good matches do not surface up No advanced features: suggestions, facets, boolean op’s etc. Home-brewed: hard to maintain the code, hard to extend Difficult to debug… EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 EPrints Xapian Search Introduced in 3.3 Only integrated with the simple search Little flexibility in controlling what is indexed Advanced features “not really” enabled Searches every fields (“text_index” not respected) But the idea is good & worth building upon EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Indexing Attempts to re-use EPrints’ default configuration: datasets’ field defintion (+ “text_index”) fields defined in the simple search (un-prefixed terms) But needs its own bits to define: default indexing methods (by MetaField type) facet-able indexes order-able indexes May be used to declare derived indexes – examples: “open_access”: to filter references from open full-text documents “year”: to filter by year of publication (rather than by date) “image_orientation”: if you had an archive of images, you could extract the orientation via EXIF EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Indexing - Classes Xapian::Index Config Xapian DB IndexMethod OrderMethod Fulltext Name, etc. Alpha. Name, etc. EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Indexing – Extra information Indexes are prefixed by “_” e.g. “_title” so we can sanitise the user query – otherwise users could do prefixed search (and search not necessarily allowed fields) Z notation: indicates a stemmed value or index: Z_title, Zhappi (internal Xapian convention) Script available to re-process the Xapian indexes (similar to “epadmin reindex” but doesn’t re-index the EPrints’ internal) Reserved indexes: _id: keep the internal id of the data-obj (/id/eprint/123) _dataset: to which dataset the record belongs to (‘eprint’, ‘user’…) _configuration_md5: keeps an MD5 of the conf. the item was indexed against (useful?) - _index_timestamp: when the item was last indexed EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Searching Again, attempts to re-use EPrints’ configuration: simple search (mostly for ordering methods) advanced/staff search: which fields to use (prefixed terms) Extra bits can be configured such as which facets can be used on each search (simple, advanced, …) Only indexed stuff can be searched  you cannot use a facet which has not been generated you need to re-index your data if you change the simple search def. same if you add new order-able fields EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Searching (2) Abstracted by Plugin::Search (original implementation) Tricky to make it work with EPrints’ UI because it expects an EPrints::Search object Plugin::Search::Internal is a wrapped EPrints::Search object (hack) so Plugin::Search::Xapian must emulate this behaviour EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Searching – Classes & Op. Stack /cgi/xapian Search::XapianSearch Paginate::Facets Xapian::Facets Plugin::Search::Xapian Xapian DB EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Searching – Extra information May be used in a script Exports & feeds work Can be serialised/de-serialised (including facets) so should work for Saved Searches (to test) EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Extras “Related Items” Jiadi has developed a Bootstrap-based Pagination module: more sexy supports alternative “views” of the search results EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 TO-DO’s Range searching: possible in Xapian but not yet implemented (e.g. 1..10) Some refactoring: Xapian::Index -> Xapian::Indexer Plugin::Search::Xapianv2 => Plugin::Search::Xapian (and replace the default EPrints’ Xapian implementation) Test with real life data (done to a certain extent...) Load & scalability testing (+ number of slots etc.) Multi-lang considerations (and related IndexMethod) EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 TO-DO’s – Would be nice Page displaying how a data-obj has been indexed prefixes terms facets & order-able fields Status page (cf. “Admin > Status”): DB size number of Documents indexed datasets (and how) Weighting: supported (via conf.) but un-tested in real life EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Internal Search vs Xapian Search Xapian is more of a user search The internal search is still required to: get records from the Database ($dataset->search()) this affects screens such as “Manage Deposits”, the “Review” etc. which cannot wait for items to be indexed (direct DB calls) may be needed to apply ACL’s (if some items cannot be searched): safer to use the (MySQL) DB as authority EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Debugging Xapian Plugin::Search::Xapian may be set to debug mode: shows processing and query building Xapian comes with an analysis tool, “delve” to: view the content of the Xapian DB or some selected Documents see if a term exists in the DB (and in which Documents) other info (term frequency etc.) Knowing what Xapian is searching and how a data-obj is indexed is key to debug most search-relating issues EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Using & Contributing Not quite at release stage but it is –currently- isolated so shouldn’t break your IR All the code is on GitHub: https://github.com/eprints/xapianv2 EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Demos http://puffin.ecs.soton.ac.uk/cgi/xapian Simple search / facets / export / order Simple search with boolean op’s, suggestion Advanced search / facets / export / order Related items http://vmdev1.eprints.org/cgi/xapian (more data + cached citations) http://vmdev1.eprints.org/cgi/xapian_status EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.

EPrints for Administrators Training @ University of Southampton 28th September 2011 Q&A & what’s next Let’s have a play? Code overview? Doc? EPrints Services, Web & Internet Science (WAIS) Research Group, Electronics & Computer Science, University of Southampton 2011.