New free text search engine for www.stat.fi

Background ”Project” began year 2000 First version based on Finnish, commercial Oval search engine Oval was deemed too cumbersome to develop and lacked necessary features Search for new search service was started Jussi Arpalahti 21.09.07

New Search Service: Lucene/Solr Full text search Match highlight Categories (tags, facets, ...)‏ More like this Did you mean..? Finnish language support (Lingsoft module)‏ Free Software Fast, adaptable, scalable, popular in free and commercial settings from small to large scale Jussi Arpalahti 21.09.07

Structured Search Structure means distinguishable parts of document (title, author, paragraph, creation date)‏ Searches based on structure give better results Regular HTML page poorly structured, PC Axis and CoSSI XML much better F.x. searching directly from table title and variable names statistical data can be found Jussi Arpalahti 21.09.07

Implementation Solr supports infinite amount of fields in principle => structure easy to index Search syntax supports Boole's operators, field based, fuzzy and wildcard searches Solr accepts only text, so other solutions are needed to extract structured text from documents and feed it to index Formats supported as of now: HTML, CoSSI XML, PC Axis (PX Web)‏ Jussi Arpalahti 21.09.07

Still things to do Search service is newer ”done” Documents change and their structure evolves Users searches and search result tell more about services usability than developer(s) can test Most of Solr's features are yet to be utilized Most of Statistics Finland's documents are still poorly structured: better structure -> better search service This is just the beginning! Jussi Arpalahti 21.09.07