Apache Tika End-to-End An introduction to Apache Tika, and integrating it to your application

Nick Burch Software Engineer Alfresco

Apache Tika Project which started in 2006 Grew out of the Lucene community, now widely used Provides detection of files – eg this binary blob is really a word file, that one is UTF-8 plain text Plain text, HTML and XHTML versions of a wide range of different file formats Consistent Metadata from different files Tika hides the complexity of the different formats and their libraries, instead presents a simple, powerful API Easy to use and extend

What's new? Lots of new parsers – text, office formats, publishing formats, images, audio, CAD, fonts etc Long standing parsers improved – better HTML from word for example Embedded resources and containers Use expanding – used by many SOLR users, Alfresco, lots of people crunching masses of data on Hadoop

Supported Formats Page 1 Audio – WAV, RIFF, MIDI DWG (CAD) Epub RSS and ATOM Feeds True Type Fonts HTML Images – JPEG, GIF, PNG, TIFF, Bitmap (including EXIF where found) iWork (Keynote, Pages etc) RFC822 mbox Mail

Supported Formats Page 2 Microsoft Outlook.msg Microsoft Office (Binary) – Word, PowerPoint, Excel, Visio, Publisher, Works Microsoft Office (OOXML) – Word, PowerPoint, Excel MP3 (id3 v1 and v2) CDF (Scientific Data) Open Document Format (Open Office) Old-style Open Office (.sxw etc) PDF

Supported Formats Page 3 Zip and Tar archives RDF Plain Text FLV Video XML Java class files And I probably forgot one...!

Metadata Tika provides consistent metadata across the range of parsers No need to know if it's “Last Author”, “Last Editor” or “Previous Author” in a file format, they all come back with the same metadata key Keys and values are strings, but strongly typed metadata entries provide converters to dates, ints etc

Text Content Tika generates HTML-like SAX events as it parses Uses Java SAX API Events can be captured or transformed Body Content Handler used for plain text HTML and XHTML available Can customise with your own handler, with XSLT or with E4X from JavaScript eg HTML Table → CSV

Calling Tika

// Get a content detector, and an auto- selecting Parser TikaConfig config = TikaConfig.getDefaultConfig(); ContainerAwareDetector detector = new ContainerAwareDetector( config.getMimeRepository() ); Parser parser = new AutoDetectParser(detector); // We’ll only want the plain text contents ContentHandler handler = new BodyContentHandler(); // Tell the parser what we have Metadata metadata = new Metadata(); metadata.set(Metadata.RESOURCE_NA ME_KEY, filename); // Have it processed parser.parse(input, handler, metadata, new ParseContext());

// Plain text only content handler ContentHandler handler = new BodyContentHandler(); String text = handler.toString(); // XHTML content handler SAXTransformerFactory factory = SAXTransformerFactory.newInstance(); TransformerHandler handler = factory.newTransformerHandler(); handler.getTransformer().setOutputProp erty(OutputKeys.METHOD, "xml"); handler.getTransformer().setOutputProp erty(OutputKeys.INDENT, "yes"); StringWriter sw = new StringWriter(); handler.setResult(new StreamResult(sw)); String text = sw.toString();

Tika Parsers

Parser Interface Two key methods – what mime types are supported, and do the parsing public interface Parser { Set getSupportedTypes(ParseContext context); void parse(InputStream stream, ContentHandler handler, Metadata metadata, ParseContext context) throws IOException, SAXException, TikaException; }

public class HelloWorldParser implements Parser { public Set getSupportedTypes(ParseContext context) { Set types = new HashSet (); types.add(MediaType.parse("hello/world" )); return types; } public void parse(InputStream stream, ContentHandler handler, Metadata metadata, ParseContext context) throws SAXException { XHTMLContentHandler xhtml = new XHTMLContentHandler(handler, metadata); xhtml.startDocument(); xhtml.startElement("h1"); xhtml.characters("Hello, World!"); xhtml.endElement("h1"); xhtml.endDocument(); metadata.set("hello","world"); metadata.set("title","Hello World!"); }

Demo: Tika-App

Demo: Geo-Tagged Images in Alfresco Share via Tika

Any Questions?