David Walker Princeton University In Collaboration with AT&T Research Pads: Simplified Data Processing For Scientists

2 Standard Data Formats Behind the scenes, much of this information is represented in standardized data formats Standardized data formats: – Web pages in HTML – Pictures in JPEG – Movies in MPEG – “Universal” information format XML – Standard relational database formats A plethora of data processing tools: – Visualizers (Browsers Display JPEG, HTML,...) – Query languages allow users extract information (SQL, XQuery) – Programmers get easy access through standard libraries Java XML libraries --- JAXP – Many applications handle it natively and convert back and forth MS Word

3 Ad Hoc Data Formats Massive amounts of data are stored in XML, HTML or relational databases but there’s even more data that isn’t An ad hoc data format is any nonstandard data format for which convenient parsing, querying, visualizing, transformation tools are not available – ad hoc data is everywhere.

4 Ad Hoc data from Date: 3/21/2005 1:00PM PACIFIC Investor's Business Daily ® Stock List Name: DAVE Stock Company Price Price Volume EPS RS Symbol Name Price Change % Change % Change Rating Rating AET Aetna Inc % 31% GE General Electric Co % -8% HD Home Depot Inc % 63% IBM Intl Business Machines % -13% INTC Intel Corp % -47% Data provided by William O'Neil + Co., Inc. © All Rights Reserved. Investor's Business Daily is a registered trademark of Investor's Business Daily, Inc. Reproduction or redistribution other than for personal use is prohibited. All prices are delayed at least 20 minutes.

5 Ad Hoc data from !autogenerated-by: DAG-Edit version rev 3 !saved-by: gocvs !date: Fri Mar 18 21:00:28 PST 2005 !version: $Revision: $ !type: % is_a is a !type: < part_of part of !type: ^ inverse_of inverse of !type: | disjoint_from disjoint from $Gene_Ontology ; GO: <biological_process ; GO: %behavior ; GO: ; synonym:behaviour %adult behavior ; GO: ; synonym:adult behaviour %adult feeding behavior ; GO: ; synonym:adult feeding behaviour % feeding behavior ; GO: %adult locomotory behavior ; GO: ;...

6 Ad Hoc Data in Chemistry C5=CC=CC=C5)=O)C1

7 The challenge of ad hoc data What can we do about ad hoc data? – how do we read it into programs? – how do we detect errors? – how do we correct errors? – how do we query it? – how do we view it? – how do we gather statistics on it? – how do we load it into a database? – how do we transform it into a standard format like XML? – how do we combine multiple ad data sources? – how do we filter, normalize and transform it? In short: how do we do all the things we take for granted when dealing with standard formats in a reliable, fault-tolerant and efficient, yet effortless way?

8 Enter Pads Pads: a system for Processing Ad hoc Data Sources Two main components: – a data description language for concise and precise specifications of ad hoc data formats and properties – a compiler that automatically generates a suite of data processing tools robust libraries for C programming – parser that flags all errors and automatically recovers – printing utilities – constraint checking utilities converter to XML a statistical profiler – collects stats on common values appearing in all parts of the data; records error stats visual interface & viewer (coming soon!)

9 Pads Tool Generation Architecture Pads Compiler Gene Ontology description Statistical Profiler Tool gene data Profile ACE 25% BKJ 25%... XML Formatter Tool gene data Viewer Tool gene data

10 Pads Tool Generation Architecture Pads Compiler Gene Ontology description Gene Ontology Generated Parser Pads Base Library Gene Ontology Statistical Profiler Glue code for statistical profile

11 Pads Programmer Tools Pads Compiler Gene Ontology description Gene Ontology Generated Parser Pads Base Library Ad Hoc User Program Ad Hoc User Program in C

12 The Statistical Profiler Tool for each part of a data source, profiler reports errors & most common values. from example weblog data:.length : uint good: bad: 3824 pcnt-bad: min: 35 max: avg: top 10 values out of 1000 distinct values: tracked % of values val: 3082 count: 1254 %-of-good: val: 170 count: 1148 %-of-good: val: 43 count: 1018 %-of-good:

13 The Statistical Profiler Tool ad hoc data is often poorly documented or out-of-date even the documentation of weblog data from our textbook was missing some information: good: bad: 3824 pcnt-bad: – web server sometimes return a ‘-’ instead of length of bytes, which wasn’t mentioned in the textbook data descriptions can be written in a iterative fashion – use the profiler at each stage to uncover additional information about the data and refine the description

14 PADS language Based on Type Theory – in most modern programming languages, types (int, bool, struct, object...) describe program data the source of most of my research – in Pads, types describe physical data formats, semantic properties of data, and a mapping into an internal program representation (ie, a parser) – in Pads, types include base types for ints of different kinds, strings of different kinds, dates, urls,... structs and arrays for reading sequences unions, switched unions and enums for alternatives parameterized types to express dependencies & constraints recursive types to express recursive hierarchies (coming soon!) – Can describe ASCII, binary, and mixed data formats.

15 Future Work Ad Hoc Data Transformation & Integration – language and compiler support for moving data from the format you are given to the format you really want specifying simple transforms: permuting, dropping, computing fields; normalizing representations of dates, times, places... correcting errors integrating multiple sources Pads Applications – genomics data (with Olga Troyanskaya, Princeton CS) – networking and telephony data (AT&T) – financial data (Richard Liao, Princeton ORFE)

16 Challenges of Ad Hoc Data Revisited Data arrives “as is” – Format determined by data source, not consumers. The Pads language allows consumers to describe data in just about any format. – Often has little documentation. A Pads description can serve as documentation for a data source. The statistical profiler helps analysts understand data. – Some percentage of data is “buggy.” Constraints allow consumers to express expectations about data. Parsers check for errors and say where errors are located. Ad hoc data is a rich source of information for financial analysts, chemists, biologists, computer scientists, if they could only get at it. – Pads generates a collection of useful tools automatically from data descriptions

17 Pads Summary The overarching goal of Pads is to make understanding, analyzing and transforming ad hoc data an effortless task. We do so with new programming language technology based on the principles of Type Theory. AT&T Research: Kathleen Fisher Mary Fernandez Joel Gottlieb Robert Gruber (now Google) Ricardo Medel (summer intern) Princeton: Mark Daly (UGrad) Yitzhak Mandelbaum (Grad) David Walker

End!