Topes: Enabling End-User Programmers to Validate and Reformat Data Christopher Scaffidi Key collaborators: Brad Myers, Mary Shaw Carnegie Mellon University

2 Hurricane Katrina “Person Locator” site: Many inputs unvalidated... and error-ful Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

3 Data errors reduce the usefulness of data. Even little typos impede data de-duplication. Age is not useful for flying my helicopter to come rescue you. Nor is a “city name” with 1 letter. Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

4 Hurricane Katrina sites are not alone in lacking input validation. Eg: Google Base web application –13 primary web forms –Even numeric fields accept unreasonable inputs (such as a salary of “-45”) Eg: Spreadsheets –40% of cells are non-numeric, non-date textual data –Often used to gather/organize textual data for reports Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

5Outline 1.Challenges of data validation 2.Topes Model for describing data Tools for creating/using topes 3.Evaluations 4.Conclusion Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

6 Digging into the details: real user inputs that need validation. Sources: –Interviews of Hurricane Katrina website creators –Survey of Information Week readers –Contextual inquiry of information workers who created and used websites –Logs of what admin assistants typed into browsers –Exploration of the EUSES spreadsheet corpus Validating user inputs has 3 primary challenges… Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

7 1. Inputs don’t always conform well to the simple “binary” validation model. Data is sometimes questionable… yet valid. –Eg: a suspiciously long address –In practice, person names and other proper nouns are never validated with regexps… too brittle. –Life is full of corner cases and exceptions. If code can identify questionable data, then it can double-check the data: –Ask an application end user to confirm the input –Flag the input for checking by a system administrator –Compare the value to a list of known exceptions –Call up a server and see if it can confirm the value Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

8 2. User inputs often can occur in multiple different formats. Two different strings can be equivalent. –How many ways can you write a date? –What if an end user types a date in the wrong format? –“Jan ” and “1/1/2007” mean the same thing because of the category that they are in: date. –Sometimes the interpretation is ambiguous. In real life, preferences and experience guide interpretation. If code can transform among formats (ie: not just recognize formats with regexps), then it can put data in an unambiguous format as needed. –Display result so users can check/fix interpretation Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

9 3. The meaning of data is often tied to its “parts”, not directly to its characters. Data often has parts, each with a meaning. –What are the parts of a date, 12/31/2008? –Valid data obeys intra- and inter-part constraints. –Constraints are usually platform-independent –Writing regexps requires you to translate constraints into a character sequence… tough in many cases, practically or truly impossible in others. If code could succinctly state the parts, as well as mandatory and optional constraints on the parts, wouldn’t the code be easier to write and maintain? –Especially if it was platform-independent! Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

10 Limitations of existing approaches Types do not support questionable values Grammars do not, either, nor can they reformat Information extraction algorithms rely on grammatical cues that are absent during validation Cues, Forms/3,  -calculus, Slate, pollution markers, etc, infer numerical constraints but not constraints on strings, nor are they platform-independent Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

11 Imagine a world where… Code can ask an oracle, “Is this a company name?”, and the oracle replies yes, no, almost definitely, probably not, and other shades of gray. Code allows input in any reasonable format, since the code can ask the oracle to put the input into the format that is actually needed. People teach the oracle about a new data category by concisely stating its parts and constraints. Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

12 New Approach: Topes A tope = a platform-independent abstraction describing how to recognize and transform strings in one category of data Greek word for “place,” because each corresponds to a data category with a natural place in the problem domain Validating with topes improves –Accuracy of validation –Reusability of validation code –Consistency of data formatting Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

13 A tope is a graph. Node = format, edge = transformation Notional representation for a CMU room number tope… Formal building name & room number Elliot Dunlap Smith Hall 225 Colloquial building name & room number Smith 225 Building abbreviation & room number EDSH 225 Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

14 A tope is a conceptual abstraction. A tope implementation is code. Each tope implementation has executable functions: –1 isa:string  [0,1] function per format, for recognizing instances of the format (a fuzzy set) –0 or more trf:string  string functions linking formats, for transforming values from one format to another Validation function:  (str) = max(isa f (str)) where f ranges over tope’s formats –Valid when  (str) = 1 –Invalid when  (str) = 0 –Questionable when 0 <  (str) < 1 Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

15 Common kinds of topes: enumerations and proper nouns Multi-format Enumerations, e.g: US states –“New York”, “CA”, maybe “Guam” Open-set proper nouns, e.g.: Company names –Whitelist of definitely valid names (“Google”), with alternate formats (e.g. “Google Corp”, “GOOG”) –Augmented with a pattern for promising inputs that are not yet on the whitelist Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

16 Two other common kinds of topes: numeric and hierarchical Numeric, e.g.: human masses –Numeric and in a certain range –Values slightly outside range might be questionable –Sometimes labeled with an explicit unit –Transformation usually by multiplication Hierarchical, e.g.: address lines –Parts described with other topes (e.g.: “100 Main St.” uses a numeric, a proper noun, and an enum) –Simple isas can be implemented with regexps. –Transformations involve permutation of parts, lookup tables, and changes to separators & capitalization. Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

17 Tope Development Environment (TDE) Topei Module Infers tope from examples Toped Module Enables EUPs to create/edit topes Topeg Module Generates context-free grammars and transformations Topep Module Parses data against grammars, performs transformations Plug-ins Read/write program data Robofox Web macros Vegemite/CoScripter Web macros Microsoft Excel Spreadsheets Visual Studio.NET Web applications … Introduction  Challenges  Topes  Tools  Evaluation  Conclusion Repository Stores topes for sharing/reuse

18 Toped User Interface Features Format inference Format/part names Soft constraints Value whitelists Testing features Format reusability Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

19 Integration with programming platforms Microsoft Excel: buttons and menus Visual Studio: drag-and drop code generation Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

20 Integration with programming platforms Introduction  Challenges  Topes  Tools  Evaluation  Conclusion Recommends tope for the data at hand Convenient access to reformatting

21 Other integrations to date: CoScripter, Robofox, XML/HTML library Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

22 Evaluating accuracy Implemented topes for spreadsheet data –Grouped 1712 columns of spreadsheet data (from the EUSES spreadsheet corpus) into data categories –Created 32 topes for the most common 32 data categories (~ 70% of the data) –Compared validation with topes to validation with regexps or enumerations from the web –Tope-based validation was over 3 times as accurate (for 5 formats or regexps per data category) Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

23 Evaluating reusability Reused spreadsheet-based topes on webform data –Downloaded data for 8 data categories on Google Base and 5 in Hurricane Katrina website –Reused spreadsheet-based topes on the web data –Validation was just as accurate (and sometimes even better, as the webform data was from just two sources and therefore less diverse than the spreadsheet data) Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

24 Evaluating support for data cleaning Used topes to put web data into consistent formats –Again with the 5 columns in Hurricane Katrina website –Used transformation functions to put each string into the most common format for that data category –Increased number of duplicate strings found by 10% Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

25 Evaluating usability for data validation End users validating data with single-format topes –Between-subjects lab study (early version of Toped) –8 users validated spreadsheet data with Toped; for comparison, 8 users validated with Lapis patterns –Toped users found twice as many of the typos compared to Lapis users –Topes were 50% more accurate than Lapis patterns –Toped gave significantly higher user satisfaction –(Comparison to an earlier regular expression study that had similar but not identical tasks: Toped users were faster and more accurate, but not a statistically significant difference) Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

26 Evaluating usability for data reformatting End users reformatting data with multi-format topes –Within-subjects lab study (latest version of Toped) –9 users reformatted spreadsheet data by creating & using topes; for comparison, they then did it manually –Effort of creating a tope “pays off” at only 47 strings (further reuse is essentially “free”) –Every participant strongly preferred using Toped instead of doing tasks manually Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

27 Evaluating tope recommendations Quickly recommend existing tope for data at hand –Supports keyword-based search + search-by-match (eg: topes that match “ ”) –Evaluated by searching through topes for the 32 most common data categories in EUSES spreadsheet corpus, using strings from corpus –High accuracy: Recall over 80% (result set size = 5) –Adequate speed: User is likely to have a few dozen topes on computer, taking under 1 sec to search Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

28 Conclusion: Topes improve data validation Validating with topes improves –Accuracy of validation –Consistency of data formatting –Reusability of validation code Primary contributions: –Support for ambiguous data categories –Support for reformatting values –Platform-independent, reusable validation Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

29 Future work: quality control Quality control (of topes) within topes repository –Indicators of tope reusability Eg: meaningful names given to parts in formats? Eg: plenty of test strings that match the tope? –Extension of work on identifying reusable web macros Quality control (by topes) of data exchange –Two modules (components/web services/…) may use the same kind of data, but require different formats. –Topes can automatically reformat strings on demand. –One step toward a larger goal… helping end users to create, share, and combine their code – ask for details! Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

30 Thank You… For this opportunity to present To NSF for funding Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

31 Professional programmers use lots of tricks to simplify validation code. Eg: njtransit.com Split inputs into many easy-to-validate fields. Who cares if the user has to type tabs now, or if he can’t just copy-paste into one field? Make users pick from drop-downs. Who cares if it’s faster for users to type “NJ” or “1/2007”? (Disclaimer: drop-downs sometimes are good!) I implemented this site in Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

32 Even with these tricks, writing validation is still very time-consuming. Overall, the site had over 1100 lines of JavaScript just for validation…. Plus equivalent server-side Java code (too bad code isn’t platform-independent) if (!rfcCheck (frm.primary .value)) return messageHelper(frm.primary , "Please enter a valid Primary address."); var atloc = if (atloc > 31 || atloc < frm.primary .value.length-33) return messageHelper(frm.primary , "Sorry. You may only enter 32 characters or less for your name\r\n”+ ”and 32 characters or less for your domain Introduction  Challenges  Topes  Tools  Evaluation  Conclusion

33 That was worst case. Best case: reusable regexps. Many IDEs allow the programmer to enter one regular expression for validating each input field. –Usually, this drastically reduces the amount of code, since most validation ain’t fancy. –So why don’t programmers validate most inputs? Introduction  Challenges  Topes  Tools  Evaluation  Conclusion