1. © Copyright 2010 Hewlett-Packard Development Company, L.P. 1 David Lehavi HP Labs Israel A NEW PARSING LANGUAGE FOR GUI AND VISUALLY STRUCTURED DOCUMENTS

2. © Copyright 2010 Hewlett-Packard Development Company, L.P. 2 UNIVERSAL INTERFACE FOR ALL GRAPHICAL APPLICATIONS STANDARD APPROACH: USE THE DOM AND IF THERE IS NO DOM, OR A HYBRID ENVIRONMENT ? WHY BOTHER ? –New GUI for legacy apps (additional functionality, hiding sensitive data). –Software testing (record and replay). –Accessibility (speech activated apps). Mobile Devices Web 2.0 (Flash, fragmented toolkit environment) Hybrid environments DOM inspector Object type Set/Get properties

3. © Copyright 2010 Hewlett-Packard Development Company, L.P. 3 What images do we need to understand ? VISUAL LANGUAGES –A two dimensional pixel word: bit map –A two dimensional picture word (constructed from graphical tokens) –Formal presentation: A (BC) –We only parse objects which are cut by lines. –Less restrictive than it seems at first: we may generalize and parse objects which are cut by curves (overcome the X) C B A

4. © Copyright 2010 Hewlett-Packard Development Company, L.P. 4 INTERMEZZO: USING LANGUAGE CONSTRUCTS Following Ken Thompsons work on regular expressions Universal machine Compiler Visual lexer bytecode Language definition requests tokens characters

5. © Copyright 2010 Hewlett-Packard Development Company, L.P. 5 Finding useful language constructs CHALLENGES IN GUI PARSING Expressability Regular languages are too weak to describe recursive structures. Decidability & performance Context free languages are too strong – they are undecidable. Ease of maintenance: Many GUIs, and constantly changing. Robust to lexing noise: Input may originate from screenshot analysis.

6. © Copyright 2010 Hewlett-Packard Development Company, L.P. 6 RADIO-BUTTON-SET EXAMPLE A Naïve representation: (Radio Text)* Problems: alignment, distances. RTitled_E = [ X C 0..50 Text ] RTitled_M = [ X C 0..50 Text L L 0..50 X C 0..50 Text ] RBS := V{RTitled_M *RTitled_E }

7. © Copyright 2010 Hewlett-Packard Development Company, L.P. 7 EBNF FOR REGULAR EXPRESSION VPL = REGEX + FUNCTION CALLS AND DEFINITIONS ADDING DISTANCES AND ALIGNMENTS USING A VISIBLY PUSHDOWN META LANGUAGE = | = "|" = | = = | = "*" = | ="(" ") =[V>]"(" ")" = standard = " ">" = comma separated = | | = " ">=" ( | | ) = comma separated ="Object" = | | | = ".." = [TBC] ( | | ) = ( | | ) ? = [RLC][RLC]? = "[" ?"]"

8. © Copyright 2010 Hewlett-Packard Development Company, L.P. 8 LANGUAGE & COMPILATION - EXAMPLE RTitled_M = [ Radio C 0..50 Text L L 0..50 Radio C 0..50 Text ] RTitled_E = [ Radio C 0..50 Text ] RBS := V{RTitled_M *RTitled_E } Each node is a function Concat Kleene-* Col Row Radio Text

9. © Copyright 2010 Hewlett-Packard Development Company, L.P. 9 RUNNING THE VPL CODE - EXAMPLE Line Concat Kleene- * Col Radio Text Universal VPL machine RBS

10. © Copyright 2010 Hewlett-Packard Development Company, L.P. 10 GLOBAL ROBUSTNESS TO LOCAL AMBIGUITIES Visual lexer returns atoms. Lexer assigns likelihood to any pair (atom, bounding box). We use conditional likelihood to avoid consistent errors. A compound object has heuristic likelihood VPL graph vertices are no longer functions, but co-routines (user space threads). sit on (heuristic based) priority queue, and paused when their priority is low. can be forked when they get multiple return values. 50 % LO, 50 % scroller

11. © Copyright 2010 Hewlett-Packard Development Company, L.P. 11 QUESTIONS ?