1. Understanding SVG Diagrams Interim Presentation Vivek Ramakrishna Honours Program, Computer Science vivekr@csse.monash.edu.au

2. Motivation Vast amounts of information stored in diagrams Information not accessible to all users –Blind people –Search engine agents Should be able to retrieve information in a diagram, given a query

3. Stages in diagram understanding 3. Query Answering System 1. Interpreter Context Kind of diagram Query Answer (in context) Diagrams (SVG) 2. Semantics (description)

4. Diagram Representations and Types Rasterised versus vector graphics Scalable Vector Graphics (SVG) –Vectors have types and attributes Project will investigate bar charts, organisation hierarchies and flow charts in SVG format.

5. A simple bar chart in SVG

6. Stages in diagram understanding 3. Query Answering System 1. Interpreter Context Kind of diagram Query Answer (in context) Diagrams (SVG) 2. Semantics (description) Interpreter: Understands the meaning of the diagram and produces a useful description

7. Constraint Multiset Grammars In visual domain, spatial relationships more important than sequential. P ::= P 1, …, P n where C Non terminal symbol P can be recognised by multiset of symbols P 1,…, P n whenever the attributes of the symbol satisfy the constraints C.

8. CMG for a bar “A bar comprises of a rectangle and a text label. The label is aligned centrally below the bar”. B:bar() ::= R:rectangle, T:text where ( R.mid == T.mid && R.start > T.start ) We will use CMGs to produce grammars to interpret our diagrams.

9. CIDER Toolkit developed at Monash by Tony Jansen. Interprets diagrams based on CMG specifications. We will use CIDER’s powerful compiler to parse SVG to recognise diagram tokens. Bar Chart BarsAxes Bar Rectangle X axisY axis Text Line

10. Stages in diagram understanding 3. Query Answering System 1. Interpreter Context Kind of diagram Query Answer (in context) Diagrams (SVG) 2. Semantics (description) Semantics: The key issue when extracting descriptions is the format to specify them.

11. RDF: A Description Logic Resource Description Framework Can express knowledge of a domain by first defining its terminology. Equipped with formal, logic based semantics. Allows complex relationships to be inferred despite not having been explicitly stated –In an organisation chart, can infer all superiors of an employee, without stating each relationship Disadvantage: DLs can’t handle “fuzzy” predicates

12. Stages in diagram understanding 3. Query Answering System 1. Interpreter Context Kind of diagram Query Answer (in context) Diagrams (SVG) 2. Semantics (description) Query Interface: Communicate question to system and answer to user. Different requirements for each user. We will use existing interfaces for our trials.

13. Plan Considering SVGs of bar charts, organisational hierarchies and flow charts. CIDER used to interpret CMG for each diagram. Descriptions generated in RDF. Evaluation –Get corpus of sample diagrams –Evaluate with corpus of sample queries Limitations –Grammars are inherently brittle –Description Logics are rigid. –We wish to investigate when and why failure occurs

14. Questions?