1. Extensible syntax analyzers Pavel Egorov pe@skbkontur.ru

2. Problem statements L – some formal language; T : L  R – it’s translator; L ext  L – some extended language; T ext : L ext  R ext – it’s extended translator.  L T ext (  ) = T(  ) We want compatibility!

3. Disclaimer Consider syntax analyzers only. Fixed algorithm of syntax analysis. Extension due to modification of the grammar only.

4. What is syntax analyzer? SA converts a word to a syntax tree. What is syntax tree? – Root is labeled by the axiom of the grammar – Inner nodes are labeled by non-terminals – Leaf nodes are labeled by terminals or non- terminals or 

5. Classic parse tree Grammar: A  aBd B  b B  A Word: “aabdd” Derivation: A  aBd  aAd   aaBdd  aabdd A B A B b da da

6. Classic parse tree Is it good enough? G 1 : A  abc Extended G 2 : A  aBc B  b Word: “abc” A B b ca A bca They are different!

7. Classic parse tree Is it good enough? G 1 : A  a Extended G 2 : A  aE E   Word: “a” A E  a A a They are different!

8. Cancelled tree Driven by labeled grammar Cut off some inner nodes from parse tree Cut off  -leafs from parse tree Is much better for parser extension!

9. Cancelled tree example Labeled grammar: A  a 1 B 0 d 0 B 1 B  b 1 E 0 B  x 1 E   0 Symbols labeled by 0 are not present in tree  -leafs are always labeled by 0 A B b d a  A b a parse tree cancelled tree B x B x E

10. Intermediate results Labeled grammar Syntax analyzer driven by labeled grammar Canceled tree as a result of syntax analysis So how can we extend grammars without breaking compatibility? Compatibility:  L SA ext (  ) = SA (  )

11. Extending transformations of grammars f – extending grammar transformation G - grammar L(f(G))  L(G)

12. Extending transformations ADD-transform adds some fixed new production to a grammar. EXTRACT-transform: before:A   x  y  z after:A   x B 0  z B   y

13. Extending transformations AE *  all transforms, composed from arbitrary number of ADD- and EXTRACT-transforms.

14. Results

15. Compatibility 1.Any transform from AE* doesn’t break compatibility of corresponding syntax analyzers. 2.A grammar extended by AE* transform has the same production labeling as initial grammar has on the common productions. (AE* doesn’t change labeling of unchanged productions)

16. Independent transforms G G2G2 G1G1 G 12 ???   When it is possible? How can we do it?      AE * G 2  Domain(   ) ? G 1  Domain(   ) ?

17. Independent transforms What if G 1  Domain(  2 )? Sometimes we can solve this problem! A    1 : A   C  ; C    2 : A   B  ; B    2 /  1 : A   B  ; B   C Conflict! Decision!

18.  a  fixed transform  “fixed” , which can be applied after  Sometimes we can’t fix   A   A  X  ; X   A  Y; Y  Recursive definition of the function  for  and  from AE *. Sometimes  is not defined.

19. Independent transforms G G2G2 G1G1 G 12     /     /     /     ?   /     G 21 ?

20. Independent transforms G G2G2 G1G1 G 12     /     /     /     =   /    

21. Conclusion Syntax analyzers – driven by labeled grammars – cancelled trees as a result AE* - a way to extend grammars safelly  – a way to compose independent grammar transforms Order of application of the extending transforms doesn’t matter!

22. Thank you! Questions?