1. © 2006 by «Author»; made available under the EPL v1.0 | Date | Other Information, if necessary Doug Schaefer CDT DOM What is it? What should it be?

2. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 2 Requirements  Provide information about the source code in a project  Search, content assist, refactoring, CModel, …  Accuracy  What are the acccuracy requirements? 100%, 0%?  Performance  Ideally user should never notice parser running  At least don’t disrupt other workflows  Multi-language  C, C++, others?

3. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 3 Components of DOM  AST  Abstract Syntax Tree based on grammar  Bindings  Semantic link between names in AST  Types  Type information for Bindings that have types  Locations  Text locations of AST Nodes  Index  On disk cache of names, bindings, and types  Scanner/Parser  The creator

4. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 4 AST  Structure of AST generally follows grammar as laid out in parser  Makes it easier to create.  Key elements – IASTTranslationUnit, IASTName  Minimal elements  IASTTranslationUnit is entry point  IASTName represents pathway to Bindings and Types  IASTNode is root node type  Parent/Child hierarchy of AST  Map to Locations

5. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 5 AST Continued  ASTVisitor  Walks the tree  Static members to pick options  Separate visit members for common member types  Common AST node types prefixed with IAST  Maybe too C centric at the moment  Languages extends the interfaces  ICPPAST, ICAST  Starting with IASTTranslationUnit

6. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 6 Locations  Life in preprocessor land   nodes do not necessarily map directly to source  Maintains a map of text expansions including includes, macros  AST Nodes store the ppp offset into the translation unit  ppp = post pre-processor  IASTNode.getNodeLocations() returns where the node is in the map.  IASTNode.getFileLocation() gives the file location where the text was generated.  Will return header file of macro call

7. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 7 Bindings (IBinding)  “Bind” references and declarations  Definition is C/C++ specialization of declaration  IASTName models identifiers used in bindings  IASTName.resolveBinding() to find Binding  To find refs and decls for a binding, need context  IASTTranslationUnit  Translation unit wide, searches DOM  IPDOMResolver  Project wide, searches PDOM

8. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 8 Types (IType)  Bindings often have types  E.g. variables have types  IVariable.getType()  Types are not always Bindings  Sometimes they are (e.g. ICPPClassType)  Sometimes they are not (e.g. IBasicType)  Expressions also have types  IASTExpression.getExpressionType()

9. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 9 Scanner (IScanner)  Tokenizes translation units  IToken IScanner.nextToken()  Handles preprocessor on the fly  Creates the location map  IASTPreprocessorFunctionStyleMacroDefinition  Built for speed  Minimize creation of String objects  Not very reusable for other languages  Need to factor out preprocessor from tokenizer

10. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 10 Parsers (ISourceCodeParser)  One parser per language  One per variant too?  Parser reuse?  Create IASTTranslationUnit for a ITranslationUnit  ILanguage is the interface of choice starting in 3.1  ITranslationUnit.getLanguage()  ILanguage.getASTTranslationUnit()  Also used to get an ASTCompletionNode for content assist  Returns all names that fit at a given offset in a given working copy  Old parser, org.eclipse.cdt.core.parser, is deprecated

11. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 11 Code Reader Factory  Provides flexible mechanism for finding include files  Hides details of resource management system  Does not hide details of include path  Some modes provide caching of file buffers so that files only need to be read once.

12. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 12 GNU C and C++ Parsers  Recursive Descent with backtracking, * lookahead  i.e. LL(*) – give or take  Error recovery attempts to march ahead to next sane point in source and continue.  Does not use semantic information to resolve ambiguities  Stick “ambiguity nodes” in AST where this occurs  E.g. IASTAmbiguousStatement  Resolved at binding resolution time  Binding resolution slow, only resolved when someone cares  Don’t have a good extensibility story, yet…

13. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 13 Index (IPDOM)  PDOM persists parts of AST, Bindings, and Types  CDT 3.1 was ad-hoc  Enough for Search to work in most cases  Binary flat file that is memory mapped (Database class)  Variable size records (16 byte blocks)  Divided into 16K chunks, mapped into memory separately  Utility records to organize collections  Linked List  B-Tree

14. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 14 Indexers  CDT 3.1 only has Full and Fast (or none)  Ctags has been removed (resource/technical)  Both use DOM as starting point  Walk the AST finding IASTNames and resolve their bindings  If not already, create PDOM object for Binding  If not already, create PDOM object file file containing name  Create PDOM object for Name and hook up to Binding and File  For incremental reindexing, first clear out all names for file  Does not clear out bindings, yet…  Reindex command available on project

15. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 15 Fast Indexer  Tells parser to skip over header files that are already in PDOM  PDOMCodeReaderFactory  Tells parser to look up bindings in the PDOM  when they can not be found in the DOM proper  These bindings participate in binding resolution algorithm  Without changes required to the algorithm (hidden)  Thus, the PDOM objects must implement IBinding interfaces  And, thusly, the IType interfaces  Much testing and improvements to algorithm needed to ensure this works 100% (although works most of the time now)

16. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 16 Searching PDOM  Interfaces currently in IPDOM  Use regex’s (Pattern) in findBindings()  Use visitor (IPDOMVisitor) in accept()  Cast to PDOM and start walking yourself at the PDOMLinkages  PDOM.getLinkages()  Linkage represents symbols that can be linked together  i.e. language  Make sure you acquireReadLock() first to avoid running into indexer.

17. Presentation Title | Presentation Subtitle | © 2006 by «Author»; made available under the EPL v1.0 17 Putting it all together  For example, Open Definition (OpenDefinitionAction in search)  Parse to get IASTTranslationUnit  AST_USE_INDEX | AST_SKIP_ALL_HEADERS mode  getSelectedNames to find names under selection  For reach name, resolveBinding  Returns PDOM bindings if necessary  Ask AST to get definitions for Binding  If it can’t find any, searches PDOM  Open editor on first name found  Using names file location