Documentation Generators: Internals of Doxygen John Tully

Motivation: Why document code? Required by company / administrators who examine code Required by company / administrators who examine code Typically in software engineering, code isn’t consistently worked on Typically in software engineering, code isn’t consistently worked on Co-workers (or “smart” users) look at source code for a better understanding of functionality Co-workers (or “smart” users) look at source code for a better understanding of functionality Code modified/built upon by others Code modified/built upon by others

Motivation: Why Automated? Docs much more likely to be up to date - developers only need to update comments in code (editing word docs/latex files isn’t enjoyable) Docs much more likely to be up to date - developers only need to update comments in code (editing word docs/latex files isn’t enjoyable) Reuse of comments = ½ the work Reuse of comments = ½ the work Auto formatting Auto formatting More importantly, automatic cross-referencing if the automated system is sophisticated More importantly, automatic cross-referencing if the automated system is sophisticated Huge advantage: what if recipients want different formats? Huge advantage: what if recipients want different formats? Management team (or dumb readers) Management team (or dumb readers) End users End users Software testers or other software developers Software testers or other software developers

Problem / Goals Create a simple, easy-to-use, portable, highly configurable automatic documentation generator for a variety of output formats Create a simple, easy-to-use, portable, highly configurable automatic documentation generator for a variety of output formats Real motivation: Creator (Dimitri Van Heesch) used Qt, an application development framework (popular for creating GUIs) Real motivation: Creator (Dimitri Van Heesch) used Qt, an application development framework (popular for creating GUIs) Wrote docs by hand to look exactly like Qt documentation; tried doc++ to do this automatically (not configurable enough) Wrote docs by hand to look exactly like Qt documentation; tried doc++ to do this automatically (not configurable enough) source: sourceforge

Other Doc Generators Several generators support more languages than Doxygen * Several generators support more languages than Doxygen * ROBODoc, TwinText, Natural Docs ROBODoc, TwinText, Natural Docs VB, Pascal,.NET, Perl, JavaScript, SQL VB, Pascal,.NET, Perl, JavaScript, SQL Many (including Doxygen) allow addition of new languages Many (including Doxygen) allow addition of new languages Discussion – ease use of these tools? Discussion – ease use of these tools? As far as output formats, Doxygen most versatile As far as output formats, Doxygen most versatile Also seems to be best for diagram generation Also seems to be best for diagram generation Dependency graphs, inheritance / collaboration diagrams Dependency graphs, inheritance / collaboration diagrams Certain tools may do some things that Doxygen can’t (control flow, data flow); but they’re language-specific Certain tools may do some things that Doxygen can’t (control flow, data flow); but they’re language-specific * Source for info on other generators: Wikipedia

Doxygen Information Flow Source Code Custom Stuff: - Headers - Footers - Images Doxywizard GUI Doxygen Doxyfile (Config file) Tag file Doxytag Program HTML Pages Man Pages RTF Files Latex Files XML ps or pdf MS-Word formatted dox xml parser (for custom output) external docs Source: Doxygen “Getting Started” Page

1. Documentation Source Code Custom Stuff: - Headers - Footers - Images Doxywizard GUI Doxygen Doxyfile (Config file) Tag file Doxytag Program HTML Pages Man Pages RTF Files Latex Files XML ps or pdf MS-Word formatted dox xml parser (for custom output) external docs

2. Configuration Source Code Custom Stuff: - Headers - Footers - Images Doxywizard GUI Doxygen Doxyfile (Config file) Tag file Doxytag Program HTML Pages Man Pages RTF Files Latex Files XML ps or pdf MS-Word formatted dox xml parser (for custom output) external docs

3. Parsing / Execution Source Code Custom Stuff: - Headers - Footers - Images Doxywizard GUI Doxygen Doxyfile (Config file) Tag file Doxytag Program HTML Pages Man Pages RTF Files Latex Files XML ps or pdf MS-Word formatted dox xml parser (for custom output) external docs

4. Output Options Source Code Custom Stuff: - Headers - Footers - Images Doxywizard GUI Doxygen Doxyfile (Config file) Tag file Doxytag Program HTML Pages Man Pages RTF Files Latex Files XML ps or pdf MS-Word formatted dox xml parser (for custom output) external docs

Doxygen Internals Doxygen Information Flow: high-level view of steps taken by the user to document code Doxygen Information Flow: high-level view of steps taken by the user to document code Doxygen Internals: overview of data flow as source files are parsed by Doxygen Doxygen Internals: overview of data flow as source files are parsed by Doxygen Closer look at Parsing / Execution Phase mentioned previously Closer look at Parsing / Execution Phase mentioned previously

Doxygen Internals: Data Flow Config parser: processes configuration file Config parser: processes configuration file Written in flex Written in flex Simple (since config options are pretty simple – just 5 types) Simple (since config options are pretty simple – just 5 types) C Preprocessor: fairly similar to standard preprocessor C Preprocessor: fairly similar to standard preprocessor Written in flex; uses yacc-based parser for expression evaluation Written in flex; uses yacc-based parser for expression evaluation Language parser: convert input buffer into a tree of entries Language parser: convert input buffer into a tree of entries Basically breaks up code into smaller modules that will be reorganized later Basically breaks up code into smaller modules that will be reorganized later Config Parser C Preprocessor Data Organizer Output Generators Source Parser Language Parser Documentation Parser Source: Doxygen“internals” page

Doxygen Internals: Data Flow Data organizer: Gather information from these entries Data organizer: Gather information from these entries Build dictionaries of classes, files, variables, functions, groups, etc. Build dictionaries of classes, files, variables, functions, groups, etc. Determine relationships between these entities Determine relationships between these entities Documentation parser: find comment blocks / tags in entities; feed results to output generators Documentation parser: find comment blocks / tags in entities; feed results to output generators Source parser (if source is documented): cross-referencing, syntax highlighting Source parser (if source is documented): cross-referencing, syntax highlighting Output generators: take data, which was found by language parser and organized/cross- referenced by data organizer, and generate output in specified format Output generators: take data, which was found by language parser and organized/cross- referenced by data organizer, and generate output in specified format Config Parser C Preprocessor Data Organizer Output Generators Source Parser Language Parser Documentation Parser

Debugging Doxygen For a thorough understanding of doxygen’s source code, understanding of flex important For a thorough understanding of doxygen’s source code, understanding of flex important Lexical analyzer: generates scanners (programs that recognize lexical patterns in text) Lexical analyzer: generates scanners (programs that recognize lexical patterns in text) Executable produced from scanners, which execute C code when patterns are found Executable produced from scanners, which execute C code when patterns are found Flex has debugging option to output matched rules when they’re found – easy to follow the steps doxygen is taking Flex has debugging option to output matched rules when they’re found – easy to follow the steps doxygen is taking On internals page, a tool (script) is available which turns on debugging for any flex file On internals page, a tool (script) is available which turns on debugging for any flex file

Limitations / Future Work Limitations / “wish list” of doxygen discussed during demo Limitations / “wish list” of doxygen discussed during demo More languages, more output formats, better template files More languages, more output formats, better template files Several improvements to internals also mentioned: Several improvements to internals also mentioned: Language Parser: Language Parser: One scanner / parser per language (currently one huge combined scanner) One scanner / parser per language (currently one huge combined scanner) Modulate parsing (documentation blocks vs code) Modulate parsing (documentation blocks vs code) Parse preprocessor input for extended documentation (i.e. #defines) Parse preprocessor input for extended documentation (i.e. #defines)

Limitations / Future Work Output generators – interesting future work Output generators – interesting future work Instead of using data structures (generated from the data organizer) to produce XML, use XML as an intermediate language Instead of using data structures (generated from the data organizer) to produce XML, use XML as an intermediate language More info could be extracted by various output generators – with more understandable IL, easy to create better end tools More info could be extracted by various output generators – with more understandable IL, easy to create better end tools Interactive source browsers Interactive source browsers Configurable class diagram generators Configurable class diagram generators Computing code metrics Computing code metrics

Discussion