Code Generation Tips Tricks Pitfalls 5/4/2019 A.Symons CiTR Pty Ltd

Why? Speed up development Remove bugs Consistency Less flexibility DeltaX 8X Remove bugs Consistency Less flexibility DeltaX initially had 6 people doing 6 modules in 16 weeks. In the current DeltaX 2 people in 1 week developed a module. Any bugs found can be fixed in the generator and thus be removed from all of the generated code. All file names and function names can be made consistent. Also, variable names become consistent and easier to find - e.g., finding ifIndex in the MIB, source code and database. GUIs look the same. All attributes can be supported, and things like help screens produced easily. Less flexibility - Ties in with above - coders may want to capitialize differently, use ‘_’, drop characters. Also coders may not want to support all possible attributes / states, only those in the current test/design. 5/4/2019 A.Symons CiTR Pty Ltd

Why Not? May be difficult May not be appropriate Readability of generated code Debugability of generated code Less flexibility Really do want to have an initial prototype first. May have very difficult usage which is not consistent across input files. Readability - code can often be mangled when generated (E.g., ROSE) affects debugability. Flexibility - Can be hard to put in special case code. 5/4/2019 A.Symons CiTR Pty Ltd

Example - DeltaX What High level design Detailed design API Document SNMP MIBs &Database Access SNMP Agent functionality C++ SNMP & Corba Classes ... GUI Delta X is a Network management system - consists of Design Documents Api Documents ILOG GUI CORBA Config server Postgres SQL database SNMP Agent Hardware API 5/4/2019 A.Symons CiTR Pty Ltd

Example - DeltaX EMS EMS SNMP Agent API Document High Level Design MIB Master C Code Subagent Set CB Cache access API calls 5/4/2019 A.Symons CiTR Pty Ltd

Example - DeltaX NMS NMS - C++ Corba + Ilog GUI IDL Corba Impl DB schema & access SNMP access Ilog GUI GUI C++ 5/4/2019 A.Symons CiTR Pty Ltd

Example GUI 5/4/2019 A.Symons CiTR Pty Ltd

Lines 5/4/2019 A.Symons CiTR Pty Ltd NMS Total is the total NMS. Not much was generated initially. NMS New is the current phase of the NMS - really have been applying code generation 5/4/2019 A.Symons CiTR Pty Ltd

Percent 5/4/2019 A.Symons CiTR Pty Ltd Significant parts of the API document, HLD, EMS and current NMS are now generated Most of the current NMS is now able to be generated. 90-10% rule for NMS 5/4/2019 A.Symons CiTR Pty Ltd

Tips Metadata Utility printing executables Parsing executables Integrated executables Shell ‘here documents’ 5/4/2019 A.Symons CiTR Pty Ltd

Metadata Describes the data Want it ‘compiled in’ I.e., accessible in scripts Source code Database Metadata parser & input file Metadata is the heart of code generation. This allows you to be able to print out the textual names of the attributes, structures and their types. Do need to have a Metadata parser/compiler. This is not that difficult to implement in lex/yacc. If you are specifying the input file format (e.g., internal API documentation) then keep one eye on being able to parse the file format to extract the metadata. 5/4/2019 A.Symons CiTR Pty Ltd

Parsing MibMaster etc. Bulk Code Metadata No Changes Generic Difficult to debug/change MM parses SNMP MIB definitions in ASNv1 and generates huge amounts of code (some files > 60 M) This bulk code is so large that it becomes impractical to modify/debug, although it is possible. This is mainly intended as the basis for other applications. The parsers typically have printf type statements and themselves become difficult to follow. These do however produce the metadata used in our other applications. 5/4/2019 A.Symons CiTR Pty Ltd

Integrated Based on intermediate metadata Can do complex tasks - e.g., if then else Difficult to modify An example is the ilog generator Put complex tasks in here, and put the rest in scripts 5/4/2019 A.Symons CiTR Pty Ltd

Utilities PrintMibs Prints metadata Print 17 items per attribute Flexible - uses printf ‘%$’ Absolute key 5/4/2019 A.Symons CiTR Pty Ltd

Shell ‘here document’ Use existing example Replace specifics with $variables Enclose with cat << EOF … EOF Easy to read, write and debug New age of code generation Shell scripts can be written with all of the logic at the top and then the ‘here’ document is essentially the source code with the specific names replaced with shell variables. This produces a very readable code generator, which is also then easy to modify. Similarly the generated code is very readable and debugable. This is also quick to implement and also quick to modify/re-run. Parts of the script can also be extracted for easier debugging. 5/4/2019 A.Symons CiTR Pty Ltd

Metadata Applications E.g., GOA Array of attributes Switch statements/virtual functions Very good for ‘raw’ applications Low code base size Easy to add new ones dynamically Low code ‘*ability’ Difficult to add special cases Templates and deep OO analysis also fits in here. CM used a lot of configuration to support new snmp agents. The example of the GUI screen shows that it is not so easy to generate, either through scripts or through metadata a perfect implementation. I.e., extra information and logic is required for the perfect implementation, such as this table is an ATM link and many are required for an ATM PVC. Low readability, writeability, debugability. It is much easier to code with an explicit data structure, than with an array of attributes. Probably need a rule based system to add in special cases. 5/4/2019 A.Symons CiTR Pty Ltd

Tricks How to identify Don’t try to be optimal initially Doing the same thing over and over Use generation when possible. Only use generation when possible. Cut and Paste for small amounts. Don’t try to be optimal initially Need a lot of something - a lot of screens, database tables, documents … Use metadata based solutions/templates when you are doing exactly the same thing for each one - I.e., no modifications, no special cases. Use generation when doing similar things but some special cases. Use cut and paste when some can be generated but most is special case code. 90-10 % rule - generate the 90% case code - easy. Do the 10% difficult code by hand. 5/4/2019 A.Symons CiTR Pty Ltd

Pitfalls Modifying Code Code understandability Stub Files RCS #ifdef wrappers Derived classes Registered Callbacks Don’t generate as part of the normal build Code understandability Generated code is good if you are not modifying it. However, problems occur as soon as you do modify the code. Especially if multiple coders. Stub files for code you know you are going to have to hand code. You really should keep all of the changes in one file, preferably not a generated file. Patch shell scripts RCS all code that you do modify - so that you have a chance of getting it back. If all of the code is generated then you may not understand what is going on. Especially if you have to fix the code! Really do need good documentation. 5/4/2019 A.Symons CiTR Pty Ltd

Conclusions Generation can reduce time Use 90% generated 10% hand coded Use metadata based for ‘raw’ apps Shell scripting is fast and readable. Take care when modifying generated code. 5/4/2019 A.Symons CiTR Pty Ltd

References MM hptools/yacc PrintMibs mibmaster/008.Tools/print_mibs GOA mibmaster/007/src/monitor/goa gen nms nc867.15/007/src/gen_nms ems nc867.15/007/bin 5/4/2019 A.Symons CiTR Pty Ltd