Model Development Unit Strategy Directorate 1 Developing & Maintaining Dynamic Micro-Simulation Models at DWP Sally Edwards Simon Gault ESRC – 2 nd April 2009

2 Model Development Unit Strategy Directorate Developing Models at DWP 1.Overview 2.Objectives 3.Development Standards 4.Simplifying Complexity 5.Maintenance Protocol 6.What worked well and Lessons Learned

3 Model Development Unit Strategy Directorate Introductions Simon Gault  Statistician, with a strong background in Economics  Extensive experience in developing models  Manages the Forecasting Simulation models team Sally Edwards  Systems / Business Analyst  Predominantly worked in the private sector on large scale IT systems  Manage the Genesis Simulation Model Members of a the Model Development Unit (MDU) which has responsibility for Micro-Simulation Models at DWP MDU consists of 20+ people

4 Model Development Unit Strategy Directorate Micro-Simulation Models  Dynamic Micro-Simulation Models, based on our standard architecture, Genesis:  Pensim2 private & state pension income  Inform Integrated Forecasting model for working age benefit claimants  Individual Benefit Forecasting for RP, IB, DLA/AA  Employment Model, currently being validated  Static Micro-Simulation Models:  Policy Simulation Model (PSM)  Employment and Hours Model (EHM)

5 Model Development Unit Strategy Directorate Key factors taken into Consideration during Development Key factors considered when we designed our first dynamic micro-simulation model, Pensim2 - also valid for our other models:  Simplify the complexity of the model as much as possible, using a modular approach and allowing a simple prototype model to be developed quickly  Several developers work on each model at the same time  Large staff turnover  Models are intended for long term use (e.g. Pensim2 anticipated to have year life) so most of the people using/maintaining the models are not involved in their development  Clear documentation, processes, training and user guides are essential  Excel and SAS are the preferred tools, as both are well understood and extensively used among the Analysts at DWP

6 Model Development Unit Strategy Directorate Developing Models at DWP 1.Overview 2.Objectives for Pensim2 (first DMS model) 3.Development Standards 4.Simplifying Complexity 5.Maintenance Protocol 6.What worked well and Lessons Learned

7 Model Development Unit Strategy Directorate Original Objectives for Pensim2 Pensim2 Feasibility Study document (2002) – Objectives 1.Flexibility 8. Efficiency (in use of memory) 2.Robustness 9. Timeliness 3.Transparency 10. Done with own expertise 4.User-friendliness 11. Done with own resources 5.Speed 12. Ease of handover 6.Reliable Output 13. Ease of maintenance 7.Availability 14. Independence of base data (i.e. desktop model)

8 Model Development Unit Strategy Directorate Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives:

9 Model Development Unit Strategy Directorate Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives: Excel based front end – Pensim2  User friendly, transparent code in a standard format  Easy to understand and flexible, enabling analysts to change parameters & structure of model without understanding the underlying code.  Easy to maintain and handover  Allowed independence of base data

10 Model Development Unit Strategy Directorate Decision to Split the Model Impossible to satisfy all the objectives in a single model solution so decision was made to split the model into two separate parts, satisfying most of the objectives: : Excel based front end – Pensim2  User friendly, transparent code in a standard format  Easy to understand and flexible, enabling analysts to change parameters & structure of model without understanding the underlying code.  Easy to maintain and handover  Allowed independence of base data SAS code generator - Genesis  Developed with our own expertise and own resources  Robust generator, producing reliable outputs

11 Model Development Unit Strategy Directorate Basic structure of Genesis Dynamic Micro-Simulation Models at DWP

12 Model Development Unit Strategy Directorate Developing Models at DWP 1.Overview 2.Objectives 3.Development Standards 4.Simplifying Complexity 5.Maintenance Protocol 6.What worked well and Lessons Learned

13 Model Development Unit Strategy Directorate Development Standards Two separate development teams, with different development standards adopted Genesis Model Engine  Developed by IT staff  Used standard IT project development procedures Pensim2 Analytical team  Economists and Statisticians  Less structured approach to development

14 Model Development Unit Strategy Directorate Genesis Model Engine Development Standards Highly structured approach to the development  Project Management Protocol based on Prince2 methodology, although not all aspects were needed  Documentation – 2 types  Development documentation produced at each stage of project, that was signed off and used as input to the next stage  Key Project documentation (*) - maintained and kept up-to-date with each new release  Development Processes –simple process diagrams to provide clear guidelines, particularly useful for new staff  Detailed Requirements – explaining the full functionality of the model engine. Document translates Economist language into IT language (*)

15 Model Development Unit Strategy Directorate Genesis Model Engine Development Standards  Design Diagrams and documentation – used in development stage only  Change Control process –changes estimated and impacted before being accepted/rejected/on hold (*)  Problem Log process –problems recorded, prioritised, fixed/removed from the scope/user error (*)  Validation Process - detailed Test Plan based on Requirements document, used to ensure that each requirement was fully satisfied  Test Pack ensured existing functionality continued to work when changes were made to the model engine (*)  User Guide contains details of functionality written in simple format (*)  Training materials, inc presentations and training text for self-study (*)

16 Model Development Unit Strategy Directorate Pensim2 Model Development  Pensim2 Analytical development followed a less structured approach  Generally this was appropriate, as detailed requirements were not possible to establish up-front  Effort was predominantly spent on regression analysis  Analysis results reviewed to determine the key components to include and the most appropriate sources for the assumptions  Using the standard Genesis templates, the Pensim2 model was more easy to develop, rather than if a traditional DMS model had been built

17 Model Development Unit Strategy Directorate Developing Models at DWP 1.Overview 2.Objectives 3.Development Standards 4.Simplifying Complexity 5.Maintenance Protocol 6.What worked well and Lessons Learned

18 Model Development Unit Strategy Directorate Key Aim of Genesis Structure - Simplify Complexity No Hard Coding Rule which allows complete flexibility for: Structure of the simulation, i.e. order of events processed Data structures, i.e. data tables, data relationships and variables Parameters, e.g. indexes, rates, dates Regression Equations Selection filters Static SAS code, if required (e.g. addition of new cases to a model) Ability to add / remove modules easily Each model is defined in simple Excel spreadsheets, using standard templates, that users can easily understand and modify

19 Model Development Unit Strategy Directorate Key Aim of Genesis Structure - Simplify Complexity Data Access routines  Enable easy access to data so user does not have to write complex code  Default data access will process the current object in the current simulation year  Facility to process a variable belonging to a related object, e.g. partner PAR; Relationships are not hard-coded and new relationships can easily be defined  Data manipulation routines provide commonly required data facilities, e.g. “maximum value during past 10 years” = MA10  E.g. MA10_PAR_pa_salary returns the maximum value in the past 10 years of the salary variable for the individual’s partner, taken from the pa table Keep Equations simple  Equations, filters & process flows are defined in a simple, standard format, that is unambiguous. Hence the models are easy to maintain

20 Model Development Unit Strategy Directorate Developing Models at DWP 1.Overview 2.Objectives 3.Development Standards 4.Simplifying Complexity 5.Maintenance Protocol 6.What worked well and Lessons Learned

21 Model Development Unit Strategy Directorate Change Management  Change Management Process is followed for all models  Every change goes through the formal Change Control procedure, i.e. requirements documented, impact assessed, agreed by users & signed off  Change Register listing all changes, each with a separate reference number  For each change to the model, a separate Change Control form is produced  The model code contains the Change ref number, with a comment  Some changes are repeated regularly, e.g. assumptions, so clear documentation is helpful for the repeat change  Questions arise after a change has been implemented - these can be easily answered by referencing the Change Control documentation

22 Model Development Unit Strategy Directorate Problem Management  Problem Management Process is followed for all models  Every problem/error discovered in a model goes through the formal Problem Log procedure, i.e. requirements documented, impact is assessed, agreed by users & signed off  Problem Log Register, listing all problems identified, including those that turn out to be User Errors or misunderstandings; PL ref number assigned  For each problem recorded, a separate Problem Log form is produced  Users are notified of significant problems that affect the model results  Many of the problems are minor and are picked up by the development team – these are recorded formally, even if they do not impact the results

23 Model Development Unit Strategy Directorate User Model  Our models fall into 2 categories:  Maintained & owned by the Model Development Unit  Developed by Model Development Unit; handed over to User team, who then own and maintain the model  Models owned by the Model Development Unit have a User Group and/or a Steering Group consisting of representatives from each team using the model  The purpose of these groups is to agree the content, priority and timing of changes made to the model

24 Model Development Unit Strategy Directorate User Group Purpose Specifically for Pensim2, our largest model:  Change Requests and Problem Logs are reviewed by the User Group, where appropriate  A key user sponsor is assigned to each change. The sponsor ensures that the change is accurately specified & will sign-off testing  The User Group are responsible for signing off new versions of the model before they are released  The users also have a separate User Forum where they present analysis carried out on the model outputs and explain how the results have been used

25 Model Development Unit Strategy Directorate Release Management Again, specifically for Pensim2:  New Releases are implemented approximately every 6 to 9 months  Each Release includes one or two key components, e.g. Private Pension Reform, and a bundle of smaller changes and problem fixes  The timing and content of each Release are agreed with the User Group  When a new Release is built: Each change/fix is added to the model one at a time Hence each change is reviewed and signed off separately Enables us to fully understand and agree the effect of each change. Time consuming but gives confidence in the results  Clear audit trail  All previous versions of the model remain available

26 Model Development Unit Strategy Directorate Developing Models at DWP 1.Overview 2.Objectives 3.Development standards 4.Simplifying Complexity 5.Maintenance protocol 6.What worked well and Lessons Learned

27 Model Development Unit Strategy Directorate Original Objectives Satisfied Pensim2 Feasibility Study document (2002) – Objectives 1.Flexibility 8. Efficiency (in use of memory) 2.Robustness 9. Timeliness 3.Transparency 10. Done with own expertise 4.User-friendliness 11. Done with own resources 5.Speed 12. Ease of handover 6.Reliable Output 13. Ease of maintenance 7.Availability 14. Independence of base data (i.e. desktop model)

28 Model Development Unit Strategy Directorate Objectives Partially Satisfied Pensim2 Feasibility Study document (2002) – Objectives 1.Flexibility 8. Efficiency (in use of memory) 2.Robustness 9. Timeliness 3.Transparency 10. Done with own expertise 4.User-friendliness 11. Done with own resources 5.Speed 12. Ease of handover 6.Reliable Output 13. Ease of maintenance 7.Availability 14. Independence of base data (i.e. desktop model)

29 Model Development Unit Strategy Directorate Objectives that proved more tricky Pensim2 Feasibility Study document (2002) – Objectives 1.Flexibility 8. Efficiency (in use of memory) 2.Robustness 9. Timeliness 3.Transparency 10. Done with own expertise 4.User-friendliness 11. Done with own resources 5.Speed 12. Ease of handover 6.Reliable Output 13. Ease of maintenance 7.Availability 14. Independence of base data (i.e. desktop model)

30 Model Development Unit Strategy Directorate What went well  Generic framework has proved successful and easy to use  Genesis architecture has been remarkably robust  Modules have been reused/copied from one model to another  Quicker development of new dynamic micro-simulation models, compared with a traditional build  Automated documentation facility and error checking produced using VBA, listing where variables are used and providing an overview of each module, to ensure consistency  Lessons learnt from Pensim2 development were used when developing subsequent models

31 Model Development Unit Strategy Directorate Lessons Learned Key lessons learnt from Pensim2 development:  Tried to make the model too complicated – particularly the analytical side  We should have developed a set of simple modules for a Phase 1 delivery, then added complexity to the more important modules for Phase 2  The structure of some of our modules was more complicated than the data would support, e.g. Labour Market process  Insufficient documentation produced by the Analytical team during the development (e.g. Base Data Creation)  Genesis Engine code is difficult to understand and hence modify, although it is robust and does not require much modification  Some Genesis Engine functionality was over-ambitious and subsequently dropped (e.g. Alignment of polychotomous variables)

32 Model Development Unit Strategy Directorate Any questions?