1. US Army Corps of Engineers BUILDING STRONG ® Model Certification Jodi Staebell Operational Director, Ecosystem Restoration Planning Center of Expertise Rock Island, IL August 7, 2009

2. BUILDING STRONG ® Model Certification  Process to review, improve, validate analytical tools and models  Toolbox - Ensure high quality methods and tools available to enable informed decisions  Technical quality ► Theory ► Computational correctness  System Quality  Usability

3. BUILDING STRONG ®  Information Quality Act (PL106-544)  OMB Information Quality Bulletin for Peer Review  Report of the Planning Models Improvement Task Force  Engineer Circular 1105-2-407  Updated EC under development  Protocols for Certification of Planning Models  HQ Memo Policy Guidance on Certification of Ecosystem Output Models, Aug 08 Model Certification References

4. BUILDING STRONG ®  Definition - any model or analytical tool used to ► Define problems and opportunities ► Formulate alternatives ► Evaluate effects ► Support decision-making  Planning models, not engineering models  Review is cost-shared  In-house or contracted Model Certification Basics

5. BUILDING STRONG ® Stages of Model Development  Requirements stage  Development stage  External Testing  Implementation Model categories  Corporate  Regional/local  Commercial off-the-shelf  Models developed by others Model Basics

6. BUILDING STRONG ®  Provided by model proponent  Documentation includes ► Background ► Theory, assumptions, analytical requirements, formulas ► Software/hardware, testing/validation process, ► Availability of input data, usefulness to support project analysis, tech support, training  Software/spreadsheets also provided Model Documentation

7. BUILDING STRONG ®  Planner/Formulator  Functional Field Expert(s) (internal and/or external)  Software programmer/spreadsheet auditor Model Review Team

8. BUILDING STRONG ® Ecosystem Output Models Flowchart to Determine Level of Review IEPR required? Single use of multiple use model? Detailed model review is required. Yes Goal to recommend model for certification/ approval for use on future projects Goal to approve application of model on specific project Multiple Use Single Use Model Review $60-120k 7-10 months Single use or multiple use model? No Single Use ATR Team staffed and charged to conduct model review, supplement ATR team with outside experts, as appropriate Detailed model review is required. Multiple Use Model certified/ approved for use? No Yes ATR Team reviews model application Approved models include IWR Planning Suite and USFWS Published HSI ModelsUSFWS Published HSI Models

9. BUILDING STRONG ®  Develop Model Certification Plan  Assemble Review Team (In-house/Contract)  Charge to Reviewers  Kick-off Meeting  Conduct Review  Draft Model Review Report  Meeting to discuss findings  Final Model Review Report  Revise model, as needed (lather, rinse, repeat!)  PCX recommend that HQ certify model Model Certification Process

10. BUILDING STRONG ® District request PCX Model Review Model Review Process PCX assign Model Review Manager Model Review Plan approved by HQ Model Review Contract Executed Model Review Conducted Final Review Report submitted Model Review Plan Scope/Cost/Schedule/Method Scope of Work Independent Government Estimate Contractor Cost proposal Initial Findings Meeting held Draft Review Report submitted Draft Model Review Report Revised Model Documentation Final Model Review Report Model and model documentation revised as needed District MSC PCX Headquarters Review Group Recommend model for certification/approval for use? Model Documentation Published reports Table 2 information Model software or spreadsheet

11. BUILDING STRONG ® Model Review Process (continued) PCX recommend approval to HQ PCX posts approved model and documentation in PCX Toolbox HQ certify model/appro ve for use? Yes Model and model documentation revised as needed PCX memo describing model limitations submitted to MSC Recommend model for certification/approval for use? Yes No or District MSC PCX Headquarters Review Group

12. BUILDING STRONG ® Schedule  Start SOW to Notice to Proceed – 8-12 weeks  NTP to Final Model Review Report – 15-18 weeks  Revise model – depends on PDT  PCX recommendation to HQ – 4 weeks  HQ review and certify - ? Timing  Requirements and/or development stage  Identify models at (prior to) Feasibility Scoping Meeting  Initiate model review prior to Alternative Formulations Briefing Scheduling and Timing

13. BUILDING STRONG ®  ATR – 3 reviewers extra time to review model  In-house - $30-50k  Contract - $80-140k  PCX labor ~$10-15k  Model proponent labor ► Prepare model documentation ► Assist in Model Cert contract ► Revise model and documentation Model Certification Cost

14. BUILDING STRONG ® 3 National models certified  IWR Planning Suite  Beach-FX  HEC-FDA All published USFWS Habitat Suitability Index models approved for use Many reviews of single use models Model Certification Experience

15. BUILDING STRONG ® ECO-PCX Model Reviews  Island Community Index Model, NAB  3 Everglades models  Upper Miss Fish Passage Effectiveness Index  Wetlands Value Assessment, MVN  Habitat Evaluation and Assessment Tools (HEAT) ERDC  Floodplain forest and wet coastal prairie community models, SWG  Des Plaines Community HSI and HGM, LRC  Arid Streams HGM, SPL  4 Ecological Models, MVM

16. BUILDING STRONG ® Available Models  Ecosystem Restoration Gateway, Tools Tab  http://cw-environment.usace.army.mil/tools.cfm?CoP=Restore http://cw-environment.usace.army.mil/tools.cfm?CoP=Restore

17. BUILDING STRONG ®

18.  Identify models early  Prepare model documentation  Test/validate model  Check software  Identify expertise needed  Be open to process  Pool funds for multiple-use models How can the model proponent help?

19. BUILDING STRONG ® Questions?Questions?