Critical Needs ChapterS webinar Presentation for TSSM Stakeholders and FHWA 4th in the series of 6 06 December 2017 George List, Shannon Warchol, NC State David Hale, Leidos Brandon Nevers, Kittelson Rob Bertini, USF

Deliverables Schedule Task Deliverable Due Date 4 Final Annotated Outline 04/21/2017 5 Draft Critical Need Chapters 11/21/2017 Stakeholder Webinar #4 12/21/2017 Stakeholder Webinar Meeting Minutes 01/06/2018 Final Critical Need Chapters 01/21/2018 6 Draft Intermediate Need Chapters 03/21/2018 Stakeholder Webinar #5 04/21/2018 05/06/2018 Final Intermediate Need Chapters 05/21/2018 7 TRB Annual Meeting Materials December 2017 and December 2018 Meeting Minutes from TRB Annual Meeting 4 weeks after TRB Annual Meeting Stakeholder Webinar #6 -- PFS/Task Force Review Webinar of TRB Annual Materials 07/21/2018 08/06/2018 Draft Volume V 08/21/2018 Final Volume V 09/21/2018 8 Problem Statements 10/21/2018 Government Review Period Four months added to ensure document clears 508 compliance

Summary of Feedback http://tssm.kaiproject.com/ Chapter 5: 115 comments Chapter 6: 120 comments Chapter 7: 106 comments Chapter 8: 191 comments Chapter 9: 66 comments

Overarching Feedback (all chapters) Too technical, too academic Need example problems, case studies Too product-specific in some cases… …too product-agnostic in other cases Prefer step-by-step guidance Too focused on microsimulation

Chapter 5 (Defining the Problem) Chapter Themes: Bounding the problem wisely is key to successful simulations Performance measures are needed to: Evaluate design and operation alternatives Determine if you have a good simulation model Simulation is often a key part of the solution to many analysis problems, but it is not the solution to all problems

Chapter 5 (Defining the Problem) Selecting performance measures Alternatives analysis Calibration Setting study bounds 1. Initial ID based on existing conditions 2. Review for future conditions 3. Continuing check of simulation results Scoping the role of simulation

Chapter 5 (Defining the Problem) Major Comments Received Differences in terminology and content between chapters Refer to overall technical editor to resolve conflicts/redundancies in content. FHWA reliability calibration method is not clearly described in Chapter 5. Refer reader to Chapter 8 for more info on calibration using FHWA method. FHWA reliability calibration method is not cost-effective for most users. Highlight method is to be used only if simulating reliability. Need more material on meso and macro simulation Avoided use of “micro” where guidance applies to all levels. Add bullets on meso simulation DTA Discussion not clear Clarified distinctions between DTA and mesoscopic simulation modeling

Chapter 6 (Data) INTRODUCTION DATA TYPES AND SOURCES Geometric Data | Traffic Count Data | Traffic Signal Ops Data | Calibration Data | Safety Data First-Gen Probe Data | NPMRDS Data | Big Data Sources DATA TYPES AND SOURCES Signalized Intersections/Arterials | Transit Priority Lanes | Roundabouts | Non-traditional intersections | Ped-Bike Lanes | Freeways/Interchanges | Two-lane highways | Multilane highways | Work Zones | Toll Plazas | Managed Lanes/Ramp Metering | Safety Analyses DATA REQUIREMENTS (ANALYSIS TYPES) Macroscopic | Mesoscopic | Microscopic DATA REQUIREMENTS (MODEL TYPES) Data Comprehensiveness and Reliability | Data Accuracy | Data Fusion and Aggregation | Data Quality Control DATA CHALLENGES Freeway Corridor | Freeway Interchange | Highway Corridor | Reliability | Sampling | Simulation Runs | Data Measurements with NGSIM SAMPLE DATA CASE STUDIES/ APPENDICES

Some Excerpts… Connected Vehicle Data https://ops.fhwa.dot.gov/publications/fhwahop16072/index.htm https://ops.fhwa.dot.gov/publications/fhwahop16072/index.htm

Reviewer Comments Summary No of reviewers: 8 Total comments received: 120 Responses to comments up on the Kittelson project website Chapter updates based on comments and discussions on-going

Chapter 7 (Creating the Model) Fill time, equilibrium, spatial-temporal limits Network development Geometric data Control data Traffic data Operating condition data Typical calibration data

Chapter 7 Material One-page tech briefs with visuals Fundamentals (e.g., link, node, equilibrium) Approaches to data entry (e.g., link-connector, demands vs. volumes, O-D flows, conditional turns) Concepts that overlap with other chapters

Chapter 7 Feedback Quality of diagrams and images Tool specific vs. tool agnostic Step-by-step instructions Overlapping material Add new material: Volume balancing, global vs. local parameters Optimize discussion of: Volumes vs. demands, equilibrium, editing “raw” data, Monte-Carlo analysis, diagonal movements, results-oriented analysis

Chapter 8 (VCV) Introduction Performance metrics Verification Basic tenets of the VCV process Performance metrics Options for micro, meso, and macroscopic models Verification Important considerations in checking the simulation software’s capabilities and the completeness of the input data Calibration Perspectives to consider, sequence options, assessment metrics Validation Checking robustness, assessing limitations Step-by-Step Process Recap previous material in a “do this” fashion Summary

Chapter 8 Focus (VCV) Verification Calibration Validation Software logic reflects system behavior (software developers) Input data correctly describe the system (users) At the end: logic is okay and the input data are nominally clean. Calibration Software parameters are tuned to predict observed behavior Input and output for site-specific and global system performance metrics At the end: model parameters allow the observed performance to be matched Validation Check the robustness of the model’s predictive capability Use data for scenarios not employed in calibration At the end: model robustness and limitations have been assessed.

Chapter Highlights

Chapter Highlights (2)

Chapter 8 Comments More examples Be more mindful of the audience – writing style Move as much of the dense technical material as possible to appendices Break down the introduction into smaller sections Place more stress on macro- and meso-scale modeling Clarify why certain practices are deemed important Ex-situ tests, for example, and hypothetical facility tests OD calibration/adjustment Why the ideas of warm-up, operating, and cool-down periods may be problematic

Chapter 9 (Processing, Analysis, and Presentation) Running simulation models Run times, random seeds, keeping track of model runs Data processing Output methods, data types, aggregation issues, data synthesis Analysis Definitions, statistically understanding the results, interpretation Presentation Multiple metrics, tables, charts, figures, visualization options, visualization tools, written documentation

Chapter 9 Guiding Principles Short paragraphs Heavy use of figures/tables/examples Discrepancy in comments between usefulness of seeing “staged” figures versus screenshots Internal & external links Statistics from the ground up

Chapter 9: Comments Formatting / Links to Tables & Figures October review was mid-process; formatting will be standardized for the draft review Adjustments & Additions to Examples Will be incorporated as able; figures cleaned up by editor Addition of other procedure options Will incorporate as possible while maintaining impartial to software Further explanation of statistical assumptions Incorporate while maintaining usability

Timeline Review Today July 10th ~1/3rd checkpoint Traffic analysis PFS TRB Task Force August 15th ITE SimCap Today December 6th Webinar #4 Stakeholder webinar #3 (annotated outline) April 21st Webinar #5 Start date Jan 2017 Jan 2018 Jan 2019 January 7-11 TRB Task force meeting Today Interviewed 16 PFS states TRB Mid-Year HCQSC (Sturrock) March 21st Draft intermediate need chapters November 21st Draft critical need chapters January 21st Final critical need chapters

Activities at TRB Sunday Monday Tuesday Workshop 130: Data-Driven Simulation 9:00 am – 12:00 pm Report from TSSM team Chapter Liaisons 1:00 pm – 6:00 pm One hour per chapter - With TSSM team SimSub 7:30 pm – 9:30 pm Report from TSSM team Task Force 8:00 am – 12:00 pm Brief report from liaisons Report from TSSM team Also, members of the TSSM project team will be attending: Simulation subcommittee meeting, HCQS Simulation subcommittee meeting, Freeway Ops Simulation subcommittee meeting, Traffic Signals

Questions and Answers Thank You