1. FSUTMS Calibration and Validation Standards presented to MTF Model Advancement Committee presented by Robert Schiffer, AICP Cambridge Systematics, Inc. November 10, 2009

2. 1 Presentation Overview Background Literature Review Recommended Calibration and Validation Guidelines and Standards LRTP Models with Transit Other Model Applications Calibration and Validation Best Practices Guidelines for Model Application Guidelines and Standards in Practice

3. 2 Background Follow-up to Phase I Study on model parameters Phase I Final report is available for downloading at −http://www.fsutmsonline.net/images/uploads/mtf- files/FSUTMS-Cube_Parameters.pdf Phase II Study on calibration standards included four subtasks Literature Review Model Calibration/Validation Guidelines and Standards Best Practices for Model Calibration/Validation Documentation – 1) Calibration and Validation Standards; 2) Best Practices for Model Validation; 3) Guidelines for Model Application Update to presentation at last MTF meeting in 2007

4. 3 Background (continued) Calibration versus validation −Calibration – process where models are adjusted to simulate or match observed travel behavior in the study area −Validation – procedure used to adjust models to simulate base-year traffic counts and transit ridership figures Standards versus guidelines/benchmarks −Standards – desirable accuracy levels for comparing estimated versus observed metrics −Benchmarks – documented statistical ranges from literature review, model outputs, NHTS, etc. Purposes of validation process −Level of comfort to planners, agency staff, and elected officials −Evidence that model is accurate enough for specific application −Accounts for errors in observed comparative data

5. 4 Literature Review 60+ documents reviewed Specific models −Technical reports −Model outputs Reference reports −Federal agencies/TMIP −State DOT guidelines and standards Prepared ranges of acceptability StatisticStandardBenchmarkDocument(s) Cited Population/Employment Ratio40-60%Iowa DOT Peer Review (39) Person Trips/Person3.64 – 3.87Validation and Reasonableness (14) Person Trips/Person (Urban)2.54 University of Wisconsin (16): Kentucky Statewide Model/NPTS Person Trips/Person (Rural)2.57 University of Wisconsin (16): Kentucky Statewide Model/NPTS Person Trips/HH8.5 – 10.5University of Tennessee (59) Person Trips/HH6.8 – 12.4Validation and Reasonableness (14), NCHRP 365 (15) Person Trips/DU14.1/14.5/11.8/7.6 Calibration and Adjustment 6) – population sizes: 50-100/100- 250/250-750/750k+ Person Trips/DU9.2/9.0/8.6/8.5 NCHRP 365 (15) – population sizes: 50k-200k/200k- 500k/500k-1M/1M+ Vehicle Trips/DU9.15VTRC (29) Resident/Commercial Neighborhood Trips 78.5%/21.5%VTRC (29) Person Trips/Employee 1.29 – 1.40Validation and Reasonableness (14) TAZs/Population 1 TAZ/1k PopulationIowa DOT Peer Review (39) Person Trips/TAZ 25k or lessIowa DOT Peer Review (39) Percent Trips by Purpose – HBW*18% – 27%University of Tennessee (59) Percent Trips by Purpose – HBNW47% – 54%University of Tennessee (59) Percent Trips by Purpose – NHB22% – 31%University of Tennessee (59) Percent Trips by Purpose – HBW*17% – 23%Validation and Reasonableness (14), NCHRP 365 (15) Percent Trips by Purpose – HBNW52% – 60%Validation and Reasonableness (14), NCHRP 365 (15) Percent Trips by Purpose – NHB 23% – 25%Validation and Reasonableness (14), NCHRP 365 (15) Unbalanced Attractions/Productions 0.90-1.10 Validation and Reasonableness (14) Checklist of Available Validation Standards from Literature – Trip Generation

6. 5 Recommended Guidelines and Standards Checking Input Data Socioeconomic data Visual comparisons Statistical comparisons Regionwide comparisons (below) −Persons per DU (or HH) −Employment/ population ratio −Autos/DU (or HH)County Census Data NERPM Data 20002003 Percent Difference 20002005 Duval779,618817,4805%762,674810,4936% Clay141,671157,50211%139,036167,02020% Nassau57,90361,6256%56,89764,69514% St Johns 124,458142,86915%120,738150,08424% Total1,103,6501,179,4767%1,079,3451,192,29210% StatisticBenchmarks/SettingsLowHigh Regionwide Persons/DU (or HH) 2.02.7 Regionwide Employment/Population Ratio 0.450.75 Regionwide Autos/DU (or HH) 1.752.10 Approximate Population/TAZ N/A3,000

7. 6 Recommended Guidelines and Standards Checking Input Data (continued) Highway network data Transit network data Highway and transit speed data Logical hierarchy Balance highway and transit Terminal times Logical hierarchy Phase I Report

8. 7 Recommended Guidelines and Standards Trip Generation Aggregate trip rates Person trips/TAZ Person trips/person Person trips/DU (or HH) HBW person trips/employee Total unbalanced attractions versus productions by purpose Preferred +/-10%; acceptable in some instances +/-50% Percent external-external trips by zone/station Great variation expected (4-21 percent range documented) Statistic Benchmarks a LowHigh Person Trips/TAZ N/A15,000 Person Trips/Person 3.34.0 Person Trips/DU (or HH) 8.010.0 HBW Person Trips/Employee 1.201.55 a Generally excludes nonmotorized trips; including motorized trips could increase person trips per DU up to 11.5.

9. 8 Recommended Guidelines and Standards Trip Generation (continued) Percent trips by purpose Statistic Benchmarks Low (Percent)High (Percent) Percent Trips by Purpose – HBW 1224 Percent Trips by Purpose – HBSH 1020 Percent Trips by Purpose – HBSR 912 Percent Trips by Purpose – HBSC 58 Percent Trips by Purpose – HBO a 1428 Percent Trips by Purpose – HBNW b 4560 Percent Trips by Purpose – NHB c 2033 a HBO includes a variety of special trip purposes depending on the model (e.g., airport, college, and shop). b HBNW accounts for all home-based trip purposes except HBW. c NHB includes combined purposes for NHB Work and NHB Nonwork, where appropriate.

10. 9 Recommended Guidelines and Standards Trip Distribution Average trip length by purpose Trip length frequency distributions by purpose Coincidence ratios by purpose – measures the percent of area that coincides for two trip length frequencies Statistic Benchmarks LowHigh Average Trip Length – HBW (minutes)1235 Average Trip Length – HBSH (minutes)919 Average Trip Length – HBSR (minutes)1119 Average Trip Length – HBSC (minutes)716 Average Trip Length – HBO a (minutes)820 Average Trip Length – NHB b (minutes)619 Average Trip Length – IE (minutes)2658 StatisticStandards Mean Trip Length, Observed Total Trips+/-3% Trip Length Frequency Distribution versus Observed +/-5% Coincidence Ratios by Purpose c 70% a HBO includes a variety of special trip purposes, depending on the model (e.g., airport, college, and school). b NHB includes combined purposes for NHB Work and NHB Nonwork, where appropriate. c Some lower coincidence ratios have been deemed acceptable for trip purposes that had relatively few trips and therefore higher error rates. 8% 6% 4% 2% 0% Percent of Total Trips Travel Time (in Minutes) 0102030405060 Coincidence Ratio = 0.82 Estimated (ATL = 18.2 Min) Observed (ATL = 18.9 Min)

11. 10 Recommended Guidelines and Standards Trip Distribution (continued) Percent intrazonal trips by purpose Map-based (“node-point”) charts Zone-based Number of trips Trip productions/attractions by purposeStatisticBenchmarksLowHigh Percent Intrazonal – HBW 1%4% Percent Intrazonal – HBSH 3%9% Percent Intrazonal – HBSR 4%10% Percent Intrazonal – HBSC 10%12% Percent Intrazonal – HBO a 3%7% Percent Intrazonal – NHB b 5%9% Percent Intrazonal – Total Trips 3%5% Standards StatisticAcceptablePreferable Percent Intrazonal, Observed Total Trips +/-3%+/-5% a HBO includes a variety of special trip purposes, depending on the model (e.g., airport, college, and school). b NHB includes combined purposes for NHB Work and NHB Nonwork, where appropriate.

12. 11 Recommended Guidelines and Standards Mode Choice Mode split targets (ideal) Trip purpose Mode Auto ownership level Geographic subarea Mode Zero-Vehicle Households One-Vehicle Households Two-Vehicle Households Three-Vehicle Households Walk5,0006,0004,0003,000 Bike2,0001,000500200 Drive Alone -130,000350,000200,000 Shared Ride 2 Persons 6,00015,00020,00010,000 Shared Ride 3 Persons 1,0002,0004,0002,000 Local Bus, Walk 6,0007,0004,0001,000 Local Bus, PNR -500 2,000500 Local Bus, KNR -200 Express Bus, Walk 1,0001,0001,000500 Express Bus, PNR -2,0004,0002,000 Express Bus, KNR -200500 LRT, Walk 5001,000400 LRT, PNR -300500 LRT, KNR Trip Allocation By Mode Trip Allocation By ModePolk Census 2000 Census 200020001990 Trips Percent of Trips Trips Number HBW Drive Alone 244,41479.69%188,25980.47% 84.14% One Passenger 51,46516.78%38,21216.33%Carpool15.07% Two+ Passenger 9,6373.14%7,0233.00% Total Transit 1,2060.39%4650.20%Transit0.78%

13. 12 Recommended Guidelines and Standards Mode Choice (continued) Mode splits by observed calibration targets Total area transit trips, estimated versus observed Transit trips between districts Tabular comparisons (CTPP) Desire lines Mean trip length, estimated transit trips versus observed Statistic Standards LowHigh Total Area Transit Trips versus Observed+/-1%+/- 2% Transit Trips between DistrictsCompare model trip table against CTPP or HH survey Mean Trip Length Transit Trips versus Observed+/-5%+/-15% Mode Splits by Observed Calibration Targets+/- 2% Elasticity of Demand with Respect to LOS Variables-0.6-0.1

14. 13 Recommended Guidelines and Standards Trip Assignment Volume-over-count ratios +/-1 lane percent error (calculated based on FDOT LOS Handbook) Aggregate VMT VMT/HH (60-75) VMT/person (24-32) VMT/commercial vehicle (3-25%) Statistic Standards AcceptablePreferable Freeway Volume-over-Count +/- 7%+/- 6% Arterial Volume-over-Count +/- 15%+/- 10% Collector Volume-over-Count +/- 25%+/- 20% Frontage Road Volume-over-Count +/- 25% Freeway Peak Volume-over-Count 75% of links @ +/-20%; 50% of links @ +/-10% Major Arterial Peak Volume-over-Count 75% of links @ +/-30%; 50% of links @ +/-15% Assigned VMT-over-Count Areawide +/-5%+/-2% Assigned VHT-over-Count Areawide +/-5%+/-2% Assigned VMT-over-Count by FT/AT/NL +/- 25%+/- 15% Assigned VHT-over-Count by FT/AT/NL +/- 25%+/- 15% Statistic Standards AcceptablePreferable Percent Error – LT 10,000 volume (2L road)50%25% Percent Error – 10,000-30,000 (4L road)30%20% Percent Error – 30,000-50,000 (6L road)25%15% Percent Error – 50,000-65,000 (4-6L freeway)20%10% Percent Error – 65,000-75,000 (6L freeway)15%5% Percent Error – GT 75,000 (8+L freeway)10%5%

15. 14 Recommended Guidelines and Standards Trip Assignment (continued) Screenline volume-over-count By volume and location RMSE by volume group Transit assignment validation Statistic Standards AcceptablePreferable RMSELT 5,000 AADT RMSE – LT 5,000 AADT150%45% RMSE5,000-9,999 AADT RMSE – 5,000-9,999 AADT45%35% RMSE10,000-14,999 AADT RMSE – 10,000-14,999 AADT35%27% RMSE15,000-19,999 AADT RMSE – 15,000-19,999 AADT35%25% RMSE20,000-29,999 AADT RMSE – 20,000-29,999 AADT27%15% RMSE30,000-49,999 AADT RMSE – 30,000-49,999 AADT25%15% RMSE50,000-59,999 AADT RMSE – 50,000-59,999 AADT20%10% RMSE60,000+ AADT RMSE – 60,000+ AADT19%10% RMSE Areawide 45%35% Statistic Benchmarks LowHigh Estimated-over-Observed Transit Trips+/- 9%+/- 3% Standards StatisticAcceptablePreferable Acceptable Error – Transit Screenlines+/-20%+/-10% Transit Ridership – <1,000 Passengers/Day+/-150%+/- 100% Transit Ridership – 1k-2k Passengers/Day+/- 100%+/- 65% Transit Ridership – 2k-5k Passengers/Day+/- 65%+/- 35% Transit Ridership – 5k-10k Passengers/Day+/- 35%+/- 25% Transit Ridership – 10k-20k Passengers/Day+/- 25%+/- 20% Transit Ridership – >20,000 Passengers/Day+/- 20%+/- 15%

16. 15 Recommended Guidelines and Standards Other Model Applications – LRTP Highway Only Models Same default guidelines and standards except Replace mode choice checks with auto occupancy comparisons against NHTS and other surveys Commercial vehicle VMT checks not likely relevant No transit assignment validation Auto Occupancy Rates PurposeCurrentModel1988FLSWM2001 NHTS FL 2001 NHTS US HBW1.101.301.061.10 HBShop1.801.551.571.80 HBSR1.942.271.791.94 HBO1.701.501.901.70 NHB1.711.581.821.71 Statistic Benchmarks/Settings LowHigh Auto Occupancy Rates – HBW1.051.10 Auto Occupancy Rates – HBSH1.501.80 Auto Occupancy Rates – HBSR1.701.90 Auto Occupancy Rates – HBO a 1.651.95 Auto Occupancy Rates – NHB b 1.601.90 a HBO includes a variety of special trip purposes, depending on the model (e.g., airport, college, and school). b NHB includes combined purposes for NHB Work and NHB Non-Work, where appropriate.

17. 16 Recommended Guidelines and Standards Other Model Applications – FTA New Starts Transit networks and pathbuilding checks Compare skim settings to on-board surveys Trip distribution checks Mode choice calibration Highway assignment checks Transit assignment checks Assign on-board survey trip table and compare ridership SUMMIT diagnostics Statistic Acceptable Range of Values LowHigh Elasticity of demand with respect to LOS variables -0.10 -0.70 IVT parameter – HBW* -0.02 -0.03 IVT parameter – HBNW* 0.1 to 0.5*CIVT HBW trips IVT parameter – NHB* ~CIVT HBW trips RatioOVT/IVT parameters – HBW* Ratio – OVT/IVT parameters – HBW*2.0 3.0 RatioOVT/IVT parameters – HBNW* Ratio – OVT/IVT parameters – HBNW*2.0 3.0 RatioOVT/IVT parameters – NHB* Ratio – OVT/IVT parameters – NHB*2.0 3.0 Implied value of time – Percent of income 25% 33% Implied value of time – HBW $2.00 $7.00 Implied value of time – HBNW $0.50 $5.00 Implied value of time – NHB $0.20 $5.00 * FTA published guideline.

18. 17 Recommended Guidelines and Standards Other Model Applications – Subarea Models Prerequisite – approved regional model validation Input data – focus on socioeconomic and network data Trip generation – review and compare subarea versus regional model aggregate trip rates Trip distribution – compare subarea versus regional average trip length and percent intrazonal trips by purpose Mode choice – check subarea mode shares versus regional Trip assignment – volume-over-count (v-o-c), percent error, VMT and VHT v-o-c, v-o-c by screenline/cutline, and RMSE

19. 18 Recommended Guidelines and Standards Other Model Applications – Corridor Models Same subarea model validation checks Input data – focus on network details surrounding corridor Trip generation – review corridor productions and attractions by zone Trip distribution – desire line analyses Mode choice – review of mode shares within study area Trip assignment – more stringent standards for v-o-c, v-o-c by screenline/cutlineStatisticStandardsAcceptablePreferable Freeway Volume-over-Count +/- 6% +/- 5% Arterial Volume-over-Count +/- 10% +/- 7% Collector Volume-over-Count +/- 15% +/- 10% Frontage Rd Volume-over-Count +/- 20% +/- 15%

20. 19 Recommended Guidelines and Standards Other Model Applications – Models for Impact Studies Input data SE data – site, nearby zone assumptions, pop/TAZ Networks – verify coding, path traces from site Transit – access coding, headways, stop locations near site Trip generation – document trip rate assumptions Trip distribution – district summaries Mode choice – check ITE trips versus model trips Trip assignment – select zone and select link, turn volumes

21. 20 Calibration and Validation Best Practices Steps in Model Validation and Calibration Process Iterative process Process must acknowledge Availability of behavioral data (or lack thereof) Regional issues to be evaluated with the model Need for future year sensitivity testing Must validate each step (i.e., not just assignment) Inventory Institutional Framework Data Checking Initiate Primary Data Collection Model Estimation Secondary Data Collection Model Implementation Model Calibration Model Validation Model Application Iterate Continual Model Maintenance, Application Satisfactory Results? NO YES

22. 21 Calibration and Validation Best Practices Guidance on Validation and Adjustment Matching base year statistics is not sufficient to say model is validated Ideal combination of skills Local area knowledge Familiarity with sources for transferable parameters Understanding of what constitutes acceptable results Experience with cause and effect of model adjustments Quality and availability of data Validation and Reasonableness Checks Iterate EstimationCalibration ValidationApplication

23. 22 Calibration and Validation Best Practices Special Validation Considerations Requirements by study type FTA New Starts projects Subarea and corridor validation Site impact studies Other validation practices Adjustment of congested speeds Use of special generators Transferable parameters Impact of new paradigms Study Area Urban Model Area Study Area Urban Model Area 1 Study Area Within One Model Area Urban Model Area 2 Study Areas Within Two Model Areas Study Area Urban Model Area Study Area Statewide Model Study Area Outside Urban Model Areas Statewide Model General Purpose Lanes HOV Lanes Would not cross traffic to take loop ramp Cannot cross median to access driveway (centroid connector) Example of Coding Prohibitors for Proper Access Example of Coding Penalties for HOV Lanes Red lines represent penalized or prohibited movements.

24. 23 Guidelines for Model Applications Stability of Model Parameters Static versus dynamic parameter settings Generally “locked down” during validation Limited trend data to back up values for future years History has shown changes in… Nonhome-based trip rates Trip length frequencies Auto occupancy rates BPR curves

25. 24 Guidelines for Model Applications Typical Model Applications and Guidelines Developed list of 14 different model application types and relevant model guidance unique to each (e.g., forecasting external trips for MPO LRTP Updates, etc.) MPO LRTP Updates Comprehensive Plans SIS/FIHS Planning Campus Master Plans Concurrency Applications Development of Regional Impacts (DRIs) Congestion Management Systems Air Quality and Climate Change Corridor Studies  Corridor Feasibility Studies;  FTA New Starts/Small Starts Applications;  Project Development and Environment (PD&E) Studies;  Interstate Master Plans;  Interchange Justification/Modification Reports (IJR/IMR); and  Toll Feasibility Studies.

26. 25 Guidelines for Model Applications Model Application Checks Review logic of demographic forecasts at region and subarea Generate color-coded plots of highway network characteristics Compare base and future year trip productions and attractions by purpose at the regional and subarea level Compare base and future year trip distribution patterns Review logic of changes in mode splits resulting from scenario testing that would seemingly benefit one mode over another Compare traffic estimates on specific corridors and screenlines between base and future years and build and no-build conditions

27. 26 Guidelines and Standards in Practice FDOT has circulated Final Report to some MTF members Implemented validation checklist and new standards FDOT D2 I-295 Master Plan – NERPM corridor validation FDOT Statewide Model – base year 2005 validation Chattanooga 2035 LRTP – base year 2007 validation In use – Gainesville 2035 LRTP and San Juan model check Seeking concurrence to upload Final Report to Florida model users web site

28. 27 Questions and Comments… Model StepModel Statistic to Evaluate Acceptable Range of Values Low High Accuracy Standard Recommended Comparisons and Calculation Methods/Comments Input DataSocioeconomic DataCube, GIS Visual and Statistical Comparisons/Checks Document checks for households and employment Persons/DU (or HH)2.002.70N/ANHTS > 2.46 FL – 2.59 U.S. Employment/Population Ratio0.350.75N/A Autos/DU (or HH)1.752.10N/A Approximate Population/TAZN/A3,000N/ARecommendations from TAZ White Paper Highway Network DataCube, GIS visual and statistical comparisons/checks Check hwy network, prohibitors, tolls, paths Highway Speed Dataensure logical hierarchy by AT/FT/NL; survey chk Will provide acceptable ranges in sep. table Transit Network Datachk access links; chk routing against GIS dataChecks for transit network, access, paths