SACRAMENTO AREA COUNCIL OF GOVERMENTS ACCOUNTING FOR DEMAND SIMULATION RANDOM VARIATIONS IN PROJECT-LEVEL PERFORMANCE ASSESSMENT Shengyi Gao SACOG sgao@sacog.org Garett Ballard-Rosa gballard-rosa@sacog.org

SACRAMENTO AREA COUNCIL OF GOVERMENTS Motivation Problem: In benefit cost analysis at project level, the noise caused by the random variation in activity-based, demand simulation models is significant enough to overwhelm project effects. Objective: Estimate the range of variation for key measures of project effects, and find a practical approach to managing noise and random variation that can be applied at project level benefit cost analysis.

Literature Bradley, Bowman, and Lawton (1999) SACRAMENTO AREA COUNCIL OF GOVERMENTS Literature Bradley, Bowman, and Lawton (1999) --Random sampling error: <2% in Portland’s ABM Castiglione, Freedman, and Bradley (2003) --Variations decrease sharply after 10 runs in SF ABM

Literature cont’ Bowman, Bradley, and Gibb (2006) SACRAMENTO AREA COUNCIL OF GOVERMENTS Literature cont’ Bowman, Bradley, and Gibb (2006) --The upper level models of longer term decisions and activity/tour generation are sensitive to network accessibility and a variety of land use variables. --“If this feature is used throughout the simulation, then a change in just one decision by one individual will change the entire sequence of random numbers that are used for all subsequent simulated individuals.” But SACSIM is programmed to avoid this situation.

Part I: Explore random variation across multiple model runs SACRAMENTO AREA COUNCIL OF GOVERMENTS Part I: Explore random variation across multiple model runs Run SACSIM 10 times with different random seeds from a single base run Calculate the variations of skims, VMT, and emissions from a single run

SACRAMENTO AREA COUNCIL OF GOVERMENTS Skim Variation across 10 runs of SACSIM Auto Drive Alone skim (p.m peak) Skim time band 5 min 10 min 20 min 30 min Average Variation -0.02% 0.07% 0.10% Lower Bound -1.5% -2.6% -1.4% -0.9% Upper Bound 2.3% 1.9% 1.4% 1.0%

Skim Variation across 10 runs of SACSIM Auto HOV skim (p.m peak) SACRAMENTO AREA COUNCIL OF GOVERMENTS Skim Variation across 10 runs of SACSIM Auto HOV skim (p.m peak) Skim time band 5 min 10 min 20 min 30 min Average Variation -0.02% -0.04% -0.03% Lower Bound -1.5% -2.6% -1.4% -0.9% Upper Bound 2% 1.9% 1.4% 1.1%

SACRAMENTO AREA COUNCIL OF GOVERMENTS Skim Variation across 10 runs of SACSIM In-Vehicle Transit Time skim (walk to transit, mid day) Skim time band 10 min 20 min 40 min 60 min Average Variation 0.02% -0.01% 0.01% Lower Bound -25% -29% -29 % -14% Upper Bound 56% 17% 40% 24%

SACRAMENTO AREA COUNCIL OF GOVERMENTS Skim Variation across 10 runs of SACSIM Transit Transfer Time skim (walk to transit, mid day) Skim time band 5 min 10 min 20 min 30 min Average Variation -0.21% -0.23% 0.01% -0.01% Lower Bound -94% -60% -25% -12% Upper Bound 81% 48% 40% 36%

Skim Variation across 10 runs of SACSIM Walk to Transit skim (mid day) SACRAMENTO AREA COUNCIL OF GOVERMENTS Skim Variation across 10 runs of SACSIM Walk to Transit skim (mid day) Skim time band 5 min 10 min 20 min 30 min Average Variation -0.30% -0.07% 0.06% 0.07% Lower Bound -107% -70% -38% -24% Upper Bound 12% 20% 39% 34%

SACRAMENTO AREA COUNCIL OF GOVERMENTS Skim Variation across 10 runs of SACSIM Transit Initial Wait skim (mid day) Skim time band 10 min 15 min 25 min Average Variation 0.04% 0.02% 0.06% Lower Bound -86% -62% -32% Upper Bound 35% 41% 44%

SACRAMENTO AREA COUNCIL OF GOVERMENTS Skim Variation across 10 runs of SACSIM # of Transit Transfers (mid day) Skim time band 10 min 15 min 25 min Average Variation 0% Lower Bound -72% -42% -21% Upper Bound 65% 42% 33%

Part II: Random Variation and Project-Level Performance Assessment SACRAMENTO AREA COUNCIL OF GOVERMENTS Part II: Random Variation and Project-Level Performance Assessment

BCA measures come from SACSIM SACRAMENTO AREA COUNCIL OF GOVERMENTS BCA measures come from SACSIM Travel time savings (auto, freight, transit, bike/ped) Reliability benefit (buffer hours) Travel operating and ownership costs Collisions Physical activity Air quality Etc.

First check: Do results pass reasonableness test? SACRAMENTO AREA COUNCIL OF GOVERMENTS First check: Do results pass reasonableness test? SOV time Transit travel time VMT Auto ownership Run 1 Run 2 Run 3 Run 4 Run 5

Possible approach moving forward? SACRAMENTO AREA COUNCIL OF GOVERMENTS Possible approach moving forward? Thresholds from assessing random variation in each BCA metric Test for statistical significance (t test) by BCA metric at TAZ level Exclude where P/NP difference inside threshold Include where P/NP difference outside of threshold

Example of BCA Approach SACRAMENTO AREA COUNCIL OF GOVERMENTS Example of BCA Approach

Daily VMT Change 5 runs no filter 5 runs with filter Run 1 8,000 9,100 SACRAMENTO AREA COUNCIL OF GOVERMENTS Daily VMT Change 5 runs no filter 5 runs with filter Run 1 8,000 9,100 Run 2 4,800 10,200 Run 3 10,000 9,000 Run 4 -17,600 11,000 Run 5 -16,300 9,600 Average -2,200 9,780

SACRAMENTO AREA COUNCIL OF GOVERMENTS Takeaways There is a need to account for demand simulation variation in project performance assessment Threshold testing: Is there a point where lose signal to ‘noise’? How much an issue based on project scale? Certain performance measures seem more stable than others SOV travel time Transit travel time Auto ownership, active health measure

Shengyi Gao SACOG sgao@sacog.org Garett Ballard-Rosa SACRAMENTO AREA COUNCIL OF GOVERMENTS Contact Info Shengyi Gao SACOG sgao@sacog.org Garett Ballard-Rosa gballard-rosa@sacog.org