1. SACRAMENTO AREA COUNCIL OF GOVERMENTS
ACCOUNTING FOR DEMAND SIMULATION RANDOM VARIATIONS IN PROJECT-LEVEL PERFORMANCE ASSESSMENT
Shengyi Gao
SACOG
Garett Ballard-Rosa

2. SACRAMENTO AREA COUNCIL OF GOVERMENTS
Motivation
Problem: In benefit cost analysis at project level, the noise caused by the random variation in activity-based, demand simulation models is significant enough to overwhelm project effects.
Objective: Estimate the range of variation for key measures of project effects, and find a practical approach to managing noise and random variation that can be applied at project level benefit cost analysis.

3. Literature Bradley, Bowman, and Lawton (1999)
SACRAMENTO AREA COUNCIL OF GOVERMENTS
Literature
Bradley, Bowman, and Lawton (1999)
--Random sampling error: <2% in Portland’s ABM
Castiglione, Freedman, and Bradley (2003)
--Variations decrease sharply after 10 runs in SF ABM

4. Literature cont’ Bowman, Bradley, and Gibb (2006)
SACRAMENTO AREA COUNCIL OF GOVERMENTS
Literature cont’
Bowman, Bradley, and Gibb (2006)
--The upper level models of longer term decisions and activity/tour generation are sensitive to network accessibility and a variety of land use variables.
--“If this feature is used throughout the simulation, then a change in just one decision by one individual will change the entire sequence of random numbers that are used for all subsequent simulated individuals.” But SACSIM is programmed to avoid this situation.

5. Part I: Explore random variation across multiple model runs
SACRAMENTO AREA COUNCIL OF GOVERMENTS
Part I: Explore random variation across multiple model runs
Run SACSIM 10 times with different random seeds from a single base run
Calculate the variations of skims, VMT, and emissions from a single run

6. SACRAMENTO AREA COUNCIL OF GOVERMENTS
Skim Variation across 10 runs of SACSIM Auto Drive Alone skim (p.m peak)
Skim time band
5 min
10 min
20 min
30 min
Average Variation
-0.02%
0.07%
0.10%
Lower Bound
-1.5%
-2.6%
-1.4%
-0.9%
Upper Bound
2.3%
1.9%
1.4%
1.0%

7. Skim Variation across 10 runs of SACSIM Auto HOV skim (p.m peak)
SACRAMENTO AREA COUNCIL OF GOVERMENTS
Skim Variation across 10 runs of SACSIM Auto HOV skim (p.m peak)
Skim time band
5 min
10 min
20 min
30 min
Average Variation
-0.02%
-0.04%
-0.03%
Lower Bound
-1.5%
-2.6%
-1.4%
-0.9%
Upper Bound 2%
1.9%
1.4%
1.1%

8. SACRAMENTO AREA COUNCIL OF GOVERMENTS
Skim Variation across 10 runs of SACSIM In-Vehicle Transit Time skim (walk to transit, mid day)
Skim time band
10 min
20 min
40 min
60 min
Average Variation
0.02%
-0.01%
0.01%
Lower Bound
-25%
-29%
-29 %
-14%
Upper Bound
56%
17%
40%
24%

9. SACRAMENTO AREA COUNCIL OF GOVERMENTS
Skim Variation across 10 runs of SACSIM Transit Transfer Time skim (walk to transit, mid day)
Skim time band
5 min
10 min
20 min
30 min
Average Variation
-0.21%
-0.23%
0.01%
-0.01%
Lower Bound
-94%
-60%
-25%
-12%
Upper Bound
81%
48%
40%
36%

10. Skim Variation across 10 runs of SACSIM Walk to Transit skim (mid day)
SACRAMENTO AREA COUNCIL OF GOVERMENTS
Skim Variation across 10 runs of SACSIM Walk to Transit skim (mid day)
Skim time band
5 min
10 min
20 min
30 min
Average Variation
-0.30%
-0.07%
0.06%
0.07%
Lower Bound
-107%
-70%
-38%
-24%
Upper Bound
12%
20%
39%
34%

11. SACRAMENTO AREA COUNCIL OF GOVERMENTS
Skim Variation across 10 runs of SACSIM Transit Initial Wait skim (mid day)
Skim time band
10 min
15 min
25 min
Average Variation
0.04%
0.02%
0.06%
Lower Bound
-86%
-62%
-32%
Upper Bound
35%
41%
44%

12. SACRAMENTO AREA COUNCIL OF GOVERMENTS
Skim Variation across 10 runs of SACSIM # of Transit Transfers (mid day)
Skim time band
10 min
15 min
25 min
Average Variation 0%
Lower Bound
-72%
-42%
-21%
Upper Bound
65%
42%
33%

13. Part II: Random Variation and Project-Level Performance Assessment
SACRAMENTO AREA COUNCIL OF GOVERMENTS
Part II: Random Variation and Project-Level Performance Assessment

14. BCA measures come from SACSIM
SACRAMENTO AREA COUNCIL OF GOVERMENTS
BCA measures come from SACSIM
Travel time savings (auto, freight, transit, bike/ped)
Reliability benefit (buffer hours)
Travel operating and ownership costs
Collisions
Physical activity
Air quality
Etc.

15. First check: Do results pass reasonableness test?
SACRAMENTO AREA COUNCIL OF GOVERMENTS
First check: Do results pass reasonableness test?
SOV time
Transit travel time
VMT
Auto ownership
Run 1
Run 2
Run 3
Run 4
Run 5

16. Possible approach moving forward?
SACRAMENTO AREA COUNCIL OF GOVERMENTS
Possible approach moving forward?
Thresholds from assessing random variation in each BCA metric
Test for statistical significance (t test) by BCA metric at TAZ level
Exclude where P/NP difference inside threshold
Include where P/NP difference outside of threshold

17. Example of BCA Approach
SACRAMENTO AREA COUNCIL OF GOVERMENTS
Example of BCA Approach

18. Daily VMT Change 5 runs no filter 5 runs with filter Run 1 8,000 9,100
SACRAMENTO AREA COUNCIL OF GOVERMENTS
Daily VMT Change
5 runs no filter
5 runs with filter
Run 1
8,000
9,100
Run 2
4,800
10,200
Run 3
10,000
9,000
Run 4
-17,600
11,000
Run 5
-16,300
9,600
Average
-2,200
9,780

19. SACRAMENTO AREA COUNCIL OF GOVERMENTS
Takeaways
There is a need to account for demand simulation variation in project performance assessment
Threshold testing: Is there a point where lose signal to ‘noise’? How much an issue based on project scale?
Certain performance measures seem more stable than others
SOV travel time
Transit travel time
Auto ownership, active health measure

20. Shengyi Gao SACOG sgao@sacog.org Garett Ballard-Rosa
SACRAMENTO AREA COUNCIL OF GOVERMENTS
Contact Info
Shengyi Gao
SACOG
Garett Ballard-Rosa