Experiences running Dynamic Traffic Assignment Simulations at scale using HPC Infrastructure Amit Gupta Joint work with: Weijia Xu Texas Advanced Computing Center Kenneth Perrine, Dennis Bell, Natalia Ruiz-Juri Network Modeling Center Center for Transportation Research

Dynamic Traffic Assignment Used to Model complex interactions between –Traveller   Traveller –Traveller   Road Infrastructure Usually Simulation Based Shortest Path calculations –Used to model traveller behavior to congestion –Intensive part of the process –Time dependent Traffic conditions change 1

Road Networks & Travel Time Turn Movement Delays Signal Delays 2 Prohibitions –(One Ways, Closed Roads) Tolls, Vehicle Types * Time Dependent *

3 Transportation: Road Networks

Dynamic Traffic Assignment Static Traffic Assignment –Inflow = Outflow Dynamic Traffic Assignment –More realistic modeling of congestion Lambda –Fraction of vehicles that change route Equilibrium –For each O/D pair, all routes experience the same travel time. Different route choice cant lower cost. Gap - Measure of proximity to equilibrium 4

DTA Workflow 5

Our Approach Example Profile VISTA –DTA and Simulation Framework –Popular in Transportation Research “strace –c” Cross verified with other profiling (iostat) 6

Example Runtime Profile I/O Bottleneck –Initial profiling Upto 77% time spent here –Primarily Route File activity Moved target IO to local Ramdisk –Most I/O ops in-memory 7

Our Approach Computational Bottleneck –Profiling indicated TDSP( Update Labels ) is the main computational Load 8

TDSP Is a “Label Correcting” algorithm Has 2 main parts –Update Labels Incorporates the activity of most recent simulation Changes the cost (label) values at nodes Causes upstream costs to change Maintains pointers to downstream nodes –Search for Routes Back traces the downstream pointers Constructs the shortest path topology 9

TDSP 10 Origin Destination

Update Labels 11 Origin Destination

TDSP Challenges For precision –Larger number of labels maintained –Implies Computational overhead Simulation activity changes travel time –Label values keep changing –Shortest paths also change –Stored on file and looked up –Small, Random reads. I/O overhead 12

13

Update Labels: Queuing Scheme Scan Eligible List –All Label Correcting algorithms use it –Intermediate structure to hold nodes for later VISTA used Dequeue scheme –“Visited nodes” inserted to the front –No prioritization otherwise –Order of Nodes depends on order in which Links are visited –“Feedback effect” during label updates 14

15

Priority Queue Order of Node evaluation matters Prioritize Nodes during evaluation –Use the first time label –Nodes label values converge faster –Reduces overall time for Update Labels => TDSP finishes faster –Average case is improved Already congested link stays that way –Analysis periods like “Rush Hour” time windows –Common for many research investigations 16

17

18

Datasets NetworkNodesLinksO/D PairsDemand (Vehicles) Downtown Austin South Austin Central Austin Austin Region AM Austin Region PM

20

21

Summary Large performance gains (6x-12x) –Relatively minor code modifications –Changes to application architecture Other things attempted –Using Accelerators (Xeon Phi) Offload calculations at every hop Overhead is too high –Different Label for priority Results similar Depends on demand 22

Other code in workflow Update cost parallelization –Runs a different test for different lambda values –Used in analysis –Test data structure Deep Tree like hierarchy –Simulation Results »Graph topology »… –File Store –… Not trivial to serialize Simple solution using MPI_Scatter/Gather not possible 23

Future Work Web portal to kick off analysis jobs on TACC infrastructure Examine possibility of serialization for lambda test related data structures Architectures moving away from offload model –Determine tradeoffs in Knights Landing Investigate using Hadoop/Spark based graph engines for more scalable solutions 24

Acknowledgement Current –XSEDE ECSS –NSF Stampede Supercomputer Previous Funding –Texas Department of Transportation –CAMPO Capital Area Metropolitan Planning Organization Questions ?... 25