1. Traffic Light Control Using Reinforcement Learning
Final Presentation
Traffic Light Control Using Reinforcement Learning
Daniel Goldberg
Andrew Elstein

2. The Problem
Traffic congestion is estimated to cost Americans $121 billion in lost productivity fuel, and other costs.
Traffic Lights are imperfect and contribute to this
Usually statically controlled
A better method of controlling them can reduce waiting times significantly

3. Approach
Implement a “Reinforcement Learning” (RL) algorithm to control traffic lights
Create a simulation of traffic to tweak and test traffic light optimizations

4. Implementation
If minor adjustments were made to the algorithm, it could operate within existing infrastructure
Optimally, a camera system and would be added

5. Simulation
Insert picture of visualization

6. Simulation Structure
To build the simulation we created the follow Data Structures:
Cars
Position, Destination, Velocity, Map, Color
Roads
Lanes
Individual Cells
Intersection location matrix
Intersections
Position, Traffic Lights
In total, the simulation is coded in
MATLAB with 3100 lines of code
Cars Struct

7. Simulation Dynamics Cars are spawned randomly
They follow an randomly generated path to destination
Cars follow normal traffic rules
Road Cells are discretized to easily simulate traffic, only one car can exist in each road cell. Cars move ahead one or two cells in each time-step, depending on the car's max velocity and whether there is an open spot.

8. Demo

9. Reinforcement Learning
Weiring - Multi-Agent Reinforcement Learning for Traffic Light Control
It introduced an objective function to minimize or maximize a goal value
𝑄 𝑡𝑙,𝑝,𝑑 , 𝐿 = (𝑡 𝑙 ′ , 𝑝 ′ ) 𝑃 𝑡𝑙,𝑝,𝑑 ,𝐿, 𝑡 𝑙 ′ , 𝑝 ′ ∗(𝑅 𝑡𝑙,𝑝 , 𝑡 𝑙 ′ , 𝑝 ′ +𝛾𝑉( 𝑡 𝑙 ′ , 𝑝 ′ ,𝑑 ) )
𝑉 𝑡𝑙,𝑝,𝑑 = 𝐿 𝑃 𝐿| 𝑡𝑙,𝑝,𝑑 𝑄 𝑡𝑙,𝑝,𝑑 , 𝐿
𝑅 𝑡𝑙,𝑝 , 𝑡 𝑙 ′ , 𝑝 ′ = 𝑖𝑓 𝑡𝑙,𝑝 =[𝑡 𝑙 ′ , 𝑝 ′ ] 𝑖𝑓 𝑡𝑙,𝑝 ≠[𝑡 𝑙 ′ , 𝑝 ′ ]
tl = traffic light
p = current position
d = destination
L = light decision
𝛾 = discounting constant
‘ = next

10. Reinforcement Learning Theory
Coordinating a system of lights to respond to current conditions can reap exceptional benefit
The theory cleverly merges probability, game theory and machine learning to efficiently control traffic
In our case, the expected value of each of a light’s possible states are calculated
With this value function a game is played to maximize it, in turn minimizing waiting time

11. Results
Wrote a script to compare the smart algorithm to static On-Off-On-Off lights. Our algorithm reduced average waiting time—and thus traveling time— for a system with any number of cars Travelling time for our implementation was reduced by an average of 10%. There was a 15% reduction for sparse traffic systems from a static control, but only a 3% decrease for heavy congestion.

12. Results cont.

13. Extensions Fairness-weighted objective: ω = weighting constant
ω = weighting constant
t = current time
ti = time of arrival for car i
If F(t) > 1, cars on road 1 get to go
If F(t) < 1, cars on road 2 get to go

14. Further Extensions Car Path optimization and rerouting
Model expansion to traverse an entire city
Inter-traffic-light communication
Retesting with increased computational resources for modeling accuracy and robustness

15. RL In the News
Samah El-Tantawy, 2012 PhD recipient from the University of Toronto, won the 2013 IEEE best dissertation award for her research in RL.
Her RL traffic model showed reduced rush-hour travel times by as much as 26 percent and is working on monetizing her research with small MARLIN-ATSC (Multi-agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers) computers.
Multi-agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers.

16. Challenges
Difficult to understand data structures and how they would interact
Object Oriented Approach vs. MATLAB’s index-based structures
Understand how cars would interact with each other
Understanding RL algorithm
Adapting our model to use RL algorithm
Limited computational resources