1. Adaptive Dynamic Bipartite Graph Matching: A Reinforcement Learning Approach
Yansheng Wang 1, Yongxin Tong 1, Cheng Long 2, Pan Xu 3, Ke Xu 1, Weifeng Lv 1
1 Beihang University
2 Nanyang Technological University
3 University of Maryland, College Park
Good afternoon everyone. I am Yansheng Wang from Beihang University. Today I’ll present our work Adaptive Dynamic Bipartite Graph Matching:A Reinforcement Learning Approach. This is a collaborative work with Yongxin, Ke and Weifeng from Beihang University, Cheng from Nanyang Technological University and Pan from University of Maryland, College Park

2. Outline Background and Motivation Problem Statement Our Solutions
Experiments
Conclusion
This is the outline

3. Outline Background and Motivation Problem Statement Our Solutions
Experiments
Conclusion
I will first introduce the background and the motivation

4. Solved by the Hungarian method in polynomial time
Background
Bipartite graph matching
Traditional applications
Assignment problem, vehicle scheduling problem, etc.
Perform well in offline scenarios 3 1 2 4 5 6
Solved by the Hungarian method in polynomial time
I think everyone is quite familiar with the bipartite graph matching problem;
This is a weighted bipartite graph and we want to match the nodes in order to maximize the total weights
The problem can be solved by the Hungarian algorithm in polynomial time, which is first developed by Harold Kuhn in 1950s
It has been applied to various problems like…, and its good performance in offline scenarios has been validated
Harold W. Kuhn

5. Online matching is more and more important
Background
Emergence of online scenarios
Transportation
Medical
Economic
Taxi-hailing,
Ride sharing, …
Mutual blood donation,
Kidney exchange, …
Two-sided market,
Crowdsourcing, …
However, with the emergence of mobile Internet and sharing economy, online scenario is becoming more and more common for matching problems, where the supply and demand come into a system dynamically.
For example, xxx,xxx in xxx area,… , they all have real-world apps, so online matching is becoming more and more important
Online matching is more and more important

6. Existing Research Problem model: online bipartite matching
Solution: online algorithms under instantaneous constraint 1
Objective: maximize the sum of weights
Nodes arrive and leave dynamically 2 2 3 3 4
Match as soon as the node arrives
(instantaneous constraint) 4 1 5 2
There are some existing research on this problem, which often models it with online bipartite matching, where nodes arrive and leave dynamically and the objective is to maximize the sum of weights between matching pairs. Also, it assumes that each node needs to be matched as soon as its arrival, which is also known as the spontaneous constraint.
And existing solutions are often to design online algorithms under this constraint 6
Y. Tong et al, Online mobile microtask allocation in spatial crowdsourcing. In ICDE 2016.

7. Motivation Instantaneous constraint is too strong sometimes
If nodes can wait (match in a batch manner)
More information can be gathered
Likely to meet better candidates in the future
17:00
17:30
I have a task of
labeling 500
pictures
Accuracy:70%
Accuracy:95%
Waiting for response
2 drivers nearby have
received your order
Passengers can wait for a short time before being served
Requseters are willing to wait for more reliable workers
However, from our point of view, the spontaneous constraint is too strong sometimes.
For example, in a taxi hailing app, passengers can…,
So if nodes can …
There are also existing works using fixed batch-based methods
L. Kazemi et al. Geocrowd: enabling query answering with spatial crowdsourcing. In GIS 2012.

8. Limitation of existing work
Strong assumptions: instantaneous constraint
Batch manner: fixed batch and lacking in global theoretical guarantee
Let’s conclude the limitation…
Y. Tong et al, Online mobile microtask allocation in spatial crowdsourcing. In ICDE 2016.
Y. Tong et al, Flexible dynamic task assignment in real-time spatial data. In VLDB 2017.
P. Cheng et al, An experimental evaluation of task assignment in spatial crowdsourcing. In VLDB 2018
L. Kazemi et al. Geocrowd: enabling query answering with spatial crowdsourcing. In GIS 2012.
L. Kazemi et al. GeoTruCrowd: trustworthy query answering with spatial crowdsourcing. In GIS 2013.

9. Contribution Devise a novel adaptive batch-based framework
Analyze the global theoretical guarantee
Propose an effective and efficient reinforcement learning based solution
We make the following contribution in this paper…

10. Outline Background and Motivation Problem Statement Our Solutions
Experiments
Conclusion
Next, we will state our problem formally

11. Problem Statement Dynamic bipartite graph 𝑩=(𝑳,𝑹,𝑬) 𝑹= 𝒋∈ ℕ ∗
Arrival time 3 1 2 4 5 6
𝑹= 𝒋∈ ℕ ∗
𝑳={𝒊∈ ℕ ∗ }
We call such a graph B=…

12. Problem Statement Dynamic bipartite graph 𝑬⊆𝑳×𝑹 𝒘(𝟑,𝟏)=𝟐 𝒘(𝟒,𝟓)=𝟎 1 2
Arrival time 3 1 2 4 5
𝒘(𝟑,𝟏)=𝟐
𝒘(𝟒,𝟓)=𝟎
E is the edge set 2 6

13. Node 3 will vanish at time step 6
Problem Statement
Dynamic bipartite graph
Duration of nodes:
Node 3 will vanish at time step 6
Arrival time 3 1 2 4 5 6 5 3
Lifetime of
node 3 1
Every node has a duration… 2 2 6 4

14. Problem Statement Dynamic bipartite graph Matching allocation:
Arrival time 1 6
Matching allocation:
𝑴={ 𝟑,𝟏 , 𝟒,𝟐 ,(𝟔,𝟓)} 2 2 5 3 3 3 4 1 4 1
In online scenario, the graph is dynamically changing as follow 5 2 2 6 4
Utility score:
𝑼 𝑩,𝑴 =𝟐+𝟑+𝟐=𝟖

15. Problem Statement Dynamic Bipartite Graph Matching (DBGM) Problem
Given dynamic bipartite graph 𝑩,where each node appears in online scenario
Find matching allocation 𝑴 to maximize the total utility, i.e.,
𝐦𝐚𝐱 𝑴 𝑼(𝑩,𝑴)
Decisions can be made freely.
(Without assumption of instantaneous constraint)
Now we can define the xxx problem as follow

16. Theoretical guarantee of an online algorithm in worst case
Problem Statement
Evaluation Metric
Competitive ratio in adversarial model
𝑪𝑹= 𝐦𝐢𝐧 𝑩 𝑼(𝑩,𝑴) 𝑶𝒑𝒕(𝑩)
Theoretical guarantee of an online algorithm in worst case
Offline optimum
We use the CR in AM as the evaluation

17. Outline Background and Motivation Problem Statement Our Solutions
Experiments
Conclusion
Next we will introduce our solution to the problem, namely the adaptive batch-based solution

18. Our framework We propose an adaptive batch-based framework 1 6
Time of arrival 1 6
Cumulate the dynamically coming nodes into a batch
Match all the nodes in the batch 2 2 5 3 3 3 4 1 4 1
First, we propose the ABB framework, which consists of the following steps
Adaptively adjust the size of batch 5 2 2 6 4

19. Our framework
We propose an adaptive batch-based framework
Time of arrival 1 6
Cumulate the dynamically coming nodes into a batch
Match all the nodes in the batch
Challenge 1: How optimal is an adaptive batch-based framework in theory? 2 2 5 3 3 3 4
Challenge 2: How to implement an optimal strategy to split batches adaptively? 1 4 1
In this framework, two challenges need to be addressed
Adaptively adjust the size of batch 5 2 2 6 4

20. Solution to the 1st Challenge
Hungarian algorithm as the in-batch algorithm 1
Time of arrival 1 6 1
Strategy
𝝈=(𝟏,𝟎,𝟎,𝟎,𝟎,𝟏,𝟏)
∈ 𝑺 𝑩 = 𝟐 𝑩 +𝟏 2 2 5 3 2 3 3 4 3 1 4
𝑼 𝑩, 𝑺 𝑩 = 𝐦𝐚𝐱 𝝈∈ 𝑺 𝑩 𝑼(𝑩,𝝈) 1 4
We will discuss about the solution to the 1st challenge first.
We formalize
Two cases… 5 2 2 5 6 4
We use the Hungarian algorithm in each batch 6

21. Solution to the 1st Challenge
Theoretical result
Our framework can achieve constant competitive ratio 𝟏 𝑪−𝟏 (Where 𝐶 is a constant)
We also prove the competitive ratio is tight
We answer the open question:
Whether a batch-based solution can achieve a global theoretical guarantee?
This is the theoretical result
Thus we answer the open question:

22. Solution to the 2nd Challenge
Observation 3 1 2 4 5 6 ?
utility gain=𝟎
𝝈 𝟏 =𝟎
A sequential decision
making problem . ?
utility gain=𝟔
𝝈 𝟒 =𝟏
Now let me introduce our solution to the 2nd challenge, that is, how to effectively implement a batch splitting strategy…
We have the following observations about the problem,…
Modeled by a
Markov decision process (MDP)

23. Solution to the 2nd Challenge
MDP modeling
State: number of left and right nodes (for simplification)
Action: whether to split the batch now (𝟎 or 𝟏)
Reward: sum of weights of the matching (𝟎 if not to match)
Baseline: Q-learning
Purpose: learning a mapping from states (𝒔) to actions (𝒂) in order to maximize the sum of rewards
Basic idea: a function 𝑸(𝒔,𝒂) to record the score of 𝒔→𝒂
Here is how we model the problem with MDP
We first apply the classical Q-learning algo as the baseline method

24. Solution to the 2nd Challenge
Baseline: Q-learning
an example of inferring (with learned parameters)
Learned Q-table 1 6
State
a=0
a=1
(0,0)
0.97
0.02
(0,1)
0.85
0.59
(1,0)
0.82
0.45
(1,1)
0.66
0.71
(0,2)
0.88
0.53
(2,0)
0.79
0.44
(1,2)
0.61
(2,1)
0.65
(2,2)
0.42 …
Here is an running example of how Q-learning makes decisions,

25. Solution to the 2nd Challenge
Baseline: Q-learning
an example of inferring (with learned parameters)
Learned Q-table 1 6
State
a=0
a=1
(0,0)
0.97
0.02
(0,1)
0.85
0.59
(1,0)
0.82
0.45
(1,1)
0.66
0.71
(0,2)
0.88
0.53
(2,0)
0.79
0.44
(1,2)
0.61
(2,1)
0.65
(2,2)
0.42 …

26. Solution to the 2nd Challenge
Baseline: Q-learning
an example of inferring (with learned parameters)
Learned Q-table 1 6
State
a=0
a=1
(0,0)
0.97
0.02
(0,1)
0.85
0.59
(1,0)
0.82
0.45
(1,1)
0.66
0.71
(0,2)
0.88
0.53
(2,0)
0.79
0.44
(1,2)
0.61
(2,1)
0.65
(2,2)
0.42 … 2 5

27. Solution to the 2nd Challenge
Baseline: Q-learning
an example of inferring (with learned parameters)
Learned Q-table 1 6
State
a=0
a=1
(0,0)
0.97
0.02
(0,1)
0.85
0.59
(1,0)
0.82
0.45
(1,1)
0.66
0.71
(0,2)
0.88
0.53
(2,0)
0.79
0.44
(1,2)
0.61
(2,1)
0.65
(2,2)
0.42 … 2 2 5 3 3 3

28. Solution to the 2nd Challenge
Baseline: Q-learning
an example of inferring (with learned parameters)
Learned Q-table 1 6
State
a=0
a=1
(0,0)
0.97
0.02
(0,1)
0.85
0.59
(1,0)
0.82
0.45
(1,1)
0.66
0.71
(0,2)
0.88
0.53
(2,0)
0.79
0.44
(1,2)
0.61
(2,1)
0.65
(2,2)
0.42 … 2 2 5 3 3 3 4 1 4

29. Solution to the 2nd Challenge
Baseline: Q-learning
an example of inferring (with learned parameters)
Learned Q-table 1 6
State
a=0
a=1
(0,0)
0.97
0.02
(0,1)
0.85
0.59
(1,0)
0.82
0.45
(1,1)
0.66
0.71
(0,2)
0.88
0.53
(2,0)
0.79
0.44
(1,2)
0.61
(2,1)
0.65
(2,2)
0.42 … 2 2 5 3 3 3 4 1 4 1 5 2

30. Solution to the 2nd Challenge
Baseline: Q-learning
an example of inferring (with learned parameters)
Learned Q-table 1 6
State
a=0
a=1
(0,0)
0.97
0.02
(0,1)
0.85
0.59
(1,0)
0.82
0.45
(1,1)
0.66
0.71
(0,2)
0.88
0.53
(2,0)
0.79
0.44
(1,2)
0.61
(2,1)
0.65
(2,2)
0.42 … 2 2 5 3 3 3 4 1 4 1 5 2 2 4 6

31. Solution to the 2nd Challenge
Baseline: Q-learning
Q-table is updated following
𝑸 𝒔,𝒂 ←𝑸 𝒔,𝒂 +𝜶[𝒓𝒆𝒘𝒂𝒓𝒅+ 𝐦𝐚𝐱 𝒂 ′ 𝑸 𝒔 ′ , 𝒂 ′ −𝑸(𝒔,𝒂)]
Bellman backups
The updating process of the Q-table follows this equation…
The Q-learning algorithm converges to optimum
given sufficient training data
Watkins C. J. , Dayan P. Q-learning. Machine learning, 1992.

32. Solution to the 2nd Challenge
Observation 1 2 6 3 4 5 1 ? 2 ?
Searching space of general Q-learning is too large, leading to inefficiency problem 3 ? 4 ? 5
However, the classical Q-learning can not capture the unique properties of our problem
We have observed that…
Therefore, the searching space… 6
Some checks are unnecessary with a
too small batch size
Many nodes will vanish with a too large batch size

33. Solution to the 2nd Challenge
Restricted Q-learning (RQL)
Only consider batches with size ∈ 𝒍 𝒎𝒊𝒏 , 𝒍 𝒎𝒂𝒙
Reformulate states and actions
state 𝒔 → (𝒔,𝒍) 𝒍: how many rounds have we waited for
action 𝒂∈{𝟎,𝟏} → 𝒂∈ 𝒍 𝒎𝒊𝒏 , 𝒍 𝒎𝒂𝒙
We have devised another RL-based solution, namely…
The idea is

34. Solution to the 2nd Challenge
An example of RQL
Suppose 𝒍 𝒎𝒊𝒏 =𝟑, 𝒍 𝒎𝒂𝒙 =𝟓
Learned Q-table 1 6
State
a=3
a=4
a=5 …
(1,2,3)
0.59
0.62
0.55
(1,2,4)
0.0
0.51
0.47
(1,2,5)
0.56
0.44
(2,1,3)
0.37
0.29
0.12
(2,2,4)
0.58
(2,2,5)
0.41
(2,3,5)
0.42
𝒍=𝟏,skip
Let’s see a running example of RQL

35. Solution to the 2nd Challenge
An example of RQL
Suppose 𝒍 𝒎𝒊𝒏 =𝟑, 𝒍 𝒎𝒂𝒙 =𝟓
Learned Q-table 1 6
State
a=3
a=4
a=5 …
(1,2,3)
0.59
0.62
0.55
(1,2,4)
0.0
0.51
0.47
(1,2,5)
0.56
0.44
(2,1,3)
0.37
0.29
0.12
(2,2,4)
0.58
(2,2,5)
0.41
(2,3,5)
0.42 2 5
𝒍=𝟐,skip

36. Solution to the 2nd Challenge
An example of RQL
Suppose 𝒍 𝒎𝒊𝒏 =𝟑, 𝒍 𝒎𝒂𝒙 =𝟓
Learned Q-table 1 6
State
a=3
a=4
a=5 …
(1,2,3)
0.59
0.62
0.55
(1,2,4)
0.0
0.51
0.47
(1,2,5)
0.56
0.44
(2,1,3)
0.37
0.29
0.12
(2,2,4)
0.58
(2,2,5)
0.41
(2,3,5)
0.42 2 2 5 3 3 3
𝒍=𝟑,check table

37. Solution to the 2nd Challenge
An example of RQL
Suppose 𝒍 𝒎𝒊𝒏 =𝟑, 𝒍 𝒎𝒂𝒙 =𝟓
Learned Q-table 1 6
State
a=3
a=4
a=5 …
(1,2,3)
0.59
0.62
0.55
(1,2,4)
0.0
0.51
0.47
(1,2,5)
0.56
0.44
(2,1,3)
0.37
0.29
0.12
(2,2,4)
0.58
(2,2,5)
0.41
(2,3,5)
0.42 2 2 5 3 3 3
𝒍=𝟑,check table
Better split at 𝒍=𝟒

38. Solution to the 2nd Challenge
An example of RQL
Suppose 𝒍 𝒎𝒊𝒏 =𝟑, 𝒍 𝒎𝒂𝒙 =𝟓
Learned Q-table 1 6
State
a=3
a=4
a=5 …
(1,2,3)
0.59
0.62
0.55
(1,2,4)
0.0
0.51
0.47
(1,2,5)
0.56
0.44
(2,1,3)
0.37
0.29
0.12
(2,2,4)
0.58
(2,2,5)
0.41
(2,3,5)
0.42 2 2 5 3 3 3 4
𝒍=𝟒,check table 1 4

39. Solution to the 2nd Challenge
An example of RQL
Suppose 𝒍 𝒎𝒊𝒏 =𝟑, 𝒍 𝒎𝒂𝒙 =𝟓
Learned Q-table 1 6
State
a=3
a=4
a=5 …
(1,2,3)
0.59
0.62
0.55
(1,2,4)
0.0
0.51
0.47
(1,2,5)
0.56
0.44
(2,1,3)
0.37
0.29
0.12
(2,2,4)
0.58
(2,2,5)
0.41
(2,3,5)
0.42 2 2 5 3 3 3 4
𝒍=𝟒,check table 1 4
Better split now!

40. Solution to the 2nd Challenge
An example of RQL
Suppose 𝒍 𝒎𝒊𝒏 =𝟑, 𝒍 𝒎𝒂𝒙 =𝟓
Learned Q-table 1 6
State
a=3
a=4
a=5 …
(1,2,3)
0.59
0.62
0.55
(1,2,4)
0.0
0.51
0.47
(1,2,5)
0.56
0.44
(2,1,3)
0.37
0.29
0.12
(2,2,4)
0.58
(2,2,5)
0.41
(2,3,5)
0.42 2 2 5 3 3 3 4 1 4 1 5 3
𝒍=𝟏,skip

41. Solution to the 2nd Challenge
An example of RQL
Suppose 𝒍 𝒎𝒊𝒏 =𝟑, 𝒍 𝒎𝒂𝒙 =𝟓
Learned Q-table 1 6
State
a=3
a=4
a=5 …
(1,2,3)
0.59
0.62
0.55
(1,2,4)
0.0
0.51
0.47
(1,2,5)
0.56
0.44
(2,1,3)
0.37
0.29
0.12
(2,2,4)
0.58
(2,2,5)
0.41
(2,3,5)
0.42 2 2 5 3 3 3 4 1 4 1 5 3 2 4 6
𝒍=𝟐,skip

42. Solution to the 2nd Challenge
An example of RQL
Suppose 𝒍 𝒎𝒊𝒏 =𝟑, 𝒍 𝒎𝒂𝒙 =𝟓
Learned Q-table 1 6
State
a=3
a=4
a=5 …
(1,2,3)
0.59
0.62
0.55
(1,2,4)
0.0
0.51
0.47
(1,2,5)
0.56
0.44
(2,1,3)
0.37
0.29
0.12
(2,2,4)
0.58
(2,2,5)
0.41
(2,3,5)
0.42 2 2 5 3 3 3 4 1 4 1 5 3 2 4 6 5 1 7
𝒍=𝟑,check table

43. Solution to the 2nd Challenge
An example of RQL
Suppose 𝒍 𝒎𝒊𝒏 =𝟑, 𝒍 𝒎𝒂𝒙 =𝟓
Learned Q-table 1 6
State
a=3
a=4
a=5 …
(1,2,3)
0.59
0.62
0.55
(1,2,4)
0.0
0.51
0.47
(1,2,5)
0.56
0.44
(2,1,3)
0.37
0.29
0.12
(2,2,4)
0.58
(2,2,5)
0.41
(2,3,5)
0.42 2 2 5 3 3 3 4 1 4 1 5 3 2 4 6 5 1 7
𝒍=𝟑,check table
Better split now!

44. Outline Background and Motivation Problem Statement Our Solutions
Experiments
Conclusion
Next, we will show some experimental results

45. Experiments Datasets Compared methods Synthetic data varying
Distribution of edge weights
Graph sparsity
Duration of nodes
Arriving density of nodes
Cardinality
Scalability
Real data from Didi chuxing
about 10K nodes per hour
400K for training and 10K for testing
Compared methods
Greedy algorithm (GR)
TGOA from ICDE16
Fixed-batch algorithm (FB)
We use a synthetic dataset and a real dataset,…

46. RQL is the most effective RQL is efficient in running time
Experiments
Impact of edge distribution
RQL is the most effective
RQL is efficient in running time
The memory cost
is not high (about 23MB)
Here are results of impact of edge distributions

47. Varying the arriving density Varying the graph sparsity
Experiments
Impact of sparsity and arriving density
Varying the arriving density
of the nodes
Varying the graph sparsity

48. Varying the maximal duration of tasks/workers
Experiments
Results on real data from Didi chuxing
Here are results on real datasets
Varying the maximal duration of tasks/workers

49. Outline Background and Motivation Problem Statement Our Solutions
Experiments
Conclusion
Finally, we conclude our work

50. Conclusion
Propose a novel adaptive batch-based framework that guarantees a constant competitive ratio
Devise effective and efficient RL-based solutions to learn how to split the batches adaptively
Extensive experiments on both real and synthetic datasets show that our solution outperforms the state- of-the-arts.

51. Q & A
Thank You

52. Solution to the 1st Challenge
theoretical analysis
Assumption: Duration has upper bound 𝑪≥𝟐
Theorem 1 :
𝑪𝑹 𝒆𝒙𝒑𝒊𝒓𝒆 = 𝟏 𝑪−𝟏
Theorem 2 :
𝟏 𝑪−𝟏 ≤ 𝑪𝑹 𝒓𝒆𝒎𝒂𝒊𝒏 < 𝟐 𝑪−𝟐 (𝑪≥𝟑)
Unmatched nodes remain in the batch
Unmatched nodes expire

53. Solution framework Main idea of proof
Construct a good enough strategy using offline guide 𝑴 ∗ :
Pick the edge with the largest weight in 𝑴 ∗ and put it in a batch
Delete edges in 𝑴 ∗ that could not be matched in that batch
Pick and delete edges repeatedly until 𝑴 ∗ =∅
We have to bound number of deleted edges in each round
Two cases of deleted edges:
(𝒋−𝒊−𝟏) edges at most +
𝐂−𝟏−(𝒋−𝒊+𝟏) edges at most
= 𝑪−𝟐 edges at most

54. Solution to the 2nd Challenge
Problem: it is memory consuming
Optimization: quantization techniques
State
a=2
a=3
a=4
a=5 …
(10,10,2)
0.45
0.46
0.48
0.49
(10,11,2)
0.50
(11,10,2)
0.44
(11,11,2)
1 slot:𝑸( 𝟏𝟎~𝟏𝟏,𝟏𝟎~𝟏𝟏 ,𝟐~𝟑)
压缩后的表也画一下，complexity
1 slot:𝑸( 𝟏𝟎~𝟏𝟏,𝟏𝟎~𝟏𝟏 ,𝟒~𝟓)

55. Utility score is not damaged Even better in some cases
Experiments
Impact of quantization
Utility score is not damaged
Even better in some cases
Memory cost is
largely decreased
and remains stable

56. Spontaneous v.s. Batch-based1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 1 1 5 5 2 5 5
Now let’s see an example of spontaneous decision versus batch-based decision 2 6 6 6 6
The total utility is = 5
The total utility is = 8

57. Problem Statement Dynamic bipartite graph Online scenario
Arrival time 1 1 6
t = 1
Matching result:
𝑴={ 𝟑,𝟏 , 𝟒,𝟐 ,(𝟔,𝟓)} 2 2 2
t = 2 5 3
t = 3 3 3 3 4
t = 4 1 4 4 1 5 5
t = 5
In online scenario, the graph is dynamically changing as follow 2 2 6 6 4
t = 6
Utility score:
𝑼 𝑩,𝑴 =𝟐+𝟑+𝟐=𝟖