1. Distributed Dynamic Packet Scheduling for Handling Disturbances in Real-Time Wireless Networks
Tianyu Zhang1,3, Tao Gong2, Chuancai Gu1, Huayi Ji2, Song Han2, Qingxu Deng3, X. Sharon Hu1
1University of Notre Dame, USA
2University of Connecticut, USA
3Northeastern University, China
(Thank you chair for introduction) Good Afternoon, I would like to present paper entitled ... This is a joint work from UND, UConn and NEU.

2. Outline Motivation System model & related work
Distributed dynamic packet scheduling framework (D2-PaS)
Experimental evaluation
Here is a quick outline of my presentation,

3. Real-Time Wireless Networks (RTWNs)
Network-based Rehabilitation System
Real-Time Analytics Platform for Process Control
Nowadays, CPS are everywhere, from smart buildings to medical devices to automobiles. And as a critical role, (RTWNs) are fundamental to many CPS applications in a broad range of fields such as telerobotics, civil infrastructure monitoring and industrial automation.
Remote and Real-time Welding System
Cyber-physical Avatar

4. An Example: Mining Monitoring System
Sensors deployed underground mine
External disturbance: unexpected changes in temperature/pressure, etc.
Require more frequent monitoring/response
For example, in a mining monitoring system, a plenty of sensors are deployed in the tunnel to monitor the environment in real-time. Environment data sampled by sensors are sent to the monitering center on the ground periodicly. All thses devices are spatially distributed and connected over wireless network media to construct a RTWN.
From real applications, we found a fact that most RTWNs must deal with external disturbances in the environment such as detection of an emergency, sudden pressure changes or temperature changes. When such situation occurs, sensors must monitor and sample more frequently than usual and some necessary reponse is needed from the monitering center. For example, once we detected that the oxygen concentration was lower than a threshold value, we need to send a signal to all nodes carried by workers undergound to tell them you must leave this area immediately. Otherwise you are in danger.

5. Problem and Challenges
Requirements of a RTWN:
Environment data must be collected timely
Guaranteed data freshness
QoS: how well it satisfies real-time deadlines
Packet scheduling is critical
Challanges:
External disturbances occur unexpectedly
Network size getting bigger
From the example, we can see that there are two basic requirements of a RTWN. The first one is that environment data must be collected and sent out to the monitoring center timely. Secondly, data freshness also need to be guaranteed.
So, the Quality of Service offered by an RTWN is often measured by how well it satisfies the real-time deadlines of tasks running in the network.
In this case, packet scheduling plays a critical role in achieving the desired QoS.
Also, from the example, we can see that the environment is not always stable. External physical disturbances occur unexpectedly. This aggravates the packet scheduling problem. And with the network size getting bigger, the problem is slso being more chanllenging.

6. Design an on-line dynamic scheduling framework
What We Want to Achieve?
Design an on-line dynamic scheduling framework
Upon detecting a physical disturbance
Determine a dynamic schedule interval
Generate and distribute a temporary dynamic schedule
Meet as many periodic packet deadlines as possible
So, what we want to achieve in this paper is to design an on-line dynamic scheduling framework for handling disturbances occured in RTWNs.
Specifically, when a physical disturbance is detected and reported, firstly, we need to determine a dynamic schedule interval which can cover the disturbance duration. and then we need to generate and distribut a temporary dynamic schedule to all nodes in the network to handle this disturbance. To achieve the desired QoS, we want to meet as many periodic pakcet deadlines as possible when the disturbance is being handled.

7. Outline Motivation System model & related work
Distributed dynamic packet scheduling framework (D2-PaS)
Experimental evaluation
Next I will introduce the system model we used and some related works

8. Model Disturbances: Rhythmic Model
Nominal state
Rhythmic state
Period
Deadline t t
r0,m
r0,m+1
r0,m+R
r0,m+R+1
r0,m+R+2
...
d0,m
d0,m+1 t
Nominal state
Rhythmic state
Nominal state
First of all, we need to model the physical disturbances. There are many different models to capture workload changes. And in this paper we use the rhythmic task model proposed by professor Rajkumar's group.
In the rhythmic model, each task has two different states, the nominal state and the rhythmic state. When it is in the nominal state, it has a nominal period and nominal relative deadline. When it enters the rhythmic state, its period and relative deadline are first reduced and then gradually return to their nominal values.
Note that our D2-PaS framework is not limited to the rhythmic task model but can be applied to any task model that can provide on-line workload changing patterns to capture disturbances.
Kim, Lakshmanan and Rajkumar, ICCPS, 2012

9. System Model Infrastructure of RTWN Task model
A gateway, sensors, relay nodes and actuators sharing a channel
Task model
Broadcast task and unicast tasks (periodic and rhythmic)
Routing path: every task passes through gateway
The network model studied in this paper consists of a gateway, multiple sensors, relay nodes and actuators. All these devices are wirelessly connected sharing a unichannel.
The task set running in the network consists of a broadcast task and unicast tasks. All unicast tasks are periodic task and we further assume that during the network operation, at any time there is at most one periodic task can enter its rhythmic state. That is, there is at most one rhythmic task but this rhythmic task can be from any node and any periodic task.
The broadcast task is sent out from the gateway to all nodes for propagating some necessary information.
Every unicast task is sent out from a sensor node, passes through the gateway and arrives at an acutator node.

10. Existing Work: Static Approach
Communication schedule stays unchanged once constructed and distributed
Pros: simple and deterministic
Cons: cannot handle on-line workload changes
Now I will introduce some related works.
In current wireless networks, the most commen scheduling framework is static approach. That is, communication schedule of the network stays unchanged once it is constructed and distributed.
The pros of doing this is that it is simple to realize and it supports deterministic real-time communication.
However, apparently, static approaches cannot handle on-line network workload changes caused by physical disturbances.

11. Existing Work: Centralized Dynamic Approach
Static S
Dynamic S
Start using dynamic schedule
Static S
Go back to static schedule
Generate a dynamic schedule S
Broadcast S
Dynamic schedule generated t
Disturbance detected
The dynamic schedule must satisfy that
Each rhythmic packet must meet its deadline
Number of dropped periodic packets is minimized
# of updated slots satisfies an upper bound
To overcome the disadvantage of static approaches, we proposed a centralized dynamic approach in our previous work which called OLS.
In the centralized framework, when the disturbance is detected, the gateway generates a temporary dynamic schedule for the network in the rhythmic mode, and broadcasts the differences between the dynamic and static schedule to each node. And then the network starts using the dynamic schedule. After the rhythmic duration, the network goes back to reuse the static schedule.
The dynamic schedule generated by the gateway must satisfy that each rhythmic packet must meet its deadline, this is to guarantee the disturbance can be handled properly.
And, the number of dropped periodic packets is minimized to achieve the desired QoS.
And last, the number of updated slots must satisfy an upper bound. This is because that the updated slots in the dynamic schedule must be piggybacked to a broadcast packet and propagated to all nodes, but the payload size of a broadcast packet is always bounded. So the maximum allowed number of updated slots is limited.
Hong, Hu, Gong and Han, ECRTS, 2015

12. Drawbacks of OLS Suffer from the limit imposed on # of updated slots
More periodic packets have to be dropped
Incur high latency for generating dynamic schedules
Latency grows as the size of the network increases
Observation:
The gateway undertakes all the work to handle disturbances
Leverage local computing capability to design a distributed dynamic packet scheduling framework
Although compared with static approaches, the centralized approach can handle disturbances, it has two drawbacks. First of all, it suffers from the limit imposed on the number of updated slots. If the number of updated slots exceeds the payload size of a broadcast packet, more periodic packets have to be dropped.
Secondly, the centralized approach tends to incur high latency for generating dynamic schedules, and the latency grows quickly as the size of the network increases. If the updated schedule cannot be generated before the next broadcast slot, the rhythmic event cannot be handled properly or has to be delayed until the next broadcast slot.
We observed that the reason that the centralized approach has the limitation on the number of updated slots is the choice that the gateway undertakes all the work to handle disturbances.That is. all other nodes only need to follow the schedule generated by the gateway. Such an approach is equal to assume that device nodes have no local computing capability, however this is not true for RTWNs nowadays.
In fact, although the device nodes donot have the same computing capability as the gateway, they can make some basic computations. So, we are going to leverage such local computing capability and design a distributed scheduling framework to achieve better performance

13. Outline Motivation System model & related work
Distributed dynamic packet scheduling framework (D2-PaS)
Experimental evaluation
Next I will describe our D2-PaS framework in detail

14. D2-PaS Framework
Broadcast dropped packets and the rhythmic task info N
... N
...
Gateway
Local schedule
1. Determine the duration
tst
tsw
Rhythmic mode duration
Instead of broadcasting a schedule, we only broadcast the indices of the dropped packets
Each node generates static schedule
Each node generates dynamic schedule
Each node generates static schedule
2. Check the schedulability
Network set up
Disturbance detected
3. Determine the dropped packets
Here is an overview of our distributed dynamic packet scheduling framework, D2-PaS.
After network initialization, when a broadcast packet is received from the gateway, each node generates its own schedule locally according to some scheduling policy and then follows this local schedule to operate.
When disturbance is detected, the sensor node sends a rhythmic event report to the gateway.
Then the gateway firstly determine the rhythmic mode duration and next it checks the schedulability of the network in the rhythmic mode. If the system is overloaded, the gateway need to determine which periodic packets to be dropped to guarantee the deadlines of the rhythmic packets. Then it broadcast the information of dropped packets and the rhythmic task, that is, which task is going to enter the rhythmic state to all nodes. If the system is still schedulable even a rhythmic task enters the rhythmic state, the gateway only need to broadcast which task is the rhythmic task to all nodes. Then each node can generate the dynamic schedule using the information received from the gateway to handle the disturbance.
And after the disturbance is completely handled, the network goes back to the nominal mode and each node generates the static schedule.
So, in this way, instead of broadcasting the entire updated schedule, D2-PaS only need to broadcast the indices of the packets to be dropped when the system is overloaded.

15. Challenges and Ideas Local schedule generation
Each node cannot construct and store the entire schedule for the hyperperiod in the nominal mode
Local schedule computation should not occur when the node is supposed to send or receive packets
An efficient method is needed at the gateway to determine which packets to drop
Scheduling in the rhythmic mode
Construct one segment of the entire schedule at a time
Perform local schedule generation in the idle slots of each node
To make sure that the D2-PaS works properly, we need to tackle a few challenges. First, each node cannot construct and store the entire schedule for the hyperperiod in the nominal mode considering This is mainly because of its limited memory.
Secondly, Local schedule computation should not occur when the node is supposed to send or receive packets because generating the schedule takes time and each node only has limited computing capability,
Third, An efficient method is needed at the gateway to determine which packets to drop
The first two challenges forcus on local schedule generation at local nodes and the third one is about the scheduling in the rhythmic mode at the gateway.
To tackle the first challenge, we design a principle that each node only construct one segment of the entire schedule at a time.
To tackle the second one, we follow the principle that each node only perform local schedule generation in its own idle slots.
Then the questions need to be answered include what is the segment, how long and how to generate it incrementally.
What is the segment, how long and how to generate it incrementally?

16. Local Schedule Example
Broadcast transmissions
(2,3)
(2,1)
(1,2)
(1,1)
(0,1)
(3,1)
(3,2)
(2,2)
(0,2) 1 2 3 4 5 6 7 8 9 10 11 12 13
(2,2)
(2,3)
(3,1)
(3,2)
1st SS3
2nd SS3
3rd SS3
Schedule segment (SS) definition
Start at idle or broadcast
End at busy or before broadcast
Length of SS ≤ 2 × # of tasks passing the node
Here I use an example to show the local schedule generation.
This is the entire EDF schedule for the whole network. ...
So here we have the local schedule for this node. Now we can see that there are a lot of idle slots. So we can use these idle slots to generate local schedules.
If we want to do so, we need to define the schedule segment. In a word, we want each schedule segment start at an idle slot or a broadcast slot and end at a busy slot or a slot right before a broadcast slot.
So in this example, the first schedule segment starts from slot 0 to slot 2....Here the reason we want each schedule segment to end before a broadcast transmission is that broadcast transmission can bring some updated information. So any schedule slot after a broadcast transmission is probably need to be changed. So we do not nodes to recompute any schedule. And also we have measured in a real testbed that broadcast transmissions take much less time than any other transmissions. The rest time of a broadcast transmission slot is long enough to operate a schedule generation. So in a word, any broadcast slot can also be seen as an idle slot for the node.
Also, to make sure that the schedule segment generation can be completed within one idle slot, we proved that the length of schedule segment can be bounded. Details can be found in our paper.

17. Scheduling in the Rhythmic Mode
Local schedule generation ensures correctness in the nominal mode
Where is the end point tsw?
1. Determine the duration
tst
tsw
Rhythmic mode duration
Each node generates schedule locally
2. Check the schedulability
Network setted up
Disturbance detected
3. Determine the dropped packets
Now with the local-schedule generation approach, the network can work properly in the nominal mode. So next we need to discuss the network in the rhythmic mode.
From the overview of the execution model, the first question need to be answered by the gateway is to determine the rhythmic mode duration, that is, where is the end point of the rhythmic mode?

18. Determine the End Point
Observations:
Earlier end points incur shorter time overhead
A special time slot yields the minimum number of dropped periodic packets among all end point candidates
This special point is called No-Carry-over-Packet (NCoP)
NCoP point
Deadline miss
For the end point selection, we have the following observations. The first one is that, earlier end points incur shorter time overhead. This is easy to get. And what is more interesting is we found that a special time slot can yield the minimum number of dropped periodic packets among all end point candidates. This special point is called No-Carry over packet point, denoted as NCoP.
Here I use an example to show where is the NCoP. Consider three tasks and \tau_0 enters its rhythmic state at this point. Then we generate the EDF schedule using new periods and deadlines for the rhythmic task. Note that if a packet misses its deadline, its unfinished part is dropped but the finished part is kept. Then the first occured idle slot is the NCoP point which yields the minimum number of dropped periodic packets, we proved this in our paper.
And then we can directly select the NCoP point as the end point.

19. Determine Dropped Packets
Late-job minimization
Each job Jj = <rj, ej, dj, wj>
A job is late if it is not finished at the deadline
Minimize the total late job's weight
Dropped packets minimization
Each packet Xi,k = <ri,k, Hi, di>
Weight of each R-packet wi = np +1; Weight of each P-packet wi = 1
A packet is dropped if it s not finished at the deadline
Minimize # of dropped packets
Determing the minimum # of dropped packets
Determing the minimum # of
late jobs
The optimal schedule in terms of # of dropped packets can be found in O(n5) where n is # of packets
We use a heuristic with O(n2) and the results are good in both simulation and testbed
After the end point is selected, the next question is to determine which periodic packets to be dropped within this time duration.
In the real time scheduling literature, the problem of minimizing # of late jobs is well studied. So here what we need to do is to map our problem of determing the minimum number of dropped packets to the problem of minimizing the number of late jobs.
Here I would like to skip the details on how we map. But after the mapping, The optimal schedule in terms of # of dropped packets can be found in O(n5) where n is # of packets.
But considering an on line approach, we want the algorithm be more efficient. So we use a heuristic with O(n2) and the results is shown to be good enough in both simulation and testbet.
Lawler, E. L. Annals of Operations Research, 1990

20. Outline Motivation System model & related work
Distributed dynamic packet scheduling framework (D2-PaS)
Experimental evaluation
Simulation
Testbed
Next I will briefly show our experimental results. We evaluate D2-PaS both in simulation and on a real testbed

21. Simulation Setup Task sets Baseline algorithm
Utilization from 0.5 to 0.9
Parameters of each task: # of hops -> [2, 10]; period and deadline -> [15, 50]
Randomly select one task to be the rhythmic one
Workload of the rhythmic task is controlled by the # of rhythmic packets
Baseline algorithm
OLS: centralized method
In our simulation study, we use random periodic task sets generated according to a target nominal utilization. We vary the task set utilization from 0.5 to 0.9.
Each periodic task is generated based on the number of hops, period and relative deadline.
After the random task set is generated, we randomly pick one of them to be the rhythmic task. And we vary the number of elements in its period vector to control the workload of rhythmic task.
We compared our D2-PaS to the state-of-the-art approach OLS.

22. Average Drop Rate Based on Simulation
D2-PaS-4
D2-PaS-10
D2-PaS-16
OLS-4
OLS-10
OLS-16
This is one of the performance Comparison results and it is also the most representative one.
This figure shows the average drop rate as a function of task set nominal utilization. All solid curves denote our D2PaS and other dash lines denote OLS. Different color denotes different workload of the rhythmic task. That is, here 4, 10, 16 denote the number of elements in the rhythmic task's period vector. So for each curve, we can see that with the workload of rhythmic task getting higher, the average drop rate is also getting higher.
From the figure, it is clear that D2-PaS performs much better than OLS under all different parameter settings and D2-PaS has a very low drop rate when the nominal utilization is less than 0.9. This is mainly because that, without the limitation on the number of updated slots of the static schedule, D2-PaS can schedule both the rhythmic task and all other periodic tasks in most cases when the task set utilization is not very high.
D2-PaS performs much better than OLS from 40% to 95%

23. Testbed-Based Experiment
Use TI CC2538 EMKs to form a mesh network
Implement in OpenWSN 6TiSCH (an IoT protocol)
Modify the network stack to support D2-PaS scheduler

24. Testbed-Based Experiment
8 CC2538 wireless devices
JTAG debuggers
Here is a picture showing the running system.
We set up 8 CC2538 wireless devices. 1 as gateway and the other 7 are the mesh nodes
4 of the JTAG debug probes are connected to 4 devices so the running status of the device is collected (including the scheduling and data packet information).
A separate JTAG is setup specifically for gateway.
The devices form the network, run the D2-PaS scheduler and report the running result.

25. Testbed-Based Experiment
Runtime status is recorded and analyzed by our Cross-Device Testing and Reporting System (CD-TRS)
Running status collected by JTAG probes
Automated analysis on the recording to verify correctness
Verified that D2-PaS is capable of handling unexpected disturbances
Here shows a screen capture of the analysis result. The JTAG probes collect running status and retrieve logs from the devices in the runtime.
An automated analyzer then verify the correctness of the running schedule, also in the runtime.
Using the real testbed experiment, we validated the correctness of our proposed D2-PaS framework.

26. Summary and Future Work
Introduce an on‐line packet scheduling problem
Propose a distributed dynamic framework for handling disturbances
Validate by simulation and testbed
Future work
Consider unreliable networks
Support multiple communication channels
Design fully distributed framework without the gateway

27. Thank you! Questions?