1. Adjusting Forward Error Correction with Quality Scaling for Streaming MPEG
Huahui Wu, Mark Claypool, Robert Kinicki
Computer Science,
Worcester Polytechnic Institute
Welcome
Title, Name

2. Outline Introduction  Model & Algorithm Experiments Conclusion
NOSSDAV 2005
11/28/2018

3. Introduction-Motivation
Video Frames
Ideal figure of media streaming, server captures frames and sends to network, client reads the pipe and play it.
This figure assumes enough capacity, but that’s not always true. In most cases, current network has limited capacity
Then the server needs to scale down it’s streaming bitrate, and we call it media scaling. There are some ways to do scaling. And here is one example to do that. The server discards some frames before sending them to the network. By that means, the server reduces the bitrate, which is now suitable for the network.
So, that’s about the scaling, and another issue of media streaming is repair.
In the illustration so far, the network cloud is clear, and the data passes through the network without any problem. But that’s not always true since we might have packet loss in the network. When there are more packets than what the network can handle, the network will drop some packets randomly. Or there are noises in the physical channel, which leads to bits error.
For this reason, different types of repair techniques are used to protect the data from loss as an armor. Besides the original video flow, now there is another repair flow which is generated in server and sent to the network. When there are packet losses in the video flow, the client will use the repair data to recover the video.
Maybe you’ve already noticed, the repair flow adds more data into the network, and the network only has limited capacity as we showed here. So the server has to scaling more in the video to give some bandwidth to the repair flow. That means, repair is good, but it is not free, the more repair data you add, the less video data you can send. There is a trade-off between repair and scaling when the capacity is constrained.
Forward Error Correction
NOSSDAV 2005
11/28/2018

4. Introduction-Overview
Video Quality
Optimal Point
Adjusting FEC with Quality Scaling
Given Network and Application Environment
For each valid FEC and scaling combination
Measure the video quality
Find the optimal point
More FEC and More Scaling
NOSSDAV 2005
11/28/2018

5. Outline Introduction Model & Algorithm Experiments Conclusion
NOSSDAV 2005
11/28/2018

6. Algorithm
Exhaustively search all FEC and Scaling combination and look for the optimal point
For each FEC and Scaling combination
[S1] Estimate video frame sizes
[C] Compute streaming bitrate and make sure it’s under capacity constraint
[S2] Estimate quality scaling distortion
[S3] Use frame sizes and FEC amounts to get successfully frame transmission rate
[S4] Compute playable frame rate
[S5] Compute overall video quality
Measured by Distorted Playable Frame Rate
NOSSDAV 2005
11/28/2018

7. System Layers and Parameters
Application
Presentation
Network
NOSSDAV 2005
11/28/2018

8. [Frossard+ 01, Sakazawa+ 96]
Video Frame Sizes [S1]
Assumption 1: Video frame sizes vary exponentially with the quantization level
[Frossard+ 01, Sakazawa+ 96]
Errors are less than half packet
NOSSDAV 2005
11/28/2018

9. Capacity Constraint [C]
Total streaming bitrate:
where G is the constant GOP rate
Should be no more than the capacity constraint T
NOSSDAV 2005
11/28/2018

10. Overall Video Quality [S2~S5]
Two distortion factors
Quality Scaling
Caused by a high quantization value
Appears visually as coarse granularity in every single frame
Measured by ITS-VQM [Pinson+ 04]
Frame Loss
Caused by network packet loss
Appears jerkiness in the video playout
Measured by Playable Frame Rate
Overall Quality
Distorted Playable Frame Rate
[Wu+ 03 NOSSDAV]
NOSSDAV 2005
11/28/2018

11. Distortion from Quality Scaling [S2]
Assumption 2:
Quality scaling distortion varies exponentially with the quantization level
[Frossard+ 01]
NOSSDAV 2005
11/28/2018

12. Successfully Transmission Prob. [S3]
Media-Independent FEC
Reed-Solomon codes [Reed+ 60]
NOSSDAV 2005
11/28/2018

13. Playable Frame Rate [S4]
Total Playable Frame Rate
[Wu+ NOSSDAV03]
NOSSDAV 2005
11/28/2018

14. The Operations Research Equation
Overall Quality [S5]
Distorted Playable Frame Rate
The Operations Research Equation
NOSSDAV 2005
11/28/2018

15. Algorithm
Exhaustly search all FEC and Scaling combination and look for the optimal point
For each FEC and Scaling combination
[S1] Estimate video frame sizes (SI, SP, SB)
[C] Compute streaming bitrate and make sure it’s under capacity constraint
[S2] Estimate quality scaling distortion (D)
[S3] Use frame sizes and FEC amount to get successfully frame transmission rate (qI, qP, qB)
[S4] Compute playable frame rate (R)
[S5] Compute distorted playable frame rate (RD)
NOSSDAV 2005
11/28/2018

16. Outline Introduction Background Model & Algorithm Experiments 
Conclusion
NOSSDAV 2005
11/28/2018

17. Methodology Built a function RD() Built an optimization program
Returns the distorted playable frame rate
Built an optimization program
TCP-Friendly Bitrate Constraint
Searches repair and scaling levels
Returns the highest RD()
Compared four FEC choices
Non-FEC
Small Fixed FEC
Large Fixed FEC
Adjusted FEC
NOSSDAV 2005
11/28/2018

18. System Setting GOP: IBBPBBPBBPBBPBB Network Layer [Jaiswal+ 04]
Application Layer
tRTT
50 ms NP
4 frames per GOP s
1 Kbyte NB
10 frames per GOP p
0.01 to 0.04 RF
30 frames per sec
GOP: IBBPBBPBBPBBPBB
NOSSDAV 2005
11/28/2018

19. Distorted Playable Frame Rate
NOSSDAV 2005
11/28/2018

20. Conclusion Summary Future works
Analytically captures the frame dependencies
Derives analytical models for streaming video with FEC and quality scaling
Use OR algorithm to optimize the quality
Future works
Implementation
Other quality scaling techniques (FGS)
Quality scaling vs. temporal scaling
NOSSDAV 2005
11/28/2018

21. Adjusting FEC with Quality Scaling for Streaming MPEG
Huahui Wu, Mark Claypool, Robert Kinicki
Computer Science, Worcester Polytechnic Institute
Questions?
Welcome
Title, Name, Committee members

22. Outline Introduction Background  Model Algorithm Experiments
Conclusion
NOSSDAV 2005
11/28/2018

23. Video Compression Standard
MPEG
Popular compression standard
Intra-compression and inter-compression
Three types of frames: I, P and B
Group Of Pictures (GOP)
The first part of the background is video compression standard.
Here I use MPEG as the example, but all the research models and results should hold for other video standard such as H.26X.
In MPEG, there are two types of compression techniques and they are intra-frame compression and inter-frame compression. The intra-frame compression is very similar to the techniques used in JPEG or other picture format. It uses the similarities in a single frame. The inter-frame compression, however, use the similarities among a sequence of pictures. With these two compression methods, there are three types of frames in MPEG and they are I frame, P frame and B frame. I frame, where I stands for Intra-coded, uses intra-frame compression only and it is independent to other frames. When the client get an I frame, it can decode it and play it immediately. P frame, where P stands for Predictive-coded, uses both of intra and inter-frame encoding. It only encodes the difference from its preceding I or P frame. So P frame is dependent on its preceding I or P frame. B frame, where B stands for Bi-directionally predictive coded, also uses both of intra and inter-frame encoding. Different to P frame, it only encodes the difference from its preceding and succeeding I or P frame.
MPEG video typically repeats a sequence of I, P and B frames for the duration of a video stream and it is called as Group of Pictures (or GOP). This figure shows a sample GOP, where the second I frame denotes the start of next GOP. Because of the dependencies of I, P and B frames, the loss of P frame can make other P and B frame useless, and the loss of I frame can make the whole GOP useless. So we can tell, I frame is the most important and P frames are more important than B frames.
The subscript
NOSSDAV 2005
11/28/2018

24. Quality Scaling Quality Scaling 24KB, 10KB, 5KB
Adaptive Quantization Level
Another type of media scaling is quality scaling.
In the MPEG encoder, there is a parameter called quantization level.
This parameter controls the accuracy of data compression.
If it is large, the compression rate is high and the quality is low.
If it is small, the compression rate is low and quality is good.
Here are three pictures encoded from same source but with different quantization level.
The quantization level increases from left to right.
The sizes are 24KB, 10KB and 5KB respectively and the quality decreases.
So by adjusting the quantization level, quality scaling can be used to reduce the bitrate as well as temporal scaling.
24KB, 10KB, 5KB
NOSSDAV 2005
11/28/2018

25. Forward Error Correction (FEC)
Media-Independent FEC
Reed-Solomon codes [Reed+ 60]
Another part of the background is Forward Error Correction (FEC), which is the repair methods I study in this research.
The idea of FEC is add redundancies to the original data before sending. When there is loss, the redundancies will be used to recover the data.
There are two kinds of FEC, media-independent FEC and media dependent FEC, which I will talk about later.
Media-independent FEC doesn’t need to know the data content. It uses mathematic algorithm to generate redundant parities for the original data.
Reed-Solomon codes is a popular media-independent FEC method and this picture shows how it works.
In the first row, the original video frame is divided into K packets.
Then in the second row, the FEC generator uses a mathematical function to create N-K redundant FEC packets and now we have N packets in total.
All these N packets will be sent to a lossy network and some of them might get lost as shown in the third line.
Here I only show one packet is loss, but it could be more.
Then in the client side, which is the fourth row, the FEC decoder will check how many packets among those N packets are received successfully. If there are K or more packets are correct, the FEC decoder is able to reconstruct the original K data packets, and then the original video frame.
NOSSDAV 2005
11/28/2018

26. Capacity Constraint TCP-Friendly Flow [Padhye+ 00] Bottleneck Capacity
Dial up: 56 Kbps
DSL: 1.5 Mbps (Verizon)
Cable Modem: 3 Mbps/384 Kbps (Charter)
Video is often larger than 1.5 Mbps
The third background topic is capacity constraint.
There are two types of capacity constraint I am dealing with.
The first constraint comes from TCP-Friendly idea.
Basically, we can divide the flows in network into two types, well behaved and not well behaved.
The well behaved flow also check the status of the network, when there is a congestion, it will adjust its bitrate to reduce the probability of congestion. Usually, a flow uses TCP is well behaved. The bad behaved flow doesn’t care the network. It keeps sending data into the network no matter whether there is a congestion.
Some researchers are proposing the TCP-Friendly idea to deal with such bad behaved flows. They suggest, the network routers should be able to detect the flow’s behavior and punish the flows that doesn’t behave like a TCP flow. So the flow will be force to adjust its bitrate to a conformant TCP flow running under comparable network situation. Padhye et al derived an equation for modeling TCP’s bitrate as in the slides, where s is the packet size … And this equation will be used as the TCP-Friendly capacity constraint.
Another type of capacity constraint is the bottleneck capacity. Many Internet users are home users. There are several ways for the home user to access the Internet: Dial-up, DSL and cable modem. All these accesses have dierent bandwidth limits. Dial-up was a only choice for home users ten years ago, and it has a bandwidth limited up to 56 Kbps. Cable Modems and DSLs are the two major choices for higher bandwidth consumers today. Their capacities are several hundred Kbps or a few Mbps.
NOSSDAV 2005
11/28/2018

27. Playable Frame Rate [S4]
Playable Frame Rate (PFR) of I frames
NOSSDAV 2005
11/28/2018

28. Playable Frame Rate [S4] (cont.)
PFR of P frames
product
NOSSDAV 2005
11/28/2018

29. Playable Frame Rate [S4] (cont.)
PFR of B frames
NOSSDAV 2005
11/28/2018

30. How RD works Repair Method D R RD Q (user) QAFEC 0.17 28.55 23.78 6.89
SFEC
0.20
23.58
18.90
4.44
LFEC
0.44
30.00
16.93
3.89
NFEC
0.28
20.17
14.61
3.50
NOSSDAV 2005
11/28/2018

31. Distorted Playable Frame Rate
NOSSDAV 2005
11/28/2018

32. Introduction-Motivation
Video Frames
Ideal figure of media streaming, server captures frames and sends to network, client reads the pipe and play it.
This figure assumes enough capacity, but that’s not always true. In most cases, current network has limited capacity
Then the server needs to scale down it’s streaming bitrate, and we call it media scaling. There are some ways to do scaling. And here is one example to do that. The server discards some frames before sending them to the network. By that means, the server reduces the bitrate, which is now suitable for the network.
So, that’s about the scaling, and another issue of media streaming is repair.
In the illustration so far, the network cloud is clear, and the data passes through the network without any problem. But that’s not always true since we might have packet loss in the network. When there are more packets than what the network can handle, the network will drop some packets randomly. Or there are noises in the physical channel, which leads to bits error.
For this reason, different types of repair techniques are used to protect the data from loss as an armor. Besides the original video flow, now there is another repair flow which is generated in server and sent to the network. When there are packet losses in the video flow, the client will use the repair data to recover the video.
Maybe you’ve already noticed, the repair flow adds more data into the network, and the network only has limited capacity as we showed here. So the server has to scaling more in the video to give some bandwidth to the repair flow. That means, repair is good, but it is not free, the more repair data you add, the less video data you can send. There is a trade-off between repair and scaling when the capacity is constrained.
Forward Error Correction
NOSSDAV 2005
11/28/2018

33. Successfully Transmission Prob. [S3]
Media-Independent FEC
Reed-Solomon codes [Reed+ 60]
NOSSDAV 2005
11/28/2018