1. Evaluation of packet video quality in real time using Random Neural Networks Samir Mohamed Gerardo Rubino IRISA, Rennes, FRANCE

2. 2 Workshop RNN, ICANN’02, Madrid, 27/8/02 The problem as perceived by the receiverHow to automatically quantify the quality of the stream, as perceived by the receiver? Source Receiver stream of voice, music, video video, multimedia, … IP network

3. 3 Workshop RNN, ICANN’02, Madrid, 27/8/02 OUTLINE 1.Subjective tests 2.Objective tests 3.Our approach in 5 steps 4.The obtained performances 5.ANN vs RNN 6.Application: analyzing parameters impact 7.A view of our demo tool 8.Related work in audio 9.Ongoing research

4. 4 Workshop RNN, ICANN’02, Madrid, 27/8/02 1: The “voie royale” To quantify the quality at the receiver side … just put a human there, give her/him a scale (say, from 1 to 10), and ask her/him to evaluate the quality of a (short) part of the stream (say, some seconds).

5. 5 Workshop RNN, ICANN’02, Madrid, 27/8/02 (1:) The “voie royale” (cont.) Better: –put n humans at the receiver side, –ask them to evaluate a (short) given sequence, –take the average as the quantified quality. Still better: –give to the set of humans the sequence to evaluate plus several other sequences, in order to allow each member to adjust her/his scale

6. 6 Workshop RNN, ICANN’02, Madrid, 27/8/02 (1:) Subjective tests The previous procedure is in fact standardized: see, for instance, the norm ITU-R BT.500-10 (March 2000). Basically, the procedure is as previously described + a statistical filter of the results. Goal of the statistical filter: to eliminate bad observers. Bad observer: “one disagreeing with the majority”

7. 7 Workshop RNN, ICANN’02, Madrid, 27/8/02 (1:) Drawbacks The previous procedure gives “the truth” but doesn’t do the job automatically (so, a fortiori, not in real time neither). Moreover, performing subjective tests is costly –in logistics (you need about 20 people with appropriate characteristics) –in resources (you need an appropriate place or environment to put your set of human subjects) –and in time to perform the tests.

8. 8 Workshop RNN, ICANN’02, Madrid, 27/8/02 (1:) Drawbacks (cont.) Suppose you decide to analyze the way some factor (the loss rate in the network, for instance) affects quality: in many points for many sequences VERY EXPENSIVE –you must then evaluate the function q = f(lr) where q is the quality and lr is the loss rate in many points and for many sequences: VERY EXPENSIVE. TOO EXPENSIVE TO BE DONESuppose you want now to study q = g(lr,br,d) where br is the source bit rate and d the mean end-to-end delay: TOO EXPENSIVE TO BE DONE

9. 9 Workshop RNN, ICANN’02, Madrid, 27/8/02 2: Another possibility The other way is obviously to look for an explicit formula (or algorithm) giving q as a function of the considered factors (lr, d, etc.) in, say, O(1) time. This appears to be a formidable task, because of two factors: –no formal definition of what we want to quantify (quality) is available –the “intuitive concept”of quality depends a priori on too many factors and in a complex way

10. 10 Workshop RNN, ICANN’02, Madrid, 27/8/02 (2:) Objective tests degradationThere is an area called “objective tests” where different procedures are proposed to evaluate the degradation of a sequence. They consist of specific metrics that compare the original and the degradated streams. Even if this is not the goal here, the results of these objective tests are not satisfactory so far. How do we know? by comparing the results obtained using these techniques with “the truth”, that is, with those coming from subjective tests.

11. 11 Workshop RNN, ICANN’02, Madrid, 27/8/02 (2:) An example The MNB metric: –developped at the US Dep. of Commerce, 1997 –for voice (VoIP applications) –based on a cognition model –has been shown to behave correctly in some cases In next slide, some results of a test against subjective evaluations (from T. A. Hall, Objective speech quality measures for Internet telephony, Proc. of SPIE 2001).

12. 12 Workshop RNN, ICANN’02, Madrid, 27/8/02 (2:) Performance of MNB 2

13. 13 Workshop RNN, ICANN’02, Madrid, 27/8/02 (2:) The only exception There is one trial of measuring quality without referring to the original sequence: www.itu.int ITU “E-model” www.itu.int It has been proposed for the specific case of VoIP applications. However, the comparison with subjective evaluation says that the performance of this metric can be very poor (in next slide, some results from T. A. Hall, Objective speech quality measures for Internet telephony, Proc. of SPIE 2001).

14. 14 Workshop RNN, ICANN’02, Madrid, 27/8/02 (2:) Performance of the E-model

15. 15 Workshop RNN, ICANN’02, Madrid, 27/8/02 (2:) An example in video: the ITS metric Source: S. Voran and S. Wolf, The development and evaluation of an objective video quality assesment system that emulates human viewing panels, in IBC 1992.

16. 16 Workshop RNN, ICANN’02, Madrid, 27/8/02 3a: Our approach: first step Our goal is to take, in some sense, the best of subjective and objective approaches. first stepAs a first step, we select a set of factors or parameters that we think are important to the final perceived quality. Even if this is an a priori task, it is not a difficult one. For each parameter, we select a few representative or important values. Our goal here is to discretize the problem.

17. 17 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3a:) Our approach: first step (cont.) In our case, we selected 5 parameters: –BR: Bit Rate –BR: Bit Rate. The normalized rate of the encoder. In our environments, we considered 4 possible values (256, 512, 768 and 1024 KBps), normalized with respect to the maximal value. –FR: Frame Rate –FR: Frame Rate, in fps (frames per sec). The rate at which the original stream is encoded. Four selected values: 6, 10, 15 and 30 fps.

18. 18 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3a:) Our approach: first step (cont.) –LR: loss rate –LR: loss rate, in % (loss probability). Selected values: 0, 1, 2, 4 and 8 %. –CLP: number of consecutive loss packets –CLP: number of consecutive loss packets. We consider packets dropped in bursts of size 1, 2, 3, 4 or 5. –RA: ratio between intra macro-blocs to inter macro-blocs –RA: ratio between intra macro-blocs to inter macro-blocs. It (indirectly) measures the redundancy in the sequence. We used for this parameter five values between 0.05 and 0.45.

19. 19 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3a:) Our approach: first step (cont.) Observe that there can be interactions (in general, complex) between the parameters –for instance, here, RA is used by the encoder to protect the stream against losses (measured by LR) It must also be underlined that our approach does not depend on the specific set of chosen parameters. source parametersnetwork parametersObserve that BR, FR and RA are source parameters, and that LR and CLP are network parameters.

20. 20 Workshop RNN, ICANN’02, Madrid, 27/8/02 3b: Our approach: second step We have a set of parameters P = {  1,  2, …,  P } For parameter i we have a set of possible values:  i  {v i1, v i2, …} Any vector of the form (v 1i, v 2j, …, v Pk ) is then called a configuration.

21. 21 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3b:) Our approach: second step (cont.) In our experiments, we had P = 5 parameters The # of selected values for them were respectively 4, 4, 4, 5, 5, leading to 4  4  4  5  5 = 1600 configurations Call C the set of all possible configurations. second stepThe second step consists of selecting a reduced part of C, trying to have a “good coverage” of this set.

22. 22 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3b:) Our approach: second step (cont.) In our experiments, we selected about 100 configurations. The method we followed was not to use something like a low discrepancy sequence of points in a hypercube, but –to take into account the characteristics of the parameters –and the extreme values. Call SC the set of selected configurations

23. 23 Workshop RNN, ICANN’02, Madrid, 27/8/02 3c: Our approach: third step third stepFor the third step, we must be able to reproduce an environment (source + network) where the selected parameters can be put in any configuration we want. In our case, we achieved this using simulators and appropriated controlled (lab conditions) networks. The third step consists of reproducing each configuration  from the set SC and to send a fixed original sequence  from source to receiver.

24. 24 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3c:) Our approach: third step (cont.) 11 11 22 22 33 33 SC = {  1,  2, … }

25. 25 Workshop RNN, ICANN’02, Madrid, 27/8/02 3d: Our approach: fourth step The result is a set of versions of , each having encountered different conditions at the source and in the network. We have now a set of sequences (in our tests, about 100 sequences), and a configuration associated with each one. fourth step by definitionThe fourth step consists of performing a standard subjective test on each sequence, to build a value assumed to be, by definition, the quality of the sequence.

26. 26 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3d:) Our approach: fourth step (cont.) In symbols, we have S sequences {  1,…,  S } and, associated with sequence i, the configuration (x i1, x i2,…, x iP ). In other words, x ik is the value of the kth parameter which led to the sequence  i at the receiver. Together with this, we have the quality value of each version, coming from the subjective test;  i is the value of  i.

27. 27 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3d:) Our approach: fourth step (cont.) 11 22 33 … LR2%0%1%… FR15630… …  3.52.53.2…

28. 28 Workshop RNN, ICANN’02, Madrid, 27/8/02 3e: Our approach: fifth (last) step randomlyThe S sequences are now divided into two sets, randomly. For instance, our 100 sequences were divided into one set of about 80, and one set of about 20. To simplify, assume we renumber things such that the first set is {  1, …,  K }. The idea is then to train a Neural Network (NN) to learn function with P inputs and 1 output, which associates with the input (x k1, x k2,…, x kP ) the output  k, for k = 1,…,K.The idea is then to train a Neural Network (NN) to learn “the” function with P inputs and 1 output, which associates with the input (x k1, x k2,…, x kP ) the output  k, for k = 1,…,K.

29. 29 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3e:) Our approach: fifth (last) step (cont.) The second set of sequences is then used to validate the obtained NN (standard approach). The hope is the following: if we give to the function an input (x 1, x 2,…, x P ) which is not in the data base (taht is, not in SC ), the output y should be close to the subjective evaluation of any sequence degradated through a system (source + network) where the configuration was (x 1, x 2,…, x P ).

30. 30 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3e:) Our approach: key implicit assumption Our approach implies the following implicit assumption: For any sequence  (or for any sequence  belonging to a given family or class), the subjective quality depends only on the values of the set of chosen characterizing parameters  belonging to a given family or class), the subjective quality depends only on the values of the set of chosen characterizing parameters.

31. 31 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3e:) Our approach: key implicit assumption (cont.) This means in particular that given, say, three video sequences perhaps with very different contents, if the system configuration is the same (source bit rate, loss rate, …) then the perceived quality will be roughly the same.

32. 32 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3e:) Our approach at work Source Receiver stream of voice, music, video video, multimedia, … IP network NN asking the source for BR, FR, RA measuring LR, CLP

33. 33 Workshop RNN, ICANN’02, Madrid, 27/8/02 (3e:) Our approach at work (cont.) Receiver source parameters module BR FR RA network parameters module LR CLP NN measure of quality

34. 34 Workshop RNN, ICANN’02, Madrid, 27/8/02 4: Our results in a nutshell Once trained, the NN is supposed to behave like an average human observer face to a received sequence. Observe that the performance of the method depends on the selected parameters, but that there is no restriction in such a choice. Moreover, if a posteriori some new parameter appears to be important, it can be easily added following the same procedure.

35. 35 Workshop RNN, ICANN’02, Madrid, 27/8/02 (4:) Our results in a nutshell (cont.) The trained NN (classical and RNN) correlated remarquably well against human evaluations. They (obviously) run in negligible time. They allow us to study the behavior of quality as a function of several parameters. We did the same work for audio, with the same results. We also developped an example of application using our tool for control purposes (in audio).

36. 36 Workshop RNN, ICANN’02, Madrid, 27/8/02 (4:) Performances (RNN) during the training phase

37. 37 Workshop RNN, ICANN’02, Madrid, 27/8/02 (4:) Another view

38. 38 Workshop RNN, ICANN’02, Madrid, 27/8/02 (4:) Performances (RNN) during the validation phase

39. 39 Workshop RNN, ICANN’02, Madrid, 27/8/02 (4:) Another view

40. 40 Workshop RNN, ICANN’02, Madrid, 27/8/02 5: ANN vs RNN We used the toolbox of MATLAB implementing several standard Neural Networks techniques (Artificial Neural Networks, ANN) and a specific software for RNN. We compared their performances mainly in the respective learning abilities. We will present here some of the obtained results, which show that RNN behave better for our applications.

41. 41 Workshop RNN, ICANN’02, Madrid, 27/8/02 (5:) An interesting example: left: RNN right: ANN

42. 42 Workshop RNN, ICANN’02, Madrid, 27/8/02 (5:) RNN vs. ANN: # of hidden neurons

43. 43 Workshop RNN, ICANN’02, Madrid, 27/8/02 (5:) RNN vs. ANN: # of hidden neurons

44. 44 Workshop RNN, ICANN’02, Madrid, 27/8/02 6: Analyzing parameters impact

45. 45 Workshop RNN, ICANN’02, Madrid, 27/8/02 (6:) Analyzing parameters impact (cont.)

46. 46 Workshop RNN, ICANN’02, Madrid, 27/8/02 (6:) MPQM (well-known metric) when applying losses

47. 47 Workshop RNN, ICANN’02, Madrid, 27/8/02 (6:) ITS (well-known metric) when bit rate is varied

48. 48 Workshop RNN, ICANN’02, Madrid, 27/8/02 7: Our demo tool

49. 49 Workshop RNN, ICANN’02, Madrid, 27/8/02 8: Our work on control schemes for audio

50. 50 Workshop RNN, ICANN’02, Madrid, 27/8/02 (8:) Some simulation results BW needed by PCM BW needed by GSM BW needed by PCM BW needed by GSM

51. 51 Workshop RNN, ICANN’02, Madrid, 27/8/02 (8:) Illustrating our control application

52. 52 Workshop RNN, ICANN’02, Madrid, 27/8/02 9: Ongoing work Refining our initial models for quantitying the quality of audio and video transmission: –better loss models –exploration of new parameters (for instance, FEC) –trying to characterize streams types

53. 53 Workshop RNN, ICANN’02, Madrid, 27/8/02 (9:) Ongoing work (cont.) Coupling our approach with traffic prediction (also using neural techniques) Exploration of other applications (diffserv Work on learning algorithms for RNN (numerical analysis)