1. Summary of Results Study Period 2005-2008 ITU-T Study Group 12 Performance and Quality of Service Jean-Yves Monfort

2. International Telecommunication Union 2 ITU-T Study Group 12 Performance and Quality of Service Contents Terms of reference Highlights of achievements / Projects / FGs Future work Conclusions Supplemental slides

3. International Telecommunication Union 3 ITU-T Study Group 12 Performance and Quality of Service Terms of Reference Responsible for Recommendations on the end- to-end transmission performance of terminals and networks, in relation to the perceived quality and acceptance by users of text, data, speech, and multi-media applications. Although this work includes the related transmission implications of all networks and all telecommunication terminals, a special focus is given to IP QoS, interoperability and implications for NGN, and also includes work on performance and resource management. Lead SG for Performance and Quality of Service (QoS)

4. International Telecommunication Union 4 ITU-T Study Group 12 Performance and Quality of Service Highlights of achievements (I) NewRecommendation P.505 " One-view visualisation of quality" "Quality Pie" The test methodologies for subjective assessment of speech and audio are always evolving to take into account the new services and the new environments SG12 is assessing the quality of codecs (26 test plans and 15 test results)- Collab. with SG16 New Recommendation P.805 for conversation test. The creation of 2 Focus Groups on car communication FITCAR (closed in Jan.2008) and CARCOM(opened in June 2008) Round Robin Test on Human/artificial ears impedance Planned update of P.57

5. International Telecommunication Union 5 ITU-T Study Group 12 Performance and Quality of Service Highlights of achievements (II) One of the major area on which SG12 has the global leadership is the development of quality models and tools for speech, but now also for data and multimedia services Recommendation G.107 ("E-model") is updated on a regular basis. One of the major improvements has been to cover wideband speech. Perceptual models for narrow and wideband speech – P.862.x-series Recommendations give a complete set of models and information on how to use them. These types of models are working with reference signal. – P.862.2 Wideband extension to Recommendation P.862 – P. 862.3 Application guide for objective Quality Measurement based on Recommendation P.862

6. International Telecommunication Union 6 ITU-T Study Group 12 Performance and Quality of Service Highlights of achievements (III) "In-service non-intrusive" quality models, without reference signal. Recommendation P.564, which has also been implemented for wideband speech, specifies the minimum criteria for objective speech quality assessment models that predict the impact of observed IP network impairments on the one-way listening quality experienced by the end-user in IP/UDP/RTP-based telephony applications. A call for proposal has been published for P.NAMS (Non Intrusive assessment of performance for video streaming) 2009

7. International Telecommunication Union 7 ITU-T Study Group 12 Performance and Quality of Service Highlights of achievements (IV) Recommendations for other media than speech, e.g. for videophone (G.1070) and for multimedia (G.1050). – G.1070 provides algorithm estimating videophone quality to be used by QoS/QoE planners to ensure end user satisfaction. Recommendation – G.1050 a network model for evaluating multimedia transmission performance over IP; based on statistical models on a broad range of known network configurations. Recommendation G.1030 which provides a perceptual model for web browsing and defines a framework of tools to estimate end-to-end IP network performance for user applications.

8. International Telecommunication Union 8 ITU-T Study Group 12 Performance and Quality of Service Highlights of achievements (V) One of the most popular Recommendations (Y.1541), initiated in SG13, has been updated several times by SG12 during the study period, to cover new services and media, – e.g. Appendix VIII on effects of IP network performance on digital television transmission QoS. This fundamental Recommendation should be the core for a global end-to-end QoS for interoperability. The set of existing Recommendations has been completed by Y.1542, "a Framework for achieving End-to-End Performance objectives", linked with Y.1541.

9. International Telecommunication Union 9 ITU-T Study Group 12 Performance and Quality of Service Highlights of achievements (VI) SG12 experts and Officials has been active in Focus Groups and several Recommendations have been developed :FG-IPTV-GSI (G.1080 and 1081 have been approved and two other Rec. for monitoring are under progress), FG-NGN-GSI (Y.1542 has been approved, TR-NGN.QoS and TR-NGN.NHNperf are under progress). A new recommendation addressing Ethernet performance, Recommendation Y.1563 "Ethernet frame transfer and availability performance" has been approved in May 2008. For SLA (service level agreements) matters, G.1040 provides performance metric to be used by network service providers.

10. International Telecommunication Union 10 ITU-T Study Group 12 Performance and Quality of Service Future Work (I) For speech and audio quality : The hot topics are "bandwidth extension" (superwideband and fullband) and "binaural perception" consequences on loudness, binaural modelling, acoustical interfaces (for terminals and test equipments such as artificial ear an mouth) and test methodologies (subjective and objective test methods). Result from FG CARCOM : a specification for wideband car communication To identify end-user performance expectations and associated metrics for audio, video, text, graphics quality, and control functionality. to improve the accessibility of services.

11. International Telecommunication Union 11 ITU-T Study Group 12 Performance and Quality of Service Future Work (II) Industry needs to assess Quality of all types of media. SG 12to work on audiovisual quality, and visual and multimedia quality measurements in telecommunication services. This work to be done in close partnership with VQEG. SG12, the major actor for the development of quality modelling for speech. This leadership should be increased. development of perceptual and parametric models for voice, audio, audiovisual and multimedia, application of measurement tools and development and application of all planning and measurement tools. framework for diagnostic functions and their interaction with external objective models predicting media quality. operational aspects of QoS, for the benefits of in particular developing countries.

12. International Telecommunication Union 12 ITU-T Study Group 12 Performance and Quality of Service Future Work (III) Quality models – Next future - New objective models for quality predictions are expected for speech quality evaluation (P.OLQA) at the end of 2009 (call already sent) and for objective prediction of P.835 scores (P.ONRA). - The plans to define a model for spoken conversations have been agreed. - Regarding multimedia, the call for proponents for P.NAMS has been sent out a new Recommendation planned for 2009, Next step following P.NAMS : P.NBAMS a non-intrusive bit stream model for the assessment of performance of multimedia streaming. QoE, QoS and Perf for multimedia (IPTV, HN) G.OMVAS Opinion Model for Video and Audio Streaming applications and Recs. on IPTV Quality monitoring ( in addition to G. 1081 )

13. International Telecommunication Union 13 ITU-T Study Group 12 Performance and Quality of Service Future Work (IV) NGN performance interworking issues requiring consideration include end-to-end multimedia performance interworking and interoperability issues as well as impacts of interworking functions. To define static and dynamic apportionment of performance objectives across networks and technologies. To examine the suitability of numerical objectives of IP QoS provisional Classes 6 and 7 given in Recommendation Y.1541. – e.g. for MPEG-4 video and various Audio schemes. Urgent topics to be solved : IP Service Availability applicable in the network operators environment For QoS, QoE and Performance, SG12 plans to contribute significantly to improve the interoperability and convergence of services and networks.

14. International Telecommunication Union 14 ITU-T Study Group 12 Performance and Quality of Service Future Work (V) A special focus is given to interoperability to ensure end-to-end users' satisfaction. SG12 also provides cross-ITU coordination for the many aspects of Quality-of-Service (QoS) and Quality of Experience (QoE) and to foster consistency among ITU-T Study Groups, as well as with other related bodies such as the IETF, 3GPP so as to advance the interoperability of any QoS solutions.

15. International Telecommunication Union 15 ITU-T Study Group 12 Performance and Quality of Service Conclusion In order to cover the future plans, it is proposed to rename Study Group 12 to Performance, QoS and QoE and define its mandate as follows: Study Group 12 is responsible for Recommendations on performance, QoS and QoE for the full spectrum of terminals, networks and services ranging from speech over fixed circuit based networks to multimedia applications over networks that are mobile and packet- based. A special focus is given to interoperability to ensure end- to-end users' satisfaction. SG12 is Lead Study Group on QoS and QoE.

16. International Telecommunication Union 16 ITU-T Study Group 12 Performance and Quality of Service Supplemental Slides Management team Structure and Implementations Working Parties Statistics Workshops QoS Flyer

17. International Telecommunication Union 17 ITU-T Study Group 12 Performance and Quality of Service Management Team Judit Katona Kiss, TSB Klemens Adler (G) VP & WP2/12 Chair Chuck Dvorak (US) VP & Lead SG Resp. Gérald Lecucq (F) WP1/12 co-Chair Paul Coverdale (Ca) VP & WP3/12 Chair Catherine Quinquis (F) WP1/12 Co-Chair Missing: Jean- Jacques Massima- Landji (Gabon) VP JY Monfort,SG Chair, is behind the camera

18. International Telecommunication Union 18 ITU-T Study Group 12 Performance and Quality of Service Study Group Structure WP1 Telephonometry, Terminals and Subjective assessment Develops Recommendations on artificial heads/ears/Mouths/signals and associated test methods Develops subjective methods and test plans WP2 Performance, Planning and Tools Develops quality models for speech and multimedia (E-Model, Perceptual models, INMD,…) WP3 IP-Related QoS and resource management Defines parameters and models for QoS classes and IP performance. Develop frameworks for IP-related QoS

19. International Telecommunication Union 19 ITU-T Study Group 12 Performance and Quality of Service Study Group Implementations Focus groups FITCAR and CARCOM After 5 meetings, the specification on characteristics and test methods for narrowband speech terminals in cars has been finalised by FG FITCAR. The specification has been submitted to SG12. The draft new Recommendation P.1100, based on the specification produced by the Focus Group FITCAR had consent at the last meeting of SG12. The new Focus Group CARCOM is focused on Wideband Speech APSC-Telemov The Advisory Panel for Standards Cooperation on Telecommunications related to Motor Vehicles - APSC TELEMOV was created at the end of the previous study period as a result of the Workshop on Standardization in Telecommunication for Motor Vehicles and intended as a cooperation group on all aspects of standardization related to telecommunications within and for motor vehicles. Workshop on Standardization in Telecommunication for Motor Vehicles

20. International Telecommunication Union 20 ITU-T Study Group 12 Performance and Quality of Service Study Group Implementations SG12 Regional Group for Africa To encourage to take an active participation in work of ITU-T SG12 by African Regulators and Operators, and in the implementation of the ITU-T Recommendations on Performance/QoS and QoE… JRG on Multimedia Quality assessment has been formed from SG12 and SG9 in early 2004 with the motivation to represent the interest of ITU-T within VQEG. The work of JRG is successfully focussing on Multimedia quality aspects within VQEG Multimedia (MMQA) working group

21. International Telecommunication Union 21 ITU-T Study Group 12 Performance and Quality of Service WP 1 /12 – Telephonometry, terminals and Subjective assessment Q.4/12 Q.5/12 Hands-Free communications in vehicles Mr H.W. Gierlich Telephonometrics methodologies for handsets and headset terminals Mr L. Madec Q.7/12 Methods, tools and test plans for the subjective assessment… Mr P. Usai and Mrs C. Quinquis Q.3/12 Analysis methods using complex measurement signals…. Mr H.W. Gierlich Q.6/12 Speech transmission characteristics of speech terminals … Mr Shi Denian

22. International Telecommunication Union 22 ITU-T Study Group 12 Performance and Quality of Service WP 2/12 – Performance, Planning and Tools Q.9/12 Q.10/12 Perceptual-based objective methods for voice, audio and visual quality… Mr J. Berger Transmission planning and performance considerations… Mr V. Siply Q.20/12 Objective assessment of conversational speech Quality in networks Mr J. Pomy Objective assessment of conversational speech Quality in networks Mr J. Pomy Q.8/12 In-Service Non Intrusive Assessment of voice transmission performance Mr V. Barriac Q.14/12 Extension of the E-Model Mrs U. Jekosch and Mr S. Möller Q.12/12 Performance evaluation of services based on speech technology Mr S.Möller&MrA. Raake

23. International Telecommunication Union 23 ITU-T Study Group 12 Performance and Quality of Service WP 3/12 – IP-Related QoS andRessource management Q.13/12 Q.16/12 IP based multimedia services end to end transm. planning …. M. J. Pomy IPTV and multimedia QoE/QoS performance requirements and assessment methods A. Takahashi IPTV and multimedia QoE/QoS performance requirements and assessment methods A. Takahashi Q.19/12 Performance of IP-based networks Mr A. Morton - Mr H. Kim(associate) Speech transmission Charact. And Meas.methods for IP terminals and gateways Mr U. Brunner Q.11/12 Broadband and IP-related resource management Mr D. Mustill Q.17/12 Call processing performance Q.2/12

24. International Telecommunication Union 24 ITU-T Study Group 12 Performance and Quality of Service Statistics During the study period, Study Group 12 examined a total of 364 contributions On the basis of these documents and of a thousand of temporary documents, Study Group 12: –drew up 21 new Recommendations; –amended/revised 23 Recommendations; –developed 1 Supplement, –developed 27 Test plans and 15 Test reports in the scope of collaboration with ITU-T SG16 for speech and audio coders.

25. International Telecommunication Union 25 ITU-T Study Group 12 Performance and Quality of Service Workshops organized by SG12 (I) End-to-End QoS-QoE / 14-16 June 2006 / Geneva http://www.itu.int/ITU-T/worksem/qos/200606/index.html Organised by SG12 with the collaboration with SG2 and SG16 The sessions were dedicated to Inter-domain QoS, Interoperability testing and terminal issues, Home Networking QoS/QoE, Quality of Experience (QoE) of new services, NGN-related QoS and the accumulation of network impairements, and Quality prediction This workshop brought together more than 100 experts from all over the world who are involved in the definition and the delivery of end-to-end quality of experience (QoE) and quality of service (QoS).

26. International Telecommunication Union 26 ITU-T Study Group 12 Performance and Quality of Service Workshops organized by SG12 (II) From Speech to Audio/10-12 Sept 2008/Lannion, France http://www.itu.int/ITU-T/worksem/speechaudio/index.html The workshop was organized by WP1/12 and involved participants from ITU-T SG12, SG16, ITU-R 6C. Representatives of ETSI STQ, 3GPP SA4 also joint the workshop that was supported by France Telecom, Alcatel Lucent and DEGA (the German acoustical society) The workshop was very productive and agreed on some actions - to enhance the constructive dialogue between universities, standardization bodies and companies. Ideas to improve the future collab have been raised need to be implemented - to provide inputs to develop new methodologies covering the complete audio bandwidth from Narrow Band to Fullband New Recs. There is a need for improved collab between ITU-T and ITU-R for audio quality. Keynote Speakers Pr. J. Blauert Bochum Univ. Pr. S. Meunier Univ. Aix-Marseille

27. International Telecommunication Union 27 ITU-T Study Group 12 Performance and Quality of Service Workshops supported by SG12 The Fully Networked Car, Geneva (Palexpo) – http://www.itu.int/ITU-T/worksem/ict-auto/index.html http://www.itu.int/ITU-T/worksem/ict-auto/index.html 2-4 March 2005 7 - 9 March 2007 5 - 7 March 2008 – The next one : 4 - 5 March 2009 Opportunities and Challenges in Home Networking Geneva, 13 - 14 October 2005 2nd ETSI STQ Workshop on Wideband Speech Quality in Terminals and Networks: Assessment and Prediction, 22-23 June 2005, Mainz, Germany ETSI Workshop on 'Effects of transmission performance on Multimedia QoS' Prague, Czech Republic, 17-19 June 2008

28. International Telecommunication Union 28 ITU-T Study Group 12 Performance and Quality of Service A QoS Flyer (page 2) True Quality of Service captures the User Experience (QoE) The G.1000 Framework helps capture the many QoS dimensions, and acknowledges that QoS can be viewed from four different perspectives

29. International Telecommunication Union 29 ITU-T Study Group 12 Performance and Quality of Service Acknowledgements Thanks to the Management Team (including Rodolfo Ceruti) for the strong support, to Hans Gierlich (as Chair of the Focus Group and a Fully networked car workshop), to the efficient Rapporteurs and contributors and to Daniel Waturu for his involvement in SG12 Regional Group for Africa. Many thanks to TSB support and specially Judit and Sarah for SG12, Sami and his team for the specific websites (E-model,..), Paolo and his team for the organization of workshops, Sylvie, Kristin and Richard for the meetings facilities.

30. International Telecommunication Union 30 ITU-T Study Group 12 Performance and Quality of Service Thank you! Mr Jean-Yves Monfort France Telecom/Orange-Labs Standardization Senior Manager & Senior expert in QoS During three years, I was deputy Director for standards in France Telecom R&D. I was previously involved as deputy Head of a Lab. Dealing with speech, Audio, video and usability Quality of services, terminals and equipments One of my hobbies