ITU-T Study Group 12 Performance and Quality of Service

1
ITU-T Study Group 12Performance and Quality of
Service

• Jean-Yves Monfort

2
Contents

• Terms of reference
• Highlights of achievements / Projects / FGs
• Future work
• Conclusions
• Supplemental slides

3
Terms of Reference

• Responsible for Recommendations on the end-to-end
  transmission performance of terminals and
  networks, in relation to the perceived quality
  and acceptance by users of text, data, speech,
  and multi-media applications.
• Although this work includes the related
  transmission implications of all networks and all
  telecommunication terminals, a special focus is
  given to IP QoS, interoperability and
  implications for NGN, and also includes work on
  performance and resource management.
• Lead SG for Performance and Quality of Service
  (QoS)

4
Highlights of achievements (I)

• The test methodologies for subjective assessment
  of speech and audio are always evolving to take
  into account the new services and the new
  environments
• SG12 is assessing the quality of codecs (26 test
  plans and 15 test results)- Collab. with SG16
• New Recommendation P.805 for conversation test.
• The creation of 2 Focus Groups on car
  communication FITCAR (closed in Jan.2008) and
  CARCOM(opened in June 2008)

Round Robin Test on Human/artificial ears
impedance ? Planned update of P.57

• NewRecommendation P.505 " One-view visualisation
  of quality"? "Quality Pie"

5
Highlights of achievements (II)

• One of the major area on which SG12 has the
  global leadership is the development of quality
  models and tools for speech, but now also for
  data and multimedia services
• Recommendation G.107 ("E-model") is updated on a
  regular basis. One of the major improvements has
  been to cover wideband speech.
• Perceptual models for narrow and wideband speech
• P.862.x-series Recommendations give a complete
  set of models and information on how to use them.
  These types of models are working with reference
  signal.
• P.862.2 Wideband extension to Recommendation
  P.862
• P. 862.3 Application guide for objective Quality
  Measurement based on Recommendation P.862

6
Highlights of achievements (III)

• "In-service non-intrusive" quality models,
  without reference signal.
• Recommendation P.564, which has also been
  implemented for wideband speech, specifies the
  minimum criteria for objective speech quality
  assessment models that predict the impact of
  observed IP network impairments on the one-way
  listening quality experienced by the end-user in
  IP/UDP/RTP-based telephony applications.
• A call for proposal has been published for P.NAMS
  (Non Intrusive assessment of performance for
  video streaming) ? 2009

7
Highlights of achievements (IV)

• Recommendations for other media than speech, e.g.
  for videophone (G.1070) and for multimedia
  (G.1050).
• G.1070 provides ? algorithm estimating videophone
  quality to be used by QoS/QoE planners to ensure
  end user satisfaction. Recommendation
• G.1050 ? a network model for evaluating
  multimedia transmission performance over IP
  based on statistical models on a broad range of
  known network configurations.
• Recommendation G.1030 which provides a perceptual
  model for web browsing and defines a framework of
  tools to estimate end-to-end IP network
  performance for user applications.

8
Highlights of achievements (V)

• One of the most popular Recommendations (Y.1541),
  initiated in SG13, has been updated several times
  by SG12 during the study period, to cover new
  services and media,
• e.g. Appendix VIII on effects of IP network
  performance on digital television transmission
  QoS.
• This fundamental Recommendation should be the
  core for a global end-to-end QoS for
  interoperability.
• The set of existing Recommendations has been
  completed by Y.1542, "a Framework for achieving
  End-to-End Performance objectives", linked with
  Y.1541.

9
Highlights of achievements (VI)

• SG12 experts and Officials has been active in
  Focus Groups and several Recommendations have
  been developed FG-IPTV-GSI (G.1080 and 1081 have
  been approved and two other Rec. for monitoring
  are under progress), FG-NGN-GSI (Y.1542 has been
  approved, TR-NGN.QoS and TR-NGN.NHNperf are under
  progress).
• A new recommendation addressing Ethernet
  performance, Recommendation Y.1563 "Ethernet
  frame transfer and availability performance" has
  been approved in May 2008.
• For SLA (service level agreements) matters,
  G.1040 provides performance metric to be used by
  network service providers.

10
Future Work (I)

• For speech and audio quality The hot topics are
  "bandwidth extension" (superwideband and
  fullband) and "binaural perception"
• consequences on loudness, binaural modelling,
  acoustical interfaces (for terminals and test
  equipments such as artificial ear an mouth) and
  test methodologies (subjective and objective test
  methods).
• Result from FG CARCOM a specification for
  wideband car communication
• To identify end-user performance expectations and
  associated metrics for audio, video, text,
  graphics quality, and control functionality.
• ? to improve the accessibility of services.

11
Future Work (II)

• Industry needs to assess Quality of all types of
  media. ?SG 12to work on audiovisual quality, and
  visual and multimedia quality measurements in
  telecommunication services.
• This work to be done in close partnership with
  VQEG.
• SG12, the major actor for the development of
  quality modelling for speech. This leadership
  should be increased. ? development of perceptual
  and parametric models for voice, audio,
  audiovisual and multimedia,
• ? application of measurement tools and
  development and application of all planning and
  measurement tools. ? framework for diagnostic
  functions and their interaction with external
  objective models predicting media quality.
• ? operational aspects of QoS, for the benefits
  of in particular developing countries.

12
Future Work (III)

• Quality models Next future
• - New objective models for quality predictions
  are expected for speech quality evaluation
  (P.OLQA) at the end of 2009 (call already sent)
  and for objective prediction of P.835 scores
  (P.ONRA).
• - The plans to define a model for spoken
  conversations have been agreed.
• - Regarding multimedia, the call for proponents
  for P.NAMS has been sent out ? a new
  Recommendation planned for 2009, Next step
  following P.NAMS P.NBAMS a non-intrusive bit
  stream model for the assessment of performance of
  multimedia streaming.
• QoE, QoS and Perf for multimedia (IPTV, HN)
• ? G.OMVAS Opinion Model for Video and Audio
  Streaming applications and
• ? Recs. on IPTV Quality monitoring (in addition
  to G. 1081)

13
Future Work (IV)

• NGN performance interworking issues requiring
  consideration include end-to-end multimedia
  performance interworking and interoperability
  issues as well as impacts of interworking
  functions.
• To define static and dynamic apportionment of
  performance objectives across networks and
  technologies.
• To examine the suitability of numerical
  objectives of IP QoS provisional Classes 6 and 7
  given in Recommendation Y.1541.
• e.g. for MPEG-4 video and various Audio schemes.
• Urgent topics to be solved IP Service
  Availability applicable in the network operators
  environment
• For QoS, QoE and Performance, SG12 plans to
  contribute significantly to improve the
  interoperability and convergence of services and
  networks.

14
Future Work (V)

• A special focus is given to interoperability to
  ensure end-to-end users' satisfaction.
• SG12 also provides cross-ITU coordination for the
  many aspects of Quality-of-Service (QoS) and
  Quality of Experience (QoE) and to foster
  consistency among ITU-T Study Groups, as well as
  with other related bodies such as the IETF, 3GPP
  so as to advance the interoperability of any QoS
  solutions.

15
Conclusion

• In order to cover the future plans, it is
  proposed to rename Study Group 12 to
  Performance, QoS and QoE and define its mandate
  as follows
• Study Group 12 is responsible for Recommendations
  on performance, QoS and QoE for the full spectrum
  of terminals, networks and services ranging from
  speech over fixed circuit based networks to
  multimedia applications over networks that are
  mobile and packet-based.
• A special focus is given to interoperability to
  ensure end-to-end users' satisfaction.
• SG12 is Lead Study Group on QoS and QoE.

16
Supplemental Slides

• Management team
• Structure and Implementations
• Working Parties
• Statistics
• Workshops
• QoS Flyer

17
Management Team
Klemens Adler (G) VP WP2/12 Chair
Chuck Dvorak (US) VP Lead SG Resp.
Paul Coverdale (Ca) VP WP3/12 Chair
Catherine Quinquis (F) WP1/12 Co-Chair
Judit Katona Kiss, TSB
Gérald Lecucq (F) WP1/12 co-Chair
Missing Jean-Jacques Massima-Landji (Gabon) VP
JY Monfort ,SG Chair, is behind the camera
18
Study Group Structure

• WP1 Telephonometry, Terminals and Subjective
  assessment
• Develops Recommendations on artificial
  heads/ears/Mouths/signals and associated test
  methods
• Develops subjective methods and test plans
• WP2 Performance, Planning and Tools
• Develops quality models for speech and
  multimedia (E-Model, Perceptual models, INMD,)
• WP3 IP-Related QoS and resource management
• Defines parameters and models for QoS classes
  and IP performance.
• Develop frameworks for IP-related QoS

19
Study Group Implementations

• Focus groups FITCAR and CARCOM
• After 5 meetings, the specification on
  characteristics and test methods for narrowband
  speech terminals in cars has been finalised by FG
  FITCAR. The specification has been submitted to
  SG12. The draft new Recommendation P.1100, based
  on the specification produced by the Focus Group
  FITCAR had consent at the last meeting of SG12.
• The new Focus Group CARCOM is focused on
  Wideband Speech
• APSC-Telemov
• The Advisory Panel for Standards Cooperation on
  Telecommunications related to Motor Vehicles -
  APSC TELEMOV was created at the end of the
  previous study period as a result of the Workshop
  on Standardization in Telecommunication for Motor
  Vehicles and intended as a cooperation group on
  all aspects of standardization related to
  telecommunications within and for motor vehicles.
  

20
Study Group Implementations

• SG12 Regional Group for Africa
• To encourage to take an active participation in
  work of ITU-T SG12 by African Regulators and
  Operators, and in the implementation of the ITU-T
  Recommendations on Performance/QoS and QoE
• JRG on Multimedia Quality assessment has been
  formed from SG12 and SG9 in early 2004 with the
  motivation to represent the interest of ITU-T
  within VQEG. The work of JRG is successfully
  focussing on Multimedia quality aspects within
  VQEG Multimedia (MMQA) working group

21
WP 1 /12 Telephonometry, terminalsand
Subjective assessment
Speech transmission characteristics of speech
terminals Mr Shi Denian
Q.3/12
Q.4/12

Telephonometrics methodologies for handsets and
headset terminals Mr L. Madec
Q.5/12
Q.6/12
Q.7/12
Methods, tools and test plans for the subjective
assessment Mr P. Usai and Mrs C. Quinquis

22
WP 2/12 Performance, Planning and Tools
Extension of the E-Model Mrs U. Jekosch and Mr
S. Möller
Q.8/12
Q.9/12

Transmission planning and performance
considerations Mr V.
Siply
Q.10/12

Performance evaluation of services based on
speech technology Mr S.MöllerMrA. Raake
Q.12/12
Q.14/12
Q.20/12
Objective assessment of conversational speech
Quality in networks Mr J. Pomy
23
WP 3/12 IP-Related QoS andRessource management
Q.2/12
Q.11/12
Q.13/12
IPTV and multimedia QoE/QoS performance
requirements and assessment methods A. Takahashi
Q.16/12
Performance of IP-based networks
Mr A. Morton -Mr H. Kim(associate)
Q.17/12

Call processing performance
Q.19/12
24
Statistics

• During the study period, Study Group 12 examined
  a total of 364 contributions
• On the basis of these documents and of a thousand
  of temporary documents, Study Group 12
• drew up 21 new Recommendations
• amended/revised 23 Recommendations
• developed 1 Supplement,
• developed 27 Test plans and 15 Test reports in
  the scope of collaboration with ITU-T SG16 for
  speech and audio coders.

25
Workshops organized by SG12 (I)

• End-to-End QoS-QoE / 14-16 June 2006 / Geneva
• http//www.itu.int/ITU-T/worksem/qos/200606/index.
  html
• Organised by SG12 with the collaboration with SG2
  and SG16
• The sessions were dedicated to Inter-domain QoS,
  Interoperability testing and terminal issues,
  Home Networking QoS/QoE, Quality of Experience
  (QoE) of new services, NGN-related QoS and the
  accumulation of network impairements, and Quality
  prediction
• This workshop brought together more than 100
  experts from all over the world who are involved
  in the definition and the delivery of end-to-end
  quality of experience (QoE) and quality of
  service (QoS).

26
Workshops organized by SG12 (II)

• From Speech to Audio/10-12 Sept 2008/Lannion,
  France
• http//www.itu.int/ITU-T/worksem/speechaudio/index
  .html
• The workshop was organized by WP1/12 and involved
  participants from ITU-T SG12, SG16, ITU-R 6C.
  Representatives of ETSI STQ, 3GPP SA4 also joint
  the workshop that was supported by France
  Telecom, Alcatel Lucent and DEGA (the German
  acoustical society)

The workshop was very productive and agreed on
some actions - to enhance the constructive
dialogue between universities, standardization
bodies and companies. Ideas to improve the future
collab have been raised ?need to be implemented -
to provide inputs to develop new methodologies
covering the complete audio bandwidth from Narrow
Band to Fullband ? New Recs. There is a need for
improved collab between ITU-T and ITU-R for audio
quality.
Keynote Speakers
Pr. J. Blauert Bochum Univ.
Pr. S. Meunier Univ. Aix-Marseille
27
Workshops supported by SG12

• The Fully Networked Car, Geneva (Palexpo)
• http//www.itu.int/ITU-T/worksem/ict-auto/index.ht
  ml
• 2-4 March 2005
• 7 - 9 March 2007
• 5 - 7 March 2008
• The next one 4 - 5 March 2009
• Opportunities and Challenges in Home Networking
  Geneva, 13 - 14 October 2005
• 2nd ETSI STQ Workshop on Wideband Speech Quality
  in Terminals and Networks Assessment and
  Prediction, 22-23 June 2005, Mainz, Germany
• ETSI Workshop on 'Effects of transmission
  performance on Multimedia QoS'Prague, Czech
  Republic, 17-19 June 2008

28
A QoS Flyer (page 2)
True Quality of Service captures the User
Experience (QoE) The G.1000 Framework helps
capture the many QoS dimensions, and acknowledges
that QoS can be viewed from four different
perspectives
29
Acknowledgements

• Thanks to the Management Team (including Rodolfo
  Ceruti) for the strong support, to Hans Gierlich
  (as Chair of the Focus Group and a Fully
  networked car workshop), to the efficient
  Rapporteurs and contributors and to Daniel Waturu
  for his involvement in SG12 Regional Group for
  Africa.
• Many thanks to TSB support and specially Judit
  and Sarah for SG12, Sami and his team for the
  specific websites (E-model,..), Paolo and his
  team for the organization of workshops, Sylvie,
  Kristin and Richard for the meetings facilities.

30
Thank you!

• Mr Jean-Yves Monfort
• France Telecom/Orange-Labs
• Standardization Senior Manager Senior expert in
  QoS
• During three years, I was deputy Director for
  standards in France Telecom RD.
• I was previously involved as deputy Head of a
  Lab. Dealing with speech, Audio, video and
  usability Quality of services, terminals and
  equipments

One of my hobbies