TMC Developers Conference San Francisco Aug 03rd, 2005

1
TMC Developers ConferenceSan FranciscoAug 03rd,
2005
ST-09 Network Assurance and Testing During the
Migration to VoIP

• Andy Huckridge
• Spirent Communications.
• Chair, Interop WG, MSF

2
Agenda

• Spirent overview
• Key implementation issues
• What is Triple Play / Converged networks?
• Specifics on testing SIP
• Network Impairments and Parameters that Voice and
  Video Affect Quality
• Metrics for Measuring Voice and Video Quality and
  Performance
• Good test methodology

3
Spirent Communications

• Spirent is the test solution leader
• 1,800 employees in 14 countries
• More than 1,500 customers
• Sales and service capabilities in 30 countries

4
Implementation steps - Lab
Services Deployment

• Characterize system before trial
• Validate system scalability
• Identify capacity limits
• Measure call performance
• Automate regression testing

5
Implementation steps - Network

• Facilitate vendor selection
• Identify performance ceilings
• Enable accurate capacity planning
• End-to-end service assurance testing
• Improve operational performance
• Improve customer satisfaction

6
Key Implementation Issues

• Circuit to packet migration
• Scalability and Performance
• Voice quality
• Interoperability and conformance
• Budget pressures

7
Before you deploy!

• Network Equipment Manufacturers (Chips, IP-PBX,
  Gateways, MSs SSs)
• Characterize your system before trial
• Validate system scalability
• Identify capacity limits
• Measure call performance
• Service Providers(NSPs, SPs, ITSPs)
• Define criteria for vendor selection
• Identify performance ceilings
• Accurately plan for your capacity needs
• End-to-end service assurance testing

8
Data TransmissionNon-Real-Time Applications
Telenet
Name Resolving DNS
Web HTTP
Email and Messaging POP SMPT Exchange
Data Base MS SQL Oracle
File Transfer FTP
Music Downloading
Home control
Data Examples Internet access, Email, File
Transfer, Portals, Database Applications, Gaming,
Government Services, Online Commerce
9
Voice and Video Real-Time Applications
Gaming Single / Multiplayer
IP Music/Audio/Radio
VoD
VoIP, IP Telephony, Video Telephony G.711, G.729,
G.728, G.726, G.723 H.261, H.263, SIP, SIP-T,
H323, Skinny, MGCP, MEGACO/H.248
Real-Time Online Communications Instant
Messenger Webex Netmeeting SIP H.323
IPTV Services Broadcast, On-Demand,
Bi-directional / Interactive MPEG1, MPEG2,
MPEG4, VC1, H264
Multi-Media RTP H.264,Microsoft AVI, QuickTime
(.mov) Windows Media (.wmv, .asf), RealMedia
(.rm),
Voice Applications Phone service integrated with
video Video Applications Broadcast TV, video on
demand, distance learning
10
Converged Triple Play Data, Voice and Video
With Network Impairments
Data
Video
Dialed Digits
Configure
Configure
On Hook
Sigtran
Configure
Notify
Ring Back
Off Hook
Dial
Good Bye
TCP
Hello
SS7
CAS
SIP
Voice Conversation
Good Bye
On Hook
Connect
Off Hook
Ring
Hello
Disconnect
Connect
ISDN
MEGACO
RTP/UDP
Impairments can be heard in the voice conversation
Signaling Path
11
Testing SIP Conformance

• Comprehensive and scriptable SIP call flows
• Complete configurable SIP signaling messages
• SIP protocol analysis
• Simplified flow diagrams with visual analysis
• Comprehensive conformance test suites

12
Testing SIP Conformance

• ETSI TS 102-027-1 v2.12,Tiphon
• RFC 3261 user agent, proxy and redirect server
  compliance
• Graphical SDL and TTCN tools
• Create, edit, compile and execute simulation
  scripts and conformance tests
• Additional SIP messages beyond RFC 3261
• Included in torture tests
• Additional tests as defined by the SIP Forum

13
Testing with Configurable SIP

• Configurable SIP call setup and call teardown
• Configurable call flows and messages
• Incoming message filter
• Adaptive signaling syntax for SIP
• Improves interoperability with new drafts and
  non-conformant proprietary implementations

14
Testing with Configurable SIP

• Configurable messages
• Invite, ACK, bye, register
• Responses 1xx, 2xx, 3xx, 4xx, 5xx, 6xx
• Configurable timers, message intervals
• Enable and disable optional messages
• Re-invite, cancel, options, message, info,
  notify, subscribe, unsubscribe, update, refer,
  Prack
• Fix erroneous incoming messages on the fly with
  the search and replace method
• Allows interoperability with SIP devices
  (including drafts, non conformant, prototype)

15
Testing with Configurable SIP

• Initial INVITE
• Redirect INVITE
• Authentication INVITE
• Ack
• 100 response
• 180 response
• 200 response
• Bye
• 200 for bye
• 200 for PRACK
• Initial REGISTER
• Authentication REGISTER
• PRACK, upon request

16
SIP Message Registration Screen Shots
 
17
SIP Message Origination
 
18
SIP Message Termination
19
Incoming Message Filters
20
TOS for SIP Signaling
 
21
Diffserv for SIP Signaling
 
22
Testing SIP Robustness

• Robustness testing
• Passed does not crash, stable, or acceptable
  results
• Failed crashes, unstable, or unacceptable
  results
• Security testing
• It is crucial to identify SIP security holes
• SIP testing tool
• Tests SIP robustness and security
• Comprehensive negative test suites for SIP

23
Real Signaling with real RTP

• Capability to do signaling with audio
• Capability to perform real time measurements
• Capability of using signaling without audio
• Problems of not using real signaling
• Problems of not using real RTP streams
• Real time objective metrics

24
Testing SIP-T
VoIP Network
MGC
MGC
SIP-T
SS7
SS7
SIP-T
SIP-T
H.248/Megaco
H.248/Megaco
SIP Proxy
RTP/RTCP
Trunking Gateway
Trunking Gateway
SS7 GR303 ISDN CAS V5
1000Base-SX/LX 10/100/1000Base-T
POTS
POTS
25
SIP-T Performance Testing Suites

• Performance testing
• Validate and stress-test SS7 ISUP and SIP
  interworking with optional media, over thousands
  of emulated user agents
• SIT-T testing
• Configurable SIP-T calls with intelligent
  protocols
• QoS and CoS testing
• Optional TOS/Diffserv and VLAN options in SIP-T
  media calls, used to measure QoS with PESQ and
  e-model
• Feature testing
• Automated and configurable SIP call set-up,
  teardown, flows, messages

26
Network Impairments and Parameters that Affect
Voice and Video Quality

• Timing Drift
• Route Flapping
• Signaling protocol mismatches
• Network faults
• Link Failures
• Voice Only Impairments
• Echo
• Voice coding algorithms
• A/D and D/A Conversion
• Noise Circuit and External
• Video Only Impairments
• Video coding algorithms
• Fixed vs Variable Frame Rate

• Network Architecture
• Types of Access Links
• QoS controlled Edge Routing
• MTU Size
• Packet Loss (Frame Loss)
• Out of order packets
• One Way Delay (Latency)
• Variable Delays (Jitter)
• Background Traffic (Congestion, Bandwidth,
  Utilization, Network Load, Load Sharing)

27
IP Network Architecture
Affects Data, Voice and Video Quality
28
Network Operating With Constant Delay
Affects Voice and Video Quality
29
End to End Delay Sources
Core Network
Originating LAN
Terminating LAN
Originating Gateway
Edge Router
Terminating Gateway
Edge Router
Core Network Routers

• Fixed
• Serialization
• WAN

• Fixed
• Serialization
• WAN

• Fixed
• Switching
• Propagation
• Serialization
• Variable
• Voice contention
• Data Contention
• Video Contention

• Fixed
• Look ahead
• Encoding
• Buffer
• VAD
• Packetizing

• Fixed
• Switching
• Variable
• Voice contention
• Data Contention
• Video Contention

• Fixed
• Switching
• Variable
• Voice contention
• Data Contention
• Video Contention

• Fixed
• Decoding
• Variable
• De-jitter buffer
• Packet loss Concealment

• Algorithmic delay
• Serialization delay
• Propagation delay
• Component delay

Affects Voice and Video Quality
30
Echo Impairment on Converged network
Delay in IP Network makes Echo sound worse
Tail Circuit
TELR
IP Phone
MG
T1 Link
2 Wire
IP Network
PBX
POTS Phone
ERL
ERLE
IP Phone
Echo Canceller in MG reduces Echo Level
ERLE Echo Return Loss Enhancement ERL Echo
Return Loss TELR Talker Echo Loudness Rating
Analog 4-Wire Link
Analog 2-Wire Link
RX
RX
EM
T1 Link
Affects Voice Quality
TX
POTS Phone
Hybrid Transformer
TX
Impedance Mismatch
PBX
31
Echo Impairment on Converged network

• Electrical Coupling
• Impedance Mismatch (Hybrid)
• Acoustical Coupling
• Speakerphone

Affects Voice Quality
Converged Network
Path A to B
Path B to A
Echo Path Side A (250ms)
Echo Path Side B (250ms)
Echo is caused by impedance mismatches in hybrid
circuits (2w to 4w) and feedback between the
telephone mouth piece and ear piece
32
Effect of Delay on Voice Quality
Voice Quality
gt 25ms Echo Cancellation Required
PSTN
lt150 ms (with echo cancellation) acceptable
150-400 ms acceptable if delay expected
gt 400 ms unacceptable for most applications
33
Effect of Echo Level on Voice Quality
Less Echo
Less Echo
More Echo
More Echo
Affects Voice Quality
TELR Talker Echo Loudness Rating (Signal to
Echo Ratio)
34
Network with Variable Delays (Jitter)

• Variable processing delay
• A busy router or switch will take longer to look
  up the routing (address) table
• Queuing delay
• Network congestion

Affects Voice and Video Quality
Delay (ms)
Time (s)
35
Jitter Characteristics
Delay (ms)
Good
Delay (ms)
Bad
Severe
Delay (ms)
Affects Voice and Video Quality
36
Packet LossExample Queue Management
Threshold
Affects Voice and Video Quality
Bit Bucket
37
Speech Compression Impairment
Voice Quality
G.711 Best Quality
Common Compression Types G.711, G.729, G.728,
G.726, G.723, AMR, EVRC
38
VAD Voice Activity Detection
Timing may be different
No VAD
Affects Voice Quality
VAD
Data is intentionally not sent during times of
Silence
39
Impact Of Packet Size
Affects Data, Voice and Video Quality
10 Bytes
10 ms Speech
20 Bytes
Normal size for VoIP applications
20 ms Speech
40 Bytes
40 ms Speech
80 Bytes
80 ms Speech

• Typically Packets are kept small for best results
• Many equipment manufacturers use dynamic packet
  size to optimize for network conditions

40
Mechanisms for Assuring QOS
Data
Voice
Video
Triple Play

• Class of Service (COS) ITU-T Y.1541 defines the 5
  classes of service and their application
• Type of Services (TOS)
• TOS and COS are both elements with in an IP
  Packet
• DIFSER and RSVP provide mechanisms to improve QOS

Affects Data, Voice and Video Quality
41
TIA-921 and ITU-T G.NIMMTest Profiles Based on
QoS (Y.1541) Classes
Different test profiles for different Service
Level Agreements (SLAs)
Profile B Best Effort Managed Network Table 3
Profile C Un-Managed Network Table 4
Profile A Well Managed Network Table 2
42
Early Voice Quality Testing
43
Voice Quality Testing

• Active (Intrusive) Testing
• Sends, Receives and compares Wave Files to
  measure voice quality
• MOS (Mean Opinion Score)
• PSQM, PSQM (Perceptual Speech Quality
  Measurement)
• PESQ (Perceptual Evaluation of Speech Quality)
• R-Value and J-MOS derived from PESQ
• Passive Testing
• R-Value ITU-T P.VTQ
• Measures Voice Quality on RTP Packets
• Based on E-model
• Japan J-MOS
• Similar Techniques can be used to measure Video
  Quality
• P.563 (ITU-T recommendation) 3SQM, P-Stream
• Measures Voice Quality of Voice traffic based on
  Audio Siginal
• Provides an estimate of PSQM

44
Active (Intrusive) Voice Quality Testing
MOS, PSQM, PSQM, PESQ, R-Factor (PESQ Derived)
DUT
Send Wave Files Example (ITU-T Female Nice File
with Pilot Tone)
Receive Wave Files
Measures Voice Quality by Comparing Sent and
Received Wave files
Sent (Green) and Received (Orange) wave files
Expanded Sent (Green) and Received (Orange) wave
files
PESQ Score vs Number of PESQ Measurements Values
are different for Male, Female, different Wave
Files and different Languages
45
Passive Voice and Video Quality Testing
R-Factor/Emodel
Measure Video Quality MOS-LQ MOS-CQ MOS-PQ J-MOS
Network R User R Burst statistics Diagnostic data
PSTN
IP Network
RTP
E1/T1/E3/T3/PRI/GR303, V5,SLC96
Trunking Gateway
RTP
Measure Video Quality MOS-LQ MOS-CQ MOS-PQ J-MOS
Network R User R Burst statistics Diagnostic data
ITU-T P.VTQ
IP Telephone
46
Passive Voice Quality TestingP.563 (P-Stream,
3SQM)
RTP
DUT
Receive Audio
Estimates Voice Quality based on 3 Characteristic
of Received Audio
47
Voice Quality Measurements
Emodel
P.861 PSQM/PSQM
P.862 PESQ
PAMS
0
5
4.5
3.88
3.65
3.40
3.13
2.84
6.5
1
-0.5
48
Sample Voice Quality Test Results
MOS
PSQM
PESQ
Created by inducing packets lost
G.711
G.723.1 (6300 bps)
Comparison of Scores for G.711 and G.723.1 (6300
bps)
49
Video Quality Measurement

• Video Compression
• Video Compress schemes affect the video quality
• H.261, H.263, H.264, VC1, MPEG-1, MPEG-2, MPEG-4,
  Microsoft AVI, Windows Media (.wmv, .asf),
  RealMedia (.rm), QuickTime (.mov)
• Interactive real-time applications (e.g., video
  conferencing, voice over IP) are sensitive to
  latency and Frame Rate
• Typical Video Quality Metrics
• Objective MOS
• Blockiness
• Blur
• PSNR
• Spatial Resolution
• Temporal Resolution

• SNR
• Edge Noise
• Jerkiness
• Error Blocks
• Object Retention
• Color Reproduction Accuracy

50
Video Quality MeasurementBlockiness
Original
Blockiness
51
Video Quality MeasurementReference and Blocky
Video
Blocky
Original (Reference)
52
Video Quality MeasurementBlur
Original
Blur
53
Video Quality MeasurementNoise
Original
Noise
54
Video Quality MeasurementSpatial (Pixel)
Resolution
128X128
32X32
8X8
Spatial Resolution
Department of Computer ScienceUniversity of
Canterbury http//www.cosc.canterbury.ac.nz/peopl
e/mukundan/covn/Imgresl.htm
55
Video Quality Measurement Temporal (Motion)
Resolution
56
Video Quality MeasurementModels

• Video Quality Metrics (VQM)
• ITU-T SG9 and VQEG are working on standard
• TV Model - optimized for higher bit-rate digital
  television systems with no frame dropping (e.g.,
  MPEG-2)
• Videoconferencing Model - optimized for lower
  bit-rate videoconferencing systems that drop
  frames (e.g., H.261, H.263).
• General Model - optimized for a wide range of
  video quality (videoconferencing, TV)
• Developer Model - optimized for a wide range of
  video quality (videoconferencing, TV) with the
  added constraint of fast computation.
• PSNR Model - based on the traditional peak
  signal-to-noise-ratio (PSNR) calculation.

57
Video Quality MeasurementTechniques

• Full Reference (ITU-T J.144R and BT.1683)
• Video quality is calculated by comparing the
  received video with the complete original video
• Reduced Reference (ANSI T1.801.03-2003 and ITU-T
  J.143)
• Spatial and temporal information are calculated
  from original video and transmitted to the
  receiving end
• Video quality is calculated by comparing the
  received video with the reduced reference
• No Reference Passive
• Video quality on based only on the received
  information (picture content)
• Video quality is derived from RTP packet
  information similar to R-Factor (E-Model) for
  voice quality

58
Video Quality Measurement Full Reference
Original Video
Received Video
Transmitted Video Signal
Processor
Processor
Original Video

• Video Quality Score
• MOS
• Blockiness
• Blur
• PSNR

Full Reference
59
Video Quality Measurement Reduced Reference
Transmitted Video
Received Video
Processor
Processor
Reduced-Reference Signal Low Rate

• Video Quality Score
• MOS
• Blockiness
• Blur
• PSNR

• Reduced-Reference
• Spatial (Pixel) and Temporal (Motion) Information

60
Video Quality Measurement No-Reference
Transmitted Video
Received Video
Processor
Processor
No-Reference Picture Content or Passive
monitoring of RTP

• Video Quality Score
• MOS
• Blockiness
• Blur
• PSNR

61
Types of Testing

• Test these DUTs
• IP PBX
• Gateways
• IP Phone
• Servers
• Firewalls
• IAD

62
Types of Testing

• Speech Quality Measurements
• PESQ
• PSQM
• MOS
• R Factor
• Echo Delay
• Round Trip Delay
• Echo Return Loss
• Signal Pass Noise
• Noise Level
• Video Quality Measurements
• MOS
• Blockiness
• Blur
• PSNR

• Call Establishment
• Start Dial Signal Delay
• Post Dial Delay
• Call Duration
• Ring Duration
• Call Disconnect
• Connection Disconnect Delay
• Release on Request
• Call Statistics
• Connection set-up failures
• Connection premature disconnect
• Call completion percentages

• Transport Layer Measurements
• One-way Transmission Time
• Roundtrip Transmission Time
• Jitter
• Packets out of order
• Packet Loss

63
Video Telephony TestingDistributed Testing

• Isolate Network Problems
• Results Over Time
• Results by Group

64
Good test methodology

• Implementation, Validation Observation
• Conformance testing
• IETF 3261 new SIP RFCs
• Stress testing
• Scriptable call flow
• Bulk signaling with real RTP
• Robustness testing
• SIPPING Torture Test
• PROTOS / ETSI TIPHON
• Visual protocol analysis
• Application content decoding

65
Analyze Assure Accelerate