Quality and Performance Evaluation of VoIP End-points - PowerPoint PPT Presentation

1 / 28
About This Presentation
Title:

Quality and Performance Evaluation of VoIP End-points

Description:

Hardware End-points. Cisco, 3Com and Pingtel IP phones ... Low (mostly 80ms) for hardware IP phones. Software clients: lowest for Messenger 2000 (68.5ms) ... – PowerPoint PPT presentation

Number of Views:31
Avg rating:3.0/5.0
Slides: 29
Provided by: henningsc
Category:

less

Transcript and Presenter's Notes

Title: Quality and Performance Evaluation of VoIP End-points


1
Quality and Performance Evaluation of VoIP
End-points
  • Wenyu Jiang
  • Henning Schulzrinne
  • Columbia University
  • NYMAN 2002
  • September 3, 2002

2
Motivations
  • The quality of VoIP depends on both the network
    and the end-points
  • Extensive QoS literature on network performance,
    e.g., IntServ, DiffServ
  • Focus is on limiting network loss delay
  • Little is known about the behavior of VoIP
    end-points

3
Performance Metrics for VoIP End-points
  • Mouth-to-ear (M2E) delay
  • C.f. network delay
  • Clock skew
  • Whether it causes any voice glitches
  • Amount of clock drift
  • Silence suppression behavior
  • Whether the voice is clipped (depends much on
    hangover time)
  • Robustness to non-speech input, e.g., music
  • Robustness to packet loss
  • Voice quality under packet loss

4
Measurement Approach
  • Capture both original and output audio
  • Use adelay program to measure M2E delay
  • Assume a LAN environment by default
  • Serve as a baseline of reference, or lower bound

5
VoIP End-points Tested
  • Hardware End-points
  • Cisco, 3Com and Pingtel IP phones
  • Mediatrix 1-line SIP/PSTN Gateway
  • Software clients
  • Microsoft Messenger, NetMeeting (Win2K, WinXP)
  • Net2Phone (NT, Win2K, Win98)
  • Sipc/RAT (Solaris, Ultra-10)
  • Robust Audio Tool (RAT) from UCL as media client
  • Operating parameters
  • In most cases, codec is G.711 ?-law, packet
    interval is 20ms

6
Example M2E Delay Plot
  • 3Com to Cisco, shown with gaps gt 1sec
  • Delay adjustments correlate with gaps, despite
    3Com phone has no silence suppression

7
Visual Illustration of M2E Delay Drop, Snapshot 1
  • 3Com to Cisco 1-1 case
  • Left/upper channel is original audio
  • Highlighted section shows M2E delay (59ms)

8
Snapshot 2
  • M2E delay drops to 49ms, at time of 416

9
Snapshot 3
  • Presence of a gap during the delay change

10
Effect of RTP M-bits on Delay Adjustments
  • Cisco phone sends M-bits, whereas Pingtel phone
    does not
  • Presence of M-bits results in more adjustments

11
Sender Characteristics
  • Certain senders may introduce delay spikes,
    despite operating on a LAN

12
Average M2E Delays for IP phones and sipc
  • Averaging the M2E delay allows more compact
    presentation of end-point behaviors
  • Receiver (especially RAT) plays an important role
    in M2E delay

13
Average M2E Delays for PC Software Clients
  • Messenger 2000 wins the day
  • Its delay as receiver (68ms) is even lower than
    Messenger XP, on the same hardware
  • It also results in slightly lower delay as sender
  • NetMeeting is a lot worse (gt 400ms)
  • Messengers delay performance is similar to or
    better than a GSM mobile phone.

A B A?B B?A
MgrXP (pc) MgrXP (notebook) 109ms 120ms
Mgr2K (pc) MgrXP (notebook) 96.8ms 68.5ms
NM2K (pc) NM2K (notebook) 401ms 421ms
Mobile (GSM) PSTN (local number) 115ms 109ms
14
Delay Behaviors for PC Clients, contd.
  • Net2Phones delay is also high
  • 200-500ms
  • V1.5 reduces PC-gtPSTN delay
  • PC-to-PC calls have fairly high delays

A B A?B B?A
N2P v1.1 NT P-2 (pc2) PSTN (local number) 292ms 372ms
N2P v1.5 NT P-2 (pc2) PSTN (local number) 201ms 373ms
N2P v1.5 W2K K7 (pc) PSTN (local number) 196ms 401ms
N2P v1.5 W2K K7 (pc) N2P v1.5 W98 P-3 (notebook2) 525ms 350ms
15
Effect of Clock Skew Cisco to 3Com, Experiment
1-1
  • Symptom of playout buffer underflow
  • Waveforms are dropped
  • Occurred at point of delay adjustment
  • Bugs in software?

16
Clock Skew Rates
  • Mostly symmetric between two devices
  • RAT (Ultra-10) has unusually high drift rates, gt
    300 ppm (parts per million)
  • High clock skews confirmed in many (but not all)
    PCs and workstations

Drift Rates (in ppm) 3Com Cisco Mediatrix Pingtel RAT
3Com -8.3 55.4 43.3 41.2 -333
Cisco -55.2 -0.4 -11.8 -12.1 -381
Mediatrix -43.1 11.7 1.3 -0.8
Pingtel -40.9 12.7 2.8 -3.5 -380
RAT 343 403 376 12.3
17
Drift Rates for PC Clients
  • Drift Rates not always symmetric!
  • But appears to be consistent between Messenger
    2K/XP and Net2Phone on the same PC
  • Existence of 2 clocking circuits in sound card?

A B A?B B?A
MgrXP (pc) MgrXP (notebook) 172 87.7
Mgr2K (pc) MgrXP (notebook) 165 85.6
NM2K (pc) NM2K (notebook) ? -33?
Net2Phone NT (pc2) PSTN 290 -287
Net2Phone 2K (pc) PSTN 166 82
Mobile (GSM) PSTN 0 0
18
Packet Loss Concealment
  • Common PLC methods
  • Silence substitution (worst)
  • Packet repetition, with optional fading
  • Extrapolation (one-sided)
  • Interpolation (two-sided), best quality
  • Use deterministic bursty loss pattern
  • 3/100 means 3 consecutive losses out of every 100
    packets
  • Easier to locate packet losses
  • Tested 1/100, 3/100, 1/20, 5/100, etc.

19
PLC Behaviors
  • Loss tolerance (at 20ms interval)
  • By measuring loss-induced gaps in output audio
  • 3Com and Pingtel phones 2 packet losses
  • Cisco phone 3 packet losses
  • Level of audio distortion by packet loss
  • Inaudible at 1/100 for all 3 phones
  • Inaudible at 3/100 and 1/20 for Cisco phone, yet
    audible to very audible for the other two.
  • Cisco phone is the most robust
  • Probably uses interpolation

20
Effect of PLC on Delay
  • No affirmative effect on M2E delay
  • E.g., sipc to Pingtel

21
Silence Suppression
  • Why?
  • Saves bandwidth
  • May reduce processing power (e.g., in
    conferencing mixer)
  • Facilitates per-talkspurt delay adjustment
  • Key parameters
  • Silence detection threshold
  • Hangover time, to delay silence suppression and
    avoid end clipping of speech
  • Usually 200ms is long enough Brady 68

22
Hangover Time
  • Measured by feeding ON-OFF waveforms and monitor
    RTP packets
  • Cisco phones is the longest (2.3-2.36 sec), then
    Messenger (1.06-1.08 sec), then NetMeeting
    (0.56-0.58 sec)
  • A long hangover time is not necessarily bad, as
    it reduces voice clipping
  • Indeed, no unnatural gaps are found
  • Does waste a bit more bandwidth

23
Robustness of Silence Detectors to Music
  • On-hold music is often used in customer support
    centers
  • Need to ensure music is played without any
    interruption due to silence suppression
  • Tested with a 2.5-min long soundtrack
  • Messenger starts to generate many unwanted gaps
    at input level of -24dB
  • Cisco phone is more robust, but still fails from
    input level of -41.4dB

24
Acoustic Echo Cancellation
  • Important for hands-free/conferencing (business)
    applications
  • Primary metric Echo Return Loss (ERL)
  • Measured by LAN-sniffing RTP packets
  • Most IP phones support AEC
  • ERL depends slightly on input level and
    speaker-phone volume
  • Usually gt 40 dB (good AEC performance)

IP Phone 3Com Cisco ipDialog Pingtel Snom-100
ERL (dB) 40-45 53-? 49-54 33-42 ?-5 (no AEC)
25
M2E Delay under Jitter
  • Delay properties under the LAN environment serves
    as a baseline of reference
  • When operating over the Internet
  • Fixed portion of delay adds to M2E delay as a
    constant
  • Variable portion (jitter) has a more complex
    effect
  • Initial test
  • Used typical cable modem delay traces
  • Tested RAT to Cisco
  • No audible distortion due to late loss
  • Added delay is normal

26
M2E Delay under Jitter, contd.
  • Cisco phone generally within expectation
  • Can follow network delay change timely
  • Takes longer (10-20sec) to adapt to decreasing
    delay
  • Does not overshoot playout delay
  • More end-points to be examined

Artificial Trace
Real Trace with Spikes
27
Conclusions
  • Average M2E Adelay
  • Low (mostly lt 80ms) for hardware IP phones
  • Software clients lowest for Messenger 2000
    (68.5ms)
  • Application (receiver) most vital in determining
    delay
  • Poor implementation easily undoes good network
    QoS
  • Clock skew high on SW clients (RAT, Net2Phone)
  • Packet loss concealment quality
  • Acceptable in all 3 IP phones tested, w. Cisco
    more robust
  • Silence detector behavior
  • Long hangover time, works well for speech input
  • But may falsely predict music as silence
  • Acoustic Echo Cancellation good on most IP
    phones
  • Playout delay behavior good based on initial
    tests

28
Future Work
  • Further tests with more end-points on how jitter
    influences M2E delay
  • Measure the sensitivity (threshold) of various
    silence detectors
  • Investigate the non-symmetric clock drift
    phenomena
  • Additional experiments as more brands of VoIP
    end-points become available
Write a Comment
User Comments (0)
About PowerShow.com