Automated SIP Interoperability Testing

1
Automated SIP Interoperability Testing
Archana Rao and Henning Schulzrinne Department of
Computer Science Columbia University http//www.cs
.columbia.edu/IRT/voip-testbed/
SIP 2009 (Paris, January 2009)
2
Overview

• The problem of SIP interoperability
• Far from perfect
• Damages brand, harms customers, encourages
  single-vendor deployments
• Testbed Overview
• Architecture
• Components
• Experiments
• Interoperability study
• End-client characteristics
• Summary

3
Interoperability

Is it a real problem ?
Excerpts from an email posted on IETF RAI mailing
list

• Hi Henning,You may remember me from IETFs past
  -- I haven't attended any in some time because I
  couldn't find any really interesting projects.
  I'm finally getting into SIP. I've got Speakeasy
  VoIP service, two sipphone accounts, a Cisco 7960
  and a copy of x-ten on my Mac. And I still can't
  make it work. Voice flows in one direction only.
  I'm not even behind a NAT or firewall -- both
  machines have global addresses, with no port
  translations or firewalls.
• I've been working with Internet protocols for
  over 20 years. I've implemented and contributed
  to them. And if I can't figure out how to make
  this stuff work, how is the average grandmother
  expected to do so?
• SIP is unbelievably complex, with
  extraordinarily confusing terms. There must be
  half a dozen different "names" -- Display Name,
  User Name, Authorization User Name, etc -- and a
  dozen "proxies". Even the word "domain" is
  overloaded a half dozen different ways. This is
  ridiculous!Sorry. I just had to get this off my
  chest. Regards, Phil Karn.

Reference http//www1.ietf.org/mail-archive/web/r
ai/current/msg00082.html
4
Interoperability issues

• Why do we have interoperability problems?
• SIP is complex
• More than 150 RFCs and 500 active Internet Drafts
• Efforts like draft-ieft-sip-hitchhikers-guide
  help to a certain extent, but not sufficient
• SIP has no proposed architecture/profile
• SIP usage in variety of environments and domains
  make it impossible
• Standardization from IETF is different from that
  of ISO/ITU
• SIP is continually evolving
• Non trivial for implementers to follow the
  life-cycle from IDs to RFCs
• Walled gardens, non-standard environments
• Vendors need to make products that work in their
  customers environment
• Poor implementations, intentional
  non-interoperability etc.

5
Improving interoperability

• Current efforts to combat SIP interoperability
  issues
• SIP Interoperability (SIPit) events
• Week long gatherings of SIP implementers to test
  interoperability
• Coordinated by SIP Forum
• SIP Forum
• SIP Connect 1.0 (and 1.1) ? outsourced enterprise
  VoIP
• IETF Basic Level of Interoperability for SIP
  Services (BLISS)
• Focus on resolving interpretability issues in SIP
  features
• Line sharing, call parking, automated call
  handling and call queuing
• University of New Hampshire InterOperability
  Laboratory
• Vendor-neutral lab dedicated to testing data
  networking technologies
• 20 different testing programs, each costing about
  20,000 per year

6
Interoperability

• So, isnt the problem solved?
• These forums focus on details of advanced SIP
  features
• Basic-level of interoperability between
  innumerable SIP devices in the market
• Can I take any SIP phone and make a VoIP call,
  through any SIP service provider?

How do we go about it? 1. Identify the basic
real-world scenarios for SIP registration and
session establishment 2. Use our VoIP testbed to
configure a variety of SIP infrastructure 3.
Study the behavior of SIP end-clients and servers
for interoperability
7
Studying interoperability

• Define a minimum set of call flows constituting
  basic interoperability
• RFC 3665, RFC 3666, RFC 4317
• Use a VoIP testbed to realize a variety of
  real-world SIP infrastructure setups
• Columbia VoIP testbed
• Study the behavior of SIP end-clients and servers
  in each of these scenarios
• Interoperability matrices
• Categorization of common issues ?
• Lack of clarity in the specification
• Implementation of an older specification
• Incomplete implementation of the specification
• Incorrect implementation of the specification
• Failure against robustness tests

8
Columbia VoIP testbed

• What?
• VoIP infrastructure for experimentation,
  analysis, testing, prototyping and deployment of
  SIP/VoIP components in a variety of environments.
  
• Part of a multi-university research project
  supported by NSF
• University of North Texas, Purdue University and
  University of California, Davis

• Why?
• Therere no telecom testbeds available for
  research experimentation
• VoIP implementations experiments have lacked
  scientific rigor
• Continuously emerging standards
• Need to develop repeatable, accurate test
  frameworks
• Realistic VoIP experiments require a distributed
  testbed

9
http//www.cs.columbia.edu/IRT/voip-testbed/
http//www.cs.columbia.edu/IRT/voip-testbed/
your equipment here -)
10
Columbia VoIP testbed servers

• 5 SIP servers
• Asterisk, Pbxnsip, Sipd, SER, OpenSER
• 3 Platforms
• Microsoft Windows XP
• Linux Fedora Core 6
• Sun Solaris X
• 3-way Connectivity
• the Internet
• VPN
• PSTN

11
Columbia VoIP testbed UAs

• 20 SIP hard-phones, soft-phones, WI-FI phones
  with different capabilities
• More details www.columbia.edu/ahr2114/VoIPtestb
  ed/end-clients

12
NSF VoIP testbed
Image courtesy http//secnet.csci.unt.edu/7.15.fu
nding_application.ppt
13
Columbia VoIP testbed experiments

• Planned or ongoing
• Interoperability study
• Can any two SIP devices talk to each other?
• SIP end-client characteristics
• Signaling and codec features in SIP devices
• Robustness
• VoIP with/despite NAT
• Performance issues
• Signaling delay, voice mouth-ear delay, packet
  loss resilience
• Servers Scalability, maximum capacity
• Security
• Separate testbed for DOS and other attacks

work in progress
14
Interoperability study call flows

• Registration (RFC 3665)
• Successful new registration
• Unsuccessful registration
• Cancellation of registration
• Update of contact list

• Session Establishment (RFC 3665)
• Successful session establishment
• Session establishment through two proxies
• Session establishment with multiple proxy
  authentication
• Successful session with proxy failure
• Unsuccessful no answer
• Unsuccessful busy
• Unsuccessful no response from user agent
• Unsuccessful temporarily unavailable

• Codec Negotiation (RFC 4317)
• Audio and DTMF
• Audio, video and DTMF
• Audio and video codec reordering

15
Interoperability study matrix

• Interoperability Matrix
• Test case 1 - Successful new registration

16
Interoperability study auth name

• Issue 1 Use of different formats for
  authentication name
• Authorization Digest usernameuser1,realmprox
  y01.sipphone.com,noncexxx, urisipproxy01.si
  pphone.com,responseyyy, algorithmMD5
• Authorization Digest usernamesipuser1_at_sipphon
  e.com, realmproxy01.sipphone.
  com,noncexxx,urisipproxy01.sipphone.com,re
  sponseyyy, algorithmMD5
• Implication
• Registration and call setup failure, if both
  formats are not supported by the UAs
• Instances
• Polycom PVX/VSX, Wengophone dont register with
  3Com / sipd proxies
• Workaround
• Use Asterisk between Polycom PVX and 3Com proxy

17
Interoperability study rport
Issue 2 Behavior in the absence of rport
parameter

• Implication
• Incorrect/incomplete signaling resulting in
  multiple retransmissions and failed transaction
• Instances
• Calls to Cisco 7940 via Sipphone proxy are always
  unsuccessful
• Calls from Polycom VSX via 3com or sipd proxies
  are always unsuccessful
• Workaround
• Nothing

18
Interoperability Study codecs
Issue 3 Codec Negotiation

• Implication Audio/Video failure (mostly
  unidirectional, but bidirectional sometimes)
• Problem Instances
• Audio failure when Cisco calls voicemail service
  of sipphone provider
• Video failure when Polycom PVX calls Xlite via
  Asterisk

• Offer
• v0
• oCisco-SIPUA 28422 0 IN IP4 128.59.17.67
• s SIP Call
• t0 0
• maudio 31030 RTP/AVP 0 8 18 101
• cIN IP4 128.59.17.67
• artpmap0 PCMU/8000
• artpmap8 PCMA/8000
• artpmap18 G729/0
• afmtp18 annexbno
• artpmap101 telephone-event/8000
• afmtp101 0-15
• asendrecv

• Answer
• v0
• oCisco-SIPUA 28422 0 IN IP4 128.59.17.67
• s SIP Call
• t0 0
• maudio 31030 RTP/AVP 18 0 8 101
• cIN IP4 128.59.17.67
• artpmap18 G729/0
• afmtp18 annexbno
• artpmap0 PCMU/8000
• artpmap8 PCMA/8000
• artpmap101 telephone-event/8000
• afmtp101 0-15
• asendrecv

19
UA characteristics features

• Basic
• Configuration, number of lines, call hold, call
  forwarding, DND, NAT support
• Advanced
• Encryption, symmetric RTP, conferencing,
  audio/video codecs, PoE
• Audio quality silence suppression, echo
  cancellation, comfort noise
• Transport, signaling and support protocols TCP,
  TLS, HTTP, DHCP, DNS, TFTP, NTP
• More details at http//www.columbia.edu/ahr2114
  /VoIPTestbed/EndClients.htm

20
UA characteristics robustness
Robustness

• SIP Torture Tests (RFC4475)
• A set of SIP messages aimed at stressing the SIP
  parser, based on experiences at SIPit.
• PROTOS
• Test suite with 4500 INVITE requests, developed
  by University of Oulu

Example Results
21
UA characteristics NATs

• SIP and NAT
• Problematic for both signaling (SIP) and media
  transfer (RTP)
• Circumventing NAT effect solutions exist for
  both end-clients and proxies
• Quest to overcome this, could lead to techniques
  uglier than NAT itself!

• NAT test setup
• NAT realization - Linux iptables, CLICK router
• Studying the behavior in different scenarios
• NAT support enabled in both the proxy server and
  end-client
• NAT support enabled in only one of them etc
• Aiming to analyze the existing practices the
  good, the bad and the ugly!

22
UA with NATs
SIP devices in NAT environment
23
What now?

• Summary
• Tests indicate that basic-level interoperability
  is far from perfect
• Instances of interoperability failures
  illustrated in our IETF workshop paper
• Achieving Interoperability shouldnt be left just
  to vendors
• non-interoperability damages SIP brand, causes
  grief for third parties and harms customers

• Going ahead some ideas
• Establish designated interop liaison for each
  vendor
• calling customer support unlikely to be useful
• Encourage vendors to publish interoperability
  reports
• in standard format
• Provide self-certification tests, supported by a
  remotely accessible test rig

24
Summary

• Current status
• The VoIP testbed VPN connectivity is
  operational
• Systematic study of real-world interoperability
  issues
• Basic-level interoperability is nowhere near 100
• IETF 70 The 1st SIP Forum Workshop on SIP
  Interoperability
• Evaluating SIP end client characteristics

• Testbed plans
• Experiments on NAT, performance, security
• Make the testbed accessible and useful for
  engineers deployments
• Create a knowledge repository about SIP devices
  tools for interoperability testing