Quantifying Skype User Satisfaction

1
Quantifying Skype User Satisfaction

• Carol K. L. Wong
• 19 March, 2007
• CSC7221

2
Skype

• a P2P Internet telephony network
• gt 2 million Skype downloads
• 85 millions users worldwide

From Wikipedia.org
3
Skypes Performance

• Q Is Skype providing a good enough voice phone
  service to the users?

4
Comparison of Proposed Existing Methods
Speech quality measures Proposed User Satisfaction Index
To quantify Speech quality User satisfaction
Built upon Subjective mean opinion score (MOS) Call duration
Predictors Distortion of signals QoS factors the bit rate, network latency, network delay variations, packet loss
5
Methodology

• Collect Skype VoIP sessions and their network
  parameters
• Analysis of Call Duration and propose an
  objective index, the User Satisfaction Index
  (USI), to quantify the level of user satisfaction
  
• Validate USI by an independent set of metrics
  that quantify the interactivity and smoothness of
  a conversation.

6
Trace Collection

• Collect Skype VoIP sessions and their network
  parameters.
• Present the network setup and filtering method
  used in the traffic capture stage.
• Introduce the algorithm for extracting VoIP
  sessions from packet traces
• Strategy to sample path characteristics.
• Summarize the collected VoIP sessions.

7
Network Setup
8
Capturing Skype Traffic

• Use 2-phase filtering to identify Skype VoIP
  sessions
• filter and store possible Skype traffic on the
  disk.
• apply an off-line identification algorithm on the
  capture packet traces to extract actual Skype
  sessions.

9
Detect Possible Skype Traffic

• Known properties of Skype clients
• dynamic port number chosen randomly when the
  application is installed and can be configured by
  users Skype port
• In the login process, submits HTTP requests to a
  well-known server, ui.skype.com

10
Heuristic to Detect Skype Hosts and their Skype
Ports

• treat sender for each HTTP request sent to
  ui.skype.com as a Skype host
• choose the port number used most frequently for
  outgoing UDP packets sent from that host within
  the next 10s as the Skype port.
• classify all peers that have bi-directional
  communication with the Skype port as Skype hosts.
• maintained a table of identified Skype hosts and
  their respective Skype ports, and
• recorded all traffic sent from or to these (host,
  port) pairs.

11
Identification of VoIP Sessions

• regard An active flow as a valid VoIP session if
• The flows duration gt 10s.
• The average packet rate is within a reasonable
  range, (10, 100) pkt/s.
• The average packet size is within (30,300) bytes.
  
• The EWMA of the packet size process must be
  within (35, 500) bytes all the time.

12
Relayed Session

• Merge a pair of flow into a relayed session if
• The flows start and finish time are close to
  each other with errors lt 30s
• The ratio of their average packet rates lt 1.5
  and
• Their packet arrival processes are positively
  correlated with a coefficient gt 0.5.

13
Path Characteristics Measurement

• RTT and their jitters
• send out ICMP and traceroute-like probe packets
  to measure paths RTT while capturing Skype
  traffic
• used

14
Collected VoIP Sessions
Category Calls Hosts Cens. TCP Duration Bit Rate (mean/std) Avg. RTT (mean/std)
Direct 253 240 1 7.10 (6.43, 10.42) min 32.21 Kbps / 15.67 Kpbs 157.3 ms/ 269.0 ms
Relayed 209 369 5 9.10 (3.12,5.58) min 29.22 Kbps / 10.28 Kpbs 376.7 ms/ 292.1 ms
Total 462 570 6 8.00 (5.17,7.70) min 30.86 Kbps / 13.57 Kpbs 256.5 ms/ 300.0 ms
15
Methodology

• Collect Skype VoIP sessions and their network
  parameters
• Analysis of Call Duration and propose an
  objective index, USI, to quantify the level of
  user satisfaction
• Validate USI by an independent set of metrics
  that quantify the interactivity and smoothness of
  a conversation.

16
Analysis of Call Duration

• Develop a model to describe the relationship
  between call duration and QoS factors.
• propose an objective index, the User Satisfaction
  Index (USI) to quantify the level of user
  satisfaction.
• validate USI by voice interactivity measures.

17
Survival Analysis

• With proper transformation, the relationships of
  session time and predictors can be described well
  by the Cox Proportional Hazards model (Cox Model)
  in survival analysis.

18
Survival Curves for Sessions with Different Bit
Rate Levels
19
Survival Curves for Sessions with Different Bit
Rate Levels
Group Median (min)
1 2
3 20

• The log-rank test strongly suggests that call
  duration varies with different levels of bit
  rates.

Bit rates Last gt 40 min
lt 25 Kbps 3
gt 35 Kbps 30
20
Relation of the bit rate with call duration

• The trend of median duration shows a strong,
  consistent, positive, correlation with the bit
  rate.

21
Effect of Network Conditions

• Network conditions are also considered to be one
  of the primary factors that affect voice quality.
  
• the fluctuations in the data rate observed at the
  receiver should reflect network delay variations
  to some extent.
• used
• jitter to denote the standard deviation of the
  bit rate, and
• packet rate jitter, or pr.jitter, to denote the
  standard deviation of the packet rate.

22
Effect of Round-Trip Times

• divided sessions into 3 equal-sized groups based
  on their RTTs, and compare their lifetime
  patterns with the estimated survival functions.
• the 3 group differ significantly

Group Median duration of sessions (min)
RTTs gt 270 ms 4
RTTs 80 - 270 ms 5.2
RTTs lt 80 ms 11
23
Effect of Jitter

• Jitter has a much higher correlation with call
  duration than RTT.

24
Group Median session time (min)
Jitter gt 2 Kbps 3
1 lt Jitter lt 2 Kbps 11
Jitter lt 1 Kbps 21
These groups differs statistically
25
QoS related to Call Duration

• most of the QoS factors they defined, including
• the source rate,
• RTT, and
• jitter
• are related to call duration.

26
Collinearity

• Given that the bit rate jitter are
  significantly correlated, true source of user
  dissatisfaction is unclear.
• Use the Cox model and treat QoS factors, e.g. the
  bit rate, as risk factors or covariates i.e. as
  variables that can cause failures.
• The hazard function of each session is decided
  completely by a baseline hazard function and the
  risk factors related to that session.

27
Collinearity

• 7 factors - bit rate (br),packet rate
  (pr),jitter, pr.jitter, packet size (pktsize),
  and round trip time (rtt)

br pr jitter pr.jitter pktsize rtt
br -
pr - --- --
jitter -
pr.jitter ---
pktsize
rtt - --
/- positive or negative correlation
collinearity is computed by Kendalls t
statistic (Pearsons product moment statistic
yields similar results)
28
Collinearity

• the bit rate, packet rate, and packet size are
  strongly interrelated
• jitter and packet rate jitter are strongly
  interrelated.
• the bit rate, jitter, and RTT are retained in the
  model

29
Cox Model

• define the risk factors of a session as a risk
  vector Z
• h(tZ) h0(t) exp(btZ) h0(t)exp(Spk1bkZk)
• h(tZ) - the hazard rate at time t for a session
  with risk vector Z
• h0(t) - the baseline hazard function computed
  during the regression process
• b (b1,, bp)t - the coefficient vector that
  corresponds to the impact of risk factors.
• Zp is the pth factor of the session

30
The Cox model

• 2 sessions with risk vectors Z and Z, the
  hazard ratio
• h(tZ)/ h(tZ) exp(Spk1bkZk bkZk)
• is time-independent constant
• Hence, the validity of the model relies on the
  assumption the hazard rates for any 2 sessions
  must be in proportion all the time.

31
Sampling of QoS Factors

• In the regression modeling, we use a scalar value
  for each risk factor to capture user perceived
  quality.
• Divide the original series s into sub-series of
  length w, from which network conditions are
  sampled.
• Choose one of the min, average and max measures
  taken from sampled QoS factors having length
  s/w depending on their ability to describe
  the user perceived experience during a call.

32
Evaluation

• evaluate all kinds of measures and window sizes
  by
• fitting the extracted QoS factors into the Cox
  model and
• comparing the models log-likelihood, i.e. an
  indicator of goodness-of-fit.
• Finally, the max bit rate and min jitter are
  chosen, both sampled with a window of 30s.

33
Model Fitting

• the Cox model assumes a linear relationship
  between the covariates and the hazard function
• the impact of the covariates on the hazard
  functions with the following equation
• This corresponds to a Poisson regression model if
  h0(s) is known.

ti - the censoring status of session i, f(Z)
the estimated functional form of the covariate Z.
34

• The influence of the bit rate is not proportional
  to its magnitude - scale transformation.

35

• The RTT factor has an approximate linear impact.

36
Jitter factor
37
Verification

• Employ a more generalized Cox model that allows
  time-dependent coefficients to check the
  proportional hazard assumption by hypothesis
  tests. After adjustment, none of covariates
  reject the linearity hypothesis at a 0.1, the
  transformed variables have an approximate linear
  impact on the hazard functions.
• Use the Cox and Snell residuals ri (for session
  i) to assess the overall goodness-of-fit of the
  model. Except for a few sessions that have
  unusual call duration, most sessions fit the
  model very well.

38
Model Interpretation
Variable Coef, eCoef Std. Err. z P gt z
br.log -2.15 0.12 0.13 -16.31 0.00e00
jitter.log 1.55 4.7 0.09 16.43 0.00e00
rtt 0.36 1.4 0.18 2.02 4.29e-02
b - coeff

• define the factors relative weights as their
  contribution to the risk score, i.e., btZ.

39
Relative Influence of Difference QoS for each
session

• The degrees of user dissatisfaction caused by the
  bit rate, jitter and round-trip time are
  46531.

40
Conclusion

• Possible to improve user satisfaction by fine
  tuning the bit rate used.
• As the use of relaying does not seriously degrade
  user experience, higher round-trip times do not
  impact on users very much.
• Jitters have much more impact on user perception.
• The choice of relay node should focus more on
  network conditions, i.e., the level of
  congestion, rather than rely on network latency.

41
User Satisfaction Index (USI)

• As the risk score btZ represents the levels of
  instantaneous hang up probability, it can be seen
  as a measure of user intolerance. Accordingly,
  define the USI of a session as its minus risk
  score
• USI - btZ 2.15xlog(bit rate)
  1.55xlog(jitter) 0.36xRTT
• where the bit rate, jitter, and RTTs are sampled
  using a 2-level sampling approach

42
The prediction is based on the median USI for
each group. y-axis is logarithmic to make the
short duration groups clearer.
43
Advantages of USI over Other Objective Sound
Quality Measures

• USIs parameters are readily accessible
• the 1st and 2nd moment of the packet counting
  process
• can be obtained by simply counting the number
  and bytes of arrival packets
• the round-trip times.
• Usually available in peer-to-peer applications
  for overlay network construction and path
  selection.
• developed the USI based on passive measurement
  rather than subjective surveys, it can also
  capture sub-conscious reactions of participants,
  which may not be accessible through surveys.

44
Methodology

• Collect Skype VoIP sessions and their network
  parameters
• Analysis of Call Duration and propose an
  objective index, the User Satisfaction Index
  (USI), to quantify the level of user satisfaction
  
• Validate USI by an independent set of metrics
  that quantify the interactivity and smoothness of
  a conversation.

45
Validation

• Results of the validation tests using a set of
  independent measures derived from user
  interactivities show a strong correlation between
  the call durations and user interactivities. This
  suggests that the USI based on call duration is
  significantly representative of Skype user
  satisfaction.