Title: Advances in Network-adaptive Video Streaming
1Advances in Network-adaptive Video Streaming
- Bernd Girod
- J. Chakareski, M. Kalman, Y. J. Liang, E. Setton,
R. Zhang - Information Systems LaboratoryDepartment of
Electrical Engineering - Stanford University
2Streaming media a huge success
- Hundreds of thousands of streaming media servers
deployed - gt 1 million hours of streaming media content
produced per month - Hundreds or millions streaming media players
- RealPlayer
- Most popular Internet application second only to
Internet Explorer Media Metrix - More than 400 million unique registered users
- More than 200,000 new users per day
- Open source code
3Internet Media Streaming
Streaming client
DSL
Media Server
Internet
56K modem
wireless
- Challenges
- compression
- rate scalability
- error resiliency
- low latency
- Best-effort packet network
- low bit-rate
- variable throughput
- variable loss
- variable delay
4Outline
- What is network-adaptive video streaming?
- Better delivery of video packets by considering
source coding, signal processing, and packet
transport jointly - Application-layer joint source-channel coding
techniques for the Internet - This talk review recent advances in
- Adaptive media playout
- Rate-distortion optimal packet scheduling
- Network-adaptive packet dependency management
5Adaptive Media Playout
Idea reduce latency and packet loss
simultaneously by continuously adapting playout
deadline to network conditions
Fixed deadline
Flexible deadline
5 packet loss 2 sec average receiver buffer
Steinbach, Färber, Girod, ICIP 2001
6Modification of Playout Speed
- Video adaptation of display rate
- Audio and speech Stretching based on time-domain
interpolation algorithm WSOLA Verhelst et al.,
1993, Liang 2001
Template
Pitch- period
0
2
1
3
4
Original packet
7Audio Speed Adjustment
- Waveform Similarity Overlap Add (WSOLA) method
allows speed adjustment without changing pitch - Speech demo
- Music demo
slower 30
original
faster 30
original
slower 30
faster 30
8Reduced Pre-roll Time for Stored Streams
Probability of buffer underflow lt 1
Kalman, Steinbach, Girod, ISCAS 2002
lG1.092, lB0.42, TG20 sec, TB2 sec, TRTT220
ms
9Rate Scalability by Playout Speed Adjustment
Server
Channel (mean throughput)
(29.2 dB, 53.3 kbps)
25 kbps
20 kbps
55 kbps
50 kbps
85 kbps
95 kbps
100 kbps
(33.1 dB, 93.60.984.2 kbps)
10First Things First Smart Prefetching
Idea Send more important packets earlier to
allow for more retransmissions
Server
Internet
Client
11Streaming as a Packet Scheduling Problem
server
pre-encoded media units
transmission opportunities
time
- Which media units should be selected for
transmission, and when? - Requirements
- Meet rate constraint
- Meet latency constraint
- Maximize reconstruction quality
- Rate-distortion framework proposed, e.g., in
Podolsky, McCanne, Vetterli 2000 Miao, Ortega
2000 Chou, Miao 2001
12Markov Decision Tree for One Packet
... N transmission opportunities before
deadline
13Packet Delay Jitter and Loss
pdf
e
(1-e)
loss
k
?
delay
14Source Description
I
I
P
P
I
B
B
B
P
P
P
I
B
B
B
P
A
A
- Each media packet n is labeled by
- Bn size in bits of data unit n
- Ddn distortion reduction if n is decoded
- tn decoding deadline for n
15Source Description
I
I
P
B
P
P
I
B
B
P
P
I
B
B
B
P
A
A
- Each media packet n is labeled by
- Bn size in bits of data unit n
- Ddn distortion reduction if n is decoded
- tn decoding deadline for n
16R-D Optimized Streaming Performance
PSNR dB
- Foreman
- 120 frames
- 10 fps, I-P-P-
- H.263 2 Layer SNR scalable
- 20 frame GOP
- Copy Concealment
- 20 loss forward and back
- G-distributed delay
- ? 10 ms
- µ 50 ms
- s 23 ms
- Pre-roll 400ms
Bit-Rate kbps
17R-D Optimized Streaming with a Proxy Server
Last hop
Chakareski, Chou, Girod, Asilomar 2002, MMSP
2002
18RaDiO Edge Experiment
- Video Foreman, QCIF, 130 frames
- Compression H.264
- 3-layer temporal scalability 72144 kbps Liang,
2002 - Backbone
- Packet loss rate 10
- Delay shifted G-distribution
- Last hop
- Packet loss rate 1
- Delay shifted G-distribution
19Streaming with Diversity
Server Diversity
Packet Path Diversity
Channel 1
Channel 2
Client
Channel N
20R-D Optimized Streaming over 2 Channels
- Video Foreman, QCIF, 130 frames
- Compression H.263
- 2-layer SNR scalability 32/64 kbps
- 2 identical, independent 2-state Markov channels
- Good/bad packet loss rates 3/15
- G-distributed delays short/long
Chakareski, Girod, DCC 2003
21R-D Optimized Streaming with Accelerated
Retroactive Decoding (ARD)
PSNR, in dB
Latency 100 ms RTT 100 ms
Bit-Rate, in kbps
22R-D Optimized Streaming with Accelerated
Retroactive Decoding (ARD)Latency 100ms
R-D Optimized Streaming with Accelerated
Retroactive Decoding (ARD)
Multiple Deadlines Rate 68.8 kbps Mean PSNR
27.0 dB
Single Deadline Rate 89.0 kbps Mean PSNR 23.9 dB
23Network-adaptive Packet Dependency Management
- Very low latency no retransmissions
- Highly robust compressed representation by
network-adaptive management of packet
dependencies - Utilize ACK/NACK in source coder
- H.263 RPS, MPEG-4 NEWPRED, H.264 multiframe
prediction
Transmission error
NACK
Time
24Error Resilience vs. Coding Efficiency
P5
I
230 frames of Foreman coded using H.26L TML8.5.
Average PSNR33.4dB
25Rate-Distortion Optimal Reference Picture
Selection
Wiegand, Färber, Girod, 2000 Liang, Girod,
2002
26RD Performance of Optimal Reference Picture
Selection
- LTM buffer V5 frames
- Feedback round-trip time dfb7 frames
- Packet loss rate p10
- Comparison with P-I scheme, where each NACK
triggers insertion of I-frame
Liang, Girod, 2002
27ORPS Performance over Time Axis
28MaD Sequence at 10 packet lossNo Retransmissions
Optimal Reference Picture Selection Rate 320
kbps Mean PSNR 39.2 dB
Adaptive I-Frame Insertion Rate 320 kbps Mean
PSNR 38.2 dB
29Conclusions
- Network-adaptive video streaming jointly
optimize compression, error control, packet
transport, and decoding - Adaptive media playout real-time more
flexible than we thought - RaDiO streaming can provide virtual priority
mechanisms - Proxy servers with RaDiO can improve streaming
performance - Path Diversity can improve streaming performance
- RaDiO streaming with multiple deadlines for
accelerated retroactive decodingof late
retransmissions (catch-up decoding) - Packet dependency management allows robust
representations for streaming w/o retransmissions