High Definition Video: Workflow from Cameras to CODECs

1
High Definition Video Workflow from Cameras to
CODECs APAN Fall Meeting Xian, China Aug
2007 Michael Wellings Director, Engineering Vice
Chair, APAN HD WG
THINK FORWARD. THINK RESEARCHCHANNEL.
2
Topics of Discussion

• ResearchChannel Background
• Recent and future demonstrations
• Advanced Systems
• iHDTV Open Source Project
• HD Video Workflow Discussion

3
ResearchChannel Background
4
ResearchChannel Programs

• More than 3,200 hours of programming in theVideo
  on Demand library
• 1,100 new programs addedthis year

5
ResearchChannel Participants
Advanced Network Forum AJA Video Systems
Inc. Australia's Academic Research Network
(AARNet) Fujinon Howard Hughes Medical
Institute IBM Corporation Intel
Corporation Internet2 Johnson Johnson Library
of Congress Microsoft Research National
Academies National Academy of Engineering

• Duke University
• George Mason University
• Indiana University, School of Informatics
• Johns Hopkins University
• Massachusetts Institute of Technology
• National University of Singapore
• New York University
• Pennsylvania State University
• Rice University
• Rutgers, The State University of New Jersey
• Stanford University Medical Center
• Texas AM University
• Tulane University, A.B. Freeman School of Business

Universidad de Puerto Rico Universidade de São
Paulo University of Alaska - Fairbanks University
of Chicago University of Hawaii University of
Maryland - College Park University of
Pennsylvania University of Southern
California University of Virginia University of
Washington University of Wisconsin-Madison Virgini
a Tech Yale University
National Academy of Sciences National Institute
of Nursing Research National Institute of
Standards and Technology National Institutes of
Health National Library of Medicine National
Science Foundation Pacific Northwest
Gigapop Poznan Supercomputing and Networking
Center R1edu.org SURFnet Vulcan Northwest,
Inc. Wisconsin Public Television
6
ResearchChannel New Participants

• American Meteorological Association
• Arizona State University
• California State University,Monterey Bay
• Indiana University
• Loma Linda University
• Pacific Northwest National Laboratory
• University of Kentucky
• University of Michigan
• Isilon Systems

7
U.S. Television Distribution
DISH Network satellite system 12 million
homes Cable systems 12.9 million homes
8
Google Video
9
Technology Milestones

• August 1999First-Ever Streaming of
  High-Definition Television over the Internet
• November 1999Network Sets Record Speed
• April 2000First Live HDTV Over Internet Newscast
• July 2004First Transmission of Full Bandwidth HD
  Video Between Computers
• September 2005First-Ever Live HD Images from
  Seafloor to Land Available as IP-Based Feed
• November 2005Interactive Multipoint HD
  Videoconference Demonstrated
• November 2006Partner in World-wide testing of
  ConferenceXP

10
Technology Leading the Way
11
Continuous Ocean Observing
12
Consortia Partnerships
13

• Recent and Future Demonstrations

14
Recent Demonstrations

• Sept 2005 - iGrid San Diego
• iHDTV - USA118 Global N-Way Uncompressed
  Interactive Conferencing
• Underwater research using HD from SS Thompson
  USA119 20,000 Terabits Under the Sea
• Nov 2005 - SC05 Seattle
• iHDTV /HD Storage, capture and editing using SRB
  (Storage Resource Broker)

15
Recent Demonstrations

• Nov 2006 SC06 Tampa
• iHD1500 system
• Dec 2006 Internet2 Fall Member Meeting Chicago
• iHD1500 system
• HD CoDecs compared
• HD Camera hands-on

16
Future Demonstrations

• ResearchChannel Annual Meeting Oct07
• Cinegrid / GLIF Sept07
• Internet 2 Fall Member Meeting Oct07
• SC07 Reno Nov07
• APAN 08 Hawaii Jan08

17

• Advanced Delivery Methods and Services

18
ResearchChannel Advanced on-line Services

• ResearchChannel-HD
• Currently VC-1 SMPTE 421M
• AVC (h.264, MPEG-4 Part-10) coming soon
• Unicast
• http//media-wm-hd.cac.washington.edu/ResearchChan
  nel-HDVideo
• Multicast
• http//researchchannel.org/multicast/ResearchChann
  el-HDVideo.nsc
• Solicitation for HD content

19
ResearchChannel Advanced on-line Services

• ResearchChannel satellite feed on the network
• 233.0.73.29 MPEG2_at_ 3.2mbps
• Live 1080i HD video of Mt Rainier
• 233.0.73.26 MPEG2 MP_at_HL _at_ 20mbps
• Live 720p HD video of Mt Rainier
• http//www.researchchannel.org/tech/desktopclient
  s.asp
• HD VOD 720p WMV-HD http//www.researchchannel.org
  /visions05/hdpresentation.asp
• HD VOD 1080i MPEG2 ML_at_HL
• http//www.researchchannel.org/tech/desktophdsamp
  les.asp
• KEXP-FM Audio-on-demand uncompressed audio
  webcasting
• http//www.kexp.org

20
ResearchChannel Advanced on-line Services

• Developing VOD archive for oceanographic HD video
• Partnership with Isilon to be demonstrated at
  SC07 Reno
• KEXP-FM on-line services have a world-wide
  audience that equals Seattle broadcast listener
  numbers

21
ResearchChannel Advanced on-line Services

• Visions 05 - Emmy Award Nominee
• First-ever live HD 1080i video/IP streamed around
  the world
• SMPTE 292M multicast and unicast _at_ 1.5gbps
• MPEG 2 MP_at_HL multicast
• WMV-HD multicast and unicast

22
ResearchChannel Advanced on-line Services

• Seattle Science Foundation Project testbed for
  new
• Medical technology with HD video archiving
• seattlesciencefoundation.org
• WWAMI collaborative medical education via
• Advanced video technology
• RC HD Lab on-line
• Advanced venue on UW campus commissioned
• Successful launch of Experimental Internet
  ResearchChannel-HD

23

• iHDTV Open Source Project

24
iHDTV Software Suite

• iHDTV Explained
• iHD1500
• iHD270
• HD to the Desktop
• WindowsXP-based application suite
• Released as Open Source
• Account created on SourceForge
• iHD_Trusted_Partners_Group formed to direct
  software development

25
iHDTV Development / Deployment

• SourceForge load
• SVN Tree transferred
• iHDTV_trusted_partners formed 47 members
• 4 Core developers _at_ (UW / U-Mich / U-Wisconsin)
• Executive team to guide development will meet
  monthly
• Deploy globally
• N-way implementation this summer with 6 nodes

26
iHDTV iHD270

• iHD270 (first introduced aug 99)
• Sony HDCamtm compression
• SDTI data format 270mbps
• AJA Xena I/O
• Requires
• AJA Xena-HD capture cards
• Dual Processor P4
• Windows XP
• Sony HDCam Hardware encoder/decoder
• Gige network connection

27
iHDTV iHD270

• 1920x1080 60i 8bit 422
• Sony HDCam source material
• 995mbps active lines, 1188mbps total

28
iHDTV iHD1500

• iHD1500
• Uncompressed SMPTE 292M 422
• Data rate total approx 1.5 gbps
• Requires
• Uncompressed HD capture cards
• PCI Express platforms
• Windows XP
• 2 x gige or 1x10gige network connection
• Jumbo frame transport and routing
• Specs http//zzz.cac.washington.edu/ihdtv_wiki/in
  dex.php/Main_Page

29
iHDTV iHD1500

• Latency about 4 frames end to end plus network
  delay
• 4 frames133ms or 1/7 sec

30
iHDTV iHD1500

• iHD1500 Enhancements
• New i/o board options
• GUI front end
• Self-installing package
• Uncompressed HD Server
• Single link
• Dual link
• Quad link
• HD Screen rendering for OptIPortal integration
• NTT i-VISTO interoperability

31
High Definition Workflow

• A Report in progress

32
HD a Background
33
Sample Rates

• NTSC 422
• Y, Pb, Pr color matrix
• 4x3.38 (compromise rate from 3.58MHz) 15.5MHz
• Color difference channels _at_ 6.75MHz
• HD 422 (really 221111)
• Y, Pb, Pr color matrix
• 22x3.38 74.25MHz
• Color difference channels _at_ 37.125MHz
• 10 bit 1024 gradations
• 8 bit 256 gradations

34
Sample Rates

• 422
• Chroma sampled at ½ rate of Luma
• 411
• Chroma sampled at ¼ rate of Luma
• 420
• Chroma sampled at ½ rate of Luma on every other
  line of each field

35
Sample / Bit Rates

• Broadcast
• 420 MPEG-2 TS 19.2mbps
• DVD
• 420 MPEG-2 PS VBR
• HD DVD/Blu-Ray
• 420 VC-1 _at_ 8mbps
• 420 AVC _at_ 8mbps

36
HD Resolutions

• 720p 60Hz SMPTE 274M
• 1280x720 24p
• 1280x720 30p
• 1280x720 60p
• 1080i/p 60Hz SMPTE 296M
• 1920x1080 24p
• 1920x1080 30p
• 1920x1080 60i
• 1920x1080 60p

37
HD Bitrates Uncompressed 720p 10- bit 422

• 720p 60Hz - SMPTE 296M
• 1280x720 30p
• 742mbps
• 30MHz Y, 15MHz Chroma
• 1280x720 60p
• 1485mbps
• 30MHz Y, 15MHz Chroma

38
Progressive / Interlaced

• 1080/60i and 1080/30p occupy the same datarate
  1.485gbps
• Interlaced pictures produce spacial errors
  associated with the time difference in motion
  between fields
• This can create a smoothing effect which is
  easier on the eye during fast motion
• High Framerate progressive (60 frame or greater)
  is best

39
HD Bitrates Uncompressed 10801/p 10- bit 422

• 1080i/p 60Hz SMPTE 274M
• 1920x1080 30p
• 1485mbps
• 30MHz Y, 15MHz Chroma
• 1920x1080 60i
• 1485mbps
• 30MHz Y, 15MHz Chroma
• 1920x1080 60p
• 2970mbps
• 60MHz Y, 30MHz Chroma

40
Imaging chips

• Solid State Camera front end (tube cameras came
  first!)
• CCD Charge Coupled Device
• CMOS Complementary metal oxide semiconductor

41
Imaging chips

• CCDs collect charge at each element, and that
  charge is read out sequentially
• Charge to Voltage conversion takes place off-chip
• AgtD takes place off chip
• Digital level luminance level
• These are monochrome sensors!

42
Imaging chips

• CMOS chips collect charge at each element
• Charge to Voltage and AgtD functions take place
  on-chip
• CMOS imagers exhibit more noise than CCDs, and
  tend not to be used in higher end cameras

43
Single chip vs 3 chip

• CCDs gather charge according to the amount of
  light impinging on the element
• This charge is converted to a voltage
  representing light level
• This light level does not specifically contain
  color information
• Color reference is obtained through the use of
  filtering in front of the image sensor

44
Bayer Filter on-chip

• Bayer Filter Diagram
• 422 sampling
• Green sampled at 2x Red and Blue
• Single chip color solution

45
Color Processing

• 3 chip
• RGB output from image processing
• Digital sampling and CODECS operate on component
  video Y, Pb, Pr (Y,Cb,Cr)
• Y Luma channel
• Pb subsampled blue matrix
• Pr Subsampled red matrix

46

• NAB 2007

47
HD Cameras

• SONY XDCam ex
• 3 chip ½ imager (details unknown)
• MPEG-2 long GOP
• 30-50mbps
• Writes to Express Flash memory cards
• 120 min record time
• Target price 8000

48
HD Cameras

• Grass Valley/Thomson Infinity
• 3 chip CMOS
• 2.4 million pixels each
• Ships with DV25 and JPEG2000 codecs
• MPEG2 optional
• Writes at up to 100mbps to IOMega REVPro drives
• Target price 26K without lens
• Ships in August 07

49
New HD Cameras at NAB 07

• SONY XDCam HD
• Long GOP MPEG-2
• Writes to dual-layer BluRay Pro disk
• Up to 50mbps
• 2/3 3 chip 2million pixel
• ships in 1st QTR 08
• Target price less than 30K without lens

50
New HD Cameras at NAB 07
51
Codec comparison Project

• SMPTE StEM HD Mini-Movie
• Designed for critical evaluation of HD equipment
  as well as encoders and decoders
• Delivered as tga files, uncompressed
• Imported into Final Cut Pro

52
SMPTE Standard Evaluation Material
53
Codec comparison Project

• 7 Uncompressed Quicktime Movies on 7 sequences

54
Codec comparison Project

• 5 movies exported
• DVCPro 100
• HDV
• MPEG-4 Basic
• H.264 Part 10 (AVC)
• HD uncompressed
• All FCP I/O via AJA Kona 2 board

55
Codec comparison Project

• HD sequence played out to HDCam Tape
• recaptured 8bit 422 in FCP
• HD sequenced played out to NTT HD1000 MPEG-2
  encoder
• recaptured from output of NTT HD1000 MPEG-2
  decoder as 8bit 422, and 420

56
Codec comparison Project

• DVCPro 100 sequence recorded to tape and played
  back
• HDV sequence re-rendered in FCP to uncompressed
• HDCAM SR recorded to tape and played back
• XDCAM HD recorded to tape and played back

57
Codec comparison Project

• MPEG-4 Basic re-rendered to uncompressed in FCP
• MPEG-4 h.264 Part 10 re-rendered to uncompressed
  in FCP

58
The edit

• Five scenes of interest selected
• 10 new sequences created for each of the 9
  compressed-decompressed files plus the original
• HDV and DVCPro were scaled 133 and 150
  respectively to match the uncompressed material

59
Difference mask

• Luminance of each compressed file inverted and
  added to the original
• 50 luma field indicated no loss
• Material lost in codec process exhibited visually
  due to the subtraction process

60
The movies

• Several HD movies resulted from this project
• 9 with the edited 5 scenes each an example of a
  particular codec showing the difference mask
  full screen
• 9 with the encoded and decoded video overlayed in
  the center of the frame of the origional for
  direct comparison
• 5 complete sequences of codec representation
• The original uncompressed movie

61
(No Transcript)
62
(No Transcript)
63
CODEC Comparison
CODEC Bit Rate (Mbps) File Size (MB) Compression Ratio Type Size Color Space Encode Time
HDCam 144 n/a 10.41 Hardware 1440X1080 311 630 min
HDCam SR 440   1500/B3 Hardware 1920X1080 444 630 min
DVCProHD 115.24 5100 131 Software 1280x1080 422 17 min
XDCam HD 35     Hardware     630 min
MPEG-2 422_at_HL 30 n/a 561 Hardware 1920X1080 422 630 min
MPEG-2 MP_at_HL 420 18 n/a 931 Hardware 1920X1080 420 630 min
HDV MPEG-2 MP_at_H14 24.96 1110 601 Software 1440X1080 311 33 min
MPEG-4 H.264 7.572 344 1981 Software 1920X1080 420 1 hr 5 min
MPEG-4 Basic 5.937 270 2531 Software 1920X1080 420 24 min
VC-1 8 320 1881 Hardware 1920X1080 420 28 min
JPEG2000 46/25     Hardware      
JPEG2000 100/75     Hardware      
JPEG2000 220/190     Hardware      
64
equipment

• SONY HDW-500 VTR
• SONY HDCAM-SR VTR
• SONY XDCAM HD VTR
• Panasonic DVCPRO-100
• Apple Dual 2.7GHz G5
• 4GB RAM
• Mac OS-X 10.4.7
• Final Cut Pro 5.1.1
• AJA Kona 2 I/O
• NTT HD 1000 MPEG Encoder
• NTT HD 1000 MPEG Decoder

65
HD Multiburst Waveform Uncompressed Reference
66
HD Multiburst Waveform DVCPro 100
67
HD Multiburst Waveform SONY HDCam
68
HD Multiburst Waveform SONY HDCAM SR
69
HD Multiburst Waveform HDV
70
HD Multiburst Waveform H.264 /AVC
71
HD Multiburst Waveform SONY XDCAM HD 30
72
HD Multiburst Waveform JPEG 2K 25mbps
73
HD Multiburst Waveform JPEG 2K 75mbps
74
HD Multiburst Waveform JPEG 2K 250mbps
75
Seating Dist
76
Screens

• Small screen up close vs. large screen further
  away
• Dot pitch most important factor
• Native HD resolution displays make sense for
  larger (24 to 50 inch) sizes due to optimum
  viewing distances
• Low-res screens appear sharper from a distance
  this is due to the apparent dot pitch of the
  digital display

77
High Definition Workflow

• Apple FCP EDIT-SAN Architecure

78
Edit System Software

• Apple Final Cut Pro / Final Cut Studio 2.0
• OSX 10.4
• XSAN 1.4

79
Edit Suite Hardware Systems

• 4 MacPro Dual Quad Core Xeon platforms
• AJA Kona 3 i/o
• 3 XSERV
• 30TB XRAID

80
Shared storage

• 27.3TB shared media pool
• Incremental backup of projects nightly
• IBM Tivoli Storage Manager
• StorageTek Silo archive
• Source tape is backup for media pool currently
• Multi-resolution capable

81
HD Edit workflow
HDW-500A
SMPTE 292M
PESA 292M HD Router
XSAN
XRAID Stack
Final Cut Studio edit suites
82
X-SAN Diagram
83
ResearchChannel HD Migration

• SD cameras to be replaced with HD cameras
• Likely Sony XDCam HD
• MPEG-2 422 30-50mbps IMX
• High speed file dump mode from Pro Blu-Ray dual
  layer disc
• 43 safe zone shooting
• HD capture and edit as uncompressed HD or IMX
• Can output anamorphic, 43, 169 letterbox for SD
• HD project archive

84
Edit Room Layout
85
Edit Room Layout
86
XSERVs in Master Control
87
XRAID
88
XSAN
89
Studio M
90
Credits

• NAB 2007 reports
• Noah Pitzer
• Jamie Alls
• Jabran Soubeih
• Dave Robertson
• Jack Hoffman
• Special thanks to
• Staff engineer Jabran Soubeih for CoDec
  measurements
• Tektronix Guide to Standard and High Definition
  Digital Video Measurements
• http//www.tek.com/Measurement/App_Notes/indexes/v
  ideo_audio/registration/

91
For more information

• www.researchchannel.org
• wellings_at_washington.edu