From Videotapebased to Filebased Archives BBC D3 preservation casestudy

1
From Videotape-based to File-based Archives- BBC
D-3 preservation case-study

• James Insell
• DCC/LUCAS - 01/12/2006

2
Contents

• Overview
• Pre-Ingest - Content preparation
• Ingest
• File-Formats
• Post-Ingest - File Construction
• Post-Ingest - File Storage
• (Preserving Quality)

3
Overview

• Preservation transfers are mounted to preserve
  content held on obsolete formats and decaying
  tape.
• Benefits of creating a file-based archive can be
  gained during the transfer process.
• Case-study a work in progress. (General
  considerations followed by case-study example at
  each stage).
• Products used in case-study are named for
  illustrative purposes only as examples of
  available technology. Not to be taken as BBC
  recommendation.

4
D-3 transfer (single channel)
LTC analog timecode
SDI
SDI
D-3 VTR
LTC -gtVITC
SDI Router(16X16)
IngestPC
PAL Transform Decoder
Embedder Delay
SDI
LTODrive
DigiBeta VTR
AES digital audio
SDI
SDI
De-embedder
LeftSpeaker
SDIMonitor
Confidence monitoring(Can see either DigiBetaor
Ingest PC output)
SDI
RightSpeaker
SDI to from other ingest channels
Bar code scanner
Channel controlPC
D3 error log, barcode and database control, vtr
remote control
5
Pre-Ingest - Content Preparation

• Content needs to be assembled in a form that can
  be readily and robustly ingested.
• Bear in mind issues of preservation and quality
  (dont throw away what you cant then retrieve).
• An SDI signal is assembled to contain all
  elements that require capture. SDI provides a
  suitable interface signal at the point of ingest.
• Its alright leaving me.

6
Pre-Ingest - Videotape replay

• Variety of videotape formats
• some analogue, some digital.
• Many obsolete or near obsolete formats.
• Manufacturers dictate obsolescence through
  availability of spares and service.
• Newer formats fall obsolete faster than older
  formats.
• Importance of obtaining a good replay. Usually no
  2nd chances
• Real-time transfer. (Costly and time-consuming).

7
Archive Videotape Format Life
8
D-3 Project- D-3 videotape replay

• Archive source format
• D-3 digital composite (4fsc PAL)
• 4 x 20-bit 48kHz digital audio.
• D-3 Introduced - 1991.
• Ceased manufacture - 1998. Support ceased - June
  2006.
• Used as the 43 delivery format in BBC until Nov
  2005.
• 341,000 tapes of which 100,000 selected to be
  transferred.

9
Pre-Ingest - PAL Decoding

• PAL content requires decoding before use.
• More convenient to ingest component video than
  PAL video.
• Quality control retained if PAL decoded at
  source.
• Possibility of PAL signal reconstruction if
  complementary decoding used.

10
D-3 Project- PAL Decoding

• BBC designed Transform PAL decoder
• Best currently (or every likely to be) available.
• Complementary decoder.
• Digital In Out.
• Not commercially
• available sorry!

11
Pre-Ingest - Audio Embedding

• Synchronously embed audio into SDI signal
• - providing a common interface.
• A to D converter required for analogue sources.
• Avoidance of possible A/V sync issues
• audio coupled to video.
• Provides test point at point of ingest.
• Simplifies cabling and number of PC cards for
  ingest.

12
D-3 Project- Audio embedding

• 4 x 48kHz digital audio channels synchronously
  embedded into SDI as two AES pairs.
• CrystalVision Tandem-110 embedder/de-embedder
  used.
• 399ms audio delay incorporated compensating for
  PAL decode delay.

13
Pre-Ingest - Timecode

• Timecode serves to index video content.
• Preserved for integrity against existing notes.
• Incoming LTC Timecode embedded into SDI signal as
  VITC to provide a common interface.
• Avoidance of A/V / timecode mistiming
• timecode coupled to audio and video.
• Provides test point at point of ingest.
• Simplifies cabling and number of PC cards for
  ingest.

14
D-3 Project- Timecode

• D-3 has LTC any VITC in video.
• Original D-3 VITC passes cleanly through to point
  of ingest
• lines 19 23.
• Analogue LTC converted to VITC
• placed on lines 15 17.
• Alpermann-Velte Rubidium DT unit used for LTC
  VITC conversion and re-timing adjustment.

15
D-3 Project- SDI signal at point of Ingest

• SDI active picture area contains 10-bit digitally
  PAL decoded video.
• Line 23 contains an extended chroma burst as
  an aid to PAL reconstruction.
• 4-channel audio contained in 2 embedded audio
  groups.
• Original VITC timecode in VBI (19 21).
• Original LTC timecode as VITC in VBI (15 17).

16
D-3 Project- DVS Centaurus OEM card

• Professional SDI capture / playback card
• DVS Software Development Kit used to create
  bespoke recording software.

17
D-3 Project- Ingest PC

• Dell PowerEdge 2950 rackmount
• Dual core Xeon 3.7 GHz CPU
• High performance required.

18
D-3 Project- Ingest PC

• Temporary storage -
• SAS (Serial Attached SCSI) case
• 15 x 250GB SATA II hard drives.
• RAID 5 configuration (hardware controlled)
• 2TB total temporary storage. Equates to around
  80 hours of uncompressed material.
• High performance required.

19
D-3 Project- Software

• Generic software not up to the task
  (multi-application, inflexible, non-extensible).
• Suitable product unavailable for a factory
  environment.
• Bespoke software written in C for Ingest, MXF
  wrapping/unwrapping, Data Tape control.
• SuSe Linux 10.1 Operating System.
• Software is to be released as Open Source,
  enabling other archives to make use of the
  technology.
• Workflow control (database, bar code reading, VTR
  control etc.) under the control of a Mediaflex
  Workstation.
• Mediaflex is a software product from TransMedia
  Dynamics Ltd.
• CORBA is used to connect the two parts of the
  system.
• Other control software could be connected using
  the IDL.

20
10-bit to 8-bit Error Feedback

• PAL decoder output likely to mainly consist of
  quantisation noise in the LSBs.
• The ingested,10-bit quantised, video data is
  processed to 8-bit form.
• No visible difference to pictures.
• Saves on wasted data storage space.
• 8-bit video format easier to handle.

21
D-3 Project- Photo-Sensitive Epilepsy

• Ingested video analysed and tested for compliance
  against Ofcom PSE recommendations.
• Types of failure
• Luminance flash
• Red flash
• Spatial pattern
• Extended failure
• Software module supplied by Cambridge Research
  Systems Ltd.
• Non-compliant sections marked-up within MXF
  file.

22
D-3 Project- Descriptive metadata

• The bar code label on D-3 tape calls up the
  relevant INFAX record.
• This information is passed from Mediaflex to the
  Recorder software.
• INFAX data is wrapped, together with the other
  content, in MXF.
• In addition, Mediaflex passes the D-3 VTR
  frame-based error log for inclusion in the MXF
  file.

23
File Formats

• Can describe a variety of video and audio essence
  file formats or wrapper formats.
• Closely related to output format of codecs.
• Careful choice of codec (form of compression)
  required in order to preserve quality.

24
D-3 Project- File Format

• MXF Material Exchange Format
• Standardised File Interchange Format SMPTE 377M
• Increasingly wide adoption in production
  environments.
• Wraps the items of content (essence) along with
  metadata, in a standardised form.
• OP1a Operational Pattern
• SMPTE 384M mapping of Uncompressed video data
  into MXF.
• SMPTE 382M mapping of Broadcast Wave Audio into
  MXF Generic Container.
• BBC Archive v1.0 flavoured MXF scheme is
  registered within the BBC node of the SMPTE
  repository as D3P_D3PreservationDescriptiveScheme
  .

25
Browse Quality File Format

• Low-quality copy enables researchers to search
  and find content via networks.
• Potentially sections could be marked-up and full
  quality versions ordered up.
• Steer clear of proprietary formats and licenced
  formats where possible.
• Opt for Open Standards based, royalty free
  codecs.
• Opt for an interchangeable (easily adopted)
  format.
• No reason why browse quality material cannot be
  MXF wrapped opportunity for including
  descriptive metadata.

26
D-3 Project- Browse Quality File Format

• The BBC developed Dirac was to be used in D-3
  project, but is not quite ready for use.
• H.264 chosen as a fall-back position.
• CIF 1.2Mbps
• Streamed to a Network Attached Server.

27
Post-Ingest - Storage Format

• What physical forms of storage are available for
  the files generated?
• IT world provides solutions for data storage.
• Market driven (Broadcast market only a
  sub-section).
• Data Tape - cost-effective proven technology
  within IT world.
• Servers not yet viable for full-quality material
  (cost, size, performance).
• Future storage formats 3D crystalline?

28
D-3 Project- Storage format

• LTO-3 chosen
• 77 data tape market share.
• Shelf life 30 years.
• What about obsolescence?
• Built-in protection within LTO road-map
• LTO drive can read data tapes from at least the
  two previous generations of drive.

29
Impact of video/audio compression

• Uncompressed
• problem solved, but can you afford it?
• Lossless compression
• halve your costs.
• Lossy compression
• be greedy, save money, but wary of
  consequences.

30
Impact of compression on Picture Quality

• Uncompressed
• No degradation
• SD (166 to 270Mbps), HD (829 to 1200Mbps)
• Lossless compression
• No degradation
• SD (83Mbps), HD (414Mbps)
• Lossy compression
• Degradation varies from no perceptible
  change to unwatchable
• SD (lt83Mbps), HD (lt414Mbps)

31
Impact of lossy compression on Picture Quality

• Lossy compression in widespread use in
  Production.
• Quality vs Cost
• How low do you go?!
• Digibeta 1st generation no perceptible change
• DV 1st generation degradation already visible
  
• DVD Acceptable for inserts?
• One size doesnt fit all Image content
  dependency
• Concatenation issues cascade effects

32
Coding scheme comparisons
33
Coding scheme comparisons
34
D-3 Project Choice of compression scheme

• Preservation issues dictate that little or,
  preferably, no impact on picture quality.
• D-3 project is to use a simple uncompressed
  format for video (and audio). This has a
  standardised and open mapping into MXF.
• Predicted savings on storage (LTO-3) against
  current equivalent videotape format.