Preserving Digital Public Television http://www.ptvdigitalarchive.org/

1
Preserving DigitalPublic Televisionhttp//www.pt
vdigitalarchive.org/

• Howard Besser
• NYU Moving Image Archiving Preservation Program
• http//besser.tsoa.nyu.edu/howard
• http//www.nyu.edu/tisch/preservation/

2
NYU/Public Television Project

• 6 million project -- 50 from LC/NDIIPP
• Marry asset management to preservation
• Preserve a broad set of elements (including
  ancillary material)
• Life-cycle mgmt (add metadata as soon as a clip
  comes in)
• Establish a community of stakeholders, working
  together for preservation (stations, university,
  librarians, journalists, historians, producers,
  scholars, )
• Build an OAIS Server
• Explore appropriate file formats, wrappers, METS
  extensions
• Develop sustainable business model

3
Project Partners

• Thirteen/WNET WGBH Content and production
  expertise
• The two largest television stations in the PBS
  system
• Together produce largest percentage of national
  programs
• Both have preservation Archives
• Public Broadcasting Service More content and
  network design
• Distributes most of the national programming
• Determines and keeps broadcast versions
• New York University Facilitation and Resources
• Leadership in designing digital libraries
• Experience in process for setting standards
• Has new Masters Program in Moving Image Archives

4
We were already collaborating to solve shared
technical problems

• The public television partners have been working
  together for a long time on common issues -- such
  as digital asset management, and a metadata
  dictionary.
• Digitally-produced programs are at great risk of
  being lost --- because of the rapid changes in
  technology, lack of funds, and no preservation
  mandate.
• Expanding our efforts to encompass preservation
  was a natural extension of our progress in
  standardizing a complex digital production and
  broadcast environment.

5
Public Television was analog for a long time
6
PBS Remote Tape Storage Facility
7
Computer-printed index of1955-69 WNET Holdings
at LC
8
Public Television has becomeDigital for
Production Distribution

• though some things still need to be streamlined

9
Programs are shot, assembled and edited
digitally. Completed programs are packaged as
digital files, often with many different
elements.

• Typical AVID
• Editing Suite
• The same huge video
• files are digitized over
• and over again for
• different uses.
• A lot of video is not
• used for broadcast, but
• goes to the internet, to
• DVDs and to other
• media.

10
Local broadcasting is nearly all via digital
systems Thirteen Master Control manages one
high definition, two analog and three digital
over-the-air broadcast channels.
11
Broadcast playback is from a digital server- no
more tapes or tape machines

• All Thirteen broadcast programs are played from
  this server.
• It has to talk to the broadcast automation
  system, the satellite system, and other in-house
  and external networks.
• It is the size of a window air conditioner and
  holds 1700 hours of material.

(
12
Prior WorkPBCore

• 2 year collaborative effort
• 48 elements based on Dublin Core
• Intellectual content of a media asset or resource
  -- 13 elements
• IP-creation, creators, and usage limitations of a
  media asset or resource -- 7 elements
• Instantiation (in either digital and/or analog)
  -- 28 elements
• http//www.utah.edu/cpbmetadata/

13
Challenges of Digitalbrought the partners
together

• Formats Versions issues (including variant
  forms of content, HD/SD quality, )
• Agreements on technical standards for handing off
  files from one organization to another
• Repository design for long-term sustainability

14
Project activities include

• Completing an inventory of at-risk materials to
  better quantify our holdings and prepare for
  selection
• Reviewing best practices and most up-to-date
  developments in the field of video archiving
• Conducting facilitated discussions on key topics
  to guide setting standards and policies
• Establishing an Advisory Committee to assist with
  selection criteria
• Ingesting sample materials and testing the
  repository
• Presenting regular reports to public broadcasting
  and moving image archive community for ongoing
  feedback

15
Project Focus Areas

• Appraisal and Selection developing criteria and
  standards for what to preserve and by whom
• File Formats and Packages determining the best
  formats for our various uses, plus testing the
  suitability of file packaging for long term
  preservation
• Metadata and Related Topics specifying
  technical, descriptive and rights information
• Repository Design technical architecture,
  administrative policies and potential business
  models
• Sharing Our Findings Keeping the public
  broadcasting community involved and informed all
  along the way

16
Collaboration has been very productive

• Now have standards that allow files to flow
  digitally from producer to PBS to stations
• Studied various metadata schemes to zero in on
  whats needed for preservation
• Inventoried and located programs in non-regular
  locations
• Wrapper Roundtable aftermath
• PBS LC beginning to collaborate
• The American Archive

17
Our Talks today

• Repository Design (including Dspace and
  SRB)--James Bullen
• Pushing metadata-gathering upstream, gathering it
  during the program production process--Kara Van
  Malssen

18
Preserving DigitalPublic Televisionhttp//www.pt
vdigitalarchive.org/
19
Pushing Metadata Upstream into the Content
Production Process Preliminary Studies of Public
Television

• Howard Besser and Kara Van Malssen
• New York University

20
The Problem

• TRADITIONALLY
• Very little metadata required for preservation
  accompanies an object to a repository.
• Archives, libraries and other repositories must
  create (or re-create) most of the necessary
  metadata.
• This requires many manual hours, and significant
  resources - both time and money.
• IN THE DIGITAL WORLD
• Repositories will be unable to continue in this
  manner, as more metadata than ever is required.

21

• Once the (television) program is finished, it
  is passed on to the archive or library for safe
  keeping. Librarians will catalog and classify
  the content, possibly using a proxy copy, and
  enter the resulting informative metadata in their
  database so they can retrieve it in the future.
  However, rarely if ever is the metadata from the
  rest of the process passed onto them, except,
  perhaps, for the title, tape number, and basic
  technical information about recording formats.
  It has to be re-created, with all the associated
  risk of errors and lack of accuracy--not to
  mention the work and time involved.
• - Cox, Tadic, and Mulder, Descriptive Metadata
  for Television (2006)

22

• The necessary or additional metadata cannot be
  effectively and satisfactorily produced either as
  an afterthought post-production process on the
  publishers side or as a pre-ingest conversion
  activity at the archives end. Approaching
  e-archiving in this fashion leads to distribution
  delays and a more complex production and
  distribution scenario, with all the accompanying
  potential to introduce production delays and
  errors.
• - Yale University, YEA The Yale University
  Archive, One Year of Progress, 2002

23
Solutions?

• Preservation becoming a shared responsibility
  between content creators, distributors, curators,
  and preservationists.
• Partnerships are needed to come to unified
  solutions.
• Preservationists seek reliable metadata back
  upstream in the production workflow...

24
Site Visits

• This report is based on workflow studies at
  public television stations, between June and
  August, 2006 by NDIIPP Research Assistants
  Caroline Rubens, Paula Felix-Dider, and Kara Van
  Malssen. Workflow report completed in September
  2006.
• Additional insight was gained through metadata
  studies conducted by Mary Ide and Leah Weisse,
  Archivists at WGBH.

25
Site Visits

• WGBH, Boston - June 19-20, 2006
• Interviewees included Archive and Media Library
  staff, members of Frontline, NOVA, and American
  Experience production units, and legal dept staff

26
Site Visits

• WNET/Thirteen, New York - July 18 and August 2,
  2006
• Interviewees included the Archivist, members of
  Broadway The American Musical production unit,
  Broadcast Operations, and Broadcast Technology
  staff.
• WNET/Thirteen, Washington DC - August 15, 2006
• Religion and Ethics production unit staff
  interviewed.

27
Public Television WorkflowBasically similar to
workflows in other fields
28
Recent Workflow

• Hybrid digital/analog process. Production and
  post-production file-based. Distribution and
  archiving tape-based.
• Programs and production materials are turned over
  to the archive after the production unit no
  longer has a need for them, or a place to store
  them.
• Metadata gathering starts at this point, or
  sometime even further in the future.

29
Changes

• Public television (PTV) moving toward file-based
  workflow.
• Distribution from PBS will soon be by the Next
  Generation Interconnction System (NGIS) real
  and non-real time distribution of program files.
• Largest stations (WNET and WGBH) implementing
  digital asset management (DAM) systems.
• PTV working with the broadcast industry on the
  development of file wrappers for post-production,
  distribution, and preservation.

30
For digital preservation, this shouldnt be the
only place for metadata in the preservation
workflow! This is far too late in the cycle!

31
METADATA
METADATA
It also needs to be here!
32

• As the workflow becomes file-based, the need
  for robust and accurate metadata will become
  critical. File relationships, video codecs, bit
  rates, and rights information must be explicit,
  accurate, and immediately accessible. This will
  require a much deeper level of metadata than is
  currently captured in tape-based archives.

Obtaining the necessary metadata at the end of
production and broadcast life cycle is not
feasible. Metadata will need to be
systematically gathered during the production
lifecycle and submitted with the programs to the
preservation repository.
33
Potential Points of Metadata Capture
34
Potential Points for Metadata Capture

• Much of the necessary metadata for preservation
  is already generated by the production unit, but
  discarded after their internal use. This needs
  to be captured throughout the workflow.
• Those in the production unit are the creators
  and have first hand knowledge of who, what,
  where, when, and why the content was created. --
  Mary Ide and Leah Weisse, WGBH Archivists.

35
Creating Partnerships

• It is in everyones interest to see that the
  content is saved.
• Changes to the production workflow should be
  minimal and not disruptive.
• Implementation of DAM is the perfect time to
  begin pushing for changes in metadata delivery.
• This should provide immediate benefits to the
  producers work, as information and assets will
  be easier to find and manage.

36
Creating Partnerships

• WGBH AS A MODEL
• At production startup, the stations archive
  provides templates and check lists for eventual
  program and metadata delivery.
• Production units must deliver completed templates
  and all related materials before final payment.
• Production team completes File Maker Pro
  databases (using templates supplied by archive)
  for original footage, stock footage, stills, and
  materials used in the final edit.

37
Creating Partnerships

• WGBH MODEL
• Metadata collected not enough for digital
  preservation, but important lessons have been
  learned
• Gathering metadata from one system and trying to
  use it for maintenance on another introduces QC
  and conversion problems (e.g. Users and archives
  have different version of File Maker Pro).
• Extensive QC required because metadata coming
  from production units is often incomplete,
  inconsistent, or incorrect. Content creators
  need to be trained as well as convinced of the
  importance of metadata.

38
Creating Partnerships

• CONVINCING CONTENT CREATORS
• Create a clear financial incentive. DAMs save
  money (on stock footage and shooting) when
  editors and producers can search and retrieve
  station-owned assets right from their desk.
• Mandate from funding agencies, such as
  Corporation for Public Broadcasting (CPB) that
  preservation planning must be part of the initial
  proposal
• This is already happening in the sciences. NSF
  and NIH require long-term data sharing and
  preservation plans in research proposals.

39
Further Studies

• Test of effectiveness for suggested methods of
  metadata capture.
• Detailed study of metadata lifecycle throughout
  production and distribution workflow.
• Look into questions of accuracy
• How can we be sure the metadata is reliable?
• What methods are there to check the accuracy of
  metadata capture?

40
Conclusion

• In order to develop systems to capture more
  metadata during the production process, producers
  and curators will need to work closely together.
• Curators need to be sensitive to creators needs,
  and mindful of politics.
• Producers must be trained in creating metadata
  according to standards, and will need to realize
  the necessity of producing metadata that can be
  utilized by others.
• Many fields are working on this problem social
  science data archives, e-journals, etc.
• Ways of achieving the goals of upstream metadata
  capture should be shared between archivists and
  curators in all fields.

41
Preserving DigitalPublic Televisionhttp//www.pt
vdigitalarchive.org/