ECDL Workshop Extending Interoperability of Digital Libraries: Building on the Open Archives Initiat

1
ECDL WorkshopExtending Interoperability
ofDigital LibrariesBuilding on theOpen
Archives InitiativeLisbon September 21, 2000

• Edward A. Fox
• fox_at_vt.edu http//fox.cs.vt.edu
• CS DLRL Internet TIC
• Virginia Tech, Blacksburg, VA, USA

2
Acknowledgements (Selected)

• Sponsors CNI, DLF, Dept. of Energy, DFG, NASA,
  NSF,
• VT Faculty/Staff Anthony Atkins,
• VT Students Fernando Das Neves, George
  Fillipini, Robert France, Marcos Goncalves,
  Hussein Suleman,

3
Open Archives Initiative OAI www.openarchives.org
openarchives_at_openarchives.org
4
Program

• 9-10 Session 1 Introduction
• 1030-11 Break
• 11-1230 Session 2 Technical Details
• 1230-2 Lunch
• 2-330 Session 3 Discussion
• 330-350 Break
• 350-420 Session 4 Presentations
• 420-5 Session 5 Moving Forward

5
Program

• 9-10 Session 1 Introduction
• Introductory Remarks (Fox, Lagoze) 15 min
• Introductions from Participants
  (Fox, Lagoze) 30 min
• Historical Overview (Fox) 45 min

6
Introductory Remarks - Fox

• Welcome!
• Thanks to conference organizers
• Program/Logistics
• Latest in series of meetings that have shaped OAI
  during its first year

7
Introductory Remarks - Lagoze

8
Introductions from Participants - 1

• Straw Polls
• Training CS / LIS / Sciences / Humanities / ?
• Work Now CS / LIS / Sciences / Humanities / ?
• Location University / Industry / Gov. / Assn. /
  ?
• OAI Connection Run an archive or DL or
  collection / Manage data / Develop software /
  Standards / ?

9
Introductions from Participants - 2

• OAI Meeting Involvement Santa Fe mtg / San
  Antonio mtg / Technical Committee / Cornell mtg /
  Steering Committee
• OAI Trials Opened an archive / Developed
  software for OAI / ?
• OAI Project Wrote proposal / Plan to write a
  proposal / Have internally funded project / Have
  externally funded project

10
Introductions from Participants

• Short Statements (20 seconds per person)
• Name (pronounced slowly, clearly)
• Country
• Affiliation (institution/organization)

11
Historical Overview - Fox

• Meetings
• Santa Fe archives of the world unite
• Philosophy
• Repositories / Building on Black Boxes
• Approaches to building repositories
• VT view
• Some proposals for funding
• Development efforts

12
Open Archives Initiative (OAI)

• xxx_at_LANL, high-energy physics (Ginsparg, 1991)
• CSTR WATERS NCSTRL (Lagoze,1994)
• xxx NCSTRL CoRR collaboration (1998)
• Universal Preprint Service protoproto, Oct.
  21-22, 1999, Santa Fe led by LANL, CNI, DLF,
  Mellon --gt OAi
• Santa Fe Convention (see Feb. D-Lib Magazine
  article)
• Follow-on mtgs 6/3_at_San Antonio, 9/21_at_Lisbon
  (ECDL)
• Archives -gt Open Archives
• Support unique archive identifiers
• Implement Open Archives metadata set (DC, using
  XML)
• Implement OA harvesting protocol (derived from
  Dienst protocol)
• Register the archive
• Build tools, layer other services linking,
  searching,

13
Open Archives (protoproto)

• ArXiv Los Alamos National Lab
• CogPrints U. Southampton
• NACA NASA (reports)
• NCSTRL Cornell U.
• NDLTD Virginia Tech
• RePEc U. Surrey
• Total of around 200K records

14
Original Open Archives Members

• NASA Langley Research Cntr
• Old Dominion University
• Stanford University
• U. of Ghent
• U. of Surrey
• U. of Southampton
• Vanderbilt University
• Virginia Tech
• Washington University

• American Physical Society
• California Digital Library
• Caltech
• Coalition for Networked Info.
• Cornell University
• Harvard University
• Library of Congress
• Los Alamos Natl Lab
• Mellon Foundation

15
Open Archives Future 1st View

• EconWPA (U. Washington)
• e-biomed -gt PubMed Central (NIH)
• PubScience (DOE)
• Clinical Medicine Netprints ( other HighWire
  Press holdings )
• University ePub (California Digital Library)
• All public e-prints (MIT)
• Scholars Forum (Caltech)
• Intl CERN, Germany, India, Mexico,
• Goal millions of books/articles/reports / yr

16
OAi Philosophy

• Self-archiving submission mechanism
• Long-term storage system archive
• Open interface harvesting mechanism
• Data provider service provider
• Start with gray literature
• e-prints/pre-prints, reports, dissertations,

17
Tiered Model of Interoperability
Mediator services
Metadata harvesting
Document models
18
Repository of Digital Objects
Repository Access Protocol
handle
Metadata
terms and conditions
Digital object
19
OAI Repository Perspective
Required Protocol
Partition Structure
ID Scheme
MDO
MDO
MDO
MDO
MDO
MDO
MDO
MDO
Required OAMS
DO
DO
DO
DO
20
OAI Black Box Perspective
21
Black Box OAI-ETD Perspective


22
Approaches to Open Archives
Build By Institution
Build By Discipline
23
Approaches to Open Archives
Build By Institution
Build By Discipline
Access by
Author Category Interdisciplinary Year Language Qu
ery
24
Mechanisms

• Sharing
• Join initiative, run software
• Make metadata and archive available
• Aggregating
• By discipline
• By institution
• By genre
• Automating
• Workflow
• Harvesting and providing services
• Federated searching
• Dynamic linking (e.g., with SFX)

25
VT View of the Open Archives Initiative (OAI)

• Enable sharing of publication metadata and
  full-text by digital libraries
• Standardize low-level mechanisms to share
  contents of libraries
• Build higher-level user-centric and
  administrative services in meta-libraries
• Install organizational mechanisms to support the
  technical processes

26
Virginia Tech Projects

• MARC XML-DTD
• Computer Science Teaching Centre (CSTC)
• W3C Web Characterization Repository
• OAI Repository Explorer
• Networked Digital Library of Theses and
  Dissertations (NDLTD)
• OAI-Campus (esp. multimedia)

27
MARC XML-DTD

• XML Transport format for US-MARC records
• Standardized metadata exchange format for
  traditional library services joining OAI

28
CS Teaching Center (CSTC)

• Collection of reviewed online resources used to
  aid in teaching of Computer Science
• Supports author submission and peer-review
  process for new ACM Journal of Educational
  Resources In Computing (JERIC)
• Connected with NSDL (NSF 00-44)
• http//www.cstc.org

29
W3C Web Characterization Repository

• Online database of metadata related to
  publications, tools and data sets dealing with
  Web characterization
• Project of the Web Characterization Activity
  working group of the World-Wide-Web Consortium
  (www.w3c.org/WCA)
• http//purl.org/net/repository

30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
OAI Repository Explorer

• Serves as a compliancy test
• Allows browsing of open archives using only OAI
  protocol
• Sends requests on behalf of user, parses and
  checks responses and displays browsable interface
• Will detect most discrepancies in protocol
• http//purl.org/net/explorer

34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
OAI-Campus

• Undergrad term project for Honors course on
  digital libraries
• Aim is to have many OAs on campus
• Emphasis will be on multimedia collections
• Survey developed for campus
• http//intercom.virginia.edu/SurveySuite/Surveys/O
  AiVT

38
Funding Success

• NSF-DFG / VT-Oldenburg
• OAI research for next 3 years
• 2 countries
• 2 domains
• Physics
• Electronic theses and dissertations (ETDs)
• Evolution of existing efforts to use OAI
• Refinement of services as ontologies develop

39
Funding Failures

• NSF ITR Large
• US Dept. of Education (FIPSE) 5 sites
• Training
• Graduate students

40
(No Transcript)
41
Other Development Efforts

• Cornell Software
• Los Alamos Software
• Southampton Software
• ODU Software
• Other Software
• Registered Archives

42
Program

• 9-10 Session 1 Introduction
• 1030-11 Break
• 11-1230 Session 2 Technical Details
• 1230-2 Lunch
• 2-330 Session 3 Discussion
• 330-350 Break
• 350-420 Session 4 Presentations
• 420-5 Session 5 Moving Forward

43
Program

• 11-1230 Session 2 Technical Details
• Expanding the Scope and New Technical Agreements
  (Lagoze) 60 min
• Framing the Discussion for the Afternoon (Fox)
  30 min

44
Expanding the Scope and New Technical Agreements
- Lagoze

45
Framing the Discussionfor the Afternoon Fox - 1

• Divide into groups soon for lunch
• Sit and discuss in groups during lunch
• Groups report back in afternoon
• Present comments orally
• Lead discussion of those comments
• Groups submit report later through email

46
Framing the Discussionfor the Afternoon Fox - 2

• Possible Groups
• Political agendas and their unfolding
• Gray literature, Courseware/NSDL,
• Guiding principles for technical agenda
• What is an archive?
• What is best terminology?
• Implementation plans for OAI core

47
Framing the Discussionfor the Afternoon Fox - 3

• Possible Groups (contd)
• Requirements for OAI-related services /
  design of component-based DL
• Implementation plans for OAI-related services
• Linking OAI with other initiatives science data,
  

48
Program

• 9-10 Session 1 Introduction
• 1030-11 Break
• 11-1230 Session 2 Technical Details
• 1230-2 Lunch
• 2-330 Session 3 Discussion
• 330-350 Break
• 350-420 Session 4 Presentations
• 420-5 Session 5 Moving Forward

49
Program

• 2-330 Session 3 Discussion
• Funding Agencies/Sponsors 30 min
• General Discussion (Fox, Lagoze) 60 min

50
Funding Agencies / Sponsors

51
General Discussion

• Reactions to OAI agreements
• Applications of OAI to communities represented by
  attendees

52
Program

• 9-10 Session 1 Introduction
• 1030-11 Break
• 11-1230 Session 2 Technical Details
• 1230-2 Lunch
• 2-330 Session 3 Discussion
• 330-350 Break
• 350-420 Session 4 Presentations
• 420-5 Session 5 Moving Forward

53
Program

• 350-420 Session 4 Presentations
• Constantino Thanos (IEI-CNR, Italy)
• Robert Tansley (U. Southampton, UK)
• Eberhard Hilf (U. Oldenburg, Germany)

54
Program

• 9-10 Session 1 Introduction
• 1030-11 Break
• 11-1230 Session 2 Technical Details
• 1230-2 Lunch
• 2-330 Session 3 Discussion
• 330-350 Break
• 350-420 Session 4 Presentations
• 420-5 Session 5 Moving Forward

55
Program

• 420-5 Session 5 Moving Forward (Fox, Lagoze)
  40 min
• Plans for implementation
• Future research agendas
• Community building listservs,

56
VT - 1

• General purpose tools
• Husseins PERL implementation
• Marcos Java implementation
• OAI Repository Explorer Version 2
  http//purl.org/net/explorer

57
VT - 2

• NSDL / XXDL ?
• Bill Graves, Collegis/Eduprise
• IMS
• UNC Wilmington
• Chemistry, CS, Math,
• CSTC already involved in OAI

58
VT - 3

• MARIAN
• Evolved from CODER (1987)
• C/C version SIGIR93
• Research and production DL system
• Harvest/Gateway Dienst, Harvest, OAI, Z39.59
  OAI to Greenstone, Phronesis

59
www.theses.org
MARIAN
VTLS
OAI Protocol
Sets by subject
Sets by origin
Optional MARC
MDO
MDO
MDO
MDO
ETDs qualified DC
MDO
MDO
MDO
MDO
MDO
MDO
MDO
MDO
OAI DC
DO
DO
DO
DO
Dienst
Harvest
Z39.50
OAI - 1
OAI - 2

60
(No Transcript)
61
Program

• 420-5 Session 5 Moving Forward (Fox, Lagoze)
  40 min
• Plans for implementation
• Future research agendas
• Community building
• Closing
• Thank you!
• Future meetings JCDL2001, ECDL2001
• Online discussions