Distribution API

1
Distribution API Serge Abiteboul, Dan Vodislav,
Boris Vrdoljak, Stéphane Laurière and Radu
POP June 2005
2
Topics

• Defining the Data Model
• Integration into the General EDOS API context
• Identifying the system Components
• Description of Roles
• Subscription and Querying
• AXML / KadoP use cases

3
Data Model

• Object Package Utility
  Collection
• Collection set of Objects
• levels of recurrence in Collections' definitions

• Package
• ID (Name,
• Version_Number)
• Metadata
• Value stream
• Ex
• perl-5.8.7-1mdk.rpm
• unique ID each Package !

• Utility
• ID Name
• Metadata
• Value stream
• Ex
• README.txt
• no Version_Number!

• Collection
• ID (Name,
• Version_Number)
• Metadata
• Value list_of_ObjectIDs
• Ex
• i586/media/jpackage
• hierarchical organization

4
Cooker Instantiation

• Cooker Objects
• physical constraints

i586/install/README.txt
i586/media/main/perl-5.8.7-1mdk.rpm
5
What to store in Metadata ?

• Metadata
• Location Object's address in the system
• Date Object's creation time
• Signature SH1 checksum
• Size Number of Bytes
• . . .
• Information used for indexing and searching.
• The type of the Object (Package Utility
  Collection) is announced at publishing into the
  system.
• Part of the information to be used at the
  Physical Level of the distribution (...see next)
  Number_of_Pieces large Objects will be
  splited in Pieces in the downloading algorithm.

6
Logical level vs Physical level

• Logical level
• coarse-grained functions that realize the
  distribution system functionalities publishing,
  replication, indexing and searching.
• Physical level
• fine-grained functions that implement
  logical-level functions.
• various physical-level implementations are
  possible for the same logical-level functions.
• the Value, seen as the content, of a Package or
  Utility object is handled by the system.
• split of large Packages into Pieces according to
  their Size.

7
Mappings to the General EDOS API

• Distribution API should plug into the
• General EDOS API.
• We employed a bottom-up aproach, using
  Mandriva's code distribution model.
• Possible mapping
• ContentUnit Package Utility
• Bundle Collection
• Project Distribution

8
Components and Roles

• Publication
• Replication
• Client

9
Publication role

• ... played by the Publisher
• publishPackage (Name, Version_Number,
  Metadata.Location)
• publishUtility (Name, Metadata.Location)
• publishCollection (Name, Version_Number,
  Metadata, Value)
• insertIntoCollection (CollectionName,
  CollectionVersion_Number,
  SetOfObjectIDs)
• deleteInCollection (CollectionName,
  CollectionVersion_Number,
  SetOfObjectIDs)
• replacePackage (PackageName, PackageVersion_Number
  ,
  NewPackageVersion_Number, Metadata)

10
Replication role

• ... played by the Replication Servers
• publishReplicatedPackage (Name, Version_Number,
  
  
  Metadata.Location)
• unpublishReplicatedPackage (Name,
  Version_Number)
• publishReplicatedUtility (Name,
  Metadata.Location)
• unpublishReplicatedUtility (Name)
• publishReplicatedCollection (Name,
  Version_Number, Value)
• unpublishReplicatedCollection (Name,
  Version_Number)

11
Client role

• ... played by the Peers
• getPackage (Name, Version_Number)
• getUtility (Name)
• getCollection (Name, Version_Number)
• getLastVersionNumber (ObjectName, ObjectType)

12
Other functionalities

• Subscription
• Can be used to provide event notification and
  possible automatic download of objects.
• Channel
• corresponds to a set of packages. For each
  channel, permissions can be assigned to distinct
  users.
• concept used in Red Hat and ideas proposed and
  discussed in the recent documents from Tel Aviv
  and Geneva
• In RHN, the base channel, the developer channel
  (receives selected packages from the base
  channel, according to developer's interest),
  testingQA channel and production channel.

13
Other functionalities Subscription (2)

• Similarly, we could have different developer
  channels according to developers' preferences.
• Some developers may want to get information about
  the complete set of changed packages for a
  certain environment, while others may want only a
  specific subset.
• After subscribing to one or more channels users
  get notification of updates according to their
  interests (Tel Aviv proposal). The users could
  then get the desired packages in two ways either
  by manually selecting them or automatically.

14
Other functionalities Subscription (3)

• Methods associated to the Publication role
• createChannel (ChannelID, listOfObjectIDs,
  privileges)
• For the publication of a package to a channel
• publishPackageToChannel (PackageName,
  PackageVersion, ChannelID)
• publishUtilityToChannel (UtilityName, ChannelID)
• publishCollectionToChannel (CollectionName,
  CollectionVersion, ChannelID)
• (another option is to enrich the functions like
  publishPackage, publishUtility and
  publishCollection).

15
Other functionalities Subscription (4)

• Methods associated to the Client role
• getChannelList
• subscribeToChannel (ChannelID, userID, )
• To select a package manually, the getPackage
  function is used (similarly for getUtility and
  getCollection)
• If we want automatic downloading
• getSubscPackage,
• getSubscUtility,
• getSubscCollection
• (or functions like getPackage should be enriched).

16
Other functionalities

• Querying
• The API should enable searching for data objects
  not only according to their IDs, but also by
  other criteria such as functionality, license,
  status, size, etc.
• The metadata will be queried in order to locate
  the wanted objects.
• Using a simple query language to define querying
  metadata values.
• Methods associated with querying (on the client
  side)
• locatePackage
• locateUtility
• locateCollection
• (input is a query string)

17

• Using Active XML and KadoP

18
Active XML

• AXML is a declarative language for distributed
  information management and an infrastructure to
  support the language in a peer-to-peer
  environment
• AXML documents are XML documents that include
  calls to Web services.
• When the call is activated and the result is
  received, the resulting XML content is inserted
  in the original XML document.
• If the external sources change, the same document
  will provide different information allows rapid
  reaction to changes in data.

19
Active XML

• Call activation issues in AXML
• when to activate the call
• periodically (daily, weekly), after some event,
  on users request, push (subscription, i.e.
  continuous services)
• what to do with its result
• append, replace, fusion
• how long a service call result is valid

20
Active XML

• A peer-to-peer system
• Each peer
• Repository of AXML documents
• Server provides Web services (which can include
  queries over XML data)
• Client when invoking the embedded service calls
• And other features
• distribution and replication
• continuous services

21
KadoP

• KadoP enables publication and discovery of
  content in P2P environment
• using Distributed Hash Table (DHT)
• structured peer network - good performance in
  finding peers that have data relevant for a
  search term
• automatic maintenance of the distributed index by
  using AXML
• the DHT index is refreshed when a data item
  changes
• the DHT keeps the network of peers connected in
  the event of peers joining or leaving the
  network, peer failures etc.
• basic ontologies isA, partOf, instanceOf
• enable semantic homogenization

22
AXML / KadoP

• AXML / KadoP system can be used in EDOS for
• indexing and searching for
• package and utility locations
• (for a package/utility ID the list of locations
  where it exists)
• collection composition
• (for a collection Id the list of object IDs in
  its composition)
• managing and querying metadata in general
• subscription AXML continuous services
• update and versioning KadoP automatically
  refreshes the DHT
• obtaining and managing measurements
• etc.
• To do
• Define replication in more detail. How much
  replication ?
• Allow splitting files ?

23
Future work

• Getting more insight on existing P2P systems
• OceanStore (Tapestry), KadoP (Pastry),
• Pgrid, Chord
• JXTA
• Architectures
• Enhance the API
• Define a general model
• Prototype
• Develop a system based on AXML KadoP
• Measurement, evaluation, comparison