SAM-SRM design / specification outline.

1
SAM-SRM design / specification outline.

1. Baranovski, R. Mathur

06/29/06
2
Plot

• Apply SAM data handling policies over the generic
  storage system managed by SRM
• Premise ?
• Similarity in many supported operations
• transferring file, removing file, building
  directory listing, and retrieving file metadata
• But there are incompatibilities

3
SRM outline

• Get file. Translates logical file path into
  transfer url (location access cookie) compatible
  with requestor transport.
• Put file. Allocates space and translates file
  path into location access cookie compatible with
  requestor transport.
• Release file. Deallocates resources associated
  with calls 1) and 2)
• Copy file. Duplicate data file between 2 storage
  elements
• Must be able to create a file path on the
  storage.
• Must be able to free storage space by removing a
  file.
• May limit number of concurrent accesses by number
  that can be less than total number of requests in
  the system.
• Must allow data read access by credentials not
  originally used to copy, put, or get a file. An
  important requirement meant to coordinate and
  ensure shared storage management.
• Must allow querying for file and directory
  metadata.

4
So , what are the differences ?

• SAM managed disks need no explicit resource
  allocation outside of the program context.
• A file may implicitly (i.e. without explicit
  action from the storage user) disappear or
  change its cost of access category.
• explicit release op invoked on a resource handle
• SRM differentiate cost of data access on a per
  file basis.
• Data management is not in the same API and
  parameter space as actual data read and data
  write.

5
Summary of the idea

• Reuse SAM cache disks as the concept to
  describe SRM resource to SAM.
• SAM disks are the objects that contain memory of
  files. The same way, reuse disks to keep memory
  of files requested by SAM via SRM. - uniform data
  management policies over durable, permanent and
  volatile storages.
• Loose binding between storage and SAM.
• Adopt a mechanism to handle uncertain state of
  the files.
• Make SAM responsible for managing user load of
  SRMs.
• allocation/deallocation of the SRM handlers,
  keeping track of timeout conditions, and actual
  logical to transfer url conversion.

6
Types of storage

• Volatile storage
• may displace its own files
• SAM cache should expect that data
  delivery/pre-staging history may not be entirely
  accurate.
• finite size implicit space cleanup policies.
• advisory explicit file erasure requests from SAM.
• Durable storage
• Durable storage stores and removes file only by
  explicit concent.
• Explicitly manage the SRM space by issuing erase
  requests
• Permanent storage (dCache)
• Permanent storage never displaces files.
• the srm location must remain registered in the DB
  reflecting permanency property of the SRM.
• Data access cost of the permanently stored files
  may change!
• active and passive request prioritization
• Active request prioritization will use SRM to
  learn of access costs.
• Passive request prioritization will use SAM
  cache. Better cache hit rates while reducing SRM
  overhead.

7
Logical file location -gt transfer url mapping.

• SAM dispatches transfer urls not file logical
  path. ( see slide 4 )
• Interpret logical transfer request to invoke get
  or copy
• Analyze request source location
• If local to storage srmGet
• Otherwise use srmCopy
• SAM to SRM request will reference valid SRM
  location, transfer urls and request id.
• The outcome is cabled to SAM and to the analysis
  job.

8
Location mapping

• How not to overload storage directory structure ?
• Implement location mapping !
• The namespace replication implements common
  requirements on maximum number of files per
  location across all supported storage elements.
• How ?
• Append one of the source paths to the "base
  directory at the destination. But do it smart.
• Use the exact source path
• always-increasing path lengths
• Use the most suitable directory structure from
  the list of source locations!
• Use the path that is similar to the current
  protocol

9
Security model

• Problem ?
• Data must be shared between storage users.
• Common data handling can be enacted on behalf of
  the experiment group.
• Solution
• SAM will act on behalf of the unique principal to
  stage and remove experiment data.
• SAM will proxy actual access requests using
  users own principals.
• Ensures that SRM correctly prepares transfer urls
  in agreement with credentials that are normally
  presented by the analysis job at a time data is
  read.
• SAM use MyProxy to delegate users credentials to
  SRM.