SRM in GGF

1
Storage Resource Management Version 3.0
Fundamentals Storage Resource
Management Working Group Meeting Jefferson
Lab September 14-15, 2005 http//sdm.lbl.gov/srm-
wg
2
Finalizing SRM V3.0

• Extending functions for advance features
• Grouping functions into core and advanced
  features
• File and Space Types in Features
• File Sharing
• Releasing and removing files
• Reserving and Releasing Spaces
• Directory Management

3
Extending functions for advance features

• Issue what to do in the case of functions that
  are in core, but are also used in advanced
  features with additional functionality

• E.g. Specifying a space-token for srmCopy when
  space reservation feature is supported.
• Solution Extending parameters according to
  features

4
Extending parameters according to featuresAn
example

• 2.1. srmAbortRequest
• srmAbortRequest terminates the previously
  submitted request.
• In Out
• String userID
• String authorizationID
• String storageSystemInfo
• String requestToken
• Boolean isRemoval
• 2.1.1. Core Behavior
• a) Terminate all files in the request regardless
  of the file state. Remove files from the
  queue, and release cached files if applicable.
  Expired files are released, too.
• b) isRemoval flag is false by default. When
  isRemoval flag is true, all cached files for
  the request associated with the request token
  must be removed.
• c) Abort must be allowed to all requests with
  requestToken, including requests such as
  srmLs or srmCopy.
• d) If any file transfers are active, SRM should
  attempt to stop the transfer. If not
  possible, abort takes place after the pending
  transfers are completed.
• Behavior With Advanced Features
• 2.1.2. Copy Feature

ReturnStatusForRequest EnumStatusCode
statusCode, string
explanation
5
Grouping functions into core and advanced
features (1)
BASIC ADVANCED

• File movement
• PrepareToGet
• PrepareToPut
• Copy
• Request capabilities
• Multi-file Streaming
• Trans. Prot. Negotiation
• File lifetime negotiation
• File types
• Volatile
• Permanent
• durable

yes
yes
yes
yes
no
yes
yes
yes
yes
yes
no
yes
yes
yes
yes (for MSS)
yes
no
yes
6
Grouping functions into core and advanced
features (2)
BASIC ADVANCED

• Space reservations
• Space-time negotiation
• Space types
• Remote access
• gridFTP
• Other SRMs
• User-specified Directory
• Volatile
• Permanent
• Durable
• Terminate/suspend
• Abort file
• Abort request
• Suspend/resume request

no
yes
yes
no
no
yes
no
yes
yes
yes
yes
yes
no
yes
yes
yes
yes
yes
no
yes
7
File and Space Types in Features
2. Core Functions 2.1. srmAbortRequest 2.2.
srmAbortRequestedFiles 2.3. srmChangeFileStorageTy
pe 2.4. srmChangeFileStorageTypeStatus 2.5.
srmExtendFileLifetime 2.6. srmGetFeatures 2.7.
srmGetRequestSummary 2.8. srmGetRequestTokens 2.9.
srmGetSRMStorageInfo 2.10. srmGetTransferProtocol
s 2.11. srmLs 2.12. srmLsStatus 2.13.
srmPrepareToGet 2.14. srmPrepareToPut 2.15.
srmPutFileDone 2.16. srmPutRequestDone 2.17.
srmReleaseFiles 2.18. srmRm 2.19.
srmRmStatus 2.20. srmStatusOfGetRequest 2.21.
srmStatusOfPutRequest
4. Advanced feature set 2 Space Management 4.1.
srmCleanupFilesFromSpace 4.2. srmCompactSpace 4.3.
srmGetSpaceMetaData 4.4. srmGetSpaceTokens 4.5.
srmReleaseSpace 4.6. srmReserveSpace 4.7.
srmUpdateSpace 5. Advanced feature set 3
Directory Management 5.1. srmCp 5.2.
srmCpStatus 5.3. srmMkdir 5.4. srmMkdirStatus 5.5.
srmMv 5.6. srmMvStatus 5.7. srmRmdir 6.
Advanced feature set 4 Authorization 6.1.
srmCheckPermission 6.2. srmReassignToUser 6.3.
srmReassignToUserStatus 6.4. srmSetPermission 7.
Advanced feature set 5 Request
Administration 7.1. srmResumeRequest 7.2.
srmSuspendRequest
3. Advanced feature set 1 Remote Access 3.1.
srmRemoteCopy 3.2. srmStatusOfCopyRequest
8
File and Space Types in Features

• File types MUST be the same as the space type
  they are in
• Are space types global to all features of the
  SRM?
• e.g. Core functions support Volatile and
  Permanent, but want space reservations for
  Volatile only
• e.g. Core functions support Volatile and
  Permanent, butdirectory functions for Permanent
  only
• Should all functions in a feature support the
  same space type?
• Recommendation
• allow space types by feature
• all functions in a feature support same space
  type
• Extend discovery function to include space types
  supported
• Example
• core volatile, permanent, space_reservation
  volatile

9
File Sharing

• From the users point of view there is NO file
  sharing
• If files are physically shared, the SRM needs to
  keep
• Lifetime per user
• Space where file is per user
• State of file per user released / removed
• May need to copy files between spaces depending
  on space types
• Simplest implementation
• Copy multiple uses of files to users spaces
• Copy can be made locally avoid getting from
  remote site
• Space efficient implementation
• Spaces are virtual
• Keep track of file status virtually

10
Releasing and removing files (1)

• Files can be released (or removed) from spaces
  that are
• Implicitly assigned (Core or default)
• Only one space of each type can be implicitly
  assigned
• Need to specify space type in call
• Explicitly assigned (Space Reservation feature)
• Can have many spaces of same type
• If more than one space of a certain type exists,
  need to provide space token
• srmReleaseFiles
• Requires Request_Token, SURLs
• To release all file, need releaseAllCurrentlyPinn
  edFiles flag
• srmRm
• Does not require Request_Token, only SURLs
• Removed only for requesting user for all of
  his/her active requests

11
Releasing and removing files (2)

• srmCleanupFilesFromSpace (Space_Token)
• Releases all files in space regardless of request
• If remove flag is on files will be removed
• If files are shared, this has no effect on other
  users
• srmAbortRequestedFiles
• Releases files specified for Request_Token
• If remove flag is on files will be removed
• Files that are still in queue (not brought to
  cache) are removed from queue
• File in-transfer stop transfer if possible, and
  remove partial file otherwise complete transfer
  and release (or remove)
• srmAbortRequest
• Same as above, but applies to all files
• Aborting an srmRemoteCopy
• Need to notify remote site, of course
• Push case needs to propagate release / remove
  to remote site

12
Reserving and Releasing Spaces (1)

• Reserve space
• Can provide guarantee total (difference is
  best-effort)
• No confirmation expected, refusal is done by
  srmReleaseSpace
• Can reserve multiple spaces of the same type,
  subject to total allowed
• Lifetime of space is also negotiated
• Files in spaces should have shorter lifetime than
  space lifetime
• All files in spaces have the same type as space
  type
• Space assigned canbe discovered with
  srmGetSpaceMetadata
• Updating space
• Can request new sizes and lifetime (from current
  time)
• No consolidation of spaces
• Can compact space
• Compact space
• Space size is reduced by amount of unused space
• Released file are removed
• doDynamicCompactFromNowOn flag for dynamic
  compacting

13
Reserving and Releasing Spaces (2)

• Release space
• Only owner can release a space
• Once released cannot be used or updated
• If space not empty when release requested
• Volatile
• Released files will be removed and space
  compacted
• Files not released space release is postponed
  till they are released or lifetime expires
• Advisable to cleanup space before releasing
  space
• Durable
• Released files will be removed and space
  compacted
• Files not released space release is postponed
  till they are released
• Files not released and lifetime expires space
  release is postponed till durable-action takes
  place (e.g. archived and owner notified)
• Permanent
• Released files will be removed and space
  compacted
• Files not released stay in space, and space
  they occupy not released

14
Directory Management

• A single directory for all files types
• Files can belong to any spaces

D1
D1
D3
D2
D3
D2
D4
D4
F2 (P)
F1 (D)
F3 (V)
F1 (V)
F2 (V)
F3 (V)
F4 (D)
F5 (D)
F6 (D)
F7 (P)
F8 (P)
F4 (P)
F5 (D)
(1) Mixed file types
(2) By file type

• Supported functions srmLs, srmMkdir, srmRmdir,
  srmRm, srmMv, and srmCp (srmLs and srmRm are
  Core functions)
• Advantage of (1) no need to move files when
  file types are changed

15
V3.0 Client to Access Existing Servers
16
Using v3.0 Client to access existing SRM
implementations
SRM V3.0 Client
translator
translator
translator
SRM V1.1
SRM V1.1 Space R.
SRM V2.1
SRM V3.0 Core
SRM V3.0 CoreAF

• Need to add discovery function to translators
• Advantage can continue to use existing
  implementations