Storage Resource Managers:

1
Storage Resource Managers Essential Components
for the Grid Arie Shoshani Staff Alex Sim,
Junmin Gu, Alex Romosan, Viji Natarajan Scientifi
c Data Management Group Lawrence Berkeley
National Laboratory http//sdm.lbl.gov/srm
2
Outline

• What are Storage Resource Managers - Motivation
• General Analysis Scenario and the use of SRMs
• SRM functionality
• Real examples of working SRMs
• Advantages of using SRMs
• Conclusions and Future Work

3
Motivation

• Grid architecture needs to include reservation
  scheduling of
• Compute resources
• Storage resources
• Network resources
• Storage Resource Managers (SRMs) role in the data
  grid architecture
• Shared storage resource allocation scheduling
• Especially important for data intensive
  applications
• Often files are archived on a mass storage system
  (MSS)
• large scientific collaborations (100s of
  clients) opportunities for file sharing
• File replication and caching may be used
• Need to support non-blocking (asynchronous)
  requests

4
A multi-file request to aDisk Resource Manager
5
Accessing Remote Storage Resource Managers
6
General Analysis Scenario

Uniform SRM Interface
7
File movement functionality srmGet, srmPut,
srmReplicate
8
SRM is a Service(OGSA, CORBA, C, Java, )

• SRM functionality
• Manage space
• Negotiate and assign space to users
• Manage lifetime of spaces
• Manage files on behalf of a user
• Pin files in storage till they are released
• Manage lifetime of files
• Manage action when pins expire (depends on file
  types)
• Manage file sharing
• Policies on what should reside on a storage
  resource at any one time
• Policies on what to evict when space is needed
• Get files from remote locations when necessary
• Purpose to simplify clients task
• Manage multi-file requests
• A brokering function queue file requests,
  pre-stage when possible
• Provide grid access to/from mass storage systems
• HPSS (LBNL, ORNL, BNL), Enstore (Fermi), JasMINE
  (Jlab), Castor (CERN), MSS (NCAR),

9
Types of SRMs

• Types of storage resource managers
• Disk Resource Manager (DRM)
• Manages one or more disk resources
• Tape Resource Manager (TRM)
• Manages access to a tertiary storage system (e.g.
  HPSS)
• Hierarchical Resource Manager (HRMTRM DRM)
• An SRM that stages files from tertiary storage
  into its disk cache
• SRMs and File transfers
• SRMs DO NOT perform file transfer
• SRMs DO invoke file transfer service if needed
  (GridFTP, FTP, HTTP, )
• SRMs DO monitor transfers and recover from
  failures
• TRM from/to MSS
• DRM from/to network

10
SRM works with other SRMsas well as legacy
systemsby using GridFTP
client
Request Interpreter
Request Manager
Logical Request
Legend
GridFTP
DRM
Control path
Data Path
Berkeley
Berkeley
Chicago
Livermore
server
server
server
server
DRM
HRM
GridFTP
GridFTP
FTP
GridFTP
11
Earth Science Grid
LBNL
HPSS High Performance Storage System
ANL
CAS Community Authorization Services
NCAR
HRM Storage Resource Management
gridFTP Striped server
gridFTP server
openDAPg server
Tomcat servlet engine

MyProxy server
MCS client
MyProxy client
LLNL
DRM Storage Resource Management
RLS client
CAS client
DRM Storage Resource Management
GRAM gatekeeper
gridFTP server
ORNL
gridFTP server
gridFTP
HRM Storage Resource Management
USC-ISI
gridFTP
gridFTP server
HRM Storage Resource Management
MCS Metadata Cataloguing Services
SOAP
HPSS High Performance Storage System
RLS Replica Location Services
RMI
NCAR-MSS Mass Storage System
12
Uniformity of Interface ?Compatibility of SRMs
Client USER/APPLICATIONS
Grid Middleware
SRM
SRM
SRM
SRM
SRM
SRM
Enstore
DCache
JASMine
CASTOR
13
Screen Dump of Demo at Fermi Booth
14
Where do SRMs belongin the Grid architecture?
15
SRMs provide a brokering serviceby supporting
multi-file requests
16
DataMover SRMs use in ESG and PPDG for Robust
Muti-file replication
17
Web-Based File Monitoring Tool

• Shows
• Files already transferred- Files during
  transfer
• Files to be transferred
• Also shows for
• each file
• Source URL
• Target URL
• Transfer rate

18
File tracking helps to identify bottlenecks
Shows that archiving is the bottleneck
19
File tracking shows recovery from transient
failures
Total 45 GBs
20
File tracking shows network slowdown and recovery
Total 53 GBs
21
Types of Spaces and Files

• Space reservation services
• Spaces and files volatile, durable, permanent
• Lifetime, action at end of lifetime
• Volatile SRM owned, files can be removed if
  space needed
• Durable files cannot be removed, but
  administrator notified
• Permanent can be removed by owner only
• Directory services
• Usual unix semantics
• any type of files in directory
• Access control services
• Support owner/group/world permission
• Can only be assigned by owner
• File sharing for read-only files
• check with source for shared file permission
• File sharing for updatable files
• check with master copy for time of last update

22
SRM Methods
Space management srmReserveSpace srmReleaseSpace s
rmUpdateSpace srmCompactSpace srmGetCurrentSpace
  FileType management srmChangeFileType
Status/metadata srmGetRequestStatus srmGetFileS
tatus srmGetRequestSummary srmGetRequestID srmG
etFilesMetaData srmGetSpaceMetaData
File Movement srm(Prepare)Get srm(Prepare)Put sr
mReplicate   Lifetime management srmReleaseFiles
srmPutDone srmExtendFileLifeTime Terminate/res
ume srmAbortRequest srmAbortFile srmSuspendReques
t srmResumeRequest  
23
Summary advantages of using SRMs

• Synchronization between storage resources
• Pinning file, releasing files
• Allocating space dynamically on as needed basis
• Insulate clients from storage and network system
  failures
• Transient MSS failure
• Network failures
• Interruption of large file transfers
• Facilitate file sharing
• Eliminate unnecessary file transfers
• Support streaming model
• Use space allocation policies by SRMs no
  reservations needed
• Use explicit release by client for reuse of space
• Control number of concurrent file transfers
• From/to MSS avoid flooding MSS and thrashing
• From/to network avoid flooding and packet loss

24
Ongoing and Future Work

• Ongoing work
• Developing Standard SRM interfaces
• Particle Physics Data Grid (PPDG) project
• LBNL, TJNAF, FNAL
• European Data Grid (EDG) project
• WP2 - data management
• WP5 mass storage
• Deployment
• LBNL, BNL, ORNL, TJNAF, FNAL, CERN, (SE-England)
• Use of SRM by other agents
• Storage Resource Broker (SDSC) calling HRM to
  Stage files from HPSS
• GridFTP invoking HRM
• New Spec completed (SRM V2.1)
• directory management
• File/directory file movement
• dynamic space management
• Future work
• Access authorization community access service
  (CAS)
• On-demand space allocation, accounting, and
  charging