Storage Resource Broker SRB Introduction

1
Storage Resource Broker (SRB) Introduction

• Michael Ernst

2
Problems with data handling

• Large Datasets Large Number of Datasets Scaling
• Distributed, Heterogeneous Storage
• Replication, Coherency
• Fault Tolerance and Load Distribution
• Scheduling, Caching Data Placements
• Data Migration over Time Space
• Data/Collection
• Uniform Name Space
• Handling Legacy Data and Data/Resource Evolution
• User-friendly Interfaces

3
Data handling requirements

• Seamless access to data and information stored at
  local and remote sites
• Virtualization of data, collection and meta
  information
• Handle Dataset Scaling size number
• Integrate Data Collections Associated Metadata
• Handle Multiplicity of Platforms, Resource Data
  Types
• Handle Seamless Authentication
• Handle Access Control
• Provide Auditing Facilities
• Handle Legacy Data Methods

4
SRB

• The Storage Resource Broker is a middleware
• It virtualises resource access
• It mediates access to distributed heterogeneous
  resources
• It uses a MetaCATalog to facilitate the
  brokering
• It integrates data and metadata

MCAT
Application
SRB Server
HRM DB2, Oracle, Illustra, ObjectStore
HPSS, TSM, UniTree
UNIX, NTFS, HTTP, FTP
(dCache/Enstore, Castor added by M.Ernst / FNAL)
5
SDSC Storage Resource Broker Meta-data Catalog
Application
Resource, User
Java, NT Browsers
Prolog Python
Third-party copy
Web
User Defined
Remote Proxies
MCAT
HRM
Dublin Core
DataCutter
Application Meta-data
dCache, Castor
6
SRB Concepts

• Abstraction of User Space
• Single sign-on
• Multiple authentication schemes
• certificates, (secure) passwords, tickets, group
  permissions, roles
• Virtualisation of Resources
• Resource Location, Type Access transparency
• Logical Resource Definitions - bundling
• Abstraction of Data and Collections
• Virtual Collections Persistent Identifier and
  Global Name Space
• Replication Segmentation
• Data Discovery system application metadata
• User-defined Metadata Structural Descriptive
• Attribute-based Access (path names become
  irrelevant)
• Uniform Access Methods
• APIs, Command Line, GUI Browsers, Web-Access
  (Portal,WSDL, CGI)
• Parallel Access with both Client and
  Server-driven strategies

7
Replication

• Core Functionality
• Synchronous Replication
• Replication via Logical Resource definition
• integrated into open/create write function
• Can choose k out of n
• Associate replication with containers/collections
• Consistency
• Asynchronous Replication - Offline
• srbObjReplicate API , Sreplicate command, GUI
• Out of Band Replication - outside SRB
• Registering of Replicas using srbRegisterReplica
  API

8
Access Control

• Access Control
• Datasets
• Collections
• Resources
• Multi-level Access
• Read, Annotate,Write, Curate, Own
• Access Control for Users and Groups
• Ticket-Based Access Control
• Audit Access

9
MCAT Metadata Catalog

• Stores metadata about
• Data sets, Collections, Users, Resources, Proxy
  Methods
• Maintains replica information for data
  containers
• Provides Collection abstraction for data
• Provides Global User name space
  authentication
• Provides Authorization through ACL tickets
• Maintains Audit trail on data collections
• Maintains metadata for methods and resources
• Provides Resource Transparency - logical
  resources
• MCAT configurable for different domains
• Implemented as a relational database
• Oracle or DB2 or Sybase
• Links from meta data to actual data can be
• pointer to a file location
• An SQL query
• ..

10
Meta Data Management

• Insert, Update and Delete of Metadata
• Access Control for Metadata
• Querying across system-level, user-defined
  metadata and annotations
• Query under collections across collections
• Browsing on user-defined metadata
• Metadata supported for legacy files directories
• Extract Metadata (using proxy operations)

11
Several SRB Projects

• Digital Libraries
• UCB, Umich, UCSB, Stanford,CDL
• NSF NSDL - UCAR / DLESE
• NASA Information Power Grid
• DOE ASCI Data Visualization Corridor
• Astronomy
• National Virtual Observatory
• 2MASS Project (2 Micron All Sky Survey)
• Particle Physics
• Particle Physics Data Grid (DOE) (babar)
• SLAC Synchrotron Data Repository
• Medicine
• Digital Embryo (NLM)
• BIRN (Biomedical Informatics research network
• Earth Systems Sciences
• ESIPS (Earth sciences information partners)
• LTER (Long term ecological research)
• Persistent Archives
• NARA

12
SRB future

• Investigate difference/commonalties with tools
  such as replica manager
• Federated MCAT. Stanford Lyon are planning to
  implement a separate MCAT catalog to manage the
  digital entities across multiple systems
• Implement GridFTP (currently GridFTP less stable
  than their own transfer protocol