Managing Metadata In SRB Arcot (Raja) Rajasekar

1
Managing MetadataIn SRBArcot (Raja) Rajasekar
Storage Resource Broker
SRB Space
2
Metadata in Data Grids

• Infrastructure independence
• Enable distribution of data across multiple
  storage systems
• Provide essential capabilities needed for a
  scalable data management system
• Administrative metadata
• Logical Name Spaces
• Provides management needed
• for resources, users, files
• Collection-based data management
• Discovery through Attributes/Schema
• User-defined metadata based discovery
• Beyond textual search

3
Metadata SRB

• SRB-aware Metadata
• Metadata in external databases
• Database access independent of SRB
• SRB-aware DB Schema
• Has columns containing SRB Object URL/PathNames
• Values can be used to create SRB Object
  URL/PathNames
• Application queries DB then accesses data in SRB
• SRB-integrated Metadata
• Non-MCAT database
• Accessible through SRBs DAI interface
• SRB-aware DB Schema
• SRB Object URL/PathNames
• SRB access-controlled DB
• No native DB interface needed by Application
• MCAT-integrated Metadata
• SRB Object access integrated with Metadata query

DB
APP
SRB
FS
SRB
DB
APP
MCAT
FS
4
MCAT-integrated Metadata

• Queriable System Metadata
• Owner, FileSize, DataType, Comments,
  CreationTime, ResourceType, Location,
• Free-form Metadata (User-defined)
• Attribute-Value-Unit Triplets
• No limits on number of Triplets
• Metadata for Data, Collection, User, Resource
• Supports Attribute-list Schemas
• Useful for Interactive Data Ingestion Querying
• Dublin Core, EcoCore, Email,
• Extensible Schema Metadata
• User Tables integrated into MCAT Core Schema
• Logical Schema (aka view) exposed to user
• No limit on number of logical schemas
• Can query on subset of logical schemas

5
Extensible MCAT Schema
SRB Client
MCAT Server
External DB
External DB
MCAT DB
External DB
External DB
External DB
SRB Server
An extensible Universal Schema is exposed to
the MCAT and User
SRB Server
SRB Servers
ExtDB-6 Schema
SRB Servers
Local Disk
HPSS
Others
Object Ring Buffer
6
How to Query Use System Metadata?

• Query Commands - each command has multiple
  options
• Data SgetD, Sls, Slscont, SmodD
• Container SgetColl, SmodColl
• Resource SgetR, SmodR
• User SgetU, SmodifyUser
• Others SgetT, Schksum, Stoken
• Integrated Query and Data Access
• Scat, Sget
• Scat S.cCTIME gt 2005-07-26-15.00
• Sget A CTIME gt 2005-07-26-15.00
• Scat S.cCTIME gt 2005-07-26-15.00VNUMgt2
  

7
How to Query Use Freeform Metadata?

• Query Commands
• Smeta depregated
• Sufmeta user-friendly
• List Files Sufmeta -Q alpha 200 beta bar
• List Collections Sufmeta -Q c title like
  srb
• Insert/Modify Commands
• File MetadataSufmeta alpha 200 foo.dat
• Coll MetadataSufmeta -c beta 200 myColl
• Bulk MetadataSufmeta D -f mdFile '' (multiple
  srbObj)
• Sufmeta -f mdFile2 '' foo.dat

8
How to Query Use Freeform Metadata?

• Integrated Query Data Access
• SgetD -A "ATTRCONDD color red" ''
• this queries for data with userdefined
  attribute-value pairs the form is ltAttrNamegt
  ltOperationgt ltValuegt with spaces between them
• SgetD -A "ATTRCONDD color red ATTRCONDD shape
  like box '
• can use conditions on ATTRCONDD upto 5 times
• SgetD -A "ATTRCONDC author fred ATTRCONDD
  color red '
• Querying for multiple conditions on same
  attribute
• SgetD -A "SIZE lt15000 gt 10000 " ''
• SgetD -A "RESOURCE like 'raja' like 'du
  " ''

9
Query Attributes

• DATA name of the object
• COPY replica number of the object
• GUID GUID value of the object. This is
  a string
• DTYPE data type of the object
• COLLECTION collection where the object can
  be found
• RESOURCE resource name where the object
  resides
• RTYPE resource type where the object
  resides
• NETADDR net address of the resource
• PATH physical path name of the object
• DCOMMENTS comments on the object
• DTIME time of last modification of the
  object
• DATA_CHECKSUM checksum of the object
• USER user name
• OWNER owner of the object
• TICKET ticket issued on the object
• SIZE size of the object
• CTIME creation timestamp of the object
• DIRTY dirty-flag of the object
• VNUM version string (even though
  called number) of the object

10
How to Use Extensible Metadata?

• Query Commands
• Query for Schema Attribute Names
• Squery -E -L
• 420mets.creator -gt sekar.me_core.author
• 421mets.data_id -gt sekar.me_core.data_nu
  mber
• 422mets.object_name -gt
  sekar.me_core.data_name
• 423roadnet.data_id -gt srb.sensor.data_id
• 424roadnet.location_lat -gt
  srb.sensor.location_lat
• 425roadnet.location_lon -gt
  srb.sensor.location_lon

E show extensible attributes L show their
physical counterparts
Internal ID
External Schema Name
Database Schema/DBLink Name
Database Attribute Name
External Attribute Name
Database Table Name
11
How to Query Extensible Metadata?

• Query Commands
• Query for Metadata
• Find all values of latitudes in the sensor
  schema?
• Squery -S roadnet.location_lat
• Find all locations where wind sensors are
  located?
• Squery -S roadnet.location_lat -S
  roadnet.location_lon
• roadnet.sensor_type like 'wind'
• Query for Discovery of Data
• Find all sensor-streams that are related to wind
  sensors?
• Squery -S DATA_NAME roadnet.sensor_type
  like 'wind
• DATA_TYPE sensor stream
• Find all files related to a particular location?
• Squery -S DATA_NAME roadnet.location_lat
  70N
• roadnet.location_lon 35E

12
How to Modify Extensible Metadata?
Object Name
Collection Name

• Insertion into a Table
• SmodE -i Spwd.c /home/sekar.sdsc/mytest
  srb.validity 'sensor_namestart_dateend_datecomm
  ents' "'SCA''2004-02-10-10.30''2004-02-22-10.20
  'internal software error'
• Deletion from a Table
• SmodE -d Spwd.c /home/sekar.sdsc/mytest
  srb.validity sensor_name " 'LAB'"

Attribute Values Within
Attribute Name List
Table Name
Deletion Condition
13
For More Information

• Man Pages
• Shelp
• Sattrs
• Sufmeta (Smeta being deprecated)
• Squery
• Stoken
• Sget, Sput, SgetD, SgetColl, SgetR, SgetU, SgetT
• SmodD, SmodColl, SmodR, SmodifyUser, Sticket
• Plus other Man Pages
• ReadMe Files
• README.MCAT.ATTRIBUTES
• README.extensibleschema

14
Attribution

• SRB Team Collaborators
• srb_at_sdsc.edu
• srb-chat_at_sdsc.edu
• http//www.sdsc.edu/srb/