SAMGrid Status

1
SAMGrid Status

• Adam Lyon (FNAL-CD)
• April 29, 2004
• DØ-Grid/Remote Computing Workshop
• Imperial College, London

• Outline
• Status and Statistics
• News Bites
• Reprocessing Suggestions

2
The World of DØ SAMGrid
Active Inactive
3
DØ Sam Station Locations - USA/Canada
Active Inactive
4
DØ Sam Station Locations - Europe
Active Inactive
5
Usage over past year (high-end)
6
Usage over past year (low)
7
MC Contribution
8
MC Contribution
9
Reprocessing Contribution
10
Reprocessing Contribution
11
SAM Statistics How much data has been
analyzed?
Data from early January 6 until February 24
256 TB!
Raw
Thumbnails
8.3 Billion Events!
12
Week of the Docs

• Improving our documentation
• New SAM Websitehttp//projects.fnal.gov/samgrid
• Writing new and rewriting old docs
• Improvements to DB Schema, Station, User, Shifter
  documents
• Still much work to do (will take more weeks)

13
Improving the SAM Implementation

• New Improved Schema (expand beyond
  production-centric)
• Addition of file types
• Multiple runs per file
• Many other changes
• DB Server Rewrite
• Coming soon with 1 month commissioning period
• Integrates the new Schema
• Easier to maintain infrastructure
• Better Dimension Queries
• Richer language
• Aim for improved robustness
• Improve Caching Implementation
• Refactoring the SAM Station Code (probably a
  rewrite)
• SAM will talk "SRM" to storage elements for
  caching

14
Improving the SAM Implementation

• Improve monitoring
• samTV has been taken to the limit
• Start researching a 2nd generation monitoring
  tool
• Implement MIS
• Monitoring Information Service
• Executables send monitoring data over network (no
  log file parsing)
• Back-ends record data or later study
• Back-ends act on urgent monitoring data
  immediately
• Why? Look for problems measure efficiency and
  productivity discover new use cases automated
  accounting make SAM better

15
Contributions from CDF to SAM

• Init-Sam
• Script for easier station installation
• Tested at Fresno
• Test Harness
• Completely rewritten and much improved
• Easy configuration and operation
• True component/integration tests
• I'm using it for SAM stress testing

16
SAM Stress Testing

• Test SAM running on Linux server nodes
• Find Optimal Parameters
• Max transfers
• Time-outs
• Cache strategies
• Try different configurations
• Route all files through file server?
• Discover the best configuration and optimal
  parameters for different use cases

17
SAM Stress Testing
max transfers 5
max transfers 1
18
SAM Stress Testing
19
SAM Stress Testing
max transfers 5
max transfers 1
20
SAM Stress Testing
max transfers 5
max transfers 1
21
SAM Stress Testing
max transfers 5
max transfers 1
22
GridPP Metadata Workshop

• Hosted by Rick St. Denis, CDF/U. of Glasgow
• The GRID should learn from SAM
• We presented SAM,"SAM is a collection of related
  services based on metadata modeled on a
  relational database"
• ATLAS and LHCb presented their technologies
• HEPCAL, OGSA-DAI, ARDA, AMI, POOL and related
  metadata DB schemas
• SAM and our advice were well received
• Our use cases and usage data will be studied by
  GridPP for their simulations

23
JIM News

• In use for production MC Requests at
• Lyon, Manchester, Wisconsin
• Merging handled by JIM
• Executables delivered by SAM
• Capability for stager free operation
• Still operational issues
• Some DB server queries un-performant
• Future Plans
• Init-JIM installation script (CDF Contribution)
• Implementation of Brokering
• Conversion to VDT-Globus 2.4 (currently at 2.0)

24
Reprocessing Proposals

• Use JIM for your job submissions
• Need MC Production RunJob merged with
  Reprocessing RunJob (so can use JIM)
• Merging is now part of production (no more
  merging at FNAL)
• Run SamTV at your site for monitoring
• Contribute to new DB server methods during
  commissioning period
• Improve Accounting

25
SAMGrid Related Reprocessing Timeline

• Propose accounting queries for new DB Server by
  June 1, 2004
• Reprocessing sites must install JIM by July 1,
  2004
• Init-JIM should be ready by then
• Test your SAM Station with the Test Harness by
  July 1
• Make sure your station works before you have to
  process requests
• Perhaps convene a problem solving workshop in
  mid-July so you can work with our experts
• Will anyone come?
• Performance benchmarking of transfers to your
  station

26
Conclusions

• Over the past year 44 stations consumed
• 3.6 million files (0.46 million files remote)
• 40 billion events (3 billion events remote)
• 1.6 Petabytes of data (137 TB of data remote)
• Over the past year 25 million MC events produced
  remotely
• Over the past year 90 million events reprocessed
  remotely

27
Conclusions

• SAM is successfully handling huge amounts of data
  for DØ both at home and away
• We think the "SAM way" is the right way. The GRID
  is starting to realize this too
• We continue to improve the implementation of the
  "SAM way"
• Looking forward to a successful reprocessing with
  p17