SAMGrid Developments Part I

1
SAMGrid Developments Part I

• Sinia Veseli
• CD/D0CA

2
Completed Projects

• DB Schema Redesign
• Python API
• Batch Adapter
• D0 Online dCache TDP
• Monitoring Tools
• Data Dimensions Grammar (Rob K.)

3
Ongoing Projects

• DB Server
• C API
• Configuration Management
• Test Harness
• Grid-JIM Developments (Rob K.)
• Chains and Links (Rob K.)
• CDF DFC Migration (Rob K.)
• CDF (H Stream) Reprocessing (Rob K.)
• Station Developments (Rob K.)
• Linux Deployment of Centralized Services (Rob K.)
• Subproject Assessments (Rob K.)

4
DB Schema Redesign

• Provided support for the unified (D0/CDF) DB
  schema
• Wyatt and Adam were driving the efforts, many
  others have been involved
• Description of schema changes can be found on
  http//www-d0.fnal.gov/lyon/samSchema/
• Completed in May 2003, few minor changes
  introduced later
• One of the major reasons for the DB Server rewrite

5
Python API

• Most of the work done by Lauri
• Replacement for the old sam_user code
• Benefits
• Real Python API
• Self-documenting interfaces
• Maintainable code
• Improved performance
• Status
• Deployed in production since June 2003
• Maintenance load is very low
• Ongoing work on replacing the functionality
  relevant to file metadata

6
Batch Adapter

• Replacement for the old station batch adapters
• Features
• Flexible/configurable, supports any batch system
• Non-restrictive (i.e., does not require specific
  batch system configuration)
• Support for multiple user processes, grid job
  submission interface
• Status
• Deployed in production since June 2003
• Low maintenance load
• Possible future work
• ability to export/import adapter configuration in
  XML format

7
D0 Online dCache TDP (Tapeless Data Path)

• Work done by Lauri/Sinia
• Provides functionality of storing files via
  dCache write pools for the D0 online system
• Reduce resource contention (tape drives)
• Files are immediately available for use, even
  before they are stored on tape
• D0 requirement ability to choose between encp
  and dccp for different streams this led to
  sam_cp redesign

8
D0 Online dCache TDP

• New sam_cp features
• Refined interface ability to perform tasks
  before and after invoking of the actual file
  transfer
• New file transfer protocol management ability to
  plug in an arbitrary algorithm for determining
  which protocol to choose
• Status
• Used by D0 Online in production since November
  2003
• Operations load problems with online machines
• Possible future sam_cp work
• XML-based configuration
• Server which handles handshakes between sites

9
Monitoring Tools

• Excellent monitoring tools developed in the last
  year by Adam and Robert
• SAMTV shows what is happening with stations and
  projects (http//d0db-dev.fnal.gov/sam/samTV/samTV
  .html)
• CAB monitoring automatically reconfigure the CAB
  station depending on the problems in the cluster

10
DB Server

• Work started in June 2003 after agreement on the
  new DB schema was reached
• Developers involved Lauri, Carmenita, Steve,
  Sinia none of us worked full time the project
• Weve got lots of help from DBAs (Julie, Anil)
• mini-SAM developer databases
• Benefits
• Code base and the number of IDL interfaces were
  significantly reduced and cleaned up
• New CORBA infrastructure wrapper classes around
  IDL structs and methods, exception handling
  lead to code reuse, easier development, more
  maintainable code

11
DB Server

• Benefits
• Removal of non-atomic methods
• Multithreading capability
• Replaced fnorb with omniorb better performance,
  less bugs
• New file metadata classes more flexibility,
  easier maintenance
• New set of administrative interfaces
• Merging of the MC and Farm Request systems
• Improved autodestinations

12
DB Server

• Status
• Basic functionality is ready (metadata classes,
  station/project interfaces)
• We are getting ready to upgrade the D0
  development DB schema, start running test harness
• To be completed MC/Farm Request system merge,
  autodestinations, client code
• Deployment plan
• minimize effect on users, do not force immediate
  client upgrades
• introduce new servers and upgrade clients
  gradually as they become ready

13
DB Server

• Milestones
• Early March upgrade D0 integration DB to the
  latest schema, start integration testing
• Early April upgrade D0 production DB to the
  latest schema, start deploying new db servers -
  CDF will follow
• Next project assessment mid-March

14
C API

• Provide unified support for both D0 and CDF SAM
  C clients
• Code reuse, common CORBA infrastructure
• Faster development, lower maintenance load
• New features
• File deliveries via CORBA callbacks
• SAMTV Ready! (i.e., has complete monitoring event
  infrastructure)

15
C API

• Planned features
• Support for the multiple naming services
• Ability to influence client behavior at runtime
  (e.g., direct the client to stop requesting
  files)
• Support for staging files
• Ability to define datasets, start projects
  (needed for CDF)
• Status
• Initial version released, D0 C client
  (sam_manager) using it (sam_manager code reduced
  to 30 of its original size)
• CDF (Fedor) is looking into adapting AC to use
  the API
• Mid-March milestone add CDF-requested
  functionality
• Next project assessment mid-March

16
Configuration Management

• Goal simplify existing (complex)
  installation/configuration, eliminate code
  compatibility problems, etc.
• Work done by Art, recently joined by Adam on the
  D0 side
• Status
• Station installation scripts ready, used by CDF
  in production
• Adapting procedure for D0 in progress
• We need similar scripts for client code
• We have to revisit upgrade/downgrade procedures

17
Configuration Management

• Late February milestone both experiments should
  be using the scripts for their standard station
  installations
• Next project assessment mid-March

18
Test Harness

• Old test harness
• Has not been developed in couple of years, did
  not keep up with the SAM code development
• Hard to adapt for CDF usage
• Work on the new test framework started in
  December 2003 (Matthew, Stefan, Adam)
• Key features
• Easier, XML-based configuration
• Written in python only, uses new API, easier to
  maintain
• Cleaner output, easier monitoring
• More flexibility in creating/adding tests

19
Test Harness

• Status
• Basic infrastructure completed
• Work on creating tests is in progress
• Mid-March milestone demonstrate basic
  functionality (tests involving file
  delivery/storage)
• Next project assessment early April

20
Concluding Remarks

• To be continued by Rob