B4 Application Environment Load Balancing Job and Queue Management

1
B4Application EnvironmentLoad BalancingJob and
Queue Management

• Tim Smith CERN/IT

2
Application Environment (I)

• Q How to ensure compatibility between execution
  and development environments
• Accessing static system and application tools
• Remote Shared file system
• Local
• Tools for client synchronisation
• Pre-compiler to hide target differences
• Defining environment variables
• Group accounts Useful, but bad for security /
  auditing
• Framework for environment definition

3
Application Environment (II)

• Accessing user files and application libraries
• Remote
• Shared file system
• Put / get mechanisms
• Local
• Ship libraries with the job
• Tool for sinking to clients
• Static vs dynamically linked binaries
• Dynamic has little benefit on a 2 processor batch
• Security issues with picking up random libraries
  on scavenger nodes that are not managed centrally
• Some 3rd party libraries only available
  dynamically

4
Load Balancing

• Interactive DNS (vs scripts)
• Round robin vs Metric based
• Since cant predict future state
• Sophistication level load, sessions,
• Dealing with load anomalies true interactive
  functions
• Batch
• Over-subscribing CPUs
• Internal blocking VS Context switching
• Queue abuse
• Master Configuration
• Queue length, scheduling complexity

5
Job and Queue Management

• Delegated responsibilities
• Tuning
• Host affinities
• Priorities
• Regulation
• Difficulty of config. and interpret. of
  fair-shares
• Job dispatching
• Forecasting
• Explaining pending status
• Diagnosing and reacting to bad hosts
• Scalability issues due to job volume
• Accounting and dispatching