Sphinx Server

1
Current Sphinx Client/Server
Sphinx Server
Sphinx Client
Data Warehouse
Message Interface
Message Interface
Multi-threaded Control Process
DAG Reducer
Prediction Engine
Submitter
Planner
Generic Grid Site
Resource
Monitoring Service
Information Gatherer
2
Sphinx Client/Server with V2 Components
Sphinx Server
Sphinx Client
Data Warehouse
Message Interface
Message Interface
DAG Reducer
Multi-threaded Control Process
Prediction Engine
Submitter
Admission Control
Planner
Generic Grid Site
Resource
Data Management
Monitoring Service
Information Gatherer
3
Stand alone Sphinx Server with light Client
Sphinx Server
Sphinx Client
Data Warehouse
Message Interface
Message Interface
DAG Reducer
Multi-threaded Control Process
Prediction Engine
Admission Control
Planner
Generic Grid Site
Resource
Submitter
Data Management
Monitoring Service
Information Gatherer
4
Data Warehouse
Sphinx Server

• Message Tables
• Policy Information Tables
• Entity Accounting Tables
• Work (DAG, Job, etc) Tables
• State
• Unreduced
• Unpredicted
• Unaccepted (not admitted)
• Unplanned (not yet feasible)
• Unsent (assigned)
• Unfinished (executing)
• Remove (accounted or rejected)
• Entity
• Resource Property Requirements
• Dependency Requirements
• Do not start before event
• I/O
• QoS Requests
• Resource Assignment for next k steps

Data Warehouse
Message Interface
DAG Reducer
Multi-threaded Control Process
Prediction Engine
Admission Control
Planner
Submitter
Data Management
Information Gatherer

• What about including Virtual Data Tables?

5
Message Interface Component
Sphinx Server

• Input (incoming)
• DAG
• Requirements
• QoS requests
• Status requests
• Output (incoming)
• DAG Table (unreduced)
• Job Table (unreduced)
• application
• Requirements
• QoS
• Input (outgoing)
• Message
• Output (outgoing)
• Status updates

Data Warehouse
Message Interface
DAG Reducer
Multi-threaded Control Process
Prediction Engine
Admission Control
Planner
Submitter
Data Management
Information Gatherer
6
DAG Reducer (Decomposer?)
Sphinx Server

• Input
• DAG (unreduced)
• Available (existing) data
• Action
• Remove DAG nodes for which input data exists
• Add Jobs to Job table
• Output
• Pruned DAG (unpredicted)
• Jobs (unpredicted)

Data Warehouse
Message Interface
DAG Reducer
Multi-threaded Control Process
Prediction Engine
Admission Control
Planner
Submitter
Data Management
Information Gatherer
7
Prediction Engine Component
Sphinx Server

• Input
• unpredicted Job
• Requirements (if available)
• CPU hours
• Storage hours
• Bandwidth
• Profile/History
• Output
• Estimated Resource Usage
• unaccepted job

Data Warehouse
Message Interface
DAG Reducer
Multi-threaded Control Process
Prediction Engine
Admission Control
Planner
Submitter
Data Management
Information Gatherer
8
Admission Control Component
Sphinx Server

• Input
• Entity Account Information
• unaccepted DAG Request
• Requirements
• QoS
• Predictions of Jobs (decomposed DAG) Resource
  Usages
• Policy Constraints
• Current schedule for next k steps
• Output
• Reject DAG Request
• Remove DAG
• Remove Jobs
• Accept DAG with Estimated QoS
• Unplanned DAG
• Unplanned Jobs

Data Warehouse
Message Interface
DAG Reducer
Multi-threaded Control Process
Prediction Engine
Admission Control
Planner
Submitter
Data Management
Information Gatherer
9
Planning Component
Sphinx Server

• Job Planner (independent work that can be fully
  resolved onto resources at time t)
• Input
• unplanned Job Information
• Requirements
• Dependencies (I/O)
• Job QoS Requests
• Entity Account Information
• Policy Constraints
• Available Data
• Available Applications
• Available Resource Properties
• Prediction of Job Resource Usage
• Output
• QoS Deliverable
• unsent Job(s) schedule for the next k steps
• DAG Planner (intra-dependent work that can not
  all be fully resolved onto resources at time t)
• Input
• unplanned DAG Information
• Requirements

Data Warehouse
Message Interface
DAG Reducer
Multi-threaded Control Process
Prediction Engine
Admission Control
Planner
Submitter
Data Management
Information Gatherer
10
Job Submission Component (also part of Sphinx
Client)
Sphinx Server

• Input
• Fully planned Job
• Job state unsent
• Assigned gatekeeper and jobmanager
• I/O data movement locations
• Action
• Construct Job DAG
• Set up environment
• Import input data (if necessary)
• Publish to DMC (if necessary)
• Run application
• Export output data (if necessary)
• Publish to DMC (if necessary)
• Cleanup
• environment
• Local I/O data (if necessary)
• Submit to Condor-G/DAGMan
• Job state unfinished
• Monitor Job statistics

Data Warehouse
Message Interface
DAG Reducer
Multi-threaded Control Process
Prediction Engine
Admission Control
Planner
Submitter
Data Management
Information Gatherer
11
Data Management Components
Sphinx Server

• User Demanded Movement Module
• Input
• Source(s)
• Destination(s)
• Output
• DMC update
• Pre-Planned Movement Module (most important to
  Sphinx)
• Input
• Available Data
• Available Resources
• Prediction of Data Access Patterns
• DAG (reduced)
• Jobs (unsent)
• Job(s) schedule for next k steps
• Output
• Data Movement
• DMC update
• Prediction Module
• Input

Data Warehouse
Message Interface
DAG Reducer
Multi-threaded Control Process
Prediction Engine
Admission Control
Planner
Submitter
Data Management
Information Gatherer
12
Information Gathering Module
Sphinx Server

• Resource Property Module
• Input
• Grid Information Systems (such as MDS using GLUE
  Schema, etc)
• Output
• Resource Property views
• Job Tracking Module
• Input
• Job Monitoring information (such as Condor-G,
  etc)
• Output
• Work Views
• Grid Weather Module
• Input
• Grid Monitoring systems (such as GEMS, MonALISA,
  etc)
• Output
• Grid Weather Views
• CPU
• Storage
• Bandwidth
• Grid Weather Prediction Module

Data Warehouse
Message Interface
DAG Reducer
Multi-threaded Control Process
Prediction Engine
Admission Control
Planner
Submitter
Data Management
Information Gatherer