Developments in MCRunjob

1
Developments in MCRunjob

• Greg Graham
• CD/CMS 25-July-2003

2
MCRunjob
MCRunjob uses a modular component based
architecture. Schema Modeling Each application
or task has its own schema representation. ScriptG
enerationEach target script environment has its
own generator. External Services Each service or
interface is described internally by its schema.
3
A user who wants to run applications A,B, and
C attaches corresponding Configurators to a
Linker. The Linker verifies that dependencies
are satisfied. Once attached, the user sets
values for the various schema elements defined
in each configurator, and defines filename
rules, random seed rules, etc. The user then
executes the framework. Each Configurator may
generate scripts used to run the corresponding
application. The scripts are collected by
a ScriptGen object.
4
5 New Developments

• Shahkar project is underway.
• Tree representation of McRunjob macro scripts -
  full recursion of if/repeat blocks
• Standalone version of production tools
• Configurator contexts
• GUI for application workflow modelling

5
1. Shahkar project

• Eric Wicklund is the lead developer on Shahkar.
• Also Greg Graham, Anzar Afaq, Dave Evans, and
  Peter Love
• Shahkar elements
• BaseLinker, BaseConfigurator, Common interfaces,
  Event Logger, Generic Exceptions
• RPM distribution recently made of Shahkar
• Plan to start integration with CMS McRunjob next
  week.

6
Advantages of Shahkar

• DZero and CMS can more easily exchange technology
• CMS is getting macro script preprocessing
• DZero is getting scriptObjects, framework calls
• We both get access to each others library of
  utility Configurators
• We both benefit from cleaner code and
  documentation in Shahkar as it passes from both
  projects into the common repository.
• Anzar has created a web page...

7
2. Tree Representation of Macros

• Macro scripts now represented internally as trees
  called CommandTrees.
• CommandTrees can be manipulated internally by a
  preprocessor to generate other CommandTrees.
• Abstract planning idea from DZero
• CommandTrees can support full recursion
• Callout to Linker for evaluation of conditionals

8
Macro Script
Shahkar Proposed Command Processing Schematic
Script Reader
Command Processor
Command Tree
Abstract/ External Planning
Next Command in Tree
Interface Customizations
Callout 1
Preprocess
Handler 1
Callout 2
Command Dispatcher
Handler 2
Callout N
Handler N
List of Command Trees
Goes Into
Produces
Calls out to
9
3. Standalone Operation

• Recently, you needed an AssignmentID from CERN to
  run the production tools
• ImpalaLiteStandalone Configurator will mimic a
  real production request in two ways
• The user inputs a from AssignmentID and
  duplicates a previous assignment locally
• The user provides his own Cardfile and teplate
  script to run.
• Soon a Rogue Configurator to allow user to run
  any application.

10
4. Contexts(because Reality is more complex than
just the applications...)

• In the current MCRunjob, the user is expected to
  add all of the configurators and model all
  dependencies.
• While this is fine for the production experts, it
  is difficult for the users who only know/care
  about the applications.
• Eg. CMKIN, CMSIM, not InputPluginRefDB, not
  VDLScriptGen, etc.

11
Sample workflow diagram using ImpalaLite
InputPlugin RefDB
JobSplit Table
InputPlugin RefDB
JobSplit Table
Context
CMSIM
CMKIN
Applications
ImpalaLite ScriptGen
Context
Too complicated!!!
Depends On
12
New Approaches to Workflow Design

• Recognize elements needed in the complete
  workflow often have nothing to do with
  application
• contact Reference Database, contact replica
  catalog, specification of submission service
• We need a mechanism to separate the application
  part of the workflow from the context

13
Applications
CMS Production Context
CMKIN
InputPlugin RefDB
JobSplit Table
?
CMSIM
Application
?
?
ImpalaLite ScriptGen
CMSIM
CMKIN
Depends On
Etc...
Substitue For
14
The Mechanism is Context.What goes into a
context?

• Contexts consist of
• Pre-added configurators
• Rules for what happens when a user adds a
  configurator on his own
• Dependencies are set up
• Metadata is mapped
• Contexts are written by the experts
• but selected by the user currently. This could
  also be automated.

15
ImpalaLite Standalone Complete Context
ImpalaLite/MOP Standalone Complete Context
ImpalaLite/VDL Standalone Complete Context
ImpalaLite Standalone
ImpalaLite Standalone
ImpalaLite Standalone
InputPlugin RefDB
JobSplit Table
InputPlugin RefDB
JobSplit Table
InputPlugin RefDB
JobSplit Table
Application
Application
Application
MOP DagGen
VDLGen
ImpalaLite ScriptGen
ImpalaLite ScriptGen
ImpalaLite ScriptGen
Depends On
16
The User is Happier...

• The User only has to worry about the application
  side of things.
• Caveat 1 The User still has to know what context
  to choose.
• But this can be standardized at each center
• Or, a planner like Sphinx can choose context
• Caveat 2 Contexts are still new in McRunjob
• Need to formalize the algebra of combining
  context vs. application-side workflows
• Need to have some concept of combining or
  layering contexts

17
5. GUI for MCRunjob

• The GUI is written in Java
• User can choose a context
• User can choose Configurators to put into the
  application space
• The GUI generates an MCRunjob macro which the
  user can then run.
• GUI will be extended to
• Author contexts or combine pre-existing contexts
• Demonstration

18
Conclusion

• Future Plans
• CMS OSCAR and ORCA cfgrtrs due next week (Julia)
• Full context-driven operation in MOP, VDL, EDG
  environments within 3 weeks.
• Finalize standalone operation within 3 weeks
• Hans cardfile repository
• Rogue Configurator
• Users submitting jobs to Grid3 by SC2003?
• Finalize GUI prototype and release to users for
  comment for the Fall.