Commodity Workflows for the Grid: An Evaluation of Microsoft Windows Workflow Foundation

1
Commodity Workflows for the Grid An Evaluation
of Microsoft Windows Workflow Foundation

• Quinn S. Lewis
• Grid Computing (COSC6376)
• 14 December 2006

2
Perceived Problem Grid Usability

• If a grid interface or the working environment is
  (apparently) too difficult to use, it will not be
  adopted by many of the appropriate users
• Most scientists, engineers, and researchers will
  be (service) end users
• Currently grids are serving power users and do
  not yet benefit non-computer-technical
  researchers

3
One Approach

• Improve the usability of the tools used to
  interact with the grid
• Should be natural and easy-to-use
• Take advantage of other familiar commodity tools
  and technologies
• Windows Workflow Foundation
• Free
• Platform level component
• Used by other (Microsoft) tools

4
Workflows Defined

• Workflow is a set of activitiesthat describe the
  implementation of a process
• Workflow can be short or long running and can
  involve people and systems

5
Workflows in Technical Computing
ComputationalModeling
Real-world Data
Persistent Distributed Data
Workflow, Data Mining Algorithms
Interpretation Insight
6
Pillars for Connected System
7
Connected Systems Architecture Model
8
Similarity to OGSA
9
Workflow Foundation Architecture
10
Work Performed

• Attempt to use Windows Workflow to connect to
  grid middleware
• WSRF.NET/OGSI.NET
• Globus
• Condor
• Explore re-hosting of
  Workflow Designer
  outside of VisualStudio

11
WSRF.NET/OGSI.NET

• From the University of Virginia
• Use Microsoft .NET to implement WSRF and OGSI
• Takes advantage of .NET attributes
• Uses VisualStudio Add-In to generate stubs
  (client/server) to handle state
• Code generation tool didnt seem to work for web
  services on ports other than 80
• Requires WSE 3.0

12
Condor

• Exposes job scheduling capability as web services
  to enable
• execution of complex computational workflows
• consisting of a number of inter-related jobs
• Ported SOAPScheddApiHelper to C

13
Condor Text to Graphics

• In code
• TransactionAndStatus tx schedd.beginTransaction
  (60)
• IntAndStatus clust schedd.newCluster(tx.transac
  tion)
• IntAndStatus job schedd.newJob(tx.transaction,
• clust.integer)
• ClassAdStructAndStatus cas schedd.createJobTemp
  late(
• clust.integer, job.integer,
• owner, type, cmd, args, reqs)
• Status st schedd.commitTransaction(tx.transacti
  on)
• RequirementsAndStatus reqs_s schedd.submit(
• tx.transaction, clust.integer, job.integer,
• cas.classAd)

• With graphics

14
Unsecure Condor Setup

• Modify condor_config and run condor_restart
• ENABLE_SOAPTRUE
• ALLOW_SOAP
• HOSTALLOW_WRITE //or may get Permission
  denied from schedd.beginTransaction(30)
• QUEUE_ALL_USERS_TRUSTEDTRUE //see next slide
• Collector web services runs on port 9618
• http//ltmachinegt9618 (couldnt use localhost)
• Check CollectorLog for more info
• Schedd runs on random port
• Check \log\.schedd_address for port
• Start explicitly in condor_config or via
  collector web service (SCHEDD_ARGS-p 1134)
  Reference

15
Condor Problem

• Get "Could not create new cluster" when trying to
  IntAndStatus cluster schedd.newCluster(xact.tra
  nsaction)
• SchedLog says QMGT command failed anonymous
  user not permitted
• Undocumented answer is to add QUEUE_ALL_USERS_TRU
  STEDTRUE to condor_config since no security is
  being used

16
Another Condor Problem

• Nothing seems to execute but it is submitted and
  get strange timer errors
• ShadowLog
• 12/9 162847 (24.0) (8000) ERROR Could not
  locate valid credential for user
  'Quinn_at_COPERNICUS'
• 12/9 162847 (24.0) (8000) init_user_ids()
  failed!
• 12/9 162847 (24.0) (8000) ERROR Could not
  locate valid credential for user
  'Quinn_at_COPERNICUS'
• 12/9 162847 (24.0) (8000) init_user_ids()
  failed!
• 12/9 162847 (24.0) (8000) ERROR
  "set_user_priv() failed!" at line 517 in file
  ..\src\condor_c_util\uids.C
• SchedLog
• 12/9 162543 (pid4044) my_popen CreateProcess
  failed
• 12/9 162543 (pid4044) Failed to execute
  C\condor/bin/condor_shadow.pvm.exe, ignoring
• 12/9 162543 (pid4044) my_popen CreateProcess
  failed
• 12/9 162543 (pid4044) Failed to execute
  C\condor/bin/condor_shadow.std.exe, ignoring
• RESOLUTION. Use condor_store_cred! (Reference)
• Dont use condor_credd if you want something easy
• Will need to set a password even if you dont
  normally use one (net user Quinn ltpasswordgt)
  http//www.kellys-korner-xp.com/win_xp_passwords.h
  tm

17
Globus

• Cant consume GT4 web services from .NET
• Tried manipulating WSDL
• Discovered Microsoft and Globus didnt use some
  WS-Addressing mechanism
• Thought to use Axis and found out a C version
  exists
• Complicated install and language mixing issues

18
Closer to home UH Work

• Computational Data Grid for Scientific and
  Biomedical Applications
• Marc Garbey and Victoria Hilford
• Rapid application development (use of the
  Microsoft Visual Studio .NET technology)
• Visual modeling interfaces (forms driven
  Graphical User Interfaces)
• Database Connectivity (interface with Microsoft
  SQL Server 2005)
• Query support (clients can store, update, delete,
  retrieve database metadata)
• Context-sensitive, role-based access (Microsoft
  Windows Server 2003, ASP.NET)
• Robust security (HIPPA compliance through
  Microsofts Authentication and Authorization from
  IIS and ASP.NET)
• Connectivity to other biomedical resources (PACS,
  DICOM, XML)

19
References

• http//ws.apache.org/axis/cpp/index.html
• http//www.windowsclusters.org/condor_guide.htm
• Condor WS not possible w/ 6.8.0 and helper?
  https//lists.cs.wisc.edu/archive/condor-users/200
  6-August/msg00350.shtml