EVLA Software - Overall Architecture, Science

1
EVLA Software - Overall Architecture, Science
Online Domains

• Bryan Butler
• NRAO

2
NRAO-wide Design
Mostly Telescope-Independent Common Software
Proposal Submission And Handling
Observer Domain
Mostly Telescope-Specific Project Software
EVLA Sched
Telescope Domain
EVLA Control
Telescope Data Model
GBT Postproc
Feedback to telescope
Data Capture
Science Data Model
Telcal
Quick Look
Mostly Telescope-Independent Common Software
Science Domain
Archive
Pipeline
Export Data Format
Offline
VO
3
NRAO-wide Design

• The EVLA software system needs to conform to the
  NRAO-wide design to the extent that it has been
  developed. Unfortunately, the NRAO-wide effort
  has languished for the past two years (this is
  changing - a new e2e Project Manager Nicole
  Radziwill and e2e Project Scientist Ed
  Fomalont have been hired in CV).

4
Domains Models

• The NRAO-wide design introduced the concept of
  domains - different (large) pieces of the
  system that can be grouped sensibly. It
  presented three such domains
• Observer
• Telescope
• Science
• It also presented a number of models -
  descriptions of different (smaller) pieces of the
  system
• Observatory
• Project
• Observing
• Archive
• Telescope Data
• Science Data

5
EVLA Overall Design

• In the spring of 2004, the EVLA software group
  undertook an effort to develop and document a
  high level design (led by Morgan, Ryan, Sowinski,
  and Waters). This culminated in a completed
  design, which was reviewed by the NRAO eOC in
  June 2004. The design presented in the next few
  slides is based mostly on that effort, with
  extension and modification in the past two years.

6
EVLA Software Requirements

• The software design and implementation is driven
  by a number of requirements documents
• e2e Science Software Requirements
• Engineering Software Requirements
• Real-time System Software Requirements
• Operations Software Requirements
• These do not have everything in them (for
  instance Proposal Handling and User Database,
  which are covered in separate less
  requirementy documents), but are fairly
  complete, and notably, the e2e requirements
  document has extensive requirements for PST and
  OPT.

7
Major Elements (Models)

• The main flow of information (and processes the
  workflow or dataflow) is

A Scheduling Block is an atomic unit of
observing. It is made up of a sequence of scans
a scan is made up of source(s), resource(s)
(hardware definition - both FE and BE), timing
information, and a mode. The mode defines the
subscan(s), which are comprised of a single
source, resource, and timing information.
Proposal
Data
Project(s)
Commands
Program(s)
Schedule(s)
8
EVLA High Level Architecture
DATAFLOW
9
EVLA High Level Architecture

• Note that most of the major subsystems have a
  direct counterpart in current VLA software - we
  have a significant amount of experience in what
  is needed there. What has been lacking is the
  electronic storage and passage of information
  between subsystems, and therefore the ability to
  do much of this automatically.

10
EVLA Design Detail, Pre-Observing Science Domain
Proposal Submission Tool (PST)
Proposal
Portal
Proposal Handling Tool (PHT)
Astronomer or Staff
Authenticated Astronomer or Staff
EVLA Observing Heuristics
Project
Observation Preparation Tool (OPT)
Program Block (Set of Scheduling Blocks for one
Program)
To Observation Scheduling Tool
11
EVLA Design Detail, Pre-Observing Online Domain
From OPT
Archive
Observation Scheduling Tool (OST)
Operator
Environment
Heuristics
Metadata to DCAF
Execution State
Next SB
Archive
Executor
Operator
Metadata to DCAF
Results from TelCal
Equipment State
Sequence of Configurations Antenna Delays
To AMCS CMCS
From AMCS CMCS
12
EVLA Design Detail, Real-Time Domain
From Executor
State Counts
FOTS Receiver
Station, Baseline Boards
EVLA Antennas
RF
CBE
Lag Frames
AMCS
FF
Raw Vis
CMCS
Hardware MC
Equipment State, Data Addressing Info, Messages,
Alerts, etc.
To Archive TelCal
To DCAF
To DCAF
13
EVLA Design Detail, Post-Observing Online Domain
Astronomer or Operator
To Executor And Archive
From AMCS CMCS
From CMCS
Portal
TelCal Results
Data Capture And Format (DCAF)
TelCal
Authenticated Astronomer or Operator
SDM
SDM
MC Archive
Observation Status Monitor Tool (OSMT)
Quick Look Pipeline (QLP)
MC Archive
To Archive (?)
To Archive
14
EVLA Design Detail, Post-Observing Science Domain
Astronomer
From DCAF
From CMCS
Open Products
Archive Search and Retrieval Tool (ASRT)
Portal
Cubes (?)
Existing Proprietary Products
Image Cubes
Open Products
Post-Processing
Default Image Pipeline (DIP)
Trigger
Authenticated Astronomer
VO Astronomer
Reprocessed Proprietary Products
15
Detailed Subsystem Designs Prototypes

• For most of the major subsystems, we either have
  a very advanced prototype (Portal, PST, Executor,
  OSMT), an early prototype (PHT, OPT, OST, ASRT,
  TelCal), or at least a much more detailed design
  (DCAF). The areas where we have done little work
  and in fact have only preliminary (high level)
  designs are for the pipelines (QLP, DIP).

16
Portal, User Database, Authentication

• We know that we need some way to restrict access
  to the various tools, and the information
  available within them. We do this with a
  portal, through which users access the various
  tools. It authenticates users, and generates a
  unique token which is then used to verify a
  users login status. We have our own
  implementation of this, but recognize that we may
  need to migrate to the VO method (or have a
  translation layer).

17
Portal, User Database, Authentication
18
Portal, User Database, Authentication

• Users enter their own User Database information,
  except
• Institution information - they can only select an
  institution from a pre-set list (we want to use
  this to automatically generate reports to the
  NSF, which require precise information on
  institutions) - if the institute is not there,
  they indicate so, and we (well, I) add it.
• We allow so-called 3rd party user registrations
  during proposal preparation and submission,
  adding significant complexity (a sore spot with
  us, but demanded by operations).

19
Proposal Submission Tool (PST)

• Mainly a tool to collect form data (web browser)
• Mostly telescope independent, with resources
  the exception
• Used to enforce telescope policy
• Coupled to other EVLA tools only loosely, via
  Portal and databases.
• Currently also supports GBT.

20
PST - Main Processes

• Main Processes
• Model - retrieve and write data to database
• Controller - business logic to map user input
  (from browser) into objects which are then
  written to database
• View - the look-and-feel of the interface (done
  in browser)
• Validation of various fields - an important and
  significant part of the tool
• Help system

21
PST - Model

• The Model drives everything, so whats in there?
• science information - title, category, mode,
  abstract, scientific justification, and some
  misc. info.
• Authors, including which is the PI and contact
  author
• Sources
• Resources (telescope hardware setup)
• Sessions (a guide to SB setup)
• Student Support
• Essentially, everything that is necessary to
• Referee the proposal
• Assign telescope time (and money)
• Automatically generate SBs (mostly for novice
  users, but experienced users will use this too!)

22
PST - The View

• Although the logic is driven by the model, most
  of the programming complexity here is in the
  view. How do we do this? 6 main tabs in the
  browser, to represent the 6 main parts of the
  model. There is also some superstructure, to
  allow for higher level functions (edit/create,
  submit, print, choose telescope, etc.), but,
  again, most of the complexity is in the view for
  these 6 tabs.

23
PST - The View

• Lets look at the actual tool

24
PST - Submit

• We must support both normal deadline
  submissions, as well as Rapid Response
  submissions (this is fundamentally a policy
  issue). Upon submission, the proposal handling
  process is initiated. What is stored is a
  proposal (as represented by a Proposal Data
  Model). This is NOT the Project Data Model, but
  rather is a guide to the creation of the initial
  PDM. This is an important distinction, and
  something we came to a recent agreement on with
  ALMA (for them this is the Science View of the
  PDM).

25
Proposal Handling Tool (PHT)

• Presently, only very initial wrangling (view,
  print), and then splits into GB and VLA specific
  handling
• GB and VLA separately support
• Adding additional data (edit XML by hand or
  script)
• Fixing problems
• Pretty-printing, for referees and scheduling
  committee
• Future
• Referee process
• Scheduling committee details
• Points of interest
• Requirements for VLA are in hand, but not in the
  form of the detailed requirements for other
  areas, rather more like a User Story. We need
  to transform this to real requirements, including
  the GB process (which have not been written down
  to our knowledge).
• Does this go in the PST (with maybe part in the
  OPT) or as a separate tool?
• The fundamental output from the PHT is Projects,
  as represented by a Project Data Model.

26
Observation Preparation Tool (OPT)

• This is the process that takes a more generic
  view of a Project, and turns it into something
  that can actually be used to command the hardware
  of the telescope (and run ancillary tasks).
• The fundamental output of the OPT is Program
  Blocks (PBs) (remember that a PB is a collection
  of SBs, with some extra information - mostly
  contingencies)
• It needs to support 3 levels of user
• Novice (automatic generation of PBs for standard
  modes)
• Intermediate (this is the tough one!)
• Expert (allow for script level editing)

27
OPT - commonality with PST

• Objects in common with PST
• Sources
• Resources (hardware configuration)
• Things not in common
• Things not in OPT
• Front page stuff
• Authors
• Sessions (well, kind of - since they represent
  SBs)
• Student Support
• Things not in PST
• Scan builder and organizer
• PB organizer
• Detailed correlator setup tool
• Calibrator tool

28
OPT - Detailed Design
29
OPT - Detailed Design
Modify PB
30
OPT - Detailed Design
Modify SB
31
OPT - Current Status

• We currently have an early prototype of the GUI
  (duplicating the look-and-feel of the PST) in
  place which will authenticate users and has the
  most minimal PB functions (create, delete, etc.).
• Much of our work here, however, has been on the
  specification of the details of the objects (the
  data models) for the various parts of the
  system. We are starting here, knowing that many
  of the elements in the system will be needed here
  and will be common. These include the
  definitions of Proposal, Project, Program Block,
  and Scheduling Block, and as parts of that,
  Sources, Resources, and Scans. We start out with
  an XML document provided by a domain expert, and
  then turn that into objects. We are currently
  having detailed discussions with ALMA to attempt
  to have as much as common in these objects. It
  is clear that early parts of the process
  (Proposal) can be common, and that general
  concepts (major elements of SBs, for instance)
  can be common it is not yet clear the level to
  which true commonality will be achieved.

32
OPT - Plan

• Our plan is to have new major functionality
  available within the OPT roughly every 3 months.
  Major milestones are
• 30Apr06 framework with minimal functionality
• 31Jul06 Add VLA calibrator database access,
  initial spectral setup
• 31Oct06 Full calibrator setup, more observation
  setup
• 31Jan07 VLA mostly supported. Some
  validation/PB creation
• 30Apr07 Beginning prototype WIDAR support
• 31Jul07 VLA fully supported prototype WIDAR
  mostly supported
• 31Oct07 Prototype WIDAR fully supported

33
Observation Scheduling Tool (OST)

• We (well, Barry) have been conducting tests of
  dynamic scheduling with the VLA during the past 2
  (3?) reconfigurations. Again, we consider these
  as prototypes for the final EVLA subsystem -
  giving us invaluable information on the practical
  aspects of dynamic scheduling of a many-element
  radio interferometer.

34
OST - Block Diagram
35
OST - Lessons Learned

• Here are the lessons learned - I need the full
  list from Barry. A few things I know
• ALMA system didnt work well (as expected,
• per Allen Farris email)
• System is inordinately fond of short SBs
• Currently effort-intensive (but getting better)

36
OST - Plan

• Here is the plan. I need to get with Barry on
  this, but my understanding is that he wants the
  VLA fully dynamically scheduled by the end of 06
  (yes, he has indeed said that!).
• For the EVLA-specific part, we are assigning some
  effort beginning summer 06 to this.