SNS Infrastructure

1
SNS UpdateDANSE Workshop
Steve Miller September 20-21, 2004
2
(No Transcript)
3
SNS Software Plans

• Phase 1 initiate framework
• Currently underway
• Phase 2 Day 1 instrument support
• DANSE for ARCS
• DAVE for Backscatter Spectrometer
• ISAW for visualization
• Phase 3 NeSSI SW on-line
• Keep this development out of critical path

4
Some Plan Considerations

• Phase 1
• Portal performance and responsiveness both
  within facility and external
• Evaluate different portal types currently
  evaluating window server implementation
• Phase 2
• Can DANSE be ready for Day 1 at SNS?
• Is it necessary for DANSE to be ready on Day 1?
• Where to host reflectometer software for Day 1?
• Soon afterward need SANS and Diffractometer SW
• Phase 3
• How long to get here?

5
Current Events

• NeSSI Workshop just took place - sponsored by
  ISIS
• Collaboration holding regular video conferences
  how to better include DANSE team?
• E-notebooks setup for NeSSI and task groups
• Currently updating Functional Requirements
  document utilizing interfaces diagram
• Recent meeting at ORNL between SNS, HFIR, and
  NIST discussing 3-way DAVE software
  collaboration
• SNS-ORNL working with IPNS team to integrate ISAW
  tools within the data portal
• Leveraging existing software where it makes sense

6
NeSSI Collaboration and Workshop

• Participating Organizations
• ANSTO, Caltech, CCLRC e-Science, HFIR, IPNS,
  ISIS, J-PARC/KEK/KENS, LANL, NIST, ORNL, SNS
• Task Groups
• Access and Portals
• Visualization
• Analysis
• Data Management
• NeSSI gaining momentum

7
SNS Software Architecture Responsibilities
real time control
Intelligent Control
Automation
High Performance Computing
Diagnostics
Controls
simulation
analysis
Instrument Simulation
Materials simulation
Acquisition
Treatment
Instrument
Analysis
Sample Environment
Vis
Vis
Vis
Vis
Vis
acquisition
visualization
data
Electronic notebook
Instrument simulation
Materials simulation
Raw
Intermediate
Scientific
Database
Database
Raw
Intermediate
Scientific
Notebook
Sample environment
Simulation
Simulation
Proposal
Software repository
Publications
Documentation
Data
User Community
Web Portal to Data, Software, and Analysis
Metadata
SNS Directed
Responsibilities
Modules
Local and remote users
SNS Responsible
8
Facilities Play Bigger Role Interfaces provide
extensibility, integration, multiple frameworks
Facilities have no control over SW users run on
their own computers Using data downloaded from SNS
data
Visualization
Analysis
Minimum requirement
Users
Desktop client
User Application
Web browser
SW defines interfaces through which user desktop
programs can access facility resources
Users and SNS write components to standard well
defined interfaces
Security Interface
Access and authorization control
Function Interface
Control portal
Data portal
Analysis portal
Acquisition
SNS Validated software
New User code
Legacy code
Commercial packages
Data Management
Data Interface
Database and flat file access protocols
Computer interface
Meta data
Documentation
data
Analysis results
data
Publications
9
Some Portal Considerations

• Pros
• Offers facility single point of control for user
  software
• Enables control of user access
• Can provide user transparent access to high
  performance networking, data, and computing
• Facilities can provide users SW licenses and
  workspace
• Sending screen images rather than data
• Cons
• Requires network access
• Remote access performance adequate?
• Adequately host high end viz software?
• Can same software running via portal run via
  desktop?
• Sending screen images rather than data

10
Task Group Areas Bringing Structure to
Interfaces
data
Users Own Computers
Visualization
Analysis
Users
Desktop client
Web browser
User Application
Security Interface
Access and authorization control
Function Interface
Control portal
Data portal
Analysis portal
Acquisition
SNS Validated software
New User code
Legacy code
Commercial packages
Data Management
Data Interface
Database and flat file access protocols
Computer interface
Meta data
Documentation
data
Analysis results
data
Publications
11
Access and Portals Workshop Summary

• Portals
• Start with web portal interaction with
  functional interfaces
• Define portal interfaces to data management and
  analysis/visualization layers
• Software repository to allow download of
  released SW versions
• Access
• Determine access use cases who gets access to
  what
• Draft a policy document
• Determine methods for implementing access
  control
• Outline necessities for single-sign-on across
  facilities
• Grid access eScience, TeraGrid, federated grid

12
Visualization Workshop Summary

• Visualization to be controllable through a user
  interface or a script and specified through a
  public API
• The visualization routines need to have input via
  either file or a pointer to memory
• The controls (GUI or script) shall be separate
  from the data and the view(s) of the data
• It will be possible to use some analysis routines
  to operate on the data before passing it to the
  viewer(s)
• Needs to be easy for users to develop viz tools
• New consideration interactive visualization may
  be a tool to spawn batch processing resulting in
  new (3D or 4D) visualizations

13
Analysis Workshop Summary

• Very large data sets driving development
• Open Source and Commercial SW
• Structured software needed.
• Software repository what to put in.
• Automation and Intelligent Control.
• Interfaces to data and visualization.
• Analysis framework definition task group formed

14
Data Management Workshop Summary

• A number of key databases have been identified
• User
• Proposals
• Sample safety
• Sample environment
• Scheduling
• Metadata (icat/scat)
• Publications
• Data
• There is a pool of metadata and data available
  to populate these databases
• Expressing relationships between various data
  types (e.g. RAW, processed, published) is key to
  preserving the analysis chain

15
Where DANSE sits today
DANSE presently runs on users local sites using
data and analysis modules located there
data
Visualization
Analysis
Desktop client
Web browser
User Application
Security Interface
Access and authorization control
Function Interface
Control portal
Data portal
Analysis portal
Acquisition
UNDER CONSTRUCTION
SNS Validated software
New User code
Legacy code
Commercial packages
Data Management
Data Interface
Database and flat file access protocols
Computer interface
Meta data
Documentation
data
Analysis results
data
Publications
16
DANSE as a Portal Client
DANSE as a user application accesses data via the
data portal interface
data
Visualization
Analysis
Possibly help support ARCS instrument integration
remotely
Desktop client
Web browser
User Application
Security Interface
Access and authorization control
Function Interface
Control portal
Data portal
Analysis portal
Acquisition
SNS Validated software
New User code
Legacy code
Commercial packages
Data Management
Data Interface
Database and flat file access protocols
Computer interface
Meta data
Documentation
data
Analysis results
data
Publications
17
DANSE as a Portal Server
data
Visualization
User accesses DANSE via web portal, desktop
client or user app
Analysis
Desktop client
Web browser
User Application
Security Interface
Access and authorization control
Function Interface
Control portal
Data portal
Analysis portal
Acquisition
SNS Validated software
New User code
Legacy code
Commercial packages
Data Management
Data Interface
Database and flat file access protocols
Computer interface
Meta data
Documentation
data
Analysis results
data
Publications
18
DANSE Fully Integrated
1. DANSE runs remotely accessing resources at SNS
data
Visualization
Analysis
Desktop client
Web browser
User Application
2. DANSE fully integrated utilizing local/remote
resources
Security Interface
Access and authorization control
Function Interface
Control portal
Data portal
Analysis portal
Acquisition
SNS Validated software
New User code
Legacy code
Commercial packages
Data Management
Data Interface
Database and flat file access protocols
Computer interface
Meta data
Documentation
data
Analysis results
data
Publications
19
Using the Portal
Web Browser Interface

• Using WiredX
• Portal can run either in a browser or as a
  rootless window on client machine.
• Portal sends screen
• Images rather than data

20
Using the Portal Selecting Data
Selecting Input
User clicks on Select Input button
21
Using the Portal Launching DANSE
User clicks on DANSE button
22
Using the Portal More Applications
ISAW
Matlab
Launch more applications such as
DAVE
23
Using the Portal Metadata Search
Here RAW Metadata searched by owner
24
Visualization Components
Data Sources URL, ptr or file
Local or remote
Data Reader
Data Manipulation (local analysis)
Visualization Engine
Viewer 1 Viewer N
Renderer
UI Control
To Analysis Components
User Interface
25
Analysis Components
Data Sources URL, ptr or file
Analysis Engine
Local or remote computation
Data Writer
Data Reader
Analysis Performer
UI Control sequences performing analysis
UI Control
26
Considerations

• Interface to NeXus files and data management
  layer
• Interface to visualization support multiple
  packages
• Ability to support data provenance and pedigree
  supply metadata to database records
• Growing need to support batch processing
• How to leverage existing software such as
  Netsolve, ISAW, DAVE?
• Visual programming interface existing options
  (CCA, SCIRUN, Caffeine, Kepler, IE, )?
• Documentation architecture, interfaces, design,
  etc.
• Incorporation of DANSE into NeSSI framework via
  interfaces
• Simple, rapid installation wizard
• Legacy code incorporation wizard
• ARCS instrument status of this SW?
• SW transition plans to SNS
• Collaborative development with SNS and with NeSSI
  how to manage?
• Any licensing or software ownership issues?
• IMPACT Software incorporating new science via
  software perhaps the biggest contribution the
  DANSE project can make

27
SNS Considerations

• Minimize Resources personnel, costs, variety of
  software to maintain, etc.
• Independently change code based upon user
  feedback especially GUI applications
• Must see the big picture for SW development
• Design criteria contention user vs developer
• Support SNS Instrument Scientists
• Support NeSSI collaborations
• DANSE Integration

28
Facility Software Musts

• Software must be
• Reliable
• Easy to use, but capable
• Accessible via web or desktop
• Documented design, references, guides
• Tested with feedback paths for improvements
• Easy to train people to use the software
• low maintenance
• Scaleable and Upgradeable
• High performance where needed
• Low cost
• Facility must have means for self-sufficiency

29
NeSSI ? Culture Changes

• Mutual trust among collaborators
• Designs must be inclusive
• Leverage collaborator strengths
• Sensitive to timelines
• Common vision and goals
• Regular communication
• Stakeholders share risks equitably
• Software suite defined by natural selection
• More anonymity among facility contributions
• Teams must remain responsible to their facilities

30
Summary

• Its a good time to align DANSE framework
  development efforts within the SNS/NeSSI
  framework
• Lets work together in conjunction with NeSSI