Vicky Rowley

1
E-x-t-e-n-d-i-n-g Rocks The Creation and
Management of Grid Systems for Biomedical
ResearchOSGC Conference - May 14, 2008

• Vicky Rowley
• Solution Architect
• BIRN Coordinating Center - University of
  California San Diego

2

3
BIRN is Data Storage (SRB) and Processing, but
UMN
MGH
BWH
Stanford
Yale
Cluster
UI
UCLA
UCI
Duke
UNM
Cluster
UCSD
4
Rocks Standard vs. Rocks for BIRN

• Cluster building focus
• Data processing focus
• Lots of big clusters

• Collaboration focus
• Data storage/sharing focus
• A few relatively small clusters
• Data Grid was needed before clusters processing
  was needed

5
So what does BIRN _do_?
Large Deformation Diffeomorphic Metric Mapping
using the TeraGrid
4
JHU Shape Analysis of Segmented Structures
3
MGH Segmentation
5
BWH Visualization
Large Scale Distributed Computing
1
BIRN Data Grid
Data Donor Site (WashU)

• Preliminary Study
• 46 hippocampus data sets
• 30,000 CPU hours, 4 TB data

Scientific Goal classify patient status from
morphometric results
N45
De-identification And upload
2
6
SASHA Shape Analysis Pipeline Results
Shape-derived metrics can be used to detect
class-specific information
6 semantic dementia subjects
18 Alzheimer subjects
21 control subjects
7
The BIRN Collaboratory Today
Enabling collaborative research at 28 research
institutions comprised of 37 research groups.
8
How does Rocks make it do that?

• Installs operating system software
• Turns individual servers into a Grid
• Portals web servers
• Data grid for access management
• Compute clusters
• Database servers
• Distributes, installs and updates 3rd party,
  domain-specific scientific software packages
• Updates system software

9
What would be better?

• Add/Improve security performance monitoring
• Detect and capture configuration changes
• Track versions
• Ideally, reduce, reuse, recycle

10
Desired System Qualities

• Agile
• Fast response with updates
• Self-help for developers
• Repeatable
• Tracking of versions
• Tracking of deployments
• Modular/Flexible
• Handles unique site requirements
• Handles unique project requirements
• Customizable
• Scalable
• Highly automated
• Supports addition of several sites per year, plus
  additional projects over 5 years

• Basic System Software
• Operating System
• Security

BIRN/Rocks Software Stack

• Server Definition Software
• Apache/Tomcat
• Globus

• Application Software
• Gridsphere
• HID
• Mediator
• Scientific Applications

Custom BIRN Server
11
Whats involved in a single grid?
Rocks Central YUM Rolls RHEL4
area51 base birn birnafs
birncondor birnportal birnsrb
CentOS condor cvsserver
freesurfer gama-naregi 1.0
gama-naregi 4.1 ganglia grid
gridsphere hardwareutils hid
hpc java kernel mediator
nagios oracle postgres sciapps
sge srb34 tomcat
updates-CentOS webserver
A BIRN Grid
14 Rocks Rolls (-2 for OS) 17 Custom Rolls
Portal/Web
BIRN Rack GPOP GComp Nettools NAS
HID
Mediator
CVS, SVN SRB Repos
DB Server Registry DB UMLS HID DB
GAMA Server MyProxy Globus CAS
Testbeds
MCAT
12
Software Development Integration
13
Software Deployment

• Staging Area
• Verify interoperation of latest code
• Support demonstration of latest development
  efforts without disruption to production
• Functional system/Beta Testing

• Development Area
• Integrate software for many diverse sources
• Version control at system and sub-component
  levels
• Rolling baseline
• Integration and Functional Testing

• Production Area
• Stable
• - Reliable
• Facilitates research

Rocks Development Server
Rocks Staging Server
Rocks Production Server
CVS
SRB
14
What we love?

• Repeatability
• All the web servers are the same
• All the database servers are the same
• Flexibility
• Mix Match rolls
• Level of automation
• Experienced person can kick a server in 5
  minutes
• IPs, hostnames, software configuration done
• Open Source
• Result Not one grid - Many! Not one project -
  Many!

15
What drives us nuts?

• Turn around time for updates
• Steep learning curve
• RPM building not standard
• Build time large
• Software developers are not co-located with
  integrators
• Reinstalling to get updates is not an option
• Lack of advanced roll development training

16
More info?

• See the project website http//www.nbirn.net
• Email vrowley_at_ucsd_dot_edu