High-Performance Computing, Computational Science, and NeuroInformatics Research

1
High-Performance Computing, Computational
Science, and NeuroInformatics Research

• Allen D. Malony
• Department of Computer and Information Science
• NeuroInformatics Center (NIC)
• Computational Science Institute
• University of Oregon

2
Outline

• High-performance computing research
• Interactions and funding
• Project areas
• TAU parallel performance system
• Computational science at UO
• Projects
• Computational Science Institute
• Neuroinformatics research
• NeuroInformatics Center (NIC)
• ICONIC Grid

3
High-Performance Computing Research

• Strong associations with DOE national
  laboratories
• Los Alamos National Lab
• Lawrence Livermore National Lab
• Sandia National Lab (Livermore)
• Argonne National Lab
• National Energy Research Supercomputing Center
• DOE funding
• Office of Science, Advance Scientific Computing
• ASCI/NNSA
• NSF funding
• Academic Research Infrastructure
• Major Research Instrumentation

4
Project Areas

• Parallel performance evaluation and tools
• Parallel language systems
• Tools for parallel system and software
  interaction
• Source code analysis
• Parallel component software
• Computational services
• Grid computing
• Parallel modeling and simulation
• Scientific problem solving environments

5
TAU Parallel Performance System

• Allen D. Malony Sameer S. Shende
• Department of Computer and Information Science
• Computational Science Institute
• University of Oregon

6
Parallel Performance Research

• Tools for performance problem solving
• Empirical-based performance optimization process

PerformanceTuning
hypotheses
Performance Diagnosis
properties
Performance Experimentation
characterization
Performance Observation
7
Complexity Challenges for Performance Tools

• Computing system environment complexity
• Observation integration and optimization
• Access, accuracy, and granularity constraints
• Diverse/specialized observation
  capabilities/technology
• Restricted modes limit performance problem
  solving
• Sophisticated software development environments
• Programming paradigms and performance models
• Performance data mapping to software abstractions
• Uniformity of performance abstraction across
  platforms
• Rich observation capabilities and flexible
  configuration
• Common performance problem solving methods

8
General Problems

• How do we create robust and ubiquitous
  performance technology for the analysis and
  tuning of parallel and distributed software and
  systems in the presence of (evolving) complexity
  challenges?
• How do we apply performance technology
  effectively for the variety and diversity of
  performance problems that arise in the context of
  complex parallel and distributed computer systems?

?
9
TAU Performance System

• Tuning and Analysis Utilities
• Performance system framework for scalable
  parallel and distributed high-performance
  computing
• Targets a general complex system computation
  model
• nodes / contexts / threads
• Multi-level system / software / parallelism
• Measurement and analysis abstraction
• Integrated toolkit for performance
  instrumentation, measurement, analysis, and
  visualization
• Portable performance profiling and tracing
  facility
• Open software approach with technology
  integration
• University of Oregon , Forschungszentrum Jülich,
  LANL

10
TAU Performance System Architecture
11
TAU Performance System Status

• Computing platforms
• IBM SP / Power4, SGI Origin 2K/3K, ASCI Red, Cray
  T3E / SV-1 / X-1, HP (Compaq) SC (Tru64), HP
  Superdome (HP-UX), Sun, Hitachi SR8000, NEX
  SX-5/6, Linux clusters (IA-32/64, Alpha, PPC,
  PA-RISC, Power, Opteron), Apple (G4/5, OS X),
  Windows
• Programming languages
• C, C, Fortran 77/90/95, HPF, Java, OpenMP,
  Python
• Communication libraries
• MPI, PVM, Nexus, shmem, LAMPI, MPIJava
• Thread libraries
• pthreads, SGI sproc, Java,Windows, OpenMP

12
TAU Performance System Status (continued)

• Compilers
• Intel KAI (KCC, KAP/Pro), PGI, GNU, Fujitsu, Sun,
  Microsoft, SGI, Cray, IBM (xlc, xlf), Compaq,
  Hitachi, NEC, Intel
• Application libraries (selected)
• Blitz, A/P, PETSc, SAMRAI, Overture, PAWS
• Application frameworks (selected)
• POOMA, MC, ECMF, Uintah, VTF, UPS, GrACE
• Performance technology integrated with TAU
• PAPI, PCL, DyninstAPI, mpiP, MUSE/Magnet
• TAU full distribution (Version 2.x, web download)
• TAU performance system toolkit and users guide
• Automatic software installation and examples

13
Computational Science

• Integration of computer sciencein traditional
  sciencedisciplines
• Third model ofscientificresearch
• Application ofhigh-performancecomputation,
  algorithmsand networking
• Parallel computing
• Grid computing

14
Computational Science Projects at UO

• Geological science
• Model coupling for hydrology
• Bioinformatics
• Zebrafish Information Network (ZFIN)
• Evolution of gene families
• Oregon Bioinformatics Tool
• Neuroinformatics
• Electronic notebooks
• Domain-specific problem solving environments
• Dinosaur skeleton and motion modeling
• Computational Science Institute

15
Computational Science ? Cognitive Neuroscience

• Computational methods applied to scientific
  research
• High-performance simulation of complex phenomena
• Large-scale data analysis and visualization
• Understand functional activity of the human
  cortex
• Multiple cognitive, clinical, and medical domains
• Multiple experimental paradigms and methods
• Need for coupled/integrated modeling and analysis
• Multi-modal (electromagnetic, MR, optical)
• Physical brain models and theoretical cognitive
  models
• Need for robust tools computational informatic

16
Brain Dynamics Analysis Problem

• Identify functional components
• Different cognitive neuroscience research
  contexts
• Clinical and medical applications
• Interpret with respect to physical and cognitive
  models
• Requirements spatial (structure), temporal
  (activity)
• Imaging techniques for analyzing brain dynamics
• Blood flow neuroimaging (PET, fMRI)
• good spatial resolution ? functional brain
  mapping
• temporal limitations to tracking of dynamic
  activities
• Electromagnetic measures (EEG/ERP, MEG)
• msec temporal resolution to distinguish
  components
• spatial resolution sub-optimal (source
  localization)

17
Integrated Electromagnetic Brain Analysis
good spatial poor temporal
Cortical Activity Knowledge Base
Head Analysis
Source Analysis
Structural / Functional MRI/PET
spatial pattern recognition
temporal dynamics
Cortical Activity Model
Experiment subject
IndividualBrain Analysis
Constraint Analysis
Component Response Model
neural constraints
Dense Array EEG / MEG
temporal pattern recognition
Signal Analysis
Response Analysis
Component Response Knowledge Base
poor spatial good temporal
neuroimaging integration
18
Experimental Methodology and Tool Integration
16x256bits permillisec (30MB/m)
CT / MRI
segmentedtissues
EEG
NetStation
BrainVoyager
processed EEG
mesh generation
source localization constrained to cortical
surface
Interpolator 3D
EMSE
BESA
19
NeuroInformatics Center (NIC)

• Application of computational science methods to
  cognitive and clinical neuroscience problems
• Understand functional activity of the brain
• Help to diagnosis brain-related disorders
• Utilize high-performance computing and simulation
• Support large-scale data analysis and
  visualization
• Advance techniques for integrated neuroimaging
• Coupled modeling (EEG/ERP and MR analysis)
• Advanced statistical factor analysis
• FDM/FEM brain models (EEG, CT, MRI)
• Source localization
• Problem-solving environment for brain analysis

20
NIC Organization

• Director, Allen D. Malony
• Associate Professor, Computer and Information
  Science
• Associate Director, Don M. Tucker
• Professor, Psychology CEO, EGI
• Computational Scientist, Kevin Glass
• Ph.D., Computer Science B.S., Physics
• Computational Physicist, Sergei Turovets
• Ph.D., Computer Science B.S., Physics
• Computer Scientist, Sameer S. Shende
• Ph.D., Computer Science parallel computing
  specialist
• Mathematician, Bob Frank
• M.S., Mathematics

21
Funding Support

• BBMI federal appropriation
• DoD Telemedicine Advanced Technology Research
  Command (TATRC)
• Initial budget of approximately 750K
• Oct. 1, 2002 through March 31, 2004
• NSF Major Research Instrumentation
• ICONIC Grid, awarded
• New proposal opportunities
• NIH Human Brain Project Neuroinformatics
• NSF ITR

22
NIC Approaches

• Optimize spatial resolution
• MRI structural information
• Measurement of skull conductivity
• Convergence / co-recording with MEG and fMRI
• Optimize temporal resolution
• Use EEG/MEG time course for fMRI signal
  extraction
• Decomposition of component analysis (ICA, PCA)
• Single-trial analysis
• Computational brain models
• Boundary and finite element brain models
• Brain information databases and atlases

23
EEG/ERP Methodology

• Electroencephalogram (EEG)
• Event-Related Potential (ERP)
• Stimulus-locked measures of brain dynamics
• Generated from subject- and trial-based analysis
• Raw EEG datasets processed and analyzed
• Segmentation to time series waveforms
• Blink removal and other cleaning
• ERP analysis
• Averaging for increasing signal to noise
• Characterization with respect to trial conditions
• Results visualization
• Source localization

24
EGI Geodesics Sensor Net

• Electrical Geodesics Inc.
• Dense-array sensor technology
• 64/128/256 channels
• 256-channel geodesics sensor net
• AgCl plastic electrodes
• Carbon fiber leads
• Future optical sensors
• EGI LANL

25
EEG/ERP Experiment Management System

• Support EEG-based cognitive neuroscience research
• Based on experiment model
• Experiment type
• Subjects measured for trial types
• Management of experiment data
• Raw and processed datasets and derived statistics
• Per experiment/subject/trial database
• Secure protection and storage with selective
  access
• Analysis tools and workflows
• Generation of results (across experimental
  variables)
• Analysis processes with multi-tool workflows

26
EEG/ERP Experiment Analysis Environment
processed datasets / derived results
raw
analysis workflow


virtual services
storage resources
compute resources
27
Source Localization

• Mapping of scalp potentials to cortical
  generators
• Single time sample and time series
• Requirements
• Accurate head model and physics
• High-resolution 3D structural geometry
• Precise tissue identification and segmentation
• Correct tissue conductivity assessment
• Computational head model formulation
• Finite element model (FEM)
• Finite difference model (FDM)
• Forward problem calculation
• Dipole search strategy

28
Advanced Image Segmentation

• Native MR gives high gray-to-white matter
  contrast
• Edge detection finds region boundaries
• Segments formed by edge merger
• Color depicts tissue type
• Investigate more advance level set methods and
  hybrid methods

29
Building Finite Element Brain Models

• MRI segmentation of brain tissues
• Conductivity model
• Measure head tissue conductivity
• Electrical impedance tomography
• small currents are injectedbetween electrode
  pair
• resulting potential measuredat remaining
  electrodes
• Finite element forward solution
• Source inverse modeling
• Explicit and implicit methods
• Bayesian methodology

scalp
CSF
skull
cortex
30
Conductivity Modeling
Governing Equations ICS/BCS
Continuous Solutions
Finite-DifferenceFinite-ElementBoundary-Element
Finite-VolumeSpectral
Discretization
System of Algebraic Equations
Discrete Nodal Values
TridiagonalADISORGauss-SeidelGaussian
elimination
Equation (Matrix) Solver
? (x,y,z,t)J (x,y,z,t)B (x,y,z,t)
Approximate Solution
31
Source Localization Analysis Environment
raw


virtual services
storage resources
compute resources
32
NIC Computational Cluster (Neuronic Cluster)

• Dell computational cluster
• 16 dual-processor nodes
• 2.8 MHz Pentium Xeon
• 4 Gbyte memory
• 36 Gbyte disk
• Dual Gigabit ethernet adaptors
• 2U form factor
• Master node (same specs)
• 2 Gigabit ethernet switches
• Brain modeling
• Component analysis

33
NIC Relationships
OHSU/ OGI
Utah
LANL
Argonne
UCSD
Internet2
NCSA
Sandia
USC
Academic
Labs / Centers
Intel
IBM
UO Departments
EGI
NIC
Psychology
BDL
BEL
Industry
UO Centers/Institutes
CIS
Physics
CSI
CDSI
BBMI
CNI
NSI
34
NSF MRI Proposal

• Major Research Instrumentation (MRI)
• Acquisition of the Oregon ICONIC Grid for
  Integrated COgnitive Neuroscience Informatics and
  Computation
• PIs
• Computer Science Malony, Conery
• Psychology Tucker, Posner, Nunnally
• Senior personnel
• Computer Science Douglas, Cuny
• Psychology Neville, Awh, White
• Approximately 1.2M over three years

35
ICONIC Grid
graphics workstations
interactive, immersive viz
other campus clusters
Internet 2
Gbit Campus Backbone
CNI
NIC
NIC
CIS
CIS
4x8
16
16
2x8
2x16
SMP Server IBM p655
Graphics SMP SGI MARS
Shared Memory IBM p690
Distributed Memory IBM JS20
Distributed Memory Dell Pentium Xeon
5 Terabytes
SAN Storage System
36
Cognitive Neuroscience and ICONIC Grid

• Common questions to be explored
• Identifying brain networks
• Critical periods during normal development
• Network involvement in psychopathologies
• Training interventions in network development
• Research areas
• Development of attentional networks
• Brain plasticity in normal development and
  deprived
• Attention and emotion regulation
• Spatial working memory and selective attention
• Attention and psychopathology

37
Computer Science and ICONIC Grid

• Scheduling and resource management
• Assign hardware resources to computation tasks
• Scheduling of workloads for
• PSEs for computational science
• Provide scientists an entrée to the computational
  and data management power of the infrastructure
  without requiring specialized knowledge of
  parallel execution
• Marine seismic tomograph, molecular evolution
• Interactive / immersive three-dimensional
  visualization
• Explore multi-sensory visualization
• Merge 3D graphics with force-feedback haptics
• Parallel performance evaluation