Introductory Courses in High Performance Computing at Illinois

1
Introductory Courses in High Performance
Computing at Illinois

• David Padua

2
Our oldest course

• 420 Parallel Programming for Scientists and
  Engineers.
•  
• Course intended for non-CS majors (but many CS
  students take it). Taught once a year for the
  last 20 years.
•  
• CS 420 Parallel Progrmg Sci Engrg
• Credit 3 or 4 hours.
• Fundamental issues in design and development of
  parallel programs for various types of parallel
  computers. Various programming models according
  to both machine type and application area. Cost
  models, debugging, and performance evaluation of
  parallel programs with actual application
  examples. Same as CSE 402 and ECE 492. 3
  undergraduate hours. 3 or 4 graduate hours.
  Prerequisite CS 400 or CS 225

3
420 Parallel Programming for Scientists and
Engineers

• Machines
• Programming models
• Shared-memory
• Distributed memory
• Data parallel
• OpenMP/MPI/Fortran 90
• Clusters/Shared-memory machines/Vector
  supercomputers (in the past)
• Data parallel numerical algorithms (in Fortran
  90/MATLAB)
• Sorting/N-Body

4
Other courses

• 4xx Parallel programming. For majors
• 4xx Performance Programming. For all issues
  related to performance
•  5xx Theory of parallel computing. For advanced
  students
• 554 Parallel Numerical Algorithms.

5
4xx Parallel programming. For majors

• Overview of architectures. Architectural
  characterization of most important parallel
  systems today. Issues in effective programming of
  parallel architectures exploitation of
  parallelism, locality (cache, registers), load
  balancing, communication, overhead, consistency,
  coherency, latency avoidance. Transactional
  memories.
• Programming paradigms. Shared-memory, message
  passing, data parallel or regular, and functional
  programming paradigms. .. Message-passing
  programming. PGAS programming. Survey of
  programming languages. OpenMP, MPI, TBB, Charm,
  UPC, Co-array Fortran, High-Performance Fortran,
  NESL.
• Concepts. Basic concepts in parallel programming.
  Speedup, efficiency, redundancy, isoefficiency,
  Amdahl's law.
• Programming principles. Reactive parallel
  programming. Memory consistency. Synchronization
  strategies, critical regions, atomic updates,
  races, deadlock avoidance, prevention, livelock,
  starvation, scheduling fairness. Lock-free
  algorithms. Asynchronous algorithms. Speculation.
  Load balancing. Locality enhancement. Lock free
  algorithms. Asynchronous algorithms.
• Algorithms. Basic algorithms Element-by-element
  array operations, reductions, parallel prefix,
  linear recurrences, boolean recurrences. Systolic
  arrays, Matrix multiplication, LU decomposition,
  Jacobi relaxation, fixed point iterations.
  Sorting and searching. Graph algorithms,
  Datamining algorithms. N-Body/Particle
  simulations.

6
4xx Performance Programming.

• Sequential Performance bottlenecks CPU
  (pipelining, multiple issue processors (in-order
  and out-of order), support for speculation,
  branch prediction, execution units,
  vectorization, registers, register renaming)
  caches (temporal and spatial locality, compulsory
  misses, conflict misses, capacity misses,
  coherence misses) memory (latency, row/column,
  read/write), I/O
• Parallel performance bottlenecks Amdahl, load
  imbalance, communication, false sharing,
  granularity of communication (distributed memory)
• Optimization strategies Algorithm and program
  optimizations. Static and dynamic optimizations.
  Data dependent optimizations. Machine dependent
  and machine independent optimizations.
• Sequential program optimizations Redundancy
  elimination. Peephole optimizations. Loop
  optimizations. Branch optimizations.
• Locality optimizations. Tiling. Cache oblivious
  and cache conscious algorithms. Padding. Hardware
  and software prefetch.
• Parallel programming optimizations Brief
  introduction to parallel programming of
  shared-memory machines. Dependence graphs and
  program optimizations. Privatization, expansion,
  induction variables, wrap-around variables, loop
  fusion and loop fission. Frequently occurring
  kernels (reductions, scan, linear recurrences)
  and their parallel versions. Program
  vectorization. Multimedia extensions and their
  programming. Speculative parallel programming.
  Load balancing. Bottlenecks. Overdecomposition.
• Communication optimizations. Aggregation for
  communication. Redundant computations to save
  avoid communication. False sharing.
• Optimization for power.
• Tools for program tuning. Performance monitors.
  Profiling. Sampling. Compiler switches,
  directives and compiler feedback.
• Autotuning. Empirical search. Machine learning
  strategies for program optimization. Libbrary
  generators. ATLAS, FFTW, SPIRAL.
• Algorithm choice and tuning. Hybrid algorithms.
  Self optimizing algorithms. Sorting. Datamining.
  Numerical error and algorithm choice.