Tuesday, September 04, 2006

1
Tuesday, September 04, 2006

• I hear and I forget,
• I see and I remember,
• I do and I understand.
• -Chinese Proverb

2
Today

• Course Overview.
• Why Parallel Computing?
• Evolution of Parallel Systems.

3
CS 524 High Performance Computing

• Course URL
• http//suraj.lums.edu.pk/cs524a06
• Folder on indus
• \\indus\Common\cs524a06
• Website Check Regularly Course announcements,
  office hours, slides, resources, policies
• Course Outline

4

• Several programming exercises will be given
  throughout the course. Assignments will include
  popular programming models for shared memory and
  message passing such as OpenMP and MPI.
• The development environment will be C/C on
  UNIX.

5
Pre-requisites

• Computer Organization Assembly Language (CS
  223)
• Data Structures Algorithms (CS 213)
• Senior level standing.
• Operating Systems?

6

• Five minute rule.

7
Hunger For More Power!
8
Hunger For More Power!

• Endless quest for more and more computing power.
• However much computing power there is, it is
  never enough.

9
Why this need for greater computational power?

• Science, engineering, businesses, entertainment
  etc., all are providing the impetus.
• Scientists observe, theorize, test through
  experimentation.
• Engineers design, test prototypes, build.

10
HPC offers a new way to do science

• Computation used to approximate physical systems
  - Advantages include
• Playing with simulation parameters to study
  emergent trends
• Possible replay of a particular simulation event
• Study systems where no exact theories exist

11
Why Turn to Simulation?

• When the problem is too . . .
• Complex
• Large
• Expensive
• Dangerous

12
Why this need for greater computational power?

• Less expensive to carry out computer simulations.
• Able to simulate phenomenon that could not be
  studied by experimentation. e.g. evolution of
  universe.

13
Why this need for greater computational power?

• Problems such as
• Weather prediction.
• Aeronautics (airflow analysis, structural
  mechanics, engine efficiency etc) .
• Simulating world economy.
• Pharmaceutical (molecular modeling).
• Understanding drug receptor interactions in
  brain.
• Automotive crash simulation.
• are all computationally intensive.
• The more knowledge we acquire the more complex
  our questions become.

14
Why this need for greater computational power?

• In 1995, the first full length computer animated
  motion picture, Toy Story, was produced on a
  parallel system composed on hundreds of Sun
  workstations.
• Decreased cost
• Decreased Time (Several months on several hundred
  processors)

15
Why this need for greater computational power?

• Commercial Computing has also come to rely on
  parallel architectures.
• Computer system speed and capacity ? Scale of
  business.
• OLTP (Online transaction processing) benchmark
  represent the relation between performance and
  scale of business.
• Rate performance of system in terms of its
  throughput in transactions per minute.

16
Why this need for greater computational power?

• Vendors supplying database hardware or software
  offer multiprocessor systems that provide
  performance substantially greater than
  uniprocessor products.

17

• One solution in the past Make the clock run
  faster.
• The advance of VLSI technology allowed clock
  rates to increase and larger number of components
  to fit on a chip.
• However there are limits
• Electrical signal cannot propagate faster than
  the speed of light 30cm/nsec in vacuum and
  20cm/nsec in copper wire or optical fiber.

18

• Electrical signal cannot propagate faster than
  the speed of light 30cm/nsec in vacuum and
  20cm/nsec in copper wire or optical fiber.
• 10-GHz clock - signal path length 2cm in total
• 100-GHz clock - 2mm
• 1 THZ (1000 GHz) computer will have to be smaller
  than 100 microns if the signal has to travel from
  one end to the other and back with a single clock
  cycle.

19

• Another fundamental problem
• Heat dissipation
• The faster a computer runs more heat it
  generates
• High end Pentium systems CPU cooling system
  bigger than the CPU itself.

20
Evolution of Parallel Architecture

• New dimension added to design space Number of
  processors.
• Driven by demand for performance at acceptable
  cost.

21
Evolution of Parallel Architecture

• Advances in hardware capability enable new
  application functionality, which places a greater
  demand on the architecture.
• This cycle drives the ongoing design, engineering
  and manufacturing effort.

22
Evolution of Parallel Architecture

• Microprocessor performance has been improving at
  a rate of about 50 per year.
• A parallel machine of hundred processors can be
  viewed as providing to applications computing
  power that will be available in 10 years time.
• 1000 processors ? 20 year horizon
• The advantages of using small, inexpensive, mass
  produced processors as building blocks for
  computer systems are clear.

23
Technology trends

• With technological advance, transistors, gates
  etc have been getting smaller and faster.
• More can fit in same area.
• Processors are getting faster by making more
  effective use of ever larger volume of computing
  resources.
• Possibilities
• Place more computer system on chip including
  memory and I/O. (Building block for parallel
  architectures. System-on-a-chip)
• Or multiple processors on chip. (Parallel
  architecture on single-chip regime)

24
Microprocessor Design Trends

• Technology determines what is possible.
• Architecture translates the potential of
  technology into performance.
• Parallelism is fundamental to conventional
  computer architecture.
• Current architectural trends are leading to
  multiprocessor designs.

25
Bit level Parallelism

• From 1970 to 1986 advancements in bit-level
  parallelism
• 4bit, 8 bit, 16 bit and so-on
• Doubling the data path reduces the number of
  cycles required to perform an operation.

26
Instruction level Parallelism

• Mid 1980s to mid 1990s
• Performing portions of several machine
  instructions concurrently.
• Pipelining (kind of parallelism also)
• Fetch multiple instructions at a time and issue
  them in parallel to distinct function units in
  parallel (superscalar)

27
Instruction level Parallelism

• However
• Instruction level parallelism is worthwhile only
  if processor can be supplied with instructions
  and data fast enough.
• Gap between processor cycle time and memory cycle
  time has grown wider.
• To satisfy increasing bandwidth requirements,
  larger and larger caches are placed on chip with
  the processor.
• cache miss
• control transfer
• Limits

28

• In mid 1970s, the introduction of vector
  processors marked the beginning of modern
  supercomputing
• Perform operations on sequences of data elements
  rather than individual scalar data
• Offered advantage of at least one order of
  magnitude over conventional systems of that time.

29

• In late 1980s a new generation of systems came on
  market. These were microprocessor based
  supercomputers that initially provided about 100
  processors and increased roughly to 1000 in 1990.
  
• These aggregation of processors are known as
  massively parallel processors (MPPs).

30

• Factors behind emergence of MPPs
• Increase in performance of standard
  microprocessors
• Cost advantage
• Usage of off-the-shelf microprocessors instead
  of custom processors
• Fostered by government programs for scalable
  parallel computing using distributed memory.

31

• MPPs claimed to equal or surpass the performance
  of vector multiprocessors.
• Top500
• Lists the sites that have the 500 most powerful
  installed computer systems.
• LINPACK benchmark
• Most widely used metric of performance on
  numerical applications
• Collection of Fortran subroutines that analyze
  and solve linear equations and linear least
  squares problems

32

• Top500 (Updated twice a year since June 1993)
• In the first Top500 list there were already 156
  MPP and SIMD systems present (around 1/3rd)

33
Some memory related issues

• Time to access memory has not kept pace with CPU
  clock speeds.
• SRAM
• Each bit is stored in a latch made up of
  transistors
• Faster than DRAM, but is less dense and requires
  greater power
• DRAM
• Each bit of memory is stored as a charge on a
  capacitor
• 1GHz CPU will execute 60 instructions before a
  typical 60ns DRAM can return a single byte

34
Some memory related issues

• Hierarchy
• Cache memories
• Temporal locality
• Cache lines (64, 128, 256 bytes)

35
Parallel Architectures Memory Parallelism

• One way to increase performance is to replicate
  computers.
• Major choice is between shared memory and
  distributed memory

36
Memory Parallelism

• In mid 1980s, when 32-bit microprocessor was
  first introduced, computers containing multiple
  microprocessors sharing a common memory became
  prevalent.
• In most of these designs all processors plug into
  a common bus.
• However, a small number of processors can be
  supported by bus

37
UMA bus based SMP architecture

• If the bus is busy, when a CPU wants to read or
  write memory, the CPU waits for CPU to become
  idle.
• Contention of bus can be manageable for small
  number of processors only.
• The system will be totally limited by bandwidth
  of the bus and most of the CPUs will be idle most
  of the time.

38
UMA bus based SMP architecture

• One way to alleviate this problem is to add a
  cache to each CPU.
• Less bus traffic if most reads can be satisfied
  from the cache and system can support more CPUs.
• Single bus limits UMA microprocessor to about
  16-32 CPUs.

39
SMP

• SMP (Symmetric multiprocessor)
• Shared memory multiprocessor where the cost of
  accessing a memory location is same for all
  processors.