CSE 7381

1
Lecture 5 Introduction to Advanced Pipelining

• Prof. Fatih Koçan
• CSE 7/5381 Computer Architecture
• Fall 2002
• Adapted from Pattersons Slides

2
Advanced Pipelining and Instruction Level
Parallelism (ILP)

• ILP Overlap execution of unrelated instructions
• gcc 17 control transfer
• 5 instructions 1 branch
• Beyond single block to get more instruction level
  parallelism
• Loop level parallelism one opportunity, SW and HW
• Do examples and then explain nomenclature
• DLX Floating Point as example
• Measurements suggests R4000 performance FP
  execution has room for improvement

3
FP Loop Where are the Hazards?

• Loop LD F0,0(R1) F0vector element
• ADDD F4,F0,F2 add scalar from F2
• SD 0(R1),F4 store result
• SUBI R1,R1,8 decrement pointer 8B (DW)
• BNEZ R1,Loop branch R1!zero
• NOP delayed branch slot

Instruction Instruction Latency inproducing
result using result clock cycles FP ALU
op Another FP ALU op 3 FP ALU op Store double 2
Load double FP ALU op 1 Load double Store
double 0 Integer op Integer op 0

• Where are the stalls?

4
FP Loop Hazards
Loop LD F0,0(R1) F0vector element
ADDD F4,F0,F2 add scalar in F2
SD 0(R1),F4 store result SUBI R1,R1,8 decre
ment pointer 8B (DW) BNEZ R1,Loop branch
R1!zero NOP delayed branch slot
Instruction Instruction Latency inproducing
result using result clock cycles FP ALU
op Another FP ALU op 3 FP ALU op Store double 2
Load double FP ALU op 1 Load double Store
double 0 Integer op Integer op 0
5
FP Loop Showing Stalls
1 Loop LD F0,0(R1) F0vector element
2 stall 3 ADDD F4,F0,F2 add scalar in F2
4 stall 5 stall 6 SD 0(R1),F4 store result
7 SUBI R1,R1,8 decrement pointer 8B (DW) 8
BNEZ R1,Loop branch R1!zero
9 stall delayed branch slot
Instruction Instruction Latency inproducing
result using result clock cycles FP ALU
op Another FP ALU op 3 FP ALU op Store double 2
Load double FP ALU op 1

• 9 clocks Rewrite code to minimize stalls?

6
Revised FP Loop Minimizing Stalls
1 Loop LD F0,0(R1) 2 stall
3 ADDD F4,F0,F2 4 SUBI R1,R1,8
5 BNEZ R1,Loop delayed branch 6
SD 8(R1),F4 altered when move past SUBI
Swap BNEZ and SD by changing address of SD
Instruction Instruction Latency inproducing
result using result clock cycles FP ALU
op Another FP ALU op 3 FP ALU op Store double 2
Load double FP ALU op 1

• 6 clocks Unroll loop 4 times code to make
  faster?

7
Unroll Loop Four Times (straightforward way)
1 Loop LD F0,0(R1) 2 ADDD F4,F0,F2
3 SD 0(R1),F4 drop SUBI BNEZ 4 LD F6,-8(R1)
5 ADDD F8,F6,F2 6 SD -8(R1),F8 drop SUBI
BNEZ 7 LD F10,-16(R1) 8 ADDD F12,F10,F2
9 SD -16(R1),F12 drop SUBI BNEZ
10 LD F14,-24(R1) 11 ADDD F16,F14,F2
12 SD -24(R1),F16 13 SUBI R1,R1,32 alter to
48 14 BNEZ R1,LOOP 15 NOP 15 4 x (12)
27 clock cycles, or 6.8 per iteration Assumes
R1 is multiple of 4

• Rewrite loop to minimize stalls?

8
Unrolled Loop That Minimizes Stalls
1 Loop LD F0,0(R1) 2 LD F6,-8(R1) 3 LD F10,-16(R1
) 4 LD F14,-24(R1) 5 ADDD F4,F0,F2 6 ADDD F8,F6,F2
7 ADDD F12,F10,F2 8 ADDD F16,F14,F2 9 SD 0(R1),F4
10 SD -8(R1),F8 11 SD -16(R1),F12 12 SUBI R1,R1,
32 13 BNEZ R1,LOOP 14 SD 8(R1),F16 8-32 -24
14 clock cycles, or 3.5 per iteration When safe
to move instructions?

• What assumptions made when moved code?
• OK to move store past SUBI even though changes
  register
• OK to move loads before stores get right data?
• When is it safe for compiler to do such changes?

9
Compiler Perspectives on Code Movement

• Definitions compiler concerned about
  dependencies in program, whether or not a HW
  hazard depends on a given pipeline
• Try to schedule to avoid hazards
• (True) Data dependencies (RAW if a hazard for HW)
• Instruction i produces a result used by
  instruction j, or
• Instruction j is data dependent on instruction k,
  and instruction k is data dependent on
  instruction i.
• If depedent, cant execute in parallel
• Easy to determine for registers (fixed names)
• Hard for memory
• Does 100(R4) 20(R6)?
• From different loop iterations, does 20(R6)
  20(R6)?

10
Where are the data dependencies?
1 Loop LD F0,0(R1) 2 ADDD F4,F0,F2
3 SUBI R1,R1,8 4 BNEZ R1,Loop delayed
branch 5 SD 8(R1),F4 altered when move past
SUBI
11
Compiler Perspectives on Code Movement

• Another kind of dependence called name
  dependence two instructions use same name
  (register or memory location) but dont exchange
  data
• Antidependence (WAR if a hazard for HW)
• Instruction j writes a register or memory
  location that instruction i reads from and
  instruction i is executed first
• Output dependence (WAW if a hazard for HW)
• Instruction i and instruction j write the same
  register or memory location ordering between
  instructions must be preserved.

12
Where are the name dependencies?
1 Loop LD F0,0(R1) 2 ADDD F4,F0,F2
3 SD 0(R1),F4 drop SUBI BNEZ 4 LD F0,-8(R1)
2 ADDD F4,F0,F2 3 SD -8(R1),F4 drop SUBI
BNEZ 7 LD F0,-16(R1) 8 ADDD F4,F0,F2
9 SD -16(R1),F4 drop SUBI BNEZ
10 LD F0,-24(R1) 11 ADDD F4,F0,F2
12 SD -24(R1),F4 13 SUBI R1,R1,32 alter to
48 14 BNEZ R1,LOOP 15 NOP How can remove
them?
13
Where are the name dependencies?
1 Loop LD F0,0(R1) 2 ADDD F4,F0,F2
3 SD 0(R1),F4 drop SUBI BNEZ 4 LD F6,-8(R1)
5 ADDD F8,F6,F2 6 SD -8(R1),F8 drop SUBI
BNEZ 7 LD F10,-16(R1) 8 ADDD F12,F10,F2
9 SD -16(R1),F12 drop SUBI BNEZ
10 LD F14,-24(R1) 11 ADDD F16,F14,F2
12 SD -24(R1),F16 13 SUBI R1,R1,32 alter to
48 14 BNEZ R1,LOOP 15 NOP Called register
renaming
14
Compiler Perspectives on Code Movement

• Again Name Dependences are Hard for Memory
  Accesses
• Does 100(R4) 20(R6)?
• From different loop iterations, does 20(R6)
  20(R6)?
• Our example required compiler to know that if R1
  doesnt change then0(R1) ? -8(R1) ? -16(R1) ?
  -24(R1)
• There were no dependencies between some
  loads and stores so they could be moved by each
  other

15
Compiler Perspectives on Code Movement

• Final kind of dependence called control
  dependence
• Example
• if p1 S1
• if p2 S2
• S1 is control dependent on p1 and S2 is control
  dependent on p2 but not on p1.

16
Compiler Perspectives on Code Movement

• Two (obvious) constraints on control dependences
• An instruction that is control dependent on a
  branch cannot be moved before the branch so
  that its execution is no longer controlled by the
  branch.
• An instruction that is not control dependent on a
  branch cannot be moved to after the branch so
  that its execution is controlled by the branch.

17
Preserving Control Dependence

• Two properties of a simple pipeline
• Instructions execute in program order
• An instruction that occurs before a branch is
  executed before the branch
• The detection of control or branch hazards
• An instruction that is control dependent on a
  branch is not executed until the branch direction
  is known
• Do we have to preserve control dependence ?

18
Relaxing Control Dependence

• Control dependence is not a critical property
  that must be preserved
• The two properties critical to program
  correctness
• The exception behavior
• The data flow
• Preserved by maintaining both data and control
  dependence
• Control dependencies relaxed to get parallelism
• Preserving exception behavior
• Any changes in the ordering of instruction
  execution must not change how exceptions are
  raised in the program
• The reordering of instruction execution must not
  cause any new exceptions in the program

19
Example

• How maintaining the control and data dependences
  can prevent any new exception occurrence
• DADDU R2, R3, R4
• BEQZ R2, L1
• LW R1, 0(R2)
• L1
• Data dependence involving R2 is not maintained
• ? Change the result of the program
• Ignore control dependence move LW before BEQZ
• Data dependence is violated ?? ? NO!
• What prevents us moving LW before BEQZ ? ? Only
  control dependence

DADDU R2, R3, R4 LW R1,
0(R2) BEQZ R2, L1 L1
20
Reordering these instructions

• How maintaining the control and data dependences
  can prevent any new exception occurrence
• DADDU R2, R3, R4
• BEQZ R2, L1
• LW R1, 0(R2)
• L1
• What prevents us moving LW before BEQZ ? ? Only
  control dependence
• Just ignore the exception when the branch is
  taken
• A HW technique speculation ? allows us to
  overcome this exception problem

DADDU R2, R3, R4 LW R1,
0(R2) BEQZ R2, L1 L1
21
Preserving Data Flow

• Actual flow of data values among instructions
• Branches make the data flow dynamic
• Allowing the source of data for a given
  instruction to come from many points
• DADDU R1, R2, R3
• BEQZ R4, L
• DSUBU R1, R5, R6
• L
• OR R7, R1, R8
• Data dependence is not sufficient alone to
  preserve correctness.
• Speculation ? helps exception problem lessen
  the impact of the control dependence (while
  maintaining data flow)

22
Safe to violate control dependence ?

• Actual flow of data values among instructions
• Branches make the data flow dynamic
• Allowing the source of data for a given
  instruction to come from many points
• DADDU R1, R2, R3
• BEQZ R12, skipnext
• DSUBU R4, R5, R6
• DADDU R5, R4, R9
• skipnext OR R7, R8, R9
• R4 unused after OR instruction
• Liveness or deadness of a variable
• R4 is dead, so change value of R4 before branch
• ? Move DSUBU before branch
• ? data flow is not changed
• ? ? Code scheduling is called SPECULATION
• Compiler is betting on the outcome of the branch

23
HW Schemes Instruction Parallelism

• Why in HW at run time?
• Works when cant know real dependence at compile
  time
• Compiler simpler
• Code for one machine runs well on another
• Key idea Allow instructions behind stall to
  proceed
• DIVD F0,F2,F4
• ADDD F10,F0,F8
• SUBD F12,F8,F14
• Enables out-of-order execution gt out-of-order
  completion
• ID stage checked both for structuralScoreboard
  dates to CDC 6600 in 1963

24
HW Schemes Instruction Parallelism

• Out-of-order execution divides ID stage
• 1. Issuedecode instructions, check for
  structural hazards
• 2. Read operandswait until no data hazards, then
  read operands
• Scoreboards allow instruction to execute whenever
  1 2 hold, not waiting for prior instructions
• CDC 6600 In order issue, out of order execution,
  out of order commit ( also called completion)

25
Scoreboard Implications

• Out-of-order completion gt WAR, WAW hazards?
• Solutions for WAR
• Queue both the operation and copies of its
  operands
• Read registers only during Read Operands stage
• For WAW, must detect hazard stall until other
  completes
• Need to have multiple instructions in execution
  phase gt multiple execution units or pipelined
  execution units
• Scoreboard keeps track of dependencies, state or
  operations
• Scoreboard replaces ID, EX, WB with 4 stages

26
Four Stages of Scoreboard Control

• 1. Issuedecode instructions check for
  structural hazards (ID1)
• If a functional unit for the instruction is
  free and no other active instruction has the same
  destination register (WAW), the scoreboard issues
  the instruction to the functional unit and
  updates its internal data structure. If a
  structural or WAW hazard exists, then the
  instruction issue stalls, and no further
  instructions will issue until these hazards are
  cleared.
• 2. Read operandswait until no data hazards, then
  read operands (ID2)
• A source operand is available if no earlier
  issued active instruction is going to write it,
  or if the register containing the operand is
  being written by a currently active functional
  unit. When the source operands are available, the
  scoreboard tells the functional unit to proceed
  to read the operands from the registers and begin
  execution. The scoreboard resolves RAW hazards
  dynamically in this step, and instructions may be
  sent into execution out of order.

27
Four Stages of Scoreboard Control

• 3. Executionoperate on operands (EX)
• The functional unit begins execution upon
  receiving operands. When the result is ready, it
  notifies the scoreboard that it has completed
  execution.
• 4. Write resultfinish execution (WB)
• Once the scoreboard is aware that the
  functional unit has completed execution, the
  scoreboard checks for WAR hazards. If none, it
  writes results. If WAR, then it stalls the
  instruction.
• Example
• DIVD F0,F2,F4
• ADDD F10,F0,F8
• SUBD F8,F8,F14
• CDC 6600 scoreboard would stall SUBD until ADDD
  reads operands

28
Three Parts of the Scoreboard

• 1. Instruction statuswhich of 4 steps the
  instruction is in
• 2. Functional unit statusIndicates the state of
  the functional unit (FU). 9 fields for each
  functional unit
• BusyIndicates whether the unit is busy or not
• OpOperation to perform in the unit (e.g., or
  )
• FiDestination register
• Fj, FkSource-register numbers
• Qj, QkFunctional units producing source
  registers Fj, Fk
• Rj, RkFlags indicating when Fj, Fk are ready
• 3. Register result statusIndicates which
  functional unit will write each register, if one
  exists. Blank when no pending instructions will
  write that register

29
Detailed Scoreboard Pipeline Control
30
Scoreboard Example
31
Scoreboard Example Cycle 1
32
Scoreboard Example Cycle 2

• Issue 2nd LD?

33
Scoreboard Example Cycle 3

• Issue MULT?

34
Scoreboard Example Cycle 4
35
Scoreboard Example Cycle 5
36
Scoreboard Example Cycle 6
37
Scoreboard Example Cycle 7

• Read multiply operands?

38
Scoreboard Example Cycle 8a
39
Scoreboard Example Cycle 8b
40
Scoreboard Example Cycle 9

• Read operands for MULT SUBD? Issue ADDD?

41
Scoreboard Example Cycle 11
42
Scoreboard Example Cycle 12

• Read operands for DIVD?

43
Scoreboard Example Cycle 13
44
Scoreboard Example Cycle 14
45
Scoreboard Example Cycle 15
46
Scoreboard Example Cycle 16
47
Scoreboard Example Cycle 17

• Write result of ADDD?

48
Scoreboard Example Cycle 18
49
Scoreboard Example Cycle 19
50
Scoreboard Example Cycle 20
51
Scoreboard Example Cycle 21
52
Scoreboard Example Cycle 22
53
Scoreboard Example Cycle 61
54
Scoreboard Example Cycle 62
55
CDC 6600 Scoreboard

• Speedup 1.7 from compiler 2.5 by hand BUT slow
  memory (no cache) limits benefit
• Limitations of 6600 scoreboard
• No forwarding hardware
• Limited to instructions in basic block (small
  window)
• Small number of functional units (structural
  hazards), especailly integer/load store units
• Do not issue on structural hazards
• Wait for WAR hazards
• Prevent WAW hazards

56
Summary of Scoreboard

• Instruction Level Parallelism (ILP) in SW or HW
• Loop level parallelism is easiest to see
• SW parallelism dependencies defined for program,
  hazards if HW cannot resolve
• SW dependencies/compiler sophistication determine
  if compiler can unroll loops
• Memory dependencies hardest to determine
• HW exploiting ILP
• Works when cant know dependence at run time
• Code for one machine runs well on another
• Key idea of Scoreboard Allow instructions behind
  stall to proceed (Decode gt Issue instr read
  operands)
• Enables out-of-order execution gt out-of-order
  completion
• ID stage checked both for structural

57
Another Dynamic Algorithm Tomasulo Algorithm

• For IBM 360/91 about 3 years after CDC 6600
  (1966)
• Goal High Performance without special compilers
• Differences between IBM 360 CDC 6600 ISA
• IBM has only 2 register specifiers/instr vs. 3 in
  CDC 6600
• IBM has 4 FP registers vs. 8 in CDC 6600
• Why Study? lead to Alpha 21264, HP 8000, MIPS
  10000, Pentium II, PowerPC 604,

58
How hazards are eliminated/reduced ?

• RAW wait until operands are ready
• WAR/WAW rename all destination registers
• Including those with a pending read or write for
  an earlier instruction
• The out-of-order write does not affect any
  instructions that depend on an earlier value of
  an operand
• DIVD F0, F2, F4
• ADDD F6, F0, F8
• SD F6, 0(R1)
• SUBD F8, F10, F14
• MULD F6, F10, F8

DIVD F0, F2, F4 ADDD S, F0, F8 SD S,
0(R1) SUBD T, F10, F14 MULD F6, F10, T ST
temporary registers
59
Tomasulo Algorithm vs. Scoreboard

• Control buffers distributed with Function Units
  (FU) vs. centralized in scoreboard
• FU buffers called reservation stations have
  pending operands
• Registers in instructions replaced by values or
  pointers to reservation stations(RS) called
  register renaming
• avoids WAR, WAW hazards
• More reservation stations than registers, so can
  do optimizations compilers cant
• Results to FU from RS, not through registers,
  over Common Data Bus that broadcasts results to
  all FUs
• Load and Stores treated as FUs with RSs as well
• Integer instructions can go past branches,
  allowing FP ops beyond basic block in FP queue

60
Tomasulo Organization
FPRegisters
FP Op Queue
LoadBuffer
StoreBuffer
CommonDataBus
FP AddRes.Station
FP MulRes.Station
61
Reservation Station Components

• OpOperation to perform in the unit (e.g., or
  )
• Vj, VkValue of Source operands
• Store buffers has V field, result to be stored
• Qj, QkReservation stations producing source
  registers (value to be written)
• Note No ready flags as in Scoreboard Qj,Qk0 gt
  ready
• Store buffers only have Qi for RS producing
  result
• BusyIndicates reservation station or FU is
  busy
• Register result statusIndicates which
  functional unit will write each register, if one
  exists. Blank when no pending instructions that
  will write that register.

62
Three Stages of Tomasulo Algorithm

• 1. Issueget instruction from FP Op Queue
• If reservation station free (no structural
  hazard), control issues instr sends operands
  (renames registers).
• 2. Executionoperate on operands (EX)
• When both operands ready then execute if not
  ready, watch Common Data Bus for result
• LOAD/STORE 1. Compute effective address if base
  register is available
• 2. Place this address in load/store buffer
• Store address and data is ready ? write to
  memory
• Load address and memory is ready ? read from
  memory
• 3. Write resultfinish execution (WB)
• Write on Common Data Bus to all awaiting units
  mark reservation station available
• Normal data bus data destination (go
  to bus)
• Common data bus data source (come from bus)
• 64 bits of data 4 bits of Functional Unit
  source address
• Write if matches expected Functional Unit
  (produces result)
• Does the broadcast

63
Tomasulo Example Cycle 0
64
Tomasulo Example Cycle 1
Yes
65
Tomasulo Example Cycle 2
Note Unlike 6600, can have multiple loads
outstanding
66
Tomasulo Example Cycle 3

• Note registers names are removed (renamed) in
  Reservation Stations MULT issued vs. scoreboard
• Load1 completing what is waiting for Load1?

67
Tomasulo Example Cycle 4

• Load2 completing what is waiting for it?

68
Tomasulo Example Cycle 5
69
Tomasulo Example Cycle 6

• Issue ADDD here vs. scoreboard?

70
Tomasulo Example Cycle 7

• Add1 completing what is waiting for it?

71
Tomasulo Example Cycle 8
72
Tomasulo Example Cycle 9
73
Tomasulo Example Cycle 10

• Add2 completing what is waiting for it?

74
Tomasulo Example Cycle 11

• Write result of ADDD here vs. scoreboard?

75
Tomasulo Example Cycle 12

• Note all quick instructions complete already

76
Tomasulo Example Cycle 13
77
Tomasulo Example Cycle 14
78
Tomasulo Example Cycle 15

• Mult1 completing what is waiting for it?

79
Tomasulo Example Cycle 16

• Note Just waiting for divide

80
Tomasulo Example Cycle 55
81
Tomasulo Example Cycle 56

• Mult 2 completing what is waiting for it?

82
Tomasulo Example Cycle 57

• Again, in-oder issue, out-of-order execution,
  completion

83
Compare to Scoreboard Cycle 62

• Why takes longer on Scoreboard/6600?

84
Tomasulo v. Scoreboard(IBM 360/91 v. CDC 6600)

• Pipelined Functional Units Multiple Functional
  Units
• (6 load, 3 store, 3 , 2 x/) (1 load/store, 1
  , 2 x, 1 )
• window size 14 instructions 5 instructions
  
• No issue on structural hazard same
• WAR renaming avoids stall completion
• WAW renaming avoids stall completion
• Broadcast results from FU Write/read registers
• Control reservation stations central
  scoreboard

85
Tomasulo Drawbacks

• Complexity
• Large amount of HW
• delays of 360/91, MIPS 10000, IBM 620?
• Many associative stores (CDB) at high speed
• Tag check in reservation station buffers
• Performance limited by Common Data Bus
• Multiple CDBs gt more FU logic for parallel assoc
  stores

86
Tomasulo Loop Example

• Loop LD F0 0 R1
• MULTD F4 F0 F2
• SD F4 0 R1
• SUBI R1 R1 8
• BNEZ R1 Loop
• Assume Multiply takes 4 clocks
• Assume first load takes 8 clocks (cache miss?),
  second load takes 4 clocks (hit)
• To be clear, will show clocks for SUBI, BNEZ
• Reality, integer instructions ahead

87
Loop Example Cycle 0
88
Loop Example Cycle 1
89
Loop Example Cycle 2
90
Loop Example Cycle 3

• Note MULT1 has no registers names in RS

91
Loop Example Cycle 4
92
Loop Example Cycle 5
93
Loop Example Cycle 6

• Note F0 never sees Load1 result

94
Loop Example Cycle 7

• Note MULT2 has no registers names in RS

95
Loop Example Cycle 8
96
Loop Example Cycle 9

• Load1 completing what is waiting for it?

97
Loop Example Cycle 10

• Load2 completing what is waiting for it?

98
Loop Example Cycle 11
99
Loop Example Cycle 12
100
Loop Example Cycle 13
101
Loop Example Cycle 14

• Mult1 completing what is waiting for it?

102
Loop Example Cycle 15

• Mult2 completing what is waiting for it?

103
Loop Example Cycle 16
104
Loop Example Cycle 17
105
Loop Example Cycle 18
106
Loop Example Cycle 19
107
Loop Example Cycle 20
108
Loop Example Cycle 21
109
Tomasulo Summary

• Reservations stations renaming to larger set of
  registers buffering source operands
• Prevents registers as bottleneck
• Avoids WAR, WAW hazards of Scoreboard
• Allows loop unrolling in HW
• Not limited to basic blocks (integer units gets
  ahead, beyond branches)
• Helps cache misses as well
• Lasting Contributions
• Dynamic scheduling
• Register renaming
• Load/store disambiguation
• 360/91 descendants are Pentium II PowerPC 604
  MIPS R10000 HP-PA 8000 Alpha 21264