Martin Rinard

1
MIT 6.035Introduction to Shift-Reduce Parsing

• Martin Rinard
• Laboratory for Computer Science
• Massachusetts Institute of Technology

2
Orientation

• Specify Syntax Using Context-Free Grammar
• Nonterminals
• Terminals
• Productions
• Given a grammar, Parser Generator produces a
  parser
• Starts with input string
• Produces parse tree

Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
3
Todays Lecture

• How generated parser works
• How parser generator produces parser
• Central mechanism
• Pushdown automaton, which implements
• Shift-reduce parser

4
Pushdown Automata

• Consists of
• Pushdown stack (can have terminals and
  nonterminals)
• Finite state automaton control
• Can do one of three actions (based on state and
  input)
• Shift
• Shift current input symbol from input onto stack
• Reduce
• If symbols on top of stack match right hand side
  of some grammar production NT ? ?
• Pop symbols (?) off of the stack
• Push left hand side nonterminal (NT) onto stack
• Accept the input string

5
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Stack
Input String

(

num
)
num
num
6
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?

(

num
)
num
num
7
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
SHIFT

(

num
)
num
num
8
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
num
SHIFT

(

num
)
num
9
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
num
REDUCE

(

num
)
num
10
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Expr
REDUCE
num

(

num
)
num
11
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Expr
SHIFT
num

(

num
)
num
12
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?

Expr
SHIFT
num
(

num
)
num
13
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Op
Expr
REDUCE
num

(

num
)
num
14
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Op
Expr
SHIFT
num

(

num
)
num
15
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
(
Op
Expr
SHIFT
num


num
)
num
16
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
(
Op
Expr
SHIFT
num


num
)
num
17
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
num
(
Op
Expr
SHIFT
num


num
)
18
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Expr
(
Op
Expr
SHIFT
REDUCE
num

num

num
)
19
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Expr
(
Op
Expr
SHIFT
num

num

num
)
20
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?

Expr
(
Op
Expr
SHIFT
num

num
num
)
21
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Op
Expr
(
Op
Expr
SHIFT
REDUCE
num

num

num
)
22
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Op
Expr
(
Op
Expr
SHIFT
num

num

num
)
23
Shift-Reduce Parser Example
num
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Op
Expr
(
Op
Expr
SHIFT
num

num

)
24
Shift-Reduce Parser Example
Expr
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Op
Expr
(
Op
Expr
SHIFT
REDUCE
num

num

num
)
25
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Expr
Expr
Expr
(
Op
Op
Expr
Expr
SHIFT
REDUCE
num

num

num
)
26
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
Expr
Expr
(
Op
Op
Expr
Expr
SHIFT
num

num

num
)
27
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
)
Expr
Expr
(
Op
Op
Expr
Expr
SHIFT
num

num

num
28
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
)
Expr
Expr
Op
Expr
Expr
(
Op
Expr
REDUCE
num

num

num
29
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
)
Expr
Expr
Op
Expr
(
Expr
Op
Expr
Expr
REDUCE
num

num

num
30
Shift-Reduce Parser Example
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
)
Expr
Expr
Op
Expr
(
Expr
Op
Expr
Expr
ACCEPT!
num

num

num
31
Basic Idea

• Goal reconstruct parse tree for input string
• Read input from left to right
• Build tree in a bottom-up fashion
• Use stack to hold pending sequences of terminals
  and nonterminals

32
Potential Conflicts

• Reduce/Reduce Conflict
• Top of the stack may match RHS of multiple
  productions
• Which production to use in the reduction?
• Shift/Reduce Conflict
• Stack may match RHS of production
• But that may not be the right match
• May need to shift an input and later find a
  different reduction

33
Conflicts

• New Grammar

• Original Grammar

Expr ? Expr Op Expr Expr ? Expr - Expr Expr ?
(Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
Expr ? Expr Op Expr Expr ? (Expr) Expr ? -
Expr Expr ? num Op ? Op ? - Op ?
34
Conflicts
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
-
num
num
35
Conflicts
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
SHIFT
-
num
num
36
Conflicts
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
num
SHIFT
-
num
37
Conflicts
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
Expr
SHIFT
REDUCE
num
-
num
38
Conflicts
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
Expr
SHIFT
num
-
num
39
Conflicts
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
-
Expr
SHIFT
num
num
40
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
Options Reduce Reduce Shift
-
Expr
num
num
41
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Reduce
-
Expr
REDUCE
num
num
42
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Reduce
Expr
Expr
SHIFT
num
-
num
43
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Reduce
num
Expr
Expr
SHIFT
num
-
44
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Reduce
Expr
Expr
Expr
REDUCE
num
-
num
45
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Reduce
Expr
Expr
Expr
FAILS!
num
-
num
46
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
Both of These Actions Work Reduce Shift
-
Expr
num
num
47
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Reduce
-
Expr
num
num
48
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Reduce
Op
Expr
REDUCE
num
-
num
49
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Reduce
num
Op
Expr
SHIFT
num
-
50
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Reduce
Expr
Op
Expr
REDUCE
num
-
num
51
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Reduce
Expr
Op
Expr
Expr
REDUCE
num
-
num
52
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Reduce
Expr
Op
Expr
Expr
ACCEPT
num
-
num
53
Conflicts
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Shift
-
Expr
SHIFT
num
num
54
Conflicts
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Shift
num
-
Expr
SHIFT
num
55
Conflicts
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Shift
Expr
-
Expr
REDUCE
num
num
56
Conflicts
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Shift
Expr
Expr
Expr
REDUCE
num
-
num
57
Conflicts
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
What Happens if Choose Shift
Expr
Expr
Expr
ACCEPT
num
-
num
58
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
This Shift/Reduce Conflict Reflects Ambiguity in
Grammar
-
Expr
num
num
59
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
This Shift/Reduce Conflict Reflects Ambiguity in
Grammar
-
Expr
Eliminate by Hacking Grammar
num
num
60
Shift/Reduce/Reduce Conflict
Expr ? Expr Op Expr Expr ? Expr - Expr Expr
? (Expr) Expr ? Expr - Expr ? num Op ? Op ?
- Op ?
This Shift/Reduce Conflict Can Be Eliminated By
Lookahead of One Symbol
-
Expr
Parser Generator Should Handle It
num
num
61
Constructing a Parser

• We will construct version with no lookahead
• Key Decisions
• Shift or Reduce
• Which Production to Reduce
• Basic Idea
• Build a DFA to control shift and reduce actions
• In effect, convert grammar to pushdown automaton
• Encode finite state control in parse table

62
Parser State

• Input Token Sequence ( for end of input)
• Current State from Finite State Automaton
• Two Stacks
• State Stack (implements finite state automaton)
• Symbol Stack (terminals from input and
  nonterminals from reductions)

63
Integrating Finite State Control

• Actions
• Push Symbols and States Onto Stacks
• Reduce According to a Given Production
• Accept
• Selected action is a function of
• Current input symbol
• Current state of finite state control
• Each action specifies next state
• Implement control using parse table

64
Parse Tables

• Implements finite state control
• At each step, look up
• Tabletop of state stack input symbol
• Then carry out the action

65
Parse Table Example
Grammar
Input
State Stack
Symbol Stack

• S ? X (1)
• X ? (X ) (2)
• X ? ( ) (3)

(())
X
s0
66
Parser Tables

• Shift to sn
• Push input token into the symbol stack
• Push sn into state stack
• Advance to next input symbol

67
Parser Tables

• Reduce (n)
• Pop both stacks as many times as the number of
  symbols on the RHS of rule n
• Push LHS of rule n into symbol stack

68
Parser Tables

• Reduce (n) (continued)
• Look up
• Tabletop of the state stacktop of symbol
  stack
• Push that state (in goto part of table) onto
  state stack

69
Parser Tables

• Accept
• Stop parsing and report success

70
Parse Table In Action
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
(())
s0
71
Parse Table In Action
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
(())
s0
72
Parse Table In Action
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
())
s2
s0
(
73
Parse Table In Action
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
())
s2
s0
(
74
Parse Table In Action
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
))
s2
s2
(
s0
(
75
Parse Table In Action
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
))
s2
s2
(
s0
(
76
Parse Table In Action
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
)
s5
s2
)
s2
(
s0
(
77
Parse Table In Action
Input
State Stack
Symbol Stack
Grammar
)
s5
S ? X (1) X ? (X ) (2) X ? ( ) (3)
s2
)
s2
(
s0
(
78
Step One Pop Stacks
Input
State Stack
Symbol Stack
Grammar
)
s5
S ? X (1) X ? (X ) (2) X ? ( ) (3)
s2
)
s2
(
s0
(
79
Step One Pop Stacks
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
)
s2
s0
(
80
Step Two Push Nonterminal
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
)
s2
s0
(
81
Step Two Push Nonterminal
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
)
s2
X
s0
(
82
Step Three Use Goto, Push New State
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
)
s2
X
s0
(
83
Step Three Use Goto, Push New State
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
)
s3
s2
X
s0
(
84
Parse Table In Action
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)
)
s3
s2
X
s0
(
85
Parse Table In Action
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)

s4
s3
)
s2
X
s0
(
86
Parse Table In Action
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)

s4
s3
)
s2
X
s0
(
87
Step One Pop Stacks
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)

s4
s3
)
s2
X
s0
(
88
Step One Pop Stacks
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)

s0
89
Step Two Push Nonterminal
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)

s0
90
Step Two Push Nonterminal
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)

s0
X
91
Step Three Use Goto, Push New State
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)

s0
X
92
Step Three Use Goto, Push New State
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)

s1
s0
X
93
Accept the String!
Grammar
Input
State Stack
Symbol Stack
S ? X (1) X ? (X ) (2) X ? ( ) (3)

s1
s0
S
94
Key Concepts

• Pushdown automaton for parsing
• Stack, Finite state control
• Parse actions shift, reduce, accept
• Parse table for controlling parser actions
• Indexed by parser state and input symbol
• Entries specify action and next state
• Use state stack to help control
• Parse tree construction
• Reads input from left to right
• Bottom-up construction of parse tree