Lexical Analysis

1
Lexical Analysis

• Uses formalism of Regular Languages
• Regular Expressions
• Deterministic Finite Automata (DFA)
• Non-deterministic Finite Automata (NDFA)
• RE ? NDFA ? DFA ? minimal DFA
• (F)Lex uses RE as input, builds lexor

2
DFAs Formal Definition

• DFA M (Q, S, d, q0, F)
• Q states finite set
• S alphabet finite set
• d transition function function in Q ? S
  ? Q
• q0 initial/starting state q0 ? Q
• F final states F ? Q

3
DFAs Example

• strings over a,b with next-to-last symbol a

4
Nondeterministic Finite Automata

• Nondeterminism implies having a choice.
• Multiple possible transitions from a state on a
  symbol.
• d(q,a) is a set of states
• d Q ? S ? Pow(Q)
• Can be empty, so no need for error/nonsense
  state.
• Acceptance exist path to a final state?
• I.e., try all choices.
• Also allow transitions on no input
• d Q ? (S ? e) ? Pow(Q)

5
NFAs Formal Definition

• NFA M (Q, S, d, q0, F)
• Q states a finite set
• S alphabet a finite set
• d transition function a total function in
• Q ? (S ? e) ? Pow(Q)
• q0 initial state q0 ? Q
• F final states F ? Q

6
NFAs Example

• strings over a,b with next-to-last symbol a

7
NFAs Example

• strings over 0,1,2 having
• (either 0-or-more 0s or 0-or-more 1s)
• followed by 0-or-more 2s

8
Regular Expressions

• Regular expression (over S)
• ?
• e
• a where a?S
• rr
• r r
• r
• where r,r regular (over S)
• Notational shorthand
• r0 e, ri rri-1
• r rr

9
RE ? NFA

• Defined inductively on structure of RE.
• This construction produces NFA with single final
  state.
• 6 cases ?, e, a, rr, rr, r

10
RE ? NFA ?
qf
q0
Accepts nothing since no edge to final state.
11
RE ? NFA e
q0
12
RE ? NFA a
qf
q0
a
13
RE ? NFA rr
14
RE ? NFA rr
15
RE ? NFA r
16
RE ? NFA Example
(001)
17
RE ? NFA Notes

• Most constructions produce very large NFAs.
• Not optimal for size.
• But easy to construct.

18
NFA -gt DFA Subset Construction

• Complicated but well described in the text
• Section 3.7.1 (pp 152-155), Algorithm 3.20 (2nd
  edition)
• In section 3.6 (pp 116-121) in 1st edition

19
Minimizing DFA

• Partition states of DFA, D, into two sets, final
  states, and non-final states.
• Continue until no more partitions are needed
• For each partition, P, split the DFA states of P
  so that, for each subpartition, all DFA states in
  that partition have the same transition for each
  input symbol, x.