Lex Output and Input

1
Lex Output and Input

• Flex The Transducer
• yytext
• yyleng
• Input from File
• yyin
• yywrap()
• yylex() calls yywrap() on EOF
• Returns value 1 (no more files)
• Returns 0 (more files)
• Control passed back to yylex()
• Assumes yyin setup with new file if 0 passed
  back
• So, if multi-file input, you need to overwrite
  default yywrap()

2
Flex (Multi-)File Input

• main(argc, argv)
• if(argcgt1)
• / open first file and set yyin to that file
  (named in argv1) /
• yylex( )
• yywrap( )
• / if more files, use argv to set yyin, return
  0, prompting yylex( ) /
• / if no more files, return 1, prompting end of
  scanning /

3
The Symbol Table

• First example listed all words to be recognised
  as verbs
• Lots of hassle to add new words, and makes lex
  file very large
• Program language compilers use symbol tables
• Table with words and their type (read in
  dynamic?)
• noun cat dog mouse
• verb ate hit chased
• Function to add-word to table addWord(int type,
  char word)
• Function to lookup word int lookUp(char word)
• State to check whether we want addition to table
  or analysis
• Enumerated type for each type of word
• EnumLOOKUP, VERB, NOUN

4
Rules section

• \n statelookup
• verb stateVERB
• noun stateNOUN
• a-zA-Z
• if(state!LOOKUP) addWord(state, yytext)
• else switch(lookUp(yytext))
• case VERB printf(yytext is verb) break
• case NOUN printf(yytext is noun) break
• default printf(Dont recognise this)
  break
• . / ignore the rest/

5
User Subroutines Section

• main()
• yylex()
• struct wordchar name, int word_type struct word
  next)
• struct word word_list // pointer to it
• addWord(type, word)
• look for it in table if found, report
• alternatively, malloc space save type and save
  word
• int lookUp(word)
• search through list with strcmp()
• if found return word type

6
Running it

• Input (Result in the table being added to)
• verb hit ate chased
• noun cat dog mouse
• Input (token recognition)
• hit ate cat
• mouse chased dog

7
Writing a Basic Parser

• Flex
• Token Identification
• yylex()
• Bison
• Token Structuring
• yyparse()
• Communication
• yyparse() called by main routine
• Calls yylex() expecting next token
• yylex() returns next token
• Passing control back to yyparse() with the token
• Implies Common understanding of Tokens

8
Description of .y file

• Definition Section
• Literal Code Block
• Definitions
• All TOKENS expected from yylex( )
• Any valid C identifier can be used but convention
  is UPPERCASE
• Rules Section
• Production Rules
• Action undertaken
• Position of Start Symbol
• Code Section
• main()
• Current Parse over on TOKEN zero
• Repeatedly calls yyparse()
• yyerror()
• When Impossible to Structure

9
Minimal Parser .y file

• / Limited sentence recogniser /
• includeltstdio.hgt
• token NOUN VERB ARTICLE
• sentence ARTICLE NOUN VERB ARTICLE NOUN
• printf(Is a valid Sentence!\n)
• extern FILE yyin
• main()
• do yyparse()
• while(!feof(yyin))
• yyerror(char s)
• fprintf(stderr, s\n, s)

10
Minimal Parser (.l file)

• / this demonstrates (very) simple recognition of
  a sentence /
• include y.tab.h /token defn.s from the
  parser/
• \t / ignore whitespace /
• cat
• dog
• mouse return(NOUN)
• ate
• hit return(VERB)
• the
• a return(ARTICLE)
•  a-zA-Z
• . / ignore any other characters /

11
Minimal Parser .y file

• / Limited sentence recogniser /
• includeltstdio.hgt
• token NOUN VERB ADJECTIVE ARTICLE
• sentence ARTICLE nounphrase VERB ARTICLE
  nounphrase
• printf(Is a valid Sentence!\n)
• nounphrase NOUN adjectives NOUN
• adjectives ADJECTIVE ADJECTIVE adjectives
• main()
• / as before /
• yyerror(char s)
• / as before /

12
Getting It Working

• flex eg2.l (presume lex file is eg2.l)
• bison d eg2.y (presume yacc file is eg2.y)
• cc c lex.yy.c eg2.tab.c
• cc o example lex.yy.o eg2.tab.o ll
• example
• Type in sentences!

13
Symbol Values / Actions

• Every Terminal Symbol has a Value
• NOUN could be cat, mouse or dog depending
  on language instance.
• Every VN has a value you assign to it
• The C code you enter forms it
• Expression NUMBER NUMBER 13
• NUMBER - NUMBER 1-2
• This can happen because of default int type
• Usually not so lucky!
• Different Symbols Different Types
• YYSTYPE union typedef created by BISON to
  contain them
• YYSTYPE used in y.tab.h for Token Definitions

14
Yacc Symbol Types (.y file)

• Double vbltable26
• union double dval
• int ival
• token ltivalgt NAME
• token ltdvalgt NUMBER
• type ltdvalgt exp
• statement NAME exp vblt1 3
• exp printf(g\n, 1)
• exp exp exp 1 3
• exp - exp 1 - 2
• NUMBER 1
• NAME vblt1

15
y.tab.h file Generated

• define NAME 257
• define NUMBER 258
• typedef union
• double dval
• int ival
• YYSTYPE
• Extern YYSTYPE yylval

16
.l file associated

• include y.tab.h
• include ltmath.hgt
• extern double vblt26
• 0-9(0-9\.0-9) yylval.dvalatof(yytext)
  
• return NUMBER
• \t / ignore /
• a-z yylval.ivalyytext0-a
• return NAME
• etc.