CO524 Programming Language Technology 20089

1
CO524Programming Language Technology 2008-9

• Simon Thompson sjt_at_kent.ac.uk
• Lecture 7

2
Overview

• JFlex lexical analyser generator.
• Principles.
• Examples.
• Patterns and actions.
• Infrastructure.
• Assessment 2.

3
JFlex

• Lexical analyser generator for Java.
• In a nutshell

• look for the longest match on a collection of
  regular definitions
• on matching, perform an associated action.

4
Not just lexical analysis

• The action could be to say "I have matched a
  number/string/ "
• but could also be any program action.
• We'll look at examples of text processing here,
  and lexical analysis later on.

5
JFlex files

• / User code /
• / Options and declarations /
• / Lexical rules /

• The three sections of the JFlex file are
  separated by .
• Comments between / / or following //.
• We will not use the first section in the first
  examples.

6
JFlex example1

• / Options etc. /
• standalone
• / Lexical rules /
• . System.out.println(
• "line matched")
• \n

• The option creates a standalone program.
• A rule contains a pattern and an associated
  action.

7
JFlex example1

• / Options etc. /
• standalone
• / Lexical rules /
• . System.out.println(
• "line matched")
• \n

• Input
• test1 test1
• test2
• test333
• Output
• line matched
• line matched
• line matched

8
JFlex generates

• a class Yylex
• the constructor Yylex
• methods including yylex()
• yylex()performs a single match plus the
  associated action
• standalone main program that repeatedly applies
  yylex() to the input file

9
Using JFlex standalone

• 1. Run jflex on the .lex file
• jflex example1.lex
• 2. Generates Yylex.java run javac on it
• javac Yylex.java
• 3. Run the compiled file, Yylex.class on an input
  file (called test here)
• java Yylex test

10
Using jflex for yourself

• JFlex is installed on raptor.
• Alternatively you can download it from
• http//www.jflex.de/
• and install it on your own machine.
• Details in the very detailed manual, which is
  available from the CO524 web page.

11
JFlex patterns
12
Java in the middle section

• standalone
• // This code included in the constructor Yylex
• init
• System.out.println("lexer constructed")
• init
• // This code included in the Yylex class
• int x0
• // This code executed at the end of the input
  file
• eof
• System.out.println("end of file matched")
• System.out.println("Total lines " x)
• eof

13
Counting lines example2

• // This code included in the constructor Yylex
• init
• System.out.println("lexer constructed")
• init
• // This code included in the Yylex class
• int x0
• // This code executed at the end of the input
  file
• eof
• System.out.println("end of file matched")
• System.out.println("Total lines " x)
• eof
• // Lexical rules
• . System.out.println("line of length
  "yylength()" matched") x
• \n

14
Count lines, words, chars wc

• // This code included in the Yylex class
• int chars0
• int words0
• int lines0
• // Code executed on reaching end of file.
• eof
• System.out.println("Total chars " chars)
• System.out.println("Total words " words)
• System.out.println("Total lines " lines)
• eof
• // Lexical rules
• \t chars chars yylength()
• \t\n chars chars yylength() words
  
• \n chars lines

15
States

• The program can move between different states,
  processing input according to the particular
  state that the system is in.
• States are declared by e.g.
• states TERSE, VERBOSE
• in the middle section.

16
States and rules verbose.lex

• ltYYINITIAL,VERBOSEgtterse\n
• System.out.println("") yybegin(TERSE)
• ltTERSEgtterse\n
• System.out.println("")
• ltYYINITIAL,VERBOSEgtverbose\n
• System.out.println("")
• ltTERSEgtverbose\n
• System.out.println("") yybegin(VERBOSE)
• ltTERSEgt.
• System.out.println("line")
• ltYYINITIAL,VERBOSEgt.
• System.out.println(yytext())
• \n

17
States and rules explained

• ltYYINITIAL,VERBOSEgtterse\n
• System.out.println("") yybegin(TERSE)
• If the system is in either of the states
  YYINITIAL (the initial state) or VERBOSE then
  process the line
• terse
• by outputting a blank line and changing state to
  TERSE.

18
verbose.lex in action

• Malesuada quis, egestas quis, wisi.
• metus a feugiat porttitor, dolor
• hendrerit sed, erat lorem ipsum.
• terse
• Vestibulum eget purus vitae
• vestibulum malesuada libero.
• verbose
• Fusce urna magna,neque eget lacus.
• terse
• Cras erat.
• Aliquam pede.
• verbose
• Vestibulum usce urna magna.

• Malesuada quis, egestas quis, wisi.
• metus a feugiat porttitor, dolor
• hendrerit sed, erat lorem ipsum.
• line
• line
• Fusce urna magna,neque eget lacus.
• line
• line
• Vestibulum usce urna magna.

19
Definitions

• A definition occurs in the middle section and has
  the form
• name regular-expression
• Examples
• digit 0-9
• digits digit

20
Assessment 2 text processing

• Write a JFlex file which builds a text processing
  system.
• Commands in the system have the form
• .XX
• on a line on their own.

21
Assessment 2 commands

• .LD
• Copy the text following, line by line.
• .CD
• Centre each of the following lines to line length
  LEN (if possible).

• .LP
• Fill the following lines to at most LEN.
• .LJ
• Fill and justify the following lines to length
  LEN.

22
Filling and justification

• .LP
• Malesuada
• quis,
• egestas quis, wisi.
• metus a feugiat porttitor,
• dolor
• hendrerit sed, erat lorem
• .LJ
• Malesuada
• quis,
• egestas quis, wisi.
• metus a feugiat porttitor,
• dolor
• hendrerit sed, erat lorem

• Malesuada quis,
• egestas quis, wisi.
• metus a feugiat
• porttitor, dolor
• hendrerit sed, erat
• lorem
• Malesuada quis,
• egestas quis, wisi.
• metus a feugiat
• porttitor, dolor
• hendrerit sed, erat
• lorem

23
Assessment 2

• Supply the .lex file together with output for the
  test file, test.txt (see the web site).
• Due date Thursday week 5 (30 October).
• Submission arrangements to be confirmed.

24
Important

• Please use the forum and the anonymous question
  page for advice on the assessment.