lex Command Purpose Generates a C Language program that matches patterns for simple lexical analysis of an input stream. Syntax lex [ -t ] [ -v | -n ] [ File ... ] Description The lex command reads File or standard input, generates a C language program, and writes it to a file named lex.yy.c. This file, lex.yy.c, is a compilable C language program. The lex command uses rules and actions contained in File to gen- erate a program, lex.yy.c, which can be compiled with the cc com- mand. The compiled lex.yy.c can then receive input, break the input into the logical pieces defined by the rules in File, and run program fragments contained in the actions in File. The generated program is a C language function called yylex. The lex command stores the yylex function in a file named lex.yy.c. You can use the yylex function alone to recognize simple, one- word input, or you can use it with other C Language programs to perform more difficult input analysis functions. For example, you can use the lex command to generate a program that simplifies an input stream before sending it to a parser program generated by the yacc command. The yylex function analyzes the input stream using a program structure called a finite state machine. This structure allows the program to exist in only one state (or condition) at a time. There is a finite number of states allowed. The rules in File determine how the program moves from one state to another. If you do not specify a File, the lex command reads standard in- put. It treats multiple files as a single file. Note: Since the lex command uses fixed names for intermediate and output files, you can have only one program generated by lex in a given directory. lex Specification File The input file can contain three sections: definitions, rules, and user subroutines. Each section must be separated from the others by a line containing only the delimiter, %% (double per- cent signs). The format is: definitions %% rules %% user subroutines The purpose and format of each are described in the following sections. Definitions If you want to use variables in your rules, you must define them in this section. The variables make up the left column, and their definitions make up the right column. For example, if you want to define D as a numerical digit, you would write the fol- lowing: D [0-9] You can use a defined variable in the rules section by enclosing the variable name in {} (braces), for example: {D} In the definitions section, you can set table sizes for the resulting finite state machine. The default sizes are large enough for small programs. You may want to set larger sizes for more complex programs. %an Number of transitions is n (default 5000) %en Number of parse tree nodes is n (default 2000) %hn Number of multibyte character output slots (default is 0) %kn Number of packed character classes (default 1000) %mn Number of multibyte "character class" character output slots (default is 0) %nn Number of states is n (default 2500) %on Number of output slots (default 5000, minimum 257) %pn Number of positions is n (default 5000) %vp Percentage of slots vacant in the hash tables controlled by %h and %m (default 20, range 0 <= P < 100) If multibyte characters appear in regular expression strings, you may need to reset the output array size with the %o argument (possibly to array sizes in the range 10,000 to 20,000). This reset reflects the much larger number of characters relative to the number of single-byte characters. If multibyte characters appear in regular expressions, you must set the multibyte hash table sizes with the %h and %m arguments to sizes greater than the total number of multibyte characters contained in the lex file. Rules Once you have defined your terms, you can write the rules sec- tion. It contains strings and expressions to be matched in File yylex, and C commands to execute when a match is made. This sec- tion is required, and it must be preceded by the delimiter %% (double percent signs), whether or not you have a definitions section. The lex command does not recognize your rules without this delimiter. In this section, the left column contains the pattern in the form of a regular expression, to be recognized in an input file to the yylex subroutine. The right column contains the C program frag- ment executed when that pattern is recognized, called an action. When the lexical analyzer finds a match for the regular expres- sion, the lexical analyzer executes the action associated with that regular expression. Patterns can include extended characters with one exception: these characters cannot appear in range specifications within character class expressions surrounded by [ ] (brackets). If multibyte locales are installed on your system, patterns can in- clude characters that are part of the installed code set, with this same exception. The columns are separated by a tab or blanks. For example, if you want to search files for the keyword KEY, you can write the following: (KEY) printf("found KEY"); If you include this rule in File, the yylex lexical analyzer matches the pattern KEY and runs the printf subroutine. Each pattern can have a corresponding action, that is, a C com- mand to execute when the pattern is matched. Each statement must end with a ; (semicolon). If you use more than one statement in an action, you must enclose all of them in { } (braces). A second delimiter, %%, must follow the rules section if you have a user subroutine section. Without a specified action for a pat- tern match, the lexical analyzer copies the input pattern to the output without changing it. When the yylex lexical analyzer matches a string in the input stream, it copies the matched string to an external character ar- ray, yytext, before it executes any commands in the rules section. If a line begins with only a blank, lex copies it to the output file, lex.yy.c. If the line is in the declarations section of File, lex copies it to the declarations section of lex.yy.c. If the line is in the rules section, lex copies it to the program code section of lex.yy.c. For information on how to form regular expressions, see "Regular Expressions in the lex Command" in AIX Version 3.2 General Pro- gramming Concepts. User Subroutines The lex library defines the following subroutines as macros that you can use in the rules section of the lex specification file: input Reads a byte from yyin. unput Replaces a byte after it has been read. output Writes an output byte to yyout. winput Reads a multibyte character from yyin. wunput Replaces a multibyte character after it has been read. woutput Writes a multibyte output character to yyout. yysetlocale Calls the setlocale (LC_ALL, ""); subroutine to determine the current locale. The winput, wunput, and woutput macros have been defined to use the yywinput, yywunput, and yywoutput subroutines coded in the lex.yy.c file. For compatibility, these yy subroutines subse- quently use the input, unput, and output subroutines to read, re- place, and write the necessary number of bytes in a complete mul- tibyte character. You can override these macros by writing your own code for these routines in the user subroutines section. But if you write your own, you must undefine these macros in the definition section as follows: %{ #undef input #undef unput #undef output #undef winput #undef wunput #undef woutput #undef yysetlocale %} There is no main subroutine in lex.yy.c because the lex library contains the main subroutine that calls the yylex lexical analyzer. Therefore, if you do not include main() in the user subroutines section, when you compile lex.yy.c, you must enter cc lex.yy.c -ll, where ll calls the lex library. External names generated by the lex command all begin with the preface yy, as in yyin, yyout, yylex, and yytext. Finite State Machine The default skeleton for the finite state machine is defined in /usr/ccs/lib/lex/ncform. The user may use a personally config- ured finite state machine by setting an environment variable LEXER=PATH. The PATH variable designates the user-defined finite state machine path and file name. The lex command checks the en- vironment for this variable and, if it is set, uses the supplied path. Putting Blanks in an Expression Normally, blanks or tabs end a rule and therefore, the expression that defines a rule. However, you can enclose the blanks or tab characters in " " (quotation marks) to include them in the ex- pression. Use quotes around all blanks in expressions that are not already within sets of [ ] (brackets). Other Special Characters The lex program recognizes many of the normal C language special characters. These character sequences are: Sequence Meaning \n Newline. (Do not use the actual new-line character in an ex- pression.) \t Tab. \b Backspace. \\ Backslash. When using these special characters in an expression, you do not need to enclose them in quotes. Every character, except these special characters and the operator symbols described in " Regu- lar Expressions in the lex Command" in AIX Version 3.2 General Programming Concepts, is always a text character. Matching Rules When more than one expression can match the current input, lex chooses in the following order: 1. The longest match. 2. Among rules that match the same number of characters, the rule that occurs first. For example, if the rules integer keyword action...; [a-z]+ identifier action...; are given in that order, and integers is the input word, lex matches the input as an identifier, because [a-z]+ matches eight characters while integer matches only seven. However, if the in- put is integer, both rules match seven characters. lex selects the keyword rule because it occurs first. A shorter input, such as int, does not match the expression integer, and so lex selects the identifier rule. Matching a String Using Wildcard Characters Because lex chooses the longest match first, do not use rules containing expressions like .*. For example: '.*' might seem like a good way to recognize a string in single quotes. However, the lexical analyzer reads far ahead, looking for a distant single quote to complete the long match. If a lex- ical analyzer with such a rule gets the following input: 'first' quoted string here, 'second' here it matches: 'first' quoted string here, 'second' To find the smaller strings, first and second, use the following rule: '[\^'\n]*' This rule stops after 'first'. Errors of this type are not far reaching, because the . (period) operator does not match a new-line character. Therefore, expres- sions like .* (period asterisk)stop on the current line. Do not try to defeat this with expressions like [.\n] +. The lexical analyzer tries to read the entire input file and an internal buffer overflow occurs. Finding Strings within Strings The lex program partitions the input stream, and does not search for all possible matches of each expression. Each character is accounted for once and only once. For example, to count oc- currences of both she and he in an input text, try the following rules: she s++ he h++ \n | . ; where the last two rules ignore everything besides he and she. However, because she includes he, lex does not recognize the in- stances of he that are included in she. To override this choice, use the action REJECT. This directive tells lex to go to the next rule. lex then adjusts the position of the input pointer to where it was before the first rule was executed, and executes the second choice rule. For example, to count the included instances of he, use the following rules: she {s++; REJECT;} he {h++; REJECT;} \n | . ; After counting the occurrences of she, lex rejects the input stream and then counts the occurrences of he. Because in this case she includes he but not vice versa, you can omit the REJECT action on he. In other cases, it may be difficult to determine which input characters are in both classes. In general, REJECT is useful whenever the purpose of lex is not to partition the input stream but to detect all examples of some items in the input, and the instances of these items may overlap or include each other. Flags -n Suppresses the statistics summary. When you set your own table sizes for the finite state machine, the lex command au- tomatically produces this summary if you do not select this flag. -t Writes lex.yy.c to standard output instead of to a file. -v Provides a one-line summary of the generated finite-state- machine statistics. Exit Status This command returns the following exit values: 0 Successful completion. >0 An error occurred. Examples 1. To draw lex instructions from the file lexcommands and place the output in lex.yy.c, use the following command: lex lexcommands 2. To create a lex program that converts uppercase to lowercase, removes blanks at the end of a line, and replaces multiple blanks by single blanks, including the following in a lex command file: %% [A-Z] putchar(yytext[0]+ 'a'-'A'); [ ]+$ ; [ ]+ putchar(' '); Implementation Specifics This command is part of Base Application Development Toolkit. Files /usr/ccs/lib/libl.a Contains the run-time library. /usr/ccs/lib/lex/ncform Defines a finite state machine. Related Information The yacc command. The Example Program for the lex and yacc Programs in AIX Version 3.2 General Programming Concepts. Creating an Input Language with the lex and yacc Commands in AIX Version 3.2 General Programming Concepts.