1 '\" te
   2 .\"  Copyright (c) 1992, X/Open Company Limited  All Rights Reserved  Portions Copyright (c) 1997, Sun Microsystems, Inc.  All Rights Reserved
   3 .\" Sun Microsystems, Inc. gratefully acknowledges The Open Group for permission to reproduce portions of its copyrighted documentation. Original documentation from The Open Group can be obtained online at
   4 .\" http://www.opengroup.org/bookstore/.
   5 .\" The Institute of Electrical and Electronics Engineers and The Open Group, have given us permission to reprint portions of their documentation. In the following statement, the phrase "this text" refers to portions of the system documentation. Portions of this text are reprinted and reproduced in electronic form in the Sun OS Reference Manual, from IEEE Std 1003.1, 2004 Edition, Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 6, Copyright (C) 2001-2004 by the Institute of Electrical and Electronics Engineers, Inc and The Open Group. In the event of any discrepancy between these versions and the original IEEE and The Open Group Standard, the original IEEE and The Open Group Standard is the referee document. The original Standard can be obtained online at http://www.opengroup.org/unix/online.html.
   6 .\"  This notice shall appear on any product containing this material.
   7 .\" The contents of this file are subject to the terms of the Common Development and Distribution License (the "License").  You may not use this file except in compliance with the License.
   8 .\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing.  See the License for the specific language governing permissions and limitations under the License.
   9 .\" When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE.  If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner]
  10 .TH LEX 1 "Aug 22, 1997"
  11 .SH NAME
  12 lex \- generate programs for lexical tasks
  13 .SH SYNOPSIS
  14 .LP
  15 .nf
  16 \fBlex\fR [\fB-cntv\fR] [\fB-e\fR | \fB-w\fR] [\fB-V\fR \fB-Q\fR [y | n]] [\fIfile\fR]...
  17 .fi
  18 
  19 .SH DESCRIPTION
  20 .sp
  21 .LP
  22 The \fBlex\fR utility generates C programs to be used in lexical processing of
  23 character input, and that can be used as an interface to \fByacc\fR. The C
  24 programs are generated from \fBlex\fR source code and conform to the ISO C
  25 standard. Usually, the \fBlex\fR utility writes the program it generates to the
  26 file \fBlex.yy.c\fR. The state of this file is unspecified if \fBlex\fR exits
  27 with a non-zero exit status. See \fBEXTENDED DESCRIPTION\fR for a complete
  28 description of the \fBlex\fR input language.
  29 .SH OPTIONS
  30 .sp
  31 .LP
  32 The following options are supported:
  33 .sp
  34 .ne 2
  35 .na
  36 \fB\fB-c\fR \fR
  37 .ad
  38 .RS 12n
  39 Indicates C-language action (default option).
  40 .RE
  41 
  42 .sp
  43 .ne 2
  44 .na
  45 \fB\fB-e\fR \fR
  46 .ad
  47 .RS 12n
  48 Generates a program that can handle \fBEUC\fR characters (cannot be used with
  49 the \fB-w\fR option). \fByytext[\|]\fR is of type \fBunsigned char[\|]\fR.
  50 .RE
  51 
  52 .sp
  53 .ne 2
  54 .na
  55 \fB\fB-n\fR \fR
  56 .ad
  57 .RS 12n
  58 Suppresses the summary of statistics usually written with the \fB-v\fR option.
  59 If no table sizes are specified in the \fBlex\fR source code and the \fB-v\fR
  60 option is not specified, then \fB-n\fR is implied.
  61 .RE
  62 
  63 .sp
  64 .ne 2
  65 .na
  66 \fB\fB-t\fR \fR
  67 .ad
  68 .RS 12n
  69 Writes the resulting program to standard output instead of \fBlex.yy.c\fR.
  70 .RE
  71 
  72 .sp
  73 .ne 2
  74 .na
  75 \fB\fB-v\fR \fR
  76 .ad
  77 .RS 12n
  78 Writes a summary of \fBlex\fR statistics to the standard error. (See the
  79 discussion of \fBlex\fR table sizes under the heading \fBDefinitions in
  80 lex\fR.) If table sizes are specified in the \fBlex\fR source code, and if the
  81 \fB-n\fR option is not specified, the \fB-v\fR option may be enabled.
  82 .RE
  83 
  84 .sp
  85 .ne 2
  86 .na
  87 \fB\fB-w\fR \fR
  88 .ad
  89 .RS 12n
  90 Generates a program that can handle \fBEUC\fR characters (cannot be used with
  91 the \fB-e\fR option). Unlike the \fB-e\fR option, \fByytext[\|]\fR is of type
  92 \fBwchar_t[\|]\fR.
  93 .RE
  94 
  95 .sp
  96 .ne 2
  97 .na
  98 \fB\fB-V\fR \fR
  99 .ad
 100 .RS 12n
 101 Prints out version information on standard error.
 102 .RE
 103 
 104 .sp
 105 .ne 2
 106 .na
 107 \fB\fB\fR\fB-Q\fR\fB[y|n]\fR \fR
 108 .ad
 109 .RS 12n
 110 Prints out version information to output file \fBlex.yy.c\fR by using
 111 \fB-Qy\fR. The \fB-Qn\fR option does not print out version information and is
 112 the default.
 113 .RE
 114 
 115 .SH OPERANDS
 116 .sp
 117 .LP
 118 The following operand is supported:
 119 .sp
 120 .ne 2
 121 .na
 122 \fB\fIfile\fR \fR
 123 .ad
 124 .RS 9n
 125 A pathname of an input file. If more than one such \fIfile\fR is specified, all
 126 files will be concatenated to produce a single \fBlex\fR program. If no
 127 \fIfile\fR operands are specified, or if a \fIfile\fR operand is \fB\(mi\fR,
 128 the standard input will be used.
 129 .RE
 130 
 131 .SH OUTPUT
 132 .sp
 133 .LP
 134 The \fBlex\fR output files are described below.
 135 .SS "Stdout"
 136 .sp
 137 .LP
 138 If the \fB-t\fR option is specified, the text file of C source code output of
 139 \fBlex\fR will be written to standard output.
 140 .SS "Stderr"
 141 .sp
 142 .LP
 143 If the \fB-t\fR option is specified informational, error and warning messages
 144 concerning the contents of \fBlex\fR source code input will be written to the
 145 standard error.
 146 .sp
 147 .LP
 148 If the \fB-t\fR option is not specified:
 149 .RS +4
 150 .TP
 151 1.
 152 Informational error and warning messages concerning the contents of
 153 \fBlex\fR source code input will be written to either the standard output or
 154 standard error.
 155 .RE
 156 .RS +4
 157 .TP
 158 2.
 159 If the \fB-v\fR option is specified and the \fB-n\fR option is not
 160 specified, \fBlex\fR statistics will also be written to standard error. These
 161 statistics may also be generated if table sizes are specified with a \fB%\fR
 162 operator in the \fBDefinitions\fR \fBin\fR \fBlex\fR section (see \fBEXTENDED
 163 DESCRIPTION\fR), as long as the \fB-n\fR option is not specified.
 164 .RE
 165 .SS "Output Files"
 166 .sp
 167 .LP
 168 A text file containing C source code will be written to \fBlex.yy.c\fR, or to
 169 the standard output if the \fB-t\fR option is present.
 170 .SH EXTENDED DESCRIPTION
 171 .sp
 172 .LP
 173 Each input file contains \fBlex\fR source code, which is a table of regular
 174 expressions with corresponding actions in the form of C program fragments.
 175 .sp
 176 .LP
 177 When \fBlex.yy.c\fR is compiled and linked with the \fBlex\fR library (using
 178 the \fB\fR\fB-l\fR\fB l\fR operand with \fBc89\fR or \fBcc\fR), the resulting
 179 program reads character input from the standard input and partitions it into
 180 strings that match the given expressions.
 181 .sp
 182 .LP
 183 When an expression is matched, these actions will occur:
 184 .RS +4
 185 .TP
 186 .ie t \(bu
 187 .el o
 188 The input string that was matched is left in \fIyytext\fR as a null-terminated
 189 string; \fIyytext\fR is either an external character array or a pointer to a
 190 character string. As explained in \fBDefinitions in lex\fR, the type can be
 191 explicitly selected using the \fB%array\fR or \fB%pointer\fR declarations, but
 192 the default is \fB%array\fR.
 193 .RE
 194 .RS +4
 195 .TP
 196 .ie t \(bu
 197 .el o
 198 The external \fBint\fR \fIyyleng\fR is set to the length of the matching
 199 string.
 200 .RE
 201 .RS +4
 202 .TP
 203 .ie t \(bu
 204 .el o
 205 The expression's corresponding program fragment, or action, is executed.
 206 .RE
 207 .sp
 208 .LP
 209 During pattern matching, \fBlex\fR searches the set of patterns for the single
 210 longest possible match. Among rules that match the same number of characters,
 211 the rule given first will be chosen.
 212 .sp
 213 .LP
 214 The general format of \fBlex\fR source is:
 215 .sp
 216 .in +2
 217 .nf
 218 \fIDefinitions\fR
 219 %%
 220 \fIRules\fR
 221 %%
 222 \fIUser Subroutines\fR
 223 .fi
 224 .in -2
 225 
 226 .sp
 227 .LP
 228 The first \fB%%\fR is required to mark the beginning of the rules (regular
 229 expressions and actions); the second \fB%%\fR is required only if user
 230 subroutines follow.
 231 .sp
 232 .LP
 233 Any line in the \fBDefinitions\fR \fBin\fR \fBlex\fR section beginning with a
 234 blank character will be assumed to be a C program fragment and will be copied
 235 to the external definition area of the \fBlex.yy.c\fR file. Similarly, anything
 236 in the \fBDefinitions\fR \fBin\fR \fBlex\fR section included between delimiter
 237 lines containing only \fB%{\fR and \fB%}\fR will also be copied unchanged to
 238 the external definition area of the \fBlex.yy.c\fR file.
 239 .sp
 240 .LP
 241 Any such input (beginning with a blank character or within \fB%{\fR and
 242 \fB%}\fR delimiter lines) appearing at the beginning of the \fIRules\fR section
 243 before any rules are specified will be written to \fBlex.yy.c\fR after the
 244 declarations of variables for the \fByylex\fR function and before the first
 245 line of code in \fByylex\fR. Thus, user variables local to \fByylex\fR can be
 246 declared here, as well as application code to execute upon entry to
 247 \fByylex\fR.
 248 .sp
 249 .LP
 250 The action taken by \fBlex\fR when encountering any input beginning with a
 251 blank character or within \fB%{\fR and \fB%}\fR delimiter lines appearing in
 252 the \fIRules\fR section but coming after one or more rules is undefined. The
 253 presence of such input may result in an erroneous definition of the \fByylex\fR
 254 function.
 255 .SS "Definitions in lex"
 256 .sp
 257 .LP
 258 \fBDefinitions\fR \fBin\fR \fBlex\fR appear before the first \fB%%\fR
 259 delimiter. Any line in this section not contained between \fB%{\fR and \fB%}\fR
 260 lines and not beginning with a blank character is assumed to define a \fBlex\fR
 261 substitution string. The format of these lines is:
 262 .sp
 263 .in +2
 264 .nf
 265 \fIname   substitute\fR
 266 .fi
 267 .in -2
 268 .sp
 269 
 270 .sp
 271 .LP
 272 If a \fIname\fR does not meet the requirements for identifiers in the ISO C
 273 standard, the result is undefined. The string \fIsubstitute\fR will replace the
 274 string \fI{\fR \fIname\fR \fI}\fR when it is used in a rule. The \fIname\fR
 275 string is recognized in this context only when the braces are provided and when
 276 it does not appear within a bracket expression or within double-quotes.
 277 .sp
 278 .LP
 279 In the \fBDefinitions\fR \fBin\fR \fBlex\fR section, any line beginning with a
 280 \fB%\fR (percent sign) character and followed by an alphanumeric word beginning
 281 with either \fBs\fR or \fBS\fR defines a set of start conditions. Any line
 282 beginning with a \fB%\fR followed by a word beginning with either \fBx\fR or
 283 \fBX\fR defines a set of exclusive start conditions. When the generated scanner
 284 is in a \fB%s\fR state, patterns with no state specified will be also active;
 285 in a \fB%x\fR state, such patterns will not be active. The rest of the line,
 286 after the first word, is considered to be one or more blank-character-separated
 287 names of start conditions. Start condition names are constructed in the same
 288 way as definition names. Start conditions can be used to restrict the matching
 289 of regular expressions to one or more states as described in \fBRegular
 290 expressions in lex\fR.
 291 .sp
 292 .LP
 293 Implementations accept either of the following two mutually exclusive
 294 declarations in the \fBDefinitions\fR \fBin\fR \fBlex\fR section:
 295 .sp
 296 .ne 2
 297 .na
 298 \fB\fB%array\fR \fR
 299 .ad
 300 .RS 13n
 301 Declare the type of \fIyytext\fR to be a null-terminated character array.
 302 .RE
 303 
 304 .sp
 305 .ne 2
 306 .na
 307 \fB\fB%pointer\fR \fR
 308 .ad
 309 .RS 13n
 310 Declare the type of \fIyytext\fR to be a pointer to a null-terminated character
 311 string.
 312 .RE
 313 
 314 .sp
 315 .LP
 316 \fBNote:\fR When using the \fB%pointer\fR option, you may not also use the
 317 \fByyless\fR function to alter \fIyytext\fR.
 318 .sp
 319 .LP
 320 \fB%array\fR is the default. If \fB%array\fR is specified (or neither
 321 \fB%array\fR nor \fB%pointer\fR is specified), then the correct way to make an
 322 external reference to \fIyyext\fR is with a declaration of the form:
 323 .sp
 324 .LP
 325 \fBextern char\fR\fI yytext\fR\fB[\|]\fR
 326 .sp
 327 .LP
 328 If \fB%pointer\fR is specified, then the correct external reference is of the
 329 form:
 330 .sp
 331 .LP
 332 \fBextern char *\fR\fIyytext\fR\fB;\fR
 333 .sp
 334 .LP
 335 \fBlex\fR will accept declarations in the \fBDefinitions in lex\fR section for
 336 setting certain internal table sizes. The declarations are shown in the
 337 following table.
 338 .sp
 339 .LP
 340 \fBTable\fR \fBSize\fR \fBDeclaration\fR \fBin\fR \fBlex\fR
 341 .sp
 342 
 343 .sp
 344 .TS
 345 box;
 346 c c c
 347 l l l .
 348 \fBDeclaration\fR       \fBDescription\fR       \fBDefault\fR
 349 _
 350 \fB%p\fR\fIn\fR Number of positions     2500
 351 \fB%n\fR\fIn\fR Number of states        500
 352 \fB%a\fR\fI n\fR        Number of transitions   2000
 353 \fB%e\fR\fIn\fR Number of parse tree nodes      1000
 354 \fB%k\fR\fIn\fR Number of packed character classes      10000
 355 \fB%o\fR\fIn\fR Size of the output array        3000
 356 .TE
 357 
 358 .sp
 359 .LP
 360 Programs generated by \fBlex\fR need either the \fB-e\fR or \fB-w\fR option to
 361 handle input that contains \fBEUC\fR characters from supplementary codesets. If
 362 neither of these options is specified, \fByytext\fR is of the type
 363 \fBchar[\|]\fR, and the generated program can handle only \fBASCII\fR
 364 characters.
 365 .sp
 366 .LP
 367 When the \fB-e\fR option is used, \fByytext\fR is of the type \fBunsigned\fR
 368 \fBchar[\|]\fR and \fByyleng\fR gives the total number of \fIbytes\fR in the
 369 matched string. With this option, the macros \fBinput()\fR,
 370 \fBunput(\fIc\fR)\fR, and \fBoutput(\fIc\fR)\fR should do a byte-based
 371 \fBI/O\fR in the same way as with the regular \fBASCII\fR \fBlex\fR. Two more
 372 variables are available with the \fB-e\fR option, \fByywtext\fR and
 373 \fByywleng\fR, which behave the same as \fByytext\fR and \fByyleng\fR would
 374 under the \fB-w\fR option.
 375 .sp
 376 .LP
 377 When the \fB-w\fR option is used, \fByytext\fR is of the type \fBwchar_t[\|]\fR
 378 and \fByyleng\fR gives the total number of \fIcharacters\fR in the matched
 379 string.  If you supply your own \fBinput()\fR, \fBunput(\fIc\fR)\fR, or
 380 \fBoutput(\fR\fIc\fR\fB)\fR macros with this option, they must return or accept
 381 \fBEUC\fR characters in the form of wide character (\fBwchar_t\fR). This allows
 382 a different interface between your program and the lex internals, to expedite
 383 some programs.
 384 .SS "Rules in lex"
 385 .sp
 386 .LP
 387 The \fBRules\fR \fBin\fR \fBlex\fR source files are a table in which the left
 388 column contains regular expressions and the right column contains actions (C
 389 program fragments) to be executed when the expressions are recognized.
 390 .sp
 391 .in +2
 392 .nf
 393 \fIERE action\fR
 394 \fIERE action\fR
 395 \&...
 396 .fi
 397 .in -2
 398 
 399 .sp
 400 .LP
 401 The extended regular expression (ERE) portion of a row will be separated from
 402 \fIaction\fR by one or more blank characters. A regular expression containing
 403 blank characters is recognized under one of the following conditions:
 404 .RS +4
 405 .TP
 406 .ie t \(bu
 407 .el o
 408 The entire expression appears within double-quotes.
 409 .RE
 410 .RS +4
 411 .TP
 412 .ie t \(bu
 413 .el o
 414 The blank characters appear within double-quotes or square brackets.
 415 .RE
 416 .RS +4
 417 .TP
 418 .ie t \(bu
 419 .el o
 420 Each blank character is preceded by a backslash character.
 421 .RE
 422 .SS "User Subroutines in lex"
 423 .sp
 424 .LP
 425 Anything in the user subroutines section will be copied to \fBlex.yy.c\fR
 426 following \fByylex\fR.
 427 .SS "Regular Expressions     in lex"
 428 .sp
 429 .LP
 430 The \fBlex\fR utility supports the set of Extended Regular Expressions (EREs)
 431 described on \fBregex\fR(5) with the following additions and exceptions to the
 432 syntax:
 433 .sp
 434 .ne 2
 435 .na
 436 \fB\fB\|.\|.\|.\fR \fR
 437 .ad
 438 .RS 14n
 439 Any string enclosed in double-quotes will represent the characters within the
 440 double-quotes as themselves, except that backslash escapes (which appear in the
 441 following table) are recognized. Any backslash-escape sequence is terminated by
 442 the closing quote. For example, "\|\e\|01""1" represents a single string: the
 443 octal value 1 followed by the character 1.
 444 .RE
 445 
 446 .sp
 447 .LP
 448 \fI<\fR\fIstate\fR\fI>\fR\fIr\fR
 449 .sp
 450 .ne 2
 451 .na
 452 \fB<\fIstate1\fR, \fIstate2\fR, \|.\|.\|.\|>\fIr\fR\fR
 453 .ad
 454 .sp .6
 455 .RS 4n
 456 The regular expression \fIr\fR will be matched only when the program is in one
 457 of the start conditions indicated by \fIstate\fR, \fIstate1\fR, and so forth.
 458 For more information, see \fBActions in lex\fR. As an exception to the
 459 typographical conventions of the rest of this document, in this case
 460 <\fIstate\fR> does not represent a metavariable, but the literal angle-bracket
 461 characters surrounding a symbol. The start condition is recognized as such only
 462 at the beginning of a regular expression.
 463 .RE
 464 
 465 .sp
 466 .ne 2
 467 .na
 468 \fB\fIr\fR/\fIx\fR \fR
 469 .ad
 470 .sp .6
 471 .RS 4n
 472 The regular expression \fIr\fR will be matched only if it is followed by an
 473 occurrence of regular expression \fIx\fR. The token returned in \fIyytext\fR
 474 will only match \fIr\fR. If the trailing portion of \fIr\fR matches the
 475 beginning of \fIx\fR, the result is unspecified. The \fIr\fR expression cannot
 476 include further trailing context or the \fB$\fR (match-end-of-line) operator;
 477 \fIx\fR cannot include the \fB^\fR (match-beginning-of-line) operator, nor
 478 trailing context, nor the \fB$\fR operator. That is, only one occurrence of
 479 trailing context is allowed in a \fBlex\fR regular expression, and the \fB^\fR
 480 operator only can be used at the beginning of such an expression. A further
 481 restriction is that the trailing-context operator \fB/\fR (slash) cannot be
 482 grouped within parentheses.
 483 .RE
 484 
 485 .sp
 486 .ne 2
 487 .na
 488 \fB\fB{\fR\fIname\fR\fB}\fR \fR
 489 .ad
 490 .sp .6
 491 .RS 4n
 492 When \fIname\fR is one of the substitution symbols from the \fIDefinitions\fR
 493 section, the string, including the enclosing braces, will be replaced by the
 494 \fIsubstitute\fR value. The \fIsubstitute\fR value will be treated in the
 495 extended regular expression as if it were enclosed in parentheses. No
 496 substitution will occur if \fB{\fR\fIname\fR\fB}\fR occurs within a bracket
 497 expression or within double-quotes.
 498 .RE
 499 
 500 .sp
 501 .LP
 502 Within an \fBERE,\fR a backslash character (\fB\|\e\e\fR, \fB\e\|a\fR,
 503 \fB\e\|b\fR, \fB\e\|f\fR, \fB\e\|n\fR, \fB\e\|r\fR, \fB\e\|t\fR, \fB\e\|v\fR)
 504 is considered to begin an escape sequence. In addition, the escape sequences in
 505 the following table will be recognized.
 506 .sp
 507 .LP
 508 A literal newline character cannot occur within an \fBERE;\fR the escape
 509 sequence \fB\e\|n\fR can be used to represent a newline character. A newline
 510 character cannot be matched by a period operator.
 511 .sp
 512 .LP
 513 \fBEscape Sequences in lex\fR
 514 .sp
 515 
 516 .sp
 517 .TS
 518 box;
 519 c c c
 520 c c c .
 521 Escape Sequences in lex
 522 _
 523 Escape Sequence Description     Meaning
 524 _
 525 \e\fIdigits\fR  T{
 526 A backslash character followed by the longest sequence of one, two or three octal-digit characters (01234567). Ifall of the digits are 0, (that is, representation of the NUL character), the behavior is undefined.
 527 T}      T{
 528 The character whose encoding is represented by the one-, two- or three-digit octal integer. Multi-byte characters require multiple, concatenated escape sequences of this type, including the leading \e for each byte.
 529 T}
 530 _
 531 \e\fBx\fR\fIdigits\fR   T{
 532 A backslash character followed by the longest sequence of hexadecimal-digit characters (01234567abcdefABCDEF). If all of the digits are 0, (that is, representation of the NUL character), the behavior is undefined.
 533 T}      T{
 534 The character whose encoding is represented by the hexadecimal integer.
 535 T}
 536 _
 537 \e\fIc\fR       T{
 538 A backslash character followed by any character not described in this table.  (\e\e, \ea, \eb, \ef, \een, \er, \et, \ev).
 539 T}      The character c, unchanged.
 540 .TE
 541 
 542 .sp
 543 .LP
 544 The order of precedence given to extended regular expressions for \fBlex\fR is
 545 as shown in the following table, from high to low.
 546 .sp
 547 .ne 2
 548 .na
 549 \fB\fBNote\fR: \fR
 550 .ad
 551 .RS 10n
 552 The escaped characters entry is not meant to imply that these are operators,
 553 but they are included in the table to show their relationships to the true
 554 operators. The start condition, trailing context and anchoring notations have
 555 been omitted from the table because of the placement restrictions described in
 556 this section; they can only appear at the beginning or ending of an \fBERE.\fR
 557 .RE
 558 
 559 .sp
 560 
 561 .sp
 562 .TS
 563 box;
 564 c c
 565 l l .
 566 ERE Precedence in lex
 567 _
 568 \fIcollation-related bracket symbols\fR \fB[= =]  [: :]  [. .]\fR
 569 \fIescaped characters\fR        \fB\e<\fR\fIspecial character\fR>
 570 \fIbracket expression\fR        \fB[ ]\fR
 571 \fIquoting\fR   \fB".\|.\|."\fR
 572 \fIgrouping\fR  \fB()\fR
 573 \fIdefinition\fR        \fB{\fR\fIname\fR}
 574 \fIsingle-character RE duplication\fR   \fB* + ?\fR
 575 \fIconcatenation\fR     
 576 \fIinterval expression\fR       \fB{\fR\fIm\fR,\fIn\fR}
 577 \fIalternation\fR       \fB|\fR
 578 .TE
 579 
 580 .sp
 581 .LP
 582 The \fBERE\fR anchoring operators (\fB\|^\fR and \fB$\fR\|) do not appear in
 583 the table. With \fBlex\fR regular expressions, these operators are restricted
 584 in their use: the \fB^\fR operator can only be used at the beginning of an
 585 entire regular expression, and the \fB$\fR operator only at the end. The
 586 operators apply to the entire regular expression. Thus, for example, the
 587 pattern (\fB^abc)|(def$\fR) is undefined; it can instead be written as two
 588 separate rules, one with the regular expression \fB^abc\fR and one with
 589 \fBdef$\fR, which share a common action via the special \fB|\fR action (see
 590 below). If the pattern were written \fB^abc|def$\fR, it would match either of
 591 \fBabc\fR or \fBdef\fR on a line by itself.
 592 .sp
 593 .LP
 594 Unlike the general \fBERE\fR rules, embedded anchoring is not allowed by most
 595 historical \fBlex\fR implementations. An example of embedded anchoring would be
 596 for patterns such as (^)foo($) to match \fBfoo\fR when it exists as a complete
 597 word. This functionality can be obtained using existing \fBlex\fR features:
 598 .sp
 599 .in +2
 600 .nf
 601 ^foo/[ \e\|n]|
 602 " foo"/[ \e\|n]    /* found foo as a separate word */
 603 .fi
 604 .in -2
 605 
 606 .sp
 607 .LP
 608 Notice also that \fB$\fR is a form of trailing context (it is equivalent to
 609 \fB/\e\|n\fR and as such cannot be used with regular expressions containing
 610 another instance of the operator (see the preceding discussion of trailing
 611 context).
 612 .sp
 613 .LP
 614 The additional regular expressions trailing-context operator \fB/\fR (slash)
 615 can be used as an ordinary character if presented within double-quotes,
 616 \fB"\|/\|"\fR; preceded by a backslash, \fB\e\|/\fR; or within a bracket
 617 expression, \fB[\|/\|]\fR. The start-condition \fB<\fR and \fB>\fR operators
 618 are special only in a start condition at the beginning of a regular expression;
 619 elsewhere in the regular expression they are treated as ordinary characters.
 620 .sp
 621 .LP
 622 The following examples clarify the differences between \fBlex\fR regular
 623 expressions and regular expressions appearing elsewhere in this document. For
 624 regular expressions of the form \fIr\fR/\fIx\fR, the string matching \fIr\fR is
 625 always returned; confusion may arise when the beginning of \fIx\fR matches the
 626 trailing portion of \fIr\fR. For example, given the regular expression a*b/cc
 627 and the input \fBaaabcc\fR, \fIyytext\fR would contain the string \fBaaab\fR on
 628 this match. But given the regular expression x*/xy and the input \fBxxxy\fR,
 629 the token \fBxxx\fR, not \fBxx\fR, is returned by some implementations because
 630 \fBxxx\fR matches x*.
 631 .sp
 632 .LP
 633 In the rule ab*/bc, the b* at the end of \fIr\fR will extend \fIr\fR's match
 634 into the beginning of the trailing context, so the result is unspecified. If
 635 this rule were ab/bc, however, the rule matches the text \fBab\fR when it is
 636 followed by the text \fBbc\fR. In this latter case, the matching of \fIr\fR
 637 cannot extend into the beginning of \fIx\fR, so the result is specified.
 638 .SS "Actions in lex"
 639 .sp
 640 .LP
 641 The action to be taken when an \fBERE\fR is matched can be a C program fragment
 642 or the special actions described below; the program fragment can contain one or
 643 more C statements, and can also include special actions. The empty C statement
 644 \fB;\fR is a valid action; any string in the \fBlex.yy.c\fR input that matches
 645 the pattern portion of such a rule is effectively ignored or skipped. However,
 646 the absence of an action is not valid, and the action \fBlex\fR takes in such a
 647 condition is undefined.
 648 .sp
 649 .LP
 650 The specification for an action, including C statements and special actions,
 651 can extend across several lines if enclosed in braces:
 652 .sp
 653 .in +2
 654 .nf
 655 ERE <one or more blanks> { program statement
 656 program statement }
 657 .fi
 658 .in -2
 659 .sp
 660 
 661 .sp
 662 .LP
 663 The default action when a string in the input to a \fBlex.yy.c\fR program is
 664 not matched by any expression is to copy the string to the output. Because the
 665 default behavior of a program generated by \fBlex\fR is to read the input and
 666 copy it to the output, a minimal \fBlex\fR source program that has just
 667 \fB%%\fR generates a C program that simply copies the input to the output
 668 unchanged.
 669 .sp
 670 .LP
 671 Four special actions are available:
 672 .sp
 673 .in +2
 674 .nf
 675 |       ECHO;      REJECT;      BEGIN
 676 .fi
 677 .in -2
 678 .sp
 679 
 680 .sp
 681 .ne 2
 682 .na
 683 \fB|\fR
 684 .ad
 685 .RS 12n
 686 The action | means that the action for the next rule is the action for this
 687 rule. Unlike the other three actions, | cannot be enclosed in braces or be
 688 semicolon-terminated. It must be specified alone, with no other actions.
 689 .RE
 690 
 691 .sp
 692 .ne 2
 693 .na
 694 \fB\fBECHO;\fR \fR
 695 .ad
 696 .RS 12n
 697 Writes the contents of the string \fIyytext\fR on the output.
 698 .RE
 699 
 700 .sp
 701 .ne 2
 702 .na
 703 \fB\fBREJECT;\fR \fR
 704 .ad
 705 .RS 12n
 706 Usually only a single expression is matched by a given string in the input.
 707 \fBREJECT\fR means "continue to the next expression that matches the current
 708 input," and causes whatever rule was the second choice after the current rule
 709 to be executed for the same input. Thus, multiple rules can be matched and
 710 executed for one input string or overlapping input strings. For example, given
 711 the regular expressions \fBxyz\fR and \fBxy\fR and the input \fBxyz\fR, usually
 712 only the regular expression \fBxyz\fR would match. The next attempted match
 713 would start after z. If the last action in the \fBxyz\fR rule is \fBREJECT\fR ,
 714 both this rule and the \fBxy\fR rule would be executed. The \fBREJECT\fR action
 715 may be implemented in such a fashion that flow of control does not continue
 716 after it, as if it were equivalent to a \fBgoto\fR to another part of
 717 \fByylex\fR. The use of \fBREJECT\fR may result in somewhat larger and slower
 718 scanners.
 719 .RE
 720 
 721 .sp
 722 .ne 2
 723 .na
 724 \fB\fBBEGIN\fR \fR
 725 .ad
 726 .RS 12n
 727 The action:
 728 .sp
 729 \fBBEGIN\fR \fInewstate\fR\fB;\fR
 730 .sp
 731 switches the state (start condition) to \fInewstate\fR. If the string
 732 \fInewstate\fR has not been declared previously as a start condition in the
 733 \fBDefinitions\fR \fBin\fR \fBlex\fR section, the results are unspecified. The
 734 initial state is indicated by the digit \fB0\fR or the token \fBINITIAL\fR.
 735 .RE
 736 
 737 .sp
 738 .LP
 739 The functions or macros described below are accessible to user code included in
 740 the \fBlex\fR input. It is unspecified whether they appear in the C code output
 741 of \fBlex\fR, or are accessible only through the \fB\fR\fB-l\fR\fB l\fR operand
 742 to \fBc89\fR or \fBcc\fR (the \fBlex\fR library).
 743 .sp
 744 .ne 2
 745 .na
 746 \fB\fBint\fR \fByylex(void)\fR \fR
 747 .ad
 748 .RS 21n
 749 Performs lexical analysis on the input; this is the primary function generated
 750 by the \fBlex\fR utility. The function returns zero when the end of input is
 751 reached; otherwise it returns non-zero values (tokens) determined by the
 752 actions that are selected.
 753 .RE
 754 
 755 .sp
 756 .ne 2
 757 .na
 758 \fB\fBint\fR \fByymore(void)\fR \fR
 759 .ad
 760 .RS 21n
 761 When called, indicates that when the next input string is recognized, it is to
 762 be appended to the current value of \fIyytext\fR rather than replacing it; the
 763 value in \fIyyleng\fR is adjusted accordingly.
 764 .RE
 765 
 766 .sp
 767 .ne 2
 768 .na
 769 \fB\fBint\fR\fIyyless(int\fR\fB n\fR\fI)\fR \fR
 770 .ad
 771 .RS 21n
 772 Retains \fIn\fR initial characters in \fIyytext\fR, NUL-terminated, and treats
 773 the remaining characters as if they had not been read; the value in
 774 \fIyyleng\fR is adjusted accordingly.
 775 .RE
 776 
 777 .sp
 778 .ne 2
 779 .na
 780 \fB\fBint\fR \fBinput(void)\fR \fR
 781 .ad
 782 .RS 21n
 783 Returns the next character from the input, or zero on end-of-file. It obtains
 784 input from the stream pointer \fIyyin\fR, although possibly via an intermediate
 785 buffer. Thus, once scanning has begun, the effect of altering the value of
 786 \fIyyin\fR is undefined. The character read is removed from the input stream of
 787 the scanner without any processing by the scanner.
 788 .RE
 789 
 790 .sp
 791 .ne 2
 792 .na
 793 \fB\fBint\fR \fBunput(int\fR \fB\fIc\fR\fR\fB)\fR \fR
 794 .ad
 795 .RS 21n
 796 Returns the character \fIc\fR to the input; \fIyytext\fR and \fIyyleng\fR are
 797 undefined until the next expression is matched. The result of using \fIunput\fR
 798 for more characters than have been input is unspecified.
 799 .RE
 800 
 801 .sp
 802 .LP
 803 The following functions appear only in the \fBlex\fR library accessible through
 804 the \fB\fR\fB-l\fR\fB l\fR operand; they can therefore be redefined by a
 805 portable application:
 806 .sp
 807 .ne 2
 808 .na
 809 \fB\fBint\fR \fByywrap(void)\fR \fR
 810 .ad
 811 .sp .6
 812 .RS 4n
 813 Called by \fByylex\fR at end-of-file; the default \fByywrap\fR always will
 814 return 1. If the application requires \fByylex\fR to continue processing with
 815 another source of input, then the application can include a function
 816 \fByywrap\fR, which associates another file with the external variable
 817 \fBFILE\fR *\fIyyin\fR and will return a value of zero.
 818 .RE
 819 
 820 .sp
 821 .ne 2
 822 .na
 823 \fB\fBint\fR \fBmain(int\fR \fB\fIargc\fR,\fR \fBchar\fR \fB*\fIargv\fR[\|])\fR
 824 \fR
 825 .ad
 826 .sp .6
 827 .RS 4n
 828 Calls \fByylex\fR to perform lexical analysis, then exits. The user code can
 829 contain \fBmain\fR to perform application-specific operations, calling
 830 \fByylex\fR as applicable.
 831 .RE
 832 
 833 .sp
 834 .LP
 835 The reason for breaking these functions into two lists is that only those
 836 functions in \fBlibl.a\fR can be reliably redefined by a portable application.
 837 .sp
 838 .LP
 839 Except for \fBinput\fR, \fBunput\fR and \fBmain\fR, all external and static
 840 names generated by \fBlex\fR begin with the prefix \fByy\fR or \fBYY\fR.
 841 .SH USAGE
 842 .sp
 843 .LP
 844 Portable applications are warned that in the \fBRules in lex\fR section, an
 845 \fBERE\fR without an action is not acceptable, but need not be detected as
 846 erroneous by \fBlex\fR. This may result in compilation or run-time errors.
 847 .sp
 848 .LP
 849 The purpose of \fBinput\fR is to take characters off the input stream and
 850 discard them as far as the lexical analysis is concerned. A common use is to
 851 discard the body of a comment once the beginning of a comment is recognized.
 852 .sp
 853 .LP
 854 The \fBlex\fR utility is not fully internationalized in its treatment of
 855 regular expressions in the \fBlex\fR source code or generated lexical analyzer.
 856 It would seem desirable to have the lexical analyzer interpret the regular
 857 expressions given in the \fBlex\fR source according to the environment
 858 specified when the lexical analyzer is executed, but this is not possible with
 859 the current \fBlex\fR technology. Furthermore, the very nature of the lexical
 860 analyzers produced by \fBlex\fR must be closely tied to the lexical
 861 requirements of the input language being described, which will frequently be
 862 locale-specific anyway. (For example, writing an analyzer that is used for
 863 French text will not automatically be useful for processing other languages.)
 864 .SH EXAMPLES
 865 .LP
 866 \fBExample 1 \fRUsing lex
 867 .sp
 868 .LP
 869 The following is an example of a \fBlex\fR program that implements a
 870 rudimentary scanner for a Pascal-like syntax:
 871 
 872 .sp
 873 .in +2
 874 .nf
 875 %{
 876 /* need this for the call to atof() below */
 877 #include <math.h>
 878 /* need this for printf(), fopen() and stdin below */
 879 #include <stdio.h>
 880 %}
 881 
 882 DIGIT    [0-9]
 883 ID       [a-z][a-z0-9]*
 884 %%
 885 
 886 {DIGIT}+        {
 887                        printf("An integer: %s (%d)\en", yytext,
 888                        atoi(yytext));
 889                        }
 890 
 891 {DIGIT}+"."{DIGIT}*    {
 892                        printf("A float: %s (%g)\en", yytext,
 893                        atof(yytext));
 894                        }
 895 
 896 if|then|begin|end|procedure|function        {
 897                        printf("A keyword: %s\en", yytext);
 898                        }
 899 
 900 {ID}                   printf("An identifier: %s\en", yytext);
 901 
 902 "+"|"-"|"*"|"/"        printf("An operator: %s\en", yytext);
 903 
 904 "{"[^}\en]*"}"         /* eat up one-line comments */
 905 
 906 [ \et\en]+               /* eat up white space */
 907 
 908 \&.                      printf("Unrecognized character: %s\en", yytext);
 909 
 910 %%
 911 
 912 int main(int argc, char *argv[\|])
 913 {
 914                       ++argv, --argc;  /* skip over program name */
 915                       if (argc > 0)
 916                                   yyin = fopen(argv[0], "r");
 917                       else
 918                       yyin = stdin;
 919         
 920                       yylex();
 921 }
 922 .fi
 923 .in -2
 924 .sp
 925 
 926 .SH ENVIRONMENT VARIABLES
 927 .sp
 928 .LP
 929 See \fBenviron\fR(5) for descriptions of the following environment variables
 930 that affect the execution of \fBlex\fR: \fBLANG\fR, \fBLC_ALL\fR,
 931 \fBLC_COLLATE\fR, \fBLC_CTYPE\fR, \fBLC_MESSAGES\fR, and \fBNLSPATH\fR.
 932 .SH EXIT STATUS
 933 .sp
 934 .LP
 935 The following exit values are returned:
 936 .sp
 937 .ne 2
 938 .na
 939 \fB\fB0\fR \fR
 940 .ad
 941 .RS 7n
 942 Successful completion.
 943 .RE
 944 
 945 .sp
 946 .ne 2
 947 .na
 948 \fB\fB>0\fR \fR
 949 .ad
 950 .RS 7n
 951 An error occurred.
 952 .RE
 953 
 954 .SH ATTRIBUTES
 955 .sp
 956 .LP
 957 See \fBattributes\fR(5) for descriptions of the following attributes:
 958 .sp
 959 
 960 .sp
 961 .TS
 962 box;
 963 c | c
 964 l | l .
 965 ATTRIBUTE TYPE  ATTRIBUTE VALUE
 966 _
 967 Interface Stability     Standard
 968 .TE
 969 
 970 .SH SEE ALSO
 971 .sp
 972 .LP
 973 \fByacc\fR(1), \fBattributes\fR(5), \fBenviron\fR(5), \fBregex\fR(5),
 974 \fBstandards\fR(5)
 975 .SH NOTES
 976 .sp
 977 .LP
 978 If routines such as \fByyback()\fR, \fByywrap()\fR, and \fByylock()\fR
 979 in \fB\|.l\fR (ell) files are to be external C functions, the command line to
 980 compile a C++ program must define the \fB__EXTERN_C__\fR macro. For example:
 981 .sp
 982 .in +2
 983 .nf
 984 example%  \fBCC -D__EXTERN_C__ ... file\fR
 985 .fi
 986 .in -2
 987 .sp
 988