Print this page
2926 lex ignores -Y
| Split |
Close |
| Expand all |
| Collapse all |
--- old/usr/src/man/man1/lex.1
+++ new/usr/src/man/man1/lex.1
1 1 '\" te
2 +.\" Copyright (c) 2014 Gary Mills
2 3 .\" Copyright (c) 1992, X/Open Company Limited All Rights Reserved Portions Copyright (c) 1997, Sun Microsystems, Inc. All Rights Reserved
3 4 .\" Sun Microsystems, Inc. gratefully acknowledges The Open Group for permission to reproduce portions of its copyrighted documentation. Original documentation from The Open Group can be obtained online at
4 5 .\" http://www.opengroup.org/bookstore/.
5 6 .\" The Institute of Electrical and Electronics Engineers and The Open Group, have given us permission to reprint portions of their documentation. In the following statement, the phrase "this text" refers to portions of the system documentation. Portions of this text are reprinted and reproduced in electronic form in the Sun OS Reference Manual, from IEEE Std 1003.1, 2004 Edition, Standard for Information Technology -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 6, Copyright (C) 2001-2004 by the Institute of Electrical and Electronics Engineers, Inc and The Open Group. In the event of any discrepancy between these versions and the original IEEE and The Open Group Standard, the original IEEE and The Open Group Standard is the referee document. The original Standard can be obtained online at http://www.opengroup.org/unix/online.html.
6 7 .\" This notice shall appear on any product containing this material.
7 8 .\" The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License.
8 9 .\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License.
9 10 .\" When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner]
10 -.TH LEX 1 "Aug 22, 1997"
11 +.TH LEX 1 "Jan 1, 2014"
11 12 .SH NAME
12 13 lex \- generate programs for lexical tasks
13 14 .SH SYNOPSIS
14 15 .LP
15 16 .nf
16 -\fBlex\fR [\fB-cntv\fR] [\fB-e\fR | \fB-w\fR] [\fB-V\fR \fB-Q\fR [y | n]] [\fIfile\fR]...
17 +\fBlex\fR [\fB-cntv\fR] [\fB-e\fR | \fB-w\fR] [\fB-V\fR \fB-Q\fR [y | n]] [\fB-Y\fR \fIdirectory\fR]] [\fIfile\fR]...
17 18 .fi
18 19
19 20 .SH DESCRIPTION
20 21 .sp
21 22 .LP
22 23 The \fBlex\fR utility generates C programs to be used in lexical processing of
23 24 character input, and that can be used as an interface to \fByacc\fR. The C
24 25 programs are generated from \fBlex\fR source code and conform to the ISO C
25 26 standard. Usually, the \fBlex\fR utility writes the program it generates to the
26 27 file \fBlex.yy.c\fR. The state of this file is unspecified if \fBlex\fR exits
27 28 with a non-zero exit status. See \fBEXTENDED DESCRIPTION\fR for a complete
28 29 description of the \fBlex\fR input language.
29 30 .SH OPTIONS
30 31 .sp
31 32 .LP
32 33 The following options are supported:
33 34 .sp
34 35 .ne 2
35 36 .na
36 37 \fB\fB-c\fR \fR
37 38 .ad
38 39 .RS 12n
39 40 Indicates C-language action (default option).
40 41 .RE
41 42
42 43 .sp
43 44 .ne 2
44 45 .na
45 46 \fB\fB-e\fR \fR
46 47 .ad
47 48 .RS 12n
48 49 Generates a program that can handle \fBEUC\fR characters (cannot be used with
49 50 the \fB-w\fR option). \fByytext[\|]\fR is of type \fBunsigned char[\|]\fR.
50 51 .RE
51 52
52 53 .sp
53 54 .ne 2
54 55 .na
55 56 \fB\fB-n\fR \fR
56 57 .ad
57 58 .RS 12n
58 59 Suppresses the summary of statistics usually written with the \fB-v\fR option.
59 60 If no table sizes are specified in the \fBlex\fR source code and the \fB-v\fR
60 61 option is not specified, then \fB-n\fR is implied.
61 62 .RE
62 63
63 64 .sp
64 65 .ne 2
65 66 .na
66 67 \fB\fB-t\fR \fR
67 68 .ad
68 69 .RS 12n
69 70 Writes the resulting program to standard output instead of \fBlex.yy.c\fR.
70 71 .RE
71 72
72 73 .sp
73 74 .ne 2
74 75 .na
75 76 \fB\fB-v\fR \fR
76 77 .ad
77 78 .RS 12n
78 79 Writes a summary of \fBlex\fR statistics to the standard error. (See the
79 80 discussion of \fBlex\fR table sizes under the heading \fBDefinitions in
80 81 lex\fR.) If table sizes are specified in the \fBlex\fR source code, and if the
81 82 \fB-n\fR option is not specified, the \fB-v\fR option may be enabled.
82 83 .RE
83 84
84 85 .sp
85 86 .ne 2
86 87 .na
87 88 \fB\fB-w\fR \fR
88 89 .ad
89 90 .RS 12n
90 91 Generates a program that can handle \fBEUC\fR characters (cannot be used with
91 92 the \fB-e\fR option). Unlike the \fB-e\fR option, \fByytext[\|]\fR is of type
92 93 \fBwchar_t[\|]\fR.
93 94 .RE
94 95
95 96 .sp
96 97 .ne 2
97 98 .na
98 99 \fB\fB-V\fR \fR
99 100 .ad
100 101 .RS 12n
101 102 Prints out version information on standard error.
102 103 .RE
103 104
104 105 .sp
|
↓ open down ↓ |
78 lines elided |
↑ open up ↑ |
105 106 .ne 2
106 107 .na
107 108 \fB\fB\fR\fB-Q\fR\fB[y|n]\fR \fR
108 109 .ad
109 110 .RS 12n
110 111 Prints out version information to output file \fBlex.yy.c\fR by using
111 112 \fB-Qy\fR. The \fB-Qn\fR option does not print out version information and is
112 113 the default.
113 114 .RE
114 115
116 +.sp
117 +.ne 2
118 +.na
119 +\fB\fB\fR\fB-Y\fR \fBdirectory\fR \fR
120 +.ad
121 +.RS 12n
122 +Designates an alternate directory that contains the driver files
123 +used by \fBlex\fR.
124 +.RE
125 +
115 126 .SH OPERANDS
116 127 .sp
117 128 .LP
118 129 The following operand is supported:
119 130 .sp
120 131 .ne 2
121 132 .na
122 133 \fB\fIfile\fR \fR
123 134 .ad
124 135 .RS 9n
125 136 A pathname of an input file. If more than one such \fIfile\fR is specified, all
126 137 files will be concatenated to produce a single \fBlex\fR program. If no
127 138 \fIfile\fR operands are specified, or if a \fIfile\fR operand is \fB\(mi\fR,
128 139 the standard input will be used.
129 140 .RE
130 141
131 142 .SH OUTPUT
132 143 .sp
133 144 .LP
134 145 The \fBlex\fR output files are described below.
135 146 .SS "Stdout"
136 147 .sp
137 148 .LP
138 149 If the \fB-t\fR option is specified, the text file of C source code output of
139 150 \fBlex\fR will be written to standard output.
140 151 .SS "Stderr"
141 152 .sp
142 153 .LP
143 154 If the \fB-t\fR option is specified informational, error and warning messages
144 155 concerning the contents of \fBlex\fR source code input will be written to the
145 156 standard error.
146 157 .sp
147 158 .LP
148 159 If the \fB-t\fR option is not specified:
149 160 .RS +4
150 161 .TP
151 162 1.
152 163 Informational error and warning messages concerning the contents of
153 164 \fBlex\fR source code input will be written to either the standard output or
154 165 standard error.
155 166 .RE
156 167 .RS +4
157 168 .TP
158 169 2.
159 170 If the \fB-v\fR option is specified and the \fB-n\fR option is not
160 171 specified, \fBlex\fR statistics will also be written to standard error. These
161 172 statistics may also be generated if table sizes are specified with a \fB%\fR
162 173 operator in the \fBDefinitions\fR \fBin\fR \fBlex\fR section (see \fBEXTENDED
163 174 DESCRIPTION\fR), as long as the \fB-n\fR option is not specified.
164 175 .RE
165 176 .SS "Output Files"
166 177 .sp
167 178 .LP
168 179 A text file containing C source code will be written to \fBlex.yy.c\fR, or to
169 180 the standard output if the \fB-t\fR option is present.
170 181 .SH EXTENDED DESCRIPTION
171 182 .sp
172 183 .LP
173 184 Each input file contains \fBlex\fR source code, which is a table of regular
174 185 expressions with corresponding actions in the form of C program fragments.
175 186 .sp
176 187 .LP
177 188 When \fBlex.yy.c\fR is compiled and linked with the \fBlex\fR library (using
178 189 the \fB\fR\fB-l\fR\fB l\fR operand with \fBc89\fR or \fBcc\fR), the resulting
179 190 program reads character input from the standard input and partitions it into
180 191 strings that match the given expressions.
181 192 .sp
182 193 .LP
183 194 When an expression is matched, these actions will occur:
184 195 .RS +4
185 196 .TP
186 197 .ie t \(bu
187 198 .el o
188 199 The input string that was matched is left in \fIyytext\fR as a null-terminated
189 200 string; \fIyytext\fR is either an external character array or a pointer to a
190 201 character string. As explained in \fBDefinitions in lex\fR, the type can be
191 202 explicitly selected using the \fB%array\fR or \fB%pointer\fR declarations, but
192 203 the default is \fB%array\fR.
193 204 .RE
194 205 .RS +4
195 206 .TP
196 207 .ie t \(bu
197 208 .el o
198 209 The external \fBint\fR \fIyyleng\fR is set to the length of the matching
199 210 string.
200 211 .RE
201 212 .RS +4
202 213 .TP
203 214 .ie t \(bu
204 215 .el o
205 216 The expression's corresponding program fragment, or action, is executed.
206 217 .RE
207 218 .sp
208 219 .LP
209 220 During pattern matching, \fBlex\fR searches the set of patterns for the single
210 221 longest possible match. Among rules that match the same number of characters,
211 222 the rule given first will be chosen.
212 223 .sp
213 224 .LP
214 225 The general format of \fBlex\fR source is:
215 226 .sp
216 227 .in +2
217 228 .nf
218 229 \fIDefinitions\fR
219 230 %%
220 231 \fIRules\fR
221 232 %%
222 233 \fIUser Subroutines\fR
223 234 .fi
224 235 .in -2
225 236
226 237 .sp
227 238 .LP
228 239 The first \fB%%\fR is required to mark the beginning of the rules (regular
229 240 expressions and actions); the second \fB%%\fR is required only if user
230 241 subroutines follow.
231 242 .sp
232 243 .LP
233 244 Any line in the \fBDefinitions\fR \fBin\fR \fBlex\fR section beginning with a
234 245 blank character will be assumed to be a C program fragment and will be copied
235 246 to the external definition area of the \fBlex.yy.c\fR file. Similarly, anything
236 247 in the \fBDefinitions\fR \fBin\fR \fBlex\fR section included between delimiter
237 248 lines containing only \fB%{\fR and \fB%}\fR will also be copied unchanged to
238 249 the external definition area of the \fBlex.yy.c\fR file.
239 250 .sp
240 251 .LP
241 252 Any such input (beginning with a blank character or within \fB%{\fR and
242 253 \fB%}\fR delimiter lines) appearing at the beginning of the \fIRules\fR section
243 254 before any rules are specified will be written to \fBlex.yy.c\fR after the
244 255 declarations of variables for the \fByylex\fR function and before the first
245 256 line of code in \fByylex\fR. Thus, user variables local to \fByylex\fR can be
246 257 declared here, as well as application code to execute upon entry to
247 258 \fByylex\fR.
248 259 .sp
249 260 .LP
250 261 The action taken by \fBlex\fR when encountering any input beginning with a
251 262 blank character or within \fB%{\fR and \fB%}\fR delimiter lines appearing in
252 263 the \fIRules\fR section but coming after one or more rules is undefined. The
253 264 presence of such input may result in an erroneous definition of the \fByylex\fR
254 265 function.
255 266 .SS "Definitions in lex"
256 267 .sp
257 268 .LP
258 269 \fBDefinitions\fR \fBin\fR \fBlex\fR appear before the first \fB%%\fR
259 270 delimiter. Any line in this section not contained between \fB%{\fR and \fB%}\fR
260 271 lines and not beginning with a blank character is assumed to define a \fBlex\fR
261 272 substitution string. The format of these lines is:
262 273 .sp
263 274 .in +2
264 275 .nf
265 276 \fIname substitute\fR
266 277 .fi
267 278 .in -2
268 279 .sp
269 280
270 281 .sp
271 282 .LP
272 283 If a \fIname\fR does not meet the requirements for identifiers in the ISO C
273 284 standard, the result is undefined. The string \fIsubstitute\fR will replace the
274 285 string \fI{\fR \fIname\fR \fI}\fR when it is used in a rule. The \fIname\fR
275 286 string is recognized in this context only when the braces are provided and when
276 287 it does not appear within a bracket expression or within double-quotes.
277 288 .sp
278 289 .LP
279 290 In the \fBDefinitions\fR \fBin\fR \fBlex\fR section, any line beginning with a
280 291 \fB%\fR (percent sign) character and followed by an alphanumeric word beginning
281 292 with either \fBs\fR or \fBS\fR defines a set of start conditions. Any line
282 293 beginning with a \fB%\fR followed by a word beginning with either \fBx\fR or
283 294 \fBX\fR defines a set of exclusive start conditions. When the generated scanner
284 295 is in a \fB%s\fR state, patterns with no state specified will be also active;
285 296 in a \fB%x\fR state, such patterns will not be active. The rest of the line,
286 297 after the first word, is considered to be one or more blank-character-separated
287 298 names of start conditions. Start condition names are constructed in the same
288 299 way as definition names. Start conditions can be used to restrict the matching
289 300 of regular expressions to one or more states as described in \fBRegular
290 301 expressions in lex\fR.
291 302 .sp
292 303 .LP
293 304 Implementations accept either of the following two mutually exclusive
294 305 declarations in the \fBDefinitions\fR \fBin\fR \fBlex\fR section:
295 306 .sp
296 307 .ne 2
297 308 .na
298 309 \fB\fB%array\fR \fR
299 310 .ad
300 311 .RS 13n
301 312 Declare the type of \fIyytext\fR to be a null-terminated character array.
302 313 .RE
303 314
304 315 .sp
305 316 .ne 2
306 317 .na
307 318 \fB\fB%pointer\fR \fR
308 319 .ad
309 320 .RS 13n
310 321 Declare the type of \fIyytext\fR to be a pointer to a null-terminated character
311 322 string.
312 323 .RE
313 324
314 325 .sp
315 326 .LP
316 327 \fBNote:\fR When using the \fB%pointer\fR option, you may not also use the
317 328 \fByyless\fR function to alter \fIyytext\fR.
318 329 .sp
319 330 .LP
320 331 \fB%array\fR is the default. If \fB%array\fR is specified (or neither
321 332 \fB%array\fR nor \fB%pointer\fR is specified), then the correct way to make an
322 333 external reference to \fIyyext\fR is with a declaration of the form:
323 334 .sp
324 335 .LP
325 336 \fBextern char\fR\fI yytext\fR\fB[\|]\fR
326 337 .sp
327 338 .LP
328 339 If \fB%pointer\fR is specified, then the correct external reference is of the
329 340 form:
330 341 .sp
331 342 .LP
332 343 \fBextern char *\fR\fIyytext\fR\fB;\fR
333 344 .sp
334 345 .LP
335 346 \fBlex\fR will accept declarations in the \fBDefinitions in lex\fR section for
336 347 setting certain internal table sizes. The declarations are shown in the
337 348 following table.
338 349 .sp
339 350 .LP
340 351 \fBTable\fR \fBSize\fR \fBDeclaration\fR \fBin\fR \fBlex\fR
341 352 .sp
342 353
343 354 .sp
344 355 .TS
345 356 box;
346 357 c c c
347 358 l l l .
348 359 \fBDeclaration\fR \fBDescription\fR \fBDefault\fR
349 360 _
350 361 \fB%p\fR\fIn\fR Number of positions 2500
351 362 \fB%n\fR\fIn\fR Number of states 500
352 363 \fB%a\fR\fI n\fR Number of transitions 2000
353 364 \fB%e\fR\fIn\fR Number of parse tree nodes 1000
354 365 \fB%k\fR\fIn\fR Number of packed character classes 10000
355 366 \fB%o\fR\fIn\fR Size of the output array 3000
356 367 .TE
357 368
358 369 .sp
359 370 .LP
360 371 Programs generated by \fBlex\fR need either the \fB-e\fR or \fB-w\fR option to
361 372 handle input that contains \fBEUC\fR characters from supplementary codesets. If
362 373 neither of these options is specified, \fByytext\fR is of the type
363 374 \fBchar[\|]\fR, and the generated program can handle only \fBASCII\fR
364 375 characters.
365 376 .sp
366 377 .LP
367 378 When the \fB-e\fR option is used, \fByytext\fR is of the type \fBunsigned\fR
368 379 \fBchar[\|]\fR and \fByyleng\fR gives the total number of \fIbytes\fR in the
369 380 matched string. With this option, the macros \fBinput()\fR,
370 381 \fBunput(\fIc\fR)\fR, and \fBoutput(\fIc\fR)\fR should do a byte-based
371 382 \fBI/O\fR in the same way as with the regular \fBASCII\fR \fBlex\fR. Two more
372 383 variables are available with the \fB-e\fR option, \fByywtext\fR and
373 384 \fByywleng\fR, which behave the same as \fByytext\fR and \fByyleng\fR would
374 385 under the \fB-w\fR option.
375 386 .sp
376 387 .LP
377 388 When the \fB-w\fR option is used, \fByytext\fR is of the type \fBwchar_t[\|]\fR
378 389 and \fByyleng\fR gives the total number of \fIcharacters\fR in the matched
379 390 string. If you supply your own \fBinput()\fR, \fBunput(\fIc\fR)\fR, or
380 391 \fBoutput(\fR\fIc\fR\fB)\fR macros with this option, they must return or accept
381 392 \fBEUC\fR characters in the form of wide character (\fBwchar_t\fR). This allows
382 393 a different interface between your program and the lex internals, to expedite
383 394 some programs.
384 395 .SS "Rules in lex"
385 396 .sp
386 397 .LP
387 398 The \fBRules\fR \fBin\fR \fBlex\fR source files are a table in which the left
388 399 column contains regular expressions and the right column contains actions (C
389 400 program fragments) to be executed when the expressions are recognized.
390 401 .sp
391 402 .in +2
392 403 .nf
393 404 \fIERE action\fR
394 405 \fIERE action\fR
395 406 \&...
396 407 .fi
397 408 .in -2
398 409
399 410 .sp
400 411 .LP
401 412 The extended regular expression (ERE) portion of a row will be separated from
402 413 \fIaction\fR by one or more blank characters. A regular expression containing
403 414 blank characters is recognized under one of the following conditions:
404 415 .RS +4
405 416 .TP
406 417 .ie t \(bu
407 418 .el o
408 419 The entire expression appears within double-quotes.
409 420 .RE
410 421 .RS +4
411 422 .TP
412 423 .ie t \(bu
413 424 .el o
414 425 The blank characters appear within double-quotes or square brackets.
415 426 .RE
416 427 .RS +4
417 428 .TP
418 429 .ie t \(bu
419 430 .el o
420 431 Each blank character is preceded by a backslash character.
421 432 .RE
422 433 .SS "User Subroutines in lex"
423 434 .sp
424 435 .LP
425 436 Anything in the user subroutines section will be copied to \fBlex.yy.c\fR
426 437 following \fByylex\fR.
427 438 .SS "Regular Expressions in lex"
428 439 .sp
429 440 .LP
430 441 The \fBlex\fR utility supports the set of Extended Regular Expressions (EREs)
431 442 described on \fBregex\fR(5) with the following additions and exceptions to the
432 443 syntax:
433 444 .sp
434 445 .ne 2
435 446 .na
436 447 \fB\fB\|.\|.\|.\fR \fR
437 448 .ad
438 449 .RS 14n
439 450 Any string enclosed in double-quotes will represent the characters within the
440 451 double-quotes as themselves, except that backslash escapes (which appear in the
441 452 following table) are recognized. Any backslash-escape sequence is terminated by
442 453 the closing quote. For example, "\|\e\|01""1" represents a single string: the
443 454 octal value 1 followed by the character 1.
444 455 .RE
445 456
446 457 .sp
447 458 .LP
448 459 \fI<\fR\fIstate\fR\fI>\fR\fIr\fR
449 460 .sp
450 461 .ne 2
451 462 .na
452 463 \fB<\fIstate1\fR, \fIstate2\fR, \|.\|.\|.\|>\fIr\fR\fR
453 464 .ad
454 465 .sp .6
455 466 .RS 4n
456 467 The regular expression \fIr\fR will be matched only when the program is in one
457 468 of the start conditions indicated by \fIstate\fR, \fIstate1\fR, and so forth.
458 469 For more information, see \fBActions in lex\fR. As an exception to the
459 470 typographical conventions of the rest of this document, in this case
460 471 <\fIstate\fR> does not represent a metavariable, but the literal angle-bracket
461 472 characters surrounding a symbol. The start condition is recognized as such only
462 473 at the beginning of a regular expression.
463 474 .RE
464 475
465 476 .sp
466 477 .ne 2
467 478 .na
468 479 \fB\fIr\fR/\fIx\fR \fR
469 480 .ad
470 481 .sp .6
471 482 .RS 4n
472 483 The regular expression \fIr\fR will be matched only if it is followed by an
473 484 occurrence of regular expression \fIx\fR. The token returned in \fIyytext\fR
474 485 will only match \fIr\fR. If the trailing portion of \fIr\fR matches the
475 486 beginning of \fIx\fR, the result is unspecified. The \fIr\fR expression cannot
476 487 include further trailing context or the \fB$\fR (match-end-of-line) operator;
477 488 \fIx\fR cannot include the \fB^\fR (match-beginning-of-line) operator, nor
478 489 trailing context, nor the \fB$\fR operator. That is, only one occurrence of
479 490 trailing context is allowed in a \fBlex\fR regular expression, and the \fB^\fR
480 491 operator only can be used at the beginning of such an expression. A further
481 492 restriction is that the trailing-context operator \fB/\fR (slash) cannot be
482 493 grouped within parentheses.
483 494 .RE
484 495
485 496 .sp
486 497 .ne 2
487 498 .na
488 499 \fB\fB{\fR\fIname\fR\fB}\fR \fR
489 500 .ad
490 501 .sp .6
491 502 .RS 4n
492 503 When \fIname\fR is one of the substitution symbols from the \fIDefinitions\fR
493 504 section, the string, including the enclosing braces, will be replaced by the
494 505 \fIsubstitute\fR value. The \fIsubstitute\fR value will be treated in the
495 506 extended regular expression as if it were enclosed in parentheses. No
496 507 substitution will occur if \fB{\fR\fIname\fR\fB}\fR occurs within a bracket
497 508 expression or within double-quotes.
498 509 .RE
499 510
500 511 .sp
501 512 .LP
502 513 Within an \fBERE,\fR a backslash character (\fB\|\e\e\fR, \fB\e\|a\fR,
503 514 \fB\e\|b\fR, \fB\e\|f\fR, \fB\e\|n\fR, \fB\e\|r\fR, \fB\e\|t\fR, \fB\e\|v\fR)
504 515 is considered to begin an escape sequence. In addition, the escape sequences in
505 516 the following table will be recognized.
506 517 .sp
507 518 .LP
508 519 A literal newline character cannot occur within an \fBERE;\fR the escape
509 520 sequence \fB\e\|n\fR can be used to represent a newline character. A newline
510 521 character cannot be matched by a period operator.
511 522 .sp
512 523 .LP
513 524 \fBEscape Sequences in lex\fR
514 525 .sp
515 526
516 527 .sp
517 528 .TS
518 529 box;
519 530 c c c
520 531 c c c .
521 532 Escape Sequences in lex
522 533 _
523 534 Escape Sequence Description Meaning
524 535 _
525 536 \e\fIdigits\fR T{
526 537 A backslash character followed by the longest sequence of one, two or three octal-digit characters (01234567). Ifall of the digits are 0, (that is, representation of the NUL character), the behavior is undefined.
527 538 T} T{
528 539 The character whose encoding is represented by the one-, two- or three-digit octal integer. Multi-byte characters require multiple, concatenated escape sequences of this type, including the leading \e for each byte.
529 540 T}
530 541 _
531 542 \e\fBx\fR\fIdigits\fR T{
532 543 A backslash character followed by the longest sequence of hexadecimal-digit characters (01234567abcdefABCDEF). If all of the digits are 0, (that is, representation of the NUL character), the behavior is undefined.
533 544 T} T{
534 545 The character whose encoding is represented by the hexadecimal integer.
535 546 T}
536 547 _
537 548 \e\fIc\fR T{
538 549 A backslash character followed by any character not described in this table. (\e\e, \ea, \eb, \ef, \een, \er, \et, \ev).
539 550 T} The character c, unchanged.
540 551 .TE
541 552
542 553 .sp
543 554 .LP
544 555 The order of precedence given to extended regular expressions for \fBlex\fR is
545 556 as shown in the following table, from high to low.
546 557 .sp
547 558 .ne 2
548 559 .na
549 560 \fB\fBNote\fR: \fR
550 561 .ad
551 562 .RS 10n
552 563 The escaped characters entry is not meant to imply that these are operators,
553 564 but they are included in the table to show their relationships to the true
554 565 operators. The start condition, trailing context and anchoring notations have
555 566 been omitted from the table because of the placement restrictions described in
556 567 this section; they can only appear at the beginning or ending of an \fBERE.\fR
557 568 .RE
558 569
559 570 .sp
560 571
561 572 .sp
562 573 .TS
563 574 box;
564 575 c c
565 576 l l .
566 577 ERE Precedence in lex
567 578 _
568 579 \fIcollation-related bracket symbols\fR \fB[= =] [: :] [. .]\fR
569 580 \fIescaped characters\fR \fB\e<\fR\fIspecial character\fR>
570 581 \fIbracket expression\fR \fB[ ]\fR
571 582 \fIquoting\fR \fB".\|.\|."\fR
572 583 \fIgrouping\fR \fB()\fR
573 584 \fIdefinition\fR \fB{\fR\fIname\fR}
574 585 \fIsingle-character RE duplication\fR \fB* + ?\fR
575 586 \fIconcatenation\fR
576 587 \fIinterval expression\fR \fB{\fR\fIm\fR,\fIn\fR}
577 588 \fIalternation\fR \fB|\fR
578 589 .TE
579 590
580 591 .sp
581 592 .LP
582 593 The \fBERE\fR anchoring operators (\fB\|^\fR and \fB$\fR\|) do not appear in
583 594 the table. With \fBlex\fR regular expressions, these operators are restricted
584 595 in their use: the \fB^\fR operator can only be used at the beginning of an
585 596 entire regular expression, and the \fB$\fR operator only at the end. The
586 597 operators apply to the entire regular expression. Thus, for example, the
587 598 pattern (\fB^abc)|(def$\fR) is undefined; it can instead be written as two
588 599 separate rules, one with the regular expression \fB^abc\fR and one with
589 600 \fBdef$\fR, which share a common action via the special \fB|\fR action (see
590 601 below). If the pattern were written \fB^abc|def$\fR, it would match either of
591 602 \fBabc\fR or \fBdef\fR on a line by itself.
592 603 .sp
593 604 .LP
594 605 Unlike the general \fBERE\fR rules, embedded anchoring is not allowed by most
595 606 historical \fBlex\fR implementations. An example of embedded anchoring would be
596 607 for patterns such as (^)foo($) to match \fBfoo\fR when it exists as a complete
597 608 word. This functionality can be obtained using existing \fBlex\fR features:
598 609 .sp
599 610 .in +2
600 611 .nf
601 612 ^foo/[ \e\|n]|
602 613 " foo"/[ \e\|n] /* found foo as a separate word */
603 614 .fi
604 615 .in -2
605 616
606 617 .sp
607 618 .LP
608 619 Notice also that \fB$\fR is a form of trailing context (it is equivalent to
609 620 \fB/\e\|n\fR and as such cannot be used with regular expressions containing
610 621 another instance of the operator (see the preceding discussion of trailing
611 622 context).
612 623 .sp
613 624 .LP
614 625 The additional regular expressions trailing-context operator \fB/\fR (slash)
615 626 can be used as an ordinary character if presented within double-quotes,
616 627 \fB"\|/\|"\fR; preceded by a backslash, \fB\e\|/\fR; or within a bracket
617 628 expression, \fB[\|/\|]\fR. The start-condition \fB<\fR and \fB>\fR operators
618 629 are special only in a start condition at the beginning of a regular expression;
619 630 elsewhere in the regular expression they are treated as ordinary characters.
620 631 .sp
621 632 .LP
622 633 The following examples clarify the differences between \fBlex\fR regular
623 634 expressions and regular expressions appearing elsewhere in this document. For
624 635 regular expressions of the form \fIr\fR/\fIx\fR, the string matching \fIr\fR is
625 636 always returned; confusion may arise when the beginning of \fIx\fR matches the
626 637 trailing portion of \fIr\fR. For example, given the regular expression a*b/cc
627 638 and the input \fBaaabcc\fR, \fIyytext\fR would contain the string \fBaaab\fR on
628 639 this match. But given the regular expression x*/xy and the input \fBxxxy\fR,
629 640 the token \fBxxx\fR, not \fBxx\fR, is returned by some implementations because
630 641 \fBxxx\fR matches x*.
631 642 .sp
632 643 .LP
633 644 In the rule ab*/bc, the b* at the end of \fIr\fR will extend \fIr\fR's match
634 645 into the beginning of the trailing context, so the result is unspecified. If
635 646 this rule were ab/bc, however, the rule matches the text \fBab\fR when it is
636 647 followed by the text \fBbc\fR. In this latter case, the matching of \fIr\fR
637 648 cannot extend into the beginning of \fIx\fR, so the result is specified.
638 649 .SS "Actions in lex"
639 650 .sp
640 651 .LP
641 652 The action to be taken when an \fBERE\fR is matched can be a C program fragment
642 653 or the special actions described below; the program fragment can contain one or
643 654 more C statements, and can also include special actions. The empty C statement
644 655 \fB;\fR is a valid action; any string in the \fBlex.yy.c\fR input that matches
645 656 the pattern portion of such a rule is effectively ignored or skipped. However,
646 657 the absence of an action is not valid, and the action \fBlex\fR takes in such a
647 658 condition is undefined.
648 659 .sp
649 660 .LP
650 661 The specification for an action, including C statements and special actions,
651 662 can extend across several lines if enclosed in braces:
652 663 .sp
653 664 .in +2
654 665 .nf
655 666 ERE <one or more blanks> { program statement
656 667 program statement }
657 668 .fi
658 669 .in -2
659 670 .sp
660 671
661 672 .sp
662 673 .LP
663 674 The default action when a string in the input to a \fBlex.yy.c\fR program is
664 675 not matched by any expression is to copy the string to the output. Because the
665 676 default behavior of a program generated by \fBlex\fR is to read the input and
666 677 copy it to the output, a minimal \fBlex\fR source program that has just
667 678 \fB%%\fR generates a C program that simply copies the input to the output
668 679 unchanged.
669 680 .sp
670 681 .LP
671 682 Four special actions are available:
672 683 .sp
673 684 .in +2
674 685 .nf
675 686 | ECHO; REJECT; BEGIN
676 687 .fi
677 688 .in -2
678 689 .sp
679 690
680 691 .sp
681 692 .ne 2
682 693 .na
683 694 \fB|\fR
684 695 .ad
685 696 .RS 12n
686 697 The action | means that the action for the next rule is the action for this
687 698 rule. Unlike the other three actions, | cannot be enclosed in braces or be
688 699 semicolon-terminated. It must be specified alone, with no other actions.
689 700 .RE
690 701
691 702 .sp
692 703 .ne 2
693 704 .na
694 705 \fB\fBECHO;\fR \fR
695 706 .ad
696 707 .RS 12n
697 708 Writes the contents of the string \fIyytext\fR on the output.
698 709 .RE
699 710
700 711 .sp
701 712 .ne 2
702 713 .na
703 714 \fB\fBREJECT;\fR \fR
704 715 .ad
705 716 .RS 12n
706 717 Usually only a single expression is matched by a given string in the input.
707 718 \fBREJECT\fR means "continue to the next expression that matches the current
708 719 input," and causes whatever rule was the second choice after the current rule
709 720 to be executed for the same input. Thus, multiple rules can be matched and
710 721 executed for one input string or overlapping input strings. For example, given
711 722 the regular expressions \fBxyz\fR and \fBxy\fR and the input \fBxyz\fR, usually
712 723 only the regular expression \fBxyz\fR would match. The next attempted match
713 724 would start after z. If the last action in the \fBxyz\fR rule is \fBREJECT\fR ,
714 725 both this rule and the \fBxy\fR rule would be executed. The \fBREJECT\fR action
715 726 may be implemented in such a fashion that flow of control does not continue
716 727 after it, as if it were equivalent to a \fBgoto\fR to another part of
717 728 \fByylex\fR. The use of \fBREJECT\fR may result in somewhat larger and slower
718 729 scanners.
719 730 .RE
720 731
721 732 .sp
722 733 .ne 2
723 734 .na
724 735 \fB\fBBEGIN\fR \fR
725 736 .ad
726 737 .RS 12n
727 738 The action:
728 739 .sp
729 740 \fBBEGIN\fR \fInewstate\fR\fB;\fR
730 741 .sp
731 742 switches the state (start condition) to \fInewstate\fR. If the string
732 743 \fInewstate\fR has not been declared previously as a start condition in the
733 744 \fBDefinitions\fR \fBin\fR \fBlex\fR section, the results are unspecified. The
734 745 initial state is indicated by the digit \fB0\fR or the token \fBINITIAL\fR.
735 746 .RE
736 747
737 748 .sp
738 749 .LP
739 750 The functions or macros described below are accessible to user code included in
740 751 the \fBlex\fR input. It is unspecified whether they appear in the C code output
741 752 of \fBlex\fR, or are accessible only through the \fB\fR\fB-l\fR\fB l\fR operand
742 753 to \fBc89\fR or \fBcc\fR (the \fBlex\fR library).
743 754 .sp
744 755 .ne 2
745 756 .na
746 757 \fB\fBint\fR \fByylex(void)\fR \fR
747 758 .ad
748 759 .RS 21n
749 760 Performs lexical analysis on the input; this is the primary function generated
750 761 by the \fBlex\fR utility. The function returns zero when the end of input is
751 762 reached; otherwise it returns non-zero values (tokens) determined by the
752 763 actions that are selected.
753 764 .RE
754 765
755 766 .sp
756 767 .ne 2
757 768 .na
758 769 \fB\fBint\fR \fByymore(void)\fR \fR
759 770 .ad
760 771 .RS 21n
761 772 When called, indicates that when the next input string is recognized, it is to
762 773 be appended to the current value of \fIyytext\fR rather than replacing it; the
763 774 value in \fIyyleng\fR is adjusted accordingly.
764 775 .RE
765 776
766 777 .sp
767 778 .ne 2
768 779 .na
769 780 \fB\fBint\fR\fIyyless(int\fR\fB n\fR\fI)\fR \fR
770 781 .ad
771 782 .RS 21n
772 783 Retains \fIn\fR initial characters in \fIyytext\fR, NUL-terminated, and treats
773 784 the remaining characters as if they had not been read; the value in
774 785 \fIyyleng\fR is adjusted accordingly.
775 786 .RE
776 787
777 788 .sp
778 789 .ne 2
779 790 .na
780 791 \fB\fBint\fR \fBinput(void)\fR \fR
781 792 .ad
782 793 .RS 21n
783 794 Returns the next character from the input, or zero on end-of-file. It obtains
784 795 input from the stream pointer \fIyyin\fR, although possibly via an intermediate
785 796 buffer. Thus, once scanning has begun, the effect of altering the value of
786 797 \fIyyin\fR is undefined. The character read is removed from the input stream of
787 798 the scanner without any processing by the scanner.
788 799 .RE
789 800
790 801 .sp
791 802 .ne 2
792 803 .na
793 804 \fB\fBint\fR \fBunput(int\fR \fB\fIc\fR\fR\fB)\fR \fR
794 805 .ad
795 806 .RS 21n
796 807 Returns the character \fIc\fR to the input; \fIyytext\fR and \fIyyleng\fR are
797 808 undefined until the next expression is matched. The result of using \fIunput\fR
798 809 for more characters than have been input is unspecified.
799 810 .RE
800 811
801 812 .sp
802 813 .LP
803 814 The following functions appear only in the \fBlex\fR library accessible through
804 815 the \fB\fR\fB-l\fR\fB l\fR operand; they can therefore be redefined by a
805 816 portable application:
806 817 .sp
807 818 .ne 2
808 819 .na
809 820 \fB\fBint\fR \fByywrap(void)\fR \fR
810 821 .ad
811 822 .sp .6
812 823 .RS 4n
813 824 Called by \fByylex\fR at end-of-file; the default \fByywrap\fR always will
814 825 return 1. If the application requires \fByylex\fR to continue processing with
815 826 another source of input, then the application can include a function
816 827 \fByywrap\fR, which associates another file with the external variable
817 828 \fBFILE\fR *\fIyyin\fR and will return a value of zero.
818 829 .RE
819 830
820 831 .sp
821 832 .ne 2
822 833 .na
823 834 \fB\fBint\fR \fBmain(int\fR \fB\fIargc\fR,\fR \fBchar\fR \fB*\fIargv\fR[\|])\fR
824 835 \fR
825 836 .ad
826 837 .sp .6
827 838 .RS 4n
828 839 Calls \fByylex\fR to perform lexical analysis, then exits. The user code can
829 840 contain \fBmain\fR to perform application-specific operations, calling
830 841 \fByylex\fR as applicable.
831 842 .RE
832 843
833 844 .sp
834 845 .LP
835 846 The reason for breaking these functions into two lists is that only those
836 847 functions in \fBlibl.a\fR can be reliably redefined by a portable application.
837 848 .sp
838 849 .LP
839 850 Except for \fBinput\fR, \fBunput\fR and \fBmain\fR, all external and static
840 851 names generated by \fBlex\fR begin with the prefix \fByy\fR or \fBYY\fR.
841 852 .SH USAGE
842 853 .sp
843 854 .LP
844 855 Portable applications are warned that in the \fBRules in lex\fR section, an
845 856 \fBERE\fR without an action is not acceptable, but need not be detected as
846 857 erroneous by \fBlex\fR. This may result in compilation or run-time errors.
847 858 .sp
848 859 .LP
849 860 The purpose of \fBinput\fR is to take characters off the input stream and
850 861 discard them as far as the lexical analysis is concerned. A common use is to
851 862 discard the body of a comment once the beginning of a comment is recognized.
852 863 .sp
853 864 .LP
854 865 The \fBlex\fR utility is not fully internationalized in its treatment of
855 866 regular expressions in the \fBlex\fR source code or generated lexical analyzer.
856 867 It would seem desirable to have the lexical analyzer interpret the regular
857 868 expressions given in the \fBlex\fR source according to the environment
858 869 specified when the lexical analyzer is executed, but this is not possible with
859 870 the current \fBlex\fR technology. Furthermore, the very nature of the lexical
860 871 analyzers produced by \fBlex\fR must be closely tied to the lexical
861 872 requirements of the input language being described, which will frequently be
862 873 locale-specific anyway. (For example, writing an analyzer that is used for
863 874 French text will not automatically be useful for processing other languages.)
864 875 .SH EXAMPLES
865 876 .LP
866 877 \fBExample 1 \fRUsing lex
867 878 .sp
868 879 .LP
869 880 The following is an example of a \fBlex\fR program that implements a
870 881 rudimentary scanner for a Pascal-like syntax:
871 882
872 883 .sp
873 884 .in +2
874 885 .nf
875 886 %{
876 887 /* need this for the call to atof() below */
877 888 #include <math.h>
878 889 /* need this for printf(), fopen() and stdin below */
879 890 #include <stdio.h>
880 891 %}
881 892
882 893 DIGIT [0-9]
883 894 ID [a-z][a-z0-9]*
884 895 %%
885 896
886 897 {DIGIT}+ {
887 898 printf("An integer: %s (%d)\en", yytext,
888 899 atoi(yytext));
889 900 }
890 901
891 902 {DIGIT}+"."{DIGIT}* {
892 903 printf("A float: %s (%g)\en", yytext,
893 904 atof(yytext));
894 905 }
895 906
896 907 if|then|begin|end|procedure|function {
897 908 printf("A keyword: %s\en", yytext);
898 909 }
899 910
900 911 {ID} printf("An identifier: %s\en", yytext);
901 912
902 913 "+"|"-"|"*"|"/" printf("An operator: %s\en", yytext);
903 914
904 915 "{"[^}\en]*"}" /* eat up one-line comments */
905 916
906 917 [ \et\en]+ /* eat up white space */
907 918
908 919 \&. printf("Unrecognized character: %s\en", yytext);
909 920
910 921 %%
911 922
912 923 int main(int argc, char *argv[\|])
913 924 {
914 925 ++argv, --argc; /* skip over program name */
915 926 if (argc > 0)
916 927 yyin = fopen(argv[0], "r");
917 928 else
918 929 yyin = stdin;
919 930
920 931 yylex();
921 932 }
922 933 .fi
923 934 .in -2
924 935 .sp
925 936
926 937 .SH ENVIRONMENT VARIABLES
927 938 .sp
928 939 .LP
929 940 See \fBenviron\fR(5) for descriptions of the following environment variables
930 941 that affect the execution of \fBlex\fR: \fBLANG\fR, \fBLC_ALL\fR,
931 942 \fBLC_COLLATE\fR, \fBLC_CTYPE\fR, \fBLC_MESSAGES\fR, and \fBNLSPATH\fR.
932 943 .SH EXIT STATUS
933 944 .sp
934 945 .LP
935 946 The following exit values are returned:
936 947 .sp
937 948 .ne 2
938 949 .na
939 950 \fB\fB0\fR \fR
940 951 .ad
941 952 .RS 7n
942 953 Successful completion.
943 954 .RE
944 955
945 956 .sp
946 957 .ne 2
947 958 .na
948 959 \fB\fB>0\fR \fR
949 960 .ad
950 961 .RS 7n
951 962 An error occurred.
952 963 .RE
953 964
954 965 .SH ATTRIBUTES
955 966 .sp
956 967 .LP
957 968 See \fBattributes\fR(5) for descriptions of the following attributes:
958 969 .sp
959 970
960 971 .sp
961 972 .TS
962 973 box;
963 974 c | c
964 975 l | l .
965 976 ATTRIBUTE TYPE ATTRIBUTE VALUE
966 977 _
967 978 Interface Stability Standard
968 979 .TE
969 980
970 981 .SH SEE ALSO
971 982 .sp
972 983 .LP
973 984 \fByacc\fR(1), \fBattributes\fR(5), \fBenviron\fR(5), \fBregex\fR(5),
974 985 \fBstandards\fR(5)
975 986 .SH NOTES
976 987 .sp
977 988 .LP
978 989 If routines such as \fByyback()\fR, \fByywrap()\fR, and \fByylock()\fR
979 990 in \fB\|.l\fR (ell) files are to be external C functions, the command line to
980 991 compile a C++ program must define the \fB__EXTERN_C__\fR macro. For example:
981 992 .sp
982 993 .in +2
983 994 .nf
984 995 example% \fBCC -D__EXTERN_C__ ... file\fR
985 996 .fi
986 997 .in -2
987 998 .sp
988 999
|
↓ open down ↓ |
864 lines elided |
↑ open up ↑ |
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX