1 .\" "
2 .\" " The contents of this file are subject to the terms of the
3 .\" " Common Development and Distribution License (the "License").
4 .\" " You may not use this file except in compliance with the License.
5 .\" "
6 .\" " You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
7 .\" " or http://www.opensolaris.org/os/licensing.
8 .\" " See the License for the specific language governing permissions
9 .\" " and limitations under the License.
10 .\" "
11 .\" " When distributing Covered Code, include this CDDL HEADER in each
12 .\" " file and include the License file at usr/src/OPENSOLARIS.LICENSE.
13 .\" " If applicable, add the following below this CDDL HEADER, with the
14 .\" " fields enclosed by brackets "[]" replaced with your own identifying
15 .\" " information: Portions Copyright [yyyy] [name of copyright owner]
16 .\" "
17 .\" " CDDL HEADER END
18 .\" "
19 .\" "Copyright (c) 1999, 2010, Oracle and/or its affiliates. All rights reserved.
20 .\" "Copyright 2012 Joshua M. Clulow <josh@sysmgr.org>
21 .\" "
22 .TH nightly 1 "6 July 2010"
23 .SH NAME
24 .I nightly
25 \- build an OS-Net consolidation overnight
26 .SH SYNOPSIS
27 \fBnightly [-in] [-V VERS] <env_file>\fP
28 .LP
29 .SH DESCRIPTION
30 .IX "OS-Net build tools" "nightly" "" "\fBnightly\fP"
31 .LP
32 .I nightly,
33 the mother of all build scripts,
34 can bringover, build, archive, package, error check, and
35 generally do everything it takes to
36 turn OS/Net consolidation source code into useful stuff.
37 It is customizable to permit you to run anything from a
38 simple build to all of the cross-checking a gatekeeper
39 needs. The advantage to using
40 .I nightly
41 is that you build things correctly, consistently and
42 automatically, with the best practices; building with
43 .I nightly
44 can mean never having to say you're sorry to your
45 gatekeeper.
46 .LP
47 More
48 specifically,
49 .I nightly
50 performs the following tasks, in order, if
51 all these things are desired:
52 .LP
53 .RS
54 .TP
55 \(bu
56 perform a "make clobber" to clean up old binaries
57 .TP
58 \(bu
59 bringover from the identified parent gate/clone
60 .TP
61 \(bu
62 perform non-DEBUG and DEBUG builds
63 .TP
64 \(bu
65 list proto area files and compare with previous list
66 .TP
67 \(bu
68 copy updated proto area to parent
69 .TP
70 \(bu
71 list shared lib interface and compare with previous list
72 .TP
73 \(bu
74 perform a "make lint" of the kernel and report errors
75 .TP
76 \(bu
77 perform a "make check" to report hdrchk/cstyle errors
78 .TP
79 \(bu
80 report the presence of any core files
81 .TP
82 \(bu
83 check the ELF runtime attributes of all dynamic objects
84 .TP
85 \(bu
86 check for unreferenced files
87 .TP
88 \(bu
89 report on which proto area objects have changed (since the last build)
90 .TP
91 \(bu
92 report the total build time
93 .TP
94 \(bu
95 save a detailed log file for reference
96 .TP
97 \(bu
98 mail the user a summary of the completed build
99 .RE
100 .LP
101 The actions of the script are almost completely determined by
102 the environment variables in the
103 .I env
104 file, the only necessary argument. Ths only thing you really
105 need to use
106 .I nightly
107 is an
108 .I env
109 file that does what you want.
110 .LP
111 Like most of the other build tools in usr/src/tools, this script tends
112 to change on a fairly regular basis; do not expect to be able to build
113 OS/Net with a version of nightly significantly older than your source
114 tree. It has what is effectively a Consolidation Private relationship
115 to other build tools and with many parts of the OS/Net makefiles,
116 although it may also be used to build other consolidations.
117 .LP
118 .SH NIGHTLY_OPTIONS
119 The environment variable NIGHTLY_OPTIONS controls the actions
120 .I nightly
121 will take as it proceeds.
122 The -i, -n, +t and -V options may also be used from the command
123 line to control the actions without editing your environment file.
124 The -i and -n options complete the build more quickly by bypassing
125 some actions. If NIGHTLY_OPTIONS is not set, then "-Bmt" build
126 options will be used.
127
128 .B Basic action options
129 .TP 10
130 .B \-D
131 Do a build with DEBUG on (non-DEBUG is built by default)
132 .TP
133 .B \-F
134 Do _not_ do a non-DEBUG build (use with -D to get just a DEBUG build)
135 .TP
136 .B \-M
137 Do not run pmodes (safe file permission checker)
138 .TP
139 .B \-i
140 Do an incremental build, suppressing the "make clobber" that by
141 default removes all existing binaries and derived files. From the
142 command line, -i also suppresses the lint pass and the cstyle/hdrchk
143 pass
144 .TP
145 .B \-n
146 Suppress the bringover so that the build will start immediately with
147 current source code
148 .TP
149 .B \-o
150 Do an "old style" (pre-S10) build using root privileges to set OWNER
151 and GROUP from the Makefiles.
152 .TP
153 .B \-p
154 Create packages for regular install
155 .TP
156 .B \-U
157 Update proto area in the parent workspace
158 .TP
159 .B \-u
160 Update the parent workspace with files generated by the build, as follows.
161 .RS
162 .TP
163 \(bu
164 Copy proto_list_${MACH} and friends to usr/src in the parent.
165 .TP
166 \(bu
167 When used with -f, build a usr/src/unrefmaster.out in
168 the parent by merging all the usr/src/unref-${MACH}.out files in the
169 parent.
170 .TP
171 \(bu
172 When used with -A or -r, copy the contents of the resulting
173 ELF-data.${MACH} directory to usr/src/ELF-data.${MACH} in the parent
174 workspace.
175 .RE
176 .TP
177 .B \-m
178 Send mail to $MAILTO at end of build
179 .TP
180 .B \-t
181 Build and use the tools in $SRC/tools (default setting).
182 .TP
183 .B \+t
184 Use the build tools in "$ONBLD_TOOLS/bin".
185
186 .LP
187 .B Code checking options
188 .TP 10
189 .B \-A
190 Check for ABI discrepancies in .so files.
191 It is only required for shared object developers when there is an
192 addition, deletion or change of interface in the .so files.
193 .TP
194 .B \-C
195 Check for cstyle/hdrchk errors
196 .TP
197 .B \-f
198 Check for unreferenced files. Since the full workspace must be built
199 in order to accurately identify unreferenced files, -f is ignored for
200 incremental (-i) builds, or builds that do not include -l, and -p.
201 .TP
202 .B \-r
203 Check the ELF runtime attributes of all dynamic objects
204 .TP
205 .B \-l
206 Do "make lint" in $LINTDIRS (default: $SRC n)
207 .TP
208 .B \-N
209 Do not run protocmp or checkpaths (note: this option is not
210 recommended, especially in conjunction with the \-p option)
211 .TP
212 .B \-W
213 Do not report warnings (for freeware gate ONLY)
214 .TP
215 .B \-w
216 Report which proto area objects differ between this and the last build.
217 See wsdiff(1) for details. Note that the proto areas used for comparison
218 are the last ones constructed as part of the build. As an example, if both
219 a non-debug and debug build are performed (in that order), then the debug
220 proto area will be used for comparison (which might not be what you want).
221 .LP
222 .B Groups of options
223 .TP 10
224 .B \-G
225 Gate keeper default group of options (-u)
226 .TP
227 .B \-I
228 Integration engineer default group of options (-mpu)
229 .TP
230 .B \-R
231 Default group of options for building a release (-mp)
232
233 .LP
234 .B Source Build options
235 .TP 10
236 .B \-S E | D | H
237 Build the Export, Domestic, or Hybrid source product. Only Export and
238 Domestic are truly buildable at this time.
239 .TP 10
240 .B \-S O
241 Simulate an OpenSolaris build on a full tree. This can be used by
242 internal developers to ensure that they haven't broken the build for
243 external developers.
244 .LP
245 Source build options only make sense for a full internal tree (open
246 and closed source). Only one source build option can be specified at
247 a time.
248
249 .LP
250 .B Miscellaneous options
251 .TP 10
252 .B \-O
253 generate deliverables for OpenSolaris. Tarballs containing signed
254 cryptographic binaries and binaries
255 of closed-source components are put in $CODEMGR_WS. (The
256 cryptographic tarballs are copies of the
257 ones that are put in the parent directory of
258 $PKGARCHIVE.)
259 .TP 10
260 .B \-V VERS
261 set the build version string to VERS, overriding VERSION
262 .TP
263 .B \-X
264 Copies the proto area and packages from the IHV and IHV-bin gates into the
265 nightly proto and package areas. This is only available on i386. See
266 .B REALMODE ENVIRONMENT VARIABLES
267 and
268 .B BUILDING THE IHV WORKSPACE
269 below.
270
271 .LP
272 .SH ENVIRONMENT VARIABLES
273 .LP
274 Here is a list of prominent environment variables that
275 .I nightly
276 references and the meaning of each variable.
277 .LP
278 .RE
279 .B CODEMGR_WS
280 .RS 5
281 The root of your workspace, including whatever metadata is kept by
282 the source code management system. This is the workspace in which the
283 build will be done.
284 .RE
285 .LP
286 .B PARENT_WS
287 .RS 5
288 The root of the workspace that is the parent of the
289 one being built. This is particularly relevant for configurations
290 with a main
291 workspace and build workspaces underneath it; see the
292 \-u and \-U
293 options as well as the PKGARCHIVE environment variable, for more
294 information.
295 .RE
296 .LP
297 .B BRINGOVER_WS
298 .RS 5
299 This is the workspace from which
300 .I nightly
301 will fetch sources to either populate or update your workspace;
302 it defaults to $CLONE_WS.
303 .RE
304 .LP
305 .B CLOSED_BRINGOVER_WS
306 .RS 5
307 A full Mercurial workspace has two repositories: one for open source
308 and one for closed source. If this variable is non-null,
309 .I nightly
310 will pull from the repository that it names to get the closed source.
311 It defaults to $CLOSED_CLONE_WS.
312 .LP
313 If $CODEMGR_WS already exists and contains only the open repository,
314 .I nightly
315 will ignore this variable; you'll need to pull the closed repository
316 by hand if you want it.
317 .RE
318 .LP
319 .B CLONE_WS
320 .RS 5
321 This is the workspace from which
322 .I nightly
323 will fetch sources by default. This is
324 often distinct from the parent, particularly if the parent is a gate.
325 .RE
326 .LP
327 .B CLOSED_CLONE_WS
328 .RS 5
329 This is the default closed-source Mercurial repository that
330 .I nightly
331 might pull from (see
332 .B CLOSED_BRINGOVER_WS
333 for details).
334 .RE
335 .LP
336 .B SRC
337 .RS 5
338 Root of OS-Net source code, referenced by the Makefiles. It is
339 the starting point of build activity. It should be expressed
340 in terms of $CODEMGR_WS.
341 .RE
342 .LP
343 .B ROOT
344 .RS 5
345 Root of the proto area for the build. The makefiles direct
346 installation of build products to this area and
347 direct references to these files by builds of commands and other
348 targets. It should be expressed in terms of $CODEMGR_WS.
349 .LP
350 If $MULTI_PROTO is "no", $ROOT may contain a DEBUG or non-DEBUG
351 build. If $MULTI_PROTO is "yes", $ROOT contains the DEBUG build and
352 $ROOT-nd contains the non-DEBUG build.
353 .LP
354 For OpenSolaris deliveries (\fB\-O\fR), $ROOT-closed contains a parallel
355 proto area containing the DEBUG build of just usr/closed components, and
356 $ROOT-nd-closed contains the non-DEBUG equivalent.
357 .RE
358 .LP
359 .B TOOLS_ROOT
360 .RS 5
361 Root of the tools proto area for the build. The makefiles direct
362 installation of tools build products to this area. Unless \fB+t\fR
363 is part of $NIGHTLY_OPTIONS, these tools will be used during the
364 build.
365 .LP
366 As built by nightly, this will always contain non-DEBUG objects.
367 Therefore, this will always have a -nd suffix, regardless of
368 $MULTI_PROTO.
369 .RE
370 .LP
371 .B MACH
372 .RS 5
373 The instruction set architecture of the build machine as given
374 by \fIuname -p\fP, e.g. sparc, i386.
375 .RE
376 .LP
377 .B LOCKNAME
378 .RS 5
379 The name of the file used to lock out multiple runs of
380 .IR nightly .
381 This should generally be left to the default setting.
382 .RE
383 .LP
384 .B ATLOG
385 .RS 5
386 The location of the log directory maintained by
387 .IR nightly .
388 This should generally be left to the default setting.
389 .RE
390 .LP
391 .B LOGFILE
392 .RS 5
393 The name of the log file in the $ATLOG directory maintained by
394 .IR nightly .
395 This should generally be left to the default setting.
396 .RE
397 .LP
398 .B STAFFER
399 .RS 5
400 The non-root account to use on the build machine for the
401 bringover from the clone or parent workspace.
402 This may not be the same identify used by the SCM.
403 .RE
404 .LP
405 .B MAILTO
406 .RS 5
407 The address to be used to send completion e-mail at the end of
408 the build (for the \-m option).
409 .RE
410 .LP
411 .B MAILFROM
412 .RS 5
413 The address to be used for From: in the completion e-mail at the
414 end of the build (for the \-m option).
415 .RE
416 .LP
417 .B REF_PROTO_LIST
418 .RS 5
419 Name of file used with protocmp to compare proto area contents.
420 .RE
421 .LP
422 .B PARENT_ROOT
423 .RS 5
424 The parent root, which is the destination for copying the proto
425 area(s) when using the \-U option.
426 .RE
427 .LP
428 .B PARENT_TOOLS_ROOT
429 .RS 5
430 The parent tools root, which is the destination for copying the tools
431 proto area when using the \-U option.
432 .RE
433 .LP
434 .B RELEASE
435 .RS 5
436 The release version number to be used; e.g., 5.10.1 (Note: this is set
437 in Makefile.master and should not normally be overridden).
438 .RE
439 .LP
440 .B VERSION
441 .RS 5
442 The version text string to be used; e.g., "onnv:`date '+%Y-%m-%d'`".
443 .RE
444 .LP
445 .B RELEASE_DATE
446 .RS 5
447 The release date text to be used; e.g., October 2009. If not set in
448 your environment file, then this text defaults to the output from
449 $(LC_ALL=C date +"%B %Y"); e.g., "October 2009".
450 .RE
451 .LP
452 .B INTERNAL_RELEASE_BUILD
453 .RS 5
454 See Makefile.master - but it mostly controls id strings. Generally,
455 let
456 .I nightly
457 set this for you.
458 .RE
459 .LP
460 .B RELEASE_BUILD
461 .RS 5
462 Define this to build a release with a non-DEBUG kernel.
463 Generally, let
464 .I nightly
465 set this for you based on its options.
466 .RE
467 .LP
468 .B PKGARCHIVE
469 .RS 5
470 The destination for packages. This may be relative to
471 $CODEMGR_WS for private packages or relative to $PARENT_WS
472 if you have different workspaces for different architectures
473 but want one hierarchy of packages.
474 .RE
475 .LP
476 .B MAKEFLAGS
477 .RS 5
478 Set default flags to make; e.g., -k to build all targets regardless of errors.
479 .RE
480 .LP
481 .B UT_NO_USAGE_TRACKING
482 .RS 5
483 Disables usage reporting by listed Devpro tools. Otherwise it sends mail
484 to some Devpro machine every time the tools are used.
485 .RE
486 .LP
487 .B LINTDIRS
488 .RS 5
489 Directories to lint with the \-l option.
490 .RE
491 .LP
492 .B BUILD_TOOLS
493 .RS 5
494 BUILD_TOOLS is the root of all tools including the compilers; e.g.,
495 /ws/onnv-tools. It is used by the makefile system, but not nightly.
496 .RE
497 .LP
498 .B ONBLD_TOOLS
499 .RS 5
500 ONBLD_TOOLS is the root of all the tools that are part of SUNWonbld; e.g.,
501 /ws/onnv-tools/onbld. By default, it is derived from
502 .BR BUILD_TOOLS .
503 It is used by the makefile system, but not nightly.
504 .RE
505 .LP
506 .B SPRO_ROOT
507 .RS 5
508 The gate-defined default location for the Sun compilers, e.g.
509 /ws/onnv-tools/SUNWspro. By default, it is derived from
510 .BR BUILD_TOOLS .
511 It is used by the makefile system, but not nightly.
512 .RE
513 .LP
514 .B JAVA_ROOT
515 .RS 5
516 The location for the java compilers for the build, generally /usr/java.
517 .RE
518 .LP
519 .B OPTHOME
520 .RS 5
521 The gate-defined default location of things formerly in /opt; e.g.,
522 /ws/onnv-tools. This is used by nightly, but not the makefiles.
523 .RE
524 .LP
525 .B TEAMWARE
526 .RS 5
527 The gate-defined default location for the Teamware tools; e.g.,
528 /ws/onnv-tools/SUNWspro. By default, it is derived from
529 .BR OPTHOME .
530 This is used by nightly, but not the makefiles. There is no
531 corresponding variable for Mercurial or Subversion, which are assumed
532 to be installed in the default path.
533 .RE
534 .LP
535 .B OPEN_SRCDIR
536 .RS 5
537 The open source tree is copied to this directory when simulating an
538 OpenSolaris build (\fB\-S O\fR). It defaults to $CODEMGR_WS/open_src.
539 .LP
540 .RE
541 .B ON_CLOSED_BINS
542 .RS 5
543 OpenSolaris builds do not contain the closed source tree. Instead,
544 the developer downloads a closed binaries tree and unpacks it.
545 .B ON_CLOSED_BINS
546 tells nightly
547 where to find these closed binaries, so that it can add them into the
548 build.
549 .LP
550 .RE
551 .B ON_CRYPTO_BINS
552 .RS 5
553 This is the path to a compressed tarball that contains debug
554 cryptographic binaries that have been signed to allow execution
555 outside of Sun, e.g., $PARENT_WS/packages/$MACH/on-crypto.$MACH.bz2.
556 .I nightly
557 will automatically adjust the path for non-debug builds. This tarball
558 is needed if the closed-source tree is not present. Also, it is
559 usually needed when generating OpenSolaris deliverables from a project
560 workspace. This is because most projects do not have access to the
561 necessary key and certificate that would let them sign their own
562 cryptographic binaries.
563 .LP
564 .RE
565 .B CHECK_PATHS
566 .RS 5
567 Normally, nightly runs the 'checkpaths' script to check for
568 discrepancies among the files that list paths to other files, such as
569 exception lists and req.flg. Set this flag to 'n' to disable this
570 check, which appears in the nightly output as "Check lists of files."
571 .RE
572 .LP
573 .B CHECK_DMAKE
574 .RS 5
575 Nightly validates that the version of dmake encountered is known to be
576 safe to use. Set this flag to 'n' to disable this test, allowing any
577 version of dmake to be used.
578 .RE
579 .LP
580 .B MULTI_PROTO
581 .RS 5
582 If "no" (the default),
583 .I nightly
584 will reuse $ROOT for both the DEBUG and non-DEBUG builds. If "yes",
585 the DEBUG build will go in $ROOT and the non-DEBUG build will go in
586 $ROOT-nd. Other values will be treated as "no". Use of the
587 .B \-O
588 flag forces MULTI_PROTO to "yes".
589 .RE
590 .LP
591 .SH NIGHTLY HOOK ENVIRONMENT VARIABLES
592 .LP
593 Several optional environment variables may specify commands to run at
594 various points during the build. Commands specified in the hook
595 variable will be run in a subshell; command output will be appended to
596 the mail message and log file. If the hook exits with a non-zero
597 status, the build is aborted immediately. Environment variables
598 defined in the environment file will be available.
599 .LP
600 .B SYS_PRE_NIGHTLY
601 .RS 5
602 Run just after the workspace lock is acquired. This is reserved for
603 per-build-machine customizations and should be set only in /etc/nightly.conf
604 .RE
605 .LP
606 .B PRE_NIGHTLY
607 .RS 5
608 Run just after SYS_PRE_NIGHTLY.
609 .RE
610 .LP
611 .B PRE_BRINGOVER
612 .RS 5
613 Run just before bringover is started; not run if no bringover is done.
614 .RE
615 .LP
616 .B POST_BRINGOVER
617 .RS 5
618 Run just after bringover completes; not run if no bringover is done.
619 .RE
620 .LP
621 .B POST_NIGHTLY
622 .RS 5
623 Run after the build completes, with the return status of nightly - one
624 of "Completed", "Interrupted", or "Failed" - available in the
625 environment variable NIGHTLY_STATUS.
626 .RE
627 .LP
628 .B SYS_POST_NIGHTLY
629 .RS 5
630 This is reserved for per-build-machine customizations, and runs
631 immedately after POST_NIGHTLY.
632 .RE
633 .LP
634 .SH REALMODE ENVIRONMENT VARIABLES
635 .LP
636 The following environment variables referenced by
637 .I nightly
638 are only required when the -X option is used.
639 .LP
640 .RE
641 .B IA32_IHV_WS
642 .RS 5
643 Reference to the IHV workspace containing IHV driver binaries.
644 The IHV workspace must be fully built before starting the ON realmode build.
645 .LP
646 .RE
647 .B IA32_IHV_ROOT
648 .RS 5
649 Reference to the IHV workspace proto area.
650 The IHV workspace must be fully built before starting the ON realmode build.
651 .LP
652 .RE
653 .B IA32_IHV_PKGS
654 .RS 5
655 Reference to the IHV workspace packages. If this is empty or the directory
656 is non-existent, then nightly will skip copying the packages.
657 .LP
658 .RE
659 .B IA32_IHV_BINARY_PKGS
660 .RS 5
661 Reference to binary-only IHV packages. If this is empty or the directory
662 is non-existent, then nightly will skip copying the packages.
663 .LP
664 .RE
665 .B SPARC_RM_PKGARCHIVE
666 .RS 5
667 Destination for sparc realmode package SUNWrmodu.
668 Yes, this sparc package really is built on x86.
669 .SH FILES
670 .LP
671 .RS 5
672 /etc/nightly.conf
673 .RE
674 .LP
675 If present, nightly executes this file just prior to executing the
676 .I env
677 file.
678 .SH BUILDING THE IHV WORKSPACE
679 .LP
680 The IHV workspace can be built with
681 .I nightly.
682 The recommended options are:
683 .LP
684 .RS 5
685 NIGHTLY_OPTIONS="-pmWN"
686 .RE
687 .LP
688 None of the realmode environment variables needed for ON realmode builds
689 are required to build the IHV workspace.
690 .SH EXAMPLES
691 .LP
692 Start with the example file in usr/src/tools/env/developer.sh
693 (or gatekeeper.sh), copy to myenv and make your changes.
694 .LP
695 .PD 0
696 # grep NIGHTLY_OPTIONS myenv
697 .LP
698 NIGHTLY_OPTIONS="-ACrlapDm"
699 .LP
700 export NIGHTLY_OPTIONS
701 .LP
702 # /opt/onbld/bin/nightly -i myenv
703 .PD
704 .LP
705 .SH SEE ALSO
706 .BR bldenv (1)