TH_DEFINE(1M) Maintenance Commands TH_DEFINE(1M) NNAAMMEE th_define - create fault injection test harness error specifications SSYYNNOOPPSSIISS tthh__ddeeffiinnee [--nn _n_a_m_e --ii _i_n_s_t_a_n_c_e| --PP _p_a_t_h] [--aa _a_c_c___t_y_p_e_s] [--rr _r_e_g___n_u_m_b_e_r] [--ll _o_f_f_s_e_t [_l_e_n_g_t_h]] [--cc _c_o_u_n_t [_f_a_i_l_c_o_u_n_t]] [--oo _o_p_e_r_a_t_o_r [_o_p_e_r_a_n_d]] [--ff _a_c_c___c_h_k] [--ww _m_a_x___w_a_i_t___p_e_r_i_o_d [_r_e_p_o_r_t___i_n_t_e_r_v_a_l]] oorr tthh__ddeeffiinnee [--nn _n_a_m_e --ii _i_n_s_t_a_n_c_e| --PP _p_a_t_h] [--aa log [_a_c_c___t_y_p_e_s] [--rr _r_e_g___n_u_m_b_e_r] [--ll _o_f_f_s_e_t [_l_e_n_g_t_h]]] [--cc _c_o_u_n_t [_f_a_i_l_c_o_u_n_t]] [--ss _c_o_l_l_e_c_t___t_i_m_e] [--pp _p_o_l_i_c_y] [--xx _f_l_a_g_s] [--CC _c_o_m_m_e_n_t___s_t_r_i_n_g] [--ee _f_i_x_u_p___s_c_r_i_p_t [_a_r_g_s]] oorr tthh__ddeeffiinnee [--hh] DDEESSCCRRIIPPTTIIOONN The tthh__ddeeffiinnee utility provides an interface to the bbuuss__ooppss fault injection bbooffii device driver for defining error injection specifications (referred to as errdefs). An errdef corresponds to a specification of how to corrupt a device driver's accesses to its hardware. The command line arguments determine the precise nature of the fault to be injected. If the supplied arguments define a consistent errdef, the tthh__ddeeffiinnee process will store the errdef with the bbooffii driver and suspend itself until the criteria given by the errdef become satisfied (in practice, this will occur when the access counts go to zero). You use the tthh__mmaannaaggee(1M) command with the ssttaarrtt option to activate the resulting errdef. The effect of tthh__mmaannaaggee with the ssttaarrtt option is that the bbooffii driver acts upon the errdef by matching the number of hardware accesses--specified in _c_o_u_n_t, that are of the type specified in _a_c_c___t_y_p_e_s, made by instance number _i_n_s_t_a_n_c_e--of the driver whose name is _n_a_m_e, (or by the driver instance specified by _p_a_t_h) to the register set (or DMA handle) specified by _r_e_g___n_u_m_b_e_r, that lie within the range _o_f_f_s_e_t to _o_f_f_s_e_t + _l_e_n_g_t_h from the beginning of the register set or DMA handle. It then applies _o_p_e_r_a_t_o_r and _o_p_e_r_a_n_d to the next _f_a_i_l_c_o_u_n_t matching accesses. If _a_c_c___t_y_p_e_s includes lloogg, tthh__ddeeffiinnee runs in automatic test script generation mode, and a set of test scripts (written in the Korn shell) is created and placed in a sub-directory of the current directory with the name _<_d_r_i_v_e_r_>.test._<_i_d_> (for example, ggllmm..tteesstt..997788117777110066). A separate, executable script is generated for each access handle that matches the logging criteria. The log of accesses is placed at the top of each script as a record of the session. If the current directory is not writable, file output is written to standard output. The base name of each test file is the driver name, and the extension is a number that discriminates between different access handles. A control script (with the same name as the created test directory) is generated that will run all the test scripts sequentially. Executing the scripts will install, and then activate, the resulting error definitions. Error definitions are activated sequentially and the driver instance under test is taken offline and brought back online before each test (refer to the --ee option for more information). By default, logging applies to all PPIIOO accesses, all interrupts, and all DMA accesses to and from areas mapped for both reading and writing. You can constrain logging by specifying additional _a_c_c___t_y_p_e_s, _r_e_g___n_u_m_b_e_r, _o_f_f_s_e_t and _l_e_n_g_t_h. Logging will continue for _c_o_u_n_t matching accesses, with an optional time limit of _c_o_l_l_e_c_t___t_i_m_e seconds. Either the --nn or --PP option must be provided. The other options are optional. If an option (other than --aa) is specified multiple times, only the final value for the option is used. If an option is not specified, its associated value is set to an appropriate default, which will provide maximal error coverage as described below. OOPPTTIIOONNSS The following options are available: --nn _n_a_m_e Specify the name of the driver to test. (String) --ii _i_n_s_t_a_n_c_e Test only the specified driver instance (-1 matches all instances of driver). (Numeric) --PP _p_a_t_h Specify the full device path of the driver to test. (String) --rr _r_e_g___n_u_m_b_e_r Test only the given register set or DMA handle (-1 matches all register sets and DMA handles). (Numeric) --aa _a_c_c___t_y_p_e_s Only the specified access types will be matched. Valid values for the _a_c_c___t_y_p_e_s argument are lloogg, ppiioo, ppiioo__rr, ppiioo__ww, ddmmaa, ddmmaa__rr, ddmmaa__ww and iinnttrr. Multiple access types, separated by spaces, can be specified. The default is to match all hardware accesses. If _a_c_c___t_y_p_e_s is set to lloogg, logging will match all PPIIOO accesses, interrupts and DMA accesses to and from areas mapped for both reading and writing. lloogg can be combined with other _a_c_c___t_y_p_e_s, in which case the matching condition for logging will be restricted to the specified additional _a_c_c___t_y_p_e_s. Note that ddmmaa__rr will match only DMA handles mapped for reading only; ddmmaa__ww will match only DMA handles mapped for writing only; ddmmaa will match only DMA handles mapped for both reading and writing. --ll _o_f_f_s_e_t [[_l_e_n_g_t_h]] Constrain the range of qualifying accesses. The _o_f_f_s_e_t and _l_e_n_g_t_h arguments indicate that any access of the type specified with the --aa option, to the register set or DMA handle specified with the --rr option, lie at least _o_f_f_s_e_t bytes into the register set or DMA handle and at most _o_f_f_s_e_t + _l_e_n_g_t_h bytes into it. The default for _o_f_f_s_e_t is 0. The default for _l_e_n_g_t_h is the maximum value that can be placed in an ooffffsseett__tt C data type (see ttyyppeess..hh). Negative values are converted into unsigned quantities. Thus, tthh__ddeeffiinnee --ll 0 --11 is maximal. --cc _c_o_u_n_t[[_f_a_i_l_c_o_u_n_t]] Wait for _c_o_u_n_t number of matching accesses, then apply an operator and operand (see the --oo option) to the next _f_a_i_l_c_o_u_n_t number of matching accesses. If the access type (see the --aa option) includes logging, the number of logged accesses is given by _c_o_u_n_t + _f_a_i_l_c_o_u_n_t - 1. The -1 is required because the last access coincides with the first faulting access. Note that access logging may be combined with error injection if _f_a_i_l_c_o_u_n_t and _o_p_e_r_a_t_o_r are nonzero and if the access type includes logging and any of the other access types (ppiioo, ddmmaa and iinnttrr) See the description of access types in the definition of the --aa option, above. When the _c_o_u_n_t and _f_a_i_l_c_o_u_n_t fields reach zero, the status of the errdef is reported to standard output. When all active errdefs created by the tthh__ddeeffiinnee process complete, the process exits. If _a_c_c___t_y_p_e_s includes lloogg, _c_o_u_n_t determines how many accesses to log. If _c_o_u_n_t is not specified, a default value is used. If _f_a_i_l_c_o_u_n_t is set in this mode, it will simply increase the number of accesses logged by a further _f_a_i_l_c_o_u_n_t - 1. --oo _o_p_e_r_a_t_o_r [[_o_p_e_r_a_n_d]] For qualifying PIO read and write accesses, the value read from or written to the hardware is corrupted according to the value of _o_p_e_r_a_t_o_r: EEQQ _o_p_e_r_a_n_d is returned to the driver. OORR _o_p_e_r_a_n_d is bitwise ORed with the real value. AANNDD _o_p_e_r_a_n_d is bitwise ANDed with the real value. XXOORR _o_p_e_r_a_n_d is bitwise XORed with the real value. For PIO write accesses, the following operator is allowed: NNOO Simply ignore the driver's attempt to write to the hardware. Note that a driver performs PIO via the ddddii__ggeett_X(), ddddii__ppuutt_X(), ddddii__rreepp__ggeett_X() and ddddii__rreepp__ppuutt_X() routines (where _X is 8, 16, 32 or 64). Accesses made using ddddii__ggeett_X() and ddddii__ppuutt_X() are treated as a single access, whereas an access made using the ddddii__rreepp__**(9F) routines are broken down into their respective number of accesses, as given by the _r_e_p_c_o_u_n_t parameter to these DDI calls. If the access is performed via a DMA handle, _o_p_e_r_a_t_o_r and _v_a_l_u_e are applied to every access that comprises the DMA request. If interference with interrupts has been requested then the operator may take any of the following values: DDEELLAAYY After _c_o_u_n_t accesses (see the --cc option), delay delivery of the next _f_a_i_l_c_o_u_n_t number of interrupts for _o_p_e_r_a_n_d number of microseconds. LLOOSSEE After _c_o_u_n_t number of interrupts, fail to deliver the next _f_a_i_l_c_o_u_n_t number of real interrupts to the driver. EEXXTTRRAA After _c_o_u_n_t number of interrupts, start delivering _o_p_e_r_a_n_d number of extra interrupts for the next _f_a_i_l_c_o_u_n_t number of real interrupts. The default value for _o_p_e_r_a_n_d and _o_p_e_r_a_t_o_r is to corrupt the data access by flipping each bit (XOR with -1). --ff _a_c_c___c_h_k If the _a_c_c___c_h_k parameter is set to 1 or ppiioo, then the driver's calls to ddddii__cchheecckk__aacccc__hhaannddllee(9F) return DDDDII__FFAAIILLUURREE when the access count goes to 1. If the _a_c_c___c_h_k parameter is set to 2 or ddmmaa, then the driver's calls to ddddii__cchheecckk__ddmmaa__hhaannddllee(9F) return DDDDII__FFAAIILLUURREE when the access count goes to 1. --ww _m_a_x___w_a_i_t___p_e_r_i_o_d [[_r_e_p_o_r_t___i_n_t_e_r_v_a_l]] Constrain the period for which an error definition will remain active. The option applies only to non-logging errdefs. If an error definition remains active for _m_a_x___w_a_i_t___p_e_r_i_o_d seconds, the test will be aborted. If _r_e_p_o_r_t___i_n_t_e_r_v_a_l is set to a nonzero value, the current status of the error definition is reported to standard output every _r_e_p_o_r_t___i_n_t_e_r_v_a_l seconds. The default value is zero. The status of the errdef is reported in parsable format (eight fields, each separated by a colon (::) character, the last of which is a string enclosed by double quotes and the remaining seven fields are integers): _f_t:_m_t:_a_c:_f_c:_c_h_k:_e_c:_s:_"_m_e_s_s_a_g_e_" which are defined as follows: _f_t The UTC time when the fault was injected. _m_t The UTC time when the driver reported the fault. _a_c The number of remaining non-faulting accesses. _f_c The number of remaining faulting accesses. _c_h_k The value of the _a_c_c___c_h_k field of the errdef. _e_c The number of fault reports issued by the driver against this errdef (_m_t holds the time of the initial report). _s The severity level reported by the driver. _"_m_e_s_s_a_g_e_" Textual reason why the driver has reported a fault. --hh Display the command usage string. --ss _c_o_l_l_e_c_t___t_i_m_e If _a_c_c___t_y_p_e_s is given with the --aa option and includes lloogg, the errdef will log accesses for _c_o_l_l_e_c_t___t_i_m_e seconds (the default is to log until the log becomes full). Note that, if the errdef specification matches multiple driver handles, multiple logging errdefs are registered with the bbooffii driver and logging terminates when all logs become full or when _c_o_l_l_e_c_t___t_i_m_e expires or when the associated errdefs are cleared. The current state of the log can be checked with the tthh__mmaannaaggee(1M) command, using the bbrrooaaddccaasstt parameter. A log can be terminated by running tthh__mmaannaaggee(1M) with the cclleeaarr__eerrrrddeeffss option or by sending a SSIIGGAALLRRMM signal to the tthh__ddeeffiinnee process. See aallaarrmm(2) for the semantics of SSIIGGAALLRRMM. --pp _p_o_l_i_c_y Applicable when the _a_c_c___t_y_p_e_s option includes lloogg. The parameter modifies the policy used for converting from logged accesses to errdefs. All policies are inclusive: o Use rraarree to bias error definitions toward rare accesses (default). o Use ooppeerraattoorr to produce a separate error definition for each operator type (default). o Use ccoommmmoonn to bias error definitions toward common accesses. o Use mmeeddiiaann to bias error definitions toward median accesses. o Use mmaaxxiimmaall to produce multiple error definitions for duplicate accesses. o Use uunnbbiiaasseedd to create unbiased error definitions. o Use oonneebbyyttee, ttwwoobbyyttee, ffoouurrbbyyttee, or eeiigghhttbbyyttee to select errdefs corresponding to 1, 2, 4 or 8 byte accesses (if chosen, the --xxrr option is enforced in order to ensure that ddddii__rreepp__**(()) calls are decomposed into mmuullttiippllee ssiinnggllee aacccceesssseess). o Use mmuullttiibbyyttee to create error definitions for multibyte accesses performed using ddddii__rreepp__ggeett**(()) and ddddii__rreepp__ppuutt**(()). Policies can be combined by adding together these options. See the NOTES section for further information. --xx _f_l_a_g_s Applicable when the _a_c_c___t_y_p_e_s option includes lloogg. The _f_l_a_g_s parameter modifies the way in which the bbooffii driver logs accesses. It is specified as a string containing any combination of the following letters: ww Continuous logging (that is, the log will wrap when full). tt Timestamp each log entry (access times are in seconds). rr Log repeated I/O as individual accesses (for example, a ddddii__rreepp__ggeett1166(9F) call which has a repcount of _N is logged _N times with each transaction logged as size 2 bytes. Without this option, the default logging behavior is to log this access once only, with a transaction size of twice the _r_e_p_c_o_u_n_t). --CC _c_o_m_m_e_n_t___s_t_r_i_n_g Applicable when the _a_c_c___t_y_p_e_s option includes lloogg. It provides a comment string to be placed in any generated test scripts. The string must be enclosed in double quotes. --ee _f_i_x_u_p___s_c_r_i_p_t [[_a_r_g_s]] Applicable when the _a_c_c___t_y_p_e_s option includes lloogg. The output of a logging errdefs is to generate a test script for each driver access handle. Use this option to embed a command in the resulting script before the errors are injected. The generated test scripts will take an instance offline and bring it back online before injecting errors in order to bring the instance into a known fault-free state. The executable _f_i_x_u_p___s_c_r_i_p_t will be called twice with the set of optional _a_r_g_s-- once just before the instance is taken offline and again after the instance has been brought online. The following variables are passed into the environment of the called executable: DDRRIIVVEERR__PPAATTHH Identifies the device path of the instance. DDRRIIVVEERR__IINNSSTTAANNCCEE Identifies the instance number of the device. DDRRIIVVEERR__UUNNCCOONNFFIIGGUURREE Has the value 1 when the instance is about to be taken offline. DDRRIIVVEERR__CCOONNFFIIGGUURREE Has the value 1 when the instance has just been brought online. Typically, the executable ensures that the device under test is in a suitable state to be taken offline (unconfigured) or in a suitable state for error injection (for example configured, error free and servicing a workload). A minimal script for a network driver could be: #!/bin/ksh driver=xyznetdriver ifnum=$driver$DRIVER_INSTANCE if [[ $DRIVER_CONFIGURE = 1 ]]; then ifconfig $ifnum plumb ifconfig $ifnum ... ifworkload start $ifnum elif [[ $DRIVER_UNCONFIGURE = 1 ]]; then ifworkload stop $ifnum ifconfig $ifnum down ifconfig $ifnum unplumb fi exit $? The --ee option must be the last option on the command line. If the --aa lloogg option is selected but the --ee option is not given, a default script is used. This script repeatedly attempts to detach and then re-attach the device instance under test. EEXXAAMMPPLLEESS EExxaammpplleess ooff EErrrroorr DDeeffiinniittiioonnss tthh__ddeeffiinnee --nn ffoooo --ii 11 --aa lloogg Logs all accesses to all handles used by instance 1 of the ffoooo driver while running the default workload (attaching and detaching the instance). Then generates a set of test scripts to inject appropriate errdefs while running that default workload. tthh__ddeeffiinnee --nn ffoooo --ii 11 --aa lloogg ppiioo Logs PIO accesses to each PIO handle used by instance 1 of the ffoooo driver while running the default workload (attaching and detaching the instance). Then generates a set of test scripts to inject appropriate errdefs while running that default workload. tthh__ddeeffiinnee --nn ffoooo --ii 11 --pp oonneebbyyttee mmeeddiiaann --ee ffiixxuupp aarrgg --nnooww Logs all accesses to all handles used by instance 1 of the ffoooo driver while running the workload defined in the fixup script ffiixxuupp with arguments aarrgg and --nnooww. Then generates a set of test scripts to inject appropriate errdefs while running that workload. The resulting error definitions are requested to focus upon single byte accesses to locations that are accessed a mmeeddiiaann number of times with respect to frequency of access to I/O addresses. tthh__ddeeffiinnee --nn ssee --ll 00xx2200 11 --aa ppiioo__rr --oo OORR 00xx44 --cc 1100 11000000 Simulates a stuck serial chip command by forcing 1000 consecutive read accesses made by any instance of the ssee driver to its command status register, thereby returning status busy. tthh__ddeeffiinnee --nn ffoooo --ii 33 --rr 11 --aa ppiioo__rr --cc 00 11 --ff 11 --oo OORR 00xx110000 Causes 0x100 to be ORed into the next physical I/O read access from any register in register set 1 of instance 3 of the ffoooo driver. Subsequent calls in the driver to ddddii__cchheecckk__aacccc__hhaannddllee(()) return DDDDII__FFAAIILLUURREE. tthh__ddeeffiinnee --nn ffoooo --ii 33 --rr 11 --aa ppiioo__rr --cc 00 11 --oo OORR 00xx00 Causes 0x0 to be ORed into the next physical I/O read access from any register in register set 1 of instance 3 of the ffoooo driver. This is of course a no-op. tthh__ddeeffiinnee --nn ffoooo --ii 33 --rr 11 --ll 00xx88110000 11 --aa ppiioo__rr --cc 00 1100 --oo EEQQ 00xx7700000033 Causes the next ten next physical I/O reads from the register at offset 0x8100 in register set 1 of instance 3 of the ffoooo driver to return 0x70003. tthh__ddeeffiinnee --nn ffoooo --ii 33 --rr 11 --ll 00xx88110000 11 --aa ppiioo__ww --cc 110000 33 --oo AANNDD 00xxffffffffffffffffffffffffeeffffff The next 100 physical I/O writes to the register at offset 0x8100 in register set 1 of instance 3 of the ffoooo driver take place as normal. However, on each of the three subsequent accesses, the 0x1000 bit will be cleared. tthh__ddeeffiinnee --nn ffoooo --ii 33 --rr 11 --ll 00xx88110000 00xx1100 --aa ppiioo__rr --cc 00 11 --ff 11 --oo XXOORR 77 Causes the bottom three bits to have their values toggled for the next physical I/O read access to registers with offsets in the range 0x8100 to 0x8110 in register set 1 of instance 3 of the ffoooo driver. Subsequent calls in the driver to ddddii__cchheecckk__aacccc__hhaannddllee(()) return DDDDII__FFAAIILLUURREE. tthh__ddeeffiinnee --nn ffoooo --ii 33 --aa ppiioo__ww --cc 00 11 --oo NNOO 00 Prevents the next physical I/O write access to any register in any register set of instance 3 of the ffoooo driver from going out on the bus. tthh__ddeeffiinnee --nn ffoooo --ii 33 --ll 00 88119922 --aa ddmmaa__rr --cc 00 11 --oo OORR 77 Causes 0x7 to be ORed into each lloonngg lloonngg in the first 8192 bytes of the next DMA read, using any DMA handle for instance 3 of the ffoooo driver. tthh__ddeeffiinnee --nn ffoooo --ii 33 --rr 22 --ll 00 88 --aa ddmmaa__rr --cc 00 11 --oo OORR 00xx77007700770077007700770077007700 Causes 0x70 to be ORed into each byte of the first lloonngg lloonngg of the next DMA read, using the DMA handle with sequential allocation number 2 for instance 3 of the ffoooo driver. tthh__ddeeffiinnee --nn ffoooo --ii 33 --ll 225566 225566 --aa ddmmaa__ww --cc 00 11 --ff 22 --oo OORR 77 Causes 0x7 to be ORed into each lloonngg lloonngg in the range from offset 256 to offset 512 of the next DMA write, using any DMA handle for instance 3 of the ffoooo driver. Subsequent calls in the driver to ddddii__cchheecckk__ddmmaa__hhaannddllee(()) return DDDDII__FFAAIILLUURREE. tthh__ddeeffiinnee --nn ffoooo --ii 33 --rr 00 --ll 00 88 --aa ddmmaa__ww --cc 110000 33 --oo AANNDD 00xxffffffffffffffffffffffffeeffffff The next 100 DMA writes using the DMA handle with sequential allocation number 0 for instance 3 of the ffoooo driver take place as normal. However, on each of the three subsequent accesses, the 0x1000 bit will be cleared in the first lloonngg lloonngg of the transfer. tthh__ddeeffiinnee --nn ffoooo --ii 33 --aa iinnttrr --cc 00 66 --oo LLOOSSEE 00 Causes the next six interrupts for instance 3 of the ffoooo driver to be lost. tthh__ddeeffiinnee --nn ffoooo --ii 33 --aa iinnttrr --cc 3300 11 --oo EEXXTTRRAA 1100 When the thirty-first subsequent interrupt for instance 3 of the ffoooo driver occurs, a further ten interrupts are also generated. tthh__ddeeffiinnee --nn ffoooo --ii 33 --aa iinnttrr --cc 00 11 --oo DDEELLAAYY 11002244 Causes the next interrupt for instance 3 of the ffoooo driver to be delayed by 1024 microseconds. NNOOTTEESS The policy option in the tthh__ddeeffiinnee --pp syntax determines how a set of logged accesses will be converted into the set of error definitions. Each logged access will be matched against the chosen policies to determine whether an error definition should be created based on the access. Any number of policy options can be combined to modify the generated error definitions. BByytteewwiissee PPoolliicciieess These select particular I/O transfer sizes. Specifying a byte policy will exclude other byte policies that have not been chosen. If none of the byte type policies is selected, all transfer sizes are treated equally. Otherwise, only those specified transfer sizes will be selected. oonneebbyyttee Create errdefs for one byte accesses (ddddii__ggeett88(())) ttwwoobbyyttee Create errdefs for two byte accesses (ddddii__ggeett1166(())) ffoouurrbbyyttee Create errdefs for four byte accesses (ddddii__ggeett3322(())) eeiigghhttbbyyttee Create errdefs for eight byte accesses (ddddii__ggeett6644(())) mmuullttiibbyyttee Create errdefs for repeated byte accesses (ddddii__rreepp__ggeett**(())) FFrreeqquueennccyy ooff AAcccceessss PPoolliicciieess The frequency of access to a location is determined according to the access type, location and transfer size (for example, a two-byte read access to address A is considered distinct from a four-byte read access to address A). The algorithm is to count the number of accesses (of a given type and size) to a given location, and find the locations that were most and least accessed (let _m_a_x_a and _m_i_n_a be the number of times these locations were accessed, and _m_e_a_n the total number of accesses divided by total number of locations that were accessed). Then a rare access is a location that was accessed less than _(_m_e_a_n _- _m_i_n_a_) _/ _3 _+ _m_i_n_a times. Similarly for the definition of common accesses: _m_a_x_a _- _(_m_a_x_a _- _m_e_a_n_) _/ _3 A location whose access patterns lies within these cutoffs is regarded as a location that is accessed with median frequency. rraarree Create errdefs for locations that are rarely accessed. ccoommmmoonn Create errdefs for locations that are commonly accessed. mmeeddiiaann Create errdefs for locations that are accessed a median frequency. PPoolliicciieess ffoorr MMiinniimmiizziinngg eerrrrddeeffss If a transaction is duplicated, either a single or multiple errdefs will be written to the test scripts, depending upon the following two policies: mmaaxxiimmaall Create multiple errdefs for locations that are repeatedly accessed. uunnbbiiaasseedd Create a single errdef for locations that are repeatedly accessed. ooppeerraattoorrss For each location, a default operator and operand is typically applied. For maximal test coverage, this default may be modified using the ooppeerraattoorrss policy so that a separate errdef is created for each of the possible corruption operators. SSEEEE AALLSSOO kkiillll(1), tthh__mmaannaaggee(1M), aallaarrmm(2), ddddii__cchheecckk__aacccc__hhaannddllee(9F), ddddii__cchheecckk__ddmmaa__hhaannddllee(9F) April 9, 2016 TH_DEFINE(1M)