1 Test suite
   2 ##########
   3 
   4 Sparse has a number of test cases in its validation directory. The test-suite
   5 script aims at making automated checking of these tests possible. It works by
   6 embedding tags in C comments in the test cases.
   7 
   8 Tag's syntax
   9 ============
  10 
  11 ``check-name:`` *name*
  12 
  13         Name of the test. This is the only mandatory tag.
  14 
  15 ``check-description:`` *description ...*
  16 
  17         A description of what the test checks.
  18 
  19 ``check-command:`` *command arg ...*
  20 
  21         There are different kinds of tests. Some can validate the sparse
  22         preprocessor, while others will use sparse, cgcc, or even other backends
  23         of the library. check-command allows you to give a custom command to
  24         run the test-case.
  25         The ``$file`` string is special. It will be expanded to the file name at
  26         run time.
  27         It defaults to ``sparse $file``.
  28 
  29 ``check-arch-ignore:`` *arch[|...]*
  30 
  31 ``check-arch-only:`` *arch[|...]*
  32 
  33         Ignore the test if the current architecture (as returned by ``uname -m``)
  34         matches or not one of the archs given in the pattern.
  35 
  36 ``check-assert:`` *condition*
  37 
  38         Ignore the test if the given condition is false when evaluated as a
  39         static assertion (``_Static_assert``).
  40 
  41 ``check-cpp-if:`` *condition*
  42 
  43         Ignore the test if the given condition is false when evaluated
  44         by sparse's pre-processor.
  45 
  46 ``check-exit-value:`` *value*
  47 
  48         The expected exit value of check-command. It defaults to 0.
  49 
  50 ``check-timeout:`` *timeout*
  51 
  52         The maximum expected duration of check-command, in seconds.
  53         It defaults to 1.
  54 
  55 ``check-output-start`` / ``check-output-end``
  56 
  57         The expected output (stdout and stderr) of check-command lies between
  58         those two tags. It defaults to no output.
  59 
  60 ``check-output-ignore`` / ``check-error-ignore``
  61 
  62         Don't check the expected output (stdout or stderr) of check-command
  63         (useful when this output is not comparable or if you're only interested
  64         in the exit value).  By default this check is done.
  65 
  66 ``check-known-to-fail``
  67 
  68         Mark the test as being known to fail.
  69 
  70 ``check-output-contains:`` *pattern*
  71 
  72         Check that the output (stdout) contains the given pattern.
  73         Several such tags can be given, in which case the output
  74         must contains all the patterns.
  75 
  76 ``check-output-excludes:`` *pattern*
  77 
  78         Similar than the above one, but with opposite logic.
  79         Check that the output (stdout) doesn't contain the given pattern.
  80         Several such tags can be given, in which case the output
  81         must contains none of the patterns.
  82 
  83 ``check-output-pattern(``\ *nbr*\ ``):`` *pattern*
  84 
  85 ``check-output-pattern(``\ *min*\ ``,``\ *max*\ ``):`` *pattern*
  86 
  87         Similar to the contains/excludes above, but with full control
  88         of the number of times the pattern should occur in the output.
  89         If *min* or *max* is ``-`` the corresponding check is ignored.
  90 
  91 Using test-suite
  92 ================
  93 
  94 The test-suite script is called through the check target of the Makefile. It
  95 will try to check every test case it finds (``find validation -name '*.c'``).
  96 It can be called to check a single test with::
  97 
  98         $ cd validation
  99         $ ./test-suite single preprocessor/preprocessor1.c
 100              TEST     Preprocessor #1 (preprocessor/preprocessor1.c)
 101         preprocessor/preprocessor1.c passed !
 102 
 103 
 104 Writing a test
 105 ==============
 106 
 107 The test-suite comes with a format command to make a test easier to write::
 108 
 109         test-suite format [-a] [-l] [-f] file [name [cmd]]
 110 
 111 `name:`  check-name value
 112         If no name is provided, it defaults to the file name.
 113 
 114 `cmd:`   check-command value
 115         If no cmd is provided, it defaults to ``sparse $file``.
 116 
 117 The output of the test-suite format command can be redirected into the
 118 test case to create a test-suite formatted file.::
 119 
 120         $ ./test-suite format bad-assignment.c Assignment >> bad-assignment.c
 121         $ cat !$
 122         cat bad-assignment.c
 123         /*
 124          * check-name: bad assignment
 125          *
 126          * check-command: sparse $file
 127          * check-exit-value: 1
 128          *
 129          * check-output-start
 130         bad-assignment.c:3:6: error: Expected ; at end of statement
 131         bad-assignment.c:3:6: error: got \
 132          * check-output-end
 133          */
 134 
 135 The same effect without the redirection can be achieved by using the ``-a``
 136 option.
 137 
 138 You can define the check-command you want to use for the test.::
 139 
 140         $ ./test-suite format -a validation/preprocessor2.c "Preprocessor #2" \
 141                         "sparse -E \$file"
 142         $ cat !$
 143         cat validation/preprocessor2.c
 144         /*
 145          * This one we happen to get right.
 146          *
 147          * It should result in a simple
 148          *
 149          *      a + b
 150          *
 151          * for a proper preprocessor.
 152          */
 153         #define TWO a, b
 154 
 155         #define UNARY(x) BINARY(x)
 156         #define BINARY(x, y) x + y
 157 
 158         UNARY(TWO)
 159         /*
 160          * check-name: Preprocessor #2
 161          *
 162          * check-command: sparse -E $file
 163          * check-exit-value: 0
 164          *
 165          * check-output-start
 166 
 167         a + b
 168          * check-output-end
 169          */