Command-line flags¶
Basic usage¶
sampletester TEST.yaml [TEST.yaml ...] [MANIFEST.manifest.yaml ...] [--envs=REGEX] [--suites=REGEX] [--cases=REGEX] [--fail-fast]
where:
- there can be any number of
TEST.yaml
testplan files - there can be any number of
MANIFEST.manifest.yaml
manifest files --envs
,--suites
, and--cases
are Python-style regular expressions (beware shell-escapes!) to select which environments, suites, and cases to run, based on their names. All the environemnts, suites, or cases will be selected to run by default if the corresponding flag is not set. Note that if an environment is not selected, its suites are not selected regardless of--suites
; if a suite is not selected, its testcases are not selected regardless of--cases
.--fail-fast
makes execution stop as soon as a failing test case is encountered, without executing any remaining test cases.
Controlling the output¶
In all cases, sample-tester
exits with a non-zero code if there were any errors in the flags, test config, or test execution.
In addition, by default sampletester
prints the status of test cases to stdout. This output is controlled by the following flags:
--verbosity
(-v
): controls how much output to show for passing tests. The default is a “summary” view, but “quiet” (no output) and “detailed” (full case output) options are available.--suppress_failures
(-f
): Overrides the default behavior of showing output for failing test cases, regardless of the--verbosity
setting--xunit=FILE
outputs a test summary in xUnit format toFILE
(use-
for stdout).
Advanced usage¶
The tester uses a “convention” to match sample names in the testplan
to actual, specific files on disk for given languages and
environments. Each convention may choose to take some set-up
arguments. You can specify an alternate convention and/or convention
arguments via the flag --convention=CONVENTION:ARG,ARGS
. The
default convention is tag:sample
, which uses the
sample
key in the manifest files. To use, say, the target
key in the manifest, simply pass --convention=tag:target
.
If you want to define an additional convention, refer to the
documention in the repo on how to do so. If you do have such an
additional convention defined, you may use the --convention
flag
to select it and give it any desired arguments, as above.