Without quotes, when the variable is empty, the shell will see three
tokens: [, -n, ]. After skipping ] as usual, it will see a single token,
so it will consider it not as command, but a string to be tested for "is
it empty", and since "-n" is not empty, the command will return true.
With quotes it see 4 tokens: [, -n, <empty string>, ] and interprets -n
as desired.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
Now the output of
grep '^[^ {}#]' tests/scripts/all-*.sh | grep -v '()'
is clean, with the only results being:
- shopt -s extglob which is needing for parsing the rest
- usage message
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
The only occurrences found by git grep were the definitions and the
comment saying these are kept for backwards compatibility.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
Don't interleave defining functions with running some code.
The only exception is calling shopt, which needs to come first as it
affects how the following function definitions are parsed.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
This change is not correct for 3.6 and will be reverted. It is only
applied as a temporary measure to avoid conflicts while cherry-pick
changes from development.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
In 3.6, compared with the state of development that this work was based on,
we're doing more testing with TLS12_PRF and TLS12_PSK_TO_MS disabled, thanks
to some configurations with MBEDTLS_PSA_CRYPTO_CONFIG disabled.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
MBEDTLS_ECP_NO_FALLBACK is only relevant when an ALT implementation of ECP
is used. We don't test ALT implementations so we don't test
MBEDTLS_ECP_NO_FALLBACK either.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
For each ignore list entry, link to a GitHub issue for its resolution,
except for a few configurations which there is a good reason to leave
uncovered.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
For each ignore list entry, link to a GitHub issue for its resolution,
except for ssl-opt Valgrind tests which we never intend to run on the CI.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Ignore certain test cases which either should not be generated or should be
executed. For each ignore list entry, link to a GitHub issue whose
definition of done includes removing the entry.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Some negative tests involving unsupported mechanisms are executed, because
they're testing what happens if the mechanism is unsupported. Refine the
ignore list for `test_suite_psa_crypto_generate_key.generated` and
`test_suite_psa_crypto_op_fail.generated` accordingly.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This clears more than half of the test cases that are not executed.
This also captures a few negative test cases that are executed.
Subsequent commits will refine the filtering.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Currently, many test cases are not executed. A follow-up pull request will
take care of that. In the meantime, continue allowing partial test coverage.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Move `collect_test_cases.py` (split from `check_test_cases.py`),
`check_test_cases.py`, and `outcome_analysis.py` (split from
`analyze_outcomes.py`) to the framework repository.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Place the code of outcome analysis (auxiliary functions, tasks, command line
entry point) into a separate module, which will be moved to the
version-independent framework repository so that it can be shared between
maintained branches. Keep the branch-specific list of driver components and
ignore lists in the per-repository script.
We keep the executable script at `tests/scripts/analyze_outcomes.py`. It's
simpler that way, because that path is hard-coded in CI scripts.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Always have tasks_list be a list, not potentially some fancier iterable.
Bypass mypy's somewhat legitimate complaint about REFERENCE and DRIVER in
task_class: they could potentially be instance attributes, but we rely on
them being class attributes. Python does normally guarantee their existence
as class attributes (unless a derived class explicitly deletes them), but
they could be overridden by an instance attribute; that's just something
we don't do, so the class attribute's value is legitimate. We can't
expect mypy to know that, so work around its complaint.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Use different names for task name, a task class and a task instance. The
interpreter doesn't care, but it's less confusing for both humans and type
checkers.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The ignore list for coverage only has two test cases out of ~10000 that are
currently reported as not executed. This is a drop in the sea and not
useful. Remove them so that the class can be used generically. A follow-up
will construct a comprehensive ignore list.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Move the test case collection code out of check_test_cases.py and into its
own module. This allows outcome analysis to depend only on the new module
and not on check_test_cases.py.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
