Ignore certain test cases which either should not be generated or should be
executed. For each ignore list entry, link to a GitHub issue whose
definition of done includes removing the entry.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Some negative tests involving unsupported mechanisms are executed, because
they're testing what happens if the mechanism is unsupported. Refine the
ignore list for `test_suite_psa_crypto_generate_key.generated` and
`test_suite_psa_crypto_op_fail.generated` accordingly.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This clears more than half of the test cases that are not executed.
This also captures a few negative test cases that are executed.
Subsequent commits will refine the filtering.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Currently, many test cases are not executed. A follow-up pull request will
take care of that. In the meantime, continue allowing partial test coverage.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Move `collect_test_cases.py` (split from `check_test_cases.py`),
`check_test_cases.py`, and `outcome_analysis.py` (split from
`analyze_outcomes.py`) to the framework repository.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Place the code of outcome analysis (auxiliary functions, tasks, command line
entry point) into a separate module, which will be moved to the
version-independent framework repository so that it can be shared between
maintained branches. Keep the branch-specific list of driver components and
ignore lists in the per-repository script.
We keep the executable script at `tests/scripts/analyze_outcomes.py`. It's
simpler that way, because that path is hard-coded in CI scripts.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Always have tasks_list be a list, not potentially some fancier iterable.
Bypass mypy's somewhat legitimate complaint about REFERENCE and DRIVER in
task_class: they could potentially be instance attributes, but we rely on
them being class attributes. Python does normally guarantee their existence
as class attributes (unless a derived class explicitly deletes them), but
they could be overridden by an instance attribute; that's just something
we don't do, so the class attribute's value is legitimate. We can't
expect mypy to know that, so work around its complaint.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Use different names for task name, a task class and a task instance. The
interpreter doesn't care, but it's less confusing for both humans and type
checkers.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The ignore list for coverage only has two test cases out of ~10000 that are
currently reported as not executed. This is a drop in the sea and not
useful. Remove them so that the class can be used generically. A follow-up
will construct a comprehensive ignore list.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Move the test case collection code out of check_test_cases.py and into its
own module. This allows outcome analysis to depend only on the new module
and not on check_test_cases.py.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
To prepare for the move of check_test_cases.py to the version-independent
framework repository, make the file in mbedtls-3.6 identical to the file
in development.
In development, check_test_cases.py now looks for tests under tf-psa-crypto.
This is useless but harmless in mbedtls-3.6.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates the data for driver-vs-reference analysis and gets rid
of the transitional code that was using the old form of the data.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates the code for driver-vs-reference analysis. To
facilitate review, this commit preserves the layout of the data that
parametrizes each task. The next commit will migrate the data.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on replacing the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates test coverage analysis.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Start replacing the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit introduces some transition code.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Fix `tls13-compat.sh` changing based on exactly how
`generate_tls13_compat_tests.py` was run (e.g. from which directory). This
made `check-generated-files.sh` behave differently from `make`. The script
has no official variations of the content of its output file, so we don't
need to record the full command line.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
`tests/opt-testcases/tls13-compat.sh` is supposed to be automatically
generated by `tests/scripts/generate_tls13_compat_tests.py`. So far, the
output has been updated by running the script manually and committing the
output. Switch to using our framework for generated files.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
With no options, update the output file (former behavior with -a).
Pass -1 to generate a single test case.
Also have the intended output file location as the default.
This way, you can just run the script after updating it, without having to
know the details of the directory structure.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE is no longer required, except in test
cases that are specifically about it. This commit removes the requirement in
tls13-compat.sh (which does not have test cases that actually depend on the
feature).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Rename the existing component_test_tfm_config which tests a modified version
of config-tfm.h for the sake of driver-vs-reference comparison.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The *.sh files in opt-testcases cannot be executed directly: they can only
be sourced by ssl-opt.sh. So don't make them executable and don't give them
a shebang line.
Also make sure that the first paragraph of each file is a short description.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Add a test that does some basic validation of the pkg-config files.
Example run:
./tests/scripts/all.sh test_cmake_as_package
<snip>
******************************************************************
* test_cmake_as_package: build: cmake 'as-package' build
* Wed Sep 11 16:22:09 UTC 2024
******************************************************************
cmake .
make
Built against Mbed TLS 3.6.1
testing package config file: mbedtls ... passed
testing package config file: mbedx509 ... passed
testing package config file: mbedcrypto ... passed
make clean
Signed-off-by: Bill Roberts <bill.roberts@arm.com>
