Run the function on a few different curves with shorter, just-right and
larger buffer sizes. Perform some basic sanity checks on the output (which
is random).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Do not assume that tests with !PSA_WANT_ALG_TLS12_PRF will not
be executed in AnalyzeCoverage task.
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
With the `depend.py` using the crypto config the `PBKDF2_HMAC` can be
enabled so thest cases can be run.
The equivalence (synonym) between `PSA_WANT_ALG_RSA_PSS_ANY_SALT` and
`PSA_WANT_ALG_RSA_PSS` is now done properly, the test can be run.
Dependencies for `MBEDTLS_CIPHER_PADDING_PKCS7' has been updated and
now it can be actually disabled.
Signed-off-by: Gabor Mezei <gabor.mezei@arm.com>
TLS only supports actual restartable ECDH with the legacy code that's going
away, not with the MBEDTLS_USE_PSA_CRYPTO code that's becoming the only
variant. This leaves a few test cases that validate restartable ECDH in TLS
as desirable, but not currently able to pass.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
For each ignore list entry, link to a GitHub issue for its resolution,
except for a few configurations which there is a good reason to leave
uncovered.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
For each ignore list entry, link to a GitHub issue for its resolution,
except for ssl-opt Valgrind tests which we never intend to run on the CI.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Ignore certain test cases which either should not be generated or should be
executed. For each ignore list entry, link to a GitHub issue whose
definition of done includes removing the entry.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Some negative tests involving unsupported mechanisms are executed, because
they're testing what happens if the mechanism is unsupported. Refine the
ignore list for `test_suite_psa_crypto_generate_key.generated` and
`test_suite_psa_crypto_op_fail.generated` accordingly.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This clears more than half of the test cases that are not executed.
This also captures a few negative test cases that are executed.
Subsequent commits will refine the filtering.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Currently, many test cases are not executed. A follow-up pull request will
take care of that. In the meantime, continue allowing partial test coverage.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Place the code of outcome analysis (auxiliary functions, tasks, command line
entry point) into a separate module, which will be moved to the
version-independent framework repository so that it can be shared between
maintained branches. Keep the branch-specific list of driver components and
ignore lists in the per-repository script.
We keep the executable script at `tests/scripts/analyze_outcomes.py`. It's
simpler that way, because that path is hard-coded in CI scripts.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Always have tasks_list be a list, not potentially some fancier iterable.
Bypass mypy's somewhat legitimate complaint about REFERENCE and DRIVER in
task_class: they could potentially be instance attributes, but we rely on
them being class attributes. Python does normally guarantee their existence
as class attributes (unless a derived class explicitly deletes them), but
they could be overridden by an instance attribute; that's just something
we don't do, so the class attribute's value is legitimate. We can't
expect mypy to know that, so work around its complaint.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Use different names for task name, a task class and a task instance. The
interpreter doesn't care, but it's less confusing for both humans and type
checkers.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The ignore list for coverage only has two test cases out of ~10000 that are
currently reported as not executed. This is a drop in the sea and not
useful. Remove them so that the class can be used generically. A follow-up
will construct a comprehensive ignore list.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Move the test case collection code out of check_test_cases.py and into its
own module. This allows outcome analysis to depend only on the new module
and not on check_test_cases.py.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates the data for driver-vs-reference analysis and gets rid
of the transitional code that was using the old form of the data.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates the code for driver-vs-reference analysis. To
facilitate review, this commit preserves the layout of the data that
parametrizes each task. The next commit will migrate the data.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on replacing the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates test coverage analysis.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Start replacing the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit introduces some transition code.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Rename the existing component_test_tfm_config which tests a modified version
of config-tfm.h for the sake of driver-vs-reference comparison.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The header line said “driver vs reference” whereas the error line said
“reference -> driver”. Updated the header line to be consistent with the
error line.
Signed-off-by: Sam Berry <sam.berry@arm.com>
The driver-vs-reference checks compare test results in different
configurations. Ignore the test results that report differences in
configurations that were the point of the comparison. Do compare other
configuration reports: this will let us know if the configurations diverge
in an unexpected way.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
