Place the code of outcome analysis (auxiliary functions, tasks, command line
entry point) into a separate module, which will be moved to the
version-independent framework repository so that it can be shared between
maintained branches. Keep the branch-specific list of driver components and
ignore lists in the per-repository script.
We keep the executable script at `tests/scripts/analyze_outcomes.py`. It's
simpler that way, because that path is hard-coded in CI scripts.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Always have tasks_list be a list, not potentially some fancier iterable.
Bypass mypy's somewhat legitimate complaint about REFERENCE and DRIVER in
task_class: they could potentially be instance attributes, but we rely on
them being class attributes. Python does normally guarantee their existence
as class attributes (unless a derived class explicitly deletes them), but
they could be overridden by an instance attribute; that's just something
we don't do, so the class attribute's value is legitimate. We can't
expect mypy to know that, so work around its complaint.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Use different names for task name, a task class and a task instance. The
interpreter doesn't care, but it's less confusing for both humans and type
checkers.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The ignore list for coverage only has two test cases out of ~10000 that are
currently reported as not executed. This is a drop in the sea and not
useful. Remove them so that the class can be used generically. A follow-up
will construct a comprehensive ignore list.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Move the test case collection code out of check_test_cases.py and into its
own module. This allows outcome analysis to depend only on the new module
and not on check_test_cases.py.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates the data for driver-vs-reference analysis and gets rid
of the transitional code that was using the old form of the data.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates the code for driver-vs-reference analysis. To
facilitate review, this commit preserves the layout of the data that
parametrizes each task. The next commit will migrate the data.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on replacing the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates test coverage analysis.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Start replacing the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit introduces some transition code.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Rename the existing component_test_tfm_config which tests a modified version
of config-tfm.h for the sake of driver-vs-reference comparison.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The header line said “driver vs reference” whereas the error line said
“reference -> driver”. Updated the header line to be consistent with the
error line.
Signed-off-by: Sam Berry <sam.berry@arm.com>
The driver-vs-reference checks compare test results in different
configurations. Ignore the test results that report differences in
configurations that were the point of the comparison. Do compare other
configuration reports: this will let us know if the configurations diverge
in an unexpected way.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Custom-e RSA key generation is not yet supported in the test driver, and we
don't support fallback from the test driver to the built-in
implementation (even though we're testing with MBEDTLS_RSA_C). So for the
time being, only run psa_generate_key_ext test cases for RSA with a custom
public exponent when using the built-in implementation.
Add a test case to validate that psa_generate_key_ext test cases for RSA
with a custom public exponent returns NOT_SUPPORTED (rather than silently
doing the wrong thing) when not using built-in RSA (which is subtly
different from when having accelerated RSA: if both are enabled, which we
currently don't do in all.sh, then this should be supported and this is
validated by the test cases above).
This wart will be resolved when we add support for drivers with a
generate_key_ext entry point.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This commit also:
- rename the reference component as component_test_full_block_cipher_legacy_dispatch()
- add a common configuration function, named common_block_cipher_dispatch() that
is used from both accelerated and reference components
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
