- the script now only terminates in case of hard faults
- each task is assigned a log
- this log tracks messages, warning and errors
- when task completes, errors and warnings are listed and
messages are appended to the main log
- on exit the main log is printed and the proper return value
is returned
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
Restore guards from the previous release, instead of the new, more
permissive guards.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
By default, we disable ASN1_[PARSE/WRITE]_C in common_tfm_config.
In fact, this is what happens for accelerated p256m driver, which
means all asn1[parse/write] tests are skipped in driver_accel test.
However, those two macros are automatically enabled for built-in
ECDSA via PSA, which means all asn1[parse/write] tests are passed
in tfm_config test.
This commit simply ignores the whole asn1[parse/write] test suite
when analyzing between driver and reference.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
{} are valid characters in test description, but they're not escaped
properly in python string format(). To resolve the bug of KeyError
when it tries to log test description which contains {}, we replace
{XXX} format with {{XXX}} in order to escape {} in python string
format() properly.
In addition, the calls to Results.log() are also handled to avoid
similar potential problems.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
* Don't break string literals in the allow list definition
* Comment each test that belongs to the allow list is there.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
* Turn the warnings produced when finding non-executed tests that
are not in the allow list into errors.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
Introduce the --require-full-coverage in analyze_outcomes.py so that
when analyze_outcomes.py --require-full-coverage is called, those
tests that are not executed and are not in the allowed list issue an
error instead of a warning.
Note that it is useful to run analyze_outcomes.py on incomplete test
results, so this error mode needs to remain optional in the long
term.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
The allow list explicits which test cases are allowed to not be
executed when testing. This may be, for example, because a feature
is yet to be developed but the test for that feature is already in
our code base.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
This includes also:
- auto enabling ECP_LIGHT when MBEDTLS_PK_PARSE_EC_COMPRESSED is
defined
- replacing ECP_LIGHT guards with PK_PARSE_EC_COMPRESSED in pkparse
- disabling PK_PARSE_EC_COMPRESSED in tests with accelarated EC curves
(it get disabled also in the reference components because we want
to achieve test parity)
- remove skipped checks in analyze_outcomes.py
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
The exception in analyze_outcomes.py follows previous commit in which
a test in test_suite_pkparse was set with the ECP_C guard for a different
parsing of the private key between the legacy and PSA implementations.
The wrong guard in test_suite_ecp.function instead was erroneously
added in a past commit and it was setting a non-existing symbol of
mbedTLS so those tests were basically never executed.
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
Only some test cases are skipped for which ECP_C is mandatory,
but the other ones are included.
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
Actually this adds both the accelerated test as well as the
reference. Both of them are used to evaluate the driver's
coverage with analyze_outcomes.py script.
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
