Place the code of outcome analysis (auxiliary functions, tasks, command line
entry point) into a separate module, which will be moved to the
version-independent framework repository so that it can be shared between
maintained branches. Keep the branch-specific list of driver components and
ignore lists in the per-repository script.
We keep the executable script at `tests/scripts/analyze_outcomes.py`. It's
simpler that way, because that path is hard-coded in CI scripts.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Always have tasks_list be a list, not potentially some fancier iterable.
Bypass mypy's somewhat legitimate complaint about REFERENCE and DRIVER in
task_class: they could potentially be instance attributes, but we rely on
them being class attributes. Python does normally guarantee their existence
as class attributes (unless a derived class explicitly deletes them), but
they could be overridden by an instance attribute; that's just something
we don't do, so the class attribute's value is legitimate. We can't
expect mypy to know that, so work around its complaint.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Use different names for task name, a task class and a task instance. The
interpreter doesn't care, but it's less confusing for both humans and type
checkers.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The ignore list for coverage only has two test cases out of ~10000 that are
currently reported as not executed. This is a drop in the sea and not
useful. Remove them so that the class can be used generically. A follow-up
will construct a comprehensive ignore list.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Move the test case collection code out of check_test_cases.py and into its
own module. This allows outcome analysis to depend only on the new module
and not on check_test_cases.py.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
To prepare for the move of check_test_cases.py to the version-independent
framework repository, make the file in mbedtls-3.6 identical to the file
in development.
In development, check_test_cases.py now looks for tests under tf-psa-crypto.
This is useless but harmless in mbedtls-3.6.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates the data for driver-vs-reference analysis and gets rid
of the transitional code that was using the old form of the data.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates the code for driver-vs-reference analysis. To
facilitate review, this commit preserves the layout of the data that
parametrizes each task. The next commit will migrate the data.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Work on replacing the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit migrates test coverage analysis.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Start replacing the stringly typed KNOWN_TASKS by classes for each category
of tasks, with a structure that matches the behavior.
This commit introduces some transition code.
No intended behavior change.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Now that dtls_server doesn't print "error" when it receives stray messages
while it's waiting for a second handshake, have the tests fail if "error" is
printed for some other reason.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
If MBEDTLS_ERR_SSL_UNEXPECTED_MESSAGE happens during the handshake, don't
show it as an "error". It might be an error, but it might also be a fact of
life if it happens during the second or more handshake: it can be a
duplicated packet or a close_notify alert from the previous connection,
which is hard to avoid and harmless.
Fixes#9652.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
When building with `configs/config-suite-b.h`, the SSL I/O buffer size is
1024 bytes. Experimentally, this isn't quite enough for the test certificate
that we use: the server aborts the handshake with
`MBEDTLS_ERR_SSL_BUFFER_TOO_SMALL` raised from
`mbedtls_ssl_write_certificate()`. State an ad hoc minimum output buffer
size to skip testing `ssl_server` in `config-suite-b`.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
dtls_client connects to "localhost", which is usually IPv6 on modern
systems. On our CI, $OPENSSL is OpenSSL 1.0.2g which doesn't support IPv6.
Pitching dtls_client against $OPENSSL works on the CI at the moment, but
only because the CI runs in Docker with default network settings which has
IPv6 disabled. This would stop working if we changed the CI's Docker setup,
and the test case is likely to fail on a developer machine. So switch the
test case to using $OPENSSL_NEXT (which is a version of OpenSSL that has
IPv6 support).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This is necessary when testing against OpenSSL 1.0.2g.
In the server, flush more often. Otherwise, when stdout is redirected to a
file, the server gets killed before it writes important information, such as
the logs that we expect in the test cases.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
GnuTLS 3.4.x doesn't allow repeated `-p PORT` arguments.
OpenSSL 1.0.2 has different logs. For TLS 1.2 test cases, use a line that
is present in logs from OpenSSL 1.0.2g, 3.3.0 and presumably all versions
in between.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test against both OpenSSL and GnuTLS.
Don't use a proxy. It's not particularly useful here, and would complicate
figuring out port numbers.
Clean up compile-time requirements dtls_server.c: any certificate-based key
exchange is ok, so don't insist on built-in RSA.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_fork_server with both TLS 1.2 and TLS 1.3.
Test against both OpenSSL and GnuTLS.
In the server, flush more often. Otherwise, when stdout is redirected to a
file, the server gets killed before it writes important information, such as
the logs that we expect in the test cases.
In the server, only write output for 10 seconds, not 100. That's enough time
to start concurrent clients if desired. 100 seconds causes ssl-opt to take a
very long time when the client actually listens to the whole input (which
`gnutls-cli` does, but not `openssl s_client`).
Clean up compile-time requirements in ssl_fork_server.c: any certificate-based
key exchange is ok, so don't insist on built-in RSA.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_pthread_server with both TLS 1.2 and TLS 1.3.
Test against both OpenSSL and GnuTLS.
In the server, flush more often. Otherwise, when stdout is redirected to a
file, the server gets killed before it writes important information, such as
the logs that we expect in the test cases.
Clean up compile-time requirements in ssl_pthread_server.c: any certificate-based
key exchange is ok, so don't insist on built-in RSA.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_server with both TLS 1.2 and TLS 1.3.
Test against both OpenSSL and GnuTLS.
Clean up compile-time requirements in ssl_server.c: any certificate-based
key exchange is ok, so don't insist on built-in RSA.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test against both OpenSSL and GnuTLS.
Don't use a proxy. It's not particularly useful here, and would complicate
figuring out port numbers.
Clean up compile-time requirements in dtls_client.c: any certificate-based
key exchange is ok, so don't insist on built-in RSA.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_client1 with both TLS 1.2 and TLS 1.3.
Test against both OpenSSL and GnuTLS.
Clean up compile-time requirements in ssl_client1.c: any certificate-based
key exchange is ok, so don't insist on built-in RSA.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
