Alert if all tests are filtered out or skipped: that probably indicates a
test script that set up an unintended configuration or an overly strict
filter. You can pass `--min 0` to bypass this check. You can pass `--min`
with a larger value to require that many test cases to run.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Skipping DTLS 1.2 with old versions was already done, but now properly
test support only once and use the results.
Note that historically, this script's policy was that it's the user's
job to find the right value of -e (EXCLUDE) for their version for
OpenSSL & config. Now it's a weird mix of that and the script doing some
detection and skipping.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
When a test script has multiple suites, it is not
true to determine the suite name from the file name
of the script. We need the script to list the suite
name for every test cases.
Signed-off-by: Pengyu Lv <pengyu.lv@arm.com>
The -l option mentioned in previous commits for both ssl-opt.sh and
compat.sh scripts should only be a --list-test-cases option.
Remove -l option from the help list.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
walk_compat_sh and walk_opt_sh are basically the same now, so:
* Merge them into one function.
* Use the --list-test-cases option for both of them.
* Rename this merged function as collect_from_script which seems
more appropriate as since it isn't iterating the script but
calling it.
Signed-off-by: Tomás González <tomasagustin.gonzalezorlando@arm.com>
The official spelling of the trade mark changed from all-lowercase "mbed"
to normal proper noun capitalization "Mbed" a few years ago. We've been
using the new spelling in new text but still have the old spelling in a
lot of text. This commit updates most occurrences of "mbed TLS":
```
sed -i -e 's/mbed TLS/Mbed TLS/g' $(git ls-files ':!ChangeLog' ':!tests/data_files/**' ':!tests/suites/*.data' ':!programs/x509/*' ':!configs/tfm*')
```
Justification for the omissions:
* `ChangeLog`: historical text.
* `test/data_files/**`, `tests/suites/*.data`, `programs/x509/*`: many
occurrences are significant names in certificates and such. Changing
the spelling would invalidate many signatures and tests.
* `configs/tfm*`: this is an imported file. We'll follow the upstream
updates.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This commit includes:
- use subprocess.check_output to report error and capture return
value
- add comment as a reminder for option --list-test-case
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
uniform_title is used to print identical format of $TITLE between
--list-test-case and run_client. In such way, no matter how $TITLE
is developed, --list-test-case will in the same format of test case
description as stored in OUTCOME.CSV.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
If the environment variable MBEDTLS_TEST_OUTCOME_FILE is set,
the test outcome file records each test case in a single line
with the format of
PLATFORM;CONFIGURATION;compat;TEST CASE DESCRIPTION;RESULT;[CAUSE]
- CONFIGURATION comes from MBEDTLS_TEST_CONFIGURATION to record
configuration of each test case
- PLATFORM is either set by users or calculated from test
platform
- RESULT is one of PASS, FAIL or SKIP. If test case fails,
srv_out/cli_out follows as FAILURE CAUSE.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
Test case description is printed by different block of code. This
causes code maintenance harder since we need to maintain two parts
of code with same functionality. print_test_title is used to
control test case description in compat.sh
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
The option --list-test-case lists all potential test cases without
executing them. The test case description is identical with $TITLE
during test case execution.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
The mechanism of detecting unsupported ciphersuites
for OpenSSL client doesn't work on a modern OpenSSL.
At least, it fails on Travis CI which is installed
with OpenSSL 1.1.1f. So we need to skip ECDH cipher-
suites for O->m.
Signed-off-by: Pengyu Lv <pengyu.lv@arm.com>
This commit add support to detect if openssl used for testing
supports static ECDH key exchange. Skip the ciphersutes if
openssl doesn't support them.
Signed-off-by: Pengyu Lv <pengyu.lv@arm.com>
Since PSK cipher suites do not allow client certificate verification,
PSK test cases should be executed under VERIFY=NO. SUB_VERIFIES is
used to constrain verification option for PSK tests.
With aforementioned change, the latter check of
$VERIFY=YES && $TYPE!=PSK is redundant so it's removed.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
With the introduction of PSK_TESTS,
- Either `compat.sh -V NO` or `compat.sh -V YES` runs the PSK tests
- `compat.sh` or `compat.sh -V "NO YES"` runs PSK tests only once
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
There is no need to provide CA file in PSK. Thus VERIFY is
meaningless for PSK. This change omits the arguments passed to
the client and server for $VERIFY=YES.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
Don't use external programs for string manipulation that the shell can do.
This makes the script a little faster (~10% when testing PSK).
For this commit, I only looked at code run in the innermost loop.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Python has a high startup cost, so go back to invoking it only once per
server start, rather than once per client start. This is a measurable
performance improvement (running time ~*0.5 with PSK, less dramatic with
asymmetric crypto).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
We can detect PSK based on $TYPE. This allows more flexibility in how cipher
suites are spelled.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The argument passed to translate_ciphers.py is calculated from $1 in
run_client instead of passed as third argument.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
