[PATCH] Add script to run basic tests with all supported HWF combinations

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH] Add script to run basic tests with all supported HWF combinations

Jussi Kivilinna-2
* tests/basic_all_hwfeature_combinations.sh: New.
* tests/Makefile.am: Add basic_all_hwfeature_combinations.sh.
--

Signed-off-by: Jussi Kivilinna <[hidden email]>
---
 0 files changed

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 1744ea79..eee24faa 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -60,7 +60,7 @@ EXTRA_DIST = README rsa-16k.key cavs_tests.sh cavs_driver.pl \
      t-ed25519.inp stopwatch.h hashtest-256g.in \
      sha3-224.h sha3-256.h sha3-384.h sha3-512.h \
      blake2b.h blake2s.h \
-     basic-disable-all-hwf.in
+     basic-disable-all-hwf.in basic_all_hwfeature_combinations.sh
 
 LDADD = $(standard_ldadd) $(GPG_ERROR_LIBS)
 t_lock_LDADD = $(standard_ldadd) $(GPG_ERROR_MT_LIBS)
diff --git a/tests/basic_all_hwfeature_combinations.sh b/tests/basic_all_hwfeature_combinations.sh
new file mode 100755
index 00000000..2ad99196
--- /dev/null
+++ b/tests/basic_all_hwfeature_combinations.sh
@@ -0,0 +1,73 @@
+#!/bin/bash
+# Run basic tests with all HW feature combinations
+# Copyright 2017 Jussi Kivilinna <[hidden email]>
+#
+# This file is free software; as a special exception the author gives
+# unlimited permission to copy and/or distribute it, with or without
+# modifications, as long as this notice is preserved.
+#
+# This file is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY, to the extent permitted by law; without even the
+# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+
+# Use BINEXT to set executable extension
+#  For example for Windows executables: BINEXT=.exe
+if [ "x$BINEXT" != "x" ] && [ -e "tests/version$BINEXT" ]; then
+ binext="$BINEXT"
+else
+ binext=""
+fi
+
+# Use BINPRE to set executable prefix
+#  For example to run Windows executable with WINE: BINPRE="wine "
+if [ "x$BINPRE" != "x" ]; then
+ binpre="$BINPRE"
+else
+ binpre=""
+fi
+
+# Use NJOBS to define number of parallel tasks
+if [ "x$NJOBS" != "x" ]; then
+ njobs="$NJOBS"
+else
+ # default to cpu count
+ ncpus=$(nproc --all)
+ if [ "x@cpus" != "x" ]; then
+ njobs=$ncpus
+ else
+ # could not get cpu count, use 4 parallel tasks instead
+ njobs=4
+ fi
+fi
+
+get_supported_hwfeatures() {
+ $binpre "tests/version$binext" 2>&1 | \
+ grep "hwflist" | \
+ sed -e 's/hwflist://' -e 's/:/ /g' -e 's/\x0d/\x0a/g'
+}
+
+hwfs=($(get_supported_hwfeatures))
+echo "Total HW-feature combinations: $((1<<${#hwfs[@]}))"
+for ((cbits=0; cbits < (1<<${#hwfs[@]}); cbits++)); do
+  for ((mask=0; mask < ${#hwfs[@]}; mask++)); do
+    match=$(((1<<mask) & cbits))
+    if [ "x$match" != "x0" ]; then
+      echo -n "--disable-hwf ${hwfs[$mask]} "
+    fi
+  done
+  echo ""
+done | sort | (
+  nbasic=0
+  while read opts; do
+    nbasic=$((nbasic+1))
+    curr_jobs=($(jobs -p))
+    while [ "${#curr_jobs[@]}" -ge "8" ]; do
+      sleep 1
+      curr_jobs=($(jobs -p))
+    done
+    echo "[$nbasic/$((1<<${#hwfs[@]}))] Basic test with '$opts'"
+    nice nice $binpre "tests/basic$binext" $opts &
+  done
+  wait
+)


_______________________________________________
Gcrypt-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gcrypt-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Add script to run basic tests with all supported HWF combinations

Werner Koch
Hi!

I indeed think that this test is very useful.

On Sun, 18 Jun 2017 10:23, [hidden email] said:

> +++ b/tests/basic_all_hwfeature_combinations.sh
> @@ -0,0 +1,73 @@
> +#!/bin/bash

Unfortunately we can't depend on bash.  There are enough systems w/o
bash installed.

> +#  For example for Windows executables: BINEXT=.exe
> +if [ "x$BINEXT" != "x" ] && [ -e "tests/version$BINEXT" ]; then
> + binext="$BINEXT"
> +else
> + binext=""

Hmmm, running the script on Windows would require the installation of a
Unix shell.  That is too troublesome.  With the new gcry_get_config
fucntion we can get the features form a C program and the we would only
need to fork/exec - oh well, not on Windows :-(.

tests/random also forks but skips these checks on Windows.  Which it
should not.  To avoid cluttering all code with a platform dependent
implemtation of a spawing function I would suggest to wait until we have
put such a function into libgpgrt (aka libgpg-error).  We plan this
because we several such spawning implementations in the GnuPG stack and
libgpgrt is a common dependency of all.

If you want to push your change, I would suggest to do it as
maintainer-only tool _for now_ so that it is not run by "make check".


Salam-Shalom,

   Werner

--
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.

_______________________________________________
Gcrypt-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gcrypt-devel

attachment0 (233 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Add script to run basic tests with all supported HWF combinations

Jussi Kivilinna-2
On 23.06.2017 11:00, Werner Koch wrote:

> Hi!
>
> I indeed think that this test is very useful.
>
> On Sun, 18 Jun 2017 10:23, [hidden email] said:
>
>> +++ b/tests/basic_all_hwfeature_combinations.sh
>> @@ -0,0 +1,73 @@
>> +#!/bin/bash
>
> Unfortunately we can't depend on bash.  There are enough systems w/o
> bash installed.
>
>> +#  For example for Windows executables: BINEXT=.exe
>> +if [ "x$BINEXT" != "x" ] && [ -e "tests/version$BINEXT" ]; then
>> + binext="$BINEXT"
>> +else
>> + binext=""
>
> Hmmm, running the script on Windows would require the installation of a
> Unix shell.  That is too troublesome.  With the new gcry_get_config
> fucntion we can get the features form a C program and the we would only
> need to fork/exec - oh well, not on Windows :-(

Yes, that would be problematic if testing on Windows. One can instead
cross-compile win32 & win64 binaries on Linux and running tests
on command-line with Wine (+ binfmt plugin) which works quite
nicely. I've also found Qemu+binfmt to be quite useful when testing
code with different architectures, although there is some caveats
especially with missing unaligned memory access problems when
running program in Qemu on top of x86.

>
> tests/random also forks but skips these checks on Windows.  Which it
> should not.  To avoid cluttering all code with a platform dependent
> implemtation of a spawing function I would suggest to wait until we have
> put such a function into libgpgrt (aka libgpg-error).  We plan this
> because we several such spawning implementations in the GnuPG stack and
> libgpgrt is a common dependency of all.>
> If you want to push your change, I would suggest to do it as
> maintainer-only tool _for now_ so that it is not run by "make check".
>

Dedicated program would be nicer, but I'm not sure if that would
be worth of the extra effort. Problem with this check is that
it can take quite a long time and I would not recommend to
run it as part of "make check". For example, with current Intel
Skylake CPU, number of different HWF combinations is 4096 and
running all basic tests can take tens of minutes.

-Jussi

_______________________________________________
Gcrypt-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gcrypt-devel
Loading...