Quantcast

Re: [gnutls-devel] [PATCH] tests: skip tests that requires tools if tools are disabled

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] [PATCH] tests: skip tests that requires tools if tools are disabled

Nikos Mavrogiannopoulos-2
What I am afraid of such a patch is having a test being skipped when
it shouldn't. For which platform do you need this patch? Can it be
worked around by not installing the tools instead?

On Sat, Dec 31, 2016 at 4:14 AM, Alon Bar-Lev <[hidden email]> wrote:

> building with --disable-tools should not cause test failure.
>
> Signed-off-by: Alon Bar-Lev <[hidden email]>
> ---
>  tests/cert-tests/aki                          |  5 +++++
>  tests/cert-tests/certtool                     |  3 +++
>  tests/cert-tests/certtool-long-cn             |  4 ++++
>  tests/cert-tests/certtool-long-oids           |  4 ++++
>  tests/cert-tests/crl                          |  3 +++
>  tests/cert-tests/email                        |  3 +++
>  tests/cert-tests/openpgp-cert-parser          |  5 +++++
>  tests/cert-tests/openpgp-certs                |  9 +++++++++
>  tests/cert-tests/openpgp-selfsigs             |  4 ++++
>  tests/cert-tests/pathlen                      |  3 +++
>  tests/cert-tests/pem-decoding                 |  5 +++++
>  tests/cert-tests/pkcs12                       |  5 +++++
>  tests/cert-tests/pkcs12-utf8                  |  5 +++++
>  tests/cert-tests/pkcs7-broken-sigs            |  3 +++
>  tests/cert-tests/privkey-import               |  3 +++
>  tests/cert-tests/provable-dh                  |  4 ++++
>  tests/cert-tests/provable-dh-default          |  4 ++++
>  tests/cert-tests/provable-privkey             |  4 ++++
>  tests/cert-tests/provable-privkey-dsa2048     |  4 ++++
>  tests/cert-tests/provable-privkey-gen-default |  4 ++++
>  tests/cert-tests/provable-privkey-rsa2048     |  4 ++++
>  tests/cert-tests/sha2-dsa-test                |  4 ++++
>  tests/cert-tests/sha2-test                    |  4 ++++
>  tests/cert-tests/userid                       |  4 ++++
>  tests/fastopen.sh                             |  8 ++++++++
>  tests/key-tests/dsa                           |  5 +++++
>  tests/key-tests/ecdsa                         |  4 ++++
>  tests/key-tests/key-id                        |  4 ++++
>  tests/key-tests/pkcs8                         |  4 ++++
>  tests/key-tests/pkcs8-decode                  |  4 ++++
>  tests/key-tests/pkcs8-invalid                 |  4 ++++
>  tests/rfc2253-escape-test                     |  5 +++++
>  tests/suite/certtool-pkcs11.sh                | 16 ++++++++++++++++
>  tests/suite/chain.sh                          |  4 ++--
>  tests/suite/crl-test                          |  6 ++++++
>  tests/suite/eagain.sh                         |  4 ++++
>  tests/suite/invalid-cert.sh                   |  5 +++++
>  tests/suite/pkcs7-cat                         |  4 ++++
>  tests/suite/testdane.sh                       |  4 ++++
>  tests/suite/testpkcs11.sh                     | 16 ++++++++++++++++
>  tests/suite/testrandom.sh                     |  5 +++++
>  tests/suite/testsrn.sh                        |  8 ++++++++
>  42 files changed, 208 insertions(+), 2 deletions(-)
>
> diff --git a/tests/cert-tests/aki b/tests/cert-tests/aki
> index 3933161..1c72f0d 100755
> --- a/tests/cert-tests/aki
> +++ b/tests/cert-tests/aki
> @@ -26,6 +26,11 @@ srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  TMPFILE=aki-$$.tmp
>  DIFF="${DIFF:-diff -b -B}"
> +
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/certtool b/tests/cert-tests/certtool
> index 6a8bcc8..bc8ac7c 100755
> --- a/tests/cert-tests/certtool
> +++ b/tests/cert-tests/certtool
> @@ -23,6 +23,9 @@
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff -b -B}"
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/certtool-long-cn b/tests/cert-tests/certtool-long-cn
> index df430a2..41e7a97 100755
> --- a/tests/cert-tests/certtool-long-cn
> +++ b/tests/cert-tests/certtool-long-cn
> @@ -27,6 +27,10 @@ CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff -b -B}"
>  outfile="out.$$.pem"
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND} --error-exitcode=3"
>  fi
> diff --git a/tests/cert-tests/certtool-long-oids b/tests/cert-tests/certtool-long-oids
> index 31ea796..c2e9559 100755
> --- a/tests/cert-tests/certtool-long-oids
> +++ b/tests/cert-tests/certtool-long-oids
> @@ -28,6 +28,10 @@ DIFF="${DIFF:-diff -b -B}"
>  OUTFILE="out.$$.pem"
>  TMPFILE1="out1.$$.pem"
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND} --error-exitcode=3"
>  fi
> diff --git a/tests/cert-tests/crl b/tests/cert-tests/crl
> index 220c656..37e0ed7 100755
> --- a/tests/cert-tests/crl
> +++ b/tests/cert-tests/crl
> @@ -25,6 +25,9 @@ export TZ="UTC"
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff}"
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND} --error-exitcode=15"
>  fi
> diff --git a/tests/cert-tests/email b/tests/cert-tests/email
> index 8efe18e..ae737eb 100755
> --- a/tests/cert-tests/email
> +++ b/tests/cert-tests/email
> @@ -23,6 +23,9 @@
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF=$"{DIFF:-diff}"
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/openpgp-cert-parser b/tests/cert-tests/openpgp-cert-parser
> index c12e1be..3a4ab62 100755
> --- a/tests/cert-tests/openpgp-cert-parser
> +++ b/tests/cert-tests/openpgp-cert-parser
> @@ -25,6 +25,11 @@ set -e
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>
> +if ! test -x "${CERTTOOL}"; then
> +
> +       exit 77
> +fi
> +
>  unset RETCODE || true
>
>  fail() {
> diff --git a/tests/cert-tests/openpgp-certs b/tests/cert-tests/openpgp-certs
> index 3cee344..dfd38d9 100755
> --- a/tests/cert-tests/openpgp-certs
> +++ b/tests/cert-tests/openpgp-certs
> @@ -24,6 +24,15 @@ srcdir="${srcdir:-.}"
>  SERV="${SERV:-../../src/gnutls-serv} -q"
>  CLI="${CLI:-../../src/gnutls-cli}"
>  DEBUG=""
> +
> +if ! test -x "${SERV}"; then
> +       exit 77
> +fi
> +
> +if ! test -x "${CLI}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND} --error-exitcode=15"
>  fi
> diff --git a/tests/cert-tests/openpgp-selfsigs b/tests/cert-tests/openpgp-selfsigs
> index 7929e53..1e7c50d 100755
> --- a/tests/cert-tests/openpgp-selfsigs
> +++ b/tests/cert-tests/openpgp-selfsigs
> @@ -25,6 +25,10 @@ set -e
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  unset RETCODE || true
>
>  fail() {
> diff --git a/tests/cert-tests/pathlen b/tests/cert-tests/pathlen
> index 687ebc2..0bae79d 100755
> --- a/tests/cert-tests/pathlen
> +++ b/tests/cert-tests/pathlen
> @@ -25,6 +25,9 @@ set -e
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff -b -B}"
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/pem-decoding b/tests/cert-tests/pem-decoding
> index 2977888..b4c2a44 100755
> --- a/tests/cert-tests/pem-decoding
> +++ b/tests/cert-tests/pem-decoding
> @@ -25,6 +25,11 @@
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff -b -B}"
> +
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/pkcs12 b/tests/cert-tests/pkcs12
> index bd71ded..3b1d2c2 100755
> --- a/tests/cert-tests/pkcs12
> +++ b/tests/cert-tests/pkcs12
> @@ -24,6 +24,11 @@
>  srcdir="${srcdir:-.}"
>  top_builddir="${top_builddir:-../..}"
>  CERTTOOL="${CERTTOOL:-${top_builddir}/src/certtool${EXEEXT}}"
> +
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND} --error-exitcode=1"
>  fi
> diff --git a/tests/cert-tests/pkcs12-utf8 b/tests/cert-tests/pkcs12-utf8
> index 154d916..52a52f3 100755
> --- a/tests/cert-tests/pkcs12-utf8
> +++ b/tests/cert-tests/pkcs12-utf8
> @@ -27,6 +27,11 @@
>  srcdir="${srcdir:-.}"
>  top_builddir="${top_builddir:-../..}"
>  CERTTOOL="${CERTTOOL:-${top_builddir}/src/certtool${EXEEXT}}"
> +
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND} --error-exitcode=1"
>  fi
> diff --git a/tests/cert-tests/pkcs7-broken-sigs b/tests/cert-tests/pkcs7-broken-sigs
> index 913b2aa..272c009 100755
> --- a/tests/cert-tests/pkcs7-broken-sigs
> +++ b/tests/cert-tests/pkcs7-broken-sigs
> @@ -23,6 +23,9 @@
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff -b -B}"
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND} --error-exitcode=15"
>  fi
> diff --git a/tests/cert-tests/privkey-import b/tests/cert-tests/privkey-import
> index 084094c..41b2e45 100755
> --- a/tests/cert-tests/privkey-import
> +++ b/tests/cert-tests/privkey-import
> @@ -23,6 +23,9 @@
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff}"
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/provable-dh b/tests/cert-tests/provable-dh
> index 452a725..0676c0f 100755
> --- a/tests/cert-tests/provable-dh
> +++ b/tests/cert-tests/provable-dh
> @@ -25,6 +25,10 @@ CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff}"
>  OUTFILE=provable-dh$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/provable-dh-default b/tests/cert-tests/provable-dh-default
> index 517e226..43c2ed5 100755
> --- a/tests/cert-tests/provable-dh-default
> +++ b/tests/cert-tests/provable-dh-default
> @@ -25,6 +25,10 @@ CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff}"
>  OUTFILE=provable-dh$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/provable-privkey b/tests/cert-tests/provable-privkey
> index 4e11c61..3f94f9e 100755
> --- a/tests/cert-tests/provable-privkey
> +++ b/tests/cert-tests/provable-privkey
> @@ -25,6 +25,10 @@ CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff}"
>  OUTFILE=provable-privkey$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/provable-privkey-dsa2048 b/tests/cert-tests/provable-privkey-dsa2048
> index fa65f09..f805ab3 100755
> --- a/tests/cert-tests/provable-privkey-dsa2048
> +++ b/tests/cert-tests/provable-privkey-dsa2048
> @@ -25,6 +25,10 @@ CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff}"
>  OUTFILE=provable-privkey.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/provable-privkey-gen-default b/tests/cert-tests/provable-privkey-gen-default
> index d900c8c..3edf343 100755
> --- a/tests/cert-tests/provable-privkey-gen-default
> +++ b/tests/cert-tests/provable-privkey-gen-default
> @@ -25,6 +25,10 @@ CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff}"
>  OUTFILE=provable-privkey$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/provable-privkey-rsa2048 b/tests/cert-tests/provable-privkey-rsa2048
> index 3cd7aab..f846476 100755
> --- a/tests/cert-tests/provable-privkey-rsa2048
> +++ b/tests/cert-tests/provable-privkey-rsa2048
> @@ -25,6 +25,10 @@ CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff}"
>  OUTFILE=provable-privkey$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/cert-tests/sha2-dsa-test b/tests/cert-tests/sha2-dsa-test
> index 955d5ae..8632aef 100755
> --- a/tests/cert-tests/sha2-dsa-test
> +++ b/tests/cert-tests/sha2-dsa-test
> @@ -31,6 +31,10 @@ TMPFILE=sha2-dsa.$$.tmp
>  USERFILE=user-dsa.$$.tmp
>  VERIFYFILE=verify-dsa.$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  echo ca > $TEMPLFILE
>  echo "cn = SHA 256 CA" >> $TEMPLFILE
>
> diff --git a/tests/cert-tests/sha2-test b/tests/cert-tests/sha2-test
> index 9447518..eca12fd 100755
> --- a/tests/cert-tests/sha2-test
> +++ b/tests/cert-tests/sha2-test
> @@ -32,6 +32,10 @@ TMPFILE=sha2.$$.tmp
>  USERFILE=user.$$.tmp
>  VERIFYFILE=verify.$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  echo ca > $TEMPLFILE
>  echo "cn = SHA 512 CA" >> $TEMPLFILE
>
> diff --git a/tests/cert-tests/userid b/tests/cert-tests/userid
> index 912e0c9..e3cdef1 100755
> --- a/tests/cert-tests/userid
> +++ b/tests/cert-tests/userid
> @@ -23,6 +23,10 @@
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  "${CERTTOOL}" --certificate-info --infile "${srcdir}/data/userid.pem" >out 2>&1
>  RET=$?
>  if [ ${RET} !=  0 ]; then
> diff --git a/tests/fastopen.sh b/tests/fastopen.sh
> index 2d0bd86..3f636ad 100755
> --- a/tests/fastopen.sh
> +++ b/tests/fastopen.sh
> @@ -25,6 +25,14 @@ SERV="${SERV:-../src/gnutls-serv${EXEEXT}} -q"
>  CLI="${CLI:-../src/gnutls-cli${EXEEXT}}"
>  unset RETCODE
>
> +if ! test -x "${SERV}"; then
> +       exit 77
> +fi
> +
> +if ! test -x "${CLI}"; then
> +       exit 77
> +fi
> +
>  if test "${WINDIR}" != ""; then
>         exit 77
>  fi
> diff --git a/tests/key-tests/dsa b/tests/key-tests/dsa
> index 17a4472..5f437da 100755
> --- a/tests/key-tests/dsa
> +++ b/tests/key-tests/dsa
> @@ -27,6 +27,11 @@ DEBUG=""
>  unset RETCODE
>
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
> +
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/key-tests/ecdsa b/tests/key-tests/ecdsa
> index c6e1bd8..c593351 100755
> --- a/tests/key-tests/ecdsa
> +++ b/tests/key-tests/ecdsa
> @@ -34,6 +34,10 @@ TMPTEMPL=template.$$.tmp
>  TMPUSER=user.$$.tmp
>  VERIFYOUT=verify.$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  echo ca > $TMPTEMPL
>  echo "cn = ECDSA SHA 256 CA" >> $TMPTEMPL
>
> diff --git a/tests/key-tests/key-id b/tests/key-tests/key-id
> index f5408b0..4e60034 100755
> --- a/tests/key-tests/key-id
> +++ b/tests/key-tests/key-id
> @@ -27,6 +27,10 @@ CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  TMPFILE=key-id.$$.tmp
>  TEMPLFILE=tmpl.$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND} --error-exitcode=1"
>  fi
> diff --git a/tests/key-tests/pkcs8 b/tests/key-tests/pkcs8
> index 0bf51a2..7fee476 100755
> --- a/tests/key-tests/pkcs8
> +++ b/tests/key-tests/pkcs8
> @@ -24,6 +24,10 @@ GREP="${GREP:-grep}"
>
>  TMPFILE=tmp-key-ca.$$.p8
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND} --error-exitcode=1"
>  fi
> diff --git a/tests/key-tests/pkcs8-decode b/tests/key-tests/pkcs8-decode
> index b26ff4c..9159d20 100755
> --- a/tests/key-tests/pkcs8-decode
> +++ b/tests/key-tests/pkcs8-decode
> @@ -25,6 +25,10 @@ CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff -b -B}"
>  TMPFILE=pkcs8-decode.$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/key-tests/pkcs8-invalid b/tests/key-tests/pkcs8-invalid
> index bf2cb96..5251f1e 100755
> --- a/tests/key-tests/pkcs8-invalid
> +++ b/tests/key-tests/pkcs8-invalid
> @@ -25,6 +25,10 @@ CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
>  DIFF="${DIFF:-diff -b -B}"
>  TMPFILE=pkcs8-invalid.$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/rfc2253-escape-test b/tests/rfc2253-escape-test
> index a2c6472..71a4250 100755
> --- a/tests/rfc2253-escape-test
> +++ b/tests/rfc2253-escape-test
> @@ -23,6 +23,11 @@
>  set -e
>
>  CERTTOOL="${CERTTOOL:-../src/certtool${EXEEXT}}"
> +
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND} --error-exitcode=1"
>  fi
> diff --git a/tests/suite/certtool-pkcs11.sh b/tests/suite/certtool-pkcs11.sh
> index 7bc0784..20030a8 100755
> --- a/tests/suite/certtool-pkcs11.sh
> +++ b/tests/suite/certtool-pkcs11.sh
> @@ -26,6 +26,22 @@ SERV="${SERV:-../../src/gnutls-serv${EXEEXT}} -q"
>  CLI="${CLI:-../../src/gnutls-cli${EXEEXT}}"
>  RETCODE=0
>
> +if ! test -x "${P11TOOL}"; then
> +       exit 77
> +fi
> +
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
> +if ! test -x "${SERV}"; then
> +       exit 77
> +fi
> +
> +if ! test -x "${CLI}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute valgrind --leak-check=full"
>  fi
> diff --git a/tests/suite/chain.sh b/tests/suite/chain.sh
> index 10868f9..0ec5e2d 100755
> --- a/tests/suite/chain.sh
> +++ b/tests/suite/chain.sh
> @@ -22,8 +22,8 @@
>
>  CERTTOOL="${CERTTOOL:-../../../src/certtool${EXEEXT}}"
>
> -if ! test -z "${VALGRIND}"; then
> -       VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND} --error-exitcode=15"
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
>  fi
>
>  SUCCESS=" 1 4 7 12 15 16 17 18 24 26 27 30 33 56 57 62 63 "
> diff --git a/tests/suite/crl-test b/tests/suite/crl-test
> index a2b544c..85c40e5 100755
> --- a/tests/suite/crl-test
> +++ b/tests/suite/crl-test
> @@ -23,9 +23,15 @@
>  srcdir="${srcdir:-.}"
>  DIFF="${DIFF:-diff}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
> +
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> +
>  TMPFILE=long.$$.pem.tmp
>
>  rm -f $TMPFILE
> diff --git a/tests/suite/eagain.sh b/tests/suite/eagain.sh
> index 1663e7e..bcbaa64 100755
> --- a/tests/suite/eagain.sh
> +++ b/tests/suite/eagain.sh
> @@ -23,6 +23,10 @@
>  srcdir="${srcdir:-.}"
>  SERV="${SERV:-../../src/gnutls-serv${EXEEXT}} -q"
>
> +if ! test -x "${SERV}"; then
> +       exit 77
> +fi
> +
>  . "${srcdir}/../scripts/common.sh"
>
>  eval "${GETPORT}"
> diff --git a/tests/suite/invalid-cert.sh b/tests/suite/invalid-cert.sh
> index 00bf1e4..a13bf88 100755
> --- a/tests/suite/invalid-cert.sh
> +++ b/tests/suite/invalid-cert.sh
> @@ -22,6 +22,11 @@
>
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
> +
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/suite/pkcs7-cat b/tests/suite/pkcs7-cat
> index a034b0c..1ddfd6b 100755
> --- a/tests/suite/pkcs7-cat
> +++ b/tests/suite/pkcs7-cat
> @@ -28,6 +28,10 @@ if ! test -z "${VALGRIND}"; then
>  fi
>  OUTFILE=out-pkcs7.$$.tmp
>
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  . ${srcdir}/../scripts/common.sh
>
>  check_for_datefudge
> diff --git a/tests/suite/testdane.sh b/tests/suite/testdane.sh
> index 1dd2d90..5bef05d 100755
> --- a/tests/suite/testdane.sh
> +++ b/tests/suite/testdane.sh
> @@ -22,6 +22,10 @@ srcdir="${srcdir:-.}"
>  DANETOOL="${DANETOOL:-../../src/danetool${EXEEXT}}"
>  unset RETCODE
>
> +if ! test -x "${DANETOOL}"; then
> +       exit 77
> +fi
> +
>  # Unfortunately it is extremely fragile and fails 99% of the
>  # time.
>  if test "${WINDIR}" != ""; then
> diff --git a/tests/suite/testpkcs11.sh b/tests/suite/testpkcs11.sh
> index f416957..0e0460a 100755
> --- a/tests/suite/testpkcs11.sh
> +++ b/tests/suite/testpkcs11.sh
> @@ -26,6 +26,22 @@ SERV="${SERV:-../../src/gnutls-serv${EXEEXT}} -q"
>  CLI="${CLI:-../../src/gnutls-cli${EXEEXT}}"
>  RETCODE=0
>
> +if ! test -x "${P11TOOL}"; then
> +       exit 77
> +fi
> +
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
> +if ! test -x "${SERV}"; then
> +       exit 77
> +fi
> +
> +if ! test -x "${CLI}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute valgrind --leak-check=full"
>  fi
> diff --git a/tests/suite/testrandom.sh b/tests/suite/testrandom.sh
> index 75e4a40..7e68a71 100755
> --- a/tests/suite/testrandom.sh
> +++ b/tests/suite/testrandom.sh
> @@ -22,6 +22,11 @@
>
>  srcdir="${srcdir:-.}"
>  CERTTOOL="${CERTTOOL:-../../src/certtool${EXEEXT}}"
> +
> +if ! test -x "${CERTTOOL}"; then
> +       exit 77
> +fi
> +
>  if ! test -z "${VALGRIND}"; then
>         VALGRIND="${LIBTOOL:-libtool} --mode=execute ${VALGRIND}"
>  fi
> diff --git a/tests/suite/testsrn.sh b/tests/suite/testsrn.sh
> index 446a988..a2b10f4 100755
> --- a/tests/suite/testsrn.sh
> +++ b/tests/suite/testsrn.sh
> @@ -25,6 +25,14 @@ SERV="${SERV:-../../src/gnutls-serv${EXEEXT}} -q"
>  CLI="${CLI:-../../src/gnutls-cli${EXEEXT}}"
>  unset RETCODE
>
> +if ! test -x "${SERV}"; then
> +       exit 77
> +fi
> +
> +if ! test -x "${CLI}"; then
> +       exit 77
> +fi
> +
>  if test "${WINDIR}" != ""; then
>         exit 77
>  fi
> --
> 2.10.2
>

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] [PATCH] tests: skip tests that requires tools if tools are disabled

Alon Bar-Lev-3
On 3 January 2017 at 09:22, Nikos Mavrogiannopoulos
<[hidden email]> wrote:
>
> What I am afraid of such a patch is having a test being skipped when
> it shouldn't. For which platform do you need this patch? Can it be
> worked around by not installing the tools instead?

Hi,

I considered several alternatives...

1. Force --enable-tools if --enable-tests in autoconf

We skip important tests that can be done for minimal installation.

2. Always build tools but modify automake to conditionally install
them based on --enable-tools

We have this huge dependency graph of autogen, guile and libxml with
their own dependencies.

3. Skip tests that requires tools

Test whatever we can without introducing more dependencies for minimal
installation.

4. Clone at least certtool which is common to all tests to
certtool-test that has no extra dependencies with simpler usage.

I chose #3 as it seems to balance between having minimal build with
minimal tests and full build and test suites.
However, #4 is the best but more than I can contribute right now.

I do not see that tests that are being skipped when explicit disabling
an option suggests that they should not be skipped... :)

Recently, in Gentoo we removed the --enable-tools by default, but
people expects that tests will succeed while these currently fails.
Maintainers will continue to run tests with tools enabled.

Regards,
Alon

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] [PATCH] tests: skip tests that requires tools if tools are disabled

Nikos Mavrogiannopoulos-2
On Tue, Jan 3, 2017 at 8:51 AM, Alon Bar-Lev <[hidden email]> wrote:

> I considered several alternatives...
>
> 1. Force --enable-tools if --enable-tests in autoconf
>
> We skip important tests that can be done for minimal installation.
>
> 2. Always build tools but modify automake to conditionally install
> them based on --enable-tools
>
> We have this huge dependency graph of autogen, guile and libxml with
> their own dependencies.
>
> 3. Skip tests that requires tools
>
> Test whatever we can without introducing more dependencies for minimal
> installation.
>
> 4. Clone at least certtool which is common to all tests to
> certtool-test that has no extra dependencies with simpler usage.
>
> I chose #3 as it seems to balance between having minimal build with
> minimal tests and full build and test suites.
> However, #4 is the best but more than I can contribute right now.

Ok, let's test it. Would you like to submit it as a merge request in
gitlab? That will make it run through the CI and we can verify that no
test cases are skipped unintentionally.

regards,
Nikos

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] [PATCH] tests: skip tests that requires tools if tools are disabled

Alon Bar-Lev-3
On 3 January 2017 at 10:51, Nikos Mavrogiannopoulos
<[hidden email]> wrote:

>
> On Tue, Jan 3, 2017 at 8:51 AM, Alon Bar-Lev <[hidden email]> wrote:
> > I considered several alternatives...
> >
> > 1. Force --enable-tools if --enable-tests in autoconf
> >
> > We skip important tests that can be done for minimal installation.
> >
> > 2. Always build tools but modify automake to conditionally install
> > them based on --enable-tools
> >
> > We have this huge dependency graph of autogen, guile and libxml with
> > their own dependencies.
> >
> > 3. Skip tests that requires tools
> >
> > Test whatever we can without introducing more dependencies for minimal
> > installation.
> >
> > 4. Clone at least certtool which is common to all tests to
> > certtool-test that has no extra dependencies with simpler usage.
> >
> > I chose #3 as it seems to balance between having minimal build with
> > minimal tests and full build and test suites.
> > However, #4 is the best but more than I can contribute right now.
>
> Ok, let's test it. Would you like to submit it as a merge request in
> gitlab? That will make it run through the CI and we can verify that no
> test cases are skipped unintentionally.
>
> regards,
> Nikos

Done[1]

[1] https://gitlab.com/gnutls/gnutls/merge_requests/216

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] [PATCH] tests: skip tests that requires tools if tools are disabled

Alon Bar-Lev-3
Hi Nikos,

Will these patches be able to make it to 3.5 branch?
Will it help if I rebase?

Thanks,
Alon

https://gitlab.com/gnutls/gnutls/commit/276a6ee44d80d4d3b144a78794020c177be8f0ea
https://gitlab.com/gnutls/gnutls/commit/9f75df64d3a53e1cbad3eabf266acec7b426869a
https://gitlab.com/gnutls/gnutls/commit/9a38264f2065ae97ef3307f47c67bbee2313cb1c
https://gitlab.com/gnutls/gnutls/commit/98e49e9d0727d93a58b336dac0a78ce74558e8c8
https://gitlab.com/gnutls/gnutls/commit/41da7be67e0e0f30ea274a6f2d5af700be064b61
https://gitlab.com/gnutls/gnutls/commit/22da6ba5d5d719b125f28718861ad136fa7207af


On 3 January 2017 at 17:41, Alon Bar-Lev <[hidden email]> wrote:

>
> On 3 January 2017 at 10:51, Nikos Mavrogiannopoulos
> <[hidden email]> wrote:
> >
> > On Tue, Jan 3, 2017 at 8:51 AM, Alon Bar-Lev <[hidden email]> wrote:
> > > I considered several alternatives...
> > >
> > > 1. Force --enable-tools if --enable-tests in autoconf
> > >
> > > We skip important tests that can be done for minimal installation.
> > >
> > > 2. Always build tools but modify automake to conditionally install
> > > them based on --enable-tools
> > >
> > > We have this huge dependency graph of autogen, guile and libxml with
> > > their own dependencies.
> > >
> > > 3. Skip tests that requires tools
> > >
> > > Test whatever we can without introducing more dependencies for minimal
> > > installation.
> > >
> > > 4. Clone at least certtool which is common to all tests to
> > > certtool-test that has no extra dependencies with simpler usage.
> > >
> > > I chose #3 as it seems to balance between having minimal build with
> > > minimal tests and full build and test suites.
> > > However, #4 is the best but more than I can contribute right now.
> >
> > Ok, let's test it. Would you like to submit it as a merge request in
> > gitlab? That will make it run through the CI and we can verify that no
> > test cases are skipped unintentionally.
> >
> > regards,
> > Nikos
>
> Done[1]
>
> [1] https://gitlab.com/gnutls/gnutls/merge_requests/216

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Loading...