libgcrypt1.9.0: Failure on linking test executables

[GnuPG - Libgcrypt - Dev mailing list]

Dear gcrypt developers,

I'm trying to build libgcrypt 1.9.0 on amazonlinux2, with its
dependency library (libgpg-error) built from source and installed in a
non-default prefixed location (and libgcrypt-config is on the $PATH
when configure).
The test executables such as tests/t-secmem and tests/t-mpi-bit fail
to link in my environment with the following error:

----
/bin/sh ../libtool  --tag=CC   --mode=link gcc
-I/opt/x86_64-redhat-linux/libgpg-error/1.41/include -g -O2
-fvisibility=hidden -fno-delete-null-pointer-checks -Wall -no-install
-o t-secmem t-secmem.o ../src/libgcrypt.la ../compat/libcompat.la
libtool: link: gcc
-I/opt/x86_64-redhat-linux/libgpg-error/1.41/include -g -O2
-fvisibility=hidden -fno-delete-null-pointer-checks -Wall -o t-secmem
t-secmem.o  ../src/.libs/libgcrypt.so ../compat/.libs/libcompat.a
-Wl,-rpath -Wl,/tmp/src/tmp/x86_64-redhat-linux/ports/libgcrypt/1.9.0/libgcrypt-1.9.0/src/.libs
-Wl,-rpath -Wl,/opt/x86_64-redhat-linux/libgcrypt/1.9.0/lib
../src/.libs/libgcrypt.so: undefined reference to `gpgrt_fprintf@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpg_err_code_from_syserror@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpgrt_get_syscall_clamp@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpgrt_lock_init@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpg_err_code_from_errno@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpgrt_lock_unlock@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpgrt_b64dec_start@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to `gpgrt_rewind@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpgrt_b64dec_finish@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpgrt_lock_lock@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpg_err_set_errno@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to `gpg_strsource@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to `gpgrt_fclose@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpgrt_b64dec_proc@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to `gpgrt_fopenmem@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to `gpg_strerror@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpgrt_fclose_snatch@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to `gpgrt_ferror@GPG_ERROR_1.0'
../src/.libs/libgcrypt.so: undefined reference to
`gpgrt_lock_destroy@GPG_ERROR_1.0'
collect2: error: ld returned 1 exit status
make[2]: *** [t-secmem] Error 1
----

Seeing the log, I think we need to specify the location of
libgpg-error for the linker to correctly link these executables.
Commenting out the following lines in tests/Makefile.am seems to fix
the problem (these lines overrides LDADD to drop -lgcrypt-error):

----
pkbench_LDADD = $(standard_ldadd)
prime_LDADD = $(standard_ldadd)
t_mpi_bit_LDADD = $(standard_ldadd)
t_secmem_LDADD = $(standard_ldadd)
testapi_LDADD = $(standard_ldadd)
----

I will provide the full configure/make logs if necessary.

_______________________________________________
Gcrypt-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gcrypt-devel

[NIIBE Yutaka]

Kasumi Fukuda via Gcrypt-devel <[hidden email]> writes:
> I'm trying to build libgcrypt 1.9.0 on amazonlinux2, with its
> dependency library (libgpg-error) built from source and installed in a
> non-default prefixed location (and libgcrypt-config is on the $PATH
> when configure).
> The test executables such as tests/t-secmem and tests/t-mpi-bit fail
> to link in my environment with the following error:

Thank you for the report.

What's the version of your GCC and linker (from binutils)?

I don't see the reason why your linker emits error when linking t-secmem
with libgcrypt.so.  The t-secmem.o object has no use of any libgpg-error
sympols.  It seems that your linker tries to resolve all symbols in
linked library, which is not needed at all.

I tried with my environment (Debian GNU/Linux): Clang-3.8, GCC 6, GCC 8,
GCC 9,... with ld.gold or ld.bfd.  I was unable to replicate the linkage
error.

> Seeing the log, I think we need to specify the location of
> libgpg-error for the linker to correctly link these executables.

FWIW, it is me who droped -lgpg-error for these programs.  It is
intentional change to minimize library dependency, assuming modern
toolchain.

However, if it results build error(s) on some systems, it should be
reverted.
--

_______________________________________________
Gcrypt-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gcrypt-devel

[GnuPG - Libgcrypt - Dev mailing list]

Thank you for your clarification, Niibe san

On Thu, Jan 21, 2021 at 2:23 PM NIIBE Yutaka <[hidden email]> wrote:
> What's the version of your GCC and linker (from binutils)?

They are from the official Amazon Linux 2 repository.

----
bash-4.2# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/7/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap
--enable-languages=c,c++,objc,obj-c++,fortran,ada,go,lto --prefix=/usr
--mandir=/usr/share/man --infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared
--enable-threads=posix --enable-checking=release --enable-multilib
--with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-gnu-unique-object
--enable-linker-build-id --with-gcc-major-version-only
--with-linker-hash-style=gnu --enable-plugin --enable-initfini-array
--with-isl --enable-libmpx --enable-libsanitizer
--enable-gnu-indirect-function --enable-libcilkrts --enable-libatomic
--enable-libquadmath --enable-libitm --with-tune=generic
--with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 7.3.1 20180712 (Red Hat 7.3.1-9) (GCC)

bash-4.2# ld -v
GNU ld version 2.29.1-30.amzn2

bash-4.2# ld.gold -v
GNU gold (version 2.29.1-30.amzn2) 1.14
----

> I don't see the reason why your linker emits error when linking t-secmem
> with libgcrypt.so.  The t-secmem.o object has no use of any libgpg-error
> sympols. It seems that your linker tries to resolve all symbols in
> linked library, which is not needed at all.

I think you are right. And in fact, it has no dependency on
libgpg-error symbols when I do `nm t-secmem.o` or `readelf -a
t-secmem`.

> I tried with my environment (Debian GNU/Linux): Clang-3.8, GCC 6, GCC 8,
> GCC 9,... with ld.gold or ld.bfd.  I was unable to replicate the linkage
> error.

Switching the linker to gold (by symlinking ld -> ld.gold) resolved
the issue in my case.
So something wrong with my ld.bfd?

--

_______________________________________________
Gcrypt-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gcrypt-devel

[GnuPG - Libgcrypt - Dev mailing list]

Let me share a repro step using docker:
----
#!/bin/bash
docker run -i amazon/aws-sam-cli-build-image-provided.al2 /bin/bash <<'EOF'

set -ex
cd /tmp
export MAKEFLAGS=-j$(nproc)

M=https://gnupg.org/ftp/gcrypt
curl $M/libgpg-error/libgpg-error-1.41.tar.bz2 | tar jx
curl $M/libgcrypt/libgcrypt-1.9.0.tar.bz2 | tar jx

cd /tmp/libgpg-error-1.41
./configure --prefix=/opt/libgpg-error --disable-doc
make && make install

export PATH=/opt/libgpg-error/bin:$PATH

cd /tmp/libgcrypt-1.9.0
./configure --prefix=/opt/libgcrypt --disable-doc
make && make install

EOF
----

On Thu, Jan 21, 2021 at 4:29 PM Kasumi Fukuda <[hidden email]> wrote:

>
> Thank you for your clarification, Niibe san
>
> On Thu, Jan 21, 2021 at 2:23 PM NIIBE Yutaka <[hidden email]> wrote:
> > What's the version of your GCC and linker (from binutils)?
>
> They are from the official Amazon Linux 2 repository.
>
> ----
> bash-4.2# gcc -v
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/7/lto-wrapper
> Target: x86_64-redhat-linux
> Configured with: ../configure --enable-bootstrap
> --enable-languages=c,c++,objc,obj-c++,fortran,ada,go,lto --prefix=/usr
> --mandir=/usr/share/man --infodir=/usr/share/info
> --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared
> --enable-threads=posix --enable-checking=release --enable-multilib
> --with-system-zlib --enable-__cxa_atexit
> --disable-libunwind-exceptions --enable-gnu-unique-object
> --enable-linker-build-id --with-gcc-major-version-only
> --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array
> --with-isl --enable-libmpx --enable-libsanitizer
> --enable-gnu-indirect-function --enable-libcilkrts --enable-libatomic
> --enable-libquadmath --enable-libitm --with-tune=generic
> --with-arch_32=x86-64 --build=x86_64-redhat-linux
> Thread model: posix
> gcc version 7.3.1 20180712 (Red Hat 7.3.1-9) (GCC)
>
> bash-4.2# ld -v
> GNU ld version 2.29.1-30.amzn2
>
> bash-4.2# ld.gold -v
> GNU gold (version 2.29.1-30.amzn2) 1.14
> ----
>
> > I don't see the reason why your linker emits error when linking t-secmem
> > with libgcrypt.so.  The t-secmem.o object has no use of any libgpg-error
> > sympols. It seems that your linker tries to resolve all symbols in
> > linked library, which is not needed at all.
>
> I think you are right. And in fact, it has no dependency on
> libgpg-error symbols when I do `nm t-secmem.o` or `readelf -a
> t-secmem`.
>
> > I tried with my environment (Debian GNU/Linux): Clang-3.8, GCC 6, GCC 8,
> > GCC 9,... with ld.gold or ld.bfd.  I was unable to replicate the linkage
> > error.
>
> Switching the linker to gold (by symlinking ld -> ld.gold) resolved
> the issue in my case.
> So something wrong with my ld.bfd?
>
> --

_______________________________________________
Gcrypt-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gcrypt-devel

[NIIBE Yutaka]

Kasumi Fukuda <[hidden email]> writes:
> Let me share a repro step using docker:

Thanks a lot.

I think that it's better practice to do "make check" before
installation.

And to do that, we need to specify LD_LIBRARY_PATH too (as well as
PATH), so that testing programs in the build procedure run correctly.

Like this:

--- libgcrypt-test-install-orig.sh 2021-01-22 11:06:34.329216354 +0900
+++ libgcrypt-test-install-fixed.sh 2021-01-22 12:29:42.725628836 +0900
@@ -11,12 +11,13 @@
 
 cd /tmp/libgpg-error-1.41
 ./configure --prefix=/opt/libgpg-error --disable-doc
-make && make install
+make && make check && make install
 
 export PATH=/opt/libgpg-error/bin:$PATH
+export LD_LIBRARY_PATH=/opt/libgpg-error/lib
 
 cd /tmp/libgcrypt-1.9.0
 ./configure --prefix=/opt/libgcrypt --disable-doc
-make && make install
+make && make check && make install
 
 EOF


And... this works, with no linking errors.

The problem is somehow complicated.  The linker checks runtime libraries
under LD_LIBRARY_PATH.


In your case, even if you don't need 'make check', please specify
LD_LIBRARY_PATH.
--

_______________________________________________
Gcrypt-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gcrypt-devel

[GnuPG - Libgcrypt - Dev mailing list]

On Fri, Jan 22, 2021 at 12:59 PM NIIBE Yutaka <[hidden email]> wrote:
>
> Kasumi Fukuda <[hidden email]> writes:
> > Let me share a repro step using docker:
>
> Thanks a lot.
>
> I think that it's better practice to do "make check" before
> installation.

Thank you for the advice. It ensures the linked executables are
actually executable.

> And to do that, we need to specify LD_LIBRARY_PATH too (as well as
> PATH), so that testing programs in the build procedure run correctly.

My understanding is that libgcypt and libgpg-error use libtool, so
RPATH (or RUNPATH) for the dependent libraries can be automatically
injected into the executable when linked.
We usually don't need to set LD_LIBRARY_PATH to run such an executable.

If I comment out the t_secmem_LDADD, the link command for t-secmem has
the necessary -rpath for libgpg-error as follows:
----
libtool: link: gcc -I/opt/libgpg-error/include -g -O2
-fvisibility=hidden -fno-delete-null-pointer-checks -Wall -o t-secmem
t-secmem.o -Wl,--disable-new-dtags  ../src/.libs/libgcrypt.so
../compat/.libs/libcompat.a -L/opt/libgpg-error/lib /
opt/libgpg-error/lib/libgpg-error.so -Wl,-rpath
-Wl,/tmp/libgcrypt-1.9.0/src/.libs -Wl,-rpath
-Wl,/opt/libgpg-error/lib -Wl,-rpath -Wl,/opt/libgcrypt/lib -Wl,-rpath
-Wl,/opt/libgpg-error/lib
----
instead of:
----
libtool: link: gcc -I/opt/libgpg-error/include -g -O2
-fvisibility=hidden -fno-delete-null-pointer-checks -Wall -o t-secmem
t-secmem.o  ../src/.libs/libgcrypt.so ../compat/.libs/libcompat.a
-Wl,-rpath -Wl,/tmp/libgcrypt-1.9.0/src/.libs -Wl,-rpath
-Wl,/opt/libgcrypt/lib
----
and `make check` works without LD_LIBRARY_PATH.

This can be virtually tested by:
  sed -i.bak 's/^\([a-z_]\+_LDADD = \)\$(standard_ldadd)$/\1$(LDADD)/'
tests/Makefile.in
before ./configure for libgcrypt.

I'm not sure I understand RPATH/RUNPATH things correctly.
But this way of building GnuPG suite has been working for older
versions and it would be inconvenient if LD_LIBRARY_PATH were required
to be set.

> Like this:
>
> --- libgcrypt-test-install-orig.sh      2021-01-22 11:06:34.329216354 +0900
> +++ libgcrypt-test-install-fixed.sh     2021-01-22 12:29:42.725628836 +0900
> @@ -11,12 +11,13 @@
>
>  cd /tmp/libgpg-error-1.41
>  ./configure --prefix=/opt/libgpg-error --disable-doc
> -make && make install
> +make && make check && make install
>
>  export PATH=/opt/libgpg-error/bin:$PATH
> +export LD_LIBRARY_PATH=/opt/libgpg-error/lib
>
>  cd /tmp/libgcrypt-1.9.0
>  ./configure --prefix=/opt/libgcrypt --disable-doc
> -make && make install
> +make && make check && make install
>
>  EOF
>
>
> And... this works, with no linking errors.
>
> The problem is somehow complicated.  The linker checks runtime libraries
> under LD_LIBRARY_PATH.
>
>
> In your case, even if you don't need 'make check', please specify
> LD_LIBRARY_PATH.
> --

_______________________________________________
Gcrypt-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gcrypt-devel

[NIIBE Yutaka]

Kasumi Fukuda wrote:
> My understanding is that libgcypt and libgpg-error use libtool, so
> RPATH (or RUNPATH) for the dependent libraries can be automatically
> injected into the executable when linked.
> We usually don't need to set LD_LIBRARY_PATH to run such an executable.

Yes and No.

For a program (like ones under tests) which is built with libtool's
-no-install option, RPATH/RUNPATH is embedded.  That's true.  This is
because such a program can be executed correctly with the particular
library currently being built (not with the one installed already on
system).

With embedded RPATH/RUNPATH, such a program can be executed with no
setting of LD_LIBRARY_PATH.  But, AFAIU, it is not for supporting no
setting of LD_LIBRARY_PATH.

Please note that LD_LIBRARY_PATH is usually set to run a program with
library under non-standard prefix.  We can't ignore this major use case.

> But this way of building GnuPG suite has been working for older
> versions and it would be inconvenient if LD_LIBRARY_PATH were required
> to be set.

People would say it's a regression, I'm afraid.  I think that this is a
corner case.


The change of mine for minimizing library dependency has its purpose to
keep up evolusion of toolchain.

Ideally, libtool itself should have been evolved, while toolchain has
been evolved gradually.
--

_______________________________________________
Gcrypt-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gcrypt-devel

[GnuPG - Libgcrypt - Dev mailing list]

On Fri, Jan 22, 2021 at 9:02 PM NIIBE Yutaka <[hidden email]> wrote:

>
> Kasumi Fukuda wrote:
> > My understanding is that libgcypt and libgpg-error use libtool, so
> > RPATH (or RUNPATH) for the dependent libraries can be automatically
> > injected into the executable when linked.
> > We usually don't need to set LD_LIBRARY_PATH to run such an executable.
>
> Yes and No.
>
> For a program (like ones under tests) which is built with libtool's
> -no-install option, RPATH/RUNPATH is embedded.  That's true.  This is
> because such a program can be executed correctly with the particular
> library currently being built (not with the one installed already on
> system).
>
> With embedded RPATH/RUNPATH, such a program can be executed with no
> setting of LD_LIBRARY_PATH.  But, AFAIU, it is not for supporting no
> setting of LD_LIBRARY_PATH.

I didn't know that.

> Please note that LD_LIBRARY_PATH is usually set to run a program with
> library under non-standard prefix.  We can't ignore this major use case.
>
> > But this way of building GnuPG suite has been working for older
> > versions and it would be inconvenient if LD_LIBRARY_PATH were required
> > to be set.
>
> People would say it's a regression, I'm afraid.  I think that this is a
> corner case.

I agree that this is just a corner case with quite an old linker, and
respect your decision not to support them.

Thank you again for your clarification.

_______________________________________________
Gcrypt-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gcrypt-devel