[gnutls-devel] symbol and library versioning

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[gnutls-devel] symbol and library versioning

Nikos Mavrogiannopoulos-2
Hi,
 With mind in the upcoming 3.6.0 release I tried to codify/clarify the
library and symbol versioning rules that are considered best practice
today.

The output is at:
https://gitlab.com/gnutls/gnutls/blob/master/CONTRIBUTING.md#symbol-and-library-versioning

My take-away from it, is that the complexity of libtool rules is
unnecessary and in fact all that is being used from it is the major
soname version (thus the minor and patch versions set via libtool are
being used by no software).

The versioning that matters is via the linker script, which is then
used by packaging software like rpm (I guess possibly dpkg as well),
to detect proper dependencies.

Did I miss something in that? Are there are best practices I missed,
or any issues in the rules I set above?

regards,
Nikos

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] symbol and library versioning

Daniel P. Berrange-2
On Thu, Aug 10, 2017 at 10:31:28AM +0200, Nikos Mavrogiannopoulos wrote:

> Hi,
>  With mind in the upcoming 3.6.0 release I tried to codify/clarify the
> library and symbol versioning rules that are considered best practice
> today.
>
> The output is at:
> https://gitlab.com/gnutls/gnutls/blob/master/CONTRIBUTING.md#symbol-and-library-versioning
>
> My take-away from it, is that the complexity of libtool rules is
> unnecessary and in fact all that is being used from it is the major
> soname version (thus the minor and patch versions set via libtool are
> being used by no software).

Yes, libtool versioning is not really useful, especially for a library
that intends to maintain API/ABI compat long term / forever.

> The versioning that matters is via the linker script, which is then
> used by packaging software like rpm (I guess possibly dpkg as well),
> to detect proper dependencies.
>
> Did I miss something in that? Are there are best practices I missed,
> or any issues in the rules I set above?

I believe your backporting rule is a bad idea.

  "if symbol gnutls_xyz with version GNUTLS_3_6_3 is backported on
   gnutls 3.3.15, it should use version GNUTLS_3_3_15."

This creates ABI breakage when you upgrade gnutls in the distro. eg

An application which links to your gnutls 3.3.15 version and uses
symbol gnutls_xyz will get a embedded symbol reference of
gnutls_xyz@GNUTLS_3_3_15.   When you then update to gnutls 3.6.3 in
the distro, the linker will be unable to resolve gnutls_xyz@GNUTLS_3_3_15,
because the library will now be exporting gnutls_xyz@GNUTLS_3_6_3
instead.

There's a few other options for backporting, but they are all awful
in their own ways:

  https://www.berrange.com/posts/2011/01/13/versioning-in-the-libvirt-library/

so for libvirt we simply refuse to ever backport public APIs to older
versions. Backports are for bug fixes only, never features.

Regards,
Daniel
--
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] symbol and library versioning

Nikos Mavrogiannopoulos-2
On Thu, Aug 10, 2017 at 11:14 AM, Daniel P. Berrange
<[hidden email]> wrote:

> On Thu, Aug 10, 2017 at 10:31:28AM +0200, Nikos Mavrogiannopoulos wrote:
>> Hi,
>>  With mind in the upcoming 3.6.0 release I tried to codify/clarify the
>> library and symbol versioning rules that are considered best practice
>> today.
>>
>> The output is at:
>> https://gitlab.com/gnutls/gnutls/blob/master/CONTRIBUTING.md#symbol-and-library-versioning
>>
>> My take-away from it, is that the complexity of libtool rules is
>> unnecessary and in fact all that is being used from it is the major
>> soname version (thus the minor and patch versions set via libtool are
>> being used by no software).
>
> Yes, libtool versioning is not really useful, especially for a library
> that intends to maintain API/ABI compat long term / forever.

I wonder whether there is a reason not to keep the libtool numbers
fixed, and follow its versioning requirements.

>> The versioning that matters is via the linker script, which is then
>> used by packaging software like rpm (I guess possibly dpkg as well),
>> to detect proper dependencies.
>>
>> Did I miss something in that? Are there are best practices I missed,
>> or any issues in the rules I set above?
>
> I believe your backporting rule is a bad idea.
>
>   "if symbol gnutls_xyz with version GNUTLS_3_6_3 is backported on
>    gnutls 3.3.15, it should use version GNUTLS_3_3_15."
>
> This creates ABI breakage when you upgrade gnutls in the distro. eg
>
> An application which links to your gnutls 3.3.15 version and uses
> symbol gnutls_xyz will get a embedded symbol reference of
> gnutls_xyz@GNUTLS_3_3_15.   When you then update to gnutls 3.6.3 in
> the distro, the linker will be unable to resolve gnutls_xyz@GNUTLS_3_3_15,
> because the library will now be exporting gnutls_xyz@GNUTLS_3_6_3
> instead.

Right. Note however that in gnutls we haven't kept ABI compatibility
over the years and this statement applies to backporting to an
ABI-incompatible version. Backporting to ABI-compatible versions is
explicitly disallowed to avoid the issues you describe in your post.

It is my intention to try keeping a single supported stable tree with
a given ABI (and hopefully I succeed in that).

> There's a few other options for backporting, but they are all awful
> in their own ways:
>
>   https://www.berrange.com/posts/2011/01/13/versioning-in-the-libvirt-library/
> so for libvirt we simply refuse to ever backport public APIs to older
> versions. Backports are for bug fixes only, never features.

Thank you, that's a nice read. I should have read that years ago.

regards,
Nikos

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] symbol and library versioning

Daniel P. Berrange-2
On Thu, Aug 10, 2017 at 01:41:55PM +0200, Nikos Mavrogiannopoulos wrote:

> On Thu, Aug 10, 2017 at 11:14 AM, Daniel P. Berrange
> <[hidden email]> wrote:
> > On Thu, Aug 10, 2017 at 10:31:28AM +0200, Nikos Mavrogiannopoulos wrote:
> >> Hi,
> >>  With mind in the upcoming 3.6.0 release I tried to codify/clarify the
> >> library and symbol versioning rules that are considered best practice
> >> today.
> >>
> >> The output is at:
> >> https://gitlab.com/gnutls/gnutls/blob/master/CONTRIBUTING.md#symbol-and-library-versioning
> >>
> >> My take-away from it, is that the complexity of libtool rules is
> >> unnecessary and in fact all that is being used from it is the major
> >> soname version (thus the minor and patch versions set via libtool are
> >> being used by no software).
> >
> > Yes, libtool versioning is not really useful, especially for a library
> > that intends to maintain API/ABI compat long term / forever.
>
> I wonder whether there is a reason not to keep the libtool numbers
> fixed, and follow its versioning requirements.

AFAIK, the per-symbol versions offer a superset of the functionality
provided by libtool soname versioning, at least on ELF platforms. If
you care about Windows DLLs or OS-X dynlibs, then perhaps the libtool
versions would have some small benefit, but I don't think its worth it
personally.

> >> The versioning that matters is via the linker script, which is then
> >> used by packaging software like rpm (I guess possibly dpkg as well),
> >> to detect proper dependencies.
> >>
> >> Did I miss something in that? Are there are best practices I missed,
> >> or any issues in the rules I set above?
> >
> > I believe your backporting rule is a bad idea.
> >
> >   "if symbol gnutls_xyz with version GNUTLS_3_6_3 is backported on
> >    gnutls 3.3.15, it should use version GNUTLS_3_3_15."
> >
> > This creates ABI breakage when you upgrade gnutls in the distro. eg
> >
> > An application which links to your gnutls 3.3.15 version and uses
> > symbol gnutls_xyz will get a embedded symbol reference of
> > gnutls_xyz@GNUTLS_3_3_15.   When you then update to gnutls 3.6.3 in
> > the distro, the linker will be unable to resolve gnutls_xyz@GNUTLS_3_3_15,
> > because the library will now be exporting gnutls_xyz@GNUTLS_3_6_3
> > instead.
>
> Right. Note however that in gnutls we haven't kept ABI compatibility
> over the years and this statement applies to backporting to an
> ABI-incompatible version. Backporting to ABI-compatible versions is
> explicitly disallowed to avoid the issues you describe in your post.

Ok, yes, if you have an .so ELF version  change between those
two versions, then apps need a rebuild already, so the problem
I describe wouldn't hit.


Regards,
Daniel
--
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] symbol and library versioning

Andreas Metzler-3
In reply to this post by Nikos Mavrogiannopoulos-2
On 2017-08-10 Nikos Mavrogiannopoulos <[hidden email]> wrote:
>  With mind in the upcoming 3.6.0 release I tried to codify/clarify the
> library and symbol versioning rules that are considered best practice
> today.

> The output is at:
> https://gitlab.com/gnutls/gnutls/blob/master/CONTRIBUTING.md#symbol-and-library-versioning

> My take-away from it, is that the complexity of libtool rules is
> unnecessary and in fact all that is being used from it is the major
> soname version (thus the minor and patch versions set via libtool are
> being used by no software).

Hello,

yes and no. ;-) Afaiui libtool versioning is an abstraction level. On e.g.
Linux the only important result is that a soname bump happens when
necessary. On other more exotic platforms libtool might (need to) do
something different.

> The versioning that matters is via the linker script, which is then
> used by packaging software like rpm (I guess possibly dpkg as well),
> to detect proper dependencies.

rpm and dpkg work differently here.
rpm looks at the library at build time and scans it for versioned symbol
definitions. If the library exports GNUTLS_3_4_0 and GNUTLS_3_4_1 the
resulting library package provides
libgnutls.so.30
libgnutls.so.30 (GNUTLS_3_4_0)
libgnutls.so.30 (GNUTLS_3_4_1)

And a package linking against gnutls which uses foo@GNUTLS_3_4_1 will
get a dependency against "libgnutls.so.30 (GNUTLS_3_4_1)".

Debian works a little bit differently. There is no benefit for dpkg
based systems in using the fine grained symbol version approach.  The
gnutls development package comes with a list of exported symbols[1] and
the respective version that a using package needs to depend on:
 gnutls_idna_reverse_map@GNUTLS_3_4 3.5.9
 gnutls_init@GNUTLS_3_4 3.5.6
 gnutls_key_generate@GNUTLS_3_4 3.5.0
If a binary uses both gnutls_key_generate and gnutls_init it will depend
on gnutls >= 3.5.6 (the maximum of 3.5.6 and 3.5.0).
There is usually a manual review process involved in updating this list
of symbols. This is evident by the gnutls_init() dependency info.
gnutls_init was introduced ages ago, however the GNUTLS_NO_TICKETS
argument was only added in 3.5.6.

> Did I miss something in that? Are there are best practices I missed,
> or any issues in the rules I set above?

| Symbol versioning as provided by libgnutls.map have several advantages.
|   1 they allow for symbol clashing between different gnutls library
|     versions being in the same address space.
|   2 they allow installers to detect the library version used for an
|     application utilizing a specific symbol
|   3 the allow introducing multiple versions of a symbol a la libc,
|     keeping the semantics of old functions while introducing new.

1 is what GnuTLS symbol-versioning in released versions (all symbols
versioned with a unified sonamed-specific label) currently accomplishes.
2 seems to be what you are aiming for, enhancing RedHat's package
dependencies.
3 Is this a goal you are envisioning of following? If not it would clear
things up to clearly state that GnuTLS is not (currently) using symbol
versioning for this purpose. (On a sidenote: Is there anybody except
glibc doing this?)

I think there is typo in the following text, the numbers seem to be off.
s/GNUTLS_3_6_2/GNUTLS_3_6_3/
| As such for every symbol introduced on a particular version, we create
| an entry in libgnutls.map based on the version and containing the new
| symbols. For example, if in version 3.6.3 we introduce symbol
| gnutls_xyz, the entry would be:
|
| GNUTLS_3_6_2 { global: gnutls_xyz; } GNUTLS_3_6_1;
|
| where GNUTLS_3_6_1 is the last version that symbols were introduced, and
| indicates a dependency of 3.6.2 to symbols of 3.6.1.



This has not been a problem with the old approach.
| Backporting new symbols to an old version which is soname compatible is
| not allowed (can cause quite some problems).
[...]

cu Andreas


[1] There is also an older, less fine grained way, where the development
ackage simply specifies "any binary build against and linking with this
version of the library gets a dependency >= x.y.z). If new symols are
introduced the depinfo is bumped and the generated dependency becomes
stricer even if the new symbol is not used.
--
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] symbol and library versioning

Nikos Mavrogiannopoulos-2
On Thu, 2017-08-10 at 14:21 +0200, Andreas Metzler wrote:

> > The versioning that matters is via the linker script, which is then
> > used by packaging software like rpm (I guess possibly dpkg as
> > well),
> > to detect proper dependencies.
>
> rpm and dpkg work differently here.
> rpm looks at the library at build time and scans it for versioned
> symbol
> definitions. If the library exports GNUTLS_3_4_0 and GNUTLS_3_4_1 the
> resulting library package provides
> libgnutls.so.30
> libgnutls.so.30 (GNUTLS_3_4_0)
> libgnutls.so.30 (GNUTLS_3_4_1)
>
> And a package linking against gnutls which uses foo@GNUTLS_3_4_1 will
> get a dependency against "libgnutls.so.30 (GNUTLS_3_4_1)".
>
> Debian works a little bit differently. There is no benefit for dpkg
> based systems in using the fine grained symbol version approach.  The
> gnutls development package comes with a list of exported symbols[1]
> and
> the respective version that a using package needs to depend on:
>  gnutls_idna_reverse_map@GNUTLS_3_4 3.5.9
>  gnutls_init@GNUTLS_3_4 3.5.6
>  gnutls_key_generate@GNUTLS_3_4 3.5.0
> If a binary uses both gnutls_key_generate and gnutls_init it will
> depend
> on gnutls >= 3.5.6 (the maximum of 3.5.6 and 3.5.0).
> There is usually a manual review process involved in updating this
> list
> of symbols. This is evident by the gnutls_init() dependency info.
> gnutls_init was introduced ages ago, however the GNUTLS_NO_TICKETS
> argument was only added in 3.5.6.

Thank you for this nice summary. My target is to have that versioning
suits both systems, and if I understand you correctly despite the
differences in packagers, that's the case.


> > Did I miss something in that? Are there are best practices I
> > missed,
> > or any issues in the rules I set above?
> > Symbol versioning as provided by libgnutls.map have several
> > advantages.
> >   1 they allow for symbol clashing between different gnutls library
> >     versions being in the same address space.
> >   2 they allow installers to detect the library version used for an
> >     application utilizing a specific symbol
> >   3 the allow introducing multiple versions of a symbol a la libc,
> >     keeping the semantics of old functions while introducing new.
>
> 1 is what GnuTLS symbol-versioning in released versions (all symbols
> versioned with a unified sonamed-specific label) currently
> accomplishes.
> 2 seems to be what you are aiming for, enhancing RedHat's package
> dependencies.

Right.

> 3 Is this a goal you are envisioning of following? If not it would
> clear
> things up to clearly state that GnuTLS is not (currently) using
> symbol
> versioning for this purpose.

Nice question. I was thinking to use that feature as tool to prevent
breaking ABI on cases where compatibility could be kept relatively
easily, or when the enhancement of a function would break all previous
callers, rather than strive keep a 100% backwards compatibility on
bugs.

> (On a sidenote: Is there anybody except
> glibc doing this?)

I'm not aware of other projects using that to the level of glibc.

> I think there is typo in the following text, the numbers seem to be
> off.
> s/GNUTLS_3_6_2/GNUTLS_3_6_3/

Thanks. It should be addressed now.

> > As such for every symbol introduced on a particular version, we
> > create
> > an entry in libgnutls.map based on the version and containing the
> > new
> > symbols. For example, if in version 3.6.3 we introduce symbol
> > gnutls_xyz, the entry would be:
> >
> > GNUTLS_3_6_2 { global: gnutls_xyz; } GNUTLS_3_6_1;
> >
> > where GNUTLS_3_6_1 is the last version that symbols were
> > introduced, and
> > indicates a dependency of 3.6.2 to symbols of 3.6.1.
>
> This has not been a problem with the old approach. 

Do you refer to the text above or the backporting issue?

regards,
Nikos

> > > >

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] symbol and library versioning

Andreas Metzler-3
On 2017-08-11 Nikos Mavrogiannopoulos <[hidden email]> wrote:
> On Thu, 2017-08-10 at 14:21 +0200, Andreas Metzler wrote:
[...]
>>> As such for every symbol introduced on a particular version, we
>>> create an entry in libgnutls.map based on the version and containing
>>> the new symbols. For example, if in version 3.6.3 we introduce
>>> symbol gnutls_xyz, the entry would be:
 
>>> GNUTLS_3_6_2 { global: gnutls_xyz; } GNUTLS_3_6_1;
 
>>> where GNUTLS_3_6_1 is the last version that symbols were introduced,
>>> and indicates a dependency of 3.6.2 to symbols of 3.6.1.

>> This has not been a problem with the old approach. 

> Do you refer to the text above or the backporting issue?

Hello,
I was refering to the backporting issue ("Backporting new symbols to an
old version which is soname compatible is not allowed"). e.g.
gnutls_pkcs7_get_embedded_data_oid() was backported to 3.4.17 from
3.5.6.

Afaiui this would be forbidden/impossible now.  If 3.4.17 exported
gnutls_pkcs7_get_embedded_data_oid@GNUTLS3_4_17 then 3.4.17 and 3.5.6.
would have been ABI incompatible, an upgrade from 3.4.x to 3.5.x would
require a rebuild of all binaries using
gnutls_pkcs7_get_embedded_data_oid.
I am not sure about the problems caused by doing it the other way round
(3.4.17 exporting gnutls_pkcs7_get_embedded_data_oid@GNUTLS3_5_6) but I
guess rpm's dependency tracking might stumble. - Or would the
differently chained version definitions (GNUTLS3_5_6 depending on
GNUTLS3_4_8 or GNUTLS3_5_5 respectively) break the ABI?

cu Andreas

--
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gnutls-devel] symbol and library versioning

Nikos Mavrogiannopoulos
On Fri, Aug 11, 2017 at 11:46 AM, Andreas Metzler <[hidden email]> wrote:

> On 2017-08-11 Nikos Mavrogiannopoulos <[hidden email]> wrote:
>> On Thu, 2017-08-10 at 14:21 +0200, Andreas Metzler wrote:
> [...]
>>>> As such for every symbol introduced on a particular version, we
>>>> create an entry in libgnutls.map based on the version and containing
>>>> the new symbols. For example, if in version 3.6.3 we introduce
>>>> symbol gnutls_xyz, the entry would be:
>
>>>> GNUTLS_3_6_2 { global: gnutls_xyz; } GNUTLS_3_6_1;
>
>>>> where GNUTLS_3_6_1 is the last version that symbols were introduced,
>>>> and indicates a dependency of 3.6.2 to symbols of 3.6.1.
>
>>> This has not been a problem with the old approach.
>
>> Do you refer to the text above or the backporting issue?
>
> Hello,
> I was refering to the backporting issue ("Backporting new symbols to an
> old version which is soname compatible is not allowed"). e.g.
> gnutls_pkcs7_get_embedded_data_oid() was backported to 3.4.17 from
> 3.5.6.
>
> Afaiui this would be forbidden/impossible now.  If 3.4.17 exported
> gnutls_pkcs7_get_embedded_data_oid@GNUTLS3_4_17 then 3.4.17 and 3.5.6.
> would have been ABI incompatible, an upgrade from 3.4.x to 3.5.x would
> require a rebuild of all binaries using
> gnutls_pkcs7_get_embedded_data_oid.

Right. That's why it is now forbidden. A work around this could be for
later 3.5.x versions to link the old symbol to the new one, though
that is still a work around, as it still has some rough points (some
3.5.x releases would still be abi incompatible).

> I am not sure about the problems caused by doing it the other way round
> (3.4.17 exporting gnutls_pkcs7_get_embedded_data_oid@GNUTLS3_5_6) but I
> guess rpm's dependency tracking might stumble. - Or would the
> differently chained version definitions (GNUTLS3_5_6 depending on
> GNUTLS3_4_8 or GNUTLS3_5_5 respectively) break the ABI?

My understanding is that the chaining has no effect on rpm tracking,
nor affect the ABI. However as you mention above that method will
break its dependency tracking.

regards,
Nikos

_______________________________________________
Gnutls-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnutls-devel
Loading...