Quantcast

[PATCH GPGME 0/5] python: misc build fixes

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH GPGME 0/5] python: misc build fixes

Alon Bar-Lev-3
This concludes the work to make the build downstream friendly. It enables
to build using downstream specific build and test using the in-tree tests.
It also support a distro wrapper instead of gpg-error.h.

Alon Bar-Lev (5):
  python: support .pydistutils.cfg mode
  python: Remove usage of PYTHON_VERSIONS
  python: Remove unneeded stats copy
  python: Read gpg-error.h using the pre-processor
  python: Support alternatate libdir for tests

 configure.ac                   |  1 -
 lang/python/Makefile.am        | 23 ++++++------
 lang/python/setup.py.in        | 85 +++++++++++++++++++++---------------------
 lang/python/tests/run-tests.py | 25 ++++++++-----
 4 files changed, 68 insertions(+), 66 deletions(-)

--
2.10.2


_______________________________________________
Gnupg-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnupg-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH GPGME 1/5] python: support .pydistutils.cfg mode

Alon Bar-Lev-3
* lang/python/setup.py.in: Do not parse arguments.

--

The distutils settings can come from either command-line or
configuration file. Parsing parameters is not working in all cases.

Signed-off-by: Alon Bar-Lev <[hidden email]>
---
 lang/python/setup.py.in | 25 ++++++++++---------------
 1 file changed, 10 insertions(+), 15 deletions(-)

diff --git a/lang/python/setup.py.in b/lang/python/setup.py.in
index 5d94c70..e50971c 100755
--- a/lang/python/setup.py.in
+++ b/lang/python/setup.py.in
@@ -19,18 +19,12 @@
 #    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307  USA
 
 from distutils.core import setup, Extension
-import argparse
 import os, os.path, sys
 import glob
 import re
 import shutil
 import subprocess
 
-# We parse a subset of the arguments.
-parser = argparse.ArgumentParser(add_help=False)
-parser.add_argument('--build-base', default='')
-options, _ = parser.parse_known_args()
-
 # Out-of-tree build of the gpg bindings.
 gpg_error_config = ["gpg-error-config"]
 gpgme_config_flags = ["--thread=pthread"]
@@ -142,8 +136,6 @@ if uname_s.startswith("MINGW32"):
 
 def in_srcdir(name):
     return os.path.join(os.environ.get("srcdir", ""), name)
-def in_build_base(name):
-    return os.path.join(options.build_base, name)
 def up_to_date(source, target):
     return (os.path.exists(target)
             and os.path.getmtime(source) <= os.path.getmtime(target))
@@ -190,6 +182,9 @@ class BuildExtFirstHack(build):
                     continue
                 sink.write(rewrite_re.sub(r'%constant long \1 = \1;'+'\n', line.strip()))
 
+    def _in_build_base(self, name):
+        return os.path.join(self.build_base, name)
+
     def _generate(self):
         print("Building python gpg module using {} and {}.".format(gpgme_h, gpg_error_h))
 
@@ -197,18 +192,18 @@ class BuildExtFirstHack(build):
         if not os.path.exists(self.build_base):
             os.makedirs(self.build_base)
 
-        self._generate_gpgme_h(gpgme_h, in_build_base("gpgme.h"))
-        self._generate_errors_i(gpg_error_h, in_build_base("errors.i"))
+        self._generate_gpgme_h(gpgme_h, self._in_build_base("gpgme.h"))
+        self._generate_errors_i(gpg_error_h, self._in_build_base("errors.i"))
 
         # Keep timestamp to avoid rebuild
-        for source, target in ((gpgme_h, in_build_base("gpgme.h")),
-                               (gpg_error_h, in_build_base("errors.i"))):
+        for source, target in ((gpgme_h, self._in_build_base("gpgme.h")),
+                               (gpg_error_h, self._in_build_base("errors.i"))):
             if not up_to_date(source, target):
                 shutil.copystat(source, target)
 
         # Copy due to http://bugs.python.org/issue2624
         # Avoid creating in srcdir
-        for source, target in ((in_srcdir(n), in_build_base(n))
+        for source, target in ((in_srcdir(n), self._in_build_base(n))
                                for n in ('gpgme.i', 'helpers.c', 'private.h', 'helpers.h')):
             if not up_to_date(source, target):
                 shutil.copy2(source, target)
@@ -221,7 +216,7 @@ class BuildExtFirstHack(build):
     def run(self):
         self._generate()
 
-        swig_sources.append(os.path.join(self.build_base, 'gpgme.i'))
+        swig_sources.extend((self._in_build_base('gpgme.i'), self._in_build_base('helpers.c')))
         swig_opts.extend(['-I' + self.build_base,
                           '-outdir', os.path.join(self.build_lib, 'gpg')])
         include_dirs.append(self.build_base)
@@ -230,7 +225,7 @@ class BuildExtFirstHack(build):
         build.run(self)
 
 py3 = [] if sys.version_info.major < 3 else ['-py3']
-swig_sources = [in_build_base('helpers.c')]
+swig_sources = []
 swig_opts = ['-threads'] + py3 + extra_swig_opts
 swige = Extension("gpg._gpgme",
                   sources = swig_sources,
--
2.10.2


_______________________________________________
Gnupg-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnupg-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH GPGME 2/5] python: Remove usage of PYTHON_VERSIONS

Alon Bar-Lev-3
In reply to this post by Alon Bar-Lev-3
* configure.ac: Remove PYTHON_VERSIONS subst.
* lang/python/Makefile.am: Use basename of python as builddir prefix.
* lang/python/tests/run-tests.py: Likewise.

--

Two variables needs be at sync PYTHONS and PYTHON_VERSIONS, these may go
out of sync in some cases, for example in Gentoo where default python is
3.4 we get:

PYTHON='/usr/bin/python2'
PYTHONS='/usr/bin/python /usr/bin/python2'
PYTHON_VERSIONS='2.7 3.4'

We can use the basename of the python interpreter to achieve similar
effect without having to sync indexes between these two variables.

Signed-off-by: Alon Bar-Lev <[hidden email]>
---
 configure.ac                   |  1 -
 lang/python/Makefile.am        | 23 +++++++++++------------
 lang/python/tests/run-tests.py |  4 ++--
 3 files changed, 13 insertions(+), 15 deletions(-)

diff --git a/configure.ac b/configure.ac
index 9974abb..becd156 100644
--- a/configure.ac
+++ b/configure.ac
@@ -474,7 +474,6 @@ if test "$found_py" = "1" -o "$found_py2" = "1" -o "$found_py3" = "1"; then
  fi
 
  AC_SUBST(PYTHONS, $PYTHONS)
- AC_SUBST(PYTHON_VERSIONS, $PYTHON_VERSIONS)
     fi
 fi
 
diff --git a/lang/python/Makefile.am b/lang/python/Makefile.am
index 4ebd214..90075f7 100644
--- a/lang/python/Makefile.am
+++ b/lang/python/Makefile.am
@@ -38,12 +38,11 @@ copystamp:
  touch $@
 
 all-local: copystamp
- set -e ; set $(PYTHONS); for VERSION in $(PYTHON_VERSIONS); do \
-  PYTHON="$$1" ; shift ; \
+ set -e ; for PYTHON in $(PYTHONS); do \
   CFLAGS="$(CFLAGS)" \
   srcdir="$(srcdir)" \
   top_builddir="$(top_builddir)" \
-    $$PYTHON setup.py build --verbose --build-base=python$${VERSION}-gpg ; \
+    $$PYTHON setup.py build --verbose --build-base="$$(basename "$${PYTHON}")-gpg" ; \
  done
 
 python$(PYTHON_VERSION)-gpg/dist/gpg-$(VERSION).tar.gz.asc: copystamp
@@ -76,26 +75,26 @@ CLEANFILES = copystamp \
 # permissions.
 clean-local:
  rm -rf -- build
- for VERSION in $(PYTHON_VERSIONS); do \
-  find python$${VERSION}-gpg* -type d ! -perm -200 -exec chmod u+w {} ';' ; \
-  rm -rf -- python$${VERSION}-gpg* ; \
+ for PYTHON in $(PYTHONS); do \
+  find "$$(basename "$${PYTHON}")-gpg" -type d ! -perm -200 -exec chmod u+w {} ';' ; \
+  rm -rf -- "$$(basename "$${PYTHON}")-gpg" ; \
  done
 
 install-exec-local:
- set -e ; set $(PYTHONS); for VERSION in $(PYTHON_VERSIONS); do \
-  PYTHON="$$1" ; shift ; \
+ set -e ; for PYTHON in $(PYTHONS); do \
   srcdir="$(srcdir)" \
   top_builddir="$(top_builddir)" \
   $$PYTHON setup.py \
   build \
-  --build-base=python$${VERSION}-gpg \
+  --build-base="$$(basename "$${PYTHON}")-gpg" \
   install \
   --prefix "$(DESTDIR)$(prefix)" \
   --verbose ; \
  done
 
 uninstall-local:
- GV=$$(echo $(VERSION) | tr - _); for PV in $(PYTHON_VERSIONS); do \
-  rm -rf -- "$(DESTDIR)$(prefix)"/lib*/python$$PV/site-packages/gpg \
-"$(DESTDIR)$(prefix)"/lib*/python$$PV/site-packages/gpg-$$GV-py$$PV.egg-info ; \
+ GV=$$(echo $(VERSION) | tr - _); for PYTHON in $(PYTHONS); do \
+  PLATLIB="$(prefix)/$$("$${PYTHON}" -c 'import sysconfig, os; print(os.path.relpath(sysconfig.get_path("platlib"), sysconfig.get_config_var("prefix")))')" ; \
+  rm -rf -- "$(DESTDIR)$${PLATLIB}/gpg" \
+ "$(DESTDIR)$${PLATLIB}"/gpg-$$GV-py*.egg-info ; \
  done
diff --git a/lang/python/tests/run-tests.py b/lang/python/tests/run-tests.py
index 9e2fb78..f236712 100644
--- a/lang/python/tests/run-tests.py
+++ b/lang/python/tests/run-tests.py
@@ -70,8 +70,8 @@ for interpreter in args.interpreters:
         [interpreter, "-c", "import sys; print('{0}.{1}'.format(sys.version_info[0], sys.version_info[1]))"]).strip().decode()
 
     pattern = os.path.join(args.builddir, "..",
-                           "python{0}-gpg".format(version),
-                           "lib*"+version)
+                           "{0}-gpg".format(os.path.basename(interpreter)),
+                           "lib*")
     builddirs = glob.glob(pattern)
     if len(builddirs) == 0:
         sys.exit("Build directory matching {0!r} not found.".format(pattern))
--
2.10.2


_______________________________________________
Gnupg-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnupg-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH GPGME 3/5] python: Remove unneeded stats copy

Alon Bar-Lev-3
In reply to this post by Alon Bar-Lev-3
* lang/python/setup.py.in: errors.i, gpgme.h are generated and always
newer than the original.

Signed-off-by: Alon Bar-Lev <[hidden email]>
---
 lang/python/setup.py.in | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/lang/python/setup.py.in b/lang/python/setup.py.in
index e50971c..f4ce64f 100755
--- a/lang/python/setup.py.in
+++ b/lang/python/setup.py.in
@@ -195,12 +195,6 @@ class BuildExtFirstHack(build):
         self._generate_gpgme_h(gpgme_h, self._in_build_base("gpgme.h"))
         self._generate_errors_i(gpg_error_h, self._in_build_base("errors.i"))
 
-        # Keep timestamp to avoid rebuild
-        for source, target in ((gpgme_h, self._in_build_base("gpgme.h")),
-                               (gpg_error_h, self._in_build_base("errors.i"))):
-            if not up_to_date(source, target):
-                shutil.copystat(source, target)
-
         # Copy due to http://bugs.python.org/issue2624
         # Avoid creating in srcdir
         for source, target in ((in_srcdir(n), self._in_build_base(n))
--
2.10.2


_______________________________________________
Gnupg-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnupg-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH GPGME 4/5] python: Read gpg-error.h using the pre-processor

Alon Bar-Lev-3
In reply to this post by Alon Bar-Lev-3
* lang/python/setup.py.in: Read gpg-error.h using the pre-processor.

--

The libgpg-error may be installed in multilib configuration in which
there is a wrapper header at /usr/include that includes the actual
header at /usr/include/*. This causes invalid errors.i generation.

Let the pre-processor extract the header content instead reading it
explicitly.

Signed-off-by: Alon Bar-Lev <[hidden email]>
---
 lang/python/setup.py.in | 60 ++++++++++++++++++++++++++++---------------------
 1 file changed, 35 insertions(+), 25 deletions(-)

diff --git a/lang/python/setup.py.in b/lang/python/setup.py.in
index f4ce64f..a1279f8 100755
--- a/lang/python/setup.py.in
+++ b/lang/python/setup.py.in
@@ -55,13 +55,6 @@ else:
     devnull = open(os.devnull, "w")
 
 try:
-    subprocess.check_call(gpg_error_config + ['--version'],
-                          stdout=devnull)
-except:
-    sys.exit("Could not find gpg-error-config.  " +
-             "Please install the libgpg-error development package.")
-
-try:
     subprocess.check_call(gpgme_config + ['--version'],
                           stdout=devnull)
 except:
@@ -84,13 +77,6 @@ if not (major > 1 or (major == 1 and minor >= 7)):
 if not gpgme_h:
     gpgme_h = os.path.join(getconfig("prefix")[0], "include", "gpgme.h")
 
-gpg_error_prefix = getconfig("prefix", config=gpg_error_config)[0]
-gpg_error_h = os.path.join(gpg_error_prefix, "include", "gpg-error.h")
-if not os.path.exists(gpg_error_h):
-    gpg_error_h = \
-        glob.glob(os.path.join(gpg_error_prefix, "include",
-                               "*", "gpg-error.h"))[0]
-
 define_macros = []
 libs = getconfig('libs')
 
@@ -150,10 +136,27 @@ def up_to_date(source, target):
 from distutils.command.build import build
 class BuildExtFirstHack(build):
 
+    def _read_header(self, header, cflags):
+        tmp_include = self._in_build_base("include1.h")
+        with open(tmp_include, 'w') as f:
+            f.write("#include <%s>" % header)
+        return subprocess.check_output(os.environ.get('CPP', 'cc -E').split() + cflags + [tmp_include]).decode('utf-8')
+
+    def _write_if_unchanged(self, target, content):
+        if os.path.exists(target):
+            with open(target) as f:
+                if f.read() == content:
+                    return
+
+        with open(target, "w") as sink:
+            sink.write(content)
+
     def _generate_gpgme_h(self, source_name, sink_name):
         if up_to_date(source_name, sink_name):
             return
 
+        print("Using gpgme.h from {}".format(source_name))
+
         deprec_func = re.compile(r'^(.*typedef.*|.*\(.*\)|[^#]+\s+.+)'
                                  + r'\s*_GPGME_DEPRECATED(_OUTSIDE_GPGME)?\(.*\);\s*',
                                  re.S)
@@ -169,31 +172,38 @@ class BuildExtFirstHack(build):
                     text = ''
             sink.write(text)
 
-    def _generate_errors_i(self, source_name, sink_name):
-        if up_to_date(source_name, sink_name):
-            return
+    def _generate_errors_i(self):
+
+        try:
+            subprocess.check_call(gpg_error_config + ['--version'],
+                                  stdout=devnull)
+        except:
+            sys.exit("Could not find gpg-error-config.  " +
+                     "Please install the libgpg-error development package.")
+
+        gpg_error_content = self._read_header("gpg-error.h", getconfig("cflags", config=gpg_error_config))
 
         filter_re = re.compile(r'GPG_ERR_[^ ]* =')
         rewrite_re = re.compile(r' *(.*) = .*')
 
-        with open(sink_name, "w") as sink, open(source_name) as source:
-            for line in source:
-                if not filter_re.search(line):
-                    continue
-                sink.write(rewrite_re.sub(r'%constant long \1 = \1;'+'\n', line.strip()))
+        errors_i_content = ''
+        for line in gpg_error_content.splitlines():
+            if not filter_re.search(line):
+                continue
+            errors_i_content += rewrite_re.sub(r'%constant long \1 = \1;'+'\n', line.strip())
+
+        self._write_if_unchanged(self._in_build_base("errors.i"), errors_i_content)
 
     def _in_build_base(self, name):
         return os.path.join(self.build_base, name)
 
     def _generate(self):
-        print("Building python gpg module using {} and {}.".format(gpgme_h, gpg_error_h))
-
         # Cleanup gpgme.h from deprecated functions and typedefs.
         if not os.path.exists(self.build_base):
             os.makedirs(self.build_base)
 
         self._generate_gpgme_h(gpgme_h, self._in_build_base("gpgme.h"))
-        self._generate_errors_i(gpg_error_h, self._in_build_base("errors.i"))
+        self._generate_errors_i()
 
         # Copy due to http://bugs.python.org/issue2624
         # Avoid creating in srcdir
--
2.10.2


_______________________________________________
Gnupg-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnupg-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH GPGME 5/5] python: Support alternatate libdir for tests

Alon Bar-Lev-3
In reply to this post by Alon Bar-Lev-3
* lang/python/tests/run-tests.py: Add --python-libdir optional
parameter.

--

This will make the python tests usable for downstream that build python
module outside of autotools build system.

Signed-off-by: Alon Bar-Lev <[hidden email]>
---
 lang/python/tests/run-tests.py | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/lang/python/tests/run-tests.py b/lang/python/tests/run-tests.py
index f236712..78a0ec2 100644
--- a/lang/python/tests/run-tests.py
+++ b/lang/python/tests/run-tests.py
@@ -51,6 +51,9 @@ parser.add_argument('--srcdir', type=str,
 parser.add_argument('--builddir', type=str,
                     default=os.environ.get("abs_builddir", ""),
                     help='Location of the tests.')
+parser.add_argument('--python-libdir', type=str,
+                    default=None,
+                    help='Optional location of the in-tree module lib directory.')
 parser.add_argument('--parallel', action="store_true", default=False,
                     help='Ignored.  For compatibility with run-tests.scm.')
 
@@ -69,18 +72,20 @@ for interpreter in args.interpreters:
     version = subprocess.check_output(
         [interpreter, "-c", "import sys; print('{0}.{1}'.format(sys.version_info[0], sys.version_info[1]))"]).strip().decode()
 
-    pattern = os.path.join(args.builddir, "..",
-                           "{0}-gpg".format(os.path.basename(interpreter)),
-                           "lib*")
-    builddirs = glob.glob(pattern)
-    if len(builddirs) == 0:
-        sys.exit("Build directory matching {0!r} not found.".format(pattern))
-    elif len(builddirs) > 1:
-        sys.exit("Multiple build directories matching {0!r} found: {1}".format(
-            pattern, builddirs))
+    if not args.python_libdir:
+        pattern = os.path.join(args.builddir, "..",
+                               "{0}-gpg".format(os.path.basename(interpreter)),
+                               "lib*")
+        libdirs = glob.glob(pattern)
+        if len(libdirs) == 0:
+            sys.exit("Build directory matching {0!r} not found.".format(pattern))
+        elif len(libdirs) > 1:
+            sys.exit("Multiple build directories matching {0!r} found: {1}".format(
+                pattern, libdirs))
+        python_libdir = libdirs[0]
 
     env = dict(os.environ)
-    env["PYTHONPATH"] = builddirs[0]
+    env["PYTHONPATH"] = python_libdir
 
     if not args.quiet:
         print("Running tests using {0} ({1})...".format(interpreter, version))
--
2.10.2


_______________________________________________
Gnupg-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnupg-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH GPGME 0/5] python: misc build fixes

Justus Winter
In reply to this post by Alon Bar-Lev-3
Hi :)

Alon Bar-Lev <[hidden email]> writes:

> This concludes the work to make the build downstream friendly. It enables
> to build using downstream specific build and test using the in-tree tests.
> It also support a distro wrapper instead of gpg-error.h.
>
> Alon Bar-Lev (5):
>   python: support .pydistutils.cfg mode
>   python: Remove usage of PYTHON_VERSIONS
>   python: Remove unneeded stats copy
>   python: Read gpg-error.h using the pre-processor
>   python: Support alternatate libdir for tests
I merged all your patches with a small fixup.  Many thanks for working
on this :)

Justus

_______________________________________________
Gnupg-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnupg-devel

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH GPGME 0/5] python: misc build fixes

Alon Bar-Lev-3
On 10 April 2017 at 16:47, Justus Winter <[hidden email]> wrote:

>
> Hi :)
>
> Alon Bar-Lev <[hidden email]> writes:
>
> > This concludes the work to make the build downstream friendly. It enables
> > to build using downstream specific build and test using the in-tree tests.
> > It also support a distro wrapper instead of gpg-error.h.
> >
> > Alon Bar-Lev (5):
> >   python: support .pydistutils.cfg mode
> >   python: Remove usage of PYTHON_VERSIONS
> >   python: Remove unneeded stats copy
> >   python: Read gpg-error.h using the pre-processor
> >   python: Support alternatate libdir for tests
>
> I merged all your patches with a small fixup.  Many thanks for working
> on this :)

Thanks!
One leftover, I sent a patch to fix.
And one test that is failing, I will open a new thread.

_______________________________________________
Gnupg-devel mailing list
[hidden email]
http://lists.gnupg.org/mailman/listinfo/gnupg-devel
Loading...