Mailing List Archive

On 20/08/18 11:30, Jasper Spaans wrote:
> I was wondering, are there any plans for a fresh release of these
> bindings, as the current version on PyPI is about six months old now,

The gist of it is that the bindings are spread by upstream bundled with
GPGME, rather than through PyPi. This is because the bindings are
tightly coupled to the GPGME library, so they should be built and
installed together.

But that's all I can tell you, I'm not well acquainted with it.

HTH,

Peter.

--
I use the GNU Privacy Guard (GnuPG) in combination with Enigmail.
You can send me encrypted mail if you want some privacy.
My key is available at <http://digitalbrains.com/2012/openpgp-key-peter>

On 20/08/2018 14:02, Peter Lebbing wrote:
> The gist of it is that the bindings are spread by upstream bundled with
> GPGME, rather than through PyPi. This is because the bindings are
> tightly coupled to the GPGME library, so they should be built and
> installed together.
>
> But that's all I can tell you, I'm not well acquainted with it.
Thanks Peter! That is a bit unusual for Python packages, but I guess
there's not much to do about it for now. Anyway, I followed your advice
and made some packages for gpgme and python, but they somehow failed
when trying to load the python module (based on current git head of gpgme):

  user@testing-gpg:~$ python3 -m gpg
  Traceback (most recent call last):
  ...
  ImportError:
/usr/local/lib/python3.5/dist-packages/gpg/_gpgme.cpython-35m-x86_64-linux-gnu.so:
undefined symbol: gpgme_data_new_from_estream


This was on a ubuntu 16.04 vm, stock packages, except libgpg-error which
I created and installed a 1.32 package for.

Doing a git bisect, the commit that causes the Python module to fail is

commit f42cd70f18d53df47cc2d027bade736377d39b71
Author: Werner Koch <wk@gnupg.org>
Date:   Thu Jul 19 09:50:30 2018 +0200

core: New interface gpgme_data_new_from_estream.

which seems to be a reasonable commit given the ImportError missing symbol.

Let me see if I can create my first patch :)

Cheers,
Jasper



_______________________________________________
Gnupg-devel mailing list
Gnupg-devel@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-devel

Hi,

On Sun, 2018-08-26 at 21:21 +0200, Jasper Spaans wrote:
> That is a bit unusual for Python packages, but I guess
> there's not much to do about it for now.
One thing one could do is upload the library along with the Python
bindings. And that's not that unusual for packages on PyPi. And it
could be done easily.

Cheers,
Tobi

_______________________________________________
Gnupg-devel mailing list
Gnupg-devel@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-devel

On Sun, Aug 26, 2018 at 09:21:31PM +0200, Jasper Spaans wrote:
> On 20/08/2018 14:02, Peter Lebbing wrote:
>> The gist of it is that the bindings are spread by upstream bundled with
>> GPGME, rather than through PyPi. This is because the bindings are
>> tightly coupled to the GPGME library, so they should be built and
>> installed together.
>>
>> But that's all I can tell you, I'm not well acquainted with it.
>
> Thanks Peter! That is a bit unusual for Python packages,

Most Python packages, certainly those on PyPI, are also written
entirely or almost entirely in Python as well. These bindings are
generated with SWIG from header files which are generated when GPGME
itself is compiled. In order to dynamically generate the bindings, we
must have the location of gpgme.h for that specific architecture,
operating system and build.

This is why in this case the PyPI copy or copies are never recommended
and, unlike so many other projects, should only be used if it's not
possible to compile GPGME and generate the bindings directly, even as
a regular user. If someone is in that position, though, there's a
much higher probability that even the PyPI option won't help them
anyway.

> but I guess there's not much to do about it for now.

Perhaps not, but I recommend you read the HOWTO for the bindings; it's
the lang/python/docs/GPGMEpythonHOWTOen.org org-mode file.

> Anyway, I followed your advice and made some packages for gpgme and
> python, but they somehow failed when trying to load the python
> module (based on current git head of gpgme):
>
> ? user@testing-gpg:~$ python3 -m gpg

Don't check the bindings using this method, it won't work because this
module is not an executable program in that sense. A better check
from a shell w=is this:

bash-4.4$ python3.7 -c "import gpg; print(gpg.version.versionstr)"
1.11.2-beta257
bash-4.4$

The output of which should match the installed library version:

bash-4.4$ gpgme-config --version
1.11.2-beta257
bash-4.4$

> ? Traceback (most recent call last):
> ? ...
> ? ImportError:
> /usr/local/lib/python3.5/dist-packages/gpg/_gpgme.cpython-35m-x86_64-linux-gnu.so:
> undefined symbol: gpgme_data_new_from_estream

Interestingly enough, I hadn't seen this error at all until this
commit: 08cd34afb762975b0273575035dacf69449ef241

I'd had up to commit 138e1dde027338358a8da19ff8592b7369494534 running
live for nearly a week (that's the commit immediately prior to your
patch).

> This was on a ubuntu 16.04 vm, stock packages, except libgpg-error which
> I created and installed a 1.32 package for.

Oh dear, mixing and matching between whatever modifications Canonical
have made with parts of current source. I guess that explains why it
broke on OS X and possibly other BSDs.

> Doing a git bisect, the commit that causes the Python module to fail is
>
> commit f42cd70f18d53df47cc2d027bade736377d39b71
> Author: Werner Koch <wk@gnupg.org>
> Date:?? Thu Jul 19 09:50:30 2018 +0200
>
> core: New interface gpgme_data_new_from_estream.
>
> which seems to be a reasonable commit given the ImportError missing symbol.

I'd be surprised if it really caused the error if that were to be
tested on any Linux system which either stuck entirely with that
distributions packages *or* entirely with building the GnuPG software
from source (yes, all of it), but *not* mixing and matching between
the two.

On Sun, Aug 26, 2018 at 10:03:31PM +0200, Jasper Spaans wrote:
> On 26/08/2018 21:21, Jasper Spaans wrote:
> > Let me see if I can create my first patch :)
> Here it is, it was just a symbol that was not being exported. Verified
> that with this change applied the Python bindings work again for me.

Maybe so, but it broke for others, whether that was due to the
exported symbols conflicting with the inner API module name
(i.e. _gpgme) or the lack of defined functions for the estream in the
core, could go either way, but I'm leaning towards it being caused by
the exported symbol not lining up with _gpgme.

I've fixed those conflicts with this commit:

1d00fb987b903e245d484bddfe3c0a0aba670ac1

That, however, required changing both of your updates, so you should
test it ASAP to see if it still works or whether Ubuntu is going to be
a point of pain in future.


Regards,
Ben

On 30/08/2018 08:44, Ben McGinnes wrote:
> I've fixed those conflicts with this commit:
> 1d00fb987b903e245d484bddfe3c0a0aba670ac1
>
> That, however, required changing both of your updates, so you should
> test it ASAP to see if it still works or whether Ubuntu is going to be
> a point of pain in future.
Indeed your fix has broken my build instead.

(I don't know what Ubuntu has to do with this; and in case my previous
message was not clear enough: I did not use any Ubuntu-provided versions
of libgpg-error or gpgme. I wouldn't be surprised if this will cause
breakage on Debian as well.)

Anyway, looking at the binary objects that were generated when building
from git head (a9863717b1b82b3077edd0db85454ba801eac9bd , one version
ahead of your commit):

$ nm
/usr/local/lib/python3.5/dist-packages/gpg/_gpgme.cpython-35m-x86_64-linux-gnu.so
| grep gpgme_data_new_from_estream
???????????????? U gpgme_data_new_from_estream
000000000004c350 t _wrap_gpgme_data_new_from_estream

The U means the symbol is undefined here and needs to be provided by
another shared library. If that is nonsense, something on the Python
side should stop referring to it.

$ nm /usr/lib/libgpgme.so | grep gpgme_data_new_from_estream
000000000000b3a0 t gpgme_data_new_from_estream

Where the small t indicates that it is a symbol known inside this
library, but not exported.
This is similar to the situation before my (now reverted) patch was applied.


Random thoughts:
- there are some traces of Jenkins in the ticket system, but is it still
actively being used? I just tried accessing jenkins.gnupg.org but that
returns "service not available"? If that Jenkins is being maintained
though, I wouldn't mind setting up some jobs to test for these kinds of
regressions.
- what version of swig are you using?


Cheers,
Jasper

_______________________________________________
Gnupg-devel mailing list
Gnupg-devel@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-devel

On Thu, Aug 30, 2018 at 04:33:24PM +0200, Jasper Spaans wrote:
> On 30/08/2018 08:44, Ben McGinnes wrote:
> > I've fixed those conflicts with this commit:
> > 1d00fb987b903e245d484bddfe3c0a0aba670ac1
> >
> > That, however, required changing both of your updates, so you should
> > test it ASAP to see if it still works or whether Ubuntu is going to be
> > a point of pain in future.
> Indeed your fix has broken my build instead.

Really? Interesting. Raise a task on dev.gnupg.org with the full
traceback and other relevant errors. I'll also need the details of
which packages are standard for the distro and which aren't (if
they're listed on the gnupg.org download page, include them here).

> (I don't know what Ubuntu has to do with this; and in case my previous
> message was not clear enough: I did not use any Ubuntu-provided versions
> of libgpg-error or gpgme.

Ah, but what of the rest of the GnuPG stack; like libassuan, libksba,
libgcrypt and GPG itself?

> I wouldn't be surprised if this will cause breakage on Debian as
> well.)

That's easy enough to check, we'll get dkg to try it out before we
release. He can test it while I nudge things closer to his wishlist
items (Installing for more than two Python releases at the same time).

> Anyway, looking at the binary objects that were generated when building
> from git head (a9863717b1b82b3077edd0db85454ba801eac9bd , one version
> ahead of your commit):
>
> $ nm
> /usr/local/lib/python3.5/dist-packages/gpg/_gpgme.cpython-35m-x86_64-linux-gnu.so
> | grep gpgme_data_new_from_estream
> ???????????????? U gpgme_data_new_from_estream
> 000000000004c350 t _wrap_gpgme_data_new_from_estream
>
> The U means the symbol is undefined here and needs to be provided by
> another shared library. If that is nonsense, something on the Python
> side should stop referring to it.

Well, originally nothing on the Python side referred to it and the
new_from_stream function is just an alias to new_from_fd because
Python doesn't differentiate betwee streams (or parts thereof) and
files or file type objects. It's all data and it's all either bytes
or strings. There's now a new_from_estream function aliased to the
stream one.

You can see much more of this via grepping the output of running
strings on the SWIG integrated binary.

> $ nm /usr/lib/libgpgme.so | grep gpgme_data_new_from_estream
> 000000000000b3a0 t gpgme_data_new_from_estream
>
> Where the small t indicates that it is a symbol known inside this
> library, but not exported.

Hang on a tic ... you've got libgpgme in /usr/lib, but the
corresponding Python modules in /usr/local/lib

> This is similar to the situation before my (now reverted) patch was
> applied.

It's not entirely reverted, it's just been modified. The
modifications prevent it trying to rename the dynamically generated
submodule for "_gpgme" (i.e. gpg._gpgme) being renamed to gpg.gpgme at
certain reference points and thus causing a fatal import error for
everyone on other platforms (when large chunks of that are imported
into gpg.core and wrapped at higher level).

> Random thoughts:
> - there are some traces of Jenkins in the ticket system, but is it still
> actively being used? I just tried accessing jenkins.gnupg.org but that
> returns "service not available"? If that Jenkins is being maintained
> though, I wouldn't mind setting up some jobs to test for these kinds of
> regressions.

I don't believe it is still active, Phabricator was supposed to take
the place of all of that, but management of that has been handed over
to (one of the) Bernhards, if I recall correctly.

> - what version of swig are you using?

The latest version; 3.0.12.


Regards,
Ben

Hi,

On Thu, 2018-08-30 at 16:44 +1000, Ben McGinnes wrote:
> In order to dynamically generate the bindings, we
> must have the location of gpgme.h for that specific architecture,
> operating system and build.
but you could still upload upload binary packages to PyPI, if you wanted
to, no? Because the technical reasons you cite should be covered by
PEP513.


Cheers,
Tobi

_______________________________________________
Gnupg-devel mailing list
Gnupg-devel@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-devel

On Fri, 31 Aug 2018 00:06, ben@adversary.org said:

>> returns "service not available"? If that Jenkins is being maintained
>> though, I wouldn't mind setting up some jobs to test for these kinds of
>> regressions.
>
> I don't believe it is still active, Phabricator was supposed to take

We had problems with the hardware and despite that the new hardware is
in place we don't use the Jenkins right now: It turned out that we spent
too much time fixing Jenkins setup problems (actually our build
scripts). We will eventually get it online again but there are more
important tasks right now.

As soon as it is back the "on vacation" note at the dev.gnupg.org link
will be removed.


Shalom-Salam,

Werner


--
Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz.

On Fri, Aug 31, 2018 at 11:23:49AM +0200, Tobias Mueller wrote:
> Hi,
>
> On Thu, 2018-08-30 at 16:44 +1000, Ben McGinnes wrote:
> > In order to dynamically generate the bindings, we
> > must have the location of gpgme.h for that specific architecture,
> > operating system and build.
> but you could still upload upload binary packages to PyPI, if you wanted
> to, no? Because the technical reasons you cite should be covered by
> PEP513.

These bindings don't meet the requirements of PEP513. Indeed, the
very purpose of the bindings explicitly breaks the manylinux policy
that PEP discusses.


Regards,
Ben

On Thu, Aug 30, 2018 at 04:33:24PM +0200, Jasper Spaans wrote:
> On 30/08/2018 08:44, Ben McGinnes wrote:
> > I've fixed those conflicts with this commit:
> > 1d00fb987b903e245d484bddfe3c0a0aba670ac1
> >
> > That, however, required changing both of your updates, so you should
> > test it ASAP to see if it still works or whether Ubuntu is going to be
> > a point of pain in future.
> Indeed your fix has broken my build instead.
>
> (I don't know what Ubuntu has to do with this; and in case my previous
> message was not clear enough: I did not use any Ubuntu-provided versions
> of libgpg-error or gpgme. I wouldn't be surprised if this will cause
> breakage on Debian as well.)

Oh, by the way, when you compiled GPGME and subsequently the Python
bindings, which C compiler did you use: GCC or Clang? Also which
version of either was used?

What is the output of: python3.5 -c "import sys; print(sys.version)"


Regards,
Ben