Mailing List Archive

"Fetchinson . via Python-list" <python-list@python.org> writes:

> I'm trying to install a binary package (tensorflow) which contains
> some binary C extensions. Now my system glibc is 2.15 but the binaries
> in the C extensions were created (apparently) with glibc 2.17. So I
> thought no problemo I installed glibc 2.17 to a custom location, built
> python2.7 from source (hopefully using my custom glibc) and installed
> pip and everything else using this custom built python. But still when
> I try to import tensorflow I get:
>
> ImportError: /lib64/libc.so.6: version `GLIBC_2.17' not found
> (required by /home/nogradi/fetch/custom/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)
>
> So apparently it's trying to use my system glibc, not the custom one.
>
> How do I tell this extension to use the custom glibc? Is it even possible?

When you import a C extension, a variant of the linker ("ld")
is used to make the linkage between the open references in
the C extension and its (now) embedding environment.
You can use the envvar "LD_LIBRARY_PATH" (using the typical path
syntax) to tell the linker where it can look for general
shared objects (such as "glibc.so").

There is also a linker command line option to tell such information
when the shared object (corresponding to the C extension) is
build.


> But maybe I have an even more basic issue: how do I link python not
> with the system glibc but with my custom glibc?

Again, you could use the "LD_LIBRARY_PATH" envvar.

There may be a way to specify the information also for the
"configure" (generating the makefiles for the Python build process).

--
https://mail.python.org/mailman/listinfo/python-list

On 2017-09-06 16:14, Fetchinson . via Python-list wrote:
> Hi folks,
>
> I'm trying to install a binary package (tensorflow) which contains
> some binary C extensions. Now my system glibc is 2.15 but the binaries
> in the C extensions were created (apparently) with glibc 2.17. So I
> thought no problemo I installed glibc 2.17 to a custom location, built
> python2.7 from source (hopefully using my custom glibc) and installed
> pip and everything else using this custom built python. But still when
> I try to import tensorflow I get:
>
> ImportError: /lib64/libc.so.6: version `GLIBC_2.17' not found
> (required by /home/nogradi/fetch/custom/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)
>
> So apparently it's trying to use my system glibc, not the custom one.
>
> How do I tell this extension to use the custom glibc? Is it even possible?

It's going to use the same libc as python, so first of all check which
libc your python interpreter is actually linked to. Maybe your custom
build didn't do quite what you wanted.

ldd `which python` # or something like that

Once you've convinced yourself that python has the correct libc, you
could try building tensorflow from source rather than installing the
binaries.

Maybe something in here helps:
https://github.com/tensorflow/tensorflow/issues/53


>
> But maybe I have an even more basic issue: how do I link python not
> with the system glibc but with my custom glibc?
>
> Cheers,
> Daniel
>
>
>


--
Thomas Jollans
--
https://mail.python.org/mailman/listinfo/python-list

On 9/7/17, Thomas Jollans <tjol@tjol.eu> wrote:
> On 2017-09-06 16:14, Fetchinson . via Python-list wrote:
>> Hi folks,
>>
>> I'm trying to install a binary package (tensorflow) which contains
>> some binary C extensions. Now my system glibc is 2.15 but the binaries
>> in the C extensions were created (apparently) with glibc 2.17. So I
>> thought no problemo I installed glibc 2.17 to a custom location, built
>> python2.7 from source (hopefully using my custom glibc) and installed
>> pip and everything else using this custom built python. But still when
>> I try to import tensorflow I get:
>>
>> ImportError: /lib64/libc.so.6: version `GLIBC_2.17' not found
>> (required by
>> /home/nogradi/fetch/custom/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)
>>
>> So apparently it's trying to use my system glibc, not the custom one.
>>
>> How do I tell this extension to use the custom glibc? Is it even
>> possible?
>
> It's going to use the same libc as python, so first of all check which
> libc your python interpreter is actually linked to. Maybe your custom
> build didn't do quite what you wanted.
>
> ldd `which python` # or something like that
>
> Once you've convinced yourself that python has the correct libc, you
> could try building tensorflow from source rather than installing the
> binaries.
>
> Maybe something in here helps:
> https://github.com/tensorflow/tensorflow/issues/53

Thanks a lot for all the comments, my problem was indeed that the
compiled python was still using the system glibc. The solution was to
set the environment variables

p=/path/to/custom/glibc
export CFLAGS=-I${p}/include
export LDFLAGS="-Wl,--rpath=${p}/lib
-Wl,--dynamic-linker=${p}/lib/ld-linux-x86-64.so.2"

And then the compiled python was using the new glibc.

Side note 1 on tensorflow: the compiled tensorflow binary uses unicode
ucs4, so for python I had to ./configure --enable-unicode=ucs4 because
the default is ucs2

Side note 2 on tensorflow: it also depends on libstdc++ and my version
was too old for that as well. Instead of compiling gcc from source
(which includes libstdc++) I copied a binary libstdc++ from a newer
linux distro and it was working fine.

And the reason, if anyone cares, I had to go through the above is that
I couldn't compile tensorflow from source.

Thanks again,
Daniel




>
>>
>> But maybe I have an even more basic issue: how do I link python not
>> with the system glibc but with my custom glibc?
>>
>> Cheers,
>> Daniel
>>
>>
>>
>
>
> --
> Thomas Jollans
> --
> https://mail.python.org/mailman/listinfo/python-list
>


--
Psss, psss, put it down! - http://www.cafepress.com/putitdown
--
https://mail.python.org/mailman/listinfo/python-list

On Friday, September 8, 2017 at 3:16:12 AM UTC+2, Fetchinson . wrote:
> On 9/7/17, Thomas Jollans <tj...@tjol.eu> wrote:
> > On 2017-09-06 16:14, Fetchinson . via Python-list wrote:
> >> Hi folks,
> >>
> >> I'm trying to install a binary package (tensorflow) which contains
> >> some binary C extensions. Now my system glibc is 2.15 but the binaries
> >> in the C extensions were created (apparently) with glibc 2.17. So I
> >> thought no problemo I installed glibc 2.17 to a custom location, built
> >> python2.7 from source (hopefully using my custom glibc) and installed
> >> pip and everything else using this custom built python. But still when
> >> I try to import tensorflow I get:
> >>
> >> ImportError: /lib64/libc.so.6: version `GLIBC_2.17' not found
> >> (required by
> >> /home/nogradi/fetch/custom/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)
> >>
> >> So apparently it's trying to use my system glibc, not the custom one.
> >>
> >> How do I tell this extension to use the custom glibc? Is it even
> >> possible?
> >
> > It's going to use the same libc as python, so first of all check which
> > libc your python interpreter is actually linked to. Maybe your custom
> > build didn't do quite what you wanted.
> >
> > ldd `which python` # or something like that
> >
> > Once you've convinced yourself that python has the correct libc, you
> > could try building tensorflow from source rather than installing the
> > binaries.
> >
> > Maybe something in here helps:
> > https://github.com/tensorflow/tensorflow/issues/53
> Thanks a lot for all the comments, my problem was indeed that the
> compiled python was still using the system glibc. The solution was to
> set the environment variables
>
> p=/path/to/custom/glibc
> export CFLAGS=-I${p}/include
> export LDFLAGS="-Wl,--rpath=${p}/lib
> -Wl,--dynamic-linker=${p}/lib/ld-linux-x86-64.so.2"
>
> And then the compiled python was using the new glibc.
>
> Side note 1 on tensorflow: the compiled tensorflow binary uses unicode
> ucs4, so for python I had to ./configure --enable-unicode=ucs4 because
> the default is ucs2
>
> Side note 2 on tensorflow: it also depends on libstdc++ and my version
> was too old for that as well. Instead of compiling gcc from source
> (which includes libstdc++) I copied a binary libstdc++ from a newer
> linux distro and it was working fine.
>
> And the reason, if anyone cares, I had to go through the above is that
> I couldn't compile tensorflow from source.
>
> Thanks again,
> Daniel
> >
> >>
> >> But maybe I have an even more basic issue: how do I link python not
> >> with the system glibc but with my custom glibc?
> >>
> >> Cheers,
> >> Daniel
> >>
> >>
> >>
> >
> >
> > --
> > Thomas Jollans
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
>
>
> --
> Psss, psss, put it down! - http://www.cafepress.com/putitdown

First of all, thank you a lot Fetchinson, it helped on my system (Centos 6) to today recompile the newest Python (3.10.7)!
Of course I had foss2018 available on the system, so I "just" had to compile glibc, zlib, and openssl 1.1.1q ... but hey it was worth it! I have my environment up and running!

FYI:
1. to me it helped that since I had nonstandard openssl location to edit Modules/Setup.dist as suggested here https://stackoverflow.com/questions/5937337/building-python-with-ssl-support-in-non-standard-location
2. to me it helped to configure with
./configure --with-libc=/tmp/glibc/lib/libc.so --prefix=/tmp/python --with-openssl=/tmp/openssl

Hope this helps someone else who will try to do the same.

Cheers
--
https://mail.python.org/mailman/listinfo/python-list