Mailing List Archive

On 05/28/2013 07:46 PM, "Tóth Attila" wrote:
> I didn't laugh my ass ofter after discovering that python stopped working
> after upgrade. Especially since the package management system depends on a
> working python instance.
> Right after emerging python-2.7.5:
> paxctl-ng -v /usr/bin/python2.7
> /usr/bin/python2.7:
> PT_PAX : -E---
> XATTR_PAX : not found
>
> Let's put there the missing "m":
> paxctl-ng -m /usr/bin/python2.7
>
> Check back:
> paxctl-ng -v /usr/bin/python2.7
> /usr/bin/python2.7:
> PT_PAX : -Em--
> XATTR_PAX : -em--
>
> If PT_PAX has E, python2.7 would not start on my system.
> Let's correct that:
> paxctl-ng -e /usr/bin/python2.7
>
> Check back:
> paxctl-ng -v /usr/bin/python2.7
> /usr/bin/python2.7:
> PT_PAX : -em--
> XATTR_PAX : -em--
>
> Now python works again.
>
> What should I properly configure the above mentioned behavior? Is it just
> my system?

Something changed in the latest python upgrades because I'm having
problems of a different nature. I'll have to investigate.


>
> I keep the habit of creating regular binary package archives.
>
> In the mean time, freshly installed firefox binary is correct:
> paxctl-ng -v /usr/lib64/firefox/bin/firefox
> /usr/lib64/firefox/bin/firefox:
> PT_PAX : -em--
> XATTR_PAX : -em--
>
> Sidenote:
> Even after running migrate-pax -m, there are binaries on the system having
> only PT_PAX marking. Example:
> migrate-pax -m
> paxctl-ng -v /usr/bin/clear
> /usr/bin/clear:
> PT_PAX : -e---
> XATTR_PAX : not found
>
> Is that expected?

Unfortunately it is very difficult to find everything that links against
everything on a system. First there's just a simple logistic problem,
going through all ELF on a system and running ldd (or readelf -d) is
time consuming and likely to miss stuff. On gentoo with portage (not
paludis!) we have linkage info in NEEDED.ELF.2 in vdb created at build
time by examing linkage info, but this also can't be everything.
Consider plugins that dlopen-ed at runtime.

So something will be missed.

BUT!

That's not what's happening there. No XATTR_PAX flags implies the
default markings which is "-e---". This is so we don't have to go
around creating xattrs on every ELF binary on your system just to get
the default. Upstream wanted it that way and it does make sense.

Hope that makes sense.

>
> Regards:
> Dw.
>


--
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

2013.Május 29.(Sze) 03:29 időpontban Anthony G. Basile ezt írta:
> On 05/28/2013 07:46 PM, "Tóth Attila" wrote:
>> If PT_PAX has E, python2.7 would not start on my system.
>> Let's correct that:
>> paxctl-ng -e /usr/bin/python2.7
>>
>> Now python works again.
>
> Something changed in the latest python upgrades because I'm having
> problems of a different nature. I'll have to investigate.
>

I wanted the community to know, that the situation looks scary for the
first time, but there's an easy fix. In case anybody else runs into this.

>>
>> Sidenote:
>> Even after running migrate-pax -m, there are binaries on the system
>> having
>> only PT_PAX marking. Example:
>> migrate-pax -m
>> paxctl-ng -v /usr/bin/clear
>> /usr/bin/clear:
>> PT_PAX : -e---
>> XATTR_PAX : not found
>>
>
> Unfortunately it is very difficult to find everything that links against
> everything on a system. First there's just a simple logistic problem,
> going through all ELF on a system and running ldd (or readelf -d) is
> time consuming and likely to miss stuff. On gentoo with portage (not
> paludis!) we have linkage info in NEEDED.ELF.2 in vdb created at build
> time by examing linkage info, but this also can't be everything.
> Consider plugins that dlopen-ed at runtime.
>
> So something will be missed.

Is there an easy command I can use to list binaries having PT_PAX flags
and missing XATTR_PAX flags?

>
> BUT!
>
> That's not what's happening there. No XATTR_PAX flags implies the
> default markings which is "-e---". This is so we don't have to go
> around creating xattrs on every ELF binary on your system just to get
> the default. Upstream wanted it that way and it does make sense.

According to my recent experience, if EMUTRAMP is enabled by a PT_PAX flag
and there's no XATTR_PAX flag present, the system will listen to the
PT_PAX flag. Can I influence this behavior to rather use the mentioned
XATTR_PAX default and don't pay attention to the PT_PAX flag?

Thanks:
Dw.

--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057

On 29 May 2013 at 1:46, "Tóth Attila" wrote:

> I didn't laugh my ass ofter after discovering that python stopped working
> after upgrade. Especially since the package management system depends on a
> working python instance.
> Right after emerging python-2.7.5:
> paxctl-ng -v /usr/bin/python2.7
> /usr/bin/python2.7:
> PT_PAX : -E---
> XATTR_PAX : not found
>
> Let's put there the missing "m":
> paxctl-ng -m /usr/bin/python2.7

python should work with EMUTRAMP enabled only, there's no need to disable
MPROTECT (unless there's more breakage than libffi but then i'd need more
information on it). note also that the above PT_PAX value tries to turn
on EMUTRAMP in softmode (beyond not disabling it in hardmode) but it'll
fail to actually do so since PAGEEXEC/SEGMEXEC are not turned on at the
same time. IOW, paxctl -zx is enough, no need for -zEx.

On 28 May 2013 at 21:29, Anthony G. Basile wrote:

> Unfortunately it is very difficult to find everything that links against
> everything on a system. First there's just a simple logistic problem,
> going through all ELF on a system and running ldd (or readelf -d) is
> time consuming and likely to miss stuff. On gentoo with portage (not
> paludis!) we have linkage info in NEEDED.ELF.2 in vdb created at build
> time by examing linkage info, but this also can't be everything.
> Consider plugins that dlopen-ed at runtime.

PaX already has code for textrel handling where it looks at mapped libraries,
it'd be easy to add detection for mismatched process vs. library PaX flags
at the same time.

let me know if you could use this (users would have to send you logs so that
you can build a database or perhaps some tool run locally could do it for
everyone) and i'll code it up.

> That's not what's happening there. No XATTR_PAX flags implies the
> default markings which is "-e---".

actually no pax.flags xattr on the file means that it'll be ignored and
the other mechanisms will be used (which can ultimately be -e--- if no
other mechanism is configured). now what you do in paxctl-ng is another
matter, i thought that it'd keep these two flags in sync, at least when
an already existing PT_PAX one is not the default value like it wasn't
in his case (EMUTRAMP was on for softmode).

On 29 May 2013 at 8:31, "Tóth Attila" wrote:

> According to my recent experience, if EMUTRAMP is enabled by a PT_PAX flag
> and there's no XATTR_PAX flag present, the system will listen to the
> PT_PAX flag. Can I influence this behavior to rather use the mentioned
> XATTR_PAX default and don't pay attention to the PT_PAX flag?

if you don't want the kernel to take PT_PAX_FLAGS into account then simply
disable its support (CONFIG_PAX_PT_PAX_FLAGS) in your kernel config ;).

On 05/29/2013 02:31 AM, "Tóth Attila" wrote:
> 2013.Május 29.(Sze) 03:29 időpontban Anthony G. Basile ezt írta:
>> On 05/28/2013 07:46 PM, "Tóth Attila" wrote:
>>> If PT_PAX has E, python2.7 would not start on my system.
>>> Let's correct that:
>>> paxctl-ng -e /usr/bin/python2.7
>>>
>>> Now python works again.
>>
>> Something changed in the latest python upgrades because I'm having
>> problems of a different nature. I'll have to investigate.
>>
>
> I wanted the community to know, that the situation looks scary for the
> first time, but there's an easy fix. In case anybody else runs into this.
>

You can pass a glob to paxctl-ng so for example

paxctl-ng -v /bin/*

will show PT_PAX and XATTR_PAX flags for all binaries in /bin.

>>>
>>> Sidenote:
>>> Even after running migrate-pax -m, there are binaries on the system
>>> having
>>> only PT_PAX marking. Example:
>>> migrate-pax -m
>>> paxctl-ng -v /usr/bin/clear
>>> /usr/bin/clear:
>>> PT_PAX : -e---
>>> XATTR_PAX : not found
>>>
>>
>> Unfortunately it is very difficult to find everything that links against
>> everything on a system. First there's just a simple logistic problem,
>> going through all ELF on a system and running ldd (or readelf -d) is
>> time consuming and likely to miss stuff. On gentoo with portage (not
>> paludis!) we have linkage info in NEEDED.ELF.2 in vdb created at build
>> time by examing linkage info, but this also can't be everything.
>> Consider plugins that dlopen-ed at runtime.
>>
>> So something will be missed.
>
> Is there an easy command I can use to list binaries having PT_PAX flags
> and missing XATTR_PAX flags?
>
>>
>> BUT!
>>
>> That's not what's happening there. No XATTR_PAX flags implies the
>> default markings which is "-e---". This is so we don't have to go
>> around creating xattrs on every ELF binary on your system just to get
>> the default. Upstream wanted it that way and it does make sense.
>
> According to my recent experience, if EMUTRAMP is enabled by a PT_PAX flag
> and there's no XATTR_PAX flag present, the system will listen to the
> PT_PAX flag. Can I influence this behavior to rather use the mentioned
> XATTR_PAX default and don't pay attention to the PT_PAX flag?
>
> Thanks:
> Dw.
>

If you have PAX_PT_PAX_FLAGS off and PAX_XATTR_PAX_FLAGS on it will only
listen to the XATTR_PAX flags.


--
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

On 05/29/2013 07:55 PM, PaX Team wrote:
> On 28 May 2013 at 21:29, Anthony G. Basile wrote:
>
>> Unfortunately it is very difficult to find everything that links against
>> everything on a system. First there's just a simple logistic problem,
>> going through all ELF on a system and running ldd (or readelf -d) is
>> time consuming and likely to miss stuff. On gentoo with portage (not
>> paludis!) we have linkage info in NEEDED.ELF.2 in vdb created at build
>> time by examing linkage info, but this also can't be everything.
>> Consider plugins that dlopen-ed at runtime.
>
> PaX already has code for textrel handling where it looks at mapped libraries,
> it'd be easy to add detection for mismatched process vs. library PaX flags
> at the same time.
>
> let me know if you could use this (users would have to send you logs so that
> you can build a database or perhaps some tool run locally could do it for
> everyone) and i'll code it up.

Yes, that would be excellent! The kernel is the obvious place to catch
actual linkings rather than the sort of userland guesswork that I've
been grappling with. What might this exe dlopen next?

>
>> That's not what's happening there. No XATTR_PAX flags implies the
>> default markings which is "-e---".
>
> actually no pax.flags xattr on the file means that it'll be ignored and
> the other mechanisms will be used (which can ultimately be -e--- if no
> other mechanism is configured). now what you do in paxctl-ng is another
> matter, i thought that it'd keep these two flags in sync, at least when
> an already existing PT_PAX one is not the default value like it wasn't
> in his case (EMUTRAMP was on for softmode).
>

When marking PpEeMmRrSs with paxctl-ng, it does keep the two sets in
sync, although there are options that allow you to change one and not
the other and so desynchronize. But it goes out of its way, where it
can, to keep the two in sync.

migrate-pax also will copy PT_PAX to XATTR_PAX flags identically with
one exception, if PT_PAX = "-e---" then no user.pax.flags xattr is
created. I am always thinking in terms of either PAX_PT_PAX_FLAGS xor
PAX_XATTR_PAX_FLAGS is on, not both. When both are on, we fall back on
what you describe. So I adopted the approach: don't copy "-e---" to
XATTR_PAX and when you reboot into a PAX_PT_PAX_FLAGS=n and
PAX_XATTR_PAX_FLAGS=y kernel, you'll get the desired behavior.

A good approach or no?


--
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

torsdag 30 maj 2013 11.13.45 skrev Anthony G. Basile:
>
> migrate-pax also will copy PT_PAX to XATTR_PAX flags identically with
> one exception, if PT_PAX = "-e---" then no user.pax.flags xattr is
> created. I am always thinking in terms of either PAX_PT_PAX_FLAGS xor
> PAX_XATTR_PAX_FLAGS is on, not both. When both are on, we fall back on
> what you describe. So I adopted the approach: don't copy "-e---" to
> XATTR_PAX and when you reboot into a PAX_PT_PAX_FLAGS=n and
> PAX_XATTR_PAX_FLAGS=y kernel, you'll get the desired behavior.
>
> A good approach or no?
To use xattr pax flags PAX_MARKINGS need to be set to XT in make.conf
else will portage default to PT when marking.
Python need EMUTRAMP enable in the kernel with newer libffi and python
and have the E mark on the binary.
/Magnus

On 05/30/2013 11:45 AM, Magnus Granberg wrote:
> torsdag 30 maj 2013 11.13.45 skrev Anthony G. Basile:
>>
>> migrate-pax also will copy PT_PAX to XATTR_PAX flags identically with
>> one exception, if PT_PAX = "-e---" then no user.pax.flags xattr is
>> created. I am always thinking in terms of either PAX_PT_PAX_FLAGS xor
>> PAX_XATTR_PAX_FLAGS is on, not both. When both are on, we fall back on
>> what you describe. So I adopted the approach: don't copy "-e---" to
>> XATTR_PAX and when you reboot into a PAX_PT_PAX_FLAGS=n and
>> PAX_XATTR_PAX_FLAGS=y kernel, you'll get the desired behavior.
>>
>> A good approach or no?
> To use xattr pax flags PAX_MARKINGS need to be set to XT in make.conf
> else will portage default to PT when marking.

Thanks Mangus for reminding us. We had to set the default portage
behaviour to just doing PT_PAX markings because there were two
remanining issues: 1) making sure that even gentoo-sources would
preserve user.pax.flags on tmpfs and 2) fixing the default behavior of
install which currently does not preserve xattrs.

The former is solved. This was needed for people who switch back and
forth between gentoo-sources and hardened-sources (there are a few, even
I do sometimes). We don't want to loose XATTR_PAX flags when emerging
with gentoo-sources.

The latter is going to bump up against GNU standards. But I have a plan :)


> Python need EMUTRAMP enable in the kernel with newer libffi and python
> and have the E mark on the binary.
> /Magnus
>

This is now on by default so a user has to deliberally turn off EMUTRAMP
rather than vice versa.


--
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

2013.Május 30.(Cs) 17:45 időpontban Magnus Granberg ezt írta:
> torsdag 30 maj 2013 11.13.45 skrev Anthony G. Basile:
>>
>> migrate-pax also will copy PT_PAX to XATTR_PAX flags identically with
>> one exception, if PT_PAX = "-e---" then no user.pax.flags xattr is
>> created. I am always thinking in terms of either PAX_PT_PAX_FLAGS xor
>> PAX_XATTR_PAX_FLAGS is on, not both. When both are on, we fall back on
>> what you describe. So I adopted the approach: don't copy "-e---" to
>> XATTR_PAX and when you reboot into a PAX_PT_PAX_FLAGS=n and
>> PAX_XATTR_PAX_FLAGS=y kernel, you'll get the desired behavior.
>>
>> A good approach or no?
> To use xattr pax flags PAX_MARKINGS need to be set to XT in make.conf
> else will portage default to PT when marking.
> Python need EMUTRAMP enable in the kernel with newer libffi and python
> and have the E mark on the binary.
> /Magnus
>
>

Thx for pointing that out.

Note, that pax-utils eclass gentoo page describes the default action
twice, differently:
http://devmanual.gentoo.org/eclass-reference/pax-utils.eclass/index.html
In the DESCRIPTION:
"To control what markings are made, set PAX_MARKINGS in
/etc/portage/make.conf to contain either "PT", "XT" or "none". The default
is to attempt both PT_PAX and XATTR_PAX."
In ECLASS VARIABLES:
"Control which markings are made: PT = PT_PAX markings, XT = XATTR_PAX
markings Default to PT markings."
It would be good to make it unambiguous.

I've appended PAX_MARKINGS="XT" to my make.conf, emerging python 3.2 dies
in install phase with the following log snippet:
---
Skipping: CDSL_CURRENT = INT_MAX
* XT PaX marking -E with paxctl-ng
* python
>>> Source compiled.
>>> Test phase [not enabled]: dev-lang/python-3.2.5

>>> Install python-3.2.5 into
/var/tmp/portage/dev-lang/python-3.2.5/image/ category dev-lang
make -j3 DESTDIR=/var/tmp/portage/dev-lang/python-3.2.5/image/ altinstall
Creating directory /usr/bin
/bin/sh: line 5: 24666 Killed
LD_LIBRARY_PATH=/var/tmp/portage/dev-lang/python-3.2.5/work/x86_64-pc-linux-gnu:
CC='x86_64-pc-linux-gnu-gcc -pthread' LDSHARED='x86_64-pc-linux-gnu-gcc
-pthread -shared -Wl,-O1 -Wl,--as-needed -L. -Wl,-O1 -Wl,--as-needed -L.'
CFLAGS=' -DNDEBUG -O2 -march=corei7-avx -pipe -fwrapv -O2
-march=corei7-avx -pipe -fwrapv ' ./python -E
/var/tmp/portage/dev-lang/python-3.2.5/work/Python-3.2.5/setup.py $quiet
build
make: *** [sharedmods] Error 137
make: Creating directory /usr/include
*** Waiting for unfinished jobs....
---

Let's check the marking on two python binaries.

First the python binary the install tries to execute in the arch directory:
paxctl-ng -v
/var/tmp/portage/dev-lang/python-3.2.5/work/x86_64-pc-linux-gnu/python
/var/tmp/portage/dev-lang/python-3.2.5/work/x86_64-pc-linux-gnu/python:
PT_PAX : -e---
XATTR_PAX : -E---

If I try to manually execute the binary in the arch directory having XT
emutramp enabled, it results in an instant kill. If I disable emutramp for
both PT and XT, the binary executes fine.

Next the python binary located in the image directory:
axctl-ng -v /var/tmp/portage/dev-lang/python-3.2.5/image/usr/bin/python3.2
/var/tmp/portage/dev-lang/python-3.2.5/image/usr/bin/python3.2:
PT_PAX : -e---
XATTR_PAX : not found

If I try to manually execute the binary in the image directory, it shows
normal behavior and display the python interpreter's prompt.

My conclusions:
On my systems XT markings make.conf entry causes troubles during the
install phase while emerging python.
The reason for the fail is that the binary gets killed instantly with
EMUTRAMP on for XT.
The binary in the image directory lack XT markings. I don't know if later
it would get further markings, but it seems to me the markings are
performed just before the install phase.

So EMUTRAMP seems to harm python's normal execution and it's possible the
necessary XT markings would not happen on the actual binary which will be
qmerged to the system - as expected.

I'm using the latest elfix from the hardened overlay, have this one
specified in my repos.conf:
---
[DEFAULT]
# eclasses provided by hardened-dev takes precedence over
# identically named eclasses that are provided by gentoo
eclass-overrides = hardened-dev

[gentoo]
eclass-overrides = hardened-dev
---
And I'm doing emerge --regen routinely after portage & layman syncs.

I would be more than happy for doing some further testing or providing
more info as needed.

Regards:
Dw.
--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057

2013.Május 30.(Cs) 01:39 időpontban PaX Team ezt írta:
>
> python should work with EMUTRAMP enabled only, there's no need to disable
> MPROTECT (unless there's more breakage than libffi but then i'd need more
> information on it). note also that the above PT_PAX value tries to turn
> on EMUTRAMP in softmode (beyond not disabling it in hardmode) but it'll
> fail to actually do so since PAGEEXEC/SEGMEXEC are not turned on at the
> same time. IOW, paxctl -zx is enough, no need for -zEx.
>

Disabling mprotect was needed for openoffice's pyuno bridge to function
long ago. It may be unnecessary these days. I have to retest.
Enabling emutramp on the binary seems to kill python on the other hand,
while it seems to operate correctly with "e" only.
--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057