Mailing List Archive

On 07/10/14 12:51, Alex Efros wrote:
> Hi!
>
> Anyone bothers to stabilize 3.14.11-r1 anytime soon because of subj?
>

Anyone = me. You can address these concerns to me personally as I am
responsible. Bugs are best so we have a public record.

I am aware of the issue. There have been too many rapid stabilizations
because of CVE-2014-3153 and other issues. It doesn't help if I
stabilize a kernel which panics on someone's hardware that I can't test
on --- security issue or not. Been there done that. There is a balance
of risk which your statement does not take into account.

--
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail : blueness@gentoo.org
GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA
GnuPG ID : F52D4BBA

Hi!

On Fri, Jul 11, 2014 at 07:55:02AM -0400, Anthony G. Basile wrote:
> > Anyone bothers to stabilize 3.14.11-r1 anytime soon because of subj?
>
> Anyone = me. You can address these concerns to me personally as I am
> responsible. Bugs are best so we have a public record.
>
> I am aware of the issue. There have been too many rapid stabilizations
> because of CVE-2014-3153 and other issues. It doesn't help if I
> stabilize a kernel which panics on someone's hardware that I can't test
> on --- security issue or not. Been there done that. There is a balance
> of risk which your statement does not take into account.

I'm sorry if my question sounds offensive to you, this wasn't intentional.

I understand the risks, but:
- Gentoo is usually slower than other distributions on this, which is sad
- Hardened kernels are special ones - if people use hardened it means they
bothers about security more than average linux user, so they more likely
to accept the risks you mentioned
- If you (I mean Gentoo devs in general, not personally you) didn't
release or stabilize such a critical security fix because of some
reasons (not well tested on some hardware, known to have issues on some
hardware, etc.) - I think you should ASAP release GLSA or news or
whatever (announcement in this maillist, at last) to force emerge to
notify users about EXACT REASONS why this security fix isn't stabilized
yet - to let THEM decide is these reasons apply to THEIR hardware and is
they ready to take such risk and update to ~ARCH (or at least give them
idea about when it expected to be stabilized and, if any, possible
recommendations how to temporary protect against this security issue
until new kernel will be stabilized)

Last point doesn't mean you should do extra work/research etc. - just
share information you already have (reasons to not stabilize right now)
and keep people updated about changes/progress.

--
WBR, Alex.

On 11/07/14 10:11 AM, Alex Efros wrote:
> Hi!
>
> On Fri, Jul 11, 2014 at 07:55:02AM -0400, Anthony G. Basile wrote:
>>> Anyone bothers to stabilize 3.14.11-r1 anytime soon because of subj?
>>
>> Anyone = me. You can address these concerns to me personally as I am
>> responsible. Bugs are best so we have a public record.
>>
>> I am aware of the issue. There have been too many rapid stabilizations
>> because of CVE-2014-3153 and other issues. It doesn't help if I
>> stabilize a kernel which panics on someone's hardware that I can't test
>> on --- security issue or not. Been there done that. There is a balance
>> of risk which your statement does not take into account.
>
> I'm sorry if my question sounds offensive to you, this wasn't intentional.
>
> I understand the risks, but:
> - Gentoo is usually slower than other distributions on this, which is sad

gentoo also has fewer devs and less manpower than other distributions,
full stop.

> - Hardened kernels are special ones - if people use hardened it means they
> bothers about security more than average linux user

true

> so they more likely to accept the risks you mentioned

false; if anything, they are *less* likely since they are likely running
production systems.

> - If you (I mean Gentoo devs in general, not personally you) didn't
> release or stabilize such a critical security fix because of some
> reasons (not well tested on some hardware, known to have issues on some
> hardware, etc.) - I think you should ASAP release GLSA or news or
> whatever (announcement in this maillist, at last) to force emerge to
> notify users about EXACT REASONS why this security fix isn't stabilized
> yet - to let THEM decide is these reasons apply to THEIR hardware and is
> they ready to take such risk and update to ~ARCH (or at least give them
> idea about when it expected to be stabilized and, if any, possible
> recommendations how to temporary protect against this security issue
> until new kernel will be stabilized)

1. we don't have enough people to release GLSAs as is, let alone
continuous progress announcements.

2. if you want to spend that much time, follow upstream itself. any such
"notifications" would merely serve to waste sec's time that could have
been spent on actually stabilizing the package.

> Last point doesn't mean you should do extra work/research etc.

yes, it does. by definition, doing more is more work.

Hi!

On Fri, Jul 11, 2014 at 12:54:09PM -0400, Alex Xu wrote:
> > - Gentoo is usually slower than other distributions on this, which is sad
> gentoo also has fewer devs and less manpower than other distributions,
> full stop.

Sure, that's why I've said it's just "sad".

> > - Hardened kernels are special ones - if people use hardened it means they
> > bothers about security more than average linux user
> true
> > so they more likely to accept the risks you mentioned
> false; if anything, they are *less* likely since they are likely running
> production systems.

There is a risk production won't boot or won't be stable with new kernel.
First one isn't risk at all, 'cos if it won't boot you'll immediately
notice this and just boot previous kernel instead - this won't even result
in noticeable interruption of your services (no more than usual when you
reboot to update kernel).
Second is a real issue, of course, but chances are it will show itself
several days or even weeks later, and it's really questionable is it good
idea to spend so much time testing new kernel "just in case" in this situation.

At same time there is a FACT (i.e. risk with 100% chance to happens) what
your production is vulnerable, and any user (or even someone with
webshell) is able to crash it or even get root and own it.

Consequences of this may be much more costly and painful than consequences
of new kernel's instability (which may happens days/weeks later) - this is
really depends on user, so ONLY USER really able to decide is it makes
sense for him to update kernel in this situation or not.

One more thing: even if kernel isn't stable, but it took days/weeks to
find this, then there is a good chance when (or even before) most users
notice this there will be available update which fix this issue.

> > - If you (I mean Gentoo devs in general, not personally you) didn't
> > release or stabilize such a critical security fix because of some
> > reasons (not well tested on some hardware, known to have issues on some
> > hardware, etc.) - I think you should ASAP release GLSA or news or
> > whatever (announcement in this maillist, at last) to force emerge to
> > notify users about EXACT REASONS why this security fix isn't stabilized
> > yet - to let THEM decide is these reasons apply to THEIR hardware and is
> > they ready to take such risk and update to ~ARCH (or at least give them
> > idea about when it expected to be stabilized and, if any, possible
> > recommendations how to temporary protect against this security issue
> > until new kernel will be stabilized)
>
> 1. we don't have enough people to release GLSAs as is, let alone
> continuous progress announcements.

ok, let's forget about GLSA.

> 2. if you want to spend that much time, follow upstream itself. any such
> "notifications" would merely serve to waste sec's time that could have
> been spent on actually stabilizing the package.

And how following upstream helps me in this situation? Linux kernel devs
already fixed this security hole and released new kernel; some other
distributions somehow quickly tested is and also released these new
kernels… but this doesn't mean Gentoo's hardened-patched kernel also
stable, and there is no "upstream" who knows about possible issues with
hardened kernel. So, who I should to follow?

> > Last point doesn't mean you should do extra work/research etc.
>
> yes, it does. by definition, doing more is more work.

No, it doesn't. When responsible Gentoo dev decide not to stabilize kernel
with critical security fix he (I suppose!) knows WHY he made this decision.
All I'm asking - to have him share these reason(s), in any way suitable
for him - if releasing GLSA is too complicates then he may write to
maillist or even tweet (I don't use Twitter, but I'll start if that will
be only channel to get such information). How many "more work" it took to
write an email or tweet each time after making (hard, I hope) decision to
not stabilize kernel yet comparing to all other work needed before making
that decision?


Again, my main point is: ONLY USER is able to assess the risks for HIS
production server, but to do this responsibly he should KNOW REASONS why
much more qualified gentoo devs decide to not stabilize this kernel yet.
Especially if there is no actual reason except "we didn't spend X days
waiting for possible bug reports" yet.

--
WBR, Alex.

Hello!

On Fri, 11 Jul 2014 20:31:11 +0300
Alex Efros <powerman@powerman.name> wrote:

> There is a risk production won't boot or won't be stable with new
> kernel. First one isn't risk at all, 'cos if it won't boot you'll
> immediately notice this and just boot previous kernel instead - this
> won't even result in noticeable interruption of your services (no
> more than usual when you reboot to update kernel).

It is not always possible to reboot with the previous kernel. There are
datacenters (and not so few) who do not give you remote console access.
So in case of boot issue or kernel panic you have to contact someone
(who is not always available btw.) and ask nicely to reboot with the old
kernel. It happened to me once with such a datacenter, that the new
kernel version had some incompatibility with Hyper-V resulting in
panic and the interruption was nasty.

> Second is a real issue, of course, but chances are it will show itself
> several days or even weeks later, and it's really questionable is it
> good idea to spend so much time testing new kernel "just in case" in
> this situation.
>
> At same time there is a FACT (i.e. risk with 100% chance to happens)
> what your production is vulnerable, and any user (or even someone with
> webshell) is able to crash it or even get root and own it.

This is not a remote vulnerability. It is a local one
(see <http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-4699>).
Usually on production servers there is no local access anyway for
arbitrary users. You do not let people to copy executables to your
server and run it afterwards. So this vulnerability is not critical at
all.

You can any time unmask the newer kernel and use it if it fits better
for you. There is no need to stabilize it blindly.

Regards,
Balint

Hi!

On Fri, Jul 11, 2014 at 09:00:35PM +0300, Balint Szente wrote:
> It is not always possible to reboot with the previous kernel. There are

Yeah, that's one more reason why user should decide is it better for him
to update on ~ARCH kernel or wait until it will be stabilized.

> This is not a remote vulnerability. It is a local one
> (see <http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-4699>).
> Usually on production servers there is no local access anyway for
> arbitrary users. You do not let people to copy executables to your
> server and run it afterwards. So this vulnerability is not critical at all.

One trivial bug in one of .php on that server may give attacker ability to
upload and run webshell, which in turn let him run exploit for that CVE.
This may be more acceptable risk for non-hardened users, but we here use
hardened and thus I suppose take even local root exploits more seriously.

Also, not all hardened system is some production server - I'm using it on
my home workstation-server, and it do have several user accounts for my
friends or co-workers. At same time, I don't like the idea to give them
root access to my workstation only because they have usual user account.

> You can any time unmask the newer kernel and use it if it fits better
> for you. There is no need to stabilize it blindly.

Sure, I've already running hardened-sources-3.14.11-r1.

--
WBR, Alex.

On 07/11/14 14:00, Balint Szente wrote:
> Hello!
>
> On Fri, 11 Jul 2014 20:31:11 +0300
> Alex Efros <powerman@powerman.name> wrote:
>
>> There is a risk production won't boot or won't be stable with new
>> kernel. First one isn't risk at all, 'cos if it won't boot you'll
>> immediately notice this and just boot previous kernel instead - this
>> won't even result in noticeable interruption of your services (no
>> more than usual when you reboot to update kernel).
>
> It is not always possible to reboot with the previous kernel. There are
> datacenters (and not so few) who do not give you remote console access.
> So in case of boot issue or kernel panic you have to contact someone
> (who is not always available btw.) and ask nicely to reboot with the old
> kernel. It happened to me once with such a datacenter, that the new
> kernel version had some incompatibility with Hyper-V resulting in
> panic and the interruption was nasty.
>

Thank you for understanding. This is precisely why I don't just jump to
stabilizing even with major security bugs --- I don't believe in
securing your severs by locking them up. Let me describe how kernels go
from the grsec/pax patches to stable: 0. I have automatic scripts for
watching upstream when it pushesout new patchsets. 1. After downloading
a new grsec/pax patchset, I compare it to the old to see what's been
added. If its a minor fix, I will replace a previous hardened-sources
version on the gentoo tree with the latest. If its major, like when
upstream merges a new PaX patchset into grsec, I keep the previous
hardened-sources version around. 2. I do a test to make sure all the
patches apply and play nicely with the gentoo specific patches ---
hardened-sources includes genpatches and there are conflicts sometimes.
3. I do a compile test. 4. I do a boot/run test on amd64/i686
hardware. 5. It goes on the tree ~arch. 6. I wait for 4 weeks and if
there are no major bugs, I stabilize.

Rapid stabilization breaks this cycle. Because I cannot test on all
different hardware, I need community feedback whether some kernel which
is marked ~arch is breaking. We need time for this feedback. I don't
think it is wise securing people's servers by locking them up, so when I
stabilize, I'm sending a message that I've done everything reasonable to
make sure this kernel will work. I want datacenters to be confident in
these kernels.

>> Second is a real issue, of course, but chances are it will show itself
>> several days or even weeks later, and it's really questionable is it
>> good idea to spend so much time testing new kernel "just in case" in
>> this situation.
>>

It takes time before an exploit like this makes it to the wild.

>> At same time there is a FACT (i.e. risk with 100% chance to happens)
>> what your production is vulnerable, and any user (or even someone with
>> webshell) is able to crash it or even get root and own it.
>
> This is not a remote vulnerability. It is a local one
> (see <http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-4699>).
> Usually on production servers there is no local access anyway for
> arbitrary users. You do not let people to copy executables to your
> server and run it afterwards. So this vulnerability is not critical at
> all.
>
> You can any time unmask the newer kernel and use it if it fits better
> for you. There is no need to stabilize it blindly.

Correct.

>
> Regards,
> Balint
>

--
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail : blueness@gentoo.org
GnuPG FP : 1FED FAD9 D82C 52A5 3BAB DC79 9384 FA6E F52D 4BBA
GnuPG ID : F52D4BBA