Mailing List Archive

On Sunday 26 September 2010, Richard Freeman wrote:
*gentoo-sources-2.6.32-r18 (21 Sep 2010)

21 Sep 2010; Mike Pagano <mpagano@gentoo.org>
+gentoo-sources-2.6.32-r18.ebuild:
Linux patch 2.6.32.22. Includes fix for CVE-2010-3301.

*gentoo-sources-2.6.35-r8 (21 Sep 2010)

21 Sep 2010; Mike Pagano <mpagano@gentoo.org>
+gentoo-sources-2.6.35-r8.ebuild:
Linux patch 2.6.35.5. Includes fix for CVE-2010-3301.

*gentoo-sources-2.6.35-r7 (16 Sep 2010)
*gentoo-sources-2.6.34-r10 (16 Sep 2010)

16 Sep 2010; Mike Pagano <mpagano@gentoo.org>
+gentoo-sources-2.6.34-r10.ebuild, +gentoo-sources-2.6.35-r7.ebuild:
Fix for exploit: IA32 Syscall Entry Point Privilege Escalation
(CVE-2010-3301)

date:
So 26. Sep 13:48:41 CEST 2010

so there has been roughly a week so far.

And the bug is not that dangerous - except when you insist on running unsecure
32bit software on a 64bit system.

On 09/26/2010 07:51 AM, Volker Armin Hemmann wrote:
> so there has been roughly a week so far.

Agreed - 10 days was the figure I mentioned. So far we're 7 over the
target of 3. Most major distros did it in less than 1.

>
> And the bug is not that dangerous - except when you insist on running unsecure
> 32bit software on a 64bit system.
>

I didn't realize that multilib amd64 wasn't a security-supported
configuration of Gentoo. Perhaps that should be documented somewhere -
like the amd64 handbook, and the multilib howto. The security page
probably should also be updated - to indicate that amd64 is a supported
arch only without multilib.

Note that you don't need to RUN any 32-bit software to be insecure - you
merely need to have support for it enabled in the kernel config.

Look, either multilib is supported, or it isn't. If it isn't, that's a
pretty big caveat that we don't document ANYWHERE. If it is, then we
have to fix bugs in line with the security guidelines.

I'm just asking for us to be up-front with our policies, and to follow
them. If we don't support multilib amd64, fine. If we do support it,
then we need to support it.

Rich

On Sun, Sep 26, 2010 at 08:16:22AM -0400, Richard Freeman wrote:
> On 09/26/2010 07:51 AM, Volker Armin Hemmann wrote:
> > so there has been roughly a week so far.
> Agreed - 10 days was the figure I mentioned. So far we're 7 over the
> target of 3. Most major distros did it in less than 1.
It WAS well within the 3 day target. The turnaround to the first fix was
just over 4 hours.

From the bug:
====
-- Comment #3 From Mike Pagano 2010-09-16 18:34:02 0000 [reply] -
Patches included in gentoo-sources versions: 2.6.32-r17, 2.6.34-r10 and
2.6.35-r7
====

Even hardened-kernel was fixed within 2 days of bug opening.

The delay is in stabilization, and I don't know why that's been held up.

--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : robbat2@gentoo.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85

On 26 September 2010 11:31, Richard Freeman <rich0@gentoo.org> wrote:
> Gentoo has been vulnerable to a highly-publicized (Guardian, Slashdot,
> the works) local privilege escalation for almost two weeks now.  (Well,
> it has been vulnerable for years, but of course we didn't know about it
> until two weeks ago.)
>
> In the bugzilla thread tracking the problem it has been mentioned a few
> times that the kernel does not receive GLSA support:
> http://bugs.gentoo.org/show_bug.cgi?id=337645

Kernels used to be covered in GLSAs.
I mourned the loss of kernel GLSAs quite a while back.
http://blog.gmane.org/gmane.linux.gentoo.security/month=20070401

Kernels used to be included, but apparently it was too much work
getting all the version kernel versions in sync.
I used to have script that emailed me applicable GLSAs, and I never
heard that they stopped including the kernel, so I was miffed when I
found out.

I still don't understand why there isn't a single security alert point
of reference that covers everything on a Gentoo box though.
What would it take to get kernels included again?

/meh.

PS. Hardened Gentoo still rocks though.

On 09/26/2010 12:56 PM, Robin H. Johnson wrote:
> Even hardened-kernel was fixed within 2 days of bug opening.
>
> The delay is in stabilization, and I don't know why that's been held up.
>

Agreed, but if it isn't stable then it isn't fixed.

I can keyword stable at anytime on amd64 (well, assuming 34-r10 is the
one to go stable - for anything else I'd have to test it first) - just
waiting for the package maintainer to request stabilization...

Rich

Excerpts from Calum's message of Sun Sep 26 19:28:01 +0200 2010:
> On 26 September 2010 11:31, Richard Freeman <rich0@gentoo.org> wrote:
> > Gentoo has been vulnerable to a highly-publicized (Guardian, Slashdot,
> > the works) local privilege escalation for almost two weeks now.  (Well,
> > it has been vulnerable for years, but of course we didn't know about it
> > until two weeks ago.)
> >
> > In the bugzilla thread tracking the problem it has been mentioned a few
> > times that the kernel does not receive GLSA support:
> > http://bugs.gentoo.org/show_bug.cgi?id=337645
>
> Kernels used to be covered in GLSAs.
> I mourned the loss of kernel GLSAs quite a while back.
> http://blog.gmane.org/gmane.linux.gentoo.security/month=20070401

I kindly request follow-up posters to not post +1's in this thread.

> […]
> I still don't understand why there isn't a single security alert point
> of reference that covers everything on a Gentoo box though.
> What would it take to get kernels included again?

Kernel sources will not be included in the GLSA system again.
The whole process was designed for userland packages, not kernel
sources.

We hope to get the kernel-check [1] utility to serve this purpose one
day.

The invitation Kurt extended to contact us and help is still standing.

[1]
http://git.overlays.gentoo.org/gitweb/?p=proj/kernel-check.git;a=summary
--
Alex Legler <a3li@gentoo.org>
Gentoo Security/Ruby

Excerpts from Richard Freeman's message of Sun Sep 26 20:17:24 +0200 2010:
> On 09/26/2010 12:56 PM, Robin H. Johnson wrote:
> > Even hardened-kernel was fixed within 2 days of bug opening.
> >
> > The delay is in stabilization, and I don't know why that's been held up.
> >
>
> Agreed, but if it isn't stable then it isn't fixed.

Bug 338855 was just filed, handling stabilization for g-s and v-s.
--
Alex Legler <a3li@gentoo.org>
Gentoo Security/Ruby

Greetings,

So what's the conclusion on what happened with bug 337645? What can we
learn from here? That everything went just fine and according to plan?
That hardly seems like a realistic assessment. If we ignore mistakes
instead of learning from them, we are doomed to repeat them.

Regarding the assessment that has been made of this exploit somehow not
being "important", I couldn't disagree more. We're not all just running
desktops at home here, clicking on random executables. There are those
of us who run Gentoo Hardened on multiuser servers, which is the first
place I would consider running Hardened to begin with. Some of these
servers simply need to support 32-bit execution as well as 64-bit,
because the users need it to do their job. It's not such a strange
requirement in the real world; just think of development environments,
for example. Yes, or proprietary tools -- as much as I would love to
have a world entirely based on FOSS, unfortunately I don't have the
luxury of pretending that proprietary software doesn't exist, or that
it's not necessary in some cases, especially when I am supporting other
users who may need it.

It would be wonderful if malicious executables and malicious users could
never exist; the world would certainly be a happier place. Sadly, things
don't work that way, and that's why we have to worry about security in
the software industry to begin with. I would think it shouldn't have to
be explained that the ability for a local user to get root access by
compiling a piece of code and running it is a very bad thing. Your own
vulnerability treatment policy ranks it as A1 level, and correctly so in
my opinion.

Whether the fix was in tree within 4 hours or 4 days is irrelevant; the
fact is that a stable release didn't come out until a week later (in the
hardened version), when a trivial fix was already in tree, is very bad.
The fact that the general-public gentoo-sources remained vulnerable to
the exploit for 3 weeks is appalling.

Yes, there were mitigating security measures *in the hardened sources*,
which broke the *publicly known* exploits. It is still hiding one's head
in the sand to assume that just because the symbols are hidden, no one
else might come up with a cleverer exploit. Just think back to the
original flaw from 2+ years ago, and the exploit techniques that scanned
the memory looking for function signatures, for example. Yes, Grsecurity
provides an additional layer of defense in depth, *on hardened-sources*,
but that's it: it's only a stop-gap measure, a best effort in case a
vulnerability appears, to minimize the possibility for damage until the
vulnerability gets fixed. The point is that a very serious vulnerability
is there: certainly the Gentoo Security team are the first people who
would want it fixed.

Then there's the case of gentoo-sources, which has no Grsecurity and
therefore no mitigation whatsoever. Surely the users of the normal
kernel (including myself on my home PC, for example) aren't mistaken for
expecting that the high standard of quality, stability and security
provided by Gentoo will still apply to them...

I am and have always been an avid defender of Gentoo, for I agree with
the philosophy behind it, and admire the consistent high quality work of
the developers and maintainers, the second-to-none documentation, the
practicality of the rolling release, the unrivaled flexibility of
Portage, and I could go on. But when we have a local root exploit, with
a known fix, that clings on for this long, it's just sad. It hurts the
Gentoo image, and leaves it wide open to unfair blanket criticism like
we saw near the end of the 337645 bug thread. Every major distribution
had a fix for this very important problem within a couple of days; it
tarnishes the otherwise magnificent security efforts undertaken by the
Gentoo dev team that 3 weeks later the exploit was still there.

So, concrete question:

The fix could have easily been backported to the stable kernels, and a
new intermediary stable release could have been launched. I even went to
the trouble of submitting a patch, the upstream fix made against
2.6.32-hardened-r9, which was the stable hardened version at the time
(not to mention stable gentoo-sources, which doesn't even have any of
the mitigation measures provided by Grsecurity). It was an obvious patch
(now that the bug was found) fixing a 2-year old regression, altering 8
lines of code, most of them in the same manner. As noted earlier, the
fix was in Portage something like the next day after the CVE came out.
So why then was it not backported and stabilized, instead of waiting for
stabilization of a new version, which had many new features and
associated bugs? And why was it not merged with gentoo-sources in the
same manner?

Regards,
Israel G. Lugo

Excerpts from Israel G. Lugo's message of Sun Oct 17 15:59:15 +0200 2010:
> So what's the conclusion on what happened with bug 337645? What can we
> learn from here? That everything went just fine and according to plan?

Are you kidding?

> That hardly seems like a realistic assessment. If we ignore mistakes
> instead of learning from them, we are doomed to repeat them.
>
> Regarding the assessment that has been made of this exploit somehow not
> being "important", I couldn't disagree more.

Er, who here assessed the issue to be not "important"?

> We're not all just running
> desktops at home here, clicking on random executables. There are those
> of us who run Gentoo Hardened on multiuser servers, which is the first
> place I would consider running Hardened to begin with. Some of these
> servers simply need to support 32-bit execution as well as 64-bit,
> because the users need it to do their job. It's not such a strange
> requirement in the real world; just think of development environments,
> for example. Yes, or proprietary tools -- as much as I would love to
> have a world entirely based on FOSS, unfortunately I don't have the
> luxury of pretending that proprietary software doesn't exist, or that
> it's not necessary in some cases, especially when I am supporting other
> users who may need it.
>
> It would be wonderful if malicious executables and malicious users could
> never exist; the world would certainly be a happier place. Sadly, things
> don't work that way, and that's why we have to worry about security in
> the software industry to begin with. I would think it shouldn't have to
> be explained that the ability for a local user to get root access by
> compiling a piece of code and running it is a very bad thing.

You seem to be assuming that we (=Gentoo people) live in a dream world
and still need to be told about hackers and the need for IA32-emul code
even today. We don't. So please stop discussing such banalities we are
all well aware of and instead get to the point.

> Your own
> vulnerability treatment policy ranks it as A1 level, and correctly so in
> my opinion.
>

Besides the fact that the VTP is still not applicable to Kernel
packages, we now do seem to rank things correctly? Can you please make
up your mind?

> Whether the fix was in tree within 4 hours or 4 days is irrelevant; the
> fact is that a stable release didn't come out until a week later (in the
> hardened version), when a trivial fix was already in tree, is very bad.
> The fact that the general-public gentoo-sources remained vulnerable to
> the exploit for 3 weeks is appalling.
>
> Yes, there were mitigating security measures *in the hardened sources*,
> which broke the *publicly known* exploits. It is still hiding one's head
> in the sand to assume that just because the symbols are hidden, no one
> else might come up with a cleverer exploit. Just think back to the
> original flaw from 2+ years ago, and the exploit techniques that scanned
> the memory looking for function signatures, for example. Yes, Grsecurity
> provides an additional layer of defense in depth, *on hardened-sources*,
> but that's it: it's only a stop-gap measure, a best effort in case a
> vulnerability appears, to minimize the possibility for damage until the
> vulnerability gets fixed. The point is that a very serious vulnerability
> is there: certainly the Gentoo Security team are the first people who
> would want it fixed.

Again: Stop with the banalities.
No one claims that workarounds or mitigations are equivalent to a fix in
the source code.
Yes, the Security team are the first ones to want things (properly)
fixed, however we cannot just jump in and fix things. We have to work
with maintainers and arch teams. This issue clearly has shown that there
are frictional losses that need to be addressed.

> Then there's the case of gentoo-sources, which has no Grsecurity and
> therefore no mitigation whatsoever. Surely the users of the normal
> kernel (including myself on my home PC, for example) aren't mistaken for
> expecting that the high standard of quality, stability and security
> provided by Gentoo will still apply to them...
>
> I am and have always been an avid defender of Gentoo, for I agree with
> the philosophy behind it, and admire the consistent high quality work of
> the developers and maintainers, the second-to-none documentation, the
> practicality of the rolling release, the unrivaled flexibility of
> Portage, and I could go on. But when we have a local root exploit, with
> a known fix, that clings on for this long, it's just sad. It hurts the
> Gentoo image, and leaves it wide open to unfair blanket criticism like
> we saw near the end of the 337645 bug thread. Every major distribution
> had a fix for this very important problem within a couple of days; it
> tarnishes the otherwise magnificent security efforts undertaken by the
> Gentoo dev team that 3 weeks later the exploit was still there.

..which is basically the same thing you said on the bug. Can we get to
the poi..

>
> So, concrete question:

..nt now? Finally! Thank God.

>
> The fix could have easily been backported to the stable kernels, and a
> new intermediary stable release could have been launched. I even went to
> the trouble of submitting a patch, the upstream fix made against
> 2.6.32-hardened-r9, which was the stable hardened version at the time
> (not to mention stable gentoo-sources, which doesn't even have any of
> the mitigation measures provided by Grsecurity). It was an obvious patch
> (now that the bug was found) fixing a 2-year old regression, altering 8
> lines of code, most of them in the same manner. As noted earlier, the
> fix was in Portage something like the next day after the CVE came out.
> So why then was it not backported and stabilized, instead of waiting for
> stabilization of a new version, which had many new features and
> associated bugs? And why was it not merged with gentoo-sources in the
> same manner?

As the maintainers are primarily responsible for providing unaffected
ebuilds, you'll need to ask the kernel team for exact reasons.

As for the 'lessons learned' part, it does seem that the resolution of
this issue was very much focused on updating to newer versions while
forgetting about alternatives (i.e. backporting). That's something that
we can do better next time.

Communication could be improved, too. Reading bug 337645, there was
little contribution from the Kernel team with regards to stabilizing
unaffected sources. That could be improved by noting progress or
problems. Then again my team (Security) could have asked for such info
earlier when there was no update.
At the same time I also have to address the users who made useless
comments on that bug. Of course, with a better resolution strategy on
our part, such things are less likely to appear, but people need to
remember that a bug tracker is not the place for random rants (which
again drain the motivation of the volunteers involved in the issue).

Something I'd like to see as a productive outcome of this issue, is a
little Kernel bug best-practices document for improving the interaction
of the various teams.

Yours most sincerely,
Alex
--
Alex Legler <a3li@gentoo.org>
Gentoo Security/Ruby

I just wanted to clarify that my intent is not to complain, or to imply
that Gentoo devs aren't working hard enough, or that "Gentoo sucks" or
anything of the sort; I may have transmitted the wrong impression in my
previous email, for which I apologize. It is precisely because I
appreciate the dedicated effort of all the Gentoo volunteers, and the
high standards of quality which this distribution has always maintained,
that I would hate to see such efforts subjected to unfair criticism due
to a few isolated procedural problems.

The problem here wasn't, in my opinion, a lack of effort by anyone; as
noted before, the fix was in the tree within hours, or within a day. The
thing is, for whatever reason, the fix only came out a contextually very
long time after that. This is what concerns me, and others I'm sure.
It's very bad for the image of Gentoo, it gives the impression that you
don't take security as seriously as others, and this -- at least in my
view -- couldn't be farther from the truth. The main reason I use Gentoo
Hardened on critical servers is precisely due to the effort and
commitment put in by the security team at every level, from the kernel
and toolchain to the user packages themselves. Nevertheless, the fact
remains that anyone using Hardened was left open to a vulnerability for
a longer time than would have been necessary, given that the fix was
already implemented within the tree. Also, I am concerned for the users
of normal gentoo-sources, who were vulnerable for a very extended period
of time.

I believe that it would be a positive thing to analyze what happened,
and try to learn from it so that next time things go better. I would
submit that sometimes, a lengthy procedure may get in the way of getting
things done; or at least, that the established procedure should be more
flexible to account for these cases.

Regards,
Israel

On 10/17/2010 02:59 PM, Israel G. Lugo wrote:
> Greetings,
>
> So what's the conclusion on what happened with bug 337645? What can we
> learn from here? That everything went just fine and according to plan?
> That hardly seems like a realistic assessment. If we ignore mistakes
> instead of learning from them, we are doomed to repeat them.
>
> [...]

On 10/17/2010 11:51 AM, Alex Legler wrote:
> Excerpts from Israel G. Lugo's message of Sun Oct 17 15:59:15 +0200 2010:
>> Your own
>> vulnerability treatment policy ranks it as A1 level, and correctly so in
>> my opinion.
>>
>
> Besides the fact that the VTP is still not applicable to Kernel
> packages, we now do seem to rank things correctly? Can you please make
> up your mind?

The VTP States:

Currently kernels are not covered by the GLSA release process.
Vulnerabilities must still be reported and will be fixed, but no GLSA
will be issued when everything is solved.

To me that sounds like we still do everything the same, but we don't
publish a GLSA when we're done. It is then suggested that the reason
for the policy is that we have shortcomings in our current tools.

It does not sound to me like we just take care of kernel root exploits
whenever we get around to it, as a matter of policy.

If we do not officially support security updates on the kernel the
webpage should be updated to explicitly state so. Of course, it would
be better to actually have a sane security policy on the kernel, even if
we can't make official GLSAs. Also, tool problems or not there is no
reason we couldn't grant somebody rights to post to the GLSA mailing
list so that they could send out manual notifications when serious
kernel vulnerabilities are fixed.

As it stands, a new gentoo-sources version was fixed, but the vulnerable
versions remain in portage and are not masked, so even users who run
emerge world often might not have realized that the need to upgrade
their kernels (as in build and install them and not just have the
sources lying around). I know I usually take my time on kernel upgrades
waiting for opportune moments, unless there is a serious issue with the
version I'm running.

This isn't about blame-finding/etc. It would be nice to look at the
overall process going forward and try to improve it. That starts by
admitting that next time we'd like to do better.

Rich

On 10/17/2010 04:51 PM, Alex Legler wrote:
> Er, who here assessed the issue to be not "important"?
>> [...]
> You seem to be assuming that we (=Gentoo people) live in a dream world
> and still need to be told about hackers and the need for IA32-emul code
> even today. We don't. So please stop discussing such banalities we are
> all well aware of and instead get to the point.
>

My remarks were not aimed at you (Alex) or the Gentoo security team in
particular. I was responding in general to several comments made both
in-list and on the bug. To name an example: "And the bug is not that
dangerous - except when you insist on running unsecure 32bit software on
a 64bit system." (Volker Armin Hemmann)


> No one claims that workarounds or mitigations are equivalent to a fix in
> the source code.

Through the process of the bug report and discussions on -security and
-hardened, it appeared to be the point of view of some people that this
wasn't a big deal, because of the Grsecurity workarounds that were
present. I was trying to argue that relying on that is dangerous and,
more importantly, that gentoo-sources never benefited from such workarounds.


> Yes, the Security team are the first ones to want things (properly)
> fixed, however we cannot just jump in and fix things. We have to work
> with maintainers and arch teams. This issue clearly has shown that there
> are frictional losses that need to be addressed.

That was my intended point.

As I tried to clarify on my second email (which I sent before receiving
your reply, and perhaps you only received it after replying), I am *not*
suggesting that Gentoo devs are incompetent, negligent, or otherwise
unqualified. My intentions were constructive; I've actually defended
Gentoo against several people over this, and it's frustrating to see
others making unfortunate public comments such as "Gentoo sucks, I'm
glad I changed to another distro", when the situation could have been
avoided altogether.

My concern is precisely because I appreciate the effort you (Gentoo
devs) place in your work, and this situation has created problems where
there shouldn't be. As a result, users were left vulnerable to a serious
exploit for a long time (as we all know, even 1 or 2 days may be more
than enough for a multiuser server to be compromised), something which
you obviously strive against. Also, the Gentoo public image of security
will have suffered by comparison, at least among those who follow CVEs
and security announcements. I'm sure you must be the first people who
don't want this to happen, so it's important to try and improve things
for next time.


> [several constructive remarks about improvements for the future]

I agree with everything you said here, and am glad to see something
positive come out of all this.

I agree that communication is key, and everyone would benefit from an
increase there. There are times when the established procedures may be
too rigid (dev makes a commit, makes formal request for testing, QA
tests, makes formal stabilization happen, etc.), especially in matters
of urgency such as with security response. Improved communication can
make things happen more smoothly, and perhaps it might not be a bad idea
for the necessary people to meet and try to reach a simpler process,
still valid but requiring a little less bureaucracy, for these kinds of
situations. In that respect, a kernel bug best-practices document might
very well help.

Best regards,
Israel

Alex Legler <a3li@gentoo.org> writes:

> As the maintainers are primarily responsible for providing unaffected
> ebuilds, you'll need to ask the kernel team for exact reasons.

In all of this we must bear in mind that the kernel team is one of the
fastest to respond to upstream changes, at least with ~arch. New
vanilla-sources are normally in the tree within hours of upstream
releasing them and gentoo-sources are seldom far behind.

On Sun, 2010-10-17 at 19:58 +0100, Graham Murray wrote:
> Alex Legler <a3li@gentoo.org> writes:
>
> > As the maintainers are primarily responsible for providing unaffected
> > ebuilds, you'll need to ask the kernel team for exact reasons.
>
> In all of this we must bear in mind that the kernel team is one of the
> fastest to respond to upstream changes, at least with ~arch. New
> vanilla-sources are normally in the tree within hours of upstream
> releasing them and gentoo-sources are seldom far behind.
>

That is quite a bold statement to make considering
vanilla-sources-2.6.36_rc7 was never put in ~arch and rc8 has been out
from upstream for 3 days now. I imagine there is a reason for that
though?