Mailing List Archive

Considering dropping the hardened toolchain
I really never wanted to send a mail like this but I don't know what
else to do. ;/

Due to low positive feedback and user input I'm considering dropping the
hardened toolchain and retiring from non commercial proactive security
efforts. ie pulling the patches developed that brings you
pie/ssp/relro/now/etc.. Doing otherwise I feel would leave you limping
along, which I feel would be a disservice to everybody. This motivation
stems from bugs that are going unresolved or being improperly fixed and
or filtered. I'm not getting the help I/we need from my own team or the
user community. This is becoming an overwhelming/stressful job for one
person to handle alone on with a user-base this large.

If you wish to see the hardened toolchain continue YOU need to step up
in the next few weeks and offer help (ie we need 2-3 really good
people).

Mail me off list for more details if your interested in helping.
A fair to strong understanding is needed of entire toolchain process.
Desired is somebody(s) that preferably also understands ELF and multi
arch assembly that wishes to see the solution developed to it's fullest
as per the goals outlined in the PaX documentation.

--
Ned Ludd <solar@gentoo.org>
Gentoo (hardened,security,infrastructure,embedded,toolchain) Developer
Re: [gentoo-dev] Considering dropping the hardened toolchain (A Quantitive Approach) [ In reply to ]
Ned Ludd wrote:

> I really never wanted to send a mail like this but I don't know what
> else to do. ;/
>
> Due to low positive feedback and user input I'm considering dropping
> the hardened toolchain

Sat Sep 18 16:14:04 CEST 2004

Subject: A Quantitive Approach

Dear Solar,

it does not surprise me that you are proposing a "change for the better"
with
dismissing the hardened toolchain.

But you should think about the impact.

Does the solution itself offer such poor quality that it should not/cannot
survive?

I think you are forgetting about one more important fact than unfixed bugs.
I know that these bugs are mainly existing due to low manpower,
which is also majorily my fault because i promised to commit to
the solution in the past and did not deliver a considerable amount of
time and
work to fulfill my responsibilities at the Gentoo Hardened team.

But before you are starting to put solutions to an end of life, i would like
you to consider the benefits and the misunderstandings from a more
"objective"
point of view:

When a beginner starts using the hardened toolchain (which is not what
it was
designed for), he or she will immediately suffer from deep impacts of our
"evil" compiler modifications.

We create documentation as easy as possible to make the "entry point" to the
solution low. But that does not mean an ape with a joystick can fly a
rocket.

You have to follow me on the thought that if someone with a professional
attitude
will see bugs and make up his/her own mind about those bugs with a
certain degree
of understanding what is going on, he or she is going to find the cause and
report it back to us without stupid comments like "your shit does not work"
(which is not a quote from a bug but what comes into my mind when i read
many
bugs, and yes, i read them)

We give the tools- the people use it. Thats the game, that is how it
always has been.
We make things easy. But we cannot make things so foolproof that you cannot
cut your throat with it.

With compilers and toolchains its different than with media players or name
servers. Either a media player works or it plays videos too fast or does
nothing- like my mplayer does sometimes. People open bugs about mplayer.

What people expect from mplayer is that it plays video.

What security people expect from a hardened toolchain is that it compiles
software in a defined and security-oriented fashion.

Peter Mazinger is working very hard and he is successfully delivering
the needed compiler modifications.
The PaX patch, provided by the PaX team, is the dark eminence (i.e. kernel)
behind the scenes that makes the things work as expected (and defined).

What are the reasons that you want to postpone those two cornerstones of
basic
security improvements over default userland and default linux kernel?

Do you call it a "failed experiment"? No.

You are just telling people to get up and support it or you will cancel it.

As a project manager for the toolchain you can order your developers to
get up
to date with open patches and take care of the user base. You are right.

But what user base and feedback do you expect?

The people using it do not file bugs, because they understand that

a) using bleeding edge software may/can break shit

b) using hardened toolchain to emerge (from a security perspective)
unneeded stuff can break more shit

Who said that the hardened toolchain can be blamed for people using it,
compiling
their desktop machines and after that joining the #channel and/or
opening bugs
without a slight degree (or as a matter of fact: interest) in
understanding or even
WORKING with the solution.

Normally, people open bugs to get something fixed.
But face it: mplayer will not be fixed.

Do you think i am proud of the bugs? You know that i am not and i know that
it hurts you to see our stuff getting so "dissed".

Trust me, they all want "security out of the box" and maybe our
documentation is misleading
and trying to "promise" that in a way. But it is not wrong, the
documentation.

Lets go somewhere else because i was discussing "low entry level" and
USE flags
for ease of customer experience already above.

We started this project as pioneers. Now we are experienced users of
our own
solution.

When people were asking me for "full scope" coverage of the hardened
solution
for the full Gentoo userland, it took me a very long time to think about
possible ways for making "transparent" defuse switches.

Then i said: it can be done, but only with my tricks. My tricks have
not been
accepted by the developers (see discussions on public lists) and i
agreed that
it was not possible without losing important factors like "visibility" of
changes and "reproducability" of toolchain and compilers involved.

But those problems arising now were created by a simple mathematical
function:

Ignorance * People * Time / Commitment to the Solution = number of open
bugs.

If the commitment to the solution by the people opening bugs were high
enough
to "keep it going", we would not have to bother about too much "ignorance
bugs". I know this sounds harsh.

And it does not mean to "insult" our customers and users. But face it.

If ignorance were near zero, this equation would hold our bug level nice
and low.
But it does not.

If the number of people were low, it would also be nice and low.

If the time had not shown the major deficiencies (which the solution clearly
has), it would also not have become such an imminent point of "turning
around"
now.

You are asking what to do. I think that is just fair because it shows your
community spirit, your true efforts and your heart for the solution and the
Gentoo team.

You don't want anybody to blame us for letting people wrestle in the mud
of a
"broken" hardened toolchain "solution".

But as is said before: this is a team of volunteers- and volunteers means:
people "sacrifice" their free time for the project and the ongoing
quality of
the project.

If quality is low, it does not mean that the developers are stubborn fools.

To give another mathematical example:

free time * commitment = quality.

If free time is low (which has been the case in my personal situation), the
quality cannot be high.

And i also know that you never suspected the commitment of team members
to be
under a needed level that the solution can "go on".

When you say: "dropping the hardened toolchain", to me it sounds like "stop
riding a dead horse".

But, in my eyes, you are underestimating the negative impact of that
decision on people
successfully using the solution.

Do you need success stories for letting it continue?
Do you need mails of people that tell you: good job, things broke left and
right of me, but i am a proud owner of a hardened gcc.

You and me know that you will never get such mails.

People do not bother. They only bitch and whine when it breaks and the
house
is burning down because they have been playing with the dynamite and at the
same time forgot the lit cigarette in their mouth.

This is what you have been doing in the last weeks: collecting examples of
"bad feedback".

And you can declare the "low positive" feedback as that any "high positive"
feedback does not exist. Of course it does not- reason is above.


Does the "large" userbase really "ad hoc" enable the hardened USE flag and
then loads of workstations go down the drain and bugs.gentoo.org suffers
major
headache from people opening hundreds of duplicate bug requests?

If you are comparing the negative impact of hardened gcc to the "mistakes"
made by some glibc and gcc updates in the past, i cannot see a reason why a
"large user base" that is reluctant to work with us and find solutions
should
be a reason for us to "retreat" from our positions of zero tolerance towards
high security systems- this includes kernel, compilers and other tools like
full runtime bounds checking (remember that the libmudflap source again
did not
make it to mainstream gcc).

To be honest, your approach is too simple.

You say: it does not work, lets drop it

I say: WORKSFORME

Peter Mazinger is putting big efforts in creating patches.
The PaX kernel can go to its full strength on a Gentoo Hardened system.


It takes people who think to implement security.

We give the tools.

People use tools. Did you know that you can cut your throat with a
screwdriver? I do not remember seeing warnings on screwdriver not to
cut your
throat while operating it.

We cannot be personally held responsible (or liable) for the ignorance of
people who want "security out of the box".

See you at the bitter end,

Alex

(And yes, this all could have been said with less than one quarter of the
character count, but i was in the mood of defensive arguments ;-)



--
gentoo-hardened@gentoo.org mailing list
Re: [gentoo-dev] Considering dropping the hardened toolchain (A Quantitive Approach) [ In reply to ]
Alex know I love and respect your opinion but with you being mostly gone
and my own business starting is suffering due to the large amount of
time the project takes I'm not seeing a whole lot of alternatives unless
we get some help from some hard hitters.

Yes we could leave the patches in and _hope_ that things continue to
work. But who is going to watch the bugs and verify that something is
not a flaw in our own design? how do we deal with fundamental design
flaws with upstream security patches when they don't care? (I know you
know this one all to well) example
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=132149

Where is our core documentation? Where is the comparative analysis of
security measures? Who is going to maintain our appropriate profiles?
Keeping up with virtuals is nearly a task in and of itself. Rolling
stages and working towards things like livecd's. These are fundamental
things that have to be done on a cycles that follow gentoo mainline. how
about aiming for getting some of these solutions to go gentoo mainline.
All suids and daemons for example should really be compiled the way we
do. But that's an uphill battle that I'm pretty sure I don't want to try
to tackle alone.

I see things like users that have completely uninstalled the
hardened-toolchain to work around small bugs when they don't have to. I
can't comment on every bug to say here is another way you can get the
same end goal without them having to dismiss the solution completely, be
that their own ignorance or ours it's a disservice to our community do
nothing.

Being on this team means more than just watching the "hardened" bugs
that get routed to us and replying to emails once every few weeks, one
has to watch all toolchain bugs and look for how things correlate to
each other and determine the right thing to do.

Other distro's are wanting to adopt similar to the same solution but
those guys need to be helped so they don't make fatal mistakes. Yes it's
a very good sign when others want to adopt it. But that's also time
consuming. Ask yourself should we keep our dear precious all to
ourselves or dedicate some time and effort to other projects of the
world. I think we should as most of these designs are being developed in
house and we by far have the most experience.
See http://d-sbd.alioth.debian.org/ for more details.

How are we going to handle the other arches.
ppc/ppc64 These two arches could be added to mix relatively easy. But
**** we can't even get one of those teams to test simple kernel patches
that were developed especially for them after multiple requests.

Peter has many questions that he needs definitive answers to. With those
questions going unanswered and him being a key player these days in your
absence we could be potentially introducing more bugs than need be.
He is/was happy with his own homegrown build system and does not really
need to support us or our efforts.

You want to help pappy? Spend your time being proactive vs defensive.
Help me train some people so the workload is reduced on us and our time
can be better spent focusing on making the next steps with more ease.

After planting the initial seeds one day and seeing things being
maintained properly long term I'd like to be just a user ya know.

On Sat, 2004-09-18 at 11:54, Alexander Gabert wrote:
> Ned Ludd wrote:
>
> > I really never wanted to send a mail like this but I don't know what
> > else to do. ;/
> >
> > Due to low positive feedback and user input I'm considering dropping
> > the hardened toolchain
>
> Sat Sep 18 16:14:04 CEST 2004
>
> Subject: A Quantitive Approach
>
> Dear Solar,
>
> it does not surprise me that you are proposing a "change for the better"
> with
> dismissing the hardened toolchain.
>
> But you should think about the impact.
>
> Does the solution itself offer such poor quality that it should not/cannot
> survive?
>
> I think you are forgetting about one more important fact than unfixed bugs.
> I know that these bugs are mainly existing due to low manpower,
> which is also majorily my fault because i promised to commit to
> the solution in the past and did not deliver a considerable amount of
> time and
> work to fulfill my responsibilities at the Gentoo Hardened team.
>
> But before you are starting to put solutions to an end of life, i would like
> you to consider the benefits and the misunderstandings from a more
> "objective"
> point of view:
>
> When a beginner starts using the hardened toolchain (which is not what
> it was
> designed for), he or she will immediately suffer from deep impacts of our
> "evil" compiler modifications.
>
> We create documentation as easy as possible to make the "entry point" to the
> solution low. But that does not mean an ape with a joystick can fly a
> rocket.
>
> You have to follow me on the thought that if someone with a professional
> attitude
> will see bugs and make up his/her own mind about those bugs with a
> certain degree
> of understanding what is going on, he or she is going to find the cause and
> report it back to us without stupid comments like "your shit does not work"
> (which is not a quote from a bug but what comes into my mind when i read
> many
> bugs, and yes, i read them)
>
> We give the tools- the people use it. Thats the game, that is how it
> always has been.
> We make things easy. But we cannot make things so foolproof that you cannot
> cut your throat with it.
>
> With compilers and toolchains its different than with media players or name
> servers. Either a media player works or it plays videos too fast or does
> nothing- like my mplayer does sometimes. People open bugs about mplayer.
>
> What people expect from mplayer is that it plays video.
>
> What security people expect from a hardened toolchain is that it compiles
> software in a defined and security-oriented fashion.
>
> Peter Mazinger is working very hard and he is successfully delivering
> the needed compiler modifications.
> The PaX patch, provided by the PaX team, is the dark eminence (i.e. kernel)
> behind the scenes that makes the things work as expected (and defined).
>
> What are the reasons that you want to postpone those two cornerstones of
> basic
> security improvements over default userland and default linux kernel?
>
> Do you call it a "failed experiment"? No.
>
> You are just telling people to get up and support it or you will cancel it.
>
> As a project manager for the toolchain you can order your developers to
> get up
> to date with open patches and take care of the user base. You are right.
>
> But what user base and feedback do you expect?
>
> The people using it do not file bugs, because they understand that
>
> a) using bleeding edge software may/can break shit
>
> b) using hardened toolchain to emerge (from a security perspective)
> unneeded stuff can break more shit
>
> Who said that the hardened toolchain can be blamed for people using it,
> compiling
> their desktop machines and after that joining the #channel and/or
> opening bugs
> without a slight degree (or as a matter of fact: interest) in
> understanding or even
> WORKING with the solution.
>
> Normally, people open bugs to get something fixed.
> But face it: mplayer will not be fixed.
>
> Do you think i am proud of the bugs? You know that i am not and i know that
> it hurts you to see our stuff getting so "dissed".
>
> Trust me, they all want "security out of the box" and maybe our
> documentation is misleading
> and trying to "promise" that in a way. But it is not wrong, the
> documentation.
>
> Lets go somewhere else because i was discussing "low entry level" and
> USE flags
> for ease of customer experience already above.
>
> We started this project as pioneers. Now we are experienced users of
> our own
> solution.
>
> When people were asking me for "full scope" coverage of the hardened
> solution
> for the full Gentoo userland, it took me a very long time to think about
> possible ways for making "transparent" defuse switches.
>
> Then i said: it can be done, but only with my tricks. My tricks have
> not been
> accepted by the developers (see discussions on public lists) and i
> agreed that
> it was not possible without losing important factors like "visibility" of
> changes and "reproducability" of toolchain and compilers involved.
>
> But those problems arising now were created by a simple mathematical
> function:
>
> Ignorance * People * Time / Commitment to the Solution = number of open
> bugs.
>
> If the commitment to the solution by the people opening bugs were high
> enough
> to "keep it going", we would not have to bother about too much "ignorance
> bugs". I know this sounds harsh.
>
> And it does not mean to "insult" our customers and users. But face it.
>
> If ignorance were near zero, this equation would hold our bug level nice
> and low.
> But it does not.
>
> If the number of people were low, it would also be nice and low.
>
> If the time had not shown the major deficiencies (which the solution clearly
> has), it would also not have become such an imminent point of "turning
> around"
> now.
>
> You are asking what to do. I think that is just fair because it shows your
> community spirit, your true efforts and your heart for the solution and the
> Gentoo team.
>
> You don't want anybody to blame us for letting people wrestle in the mud
> of a
> "broken" hardened toolchain "solution".
>
> But as is said before: this is a team of volunteers- and volunteers means:
> people "sacrifice" their free time for the project and the ongoing
> quality of
> the project.
>
> If quality is low, it does not mean that the developers are stubborn fools.
>
> To give another mathematical example:
>
> free time * commitment = quality.
>
> If free time is low (which has been the case in my personal situation), the
> quality cannot be high.
>
> And i also know that you never suspected the commitment of team members
> to be
> under a needed level that the solution can "go on".
>
> When you say: "dropping the hardened toolchain", to me it sounds like "stop
> riding a dead horse".
>
> But, in my eyes, you are underestimating the negative impact of that
> decision on people
> successfully using the solution.
>
> Do you need success stories for letting it continue?
> Do you need mails of people that tell you: good job, things broke left and
> right of me, but i am a proud owner of a hardened gcc.
>
> You and me know that you will never get such mails.
>
> People do not bother. They only bitch and whine when it breaks and the
> house
> is burning down because they have been playing with the dynamite and at the
> same time forgot the lit cigarette in their mouth.
>
> This is what you have been doing in the last weeks: collecting examples of
> "bad feedback".
>
> And you can declare the "low positive" feedback as that any "high positive"
> feedback does not exist. Of course it does not- reason is above.
>
>
> Does the "large" userbase really "ad hoc" enable the hardened USE flag and
> then loads of workstations go down the drain and bugs.gentoo.org suffers
> major
> headache from people opening hundreds of duplicate bug requests?
>
> If you are comparing the negative impact of hardened gcc to the "mistakes"
> made by some glibc and gcc updates in the past, i cannot see a reason why a
> "large user base" that is reluctant to work with us and find solutions
> should
> be a reason for us to "retreat" from our positions of zero tolerance towards
> high security systems- this includes kernel, compilers and other tools like
> full runtime bounds checking (remember that the libmudflap source again
> did not
> make it to mainstream gcc).
>
> To be honest, your approach is too simple.
>
> You say: it does not work, lets drop it
>
> I say: WORKSFORME
>
> Peter Mazinger is putting big efforts in creating patches.
> The PaX kernel can go to its full strength on a Gentoo Hardened system.
>
>
> It takes people who think to implement security.
>
> We give the tools.
>
> People use tools. Did you know that you can cut your throat with a
> screwdriver? I do not remember seeing warnings on screwdriver not to
> cut your
> throat while operating it.
>
> We cannot be personally held responsible (or liable) for the ignorance of
> people who want "security out of the box".
>
> See you at the bitter end,
>
> Alex
>
> (And yes, this all could have been said with less than one quarter of the
> character count, but i was in the mood of defensive arguments ;-)
--
Ned Ludd <solar@gentoo.org>
Gentoo (hardened,security,infrastructure,embedded,toolchain) Developer
Re: Considering dropping the hardened toolchain [ In reply to ]
Ned Ludd wrote:

>I really never wanted to send a mail like this but I don't know what
>else to do. ;/
>
>Due to low positive feedback and user input I'm considering dropping the
>hardened toolchain and retiring from non commercial proactive security
>efforts. ie pulling the patches developed that brings you
>pie/ssp/relro/now/etc..
>

I only get chance to scan the list occasionally, but have just seen
this. I can easily understand your position, but it will certainly be a
very sad day if we lose this functionality.

I don't have the time or abilities to step up to the plate myself, but I
guess the least I can offer is some interest from the user base that
this work is *very* appreciated.

As an aside, there seems to be something peculiarly difficult about
getting all these various new linux security enhancements into working
practice. I have been investigating vservers, and user-mode-linux
recently, and both seem to be rather hard to get working completely with
a modern 2.6 kernel. Hardened compiler is at least the bare min that
one can easily add to a basic install.

I hope you get the support that you need..

So long and thanks for all the patches...

Ed W

--
gentoo-hardened@gentoo.org mailing list