Mailing List Archive

On Sun, 26 Feb 2006 17:22:17 -0500 Mark Loeser <halcy0n@gentoo.org>
wrote:
| Yes, Gentoo is supposed to be fun, but we also have a responsibility
| to our users to ensure we are providing them with the best possible
| distro we can.

What, you mean the tree isn't someone's personal playground?

| * The QA team may also offer to fix obvious typos and similar minor
| issues, and silence from the package maintainers can be taken as
| agreement in such situations.

Should probably clarify that one. It's in there because there are some
situations where we find obvious typos (e.g. 'souce' instead of
'source' in DEPEND) and file a bug to alert the maintainer. If the
maintainer fixes it within a few days, there's no problem, but if not
there's no point in letting the bug sit there -- someone from QA should
be able to fix it themselves.

Equally, we don't want to just fix stuff without telling people that
they made a mistake, because then they're more likely to do it again.

| * The QA team will maintain a list of current "QA Standards". The
| list is not meant by any means to be a comprehensive document, but
| rather a dynamic document that will be updated as new problems are
| discovered.

Hrm, do we want to include the thing about the QA standards providing
rationale and explanations rather than hard rules here?

--
Ciaran McCreesh : Gentoo Developer (Wearer of the shiny hat)
Mail : ciaranm at gentoo.org
Web : http://dev.gentoo.org/~ciaranm

My personal opinion here is that a _LOT_ of this should be common sense.
But just to put in my two pennies..

On Sun, Feb 26, 2006 at 05:22:17PM -0500, Mark Loeser <halcy0n@gentoo.org> wrote:
> * The QA team's purpose is to provide cross-herd assistance in keeping
> the tree in a good state. This is done primarily by finding and pointing
> out issues to maintainers and, where necessary, taking direct action.

Please clarify "neccessary". I don't want to see repeat occurances of
non-issues bogging down real work. Also, please define around this a
clear and documented policy so when its enforced, its well defended.

> * The QA team may also offer to fix obvious typos and similar minor
> issues, and silence from the package maintainers can be taken as agreement in
> such situations.

I have no objections, on the understanding that there is a definitive
understanding of whats being changed and legitimate things aren't
accidentally replaced.

> * In case of emergency, or if package maintainers refuse to cooperate,
> the QA team may take action themselves to fix the problem.

This is part and parcel of your first point and should be included as
part of that. ie: definition of neccessary and surrounding policy.

> * In the event that a developer still insists that a package does not
> break QA standards, an appeal can be made at the next council meeting. The
> package should be dealt with per QA's request until such a time that a
> decision is made by the council.

as above.

> * In the case of disagreement on policy among QA members, the majority
> of established QA members must agree with the action.

Perhaps pushing it to an open forum on -dev/-core for consensus works
better here?

> * Just because a particular QA violation has yet to cause an issue does
> not change the fact that it is still a QA violation.

Is this a statement or a policy? I assume that if this is policy the
non-visible issue would go about appropriate scrutany, and in turn a
long-term solution made in the situation where it is not easily
resolvable/avoidable.

> * If a particular developer persistently causes breakage, the QA team
> may request that devrel re-evaluates that developer's commit rights.
> Evidence of past breakages will be presented with this request to
> devrel.

This is the case at the moment anyways isn't it? And this shouldn't be
in a QA capacity but as a herd or individual. Perhaps this is better
suited in a different proposal?

> * The QA team will maintain a list of current "QA Standards". The list
> is not meant by any means to be a comprehensive document, but rather a
> dynamic document that will be updated as new problems are discovered.
>

Can I suggest that such a document is also refered to by the policy
surrounding a violations resolution. Especially when considering a
violation which is not documented (and therefore can be fairly unknown)
so that violations not listed might be treated with more tact.

Thanks for presenting this to the list.
- John

--
Role: Gentoo Linux Kernel Lead
Gentoo Linux: http://www.gentoo.org
Public Key: gpg --recv-keys 9C745515
Key fingerprint: A0AF F3C8 D699 A05A EC5C 24F7 95AA 241D 9C74 5515

On Sun, Feb 26, 2006 at 10:58:35PM +0000, Ciaran McCreesh <ciaranm@gentoo.org> wrote:
> | * The QA team will maintain a list of current "QA Standards". The
> | list is not meant by any means to be a comprehensive document, but
> | rather a dynamic document that will be updated as new problems are
> | discovered.
>
> Hrm, do we want to include the thing about the QA standards providing
> rationale and explanations rather than hard rules here?

Either sounds fine to me as long as they are clear, simple, and where
possible brief.

--
Role: Gentoo Linux Kernel Lead
Gentoo Linux: http://www.gentoo.org
Public Key: gpg --recv-keys 9C745515
Key fingerprint: A0AF F3C8 D699 A05A EC5C 24F7 95AA 241D 9C74 5515

On Sun, 26 Feb 2006 23:11:21 +0000 johnm@gentoo.org wrote:
| On Sun, Feb 26, 2006 at 05:22:17PM -0500, Mark Loeser
| <halcy0n@gentoo.org> wrote:
| > * The QA team's purpose is to provide cross-herd assistance in
| > keeping the tree in a good state. This is done primarily by finding
| > and pointing out issues to maintainers and, where necessary, taking
| > direct action.
|
| Please clarify "neccessary". I don't want to see repeat occurances of
| non-issues bogging down real work. Also, please define around this a
| clear and documented policy so when its enforced, its well defended.

The problem is... It's impossible to document every single way in which
someone can screw up. For example, I wouldn't've thought to document
"you should not run mkdir in global scope", because I didn't think
anyone would be daft enough to do it. Policy *has* to rely upon the
basic assumption that developers won't do something crazy.

| > * The QA team may also offer to fix obvious typos and similar minor
| > issues, and silence from the package maintainers can be taken as
| > agreement in such situations.
|
| I have no objections, on the understanding that there is a definitive
| understanding of whats being changed and legitimate things aren't
| accidentally replaced.

Example of where this clause would be used, had said bug not been
fixed quickly anyway: bug #122902.

| > * In the case of disagreement on policy among QA members, the
| > majority of established QA members must agree with the action.
|
| Perhaps pushing it to an open forum on -dev/-core for consensus works
| better here?

The problem with that is, it usually ends up with too many pointless
comments from people saying how things could be fixed in the distant
future, or whining that it isn't explicitly forbidden by policy on
situations where the screwup was too weird to be documented previously.

| > * Just because a particular QA violation has yet to cause an issue
| > does not change the fact that it is still a QA violation.
|
| Is this a statement or a policy? I assume that if this is policy the
| non-visible issue would go about appropriate scrutany, and in turn a
| long-term solution made in the situation where it is not easily
| resolvable/avoidable.

This is to cover for situations where people claim that their screwups
are ok because no-one has yet reported it as broken.

--
Ciaran McCreesh : Gentoo Developer (Wearer of the shiny hat)
Mail : ciaranm at gentoo.org
Web : http://dev.gentoo.org/~ciaranm

On Sun, Feb 26, 2006 at 11:21:47PM +0000, Ciaran McCreesh <ciaranm@gentoo.org> wrote:
> On Sun, 26 Feb 2006 23:11:21 +0000 johnm@gentoo.org wrote:
> | On Sun, Feb 26, 2006 at 05:22:17PM -0500, Mark Loeser
> | <halcy0n@gentoo.org> wrote:
> | > * The QA team's purpose is to provide cross-herd assistance in
> | > keeping the tree in a good state. This is done primarily by finding
> | > and pointing out issues to maintainers and, where necessary, taking
> | > direct action.
> |
> | Please clarify "neccessary". I don't want to see repeat occurances of
> | non-issues bogging down real work. Also, please define around this a
> | clear and documented policy so when its enforced, its well defended.
>
> The problem is... It's impossible to document every single way in which
> someone can screw up. For example, I wouldn't've thought to document
> "you should not run mkdir in global scope", because I didn't think
> anyone would be daft enough to do it. Policy *has* to rely upon the
> basic assumption that developers won't do something crazy.

yeah, thats totally understandable. Its a best-efforts thing. I just
don't want neccessary to be deemed true for something which has an
arguable point with technical merit. Blatent mkdir-esque madness would
be more black than white, and I'd hope for this to try and sanitise the
gray :)

> | > * In the case of disagreement on policy among QA members, the
> | > majority of established QA members must agree with the action.
> |
> | Perhaps pushing it to an open forum on -dev/-core for consensus works
> | better here?
>
> The problem with that is, it usually ends up with too many pointless
> comments from people saying how things could be fixed in the distant
> future, or whining that it isn't explicitly forbidden by policy on
> situations where the screwup was too weird to be documented previously.

This is very much a case-by-case thing. I still feel the debate should
be better answered outside of conflicting qa members.

> | > * Just because a particular QA violation has yet to cause an issue
> | > does not change the fact that it is still a QA violation.
> |
> | Is this a statement or a policy? I assume that if this is policy the
> | non-visible issue would go about appropriate scrutany, and in turn a
> | long-term solution made in the situation where it is not easily
> | resolvable/avoidable.
>
> This is to cover for situations where people claim that their screwups
> are ok because no-one has yet reported it as broken.

I guess that also falls under the first point, based on the quality or
vagueness of the documention :)

- John

--
Role: Gentoo Linux Kernel Lead
Gentoo Linux: http://www.gentoo.org
Public Key: gpg --recv-keys 9C745515
Key fingerprint: A0AF F3C8 D699 A05A EC5C 24F7 95AA 241D 9C74 5515

Ciaran McCreesh wrote:
>
> | > * In the case of disagreement on policy among QA members, the
> | > majority of established QA members must agree with the action.
> |
> | Perhaps pushing it to an open forum on -dev/-core for consensus works
> | better here?
>
> The problem with that is, it usually ends up with too many pointless
> comments from people saying how things could be fixed in the distant
> future, or whining that it isn't explicitly forbidden by policy on
> situations where the screwup was too weird to be documented previously.
>

The rather blunt point here is to limit the power of the QA team itself.
The QA team decides what new policy to enforce, and when the QA team
can't agree "the majority of established QA members" must agree to
action. Which is in itself rather vague.

Perhaps "The majority of active QA members, where an active member is
designated as 'a QA member who responds to the corresponding mail to the
qa'". This would be similar to how the recent QA lead was chosen, mail
was sent, yay's and nay's were collected from those who cared, and then
the decision was made.

This is meant to prevent the case where the QA team ( or a subset; "the
established QA members" ) decides to make unilateral changes to the tree
( or large subset thereof ) without even necessarily talking to the
affected developers.

While you may not think that soliciting comments is useful ( and in some
limited cases I would agree with you ) giving people the opportunity to
comment also means you just covered your ass, in terms of people going
"where the hell did that come from?"

-Alec Warner

Hi Mark,

Thanks for posting this. I've a few suggestions to make (see below). I
support all the other points in your proposal.

On Sun, 2006-02-26 at 17:22 -0500, Mark Loeser wrote:
> * In case of emergency, or if package maintainers refuse to cooperate,
> the QA team may take action themselves to fix the problem.

I'd like to see this say

* In case of emergency, or after a failed appeal by the package
maintainer to the council, the QA team may take action themselves
to fix the problem, if the package maintainer(s) are unavailable
or refuse to co-operate.

That still gives the QA team the powers it needs for an enforcement
role, which is all that it needs.

> * In the event that a developer still insists that a package does not
> break QA standards, an appeal can be made at the next council meeting. The
> package should be dealt with per QA's request until such a time that a
> decision is made by the council.

I'd like to see an alternative wording:

* In the event that a developer still insists that a package does not
break QA standards, or that the QA standards are somehow defective,
an appeal can be made at the next council meeting.

If it is an emergency, then the QA team is still free to intervene
before the council meeting. But, where it isn't an emergency, there is
no need for immediate action, and so there should be no action until the
appeal has been heard.

The original wording is, imho, unfairly unbalanced. What will happen
under the original wording is that the QA team is free to force their
demands up front, and most developers will go with the flow rather than
take the trouble to take the issue to the council.

The QA team has no monopoly on what is right or wrong in Gentoo, and
neither do package maintainers. If there is a disagreement that leads
to an appeal, the onus should be on the QA team to have to present their
case to the council, as they are the ones pushing for change.

> * Just because a particular QA violation has yet to cause an issue does
> not change the fact that it is still a QA violation.

I'd support the above if the following statement was also included:

* QA standards, and the actions required to resolve them, must be
pragmatic and proportional to the impact on actual end-users of
Gentoo.

Gentoo would be best served by a QA team that spent its time tackling
issues that are urgent and important to end-users (quadrant 1 & 2
activities). A QA team that gets bogged down in the thick of thin
things is a symptom of a team that has become self-serving.

> * The QA team will maintain a list of current "QA Standards". The list
> is not meant by any means to be a comprehensive document, but rather a
> dynamic document that will be updated as new problems are discovered.

These QA standards are policy changes that affect the whole tree. The
QA team does *not* own QA policy - we all do, as contributors to Gentoo.
I'm asking the council to agree an adequate process for the approval of
QA standards by a wider group than just the QA team. Maybe changes
could be batched up into a GLEP, say once a quarter, with the option of
fast-tracking GLEPs in between for emergency QA policy changes? This
would allow the QA team to perform effectively, and have the added
benefit of ensuring that the wider developer community knew what was
coming in advance, and could "buy in" to the changes.

There's a secondary issue here for me. None of our tools are perfect;
we all have to work pragmatically within the capabilities of what we
have. Some of the real QA issues in the tree arise from the limitations
of our tools. I'd like to see the following incorporated into the QA
team's mandate.

* If the QA team wishes to push standards about specific aspects of
our tools, then those standards must be endorsed by the leads of
those tools.

Why? Because the QA team has no monopoly on what is right and wrong
with the tree. If any proposed QA standard cannot pass a simple litmus
test of the endorsement of the leads of the tools, how can it be fit for
enforcement against the wider development community? What is a package
maintainer to do? He/she can't go back to the tools team in question
and ask for a fix; all they will get is that the tools team in question
doesn't agree with the QA team, and doesn't support that particular
standard. Better to have QA standards that are strongly supported than
to have QA standards supported by just the one team.

I'd like to see the QA team's primary role stated as "educational"
first, then watchdog, then intervener in that order (emergencies
not-withstanding).

With that in mind, I suggest that the QA standards must include
educational information about the problem(s) that the standard
addresses, before they can be approved. This is in no way to challenge
the legitimacy of the agreed standards. It's to help all developers do
better QA on their own work (everyone does a better job if they
understand the reasons why). It also serves the dual purpose of helping
the next generation of QA testers when the current team members have
moved on. This could be Ciaran's opportunity to see TheDoc become an
"offical" document. There are few things worse than standards
autocratically and inflexibily applied long after the original
supporters have left, and no-one else can remember why they were
necessary in the first place.

There *may* be arguments about how much effort it takes to produce this
educational material, and I'm sympathetic to that. But for me, what it
boils down to is this: QA is something that we all need to do. I'd
rather have a QA team that's busy teaching the rest of us do things
better than spending all its time cleaning up after a developer
community that becomes more dazed and confused and left behind as time
goes on. And it also scales much better :)

This document does nothing to address the QA of anything other than the
package tree. If the scope of the team is limited to just the package
tree, then I'd like to see the team named appropriately - tree-qa
perhaps - because they would be a general, all-ranging QA team.

Best regards,
Stu
--
Stuart Herbert stuart@gentoo.org
Gentoo Developer http://www.gentoo.org/
http://blog.stuartherbert.com/

GnuGP key id# F9AFC57C available from http://pgp.mit.edu
Key fingerprint = 31FB 50D4 1F88 E227 F319 C549 0C2F 80BA F9AF C57C
--

On Sunday 26 February 2006 16:58, Ciaran McCreesh wrote:
> On Sun, 26 Feb 2006 17:22:17 -0500 Mark Loeser <halcy0n@gentoo.org>
>
> wrote:
> | Yes, Gentoo is supposed to be fun, but we also have a responsibility
> | to our users to ensure we are providing them with the best possible
> | distro we can.
>
> What, you mean the tree isn't someone's personal playground?

I do not know what he did refer to, however the following quote from
ChrisWhite's blog does come to mind:
[snip]
"Well, I was told by ciaranm to "stop treating the tree like your toy" or
something to that effect. Now, I'm going to respond to this with "Yes, it is
my toy". Before you all go phsyco and what not, let's take a look at what
Gentoo really is."
[/snip]

Following this is a rather sad rationalization as to why it is a toy.

http://www.securesystem.info/tiki-view_blog_post.php?blogId=3&postId=111

>
> | * The QA team may also offer to fix obvious typos and similar minor
> | issues, and silence from the package maintainers can be taken as
> | agreement in such situations.
>
> Should probably clarify that one. It's in there because there are some
> situations where we find obvious typos (e.g. 'souce' instead of
> 'source' in DEPEND) and file a bug to alert the maintainer. If the
> maintainer fixes it within a few days, there's no problem, but if not
> there's no point in letting the bug sit there -- someone from QA should
> be able to fix it themselves.
>
> Equally, we don't want to just fix stuff without telling people that
> they made a mistake, because then they're more likely to do it again.
>
> | * The QA team will maintain a list of current "QA Standards". The
> | list is not meant by any means to be a comprehensive document, but
> | rather a dynamic document that will be updated as new problems are
> | discovered.
>
> Hrm, do we want to include the thing about the QA standards providing
> rationale and explanations rather than hard rules here?

On Sun, 2006-02-26 at 18:41 -0500, Alec Warner wrote:
> While you may not think that soliciting comments is useful ( and in some
> limited cases I would agree with you ) giving people the opportunity to
> comment also means you just covered your ass, in terms of people going
> "where the hell did that come from?"

It also means that the QA team has learned from the lessons of the past
(ie devrel). Given Ciaran's personal and vocal history on those
lessons, one would naturally assume that he'd jump at this chance to put
his own previously stated views on how policy should be derived into
practice :)

Best regards,
Stu
--
Stuart Herbert stuart@gentoo.org
Gentoo Developer http://www.gentoo.org/
http://blog.stuartherbert.com/

GnuGP key id# F9AFC57C available from http://pgp.mit.edu
Key fingerprint = 31FB 50D4 1F88 E227 F319 C549 0C2F 80BA F9AF C57C
--

johnm@gentoo.org said:
> yeah, thats totally understandable. Its a best-efforts thing. I just
> don't want neccessary to be deemed true for something which has an
> arguable point with technical merit. Blatent mkdir-esque madness would
> be more black than white, and I'd hope for this to try and sanitise the
> gray :)

Later on we tried to address that by saying the majority of the QA team
members must agree with the action. It is normally pretty black and
white when things are necessary, and I do not know how we can accurately
describe those problems without limiting our scope.

> > The problem with that is, it usually ends up with too many pointless
> > comments from people saying how things could be fixed in the distant
> > future, or whining that it isn't explicitly forbidden by policy on
> > situations where the screwup was too weird to be documented previously.
>
> This is very much a case-by-case thing. I still feel the debate should
> be better answered outside of conflicting qa members.

Well, instead of putting the debate into an even larger crowd, this
enables the QA team to act in the way it sees best first. If people
believe we were wrong, then we give them the option to talk to the
council about one of our changes. Also, we aren't unwilling to hear
alternatives and we hope to work with the maintainer on these problems.

--
Mark Loeser - Gentoo Developer (cpp gcc-porting qa toolchain x86)
email - halcy0n AT gentoo DOT org
mark AT halcy0n DOT com
web - http://dev.gentoo.org/~halcy0n/
http://www.halcy0n.com

Alec Warner <antarus@gentoo.org> said:
> This is meant to prevent the case where the QA team ( or a subset; "the
> established QA members" ) decides to make unilateral changes to the tree
> ( or large subset thereof ) without even necessarily talking to the
> affected developers.
>
> While you may not think that soliciting comments is useful ( and in some
> limited cases I would agree with you ) giving people the opportunity to
> comment also means you just covered your ass, in terms of people going
> "where the hell did that come from?"

We don't plan on going around and making changes without discussing
issues with the maintainers. We put this in so that if the maintainer
is unwilling to work with us for some reason, that we are able to come
up with what we believe to be the best fix. As I said earlier in the
document, we plan to work as much with maintainers as possible, but
sometimes that may prove to be impossible.

--
Mark Loeser - Gentoo Developer (cpp gcc-porting qa toolchain x86)
email - halcy0n AT gentoo DOT org
mark AT halcy0n DOT com
web - http://dev.gentoo.org/~halcy0n/
http://www.halcy0n.com

Mark Loeser wrote:
> Well, instead of putting the debate into an even larger crowd, this
> enables the QA team to act in the way it sees best first. If people
> believe we were wrong, then we give them the option to talk to the
> council about one of our changes. Also, we aren't unwilling to hear
> alternatives and we hope to work with the maintainer on these problems.

As Stuart mentioned, this is not a good idea. If the maintainer
disagrees with QA-made changes, the changes should be reverted until a
higher-level decision is made. This mirrors FreeBSD policy [1], which
seems to be working quite well for them. A particularly relevant part is
this:

"Any disputed change must be backed out pending resolution of the
dispute if requested by a maintainer. Security related changes may
override a maintainer's wishes at the Security Officer's discretion."

Thanks,
Donnie

1.
http://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/rules.html

Stuart Herbert <stuart@gentoo.org> said:
> On Sun, 2006-02-26 at 17:22 -0500, Mark Loeser wrote:
> > * In case of emergency, or if package maintainers refuse to cooperate,
> > the QA team may take action themselves to fix the problem.
>
> I'd like to see this say
>
> * In case of emergency, or after a failed appeal by the package
> maintainer to the council, the QA team may take action themselves
> to fix the problem, if the package maintainer(s) are unavailable
> or refuse to co-operate.
>
> That still gives the QA team the powers it needs for an enforcement
> role, which is all that it needs.

Your change seems to imply that the QA team must wait for the council's
okay to go forth and fix the package, rather the QA team able to act on
its own. If that is the case, I don't see how we would ever be able to
get things done when someone disagrees with us.

> > * In the event that a developer still insists that a package does not
> > break QA standards, an appeal can be made at the next council meeting. The
> > package should be dealt with per QA's request until such a time that a
> > decision is made by the council.
>
> I'd like to see an alternative wording:
>
> * In the event that a developer still insists that a package does not
> break QA standards, or that the QA standards are somehow defective,
> an appeal can be made at the next council meeting.
>
> If it is an emergency, then the QA team is still free to intervene
> before the council meeting. But, where it isn't an emergency, there is
> no need for immediate action, and so there should be no action until the
> appeal has been heard.

Again, then we are going to get into the argument of the definition of
an emergency and never be able to get anything done. We really hope
problems never come down to this, which is why we left it worded this
way.

> The QA team has no monopoly on what is right or wrong in Gentoo, and
> neither do package maintainers. If there is a disagreement that leads
> to an appeal, the onus should be on the QA team to have to present their
> case to the council, as they are the ones pushing for change.

Again, this is taking away any power that the QA team might have, and
gets us stuck in the current situation where we can't do anything when
someone disagrees with us. We are hoping that most people are not going
to see us as idiots running around trying to change everything just because
we don't like it. It is expected that people will trust us to some degree,
and we are giving a way for people to challenge our decisions if they
believe we are wrong.

>
> > * Just because a particular QA violation has yet to cause an issue does
> > not change the fact that it is still a QA violation.
>
> I'd support the above if the following statement was also included:
>
> * QA standards, and the actions required to resolve them, must be
> pragmatic and proportional to the impact on actual end-users of
> Gentoo.
>
> Gentoo would be best served by a QA team that spent its time tackling
> issues that are urgent and important to end-users (quadrant 1 & 2
> activities). A QA team that gets bogged down in the thick of thin
> things is a symptom of a team that has become self-serving.

This is not quantifiable at all and will just get us into arguments with
people that disagree with us that there is a problem present.

>
> > * The QA team will maintain a list of current "QA Standards". The list
> > is not meant by any means to be a comprehensive document, but rather a
> > dynamic document that will be updated as new problems are discovered.
>
> These QA standards are policy changes that affect the whole tree. The
> QA team does *not* own QA policy - we all do, as contributors to Gentoo.
> I'm asking the council to agree an adequate process for the approval of
> QA standards by a wider group than just the QA team. Maybe changes
> could be batched up into a GLEP, say once a quarter, with the option of
> fast-tracking GLEPs in between for emergency QA policy changes? This
> would allow the QA team to perform effectively, and have the added
> benefit of ensuring that the wider developer community knew what was
> coming in advance, and could "buy in" to the changes.

Again, this bogs us down in useless process if someone wants to argue a
change that clearly breaks things, but isn't documented. It is
impossible to document every single thing that is a problem, and we are
asking for some freedom to be able to work on issues that fall under QA.

> There's a secondary issue here for me. None of our tools are perfect;
> we all have to work pragmatically within the capabilities of what we
> have. Some of the real QA issues in the tree arise from the limitations
> of our tools. I'd like to see the following incorporated into the QA
> team's mandate.
>
> * If the QA team wishes to push standards about specific aspects of
> our tools, then those standards must be endorsed by the leads of
> those tools.

So the Portage team will have to agree with us on every single problem
that is a QA issue? This seems like something we can leave alone until
it doesn't work, at which point we bring it before the council to figure
out how we can best handle this situation. Saying that another team
must approve all QA checks just seems silly.

> Why? Because the QA team has no monopoly on what is right and wrong
> with the tree. If any proposed QA standard cannot pass a simple litmus
> test of the endorsement of the leads of the tools, how can it be fit for
> enforcement against the wider development community? What is a package
> maintainer to do? He/she can't go back to the tools team in question
> and ask for a fix; all they will get is that the tools team in question
> doesn't agree with the QA team, and doesn't support that particular
> standard. Better to have QA standards that are strongly supported than
> to have QA standards supported by just the one team.

All of this seems to assume that the QA team is going to abuse its
power. So far we have had very few problems when we ask people to
fix issues that we have found for their packages. Is this going to
always be true? No, and someone needs to be able to get problems fixed
instead of just leaving things the way they are, and we want to fill
that role.

> I'd like to see the QA team's primary role stated as "educational"
> first, then watchdog, then intervener in that order (emergencies
> not-withstanding).

Which is what we plan on doing, and have been doing so far.

> With that in mind, I suggest that the QA standards must include
> educational information about the problem(s) that the standard
> addresses, before they can be approved. This is in no way to challenge
> the legitimacy of the agreed standards. It's to help all developers do
> better QA on their own work (everyone does a better job if they
> understand the reasons why). It also serves the dual purpose of helping
> the next generation of QA testers when the current team members have
> moved on. This could be Ciaran's opportunity to see TheDoc become an
> "offical" document.

We are working on a document, and hope to document all of the problems
and why they are problems. We don't plan on saying something is just
wrong and not give an explanation.


--
Mark Loeser - Gentoo Developer (cpp gcc-porting qa toolchain x86)
email - halcy0n AT gentoo DOT org
mark AT halcy0n DOT com
web - http://dev.gentoo.org/~halcy0n/
http://www.halcy0n.com

Donnie Berkholz <spyderous@gentoo.org> said:
> Mark Loeser wrote:
> > Well, instead of putting the debate into an even larger crowd, this
> > enables the QA team to act in the way it sees best first. If people
> > believe we were wrong, then we give them the option to talk to the
> > council about one of our changes. Also, we aren't unwilling to hear
> > alternatives and we hope to work with the maintainer on these problems.
>
> As Stuart mentioned, this is not a good idea. If the maintainer
> disagrees with QA-made changes, the changes should be reverted until a
> higher-level decision is made. This mirrors FreeBSD policy [1], which
> seems to be working quite well for them. A particularly relevant part is
> this:
>
> "Any disputed change must be backed out pending resolution of the
> dispute if requested by a maintainer. Security related changes may
> override a maintainer's wishes at the Security Officer's discretion."

Which is basically what we are saying. Stuart seems to be saying to
leave things "broken" and wait until a higher level decision is made.
We want to fix it/back it out/do whatever, and then come to some
resolution later if we couldn't at first.

--
Mark Loeser - Gentoo Developer (cpp gcc-porting qa toolchain x86)
email - halcy0n AT gentoo DOT org
mark AT halcy0n DOT com
web - http://dev.gentoo.org/~halcy0n/
http://www.halcy0n.com

Ciaran McCreesh <ciaranm@gentoo.org> said:
> On Sun, 26 Feb 2006 17:22:17 -0500 Mark Loeser <halcy0n@gentoo.org>
> wrote:
> | * The QA team will maintain a list of current "QA Standards". The
> | list is not meant by any means to be a comprehensive document, but
> | rather a dynamic document that will be updated as new problems are
> | discovered.
>
> Hrm, do we want to include the thing about the QA standards providing
> rationale and explanations rather than hard rules here?

Yea, I meant to add that in. I'd like for all of our standards to
include explanations and show the wrong and right way to do it.

--
Mark Loeser - Gentoo Developer (cpp gcc-porting qa toolchain x86)
email - halcy0n AT gentoo DOT org
mark AT halcy0n DOT com
web - http://dev.gentoo.org/~halcy0n/
http://www.halcy0n.com

On Sunday 26 February 2006 18:34, Mark Loeser wrote:
> Stuart Herbert <stuart@gentoo.org> said:
> > On Sun, 2006-02-26 at 17:22 -0500, Mark Loeser wrote:
> > > * In case of emergency, or if package maintainers refuse to cooperate,
> > > the QA team may take action themselves to fix the problem.
> >
> > I'd like to see this say
> >
> > * In case of emergency, or after a failed appeal by the package
> > maintainer to the council, the QA team may take action themselves
> > to fix the problem, if the package maintainer(s) are unavailable
> > or refuse to co-operate.
> >
> > That still gives the QA team the powers it needs for an enforcement
> > role, which is all that it needs.
>
> Your change seems to imply that the QA team must wait for the council's
> okay to go forth and fix the package, rather the QA team able to act on
> its own. If that is the case, I don't see how we would ever be able to
> get things done when someone disagrees with us.

give the QA team the power to fix things, likely they act on something being
broken rather than impose style changes, i do not think we want to start
discussions and appeals while broken packages remain broken

>
> > > * In the event that a developer still insists that a package does not
> > > break QA standards, an appeal can be made at the next council
> > > meeting. The package should be dealt with per QA's request until such a
> > > time that a decision is made by the council.
> >
> > I'd like to see an alternative wording:
> >
> > * In the event that a developer still insists that a package does not
> > break QA standards, or that the QA standards are somehow defective,
> > an appeal can be made at the next council meeting.
> >
> > If it is an emergency, then the QA team is still free to intervene
> > before the council meeting. But, where it isn't an emergency, there is
> > no need for immediate action, and so there should be no action until the
> > appeal has been heard.
>
> Again, then we are going to get into the argument of the definition of
> an emergency and never be able to get anything done. We really hope
> problems never come down to this, which is why we left it worded this
> way.

again, give the QA team the benefit of the doubt, if they are willing to fix
it before the maintainer, gentoo only has to gain, while hopefully everyone
learns what not to repeat in the future
>
> > The QA team has no monopoly on what is right or wrong in Gentoo, and
> > neither do package maintainers. If there is a disagreement that leads
> > to an appeal, the onus should be on the QA team to have to present their
> > case to the council, as they are the ones pushing for change.
>
> Again, this is taking away any power that the QA team might have, and
> gets us stuck in the current situation where we can't do anything when
> someone disagrees with us. We are hoping that most people are not going
> to see us as idiots running around trying to change everything just because
> we don't like it. It is expected that people will trust us to some degree,
> and we are giving a way for people to challenge our decisions if they
> believe we are wrong.

again, the QA team should have final say, and their decision should only be
appealable after the fact, not stallable with appeals before the fact
leave the tree as is (unbroken/unfixed) until the council could review any
appeal

>
> > > * Just because a particular QA violation has yet to cause an issue does
> > > not change the fact that it is still a QA violation.
> >
> > I'd support the above if the following statement was also included:
> >
> > * QA standards, and the actions required to resolve them, must be
> > pragmatic and proportional to the impact on actual end-users of
> > Gentoo.
> >
> > Gentoo would be best served by a QA team that spent its time tackling
> > issues that are urgent and important to end-users (quadrant 1 & 2
> > activities). A QA team that gets bogged down in the thick of thin
> > things is a symptom of a team that has become self-serving.
>
> This is not quantifiable at all and will just get us into arguments with
> people that disagree with us that there is a problem present.
>
> > > * The QA team will maintain a list of current "QA Standards". The list
> > > is not meant by any means to be a comprehensive document, but rather
> > > a dynamic document that will be updated as new problems are discovered.
> >
> > These QA standards are policy changes that affect the whole tree. The
> > QA team does *not* own QA policy - we all do, as contributors to Gentoo.
> > I'm asking the council to agree an adequate process for the approval of
> > QA standards by a wider group than just the QA team. Maybe changes
> > could be batched up into a GLEP, say once a quarter, with the option of
> > fast-tracking GLEPs in between for emergency QA policy changes? This
> > would allow the QA team to perform effectively, and have the added
> > benefit of ensuring that the wider developer community knew what was
> > coming in advance, and could "buy in" to the changes.
>
> Again, this bogs us down in useless process if someone wants to argue a
> change that clearly breaks things, but isn't documented. It is
> impossible to document every single thing that is a problem, and we are
> asking for some freedom to be able to work on issues that fall under QA.

the list of documented "Don't"s should be an open list, not an ultimate and
final list, just because i mess up in a way noone ever thought of documenting
before, should not give me the chance to be ignorant to the fact that i
indeed broke a package/the tree

>
> > There's a secondary issue here for me. None of our tools are perfect;
> > we all have to work pragmatically within the capabilities of what we
> > have. Some of the real QA issues in the tree arise from the limitations
> > of our tools. I'd like to see the following incorporated into the QA
> > team's mandate.
> >
> > * If the QA team wishes to push standards about specific aspects of
> > our tools, then those standards must be endorsed by the leads of
> > those tools.
>
> So the Portage team will have to agree with us on every single problem
> that is a QA issue? This seems like something we can leave alone until
> it doesn't work, at which point we bring it before the council to figure
> out how we can best handle this situation. Saying that another team
> must approve all QA checks just seems silly.
>
> > Why? Because the QA team has no monopoly on what is right and wrong
> > with the tree. If any proposed QA standard cannot pass a simple litmus
> > test of the endorsement of the leads of the tools, how can it be fit for
> > enforcement against the wider development community? What is a package
> > maintainer to do? He/she can't go back to the tools team in question
> > and ask for a fix; all they will get is that the tools team in question
> > doesn't agree with the QA team, and doesn't support that particular
> > standard. Better to have QA standards that are strongly supported than
> > to have QA standards supported by just the one team.
>
> All of this seems to assume that the QA team is going to abuse its
> power. So far we have had very few problems when we ask people to
> fix issues that we have found for their packages. Is this going to
> always be true? No, and someone needs to be able to get problems fixed
> instead of just leaving things the way they are, and we want to fill
> that role.

agreed, most objections seem to aim at preventing abuse of power rather than
commenting on how a QA team can work reliably

>
> > I'd like to see the QA team's primary role stated as "educational"
> > first, then watchdog, then intervener in that order (emergencies
> > not-withstanding).
>
> Which is what we plan on doing, and have been doing so far.

just make sure the QA team can't be paralyzed with stubborn devs filing appeal
after appeal, give them the power to act too, not just
educate/watch/intervene in security issues

>
> > With that in mind, I suggest that the QA standards must include
> > educational information about the problem(s) that the standard
> > addresses, before they can be approved. This is in no way to challenge
> > the legitimacy of the agreed standards. It's to help all developers do
> > better QA on their own work (everyone does a better job if they
> > understand the reasons why). It also serves the dual purpose of helping
> > the next generation of QA testers when the current team members have
> > moved on. This could be Ciaran's opportunity to see TheDoc become an
> > "offical" document.
>
> We are working on a document, and hope to document all of the problems
> and why they are problems. We don't plan on saying something is just
> wrong and not give an explanation.

all in all gentoo can only gain from an active, able to act QA team

Mark Loeser wrote:
> Donnie Berkholz <spyderous@gentoo.org> said:
>> Mark Loeser wrote:
>>> Well, instead of putting the debate into an even larger crowd, this
>>> enables the QA team to act in the way it sees best first. If people
>>> believe we were wrong, then we give them the option to talk to the
>>> council about one of our changes. Also, we aren't unwilling to hear
>>> alternatives and we hope to work with the maintainer on these problems.
>> As Stuart mentioned, this is not a good idea. If the maintainer
>> disagrees with QA-made changes, the changes should be reverted until a
>> higher-level decision is made. This mirrors FreeBSD policy [1], which
>> seems to be working quite well for them. A particularly relevant part is
>> this:
>>
>> "Any disputed change must be backed out pending resolution of the
>> dispute if requested by a maintainer. Security related changes may
>> override a maintainer's wishes at the Security Officer's discretion."
>
> Which is basically what we are saying. Stuart seems to be saying to
> leave things "broken" and wait until a higher level decision is made.
> We want to fix it/back it out/do whatever, and then come to some
> resolution later if we couldn't at first.

No, it's the exact opposite of what you're saying. You want to commit
first and let the maintainer bring it to the council. I'm saying the
maintainer has the right to have any non-security commit to his/her
package reverted pending a decision.

The maintainer should be the absolute authority over his/her packages,
and only the council should be able to overrule maintainer decisions in
the case of disagreement between the maintainer and anybody else.

Thanks,
Donnie

Donnie Berkholz <spyderous@gentoo.org> said:
> No, it's the exact opposite of what you're saying. You want to commit
> first and let the maintainer bring it to the council. I'm saying the
> maintainer has the right to have any non-security commit to his/her
> package reverted pending a decision.

Yea, I realize now I read it wrong :)

> The maintainer should be the absolute authority over his/her packages,
> and only the council should be able to overrule maintainer decisions in
> the case of disagreement between the maintainer and anybody else.

I think it really depends on the situation, but in general I disagree
that something should be left in a state that the QA team finds
questionable/broken. It should be a very rare occurence that this comes
up, since we don't really want to override what the maintainer says, but
I think the QA team should have this right in extreme circumstances.

--
Mark Loeser - Gentoo Developer (cpp gcc-porting qa toolchain x86)
email - halcy0n AT gentoo DOT org
mark AT halcy0n DOT com
web - http://dev.gentoo.org/~halcy0n/
http://www.halcy0n.com

Mark Loeser wrote:
> Donnie Berkholz <spyderous@gentoo.org> said:
>> The maintainer should be the absolute authority over his/her packages,
>> and only the council should be able to overrule maintainer decisions in
>> the case of disagreement between the maintainer and anybody else.
>
> I think it really depends on the situation, but in general I disagree
> that something should be left in a state that the QA team finds
> questionable/broken. It should be a very rare occurence that this comes
> up, since we don't really want to override what the maintainer says, but
> I think the QA team should have this right in extreme circumstances.

So if QA thinks one way is right, and the package maintainer thinks
another way is right, you say QA always trumps?

I'm looking at this as "innocent until proven guilty" versus "guilty
until proven innocent." When parties are in disagreement, the _current_
situation should stand until the council (or the two groups in question)
resolves it. That assumes lack of extenuating circumstances such as
security vulnerabilities or major tree breakage.

Thanks,
Donnie

On Sun, 2006-02-26 at 16:29 -0800, Donnie Berkholz wrote:
> Mark Loeser wrote:
> > Well, instead of putting the debate into an even larger crowd, this
> > enables the QA team to act in the way it sees best first. If people
> > believe we were wrong, then we give them the option to talk to the
> > council about one of our changes. Also, we aren't unwilling to hear
> > alternatives and we hope to work with the maintainer on these problems.
>
> As Stuart mentioned, this is not a good idea. If the maintainer
> disagrees with QA-made changes, the changes should be reverted until a
> higher-level decision is made. This mirrors FreeBSD policy [1], which
> seems to be working quite well for them. A particularly relevant part is
> this:
>
> "Any disputed change must be backed out pending resolution of the
> dispute if requested by a maintainer. Security related changes may
> override a maintainer's wishes at the Security Officer's discretion."

I think I agree with the part that security@ having near final say.

If I had to put a pecking order together how I think it would
look/should be would result in something like the following.

gentoo-(infra|council)
- gentoo-security
- gentoo-(devrel|base)
-gentoo-qa
- gentoo-(hardened|server)
- gentoo-(desktop|misc|maintainers|etc..)

--
Ned Ludd <solar@gentoo.org>
Gentoo Linux

--
gentoo-dev@gentoo.org mailing list

On Sun, 2006-02-26 at 19:34 -0800, Donnie Berkholz wrote:
> Mark Loeser wrote:
> > Donnie Berkholz <spyderous@gentoo.org> said:
> >> The maintainer should be the absolute authority over his/her packages,
> >> and only the council should be able to overrule maintainer decisions in
> >> the case of disagreement between the maintainer and anybody else.
> >
> > I think it really depends on the situation, but in general I disagree
> > that something should be left in a state that the QA team finds
> > questionable/broken. It should be a very rare occurence that this comes
> > up, since we don't really want to override what the maintainer says, but
> > I think the QA team should have this right in extreme circumstances.
>
> So if QA thinks one way is right, and the package maintainer thinks
> another way is right, you say QA always trumps?
>
> I'm looking at this as "innocent until proven guilty" versus "guilty
> until proven innocent." When parties are in disagreement, the _current_
> situation should stand until the council (or the two groups in question)
> resolves it. That assumes lack of extenuating circumstances such as
> security vulnerabilities or major tree breakage.

The devs asked for a council. A council was elected. The council decided
that QA trumps devs. If anybody has a problem with that they are free to
object at the next council meeting.
--
Ned Ludd <solar@gentoo.org>
Gentoo Linux

--
gentoo-dev@gentoo.org mailing list

Ned Ludd wrote:
> On Sun, 2006-02-26 at 19:34 -0800, Donnie Berkholz wrote:
>> I'm looking at this as "innocent until proven guilty" versus "guilty
>> until proven innocent." When parties are in disagreement, the _current_
>> situation should stand until the council (or the two groups in question)
>> resolves it. That assumes lack of extenuating circumstances such as
>> security vulnerabilities or major tree breakage.
>
> The devs asked for a council. A council was elected. The council decided
> that QA trumps devs. If anybody has a problem with that they are free to
> object at the next council meeting.

The council decided that QA trumps devs for documented policy. It didn't
decide, at least from what I saw, that QA could just do whatever they
feel like without any sort of change to the policy.

Thanks,
Donnie

On Sun, Feb 26, 2006 at 07:09:29PM -0500, Mark Loeser <halcy0n@gentoo.org> wrote:
> > > The problem with that is, it usually ends up with too many pointless
> > > comments from people saying how things could be fixed in the distant
> > > future, or whining that it isn't explicitly forbidden by policy on
> > > situations where the screwup was too weird to be documented previously.
> >
> > This is very much a case-by-case thing. I still feel the debate should
> > be better answered outside of conflicting qa members.
>
> Well, instead of putting the debate into an even larger crowd, this
> enables the QA team to act in the way it sees best first. If people
> believe we were wrong, then we give them the option to talk to the
> council about one of our changes. Also, we aren't unwilling to hear
> alternatives and we hope to work with the maintainer on these problems.

I've yet to read the rest of this subthread this morning, but while its
fresh in my mind I would also like to see less of a requirement from the
council. They are there purely for technical direction and not for a
teams beck and call. Regardless, I can see your point - although I would
still prefer to see a little more public discussion when the QA team are
unable to satisfactorily come to an answer between themselves and the
maintainer in question.

--
Role: Gentoo Linux Kernel Lead
Gentoo Linux: http://www.gentoo.org
Public Key: gpg --recv-keys 9C745515
Key fingerprint: A0AF F3C8 D699 A05A EC5C 24F7 95AA 241D 9C74 5515

Hi Mark,

On 2/27/06, Mark Loeser <halcy0n@gentoo.org> wrote:
> Your change seems to imply that the QA team must wait for the council's
> okay to go forth and fix the package, rather the QA team able to act on
> its own. If that is the case, I don't see how we would ever be able to
> get things done when someone disagrees with us.

In the event of a disagreement, that's exactly what I'm asking for.
Hopefully, disagreements will be rare. But, when they do arise, and
it is not an emergency, I see no reason why the QA team needs the
ability to force its point of view onto others.

> Again, then we are going to get into the argument of the definition of
> an emergency and never be able to get anything done. We really hope
> problems never come down to this, which is why we left it worded this
> way.

Me too. But it will, sooner or later, and when something isn't an
emergency, there's no reason why a change cannot wait until the
dispute has been resolved.

I have no desire to stop the QA team being able to act in an
emergency. It's in all our interests for action to be taken in an
emergency.

> > The QA team has no monopoly on what is right or wrong in Gentoo, and
> > neither do package maintainers. If there is a disagreement that leads
> > to an appeal, the onus should be on the QA team to have to present their
> > case to the council, as they are the ones pushing for change.
>
> Again, this is taking away any power that the QA team might have

I find that an interesting statement. The only power my proposals
remove is to stop you bullying people by insisting you are always
right before a peer review has happened. If there is a genuine QA
problem, and you can't convince the developer in question of that,
there's still a fair process that allows you to enforce when concensus
has failed.

Without these safeguards, my feeling is that the QA team is free to
enforce without concensus at all times. I don't believe that is in
the best interests of Gentoo, and is a significant shift in the way we
govern ourselves.

I don't see any compelling reason for the QA team to be judge, jury
and executioner, with the council acting as a post-execution appeals
panel. Wasn't devrel broken up into separate investigation and
enforcement teams over these very same concerns, raised by a member of
the QA team?

> This is not quantifiable at all and will just get us into arguments with
> people that disagree with us that there is a problem present.

If you mean that it creates grey areas where the output of automated
tools cannot always be right, I agree. If you're saying that it's
beyond your capacity as human beings to weigh up the merits of an
argument on more than just narrowly-defined technical facts, then are
you best placed to be making the decision in the first place?

If a policy is to be supportable and implementable, it has to be
reasonable, and it has to be managed by reasonable people. I feel
that what you're asking for, on this point, is more dogmatic than
reasonable.

Case in point. I've presented to you that, after two and a half
years, the situation that has sparked this debate off hasn't affected
users. I've explained that it is a third scenario, and that it is
different to the two (legitimate) scenarios that you've been busy
getting fixed. From where I'm sat, it would seem reasonable to accept
that, although this is a problem when looked at purely from a
technical point of view, this scenario isn't causing problems at this
time, and we could all move on to dealing with more important matters.
If there was a real problem, there would be enough evidence after two
and a half years for you to show me, and that would convince me (and
any other reasonable person) that I was wrong, and that action was
worth taking.

You haven't presented that evidence, or if you have, I haven't seen it
since getting up this morning.

Instead, we have a proposed QA policy that says "we're always right,
and when we're not, we still get our way until you convince the
council to let you back out the changes we demand." I think that's
unreasonable. That's why I oppose this point. To me, it smacks of
wanting to have your own way whether you're right or not. I can't
support that.

> Again, this bogs us down in useless process if someone wants to argue a
> change that clearly breaks things, but isn't documented. It is
> impossible to document every single thing that is a problem, and we are
> asking for some freedom to be able to work on issues that fall under QA.

As already mentioned, if it's not documented, how on earth do you
expect to raise the general quality of the QA done by each and every
developer? How do you expect to be able to consistently enforce your
own QA standards?

If it's not documented, then you're left with making it up as you go
along. That's in no-one's interest.

This comes back to my point about the QA team needing to be an
educational role first and foremost.

> So the Portage team will have to agree with us on every single problem
> that is a QA issue? This seems like something we can leave alone until
> it doesn't work, at which point we bring it before the council to figure
> out how we can best handle this situation. Saying that another team
> must approve all QA checks just seems silly.

I'm asserting that it isn't working. Case in point. Brian, as
co-lead of the Portage team, has said that we won't be adding
DEST_PREFIX to Portage. If the Portage team won't agree with your
interpretation of what is or isn't a valid QA check, why should you
have the right to go ahead anyway and enforce that check on the
package maintainers?

It's not silly. What do you have to fear about having your proposed
QA standards backed by key teams? If your arguments have merit, they
will be supported.

What I think is silly is the QA team claiming for itself the power to
enforce standards just on their say so.

> All of this seems to assume that the QA team is going to abuse its
> power. So far we have had very few problems when we ask people to
> fix issues that we have found for their packages. Is this going to
> always be true? No, and someone needs to be able to get problems fixed
> instead of just leaving things the way they are, and we want to fill
> that role.

I think you're already abusing that power, by calling for a package to
be removed when it's causing no trouble to anyone, and when the
workarounds create a worse user experience than what is already there.
When the developer in question declines to remove the package, your
response is a proposed QA mandate that is unbalanced.

Genuine problems need to be fixed. My proposals do not stop you from
doing that. They also ensure that we have a balanced QA approach that
genuinely serves the needs of the project.

> > I'd like to see the QA team's primary role stated as "educational"
> > first, then watchdog, then intervener in that order (emergencies
> > not-withstanding).
>
> Which is what we plan on doing, and have been doing so far.

Walk the talk. Create the educational material.

> We are working on a document, and hope to document all of the problems
> and why they are problems. We don't plan on saying something is just
> wrong and not give an explanation.

:)

Best regards,
Stu
--

--
gentoo-dev@gentoo.org mailing list