Mailing List Archive

Mark Loeser wrote:
> Here is my updated version after some feedback from people:
> * In the case of disagreement on policy among QA members, the majority
> of established QA members must agree with the action.

What is an "Established QA member"?

>
>
> I guess this won't be reviewed by the council for another month, but I'd
> like to get all of the debate out of the way now.
> Please lets keep the discussion on topic and constructive.
>
> Thanks,
>

Alec Warner <antarus@gentoo.org> said:
> Mark Loeser wrote:
> > Here is my updated version after some feedback from people:
> > * In the case of disagreement on policy among QA members, the majority
> > of established QA members must agree with the action.
>
> What is an "Established QA member"?

Listed on the QA website and approved by the current QA lead.

--
Mark Loeser - Gentoo Developer (cpp gcc-porting qa toolchain x86)
email - halcy0n AT gentoo DOT org
mark AT halcy0n DOT com
web - http://dev.gentoo.org/~halcy0n/
http://www.halcy0n.com

Mark Loeser <halcy0n@gentoo.org> said:
> Here is my updated version after some feedback from people:
>
> * The QA team's purpose is to provide cross-team assistance in keeping
> the tree in a good state. This is done primarily by finding and pointing
> out issues to maintainers and, where necessary, taking direct action.
> * In case of emergency, or if package maintainers refuse to cooperate,
> the QA team may take action themselves to fix the problem.
> * The QA team may also offer to fix obvious typos and similar minor
> issues, and silence from the package maintainers can be taken as agreement in
> such situations.
> * In the event that a developer still insists that a package does not
> break QA standards, an appeal can be made at the next council meeting. The
> package should be dealt with per QA's request until such a time that a
> decision is made by the council.
> * In the case of disagreement on policy among QA members, the majority
> of established QA members must agree with the action.
> * Just because a particular QA violation has yet to cause an issue does
> not change the fact that it is still a QA violation.
> * If a particular developer persistently causes breakage, the QA team
> may request that devrel re-evaluates that developer's commit rights.
> Evidence of past breakages will be presented with this request to
> devrel.
> * The QA team will maintain a list of current "QA Standards" with
> explanations as to why they are problems, and how to fix the problem. The
> list is not meant by any means to be a comprehensive document, but rather a
> dynamic document that will be updated as new problems are discovered. The QA
> team will also do their best to ensure all developer tools are in line with
> the current QA standards.

I'm happy enough to send the above to the Council. I think the only issue
will be with bullet point 4. I know that the QA team as a whole like the
wording this way leaving the onus on the package maintainer to prove the
merrits of their package rather then having the onus on the QA team to
prove fault. Personally I also like the wording this way (in most cases).

In the cases where a clear technical solution presents itself to the
problems the package presents that does not change the intended behavior
(unless said behavior is what is broken) and the maintainer still
refuses to apply the neceesary changes I think that the QA team should
be cleared to make them. This is all under the understanding that the
maintainer has the right to appeal in order to revert said changes.

The tougher call comes when the severity of a QA violation comes into
question. If the QA team presents a problem to a maintainer that they
believe is severe enough to warrant a package's removal and no technical
solution has presented itself to either the QA team or the maintainer to
work around the issue I think that the QA team should have the right to
hard mask the package pending an appeal. In such cases I'd almost say
that an appeal should be automatically triggered so that a decision
about the true severity of the QA issue can be ironed out. I certainly
hope that there will be few enough of these that the council will not end
up bogged down in policy decisions and QA conflict mediation.

The above also has to be done on a case by case basis, if hardmasking a
package would cause wide tree breakage itself then another choice has to
be made.

Concurrent with the above what I'd like to see is the QA team showing a
willingness to help maintainers find workarounds to particualarly sticky
violations. I'm not saying that they should have to learn the packages
inside out or that they should not be allowed to act until a solution is
found just that they should put a certain level of effort into helping
find (or concoct) a solution. This is not to say that they are not doing
this now, however, as has been said in the prior incarnation of this
thread I also believe that the stickier problems are likely to arise
because the maintainer in question did not see a better solution, so
part of QA's roll should be to help educate the developer community.

All in all the one thing I'd like to aviod is QA bugs getting closed as
invalid (one of the events that lead up to this thread). I know there
is a sense of territory with the ebuilds one maintains, but I'd really
like to see an effort made to allow the QA team to explain themselves.
If a bug gets opened up on one of your pacakges and it is unclear to you
why something is a QA issue either comment on the bug asking th QA team
to explain (and I consider pointing someone to existing offial documentation
so long as it contains an explanation of what is wrong and how to
generally fix such issues to be a valid explanation) or ping them
online.

I really hope that noone thinks the QA team is out to get them. They are
here to make the experiance better for all of us as a whole (even if
that sometimes means that the experiance for a particular package or two
needs to be a little worse). Tree QA is something that we have never had
before, at least not really (don't mean to trivialize the work that
Mr_Bones_ does). It is something that I believe we need so lets all help
with the transition instead of fighting it.

Thanks,

--
Daniel Ostrow
Gentoo Foundation Board of Trustees
Gentoo/{PPC,PPC64,DevRel}
dostrow@gentoo.org

Daniel Ostrow <dostrow@gentoo.org> said:
> The above also has to be done on a case by case basis, if hardmasking a
> package would cause wide tree breakage itself then another choice has to
> be made.

I agree. We aren't here to make a situation even worse, and we
acknowledge that we won't always get the perfect solution right away and
that flexibility is required.

--
Mark Loeser - Gentoo Developer (cpp gcc-porting qa toolchain x86)
email - halcy0n AT gentoo DOT org
mark AT halcy0n DOT com
web - http://dev.gentoo.org/~halcy0n/
http://www.halcy0n.com

[deleted]

All seems fair enough, but please fix portage qa issues before this
policy is applied (see bug http://bugs.gentoo.org/show_bug.cgi?id=123733).

Hi Mark,

This draft seems to be effectively the same as the last one.

On 3/2/06, Mark Loeser <halcy0n@gentoo.org> wrote:
> * The QA team's purpose is to provide cross-team assistance in keeping
> the tree in a good state. This is done primarily by finding and pointing
> out issues to maintainers and, where necessary, taking direct action.

Same as original version.

> * In case of emergency, or if package maintainers refuse to cooperate,
> the QA team may take action themselves to fix the problem.

Same as original version.

> * The QA team may also offer to fix obvious typos and similar minor
> issues, and silence from the package maintainers can be taken as agreement in
> such situations.

Same as original version.

> * In the event that a developer still insists that a package does not
> break QA standards, an appeal can be made at the next council meeting. The
> package should be dealt with per QA's request until such a time that a
> decision is made by the council.

Same as original version.

> * In the case of disagreement on policy among QA members, the majority
> of established QA members must agree with the action.

Same as original version.

> * Just because a particular QA violation has yet to cause an issue does
> not change the fact that it is still a QA violation.

Same as original version.

> * If a particular developer persistently causes breakage, the QA team
> may request that devrel re-evaluates that developer's commit rights.
> Evidence of past breakages will be presented with this request to
> devrel.

Same as original version.

> * The QA team will maintain a list of current "QA Standards" with
> explanations as to why they are problems, and how to fix the problem. The
> list is not meant by any means to be a comprehensive document, but rather a
> dynamic document that will be updated as new problems are discovered. The QA
> team will also do their best to ensure all developer tools are in line with
> the current QA standards.

The bit about "explanations as to why they are prblems, and how to fix
the problem" is new, as is the statement to ensure that all developer
tools are in line with the current QA standards, but otherwise this is
also the same.

I'm sorry, but personally I don't see how this draft is substantially
different from the one posted originally. It looks like you've
decided not to address the points I raised about your original draft:

* There's nothing in this policy about end users. If this QA team is
not *focused* on delivering benefit to end users, then (as has
happened this week) it becomes a self-serving team, focused instead on
what can only be described as a destructive path. No-one benefits
from that, no-one at all.

* The QA team is asking for more than it needs to perform its role.
The UNIX principle is that of "least privilege". Donnie's already
pointed out that FreeBSD is able to conduct effective QA without the
extra power that the QA team is continuing to ask for.

* There is no proposal for a process to formulate, and gain wide
approval for new QA standards. This week, there's been an example of
the QA team documenting a QA standard *after* a bug was raised about a
QA violation ... and then that document being used as if that
particular QA standard had always been in the document.

Mistakes will always be made by all developers, and good QA is
essential to Gentoo's future. We need a good QA team for Gentoo. Not
having a QA team is, in my eyes, not an option at all.

But, as this week has shown, QA members are also developers (and
human), and are just as capable of making mistakes as anyone else.

We need a quality assurance team that conducts all its activities in a
quality manner. I'm not just talking about personal behaviour, or of
any one individual. The way *everything* is done must be in a quality
manner. That should mean a high quality process for creating QA
standards, having them approved, and making sure developers know what
changes are coming and when. That should mean high quality automated
tools that cope with the real world, not some ivory tower that has no
real pay-off for users. It should mean an interpretation and
application of QA standards that is focused on how it improves matters
for real users - and not a "tick in a box" QA approach. It should
mean a team of educators, not a team out to bully with the mandate to
do so.

In twelve years of being a professional software engineer, I've never
seen a successful QA team that didn't match that description above.

Mark, in the discussions about the QA policy, your fallback
justification always seems to be "Trust us". I think this week's
events have put a big dent in the credibility of that argument, if not
holed it below the water line. If the QA team followed processes
similar to what I've described above, I believe that this week's
events wouldn't have happened. What started off as a worthy piece of
QA work, which I'm sure has fixed many real problems for users,
degenerated into something altogether unpleasant and unnecessary for
all involved. We've all gotten a week older and a week greyer out of
this. Have we fixed any real problems that stop our users installing
and running Gentoo? No, we haven't. I hope we can all (and I include
myself in that) learn something from this to prevent a repeat.

I call for Mark's proposed policy to be rejected as it stands.

Best regards,
Stu

--
gentoo-dev@gentoo.org mailing list

On Thursday 02 March 2006 03:53, Mark Loeser wrote:
> Here is my updated version after some feedback from people:
>
> * The QA team's purpose is to provide cross-team assistance in keeping
> the tree in a good state. This is done primarily by finding and
> pointing out issues to maintainers and, where necessary, taking direct
> action.
> * In case of emergency, or if package maintainers refuse to
> cooperate, the QA team may take action themselves to fix the problem.
> * The QA team may also offer to fix obvious typos and similar minor
> issues, and silence from the package maintainers can be taken as
> agreement in such situations.
> * In the event that a developer still insists that a package does not
> break QA standards, an appeal can be made at the next council
> meeting. The package should be dealt with per QA's request until such
> a time that a decision is made by the council.
> * In the case of disagreement on policy among QA members, the majority
> of established QA members must agree with the action.
> * Just because a particular QA violation has yet to cause an issue does
> not change the fact that it is still a QA violation.
> * If a particular developer persistently causes breakage, the QA team
> may request that devrel re-evaluates that developer's commit rights.
> Evidence of past breakages will be presented with this request to
> devrel.
> * The QA team will maintain a list of current "QA Standards" with
> explanations as to why they are problems, and how to fix the problem.
> The list is not meant by any means to be a comprehensive document, but
> rather a dynamic document that will be updated as new problems are
> discovered. The QA team will also do their best to ensure all
> developer tools are in line with the current QA standards.
>

I'd like to add the following two points:
* Just because breaking policy breaks a QA tool, but is guaranteed to
never break itself (formatting policy, like space vs. tab etc.) does not
increase the severity of the breakage.
* Before any enforcement is possible, QA must establish a well supported
(debated on dev-) exception policy. While it were nice if exceptions are
not needed, reality is that they can not be avoided. Therefore there
must be an exception policy.

An exception does not mean there is no violation (so appeal is
pointless). An exception means that the violation is needed because it
offers important features for the user, and the benefits outweigh the
costs. At the same time that an exception is allowed, steps should be
undertaken to get a structural solution to the problem. QA is
responsible for ensuring that the maintainer(s) of the package in
question keep on doing so.

Paul

--
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

On Thu, 2006-03-02 at 09:01 +0000, Stuart Herbert wrote:
[snip]
> * There's nothing in this policy about end users. If this QA team is
> not *focused* on delivering benefit to end users, then (as has
> happened this week) it becomes a self-serving team, focused instead on
> what can only be described as a destructive path. No-one benefits
> from that, no-one at all.
>
> * The QA team is asking for more than it needs to perform its role.
> The UNIX principle is that of "least privilege". Donnie's already
> pointed out that FreeBSD is able to conduct effective QA without the
> extra power that the QA team is continuing to ask for.
So I see two scenarios for that:
- A QA team with a purely advisory function, helping with communication.
pro: no big policy changes etc.
con: teethless QA may get ignored

- A QA team with limited executive power, fixing bugs as they are found
pro: fast turnaround times on bugs
con: resistance from developers

The second approach needs to be carefully implemented, people need to
have trust in the QA team to empower them.

> * There is no proposal for a process to formulate, and gain wide
> approval for new QA standards. This week, there's been an example of
> the QA team documenting a QA standard *after* a bug was raised about a
> QA violation ... and then that document being used as if that
> particular QA standard had always been in the document.
Communications issue. This thread should help fix the policies for that I hope.

> Mistakes will always be made by all developers, and good QA is
> essential to Gentoo's future. We need a good QA team for Gentoo. Not
> having a QA team is, in my eyes, not an option at all.
Fully agreed.
> But, as this week has shown, QA members are also developers (and
> human), and are just as capable of making mistakes as anyone else.
Obviously :-)
> We need a quality assurance team that conducts all its activities in a
> quality manner. I'm not just talking about personal behaviour, or of
> any one individual. The way *everything* is done must be in a quality
> manner. That should mean a high quality process for creating QA
> standards, having them approved, and making sure developers know what
> changes are coming and when. That should mean high quality automated
> tools that cope with the real world, not some ivory tower that has no
> real pay-off for users. It should mean an interpretation and
> application of QA standards that is focused on how it improves matters
> for real users - and not a "tick in a box" QA approach. It should
> mean a team of educators, not a team out to bully with the mandate to
> do so.
That sounds like a mission statement and should be part of QA policy

> In twelve years of being a professional software engineer, I've never
> seen a successful QA team that didn't match that description above.
>
> Mark, in the discussions about the QA policy, your fallback
> justification always seems to be "Trust us". I think this week's
> events have put a big dent in the credibility of that argument, if not
> holed it below the water line. If the QA team followed processes
> similar to what I've described above, I believe that this week's
> events wouldn't have happened. What started off as a worthy piece of
> QA work, which I'm sure has fixed many real problems for users,
> degenerated into something altogether unpleasant and unnecessary for
> all involved. We've all gotten a week older and a week greyer out of
> this. Have we fixed any real problems that stop our users installing
> and running Gentoo? No, we haven't. I hope we can all (and I include
> myself in that) learn something from this to prevent a repeat.
>
> I call for Mark's proposed policy to be rejected as it stands.
I'd like to see it extended with the ideas shown in this thread. Also
the QA team should consider ways of getting higher acceptance - I
suggest that a general vote should be done, that's about as democratic
as we can get and noone can weasel put after that (although I'm open for
other processes to give the QA team support)

Patrick
--
Stand still, and let the rest of the universe move

Paul de Vrieze <pauldv@gentoo.org> said:
> * Just because breaking policy breaks a QA tool, but is guaranteed to
> never break itself (formatting policy, like space vs. tab etc.) does not
> increase the severity of the breakage.

I had hoped something like this would have just been understood to not
be too severe, since it doesn't really break anything but coding standards.

> * Before any enforcement is possible, QA must establish a well supported
> (debated on dev-) exception policy. While it were nice if exceptions are
> not needed, reality is that they can not be avoided. Therefore there
> must be an exception policy.

I'm not sure what you mean here. You mean for each instance? In
general? In general can be difficult since it leaves a lot of things up
for interpretation. For each instance, 99% of the time an acceptable
interim solution should be able to be achieved between the QA team and
the maintainer. In situations where we can't figure out how to best
address the situation, opening the discussion up to -dev may help, but
in the end it should come down to an agreement between the maintainer
and the team.

--
Mark Loeser - Gentoo Developer (cpp gcc-porting qa toolchain x86)
email - halcy0n AT gentoo DOT org
mark AT halcy0n DOT com
web - http://dev.gentoo.org/~halcy0n/
http://www.halcy0n.com

On Thursday 02 March 2006 14:09, Mark Loeser wrote:
> Paul de Vrieze <pauldv@gentoo.org> said:
> > * Just because breaking policy breaks a QA tool, but is guaranteed to
> > never break itself (formatting policy, like space vs. tab etc.)
> > does not increase the severity of the breakage.
>
> I had hoped something like this would have just been understood to not
> be too severe, since it doesn't really break anything but coding
> standards.
>
> > * Before any enforcement is possible, QA must establish a well
> > supported (debated on dev-) exception policy. While it were nice if
> > exceptions are not needed, reality is that they can not be avoided.
> > Therefore there must be an exception policy.
>
> I'm not sure what you mean here. You mean for each instance? In
> general? In general can be difficult since it leaves a lot of things
> up for interpretation. For each instance, 99% of the time an
> acceptable interim solution should be able to be achieved between the
> QA team and the maintainer. In situations where we can't figure out
> how to best address the situation, opening the discussion up to -dev
> may help, but in the end it should come down to an agreement between
> the maintainer and the team.

The policy should be general. It could be something like "Developer and QA
discuss the exception and the solution to be used. If they do not agree
the council.... . In any case when an exception is made, appropriate bugs
are created, including a bug to request a feature that makes the
exception unneeded. When the requested feature is available and stable,
the exception becomes invalid and the feature must be used instead."

My idea is that QA can not enforce controversial things before such a
policy exists. Otherwise exceptions stay only a theoretical possibility,
and arguments continue.

Paul

--
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

On Thursday 02 March 2006 14:09, Mark Loeser wrote:
> Paul de Vrieze <pauldv@gentoo.org> said:
> > * Just because breaking policy breaks a QA tool, but is guaranteed to
> > never break itself (formatting policy, like space vs. tab etc.)
> > does not increase the severity of the breakage.
>
> I had hoped something like this would have just been understood to not
> be too severe, since it doesn't really break anything but coding
> standards.
>
> > * Before any enforcement is possible, QA must establish a well
> > supported (debated on dev-) exception policy. While it were nice if
> > exceptions are not needed, reality is that they can not be avoided.
> > Therefore there must be an exception policy.
>
> I'm not sure what you mean here. You mean for each instance? In
> general? In general can be difficult since it leaves a lot of things
> up for interpretation. For each instance, 99% of the time an
> acceptable interim solution should be able to be achieved between the
> QA team and the maintainer. In situations where we can't figure out
> how to best address the situation, opening the discussion up to -dev
> may help, but in the end it should come down to an agreement between
> the maintainer and the team.

The policy should be general. It could be something like "Developer and QA
discuss the exception and the solution to be used. If they do not agree
the council.... . In any case when an exception is made, appropriate bugs
are created, including a bug to request a feature that makes the
exception unneeded. When the requested feature is available and stable,
the exception becomes invalid and the feature must be used instead."

My idea is that QA can not enforce controversial things before such a
policy exists. Otherwise exceptions stay only a theoretical possibility,
and arguments continue.

Paul

--
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

On Thu, 2 Mar 2006 11:35:12 +0100 Paul de Vrieze <pauldv@gentoo.org>
wrote:
| * Just because breaking policy breaks a QA tool, but is guaranteed to
| never break itself (formatting policy, like space vs. tab etc.)
| does not increase the severity of the breakage.

I'd argue against this one. See, it's possible to deliberately
circumvent some of repoman's checks by doing weird whitespace and syntax
trickery. There's also no way to fix repoman short of writing a fully
functional bash parsing tool -- which is complicated enough that even
bash doesn't have one that works in some releases...

--
Ciaran McCreesh : Gentoo Developer (Wearer of the shiny hat)
Mail : ciaranm at gentoo.org
Web : http://dev.gentoo.org/~ciaranm

Ciaran McCreesh wrote:
> On Thu, 2 Mar 2006 11:35:12 +0100 Paul de Vrieze <pauldv@gentoo.org>
> wrote:
> | * Just because breaking policy breaks a QA tool, but is guaranteed to
> | never break itself (formatting policy, like space vs. tab etc.)
> | does not increase the severity of the breakage.
>
> I'd argue against this one. See, it's possible to deliberately
> circumvent some of repoman's checks by doing weird whitespace and syntax
> trickery. There's also no way to fix repoman short of writing a fully
> functional bash parsing tool -- which is complicated enough that even
> bash doesn't have one that works in some releases...

QA shouldn't have to depend on the tools you use. The final say should
be the human interaction. If doing weird white spaces breaks the tool,
but really isn't a QA issue outside of neatness, it shouldn't be waving
red flags. Yes, its probably something that should be fixed, but it
shouldn't be a critical one just because the tool is broken and can't
handle the weirdness.

--
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure | Operations Manager

---
GPG Public Key: <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1 4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net

Lance Albertson <ramereth@gentoo.org> said:
> Ciaran McCreesh wrote:
> > I'd argue against this one. See, it's possible to deliberately
> > circumvent some of repoman's checks by doing weird whitespace and syntax
> > trickery. There's also no way to fix repoman short of writing a fully
> > functional bash parsing tool -- which is complicated enough that even
> > bash doesn't have one that works in some releases...
>
> QA shouldn't have to depend on the tools you use. The final say should
> be the human interaction. If doing weird white spaces breaks the tool,
> but really isn't a QA issue outside of neatness, it shouldn't be waving
> red flags. Yes, its probably something that should be fixed, but it
> shouldn't be a critical one just because the tool is broken and can't
> handle the weirdness.

I agree. Coding standards, while they may qualify as violations, are
not as severe, but are definitely things we would like to see fixed.
They are there to make readability across ebuilds easier since
everything will be formatted the same way for a developer to see.

--
Mark Loeser - Gentoo Developer (cpp gcc-porting qa toolchain x86)
email - halcy0n AT gentoo DOT org
mark AT halcy0n DOT com
web - http://dev.gentoo.org/~halcy0n/
http://www.halcy0n.com

On Thu, 02 Mar 2006 11:35:34 -0600 Lance Albertson
<ramereth@gentoo.org> wrote:
| QA shouldn't have to depend on the tools you use.

Sure. However, the tree is far too large to check manually for many
things. If we were to do the Sekrit Tool's IUSE check manually, for
example, we'd still be in app-something, and would have missed many of
the screwups.

| The final say should
| be the human interaction. If doing weird white spaces breaks the tool,
| but really isn't a QA issue outside of neatness, it shouldn't be
| waving red flags.

The problem is, without fixing the syntax weirdness it's not possible
to tell whether red flags should be being waved for something else.

| Yes, its probably something that should be fixed,
| but it shouldn't be a critical one just because the tool is broken
| and can't handle the weirdness.

That's the thing... Doing static analysis on bash code is ludicrously
difficult. If you don't believe me, try writing a tool that will figure
out all ebuilds that have a redundant src_compile.

It's a heck of a lot easier to do if you assume that developers will
use sane syntax. Where developers don't use sane syntax, the only way
to deal with it is to check it by hand. We don't have enough developers
to do that.

--
Ciaran McCreesh : Gentoo Developer (Wearer of the shiny hat)
Mail : ciaranm at gentoo.org
Web : http://dev.gentoo.org/~ciaranm

Ciaran McCreesh wrote:
> On Thu, 02 Mar 2006 11:35:34 -0600 Lance Albertson
> <ramereth@gentoo.org> wrote:
> | QA shouldn't have to depend on the tools you use.
>
> Sure. However, the tree is far too large to check manually for many
> things. If we were to do the Sekrit Tool's IUSE check manually, for
> example, we'd still be in app-something, and would have missed many of
> the screwups.

Then fix the tool. I find it somehow ironic that a member of the QA team is
trying to force a 'work-around' just to avoid fixing the source of the problem.

--
Kind Regards,

Simon Stelling
Gentoo/AMD64 Member
--
gentoo-dev@gentoo.org mailing list

On Thu, 02 Mar 2006 19:09:28 +0100 Simon Stelling <blubb@gentoo.org>
wrote:
| Ciaran McCreesh wrote:
| > On Thu, 02 Mar 2006 11:35:34 -0600 Lance Albertson
| > <ramereth@gentoo.org> wrote:
| > | QA shouldn't have to depend on the tools you use.
| >
| > Sure. However, the tree is far too large to check manually for many
| > things. If we were to do the Sekrit Tool's IUSE check manually, for
| > example, we'd still be in app-something, and would have missed many
| > of the screwups.
|
| Then fix the tool. I find it somehow ironic that a member of the QA
| team is trying to force a 'work-around' just to avoid fixing the
| source of the problem.

How? Writing a full bash parser is extremely difficult and would be a
complete waste of time. It's far saner to assume sane syntax, and just
bail out when crazy stuff is encountered. Sane syntax is not a work
around -- it's a basic thing that should be expected from any source
file that has to be worked on by more than one person, or even one
person over a long period of time.

Syntax is already, at least to a certain extent, mandated by policy.
The question at hand is whether violations of this policy should be
effectively ignored, or whether they should be treated as potentially
severe simply because they mask other problems.

--
Ciaran McCreesh : Gentoo Developer (Wearer of the shiny hat)
Mail : ciaranm at gentoo.org
Web : http://dev.gentoo.org/~ciaranm

Ciaran McCreesh <ciaranm@gentoo.org> said:
> It's a heck of a lot easier to do if you assume that developers will
> use sane syntax. Where developers don't use sane syntax, the only way
> to deal with it is to check it by hand. We don't have enough developers
> to do that.

I don't see where anyone is saying that we shouldn't fix things to
adhere to coding standards. We are just saying it is not a, "OMG you
broke it!" problem. It is about appropriate responses to the problems
we encounter.

--
Mark Loeser - Gentoo Developer (cpp gcc-porting qa toolchain x86)
email - halcy0n AT gentoo DOT org
mark AT halcy0n DOT com
web - http://dev.gentoo.org/~halcy0n/
http://www.halcy0n.com

Ciaran McCreesh wrote:
> On Thu, 02 Mar 2006 19:09:28 +0100 Simon Stelling <blubb@gentoo.org>
> wrote:
> | Ciaran McCreesh wrote:
> | > On Thu, 02 Mar 2006 11:35:34 -0600 Lance Albertson
> | > <ramereth@gentoo.org> wrote:
> | > | QA shouldn't have to depend on the tools you use.
> | >
> | > Sure. However, the tree is far too large to check manually for many
> | > things. If we were to do the Sekrit Tool's IUSE check manually, for
> | > example, we'd still be in app-something, and would have missed many
> | > of the screwups.
> |
> | Then fix the tool. I find it somehow ironic that a member of the QA
> | team is trying to force a 'work-around' just to avoid fixing the
> | source of the problem.
>
> How? Writing a full bash parser is extremely difficult and would be a
> complete waste of time. It's far saner to assume sane syntax, and just
> bail out when crazy stuff is encountered. Sane syntax is not a work
> around -- it's a basic thing that should be expected from any source
> file that has to be worked on by more than one person, or even one
> person over a long period of time.

It should be a basic thing to expect the QA tool knows how to bail out
correctly and resume looking for more important critical issues. QA
should not revolve around the tools you use. Technical difficulties in
the QA tool dealing with weird syntax's should not provoke a red flag on
a particular package. Yes, those weirdness issues should be fixed, but
the tool should not hinder the overall outcome of the QA process.

So what if it takes too much time to fix it, then just have it ignore
that package (and mark it to be viewed later by hand) and move on to the
next package.

--
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure | Operations Manager

---
GPG Public Key: <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1 4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net

On Thu, 02 Mar 2006 13:15:48 -0600 Lance Albertson
<ramereth@gentoo.org> wrote:
| It should be a basic thing to expect the QA tool knows how to bail out
| correctly and resume looking for more important critical issues.

Sure. But what if more important critical issues are being masked by
weird syntax?

| QA should not revolve around the tools you use.

There are not enough people to check the entire tree by hand. Even if
there were, manual checks are extremely tricky to do properly. Humans
are extremely bad at picking out errors like transposed letters or a
missed captial in a package name.

The tools are extremely important. Without them most QA mistakes will
go unnoticed until they cause breakage.

| Technical difficulties in the QA tool dealing with weird syntax's
| should not provoke a red flag on a particular package.

Not flagging a weird package can lead to breakage being missed.

| So what if it takes too much time to fix it, then just have it ignore
| that package (and mark it to be viewed later by hand) and move on to
| the next package.

Doable, but not reliably, only for so long as it's a very small number
of affected packages.

Here's why manual checks don't scale. Some of the following *DEPEND
specifications are broken. Some are not. Without using a tool, pick out
which ones are broken. Then say how long it took you to do it. For
comparison, at least one QA tool can do these in well under a hundredth
of a second.

RDEPEND="|| ( ( x11-libs/libXext x11-libs/libX11 x11-libs/libXau x11-libs/libXdmcp x11-libs/libXkbui ) virtual/x11 )"

DEPEND="~app-editors/vim-core-7.0_alpha20060228 || ( x11-libs/libXext
virtual/x11 ) !aqua? ( gtk? ( >=x11-libs/gtk+-2.6 virtual/xft gnome?
( >=gnome-base/libgnomeui-2.6* ) ) !gtk? ( motif?
( x11-libs/openmotif ) !motif? ( nextaw? ( x11-libs/neXtaw ) !nextaw?
( || ( x11-libs/libXaw virtual/x11 ) ) ) ) ) !bootstrap?
( sys-devel/patch ) cscope? ( dev-util/cscope ) gpm?
( >=sys-libs/gpm-1.19.3 ) perl? ( dev-lang/perl ) python?
( dev-lang/python ) acl? ( sys-apps/acl ) ruby? ( virtual/ruby )
mzscheme? ( dev-lisp/mzscheme ) netbeans? ( dev-util/netbeans )
=sys-apps/sed-4 sys-devel/autoconf dev-util/ctags
>=sys-libs/ncurses-5.2-r2"

RDEPEND=" ( !gnome-base/gnome-core >=dev-libs/glib-2.8.6
>=x11-libs/gtk+-2.8.11 >=dev-libs/atk-1.10.3 >=x11-libs/pango-1.10.3 >=dev-libs/libxml2-2.6.23 >=dev-libs/libxslt-1.1.15 >=media-libs/audiofile-0.2.6-r1 >=media-sound/esound-0.2.36 >=x11-libs/libxklavier-2 >=media-libs/libart_lgpl-2.3.17 >=dev-libs/libIDL-0.8.6 >=gnome-base/orbit-2.12.5 >=x11-libs/libwnck-2.12.3 >=x11-wm/metacity-2.12.3 >=gnome-base/gnome-keyring-0.4.6 >=gnome-extra/gnome-keyring-manager-2.12 >=gnome-base/gnome-vfs-2.12.2 >=gnome-base/gnome-mime-data-2.4.2 >=gnome-base/gconf-2.12.1 >=net-libs/libsoup-2.2.7 >=gnome-base/libbonobo-2.10.1 >=gnome-base/libbonoboui-2.10.1 >=gnome-base/libgnome-2.12.0.1 >=gnome-base/libgnomeui-2.12 >=gnome-base/libgnomecanvas-2.12 >=gnome-base/libglade-2.5.1 >=gnome-extra/bug-buddy-2.12.1 >=gnome-base/control-center-2.12.3 >=gnome-base/eel-2.12.2 >=gnome-base/nautilus-2.12.2 =media-libs/gstreamer-0.8* =media-libs/gst-plugins-0.8* >=gnome-extra/gnome-media-2.12 >=media-sound/sound-juicer-2.12.3 >=media-video/totem-1.2.1 >=media-gfx/eog-2.12.3 >=www-client/epiphany-1.8.4.1 >=app-arch/file-roller-2.12.3 >=gnome-extra/gcalctool-5.6.31 >=gnome-extra/gconf-editor-2.12.1 >=gnome-base/gdm-2.8.0.7 >=x11-libs/gtksourceview-1.4.2 >=app-editors/gedit-2.12.1 >=app-text/evince-0.4.0 >=gnome-base/gnome-desktop-2.12.3 >=gnome-base/gnome-session-2.12.0 >=gnome-base/gnome-applets-2.12.3 >=gnome-base/gnome-panel-2.12.3 >=gnome-base/gnome-menus-2.12.0 >=x11-themes/gnome-icon-theme-2.12.1 >=x11-themes/gnome-themes-2.12.3 >=x11-themes/gtk-engines-2.6.7 >=x11-themes/gnome-backgrounds-2.12.3.1 >=x11-libs/vte-0.11.17 >=x11-terms/gnome-terminal-2.12.0 >=gnome-extra/gucharmap-1.4.4 >=gnome-base/libgnomeprint-2.12.1 >=gnome-base/libgnomeprintui-2.12.1 >=gnome-extra/gnome-utils-2.12.2 >=gnome-extra/gnome-games-2.12.3 >=gnome-base/librsvg-2.12.7 >=gnome-extra/gnome-system-monitor-2.12.2 >=gnome-base/libgtop-2.12.2 >=x11-libs/startup-notification-0.8 >=gnome-extra/gnome2-user-docs-2.8.1 >=gnome-extra/yelp-2.12.2 >=gnome-extra/zenity-2.12.1 >=net-analyzer/gnome-netstatus-2.12.0 >=net-analyzer/gnome-nettool-1.4.1 cdr? ( >=gnome-extra/nautilus-cd-burner-2.12.3 ) dvdr? ( >=gnome-extra/nautilus-cd-burner-2.12.3 ) hal? ( >=gnome-base/gnome-volume-manager-1.5.4 ) >=gnome-extra/gtkhtml-3.8.2 >=mail-client/evolution-2.4.2.1 >=gnome-extra/evolution-data-server-1.4.2.1 >=gnome-extra/evolution-webcal-2.4.1 >=net-misc/vino-2.12.0 >=app-admin/gnome-system-tools-1.4.1 >=app-admin/system-tools-backends-1.4.2 accessibility? ( >=gnome-extra/libgail-gnome-1.1.3 >=gnome-base/gail-1.8.8 >=gnome-extra/at-spi-1.6.6 >=app-accessibility/dasher-3.2.18 >=app-accessibility/gnome-mag-0.12.3 >=app-accessibility/gnome-speech-0.3.9 >=app-accessibility/gok-1.0.5 >=app-accessibility/gnopernicus-0.11.8 ) )"

--
Ciaran McCreesh : Gentoo Developer (Wearer of the shiny hat)
Mail : ciaranm at gentoo.org
Web : http://dev.gentoo.org/~ciaranm

On Thursday 02 March 2006 17:45, Ciaran McCreesh wrote:
> On Thu, 2 Mar 2006 11:35:12 +0100 Paul de Vrieze <pauldv@gentoo.org>
>
> wrote:
> | * Just because breaking policy breaks a QA tool, but is guaranteed to
> | never break itself (formatting policy, like space vs. tab etc.)
> | does not increase the severity of the breakage.
>
> I'd argue against this one. See, it's possible to deliberately
> circumvent some of repoman's checks by doing weird whitespace and syntax
> trickery. There's also no way to fix repoman short of writing a fully
> functional bash parsing tool -- which is complicated enough that even
> bash doesn't have one that works in some releases...

I'm also convinced that deliberate circumvention is easy to detect. QA should
be able to fix these issues on itself anyway. If people deliberately subvert
QA in these kinds of way, In my opinion, they are in need of a serious talk
with devrel.

Paul

--
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

On Thu, 2 Mar 2006 21:10:02 +0100 Paul de Vrieze <pauldv@gentoo.org>
wrote:
| I'm also convinced that deliberate circumvention is easy to detect.

In that case, please provide a list of cases where !arch? flags are
being used to circumvent repoman warnings, where the correct solution
would be to use use.mask. My reasonably educated guess is that this is
the most common kind of deliberate circumvention to avoid a repoman
error.

Unfortunately, detecting "foo? ( !arch? ( somepackage ) )" gets a whole
load of false positives. I already tried that one without success...

--
Ciaran McCreesh : Gentoo Developer (Wearer of the shiny hat)
Mail : ciaranm at gentoo.org
Web : http://dev.gentoo.org/~ciaranm

On Thursday 02 March 2006 19:28, Mark Loeser wrote:
> Ciaran McCreesh <ciaranm@gentoo.org> said:
> > It's a heck of a lot easier to do if you assume that developers will
> > use sane syntax. Where developers don't use sane syntax, the only way
> > to deal with it is to check it by hand. We don't have enough developers
> > to do that.
>
> I don't see where anyone is saying that we shouldn't fix things to
> adhere to coding standards. We are just saying it is not a, "OMG you
> broke it!" problem. It is about appropriate responses to the problems
> we encounter.

That is my point exactly. I do encourage QA though to make available a list of
those syntax issues that break their tools. Take it for a positive spin,
developers will then be prepared to accomodate QA. We all want quality.

Paul

--
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

On 02-03-2006 20:19:19 +0000, Ciaran McCreesh wrote:
> On Thu, 2 Mar 2006 21:10:02 +0100 Paul de Vrieze <pauldv@gentoo.org>
> wrote:
> | I'm also convinced that deliberate circumvention is easy to detect.
>
> In that case, please provide a list of cases where !arch? flags are
> being used to circumvent repoman warnings, where the correct solution

Circumvent? Can you elaborate on that? repoman does have a problem
with this, while portage does not.


--
Fabian Groffen
--
gentoo-dev@gentoo.org mailing list