Mailing List Archive

Could you (or someone else) send out the agenda and a second reminder
a short while (e.g. 1-2 days) before the actual meeting. I'd very much
appreciate that, and i guess others may too.

cheers,
Wernfried

PS: I know _i_ could volunteer as i already suggested it, but then we
want someone reliable to do this, don't we? ;-)

--
Wernfried Haas (amne) - amne at gentoo dot org
Gentoo Forums: http://forums.gentoo.org
IRC: #gentoo-forums on freenode - email: forum-mods at gentoo dot org

Mike Frysinger wrote:
> This is your monthly friendly reminder ! Same bat time (typically the
> 2nd Thursday once a month), same bat channel (#gentoo-council @
> irc.freenode.net) !

Would like the Council to discuss the current state of Gentoo Bugzilla
[1] and anon CVS/SVN [2].

Thanks.


[1] http://bugs.gentoo.org/show_bug.cgi?id=128588
[2] http://bugs.gentoo.org/show_bug.cgi?id=103664

--
Best regards,

Jakub Moc
mailto:jakub@gentoo.org
GPG signature:
http://subkeys.pgp.net:11371/pks/lookup?op=get&search=0xCEBA3D9E
Primary key fingerprint: D2D7 933C 9BA1 C95B 2C95 B30F 8717 D5FD CEBA 3D9E

... still no signature ;)

On Thu, 2006-08-03 at 11:21 +0200, Jakub Moc wrote:
> Mike Frysinger wrote:
> > This is your monthly friendly reminder ! Same bat time (typically the
> > 2nd Thursday once a month), same bat channel (#gentoo-council @
> > irc.freenode.net) !
>
> Would like the Council to discuss the current state of Gentoo Bugzilla
> [1] and anon CVS/SVN [2].

Please elaborate why you need the council to discuss
ongoing active bugs that are in progress.

--
Ned Ludd <solar@gentoo.org>
Gentoo Linux

--
gentoo-dev@gentoo.org mailing list

Ned Ludd wrote:
> On Thu, 2006-08-03 at 11:21 +0200, Jakub Moc wrote:
>> Mike Frysinger wrote:
>>> This is your monthly friendly reminder ! Same bat time (typically the
>>> 2nd Thursday once a month), same bat channel (#gentoo-council @
>>> irc.freenode.net) !
>> Would like the Council to discuss the current state of Gentoo Bugzilla
>> [1] and anon CVS/SVN [2].
>
> Please elaborate why you need the council to discuss
> ongoing active bugs that are in progress.
>

I would also like to know why the council has to be involved since
you've never sent an email to infra specifically asking the same thing
first. Sure makes sense to me that you should ask the group involved
with the work first instead of going to the top.

But since I'm typing it now, I might as well answer it.

I finally got the hardware this week for bugs, and I've been working on
bringing those boxes online. If I'm lucky, I'll have them at least
booting on their own today.

The anoncvs/svn stuff needs the attention of some knowledgeable cvs/svn
guru. We want to ensure that we cover all our grounds in the setup so
that we don't make a system that's easily DoS'able. If anyone wants to
help out with that, please contact KingTaco as he's the contact for that
project right now.

Patience is indeed a virtue.

--
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure | Operations Manager

---
GPG Public Key: <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1 4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net

> No, not really. Just that I'd expect kinda more proactive approach than
> the one demonstrated fex. in
> http://bugs.gentoo.org/show_bug.cgi?id=128588#c29 (and a bit more
> flexible approach to other alternatives, such as HW/hosting offers we've
> received before) and that have been declined for various strange reasons.

Linking to a bug where you make crazed comments about how bugs isn't
fixed!!!!1111oneone and that dammit someone should do something
now!!!1111 doesn't really help your case.

I bet if I was infra I'd be wondering what my options were since:

bugs is a pretty critical part of developing; AND
you can't just host it anywhere; AND
the hardware needed for it to perform is expensive; AND
they did not know what the problem was at first

As in, you don't just grab the first dual proc system that was offered
out of some guys basement to host bugs on. You need a dedicated host
who will stick around and provide good support should something go
wrong. You need expensive hardware ( I believe we got a blade server
with 3 blades in it, which is fscking expensive if you haven't priced
one out before ). So once again, chill out. They are working on it.

And yes bugs is slow and yes it sucks, but bitching about it doesn't
accomplish anything :x

--
gentoo-dev@gentoo.org mailing list

On Thu, 2006-08-03 at 08:06 -0500, Lance Albertson wrote:
> Ned Ludd wrote:
> > On Thu, 2006-08-03 at 11:21 +0200, Jakub Moc wrote:
> >> Mike Frysinger wrote:
> >>> This is your monthly friendly reminder ! Same bat time (typically the
> >>> 2nd Thursday once a month), same bat channel (#gentoo-council @
> >>> irc.freenode.net) !
> >> Would like the Council to discuss the current state of Gentoo Bugzilla
> >> [1] and anon CVS/SVN [2].
> >
> > Please elaborate why you need the council to discuss
> > ongoing active bugs that are in progress.
> >
>
> I would also like to know why the council has to be involved since
> you've never sent an email to infra specifically asking the same thing
> first. Sure makes sense to me that you should ask the group involved
> with the work first instead of going to the top.
>
> But since I'm typing it now, I might as well answer it.
>
> I finally got the hardware this week for bugs, and I've been working on
> bringing those boxes online.

And it's impressive hardware at a pretty kickass facility :)

> If I'm lucky, I'll have them at least
> booting on their own today.
>
> The anoncvs/svn stuff needs the attention of some knowledgeable cvs/svn
> guru. We want to ensure that we cover all our grounds in the setup so
> that we don't make a system that's easily DoS'able. If anyone wants to
> help out with that, please contact KingTaco as he's the contact for that
> project right now.

It's just about ready afaik. We just want robbat2 to review the CVS
setup and I probably need to drop a patch in the cvs pkg to disable
compression. We probably also want Pylon to review the svn setup.

> Patience is indeed a virtue.

Indeed..

--
Ned Ludd <solar@gentoo.org>
Gentoo Linux

--
gentoo-dev@gentoo.org mailing list

Ned Ludd wrote:
> On Thu, 2006-08-03 at 08:06 -0500, Lance Albertson wrote:
>>>> Would like the Council to discuss the current state of Gentoo Bugzilla
>>>> [1] and anon CVS/SVN [2].
>>> Please elaborate why you need the council to discuss
>>> ongoing active bugs that are in progress.

Progress? Erm... see below.

>> I would also like to know why the council has to be involved since
>> you've never sent an email to infra specifically asking the same thing
>> first. Sure makes sense to me that you should ask the group involved
>> with the work first instead of going to the top.

Because it's been broken for ages? Because I've asked the same on the
bug I've referred to multiple times, as did quite a few other people,
and the thing is still dead ~3 hours a day? (So uhm, the argument that
infra doesn't know about is _really_ moot.) Because users complain over
and over again? Because we are getting tons of duplicate bugs due to
bugzilla being non-responsive? Because it's wasting hours of my time
every day? Because if CVS was in the same state, you'd about have a
revolution by now?

>> The anoncvs/svn stuff needs the attention of some knowledgeable cvs/svn
>> guru. We want to ensure that we cover all our grounds in the setup so
>> that we don't make a system that's easily DoS'able. If anyone wants to
>> help out with that, please contact KingTaco as he's the contact for that
>> project right now.
>
> It's just about ready afaik. We just want robbat2 to review the CVS
> setup and I probably need to drop a patch in the cvs pkg to disable
> compression. We probably also want Pylon to review the svn setup.

Good news, would be nice if you actually responded on the bug maybe? Or
send out some status report occasionally, since the bug's been open for
~1 year now?

>> Patience is indeed a virtue.
>
> Indeed..

Sorry, having a critical facility broken for ~6 months right now =!
patience. It plain sucks.


--
Best regards,

Jakub Moc
mailto:jakub@gentoo.org
GPG signature:
http://subkeys.pgp.net:11371/pks/lookup?op=get&search=0xCEBA3D9E
Primary key fingerprint: D2D7 933C 9BA1 C95B 2C95 B30F 8717 D5FD CEBA 3D9E

... still no signature ;)

On Thu, 2006-08-03 at 16:07 +0200, Jakub Moc wrote:
> Ned Ludd wrote:
> > On Thu, 2006-08-03 at 08:06 -0500, Lance Albertson wrote:
> >>>> Would like the Council to discuss the current state of Gentoo Bugzilla
> >>>> [1] and anon CVS/SVN [2].
> >>> Please elaborate why you need the council to discuss
> >>> ongoing active bugs that are in progress.
>
> Progress? Erm... see below.
>
> >> I would also like to know why the council has to be involved since
> >> you've never sent an email to infra specifically asking the same thing
> >> first. Sure makes sense to me that you should ask the group involved
> >> with the work first instead of going to the top.
>
> Because it's been broken for ages? Because I've asked the same on the
> bug I've referred to multiple times, as did quite a few other people,
> and the thing is still dead ~3 hours a day? (So uhm, the argument that
> infra doesn't know about is _really_ moot.) Because users complain over
> and over again? Because we are getting tons of duplicate bugs due to
> bugzilla being non-responsive?

Ok this is basically bitching. Trust me we all know the current state
of things with bugzilla and it's not fun for anybody. I'm sure
however if you practice a little patience I'm sure you will be
quite pleased with the end result.

> Because it's wasting hours of my time
> every day? Because if CVS was in the same state, you'd about have a
> revolution by now?

I think you might be misunderstanding the role that the council plays.
It's a body for technical matters that effect the mainly "the code".
Daily matters of infrastructure are handled by our infra team naturally.
Funding for hardware is approved by the foundation.

> >> The anoncvs/svn stuff needs the attention of some knowledgeable cvs/svn
> >> guru. We want to ensure that we cover all our grounds in the setup so
> >> that we don't make a system that's easily DoS'able. If anyone wants to
> >> help out with that, please contact KingTaco as he's the contact for that
> >> project right now.
> >
> > It's just about ready afaik. We just want robbat2 to review the CVS
> > setup and I probably need to drop a patch in the cvs pkg to disable
> > compression. We probably also want Pylon to review the svn setup.
>
> Good news, would be nice if you actually responded on the bug maybe? Or
> send out some status report occasionally, since the bug's been open for
> ~1 year now?
>
> >> Patience is indeed a virtue.
> >
> > Indeed..
>
> Sorry, having a critical facility broken for ~6 months right now =!
> patience. It plain sucks.

You must live in that town where spare hardware and administrators
grow on the trees.

As it stands I do not see why this needs to be an agenda item for
council discussions.

--
Ned Ludd <solar@gentoo.org>
Gentoo Linux

--
gentoo-dev@gentoo.org mailing list

Jakub Moc wrote:

>>> I would also like to know why the council has to be involved since
>>> you've never sent an email to infra specifically asking the same thing
>>> first. Sure makes sense to me that you should ask the group involved
>>> with the work first instead of going to the top.
>
> Because it's been broken for ages? Because I've asked the same on the
> bug I've referred to multiple times, as did quite a few other people,
> and the thing is still dead ~3 hours a day? (So uhm, the argument that
> infra doesn't know about is _really_ moot.) Because users complain over
> and over again? Because we are getting tons of duplicate bugs due to
> bugzilla being non-responsive? Because it's wasting hours of my time
> every day? Because if CVS was in the same state, you'd about have a
> revolution by now?

There's a difference between "do you know its broke?" and "how is the
progress going?". I understood your email as "how is the progress
going?" as such, I did not see any recent comment made on the bug, nor
an email sent to us asking how the progress was going. I know you view
this as "I don't see it fixed, so that's not progress", but if you don't
think the hard work of the GNi folks of setting up some nice hardware
for this as part of the progress, then I'm sorry.

Anyways, if you would like, I can send you an update to the bug every
week until it gets completed. My guess (sparing any huge hurdles), is
that we can have the new bugs up by the end of this month. That's
including setting up the software, and testing it to make sure that the
issues that's been occurring have been addressed.

Cheers-

--
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure | Operations Manager

---
GPG Public Key: <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1 4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net

Ned Ludd wrote:
> On Thu, 2006-08-03 at 16:07 +0200, Jakub Moc wrote:
>> Because it's been broken for ages? Because I've asked the same on the
>> bug I've referred to multiple times, as did quite a few other people,
>> and the thing is still dead ~3 hours a day? (So uhm, the argument that
>> infra doesn't know about is _really_ moot.) Because users complain over
>> and over again? Because we are getting tons of duplicate bugs due to
>> bugzilla being non-responsive?
>
> Ok this is basically bitching. Trust me we all know the current state
> of things with bugzilla and it's not fun for anybody. I'm sure
> however if you practice a little patience I'm sure you will be
> quite pleased with the end result.

A little patience? As in half year is not enough?

>> Because it's wasting hours of my time
>> every day? Because if CVS was in the same state, you'd about have a
>> revolution by now?
>
> I think you might be misunderstanding the role that the council plays.
> It's a body for technical matters that effect the mainly "the code".
> Daily matters of infrastructure are handled by our infra team naturally.
> Funding for hardware is approved by the foundation.

Well, I think council should care about things that affect Gentoo as a
whole, and apparently that's not just me:

http://www.gentoo.org/proj/en/council/

<snip>
The elected Gentoo Council decides on global issues and policies that
affect multiple projects in Gentoo.
</snip>

Broken bugzilla affects every ebuild dev, affects GDP, affects bug
wranglers, affects anyone else who's using it to track outstanding
project issues. How is this continuous borkage not a global issue that
council should discuss?

> You must live in that town where spare hardware and administrators
> grow on the trees.

No, not really. Just that I'd expect kinda more proactive approach than
the one demonstrated fex. in
http://bugs.gentoo.org/show_bug.cgi?id=128588#c29 (and a bit more
flexible approach to other alternatives, such as HW/hosting offers we've
received before) and that have been declined for various strange reasons.


--
Best regards,

Jakub Moc
mailto:jakub@gentoo.org
GPG signature:
http://subkeys.pgp.net:11371/pks/lookup?op=get&search=0xCEBA3D9E
Primary key fingerprint: D2D7 933C 9BA1 C95B 2C95 B30F 8717 D5FD CEBA 3D9E

... still no signature ;)

Lance Albertson wrote:
> Anyways, if you would like, I can send you an update to the bug every
> week until it gets completed.

I'd just love to see this fixed and working, first of all... ;)

> My guess (sparing any huge hurdles), is
> that we can have the new bugs up by the end of this month. That's
> including setting up the software, and testing it to make sure that the
> issues that's been occurring have been addressed.

Wonderful.


--
Best regards,

Jakub Moc
mailto:jakub@gentoo.org
GPG signature:
http://subkeys.pgp.net:11371/pks/lookup?op=get&search=0xCEBA3D9E
Primary key fingerprint: D2D7 933C 9BA1 C95B 2C95 B30F 8717 D5FD CEBA 3D9E

... still no signature ;)

On Thu, 2006-08-03 at 10:48 -0400, Ned Ludd wrote:
> Ok this is basically bitching. Trust me we all know the current state
> of things with bugzilla and it's not fun for anybody. I'm sure
> however if you practice a little patience I'm sure you will be
> quite pleased with the end result.
I guess some more visible progress would help. I've followed the
situation where I could, it's taken more time than expected because
things didn't always go as planned. Jakub's frustration is fully
understandable, infra people have not comunicated much so it looks like
"noone" cares. (I know that some people have spent lotss of time on
fixing it, but they were busy fixing instead of writing reports ;-) )

> You must live in that town where spare hardware and administrators
> grow on the trees.
What'cha need?

I think there are many reliable sponsors that could get you quite some
hardware within a week. You just have to ask!


> As it stands I do not see why this needs to be an agenda item for
> council discussions.
I also think what's been discussed here is enough.

Patrick
--
Stand still, and let the rest of the universe move

Jakub Moc wrote:
> Broken bugzilla affects every ebuild dev, affects GDP, affects bug
> wranglers, affects anyone else who's using it to track outstanding
> project issues. How is this continuous borkage not a global issue that
> council should discuss?

What is there to discuss? Do you expect them to say 'bugzilla shall be fix0rd!'?
What would that change in reality? I, (and I guess so does solar) fail to see
what the council could effectively do in regard to this matter. You should
probably elaborate on that.

--
Kind Regards,

Simon Stelling
Gentoo/AMD64 Developer
--
gentoo-dev@gentoo.org mailing list

On Thu, 2006-08-03 at 19:03 +0200, Simon Stelling wrote:
> Jakub Moc wrote:
> > Broken bugzilla affects every ebuild dev, affects GDP, affects bug
> > wranglers, affects anyone else who's using it to track outstanding
> > project issues. How is this continuous borkage not a global issue that
> > council should discuss?
>
> What is there to discuss? Do you expect them to say 'bugzilla shall be fix0rd!'?
They could (in theory) demand that infra changes policies, i.e. active
recruiting to reduce bottlenecks or open demands for hardware. I think
it wouldn't help much though ...

> What would that change in reality? I, (and I guess so does solar) fail to see
> what the council could effectively do in regard to this matter. You should
> probably elaborate on that.
I guess the question is "What can we do when a project misbehaves and
our efforts to help are being denied?" (Please excuse the offensive
formulation, but that is afaict roughly what jakub and others
communicate when they wish to summon the council).

Any ideas how to improve communication appreciated :-)

hth,
Patrick
--
Stand still, and let the rest of the universe move

On Thu, 2006-08-03 at 13:20 +0000, Alec Warner wrote:
> > No, not really. Just that I'd expect kinda more proactive approach than
> > the one demonstrated fex. in
> > http://bugs.gentoo.org/show_bug.cgi?id=128588#c29 (and a bit more
> > flexible approach to other alternatives, such as HW/hosting offers we've
> > received before) and that have been declined for various strange reasons.
>
> Linking to a bug where you make crazed comments about how bugs isn't
> fixed!!!!1111oneone and that dammit someone should do something
> now!!!1111 doesn't really help your case.
>
> I bet if I was infra I'd be wondering what my options were since:
>
> bugs is a pretty critical part of developing; AND
yes
> you can't just host it anywhere; AND
it's not _that_ much hardware (and bandwidth) needed
> the hardware needed for it to perform is expensive; AND
for a single person yes. For a sponsor (or a group of sponsors) it may
be ok

> they did not know what the problem was at first
And even there it took some heavy prodding to get people to look at the problem.

After about half a year of waiting, with people we would consider
reliable offering pretty much everything from hosting to hardware, it's
hard to listen to the "be patient" mantra without thinking "omgwtfbbq,
it is _still_ not fixed?". Especially since bugs is considered an
important part of our infrastructure.

> As in, you don't just grab the first dual proc system that was offered
> out of some guys basement to host bugs on.
Agreed, but I'd say a webhoster with >1000 machines should know what
they are doing.

> You need a dedicated host
> who will stick around and provide good support should something go
> wrong.
Only experience can tell you how they will respond, and even reliable
sponsors could get axed if their managment changes. We have almost no
hardware in Europe, that's a huge untapped ressource ...

> You need expensive hardware ( I believe we got a blade server
> with 3 blades in it, which is fscking expensive if you haven't priced
> one out before ). So once again, chill out. They are working on it.
Dude, you don't need blades for it. Any "normal" server will do, two for DB and one for web frontend.
That we got blades is really nice and sweet, but if you check the
traffic and throughput of bugzilla (and then double or triple that for
future growth) you should still be able to do it easily.

(Note to our sponsors: you rock. Keep on rocking.)

Right now bugs is served from a 2,4Ghz P4 - that's roughly a normal
desktop box from last year.

> And yes bugs is slow and yes it sucks, but bitching about it doesn't
> accomplish anything :x
It may cause discussion that may lead to accelerated problem solving :-)

hth,

Patrick
--
Stand still, and let the rest of the universe move

Patrick Lauer wrote:

> (Note to our sponsors: you rock. Keep on rocking.)
>
> Right now bugs is served from a 2,4Ghz P4 - that's roughly a normal
> desktop box from last year.

You have no concept of where the bottle neck is. The webserver hosting
the cgi part isn't being loaded hardly at all. The database server is a
pretty beefy box, and again, its not so much a specific hardware
limitation, just more a limitation on the design of bugzilla and its
ties to mysql. We're having to 'fix' the problem by getting a
master/slave mysql db server setup which the OSL didn't have setup at
the time. This is apparently the 'solution' upstream suggests which I
think is daft, but its what we have to do.

Please stop stating solutions to problems you don't fully understand or
think you understand. I'm getting tired of all this fud being said
around about stuff people don't totally understand. Hardware from
sponsors mean nothing if they aren't utilized in a proper manner with
planning and skills. And its not really a big bottle neck of people. You
try finding people who have a ton of free time, have excellent admin
skills, gives everyone on the team a 'good vibe' and seems trustworthy.
Its not as easy as you think it is. I'm in the process of bringing on a
guy I know personally which will help the load of things a lot. Plus, he
works with me, so I can kick him literally if he slacks :). (now if only
recruiters *cough*swift*cough* could work faster ;-) )

Anyways, I'm not going to rant on about this anymore. We're working on
the problem, and you just have to be patient.

Cheers-

--
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure | Operations Manager

---
GPG Public Key: <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1 4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net

Alec Warner wrote:

> I bet if I was infra I'd be wondering what my options were since:
>
> bugs is a pretty critical part of developing; AND
> you can't just host it anywhere; AND
> the hardware needed for it to perform is expensive; AND
> they did not know what the problem was at first

Yes, its been hard to find a place that I feel totally comfortable with
moving such a service. The service at the OSL is hard to beat, but
they're loaded enough with other projects as it is. I'm hoping that
moving this will help their load as well.

> And yes bugs is slow and yes it sucks, but bitching about it doesn't
> accomplish anything :x

Exactly :-) How would you feel if I nagged you every other day about why
cvs hasn't been migrated, and that I'm bringing the council into it
because you're not fast enough and being ignorant for help? I think you
know how I'm feeling now then. I fail to see how the constant nagging
helps the situation. The logic escapes me. There are more constructive
ways of finding out the status of a problem/solution.

--
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure | Operations Manager

---
GPG Public Key: <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1 4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net

On Thu, 03 Aug 2006 13:51:55 -0500 Lance Albertson
<ramereth@gentoo.org> wrote:
| There are more constructive ways of finding out the status of a
| problem/solution.

Actually, with that in mind I think the council could be of help here...
The bugs slowdown *is* a serious problem, and Jakub's failing miserably
at getting it addressed properly. Perhaps the council could suggest a
more reasonable way of handling things -- for example, how about
requesting that certain projects (ones that are important but not
providing what developers or users would like) deliver status reports
to each meeting? That would satisfy the requests for information, and
it would spare infra from the constant puerile whining they're
receiving...

--
Ciaran McCreesh
Mail : ciaran dot mccreesh at blueyonder.co.uk


--
gentoo-dev@gentoo.org mailing list

Ciaran McCreesh wrote:
> On Thu, 03 Aug 2006 13:51:55 -0500 Lance Albertson
> <ramereth@gentoo.org> wrote:
> | There are more constructive ways of finding out the status of a
> | problem/solution.
>
> Actually, with that in mind I think the council could be of help here...
> The bugs slowdown *is* a serious problem, and Jakub's failing miserably
> at getting it addressed properly. Perhaps the council could suggest a
> more reasonable way of handling things -- for example, how about
> requesting that certain projects (ones that are important but not
> providing what developers or users would like) deliver status reports
> to each meeting? That would satisfy the requests for information, and
> it would spare infra from the constant puerile whining they're
> receiving...
>

I don't see a problem with doing something like that if its within
reason. I know this has made infra look terrible with how long its been
drawn out. Its been a failure on many different levels in the chain. I
admit that I'm not happy about it either and feel rather embarrassed
about it.

I was planning on eventually sending an email asking the devs what they
want so we (infra) have a better idea. Sometimes its hard to get a feel
for what developers need without them directly contacting us. Anyways,
thats for another time (hopefully soon).

--
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure | Operations Manager

---
GPG Public Key: <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1 4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net

On Thu, 2006-08-03 at 13:48 -0500, Lance Albertson wrote:
> Patrick Lauer wrote:
>
> > (Note to our sponsors: you rock. Keep on rocking.)
> >
> > Right now bugs is served from a 2,4Ghz P4 - that's roughly a normal
> > desktop box from last year.
>
> You have no concept of where the bottle neck is.
I have followed the discussions quite well. I think I'm quite aware of
the issues.

> The webserver hosting
> the cgi part isn't being loaded hardly at all.
Yes, only problem is that bugzilla likes to consume larger amounts of
memory, and if I'm not mistaken it's a bad interaction from a OOM killer
(to avoid the webfrontend to die) causing stale locks (which should not
happen) that causes bugzie to fall over, ja?
(I've been told a simple testcase to demonstrate that, haven't tried
myself)

> The database server is a
> pretty beefy box, and again, its not so much a specific hardware
> limitation, just more a limitation on the design of bugzilla and its
> ties to mysql. We're having to 'fix' the problem by getting a
> master/slave mysql db server setup which the OSL didn't have setup at
> the time. This is apparently the 'solution' upstream suggests which I
> think is daft, but its what we have to do.
... and one of the slowdowns was OSU being unable to get their DB cluster up and running within a reasonable timeframe. Fertilizer happens ...

> Please stop stating solutions to problems you don't fully understand or
> think you understand. I'm getting tired of all this fud being said
> around about stuff people don't totally understand.
I'm getting tired of being told "we can manage it", then having to wait
6 months to hear "almost there". We had a few people asking how they
could help, and the answer was roughly "we manage fine on our own, thank
you very much". Personally I don't mind much, but then you shouldn't
complain when people say "we could have done better" ...

> Hardware from
> sponsors mean nothing if they aren't utilized in a proper manner with
> planning and skills. And its not really a big bottle neck of people.
That sounds contradictory to me - it's not the people, it's the people?

> You
> try finding people who have a ton of free time, have excellent admin
> skills, gives everyone on the team a 'good vibe' and seems trustworthy.
For me it's easy, being a dev for more than, say, 3 months = trustworthy
Of course if you need to recruit from the outside the situation changes

> Its not as easy as you think it is.
Let me try and fail and I'll believe you ...

> I'm in the process of bringing on a
> guy I know personally which will help the load of things a lot. Plus, he
> works with me, so I can kick him literally if he slacks :). (now if only
> recruiters *cough*swift*cough* could work faster ;-) )
Cool.

> Anyways, I'm not going to rant on about this anymore. We're working on
> the problem, and you just have to be patient.
Nooooooo :-) You said the bad words again! ;-)

I hope you get everything fixed soonish, let's hope Murphy's law doesn't
try to apply here ...


Patrick
--
Stand still, and let the rest of the universe move

Hi,

On 03/08/06, Lance Albertson <ramereth@gentoo.org> wrote:
> You have no concept of where the bottle neck is. The webserver hosting
> the cgi part isn't being loaded hardly at all. The database server is a
> pretty beefy box, and again, its not so much a specific hardware
> limitation, just more a limitation on the design of bugzilla and its
> ties to mysql. We're having to 'fix' the problem by getting a
> master/slave mysql db server setup which the OSL didn't have setup at
> the time. This is apparently the 'solution' upstream suggests which I
> think is daft, but its what we have to do.

I'm curious what the problem is with bugzilla and it's db
interactions? You're suggesting a specific issue rather than general
db performance issues like fs, io scheduling, raid1, hyperthreads,
etc.?
--
gentoo-dev@gentoo.org mailing list

Chris Bainbridge wrote:
> I'm curious what the problem is with bugzilla and it's db
> interactions? You're suggesting a specific issue rather than general
> db performance issues like fs, io scheduling, raid1, hyperthreads,
> etc.?

It's most likely related to Bugzilla using MyISAM tables by default
which is fine in a small environment. However as concurrency grows along
with table size those full table locks begin to cause issues.
Additionally most www-apps are not known for their well thought out db
schema. Performance of the underlying hardware plays a part, but can be
overshadowed pretty quickly by query and table inefficiencies.

The usual fix without touching the internals of the software is doing
master/slave replication and allowing the selects to happen on the slave
and writes on the master. However you would need to change most of the
queries to point to the slave server which is usually not trivial. It
sounds like this is the path the Infra team is pursuing.

kashani
--
gentoo-dev@gentoo.org mailing list

Chris Bainbridge wrote:
> Hi,
>
> On 03/08/06, Lance Albertson <ramereth@gentoo.org> wrote:
>> You have no concept of where the bottle neck is. The webserver hosting
>> the cgi part isn't being loaded hardly at all. The database server is a
>> pretty beefy box, and again, its not so much a specific hardware
>> limitation, just more a limitation on the design of bugzilla and its
>> ties to mysql. We're having to 'fix' the problem by getting a
>> master/slave mysql db server setup which the OSL didn't have setup at
>> the time. This is apparently the 'solution' upstream suggests which I
>> think is daft, but its what we have to do.
>
> I'm curious what the problem is with bugzilla and it's db
> interactions? You're suggesting a specific issue rather than general
> db performance issues like fs, io scheduling, raid1, hyperthreads,
> etc.?

Mostly dealing with table locking and how bugzilla handles most of its
queries. If you want more details, look for r2d2, Kingtaco, or jforman
(in that order). Another issue was backups were locking up all the
tables, so for a matter of an hour or more every day, bugs was just
useless because of that. Upstream's 'solution' was to use a db cluster
and make the backup on the read-only side. Anyways, I personally don't
know a lot of the details, but feel free to ask those people mentioned
above.

--
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure | Operations Manager

---
GPG Public Key: <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1 4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net

kashani wrote:
> Chris Bainbridge wrote:
>> I'm curious what the problem is with bugzilla and it's db
>> interactions? You're suggesting a specific issue rather than general
>> db performance issues like fs, io scheduling, raid1, hyperthreads,
>> etc.?
>
> It's most likely related to Bugzilla using MyISAM tables by default
> which is fine in a small environment. However as concurrency grows along
> with table size those full table locks begin to cause issues.
> Additionally most www-apps are not known for their well thought out db
> schema. Performance of the underlying hardware plays a part, but can be
> overshadowed pretty quickly by query and table inefficiencies.
>
> The usual fix without touching the internals of the software is
> doing master/slave replication and allowing the selects to happen on the
> slave and writes on the master. However you would need to change most of
> the queries to point to the slave server which is usually not trivial.
> It sounds like this is the path the Infra team is pursuing.

1++

You got it right :-)

--
Lance Albertson <ramereth@gentoo.org>
Gentoo Infrastructure | Operations Manager

---
GPG Public Key: <http://www.ramereth.net/lance.asc>
Key fingerprint: 0423 92F3 544A 1282 5AB1 4D07 416F A15D 27F4 B742

ramereth/irc.freenode.net