Mailing List Archive

Concern with performance issues
I am not one to complain frequently or quickly. I have been a software
developer, and I know the anguish that often involves, and the meager
thanks that are often received.

However, I am becoming quite concerned about the recent performance
issues on en.wikipedia. I don't know if there are similar issues with
other projects or languages.

As users, we are told (at
http://www.livejournal.com/community/wikitech/) that the issues now is
not hardware or the MediaWiki software, but rather load balancing. We
are also told to "try again" when we receive one of many ugly errors.
We are invited to help, but given no indication how we might do that.

This situation is very unsatisfactory, and is damaging to Wikipedia.

I know that the developers are very busy, and are volunteering their
time. I do not "blame" them for the problems. I don't really see this
as a technical issue. Obviously, the solution to load balancing will be
technical, but the solution to the larger issue (reliability rapid
response to problems) is an issue of the will and resources of the
Wikimedia foundation.

We cannot allow en.wikipedia (and perhaps others) to languish for days
with horrible response times and frequent, ugly user errors. We must
quickly determine what we need to do to remedy this situation, and
quickly act to implement the remedy.

With respect and concern,

Rich Holton

(en.wikipedia:user:rholton)

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: Concern with performance issues [ In reply to ]
On Thu, 13 Jan 2005 09:25:05 -0800 (PST), Rich Holton
<rich_holton@yahoo.com> wrote:
> We are invited to help, but given no indication how we might do that.

I strongly agree with this. Can we please put up a donation banner?

The only thing more frustrating than a prolonged speed problem is
having people say "we're working it out; just come back soon". Even
if donations won't fix the immediate cause of this week's slowdown,
they will definitely improve speed in the future and will make users
feel better.

> However, I am becoming quite concerned about the recent performance
> issues on en.wikipedia. I don't know if there are similar issues with
> other projects or languages.

It seems pretty universal, unless you're connecting via the French
squids. I agree that it has been bad for an *awfully* long time, and
we should make sure that this never happens again. I've had many
quick 15-minute intervals this past week where I was at a machine with
time to work on WP projects, and that was never enough time to get
anything done.

We should also consider some method of allocating "dedicated
connections" to people for slots of time; I could make very productive
use of 5 minutes of a fast connection, rather than trying to load a
set of pages sporadically for eight hours.

+sj+
Re: Concern with performance issues [ In reply to ]
I think that one of the things we need to be absolutely certain about
is that when we do show an error page, it is not ugly but rather a
direct appeal for more resources to solve the problem. I am told that
hardware and MediaWiki software are not the problem, but load
balancing, and this takes developer time and resources to figure out.

One thing we might do here is think about how we might apply some
resources to developer/admin recruitment.

Here's the basic argument: traffic can grow very quickly. Adding new
authors is very easy. Part of the whole point of a wiki is that new
people can get involved easily. But growing the number of
developers/admins takes time, because there's a steep learning curve
for people to get involved. So it might be the case that over time,
this aspect of things lags behind.

And of course when there is too much work for too few people, it isn't
as fun anymore.

When I'm at Fosdem next month, a *big* part of my talk will be
focussed on this: a sales pitch to potential devs to get involved.

--Jimbo
Re: Concern with performance issues [ In reply to ]
--- Rich Holton <rich_holton@yahoo.com> wrote:
> We cannot allow en.wikipedia (and perhaps others) to languish for days
> with horrible response times and frequent, ugly user errors. We must
> quickly determine what we need to do to remedy this situation, and
> quickly act to implement the remedy.

We have grown to the point where money for hardware really isn't the issue.
Whenever we need money we have been able to raise it. [.ASIDE: as of 31 Dec
there was over US$30,000 in the bank and today there is over US$17,000 in
PayPal - not to mention what the German Chapter has.] In early February we have
our next donation drive planned.

The issue as I see it, is that we are now asking too much of too few volunteer
developers and none of them can be on-site at the colo. Thus I think it is the
time to consider hiring one full time or two of them part time to perform
server maintenance tasks for us that currently are not done in a timely manor
(esp the on-site stuff). Our budget is now so big ($US80,000 last quarter) that
the addition of one or two employees under contract would not be too much of a
burden. Neither would some up-front relocation money.

-- Daniel Mayer (aka mav)



__________________________________
Do you Yahoo!?
Yahoo! Mail - 250MB free storage. Do more. Manage less.
http://info.mail.yahoo.com/mail_250
Re: Concern with performance issues [ In reply to ]
It would be nice to have a proper safety margin of redundant machines
- say, one hot spare for every core machine, and one for every two
apaches/squids - before deciding that "money for hardware really isn't
the issue". I had the impression that some of the recent delays and
outages would have been less troublesome (and less noticeable), had
there been more redundancy of key components.

The $50k in available funds as of Dec 31 was a bit less than two
months' operating costs; not as much as it might seem.

--
+sj+
Re: Concern with performance issues [ In reply to ]
--- Sj <2.718281828@gmail.com> wrote:
> It would be nice to have a proper safety margin of redundant machines
> - say, one hot spare for every core machine, and one for every two
> apaches/squids - before deciding that "money for hardware really isn't
> the issue". I had the impression that some of the recent delays and
> outages would have been less troublesome (and less noticeable), had
> there been more redundancy of key components.

I said what I said based on my impressions by talking with developers; both
Jimmy and I are often pushing them to buy more servers but want to make sure we
get the largest bang for the buck.

> The $50k in available funds as of Dec 31 was a bit less than two
> months' operating costs; not as much as it might seem.

My point was that the cost of having an employee or two is now relatively cheap
given the enormity of our other expenses and income. Thus it is now possible to
talk of such a thing and not be shouted down due to its budgetary
impracticability.

-- mav

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: Concern with performance issues [ In reply to ]
I'm a bit hesitant about this. If we hire some of the existing
volunteers, we would partly be paying for what we now get in volunteer
time. And if we hire people from outside, will not lead that to
problems with people who do the same thing for free?

If we lose volunteer work for what we get in paid work, the price of
paid work might well be a lot higher than what it is at first sight.

Andre Engels


> The issue as I see it, is that we are now asking too much of too few volunteer
> developers and none of them can be on-site at the colo. Thus I think it is the
> time to consider hiring one full time or two of them part time to perform
> server maintenance tasks for us that currently are not done in a timely manor
> (esp the on-site stuff). Our budget is now so big ($US80,000 last quarter) that
> the addition of one or two employees under contract would not be too much of a
> burden. Neither would some up-front relocation money.
Re: Concern with performance issues [ In reply to ]
It depends : system administration, for exemple, is a
boring work (imho), and could be done by somebody from
outside, without special training (no need to know how
MediaWiki works, for exemple). So time can be released
for more specific (and interesting) work, like
development. Other opinions ?

Traroth

--- Andre Engels <andreengels@gmail.com> a écrit :
> I'm a bit hesitant about this. If we hire some of
> the existing
> volunteers, we would partly be paying for what we
> now get in volunteer
> time. And if we hire people from outside, will not
> lead that to
> problems with people who do the same thing for free?
>
> If we lose volunteer work for what we get in paid
> work, the price of
> paid work might well be a lot higher than what it is
> at first sight.
>
> Andre Engels







Découvrez le nouveau Yahoo! Mail : 250 Mo d'espace de stockage pour vos mails !
Créez votre Yahoo! Mail sur http://fr.mail.yahoo.com/
Re: Concern with performance issues [ In reply to ]
Andre Engels wrote:

>I'm a bit hesitant about this. If we hire some of the existing
>volunteers, we would partly be paying for what we now get in volunteer
>time. And if we hire people from outside, will not lead that to
>problems with people who do the same thing for free?
>
>If we lose volunteer work for what we get in paid work, the price of
>paid work might well be a lot higher than what it is at first sight.
>
>
That will always be a concern with any kind of hiring. We would
probably do best to contract for a fixed term of three months at a time,
with an option to renew once for another three months. After that the
person could not be hired again for at least another six months. One
thing that this would accomplish is that it would avoid having people
become dependent on the Foundation for a regular source of income.

Ec
Re: Concern with performance issues [ In reply to ]
--- Andre Engels <andreengels@gmail.com> wrote:
> I'm a bit hesitant about this. If we hire some of the existing
> volunteers, we would partly be paying for what we now get in volunteer
> time. And if we hire people from outside, will not lead that to
> problems with people who do the same thing for free?
>
> If we lose volunteer work for what we get in paid work, the price of
> paid work might well be a lot higher than what it is at first sight.

These are good points that need to be taken into consideration. However, many
other non-profits mix volunteers with paid staff with no ill-effect. So let's
see how they get things to work.

-- mav



__________________________________
Do you Yahoo!?
Read only the mail you want - Yahoo! Mail SpamGuard.
http://promotions.yahoo.com/new_mail
Re: Concern with performance issues [ In reply to ]
Daniel Mayer wrote:
> The issue as I see it, is that we are now asking too much of too few volunteer
> developers and none of them can be on-site at the colo. Thus I think it is the
> time to consider hiring one full time or two of them part time to perform
> server maintenance tasks for us that currently are not done in a timely manor
> (esp the on-site stuff). Our budget is now so big ($US80,000 last quarter) that
> the addition of one or two employees under contract would not be too much of a
> burden. Neither would some up-front relocation money.

We're doing this already, paying the guy who goes by the name "Baylink"
on IRC. More servers would also be nice -- more in every category. We're
having trouble using the hardware we have efficiently due to the lack of
system administrators and developers, but inefficiency can be
compensated for by buying more hardware.

The cheap way to get more hardware is to fix the ~7 machines that are
currently broken and under warranty. Setting up new machines is a
significant time cost to the system administrator team, it would be
better if the existing machines were fixed and put back into service as
soon as possible after breaking. The cluster configuration changes
constantly, if a machine is out of action for long enough, reconfiguring
it to match the rest of the cluster becomes as hard as adding a new machine.

There's not much any of us can do to get machines fixed besides hassle
Jimbo.

I'm told 10 machines were ordered yesterday, that will help. Here are
the details:

http://meta.wikimedia.org/wiki/Hardware_ordered_January_2005

I suspect any difference between the Paris squids and the Florida squids
will disappear when those dual-H/D machines are put into service.
However I don't think it will be long before the apache cluster needs
attention again.

JamesDay is of the opinion that we need two more database slaves worth
$10,000-$15,000 each. I'd add that to a large degree, available database
hardware limits what we can consider implementing in MediaWiki. We
typically live on the breadline, but we can always find a use for more
database hardware.

-- Tim Starling
Re: Re: Concern with performance issues [ In reply to ]
--- Tim Starling <t.starling@physics.unimelb.edu.au> wrote:

>
> The cheap way to get more hardware is to fix the ~7 machines that are
> currently broken and under warranty. Setting up new machines is a
> significant time cost to the system administrator team, it would be
> better if the existing machines were fixed and put back into service
> as
> soon as possible after breaking. The cluster configuration changes
> constantly, if a machine is out of action for long enough,
> reconfiguring
> it to match the rest of the cluster becomes as hard as adding a new
> machine.
>
> There's not much any of us can do to get machines fixed besides
> hassle
> Jimbo.
>
> I'm told 10 machines were ordered yesterday, that will help. Here are
>
> the details:
>
> http://meta.wikimedia.org/wiki/Hardware_ordered_January_2005
>

Perhaps it's time for us to consider on-site service as part of the
warranty package. It is truly dismaying to see that we are ordering 10
new machines when we have ~7 under warranty that need to be repaired.

-Rich Holton
(en.wikipedia:user:Rholton)



__________________________________
Do you Yahoo!?
Take Yahoo! Mail with you! Get it on your mobile phone.
http://mobile.yahoo.com/maildemo
Re: Concern with performance issues [ In reply to ]
On Jan 14.2005, at 09:29, Ray Saintonge wrote:

> After that the person could not be hired again for at least another
> six months. One thing that this would accomplish is that it would
> avoid having people become dependent on the Foundation for a regular
> source of income.

This is actually a very good idea.
Re: Concern with performance issues [ In reply to ]
Why, what is wrong with having "people dedendent on the Foundation for
a regular source of income."? If we were to use only contract workers,
a new person would have to learn the whole system every 3 months, and
it would also make longer term projects more difficult to
do. Contract workers, for sys admin is just crazy talk.

paz y amor,
[[User:The bellman]]

On Sun, 16 Jan 2005 22:49:26 -0800, Scott Nelson <scott@penguinstorm.com> wrote:
> On Jan 14.2005, at 09:29, Ray Saintonge wrote:
>
> > After that the person could not be hired again for at least another
> > six months. One thing that this would accomplish is that it would
> > avoid having people become dependent on the Foundation for a regular
> > source of income.

--
hit me: robin.shannon.id.au
jab me: saudade@jabber.zim.net.au

This work is licensed under the Creative Commons
Recombo Plus License. To view a copy of this license, visit
http://creativecommons.org/licenses/sampling+/1.0/
Re: Concern with performance issues [ In reply to ]
Robin Shannon wrote:

>Why, what is wrong with having "people dedendent on the Foundation for
>a regular source of income."? If we were to use only contract workers,
>a new person would have to learn the whole system every 3 months, and
>it would also make longer term projects more difficult to
>do. Contract workers, for sys admin is just crazy talk.
>
>
>
>
>>On Jan 14.2005, at 09:29, Ray Saintonge wrote:
>>
>>
>>
>>>After that the person could not be hired again for at least another
>>>six months. One thing that this would accomplish is that it would
>>>avoid having people become dependent on the Foundation for a regular
>>>source of income.
>>>
>>>
>
>
>
If anybody is interested, another FOSS project that has been paying cash
for doing development in this fashion is the Freenet Project
(http://freenet.sourceforge.net/). There are some projects that have
"gone commercial" that perhaps could be compared as well, but the point
here is that Wikipedia is growing to the point that having some sort of
professional staff might just be necessary.

I think you can set some guidelines down on who would be elegible for
recieving these funds, including perhaps a requirement that the
developer must have been doing volunteer development work for a given
number of months, nominated by the other developers, or "hired" by the
community in some fashion that has widespread approval that the
individual getting the contract really is worthy of getting paid.
Basically, this is "one of us" that is very good at what they are
doing, and already working with the volunteer developers. Sometimes
this relationship may change with money being involved, but if they
already have good relationship with the other developers before they are
hired, I don't think this would change. It would be a major mistake to
hire from outside of the group of volunteer developers if you intend to
keep the volunteer community together.

As an example, IMHO, of how volunteers have been pushed into a
second-class category under professional staff, I would give the Open
Directory Project (dmoz.org) as an example. This is to show what can go
wrong if professional staff doesn't listen to the community. I have
been a volunteer editor there for almost five years now, and for awhile
there was a huge influx of volunteer "editors" who helped to sift
through web links and edit the descriptions, and create category
classifications to organize the internet. On the whole a rather
ambitious project, and something that I would still like to stay
associated with.

I became a regional editor or a rather large category, and frankly it
was enough to keep me busy just keeping up with all of the work that I
was going through. The problem I was encountering was that the
professional staff was not really "one of us", and often ran roughshod
over the volunteers, including me. I would see changes even to the
portion of the ODP that I was responsible for, with no explaination or
warning that changes were even going to be made. When I would disagree
with the changes, I would be publically ridiculed as not understanding
what was going on, even if other volunteer editors would agree with my
viewpoint. Finally, I was going through huge turnover of volunteers who
were assisting me in sub-categories "under" the one I was working on. I
just stopped working on the project for a few months, in part over my
disgust over what has been going on, and in part due to the fact that
I've had life come up and bite me so I can't put the hours into
volunteer work like I've done in the past. In short, I've been locked
out as a volunteer editor now. Yes, I could reapply, but at this point
it is a barrier where I am not sure if I will ever be associated with
the Open Directory Project again.

Wikipedia is in a similar position where it is growing in huge numbers,
seemingly without end. It also has, for the primary purpose of what it
does, a relatively low barrier to entry for somebody new to come in and
join in the work. From my experience with the Open Directory Project, I
can also point to a time when this growth will end in terms of gaining
new people to write articles. I've also been involved in other
volunteer organizations (political, social, and youth groups) and the
need to keep the volunteers happy should never be underestimated. The
fickle thing about volunteers is that if they don't like what is going
on, they will leave quietly... often without the leaders or those "at
the top" even being able to percieve that there is a problem. If you
hire somebody in any role (developer, PR work, accounting, etc.), you
have to make sure that individual is committed to the goals of the
organization, and if possible would be a volunteer if they would not get
paid for their work. This is true for Wikipedia as it is for the Red Cross.

--
Robert Scott Horning
218 Sunstone Circle
Logan, UT 84321
(435) 753-3330
robert_horning@netzero.net
Re: Concern with performance issues [ In reply to ]
Robert Scott Horning (robert_horning@netzero.net) [050117 22:56]:

> I've also been involved in other
> volunteer organizations (political, social, and youth groups) and the
> need to keep the volunteers happy should never be underestimated. The
> fickle thing about volunteers is that if they don't like what is going
> on, they will leave quietly... often without the leaders or those "at
> the top" even being able to percieve that there is a problem. If you
> hire somebody in any role (developer, PR work, accounting, etc.), you
> have to make sure that individual is committed to the goals of the
> organization, and if possible would be a volunteer if they would not get
> paid for their work. This is true for Wikipedia as it is for the Red Cross.


I have considerable experience in managing volunteers (both as a volunteer
and being paid to manage them). One question I'd like a better answer to is
how to keep volunteers from each others' throats better ... en: is large
enough to be qualitatively different from any other wiki, anywhere, ever.


- d.
Re: Concern with performance issues [ In reply to ]
David Gerard wrote:

>Robert Scott Horning (robert_horning@netzero.net) [050117 22:56]:
>
>
>> I've also been involved in other
>>volunteer organizations (political, social, and youth groups) and the
>>need to keep the volunteers happy should never be underestimated. The
>>fickle thing about volunteers is that if they don't like what is going
>>on, they will leave quietly... often without the leaders or those "at
>>the top" even being able to percieve that there is a problem. If you
>>hire somebody in any role (developer, PR work, accounting, etc.), you
>>have to make sure that individual is committed to the goals of the
>>organization, and if possible would be a volunteer if they would not get
>>paid for their work. This is true for Wikipedia as it is for the Red Cross.
>>
>>
>I have considerable experience in managing volunteers (both as a volunteer
>and being paid to manage them). One question I'd like a better answer to is
>how to keep volunteers from each others' throats better ... en: is large
>enough to be qualitatively different from any other wiki, anywhere, ever.
>
:-D Let's face it, we're in uncharted territory. The number of
volunteers is huge, and each one has considerably more power than a
volunteer in most much smaller organizations. Many have strongly
opposing (usually sincere) views of what is right, and are totally
convinced that any other position is idiotic. Even our management
philosophies are radically different, running the entire range from
laissez-faire permissiveness to literal totalitarian strictness.

With apologies for this note of pessimism -- Good luck!

Ec
Re: Concern with performance issues [ In reply to ]
Robin Shannon wrote:

>Why, what is wrong with having "people dependent on the Foundation for
>a regular source of income."? If we were to use only contract workers,
>a new person would have to learn the whole system every 3 months, and
>it would also make longer term projects more difficult to
>do. Contract workers, for sys admin is just crazy talk.
>
>On Sun, 16 Jan 2005 22:49:26 -0800, Scott Nelson <scott@penguinstorm.com> wrote:
>
>
>>On Jan 14.2005, at 09:29, Ray Saintonge wrote:
>>
>>
>>>After that the person could not be hired again for at least another
>>>six months. One thing that this would accomplish is that it would
>>>avoid having people become dependent on the Foundation for a regular
>>>source of income.
>>>
If you are looking at this from a strictly logical mechanistic
perspective, you are of course right. Certain efficiences are a normal
by-product of system experience. This all presupposes that the employee
has a very free hand in the decisions that he must make, and that his
orders come from a single responsible source.

Wikipedia is as much a human environment as it is a technical one. Not
every member of a human environment is tolerant of being told by some
Vulcan that he is not being logical. Persons making decisions that will
have long-term effects on the community need to retain the confidence of
the community on a continuing basis. Since the majority of us are
clueless about the functioning of the underlying software we are content
to let the developers "do their thing", and to content ourselves with an
accasional whine about a bug in the system or a fantastic
"Wouldn't-it-be-great-if..." comment. Beyond that we will keep quiet
about the software unless our activities are seriously cramped.

At the same time the employee will be under pressure to produce, even if
that pressure only comes from within himself. Contracts to perform
specific pre-determined tasks that have already been approved by the
community will allow the person to focus on the task at hand, without
the need to schmooz for the purpose of continuing his employment beyond
the contract period.

Ec
Re: Concern with performance issues [ In reply to ]
--- Robin Shannon <robin.shannon@gmail.com> wrote:
> Why, what is wrong with having "people dedendent on the Foundation for
> a regular source of income."? If we were to use only contract workers,
> a new person would have to learn the whole system every 3 months, and
> it would also make longer term projects more difficult to
> do. Contract workers, for sys admin is just crazy talk.

In practice such a person would almost certainly get his/her contract renewed
each time. The contract term for each renewal would also likely get longer and
longer as we obtain more and more secure funding sources. In time we may even
be in a position to have real employees with benefits.

-- mav

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: Concern with performance issues [ In reply to ]
Robert Scott Horning wrote:

> Robin Shannon wrote:
>
>> Why, what is wrong with having "people dedendent on the Foundation for
>> a regular source of income."? If we were to use only contract workers,
>> a new person would have to learn the whole system every 3 months, and
>> it would also make longer term projects more difficult to
>> do. Contract workers, for sys admin is just crazy talk.
>>
>>> On Jan 14.2005, at 09:29, Ray Saintonge wrote:
>>>
>>>> After that the person could not be hired again for at least another
>>>> six months. One thing that this would accomplish is that it would
>>>> avoid having people become dependent on the Foundation for a regular
>>>> source of income.
>>>
>> If anybody is interested, another FOSS project that has been paying
>> cash for doing development in this fashion is the Freenet Project
>> (http://freenet.sourceforge.net/). There are some projects that have
>> "gone commercial" that perhaps could be compared as well, but the
>> point here is that Wikipedia is growing to the point that having some
>> sort of professional staff might just be necessary.
>
> I think you can set some guidelines down on who would be elegible for
> recieving these funds, including perhaps a requirement that the
> developer must have been doing volunteer development work for a given
> number of months, nominated by the other developers, or "hired" by the
> community in some fashion that has widespread approval that the
> individual getting the contract really is worthy of getting paid.
> Basically, this is "one of us" that is very good at what they are
> doing, and already working with the volunteer developers. Sometimes
> this relationship may change with money being involved, but if they
> already have good relationship with the other developers before they
> are hired, I don't think this would change. It would be a major
> mistake to hire from outside of the group of volunteer developers if
> you intend to keep the volunteer community together.
>
> As an example, IMHO, of how volunteers have been pushed into a
> second-class category under professional staff, I would give the Open
> Directory Project (dmoz.org) as an example. This is to show what can
> go wrong if professional staff doesn't listen to the community. I
> have been a volunteer editor there for almost five years now, and for
> awhile there was a huge influx of volunteer "editors" who helped to
> sift through web links and edit the descriptions, and create category
> classifications to organize the internet. On the whole a rather
> ambitious project, and something that I would still like to stay
> associated with.
> I became a regional editor or a rather large category, and frankly it
> was enough to keep me busy just keeping up with all of the work that I
> was going through. The problem I was encountering was that the
> professional staff was not really "one of us", and often ran roughshod
> over the volunteers, including me. I would see changes even to the
> portion of the ODP that I was responsible for, with no explaination or
> warning that changes were even going to be made. When I would
> disagree with the changes, I would be publically ridiculed as not
> understanding what was going on, even if other volunteer editors would
> agree with my viewpoint. Finally, I was going through huge turnover
> of volunteers who were assisting me in sub-categories "under" the one
> I was working on. I just stopped working on the project for a few
> months, in part over my disgust over what has been going on, and in
> part due to the fact that I've had life come up and bite me so I can't
> put the hours into volunteer work like I've done in the past. In
> short, I've been locked out as a volunteer editor now. Yes, I could
> reapply, but at this point it is a barrier where I am not sure if I
> will ever be associated with the Open Directory Project again.
>
> Wikipedia is in a similar position where it is growing in huge
> numbers, seemingly without end. It also has, for the primary purpose
> of what it does, a relatively low barrier to entry for somebody new to
> come in and join in the work. From my experience with the Open
> Directory Project, I can also point to a time when this growth will
> end in terms of gaining new people to write articles. I've also been
> involved in other volunteer organizations (political, social, and
> youth groups) and the need to keep the volunteers happy should never
> be underestimated. The fickle thing about volunteers is that if they
> don't like what is going on, they will leave quietly... often without
> the leaders or those "at the top" even being able to percieve that
> there is a problem. If you hire somebody in any role (developer, PR
> work, accounting, etc.), you have to make sure that individual is
> committed to the goals of the organization, and if possible would be a
> volunteer if they would not get paid for their work. This is true for
> Wikipedia as it is for the Red Cross.

Your observations are important, and reflect the reality of
organizations that depend significantly on the work of volunteers.
There is a tremendous difference between all volunteer environments, and
those with regular employees. At the same time their is evident need
that some tasks MUST be done by someone who must go beyond what we can
expect of the most competent of our volunteers. When I go on long road
trips I make a point of detouring to see small town museums. I've seen
two types with significant differences. One is owned and significantly
funded by senior governments which then hires people to do the day to
day management. Everything on display is well-organized and documented;
the facility is well-maintained and free of safety hazards; it is clean
to the point of sterility. Contrast that with a privately owned museum
with minimal support from public funds. The exhibits are chaos and
clutter with minimal documentation, buildings are often run-down
fire-hazards; one wonders what else is breeding in the exhibits besides
interest. A commercial parallel to this situation might be between the
highly structured 7-11 operation and the mom-and-pop convenience store.

Full time employees profoundly alter the nature of an organization. It
begins to put the needs of the employees above the needs of its
mission. It becomes caught up in a succession of statutory obligations
regarding the treatment of employees. The employees are there "all the
time", and consequently become more familiar with the operation than the
volunteers, a point which can be used as evidence to support their way
of doing things. Funding decisions become dominated by making sure that
there is enough in the pot to pay salaries.

If the wiki way implies promoting the right to edit as fundamental to
academic freedom of the entire population, we also need to examine
closely those other social structures that can have a bearing on that
right.

Ec
Re: Concern with performance issues [ In reply to ]
Ray Saintonge a écrit:
> David Gerard wrote:

> :-D Let's face it, we're in uncharted territory. The number of
> volunteers is huge, and each one has considerably more power than a
> volunteer in most much smaller organizations. Many have strongly
> opposing (usually sincere) views of what is right, and are totally
> convinced that any other position is idiotic. Even our management
> philosophies are radically different, running the entire range from
> laissez-faire permissiveness to literal totalitarian strictness.
>
> With apologies for this note of pessimism -- Good luck!
>
> Ec

Hmmm, this raise a immediate comment.

I am supportive of hiring someone to help, in particular to do all the
admin basic and boring work as well as helping on site in Florida. This
would relieve both Jimbo who, ihmo, can help more in other areas, and
relieve current developers from day to day work and let them focus on
improving the whole system.

I have been wondering if it was a better idea in the long run of hiring
current developers or total or need total strangers to the community. I
am undecided on this. But one thing appearing obvious and potentially
problematic to me would be that the employed would have to follow a lot
the community opinion, ie opinion of non employed. I am curious to see
that admitedly :-)

Aside from this point, a little precision :

No developer is currently employed by the Foundation. Yet.
Jimbo has been doing some interviews in that sense.

Ant
Re: Concern with performance issues [ In reply to ]
Ray Saintonge (saintonge@telus.net) [050118 07:20]:
> Robin Shannon wrote:

> >Why, what is wrong with having "people dependent on the Foundation for
> >a regular source of income."? If we were to use only contract workers,
> >a new person would have to learn the whole system every 3 months, and
> >it would also make longer term projects more difficult to
> >do. Contract workers, for sys admin is just crazy talk.

> If you are looking at this from a strictly logical mechanistic
> perspective, you are of course right. Certain efficiences are a normal
> by-product of system experience. This all presupposes that the employee
> has a very free hand in the decisions that he must make, and that his
> orders come from a single responsible source.
> Wikipedia is as much a human environment as it is a technical one. Not
[....]
> At the same time the employee will be under pressure to produce, even if
> that pressure only comes from within himself. Contracts to perform
> specific pre-determined tasks that have already been approved by the
> community will allow the person to focus on the task at hand, without
> the need to schmooz for the purpose of continuing his employment beyond
> the contract period.


If the planned employee is a sysadmin, I can tell you (as a sysadmin who
knows *lots* of sysadmins) that there are any number of highly skilled and
professional [[BOFH]]s who would *jump* at the chance to tend machines for
the Foundation. And be clueful about the Wiki way in all its interesting
glory. These people have survived Usenet, after all. Now you just need one
in St Petersburg :-D

(Dev is not my field so I cannot offer free advice worth every penny there
;-)


- d.
Re: Concern with performance issues [ In reply to ]
Two questions; does anyone have first hand knowledge of what they have
done on other major open-source projects (ie linux, openoffice.org,
mozzila etc?) How do volunteers and non-voluteers get on there. Also
if tim or brion, or any of the other really core developers are
reading this thread what are thier thoughts (as they are the ones who
will be most affected).

and finnaly an observation; contracts might be right if we were
employing them to do a specific thing, but i see that something far
more useful, would be someone whos core purpose was a facilitator. as
in, they would be making sure the 7 machines which are currently out
are working, that all the hardware is documented, that what ever the
latest emergency is, does not affect the editors, or take up developer
time. For this a more permanent type of employment is useful.
Obviously the person, should share a belief in our mission (and the
general opensource mission), and perhaps, they should be employed on a
contract for the first month or so to see if we are happy with them or
not, but after that a wage would be best.

paz y amor,
[[User:The bellman]]


On Tue, 18 Jan 2005 09:17:04 +1100, David Gerard
<fun@thingy.apana.org.au> wrote:
> If the planned employee is a sysadmin, I can tell you (as a sysadmin who
> knows *lots* of sysadmins) that there are any number of highly skilled and
> professional [[BOFH]]s who would *jump* at the chance to tend machines for
> the Foundation. And be clueful about the Wiki way in all its interesting
> glory. These people have survived Usenet, after all. Now you just need one
> in St Petersburg :-D
>
> (Dev is not my field so I cannot offer free advice worth every penny there
> ;-)
>
>
> - d.

--
hit me: robin.shannon.id.au
jab me: saudade@jabber.zim.net.au

This work is licensed under the Creative Commons
Recombo Plus License. To view a copy of this license, visit
http://creativecommons.org/licenses/sampling+/1.0/
Re: Re: Concern with performance issues [ In reply to ]
Anthere wrote:

> Ray Saintonge a écrit:
>
>> :-D Let's face it, we're in uncharted territory. The number of
>> volunteers is huge, and each one has considerably more power than a
>> volunteer in most much smaller organizations. Many have strongly
>> opposing (usually sincere) views of what is right, and are totally
>> convinced that any other position is idiotic. Even our management
>> philosophies are radically different, running the entire range from
>> laissez-faire permissiveness to literal totalitarian strictness.
>>
>> With apologies for this note of pessimism -- Good luck!
>>
>> Ec
>
> Hmmm, this raise a immediate comment.
>
> I am supportive of hiring someone to help, in particular to do all the
> admin basic and boring work as well as helping on site in Florida.
> This would relieve both Jimbo who, ihmo, can help more in other areas,
> and relieve current developers from day to day work and let them focus
> on improving the whole system.

Yes, like promotion, speaking engagenents, visiting local groups, and
all those other corporate executive duties. The travel involved is
incompatible with the need to have someone available close by to respond
to emergencies. I don't disagree with the needs assessment, which seems
to focus on the hardware rather than the software end of things.

> I have been wondering if it was a better idea in the long run of
> hiring current developers or total or need total strangers to the
> community. I am undecided on this. But one thing appearing obvious and
> potentially problematic to me would be that the employed would have to
> follow a lot the community opinion, ie opinion of non employed. I am
> curious to see that admitedly :-)

The community policy pressures are likely to be less on hardware
maintenance than on software development. The sign of a good job on the
hardware end may come when nobody notices him and everything runs
smoothly. The other issue about hardware maintenance is whether it
requires a full time person or someone who comes in maybe once a week
for routine maintenance, and is otherwise on call. For such a hardware
based position being available may be more important than community
involvement.

For software specific duties it may be more important to draw from
community members who already understand the wiki process. They need to
be diplomatic and thick-skinned at the same time. Short term contract
based hiring may be more important here, if they are ever to be
re-integrated in the general volunteer community after the contract is
finished.

Ec