Mailing List Archive

Ø Because myself and others have been frustrated by the lack of good

Ø stats on the number of active editors on the English Wikipedia, I have

Ø compiled some stats on the editing frequency on enwiki:



No worries: in only 176 days from now the English will be ready and I can
run wikistats scripts on it.

It just started 52 days ago, so let us be patient for a while ;)



http://www.infodisiac.com/cgi-bin/WikimediaDownload.pl



Nice stats though!



Erik Zachte











_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

English -> English dump

> Because myself and others have been frustrated by the lack of good
> stats on the number of active editors on the English Wikipedia, I have
> compiled some stats on the editing frequency on enwiki:

No worries: in only 176 days from now the English dump will be ready and I
can run wikistats scripts on it.
It just started 52 days ago, so let us be patient for a while ;)

http://www.infodisiac.com/cgi-bin/WikimediaDownload.pl

Nice stats though!

Erik Zachte








_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Wow, someone had more than 10,000 edits in February of 2002.

Does it look to anyone else like the first five months of 2007 and 2008 were
very busy, followed by a drop for the rest of the year? If that is whats
happened, any theories as to why?

Nathan

On Sun, Nov 30, 2008 at 5:32 AM, Robert Rohde <rarohde@gmail.com> wrote:

> Because myself and others have been frustrated by the lack of good
> stats on the number of active editors on the English Wikipedia, I have
> compiled some stats on the editing frequency on enwiki:
>
> http://en.wikipedia.org/wiki/Wikipedia:Editing_frequency
>
> I am going to forgo any extensive analysis for now. But I will say
> that these trends mostly mirror trends seen elsewhere, with a peak in
> early 2007 followed by a decline and then leveling out as we go
> towards the present.
>
> In September, 130,000 registered users and 525,000 anons made at least
> one edit to an article. If you define "active editors" as those
> making at least 20 article edits per month then 14000 registered users
> and 6000 anons met that threshold in September.
>
> -Robert Rohde
>
> _______________________________________________
> foundation-l mailing list
> foundation-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>



--
Your donations keep Wikipedia running! Support the Wikimedia Foundation
today: http://www.wikimediafoundation.org/wiki/Donate
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sun, Nov 30, 2008 at 1:12 PM, Nathan <nawrich@gmail.com> wrote:
> Wow, someone had more than 10,000 edits in February of 2002.
>
> Does it look to anyone else like the first five months of 2007 and 2008 were
> very busy, followed by a drop for the rest of the year? If that is whats
> happened, any theories as to why?
>
> Nathan

Summer break for students would be the obvious reason.

Or just good weather, generally.

You might find the inverse if you look only at Southern Hemisphere IPs.

Thanks,
Pharos

> On Sun, Nov 30, 2008 at 5:32 AM, Robert Rohde <rarohde@gmail.com> wrote:
>
>> Because myself and others have been frustrated by the lack of good
>> stats on the number of active editors on the English Wikipedia, I have
>> compiled some stats on the editing frequency on enwiki:
>>
>> http://en.wikipedia.org/wiki/Wikipedia:Editing_frequency
>>
>> I am going to forgo any extensive analysis for now. But I will say
>> that these trends mostly mirror trends seen elsewhere, with a peak in
>> early 2007 followed by a decline and then leveling out as we go
>> towards the present.
>>
>> In September, 130,000 registered users and 525,000 anons made at least
>> one edit to an article. If you define "active editors" as those
>> making at least 20 article edits per month then 14000 registered users
>> and 6000 anons met that threshold in September.
>>
>> -Robert Rohde
>>
>> _______________________________________________
>> foundation-l mailing list
>> foundation-l@lists.wikimedia.org
>> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>>
>
>
>
> --
> Your donations keep Wikipedia running! Support the Wikimedia Foundation
> today: http://www.wikimediafoundation.org/wiki/Donate
> _______________________________________________
> foundation-l mailing list
> foundation-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

If that were the major reason, wouldn't you expect to see a return to former
levels in the last four months?

Nathan

On Sun, Nov 30, 2008 at 1:17 PM, Pharos <pharosofalexandria@gmail.com>wrote:

> On Sun, Nov 30, 2008 at 1:12 PM, Nathan <nawrich@gmail.com> wrote:
> > Wow, someone had more than 10,000 edits in February of 2002.
> >
> > Does it look to anyone else like the first five months of 2007 and 2008
> were
> > very busy, followed by a drop for the rest of the year? If that is whats
> > happened, any theories as to why?
> >
> > Nathan
>
> Summer break for students would be the obvious reason.
>
> Or just good weather, generally.
>
> You might find the inverse if you look only at Southern Hemisphere IPs.
>
> Thanks,
> Pharos
>
>
>
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sun, Nov 30, 2008 at 1:24 PM, Nathan <nawrich@gmail.com> wrote:
> If that were the major reason, wouldn't you expect to see a return to former
> levels in the last four months?
>
> Nathan

Not necessarily. We have to think about psychological dynamics here.

It may well be the case that many students "quit" Wikipedia for the
summer, and only take it up again after winter break, when they are
more settled into their academic routine than in the fall.

Thanks,
Pharos

> On Sun, Nov 30, 2008 at 1:17 PM, Pharos <pharosofalexandria@gmail.com>wrote:
>
>> On Sun, Nov 30, 2008 at 1:12 PM, Nathan <nawrich@gmail.com> wrote:
>> > Wow, someone had more than 10,000 edits in February of 2002.
>> >
>> > Does it look to anyone else like the first five months of 2007 and 2008
>> were
>> > very busy, followed by a drop for the rest of the year? If that is whats
>> > happened, any theories as to why?
>> >
>> > Nathan
>>
>> Summer break for students would be the obvious reason.
>>
>> Or just good weather, generally.
>>
>> You might find the inverse if you look only at Southern Hemisphere IPs.
>>
>> Thanks,
>> Pharos
>>
>>
>>
> _______________________________________________
> foundation-l mailing list
> foundation-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sun, Nov 30, 2008 at 8:20 AM, Erik Zachte <erikzachte@infodisiac.com> wrote:
> English -> English dump
>
>> Because myself and others have been frustrated by the lack of good
>> stats on the number of active editors on the English Wikipedia, I have
>> compiled some stats on the editing frequency on enwiki:
>
> No worries: in only 176 days from now the English dump will be ready and I
> can run wikistats scripts on it.
> It just started 52 days ago, so let us be patient for a while ;)

Is there any reason at all to believe that it is more likely to finish
this time than all the previous attempts during the last two years?

I have virtually zero faith in a script that takes 230 days and where
any error wipes out all progress.

-Robert Rohde

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On 30 Nov 2008, at 20:11, Robert Rohde wrote:

> On Sun, Nov 30, 2008 at 8:20 AM, Erik Zachte
> <erikzachte@infodisiac.com> wrote:
>> English -> English dump
>>
>>> Because myself and others have been frustrated by the lack of good
>>> stats on the number of active editors on the English Wikipedia, I
>>> have
>>> compiled some stats on the editing frequency on enwiki:
>>
>> No worries: in only 176 days from now the English dump will be
>> ready and I
>> can run wikistats scripts on it.
>> It just started 52 days ago, so let us be patient for a while ;)
>
> Is there any reason at all to believe that it is more likely to finish
> this time than all the previous attempts during the last two years?
>
> I have virtually zero faith in a script that takes 230 days and where
> any error wipes out all progress.
>
> -Robert Rohde

Hold on...what? There is no recent dump of the English Wikipedia, and
there hasn't been for the last 2 years?

Please tell me I'm misunderstanding things here.

Mike

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sun, Nov 30, 2008 at 3:35 PM, Michael Peel <email@mikepeel.net> wrote:

>
> On 30 Nov 2008, at 20:11, Robert Rohde wrote:
>
> > On Sun, Nov 30, 2008 at 8:20 AM, Erik Zachte
> > <erikzachte@infodisiac.com> wrote:
> >> English -> English dump
> >>
> >>> Because myself and others have been frustrated by the lack of good
> >>> stats on the number of active editors on the English Wikipedia, I
> >>> have
> >>> compiled some stats on the editing frequency on enwiki:
> >>
> >> No worries: in only 176 days from now the English dump will be
> >> ready and I
> >> can run wikistats scripts on it.
> >> It just started 52 days ago, so let us be patient for a while ;)
> >
> > Is there any reason at all to believe that it is more likely to finish
> > this time than all the previous attempts during the last two years?
> >
> > I have virtually zero faith in a script that takes 230 days and where
> > any error wipes out all progress.
> >
> > -Robert Rohde
>
> Hold on...what? There is no recent dump of the English Wikipedia, and
> there hasn't been for the last 2 years?
>
> Please tell me I'm misunderstanding things here.
>
> Mike
>
> _______________________________________________
> foundation-l mailing list
> foundation-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>

(cc'd to wikitech-l)
I saw this the other day as well and found it odd. While enwiki dumps
do take the longest, this does seem like an _incredibly_ long time for
"All pages with complete page edit history (.bz2)" to finish (May 2009).

-Chad
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

2008/11/30 Michael Peel <email@mikepeel.net>:
> Hold on...what? There is no recent dump of the English Wikipedia, and
> there hasn't been for the last 2 years?
>
> Please tell me I'm misunderstanding things here.
>
> Mike

Not to years but yes the English Wikipedia dumps very rarely work.
De.wikipedia is starting to suffer the same issues and image database
dumps don't happen.


--
geni

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

> I saw this the other day as well and found it odd. While enwiki dumps
> do take the longest, this does seem like an _incredibly_ long time for
> "All pages with complete page edit history (.bz2)" to finish (May 2009).

Do you know how many pages enwiki has and how much edit history they
each have? It's a lot!

I think the dumps work by starting with the last successful dump and
just adding in anything that's changed, but because there haven't been
any successful dumps of the whole of enwiki in a long time, it
basically has to start from scratch, which is going to take a long
time (and means it probably won't succeed - ie. we have a catch-22).
It seems to me that (if my understanding of the problem is correct),
the answer is to devote a more powerful computer to the dump for just
this one so that we can get things moving again - I'm sure if we asked
around someone could lend us a really powerful computer for a few
weeks to do the dump on.

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sun, Nov 30, 2008 at 12:58 PM, Thomas Dalton <thomas.dalton@gmail.com> wrote:
>> I saw this the other day as well and found it odd. While enwiki dumps
>> do take the longest, this does seem like an _incredibly_ long time for
>> "All pages with complete page edit history (.bz2)" to finish (May 2009).
>
> Do you know how many pages enwiki has and how much edit history they
> each have? It's a lot!
>
> I think the dumps work by starting with the last successful dump and
> just adding in anything that's changed, but because there haven't been
> any successful dumps of the whole of enwiki in a long time, it
> basically has to start from scratch, which is going to take a long
> time (and means it probably won't succeed - ie. we have a catch-22).
> It seems to me that (if my understanding of the problem is correct),
> the answer is to devote a more powerful computer to the dump for just
> this one so that we can get things moving again - I'm sure if we asked
> around someone could lend us a really powerful computer for a few
> weeks to do the dump on.

No, dumps are total, not incremental.

It is really more than throwing a big computer at it. The dumping
process ought to be redesigned to be more fault tolerant and faster.
It is ridiculous to have a process that is expected to take months and
yet have no method of saving one's progress as it goes and restarting
it in case of trouble.

-Robert Rohde

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sun, Nov 30, 2008 at 12:56 PM, geni <geniice@gmail.com> wrote:
> 2008/11/30 Michael Peel <email@mikepeel.net>:
>> Hold on...what? There is no recent dump of the English Wikipedia, and
>> there hasn't been for the last 2 years?
>>
>> Please tell me I'm misunderstanding things here.
>>
>> Mike
>
> Not to years but yes the English Wikipedia dumps very rarely work.
> De.wikipedia is starting to suffer the same issues and image database
> dumps don't happen.

No, I think it really has been two years (September 2006 to be
precise). I'm pretty sure there have been no complete dumps this
calendar year. I believe there was one (or two?) full dump process
that claimed to run to completion in 2007 but it was later found to
have been truncated (i.e. it didn't really dump all of enwiki, only a
portion of it).

If I am wrong and there really is a more recent complete history dump
of enwiki floating around somewhere, then I'd love to hear about it,
but I don't believe that is the case.

-Robert Rohde

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

>> Not to years but yes the English Wikipedia dumps very rarely work.
>> De.wikipedia is starting to suffer the same issues and image database
>> dumps don't happen.
>
> No, I think it really has been two years (September 2006 to be
> precise). I'm pretty sure there have been no complete dumps this
> calendar year. I believe there was one (or two?) full dump process
> that claimed to run to completion in 2007 but it was later found to
> have been truncated (i.e. it didn't really dump all of enwiki, only a
> portion of it).
>
> If I am wrong and there really is a more recent complete history dump
> of enwiki floating around somewhere, then I'd love to hear about it,
> but I don't believe that is the case.

That sounds about right to me. I think the confusion may come from
there being lots of different dumps - the smaller dumps of enwiki do
succeed (occasionally, at least!), it's the full dump of every
revision of every page that fails routinely.

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

> No, dumps are total, not incremental.

The dumps are, yes, but what about the creation of the dumps? My
understanding was that they are created by taking an old dump and
adding to it.

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sun, Nov 30, 2008 at 2:39 PM, Thomas Dalton <thomas.dalton@gmail.com> wrote:
>> No, dumps are total, not incremental.
>
> The dumps are, yes, but what about the creation of the dumps? My
> understanding was that they are created by taking an old dump and
> adding to it.

No. Each dump process starts fresh from the database, they don't add
to old dumps. (Or at least they certainly didn't the last time I
looked at the dumping scripts, and I have no reason to suspect that
has changed.)

-Robert

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Thomas Dalton wrote:
>>> Not to years but yes the English Wikipedia dumps very rarely work.
>>> De.wikipedia is starting to suffer the same issues and image database
>>> dumps don't happen.
>>>
>> No, I think it really has been two years (September 2006 to be
>> precise). I'm pretty sure there have been no complete dumps this
>> calendar year. I believe there was one (or two?) full dump process
>> that claimed to run to completion in 2007 but it was later found to
>> have been truncated (i.e. it didn't really dump all of enwiki, only a
>> portion of it).
>>
>> If I am wrong and there really is a more recent complete history dump
>> of enwiki floating around somewhere, then I'd love to hear about it,
>> but I don't believe that is the case.
>>
>
> That sounds about right to me. I think the confusion may come from
> there being lots of different dumps - the smaller dumps of enwiki do
> succeed (occasionally, at least!), it's the full dump of every
> revision of every page that fails routinely.
>
> _______________________________________________
> foundation-l mailing list
> foundation-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
>
>
>
Is the data replicated anywhere outside the Tampa data centre (such as
in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could
destroy the entire en: Wikipedia.

-- Neil


_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

2008/12/1 Neil Harris <usenet@tonal.clara.co.uk>:
> Is the data replicated anywhere outside the Tampa data centre (such as
> in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could
> destroy the entire en: Wikipedia.

En probably isn't that bad. We have lost commons images already.

--
geni

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet@tonal.clara.co.uk> wrote:
> Is the data replicated anywhere outside the Tampa data centre (such as
> in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could
> destroy the entire en: Wikipedia.

There are database mirrors of every wiki, including en, as part of the
toolserver cluster in Amsterdam.

-Robert Rohde

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

It is good to hear that en: is replicated. I assume this also applies to
commons?

Still, it might be a good idea to think about a redesign of the dump
process.



On Mon, Dec 1, 2008 at 4:09 AM, Robert Rohde <rarohde@gmail.com> wrote:

> On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet@tonal.clara.co.uk>
> wrote:
> > Is the data replicated anywhere outside the Tampa data centre (such as
> > in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could
> > destroy the entire en: Wikipedia.
>
> There are database mirrors of every wiki, including en, as part of the
> toolserver cluster in Amsterdam.
>
> -Robert Rohde
>
> _______________________________________________
> foundation-l mailing list
> foundation-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sun, Nov 30, 2008 at 8:04 PM, teun spaans <teun.spaans@gmail.com> wrote:
> It is good to hear that en: is replicated. I assume this also applies to
> commons?

The database yes. For images there is as-needed caching in Amsterdam
and Korea, but my understanding is that the only complete copies of
the file store all live in Florida.

On a related point, replicated mirrors deal well enough with acts of
God (e.g. if a meteor hits the data center in Florida), but because
everything is live mirrored it doesn't deal well if the server is
accidentally instructed to delete something. That erroneous
instruction would be carried to all the mirrors as well, and you might
really lose something before the links could be severed. We really
need to have periodic stored backups in addition to the replicated
mirrors.

-Robert Rohde

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Monday 01 December 2008 04:09:11 Robert Rohde wrote:
> On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet@tonal.clara.co.uk>
wrote:
> > Is the data replicated anywhere outside the Tampa data centre (such as
> > in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could
> > destroy the entire en: Wikipedia.
>
> There are database mirrors of every wiki, including en, as part of the
> toolserver cluster in Amsterdam.

Unfortunately, enwiki mirror doesn't include article text :(

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Sun, Nov 30, 2008 at 10:29 PM, Nikola Smolenski <smolensk@eunet.yu> wrote:
> On Monday 01 December 2008 04:09:11 Robert Rohde wrote:
>> On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet@tonal.clara.co.uk>
> wrote:
>> > Is the data replicated anywhere outside the Tampa data centre (such as
>> > in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could
>> > destroy the entire en: Wikipedia.
>>
>> There are database mirrors of every wiki, including en, as part of the
>> toolserver cluster in Amsterdam.
>
> Unfortunately, enwiki mirror doesn't include article text :(

Ouch, I hadn't realized they gave up on text replication. Apparently
quite a while ago too. (That will teach me for never bothering to
learn to use the toolserver.)

So I guess we are back to the meteor impact destroys Wikipedia scenario.

-Robert Rohde

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Robert Rohde wrote:
> On Sun, Nov 30, 2008 at 10:29 PM, Nikola Smolenski <smolensk@eunet.yu> wrote:
>
>> On Monday 01 December 2008 04:09:11 Robert Rohde wrote:
>>
>>> On Sun, Nov 30, 2008 at 6:16 PM, Neil Harris <usenet@tonal.clara.co.uk>
>>>
>> wrote:
>>
>>>> Is the data replicated anywhere outside the Tampa data centre (such as
>>>> in Amsterdam or Seoul)? If not, just one fire, flood or hurricane could
>>>> destroy the entire en: Wikipedia.
>>>>
>>> There are database mirrors of every wiki, including en, as part of the
>>> toolserver cluster in Amsterdam.
>>>
>> Unfortunately, enwiki mirror doesn't include article text :(
>>
>
> Ouch, I hadn't realized they gave up on text replication. Apparently
> quite a while ago too. (That will teach me for never bothering to
> learn to use the toolserver.)
>
> So I guess we are back to the meteor impact destroys Wikipedia scenario.
>
> -Robert Rohde
>
That's scary: are there any off-site backups of the full database,
including the article text, made anywhere, other than the constantly
failing dumps? Given that Wikipedia is the Wikimedia Foundation's
principal asset, I would hope that fixing this single point of failure
would be a priority for the Foundation.

-- Neil


_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l