Mailing List Archive

Bot approval request
I'm not sure if this is the right place to ask, but I've just written a bot,
and Wikipedia policy says it should be approved before I set it loose.

I recently moved [[Brisbane, Queensland]] to [[Brisbane]], and then started
fixing the links to those pages. I got through about 100 pages and I wasn't
even a third of the way there, and I though "it would be easier to just
write a bot". So I did.

The bot takes a list of article titles, and runs a Perl search and replace
operation on the HTML escaped wikitext. The idea is to manually extract the
list of titles from Special:Whatlinkshere, and then replace all links to the
old page with links to the new page.

It will be running from username "Timbot". I've tested it on
test.wikipedia.org. It runs in "trickle" mode, i.e. like this:

while (<ARTICLES>) {
chomp($_);
DoPage($_);
if (time - $lastTime < $delay) {
sleep($delay - $lastTime + $time);
}
$lastTime = time;
}

I'm open to suggestions as to the setting of $delay. If anyone wants to see
the rest of the code, I'll send it by private email.

-- Tim Starling.


_________________________________________________________________
MSN Instant Messenger now available on Australian mobile phones. Go to
http://ninemsn.com.au/mobilecentral/hotmail_messenger.asp
Re: Bot approval request [ In reply to ]
On 3/28/03 8:15 PM, "Tim Starling" <ts4294967296@hotmail.com> wrote:

> I'm not sure if this is the right place to ask, but I've just written a bot,
> and Wikipedia policy says it should be approved before I set it loose.
>
> I recently moved [[Brisbane, Queensland]] to [[Brisbane]], and then started
> fixing the links to those pages. I got through about 100 pages and I wasn't
> even a third of the way there, and I though "it would be easier to just
> write a bot". So I did.
>
> The bot takes a list of article titles, and runs a Perl search and replace
> operation on the HTML escaped wikitext. The idea is to manually extract the
> list of titles from Special:Whatlinkshere, and then replace all links to the
> old page with links to the new page.
>
> It will be running from username "Timbot". I've tested it on
> test.wikipedia.org. It runs in "trickle" mode, i.e. like this:
>
Is it just for Brisbane, Queensland?

An easier thing to do might be to request the change be made from the
backend instead.
Re: Bot approval request [ In reply to ]
>Is it just for Brisbane, Queensland?
>
>An easier thing to do might be to request the change be made from the
>backend instead.

No, it's not just for Brisbane, Queensland, it's for whatever I need it for
in the future. In the near future, there's a few other cities and towns I
want to move. And it wouldn't be easier in terms of human effort, since the
bot's already written and ready to go. It's only ~70 lines -- it's rather
scary how easy these things are, especially since it's one of the first
things I've ever written in perl and it only took me a few hours. Of course
there's the issue of swamping RC -- if someone is happy to do it from the
backend to reduce the annoyance level, then that's alright with me.

-- Tim Starling


_________________________________________________________________
MSN Instant Messenger now available on Australian mobile phones. Go to
http://ninemsn.com.au/mobilecentral/hotmail_messenger.asp
Re: Bot approval request [ In reply to ]
On Fri, 2003-03-28 at 20:47, Tim Starling wrote:
> >Is it just for Brisbane, Queensland?
> >
> >An easier thing to do might be to request the change be made from the
> >backend instead.
>
> No, it's not just for Brisbane, Queensland, it's for whatever I need it for
> in the future. In the near future, there's a few other cities and towns I
> want to move. And it wouldn't be easier in terms of human effort, since the
> bot's already written and ready to go. It's only ~70 lines -- it's rather
> scary how easy these things are, especially since it's one of the first
> things I've ever written in perl and it only took me a few hours. Of course
> there's the issue of swamping RC -- if someone is happy to do it from the
> backend to reduce the annoyance level, then that's alright with me.
>
> -- Tim Starling
>
We'd be much better off if we add the needed functionality on the
serverside, I believe.

If you could describe exactly what the problem is that you wrote the
code to solve, that would be very helpful.

I'm not saying "don't ever run this"; I'm saying that we should think
about the consequences of this.

I really think we should err on the side of not worrying so much about
"fixing" links, if the redirects go to the right place. Redirects don't
affect the functionality for the user, and because of that, there should
be a degree of intransigence about adding complexity/running bots to fix
a problem that isn't much of a problem.

Please send me the code.
Re: Bot approval request [ In reply to ]
> (Tim Starling <ts4294967296@hotmail.com>):
> I'm not sure if this is the right place to ask, but I've just written a
> bot, and Wikipedia policy says it should be approved before I set it loose.

Looks like you've done you're homework. To run the bot the first time,
I'd set the delay to something nice and long like 30 seconds, and run
it at some off-peak hour, and go to RecentChanges and check its work
while it runs.

But if this is something that will need to be done a lot, then we
probably should look at doing it directly to the backend database,
because that will use less server bandwidth and not clutter up the
recent changes and the article histories.

And yes, I'd like to see the code before you unleash it.

--
Lee Daniel Crocker <lee@piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC
Re: Bot approval request [ In reply to ]
On Saturday 29 March 2003 12:02 am, Lee Daniel Crocker wrote:
> Looks like you've done you're homework. To run the bot the first time,
> I'd set the delay to something nice and long like 30 seconds, and run
> it at some off-peak hour, and go to RecentChanges and check its work
> while it runs.
>
> But if this is something that will need to be done a lot, then we
> probably should look at doing it directly to the backend database,
> because that will use less server bandwidth and not clutter up the
> recent changes and the article histories.
>
> And yes, I'd like to see the code before you unleash it.

Eh? Why not just "approve" the bot so that any edits made by user:Timbot do
not show-up in RC? Then Tim can run the bot at full speed and not bug
anybody. But a developer should take a look at the code first, as you say.

--mav
Re: Bot approval request [ In reply to ]
>We'd be much better off if we add the needed functionality on the
>serverside, I believe.

Perhaps, but I believe in the motto "if you want something done, do it
yourself." Who's going to write this server-side code -- you're obviously
not interested.

>If you could describe exactly what the problem is that you wrote the
>code to solve, that would be very helpful.
>
>I'm not saying "don't ever run this"; I'm saying that we should think
>about the consequences of this.
>
>I really think we should err on the side of not worrying so much about
>"fixing" links, if the redirects go to the right place. Redirects don't
>affect the functionality for the user, and because of that, there should
>be a degree of intransigence about adding complexity/running bots to fix
>a problem that isn't much of a problem.

I think you basically understand why I wrote it - to fix links after a page
move. There's a few hundred national parks articles that have been created
with a bot/template -- they form the bulk of the work. I understand your
point of view -- no, it doesn't really matter when there are working
redirects. I'm just a perfectionist, that's all.

Please understand that this bot will be make much less impact on RC than if
I did it by hand. Like I said, I did about 100 of them manually. I was able
to get them out quite quickly, doing them in batches of 15 or so I
absolutely swamped RC for about 20 minutes. No-one complained about that.

>Please send me the code.

Done, and I sent a copy to Lee as well.

_________________________________________________________________
Hotmail now available on Australian mobile phones. Go to
http://ninemsn.com.au/mobilecentral/hotmail_mobile.asp
Re: Bot approval request [ In reply to ]
>But if this is something that will need to be done a lot, then we
>probably should look at doing it directly to the backend database,
>because that will use less server bandwidth and not clutter up the
>recent changes and the article histories.

The kind of task I'm doing is done by hand fairly regularly on Wikpedia,
although perhaps on a smaller scale per job. It would be nice to provide a
proper "Special:" interface to allow contributors (perhaps just sysops) to
be able to do this more easily. However, I'm not holding my breath waiting.
I can always do it by hand (or pretend I'm doing it by hand).

As for the impact of this particular bot -- I think it will be on the scale
of hundreds (not thousands) of edits. I'm aware of the general suspicion
regarding bots around here -- I'll only use it when it's really necessary
(i.e. important enough to have been done by hand).

>And yes, I'd like to see the code before you unleash it.

Done.

-- Tim Starling


_________________________________________________________________
MSN Instant Messenger now available on Australian mobile phones. Go to
http://ninemsn.com.au/mobilecentral/hotmail_messenger.asp
Re: Bot approval request [ In reply to ]
> From: "Tim Starling" <ts4294967296@hotmail.com>
> Reply-To: wikitech-l@wikipedia.org
> Date: Sat, 29 Mar 2003 12:15:51 +1100
> To: wikitech-l@wikipedia.org
> Subject: [Wikitech-l] Bot approval request
>
>

"I'm open to suggestions as to the setting of $delay. If anyone wants to see
the rest of the code, I'll send it by private email.

-- Tim Starling."

What's up with this guy?

Fred
Re: Bot approval request [ In reply to ]
>
>"I'm open to suggestions as to the setting of $delay. If anyone wants to
>see
>the rest of the code, I'll send it by private email.
>
>-- Tim Starling."
>
>What's up with this guy?
>
>Fred

Sorry if I gave the wrong impression, but the reason I'm sending it by
private email is so that I control who gets it. As I said on User:Timbot:

:In keeping with Wikipedia policy, the source code is secret,
:however, it will be made available via email to well-behaved
:users if required

Fred, you've been here only since November, your first post to wikitech-l
was in February. With those 224 edits, you've only managed to piss me off
with a non-productive and persistent edit war over chiropractic medicine. In
that short time, you've managed to form an impression in my mind of
unilateralism and an aversion to real debate. You're not in a position to
contribute to the approval process, so why do you want the bot code? That
bot could be used very destructively if it fell into the hands of someone
interested in enforcing partisan article names. With a minor modification,
it could be made into an automated revert-bot.

Perhaps if The Cunctator or Lee have formed a better impression of you, you
can convince one of them to give you the code.

-- Tim Starling.


_________________________________________________________________
MSN Instant Messenger now available on Australian mobile phones. Go to
http://ninemsn.com.au/mobilecentral/hotmail_messenger.asp
Re: Bot approval request [ In reply to ]
Tim,

I noticed that when you applied to be a sysop you said you had never been in
an edit war, that seemed kind of funny at the time. The thing is anyone who
tries hard here to write NPOV articles will eventually bump up against
someone like RK who wants to argue. That has nothing at all to do with the
technical side. I have only a normal interest in looking at the code. If
someone who has received it would send it I would appreciate it. Paranoia
about a revert bot is really out of place.

Fred

> From: "Tim Starling" <ts4294967296@hotmail.com>
> Reply-To: wikitech-l@wikipedia.org
> Date: Sun, 30 Mar 2003 12:07:20 +1000
> To: wikitech-l@wikipedia.org
> Subject: Re: [Wikitech-l] Bot approval request
>
>>
>> "I'm open to suggestions as to the setting of $delay. If anyone wants to
>> see
>> the rest of the code, I'll send it by private email.
>>
>> -- Tim Starling."
>>
>> What's up with this guy?
>>
>> Fred
>
> Sorry if I gave the wrong impression, but the reason I'm sending it by
> private email is so that I control who gets it. As I said on User:Timbot:
>
> :In keeping with Wikipedia policy, the source code is secret,
> :however, it will be made available via email to well-behaved
> :users if required
>
> Fred, you've been here only since November, your first post to wikitech-l
> was in February. With those 224 edits, you've only managed to piss me off
> with a non-productive and persistent edit war over chiropractic medicine. In
> that short time, you've managed to form an impression in my mind of
> unilateralism and an aversion to real debate. You're not in a position to
> contribute to the approval process, so why do you want the bot code? That
> bot could be used very destructively if it fell into the hands of someone
> interested in enforcing partisan article names. With a minor modification,
> it could be made into an automated revert-bot.
>
> Perhaps if The Cunctator or Lee have formed a better impression of you, you
> can convince one of them to give you the code.
>
> -- Tim Starling.
>
>
> _________________________________________________________________
> MSN Instant Messenger now available on Australian mobile phones. Go to
> http://ninemsn.com.au/mobilecentral/hotmail_messenger.asp
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@wikipedia.org
> http://www.wikipedia.org/mailman/listinfo/wikitech-l
Re: Bot approval request [ In reply to ]
On Sat, 2003-03-29 at 01:21, Tim Starling wrote:
> >We'd be much better off if we add the needed functionality on the
> >serverside, I believe.
>
> Perhaps, but I believe in the motto "if you want something done, do it
> yourself." Who's going to write this server-side code -- you're obviously
> not interested.

Meanwhile, you obviously _are_ interested in making this ability
available. If you'd like to try your hand, I'd be delighted.

As a temporary solution, I've no objection to a bot, but it is something
that would be better off available in the software as something that
could be done in conjunction with a rename operation. Someone who wants
to do this frequently would probably prefer that to running a bot over
and over and over.

-- brion vibber (brion @ pobox.com)
Re: Bot approval request [ In reply to ]
Tim-
> :In keeping with Wikipedia policy, the source code is secret,
> :however, it will be made available via email to well-behaved
> :users if required

Can you elaborate a bit on the destructive potential of the bot? I'm not
fully sure if

1) You do not want people to use the bot for things which you don't want
to happen, but which nobody else cares to stop; or
2) You do consider it, if modified, unstoppable.

Please note that we can block whole IP ranges if necessary or switch the
wiki to read-only, so I'm not sure what a dedicated bot attack could
accomplish? If you are basically telling Fred "I don't like you, you have
opinions I disagree with, you won't get to use my code", that is certainly
your decision to make.

But if that is your logic, I do not think your bot should be approved for
running on Wikipedia at all.

Thanks for your understanding,
Erik
Re: Bot approval request [ In reply to ]
--- Tim Starling <ts4294967296@hotmail.com> wrote:
> Fred, you've been here only since November, your
> first post to wikitech-l
> was in February. With those 224 edits, you've only
> managed to piss me off
> with a non-productive and persistent edit war over
> chiropractic medicine. In
> that short time, you've managed to form an
> impression in my mind of
> unilateralism and an aversion to real debate. You're
> not in a position to
> contribute to the approval process, so why do you
> want the bot code?

Hello Tim,

what does that mean "you are not in a position to
contribute to the approval process" ???

Cheers, ant

__________________________________________________
Do you Yahoo!?
Yahoo! Platinum - Watch CBS' NCAA March Madness, live on your desktop!
http://platinum.yahoo.com
Re: Bot approval request [ In reply to ]
Erik Moeller wrote:

>Tim-
>
>>:In keeping with Wikipedia policy, the source code is secret,
>>:however, it will be made available via email to well-behaved
>>:users if required
>>
>
>Can you elaborate a bit on the destructive potential of the bot? I'm not
>fully sure if
>
>1) You do not want people to use the bot for things which you don't want
>to happen, but which nobody else cares to stop; or
>2) You do consider it, if modified, unstoppable.
>
>Please note that we can block whole IP ranges if necessary or switch the
>wiki to read-only, so I'm not sure what a dedicated bot attack could
>accomplish? If you are basically telling Fred "I don't like you, you have
>opinions I disagree with, you won't get to use my code", that is certainly
>your decision to make.
>
>But if that is your logic, I do not think your bot should be approved for
>running on Wikipedia at all.
>
I've had some serious concerns about this bot. On the surface it has to
do with changing [[Brisbane, Queensland]] to simply [[Brisbane]] with
the possibility that it could also be used for other city names. This
is a questionable use of a bot to impose a naming convention that may
not have unanimous support. If it is used in the course of an edit war,
it makes for an unequal fight between those who use a bot and those who
don't.

I'm not too sure of the solution. Perhaps such a bot should be
generally available to everybody so that someone who disagreed with the
naming convention change could just as easily revert it. Perhaps we
need to have bot generated changes appear on "Recent changes" again so
that everybody can see what is happening. Or perhaps we need to review
the bot approval process so that some vote would be needed for each
separate use of a bot.

Eclecticology
Re: Bot approval request [ In reply to ]
Brion said:

> > >We'd be much better off if we add the needed functionality on the
> > >serverside, I believe.
> >
> > Perhaps, but I believe in the motto "if you want something done, do it
> > yourself." Who's going to write this server-side code -- you're
>obviously
> > not interested.
>
>Meanwhile, you obviously _are_ interested in making this ability
>available. If you'd like to try your hand, I'd be delighted.

Alright, but in an attempt to prevent Wikipedia from completely taking over
my life, I'm going to stop making edits. I'm not sure if I can go cold
turkey, so if anyone sees me doing anything, please block me.

Now on to Eric Moeller's post:

>If you are basically telling Fred "I don't like you, you have
>opinions I disagree with, you won't get to use my code", that is certainly
>your decision to make.

Yeah, that's pretty much it. And from Eclecticology:

>I've had some serious concerns about this bot. On the surface it has to do
>with changing [[Brisbane, Queensland]] to simply [[Brisbane]] with the
>possibility that it could also be used for other city names. This is a
>questionable use of a bot to impose a naming convention that may not have
>unanimous support. If it is used in the course of an edit war, it makes
>for an unequal fight between those who use a bot and those who don't.

That's a very good point. Okay, forget the bot. Like I said above, I'll
write it on the server side. I'll start a new thread on how it should be
implemented.

>what does that mean "you are not in a position to
>contribute to the approval process" ???

I was suggesting that it's not Fred who makes the decisions, it's people
like Brion and Lee. It doesn't make much sense to me either, now that I've
calmed down. Please forget I said it.

-- Tim Starling.

_________________________________________________________________
MSN Instant Messenger now available on Australian mobile phones. Go to
http://ninemsn.com.au/mobilecentral/hotmail_messenger.asp
Re: Bot approval request [ In reply to ]
>>what does that mean "you are not in a position to
>>contribute to the approval process" ???
>
>I was suggesting that it's not Fred who makes the decisions, it's people
>like Brion and Lee. It doesn't make much sense to me either, now that I've
>calmed down. Please forget I said it.

Sorry, that quote (>>what...) was from Anthere. I didn't properly attribute
it.

_________________________________________________________________
Hotmail now available on Australian mobile phones. Go to
http://ninemsn.com.au/mobilecentral/hotmail_mobile.asp
Re: Bot approval request [ In reply to ]
The Cunctator wrote:

>We'd be much better off if we add the needed functionality on the
>serverside, I believe.
>
>
probably.

>If you could describe exactly what the problem is that you wrote the
>code to solve, that would be very helpful.
>
>I really think we should err on the side of not worrying so much about
>"fixing" links, if the redirects go to the right place.
>
I'd like some sort of system (bot, server-side, don't mind) to rename
images. for example:

1) cryptic DOS-like names, like "UKmp-nwcstle" to "Map of UK showing
Newcastle"
2) ambiguous names, like "Washington" to "washington memorial"
Re: Bot approval request [ In reply to ]
> (Tim Starling <ts4294967296@hotmail.com>):
>
> Sorry if I gave the wrong impression, but the reason I'm sending it by
> private email is so that I control who gets it. As I said on User:Timbot:
>
> :In keeping with Wikipedia policy, the source code is secret,
> :however, it will be made available via email to well-behaved
> :users if required
> ...
> Perhaps if The Cunctator or Lee have formed a better impression of you, you
> can convince one of them to give you the code.

I express no opinion about Fred, but the code is Tim's to give or
not give to whomever he chooses, and I won't usurp his prerogative.
However, I must say that I find no evidence at all of any "policy"
that bot code should be private. Indeed, I see no reason at all why
shy bots should not be as public as everything else here. Yes,
they are potentially destructive, but so is all technology. Keeping
things like that secret is just security through obscurity--if making
the code free is a problem, then it's a problem we have now already
and should think about ways to ameliorate.

--
Lee Daniel Crocker <lee@piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC