Mailing List Archive

On Fri, Aug 28, 2009 at 10:08 AM, Thomas Dalton <thomas.dalton@gmail.com>wrote:

> 2009/8/28 Anthony <wikimail@inbox.org>:
> > If you're going to do it, maybe we should work on a rough-consensus
> > objective definition of "vandalism" before you release the file,
> though...
>
> Don't we have a consensus definition already? Vandalism is bad faith
> editing. You may also want to include test edits since they are
> treated in the same way (just with different warning messages). That
> isn't objective, but it should be close enough. We can argue over a
> few borderline cases.


Well, it relies on information (intent) which we can't determine simply from
the content of the edit (sometimes it is implied if you look at the entire
behavior of the user, but that's too messy). Is a POV edit "vandalism"? I
think it has to be treated as such, at least some of the time ("Windows is
the worst operating system ever"), but there are certainly edits which are
clearly POV but the intent is unclear (many people don't know the rules).
We need to remove intent from the definition, and I suppose call it
"degraded articles". But simply saying that anything POV is vandalism would
potentially include just about any large article.

I suppose we can just list everything that's arguably vandalism and then
categorize it later though. I expect we'll come up with several different
final numbers, which I guess is okay (the only part that really needs to be
pristinely unbiased is the selection of pageviews), though I do expect some
people will adapt their definition of vandalism to fit the data.

I support the request for 5000 random pageviews (uniform distribution
> by pageview over the last 6 months) from the logs.


Seems like it could be reused for a lot of different types of studies, so
long as the researcher isn't exposed to the details of the urls before
coming up with his/her methodology. And I think the analysis of those 5000
pageviews in all sorts of ways would "crowdsource" well. I'd love to see a
"Nature Study" equivalent, analyzing the more subjective aspects of the
articles in addition to just plain old vandalized/not-vandalized.

If we can't get the 5000 random pageviews (do the logs even still exist?), I
suppose wikistats will do. They have pageviews broken down by hour, so the
non-uniformity of a single hour is probably fairly small for the popular
pages most likely to be selected. Worst part is that it's a whole lot of
data to download, and I'm not sure any shortcuts can be taken without
screwing up the non-uniformity. I considered just downloading the
projectcounts and then selecting the date-hours weighted accordingly then
downloading only the date-hour files needed, but that does potentially
introduce error if the non-article traffic isn't well correlated to the
article traffic, so I dunno. Probably a safe assumption that they are well
correlated, but I'd rather not guess. Maybe talk-page traffic is highly
correlated to increased vandalism, or decreased vandalism. It's possible,
so I'd rather be safe.
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Fri, Aug 28, 2009 at 3:55 AM, Anthony<wikimail@inbox.org> wrote:
<snip>

> Once we have the list, anyone is free to examine it any way they want, and
> show their results.  But we're talking about probably less than 200
> instances of vandalism here, so it'll be quite easy (and fun) to lambaste
> anyone whose methods produce false positives.

Comments like this discourage people like me from putting in the time
and effort to do this sort of work. Offering constructive criticism
is one thing, but looking forward to the "fun" of "lambast[ing]" the
good faith efforts of others is offensive and not in keeping with the
collaborative spirit necessary to run WMF projects.

-Robert Rohde

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

2009/8/28 Robert Rohde <rarohde@gmail.com>:
> On Fri, Aug 28, 2009 at 3:55 AM, Anthony<wikimail@inbox.org> wrote:

>> Once we have the list, anyone is free to examine it any way they want, and
>> show their results.  But we're talking about probably less than 200
>> instances of vandalism here, so it'll be quite easy (and fun) to lambaste
>> anyone whose methods produce false positives.

> Comments like this discourage people like me from putting in the time
> and effort to do this sort of work.  Offering constructive criticism
> is one thing, but looking forward to the "fun" of "lambast[ing]" the
> good faith efforts of others is offensive and not in keeping with the
> collaborative spirit necessary to run WMF projects.


I suggest that treating Anthony's foundation-l contributions as
anything other than destructive trolling is probably a waste of time.


- d.

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Fri, Aug 28, 2009 at 2:17 PM, David Gerard <dgerard@gmail.com> wrote:

> 2009/8/28 Robert Rohde <rarohde@gmail.com>:
> > On Fri, Aug 28, 2009 at 3:55 AM, Anthony<wikimail@inbox.org> wrote:
>
> >> Once we have the list, anyone is free to examine it any way they want,
> and
> >> show their results. But we're talking about probably less than 200
> >> instances of vandalism here, so it'll be quite easy (and fun) to
> lambaste
> >> anyone whose methods produce false positives.
>
> > Comments like this discourage people like me from putting in the time
> > and effort to do this sort of work. Offering constructive criticism
> > is one thing, but looking forward to the "fun" of "lambast[ing]" the
> > good faith efforts of others is offensive and not in keeping with the
> > collaborative spirit necessary to run WMF projects.
>
>
> I suggest that treating Anthony's foundation-l contributions as
> anything other than destructive trolling is probably a waste of time.


Comments like this discourage people like me from putting in the time and
effort to do this sort of work.

Okay, no they don't, but I thought it would be funny to whine about it like
that.
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

Anthony wrote:

> Umm...you would count the number of instances of vandalism?
>
> Is the question how to objectively *define* "vandalism"?

On one hand, we have a perception, as expressed by media (and by
CEO Sue Gardner, I believe), that vandalism (especially of
biographies of living people, BLP) is an increasing problem. On
the other hand, we have the habit of always asking for proofs and
measurements: Citation needed!

We can try to find out which edits are reverts, assuming that the
previous edit was an act of vandalism. That way we can conclude
which articles were vandalized and how long it took to revert
them. Add to that: How many readers viewed the vandalized
version? Vandalism is harmless if nobody watches it. It is mostly
harmless if it is obvious and childish (e.g. Barack Obama was born
on Mars, he's a space alien). When it does harm (and becomes a
problem, allegedly an increasing problem) is when it is viewed and
taken for the truth (e.g. a statement that Barack Obama was not
born in the U.S. and thus would not be a legitimate president).

Especially, it becomes a very real problem if the biographed
living person takes offense and takes legal action against the
WMF. Now, that's very easy to measure: How much money did WMF
need to spend, month by month, to resolve such conflicts,
including time to explain the process to media? That is money
that could be used to buy servers instead. A more efficient BLP
policy might render the WMF more money for servers. Very real.
Now, we only need to insert real numbers into this equation.


--
Lars Aronsson (lars@aronsson.se)
Aronsson Datateknik - http://aronsson.se

_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l

On Fri, Aug 28, 2009 at 3:44 PM, Lars Aronsson <lars@aronsson.se> wrote:

> We can try to find out which edits are reverts, assuming that the
> previous edit was an act of vandalism.


But that's a bad assumption. It gives both false positives and false
negatives, and it gives a significant number of each. I gave examples of
each above. My samples were tiny, but 38% of reverts were not reverts of
vandalism, and 40% of vandalism was not reverted by a means detected by this
strategy. And there is no reason to believe that the error is consistent
over time, so these numbers are useless when it comes to determining whether
or not the problem is increasing.

That way we can conclude
> which articles were vandalized and how long it took to revert
> them.


Your simplistic version of assuming that the previous edit was an act of
vandalism makes the conclusion of "how long it took to revert" pretty
obviously flawed, doesn't it? In your simplistic assumption (which is even
worse than the one used by Robert), you're simply measuring the average time
between edits. Any acts of vandalism which take more than one edit to find
and fix are excluded.

Now Robert's methodology wasn't quite that bad. It allowed for reverts
separated by one or more other edits. But it had no way to detect an act
of vandalism which lasted for hundreds of edits, was discovered by someone
reading the text, and was removed without reference to the original edit
with an edit summary such as "Barrack Obama was born in Hawaii". And these
acts of vandalism are the worst. They last the longest, they do the most
harm when they are read, they get the most views, etc. Any methodology
which excludes them is systemically biased.
_______________________________________________
foundation-l mailing list
foundation-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l