Mailing List Archive: Feature Proposal: Certification

Feature Proposal: Certification

Oct 30, 2002, 5:02 AM

Post #1 of 11 (1073 views)

One key problem with a wiki encyclopaedia is that there's no quality
control whatsoever. An article may have been vandalized 5 seconds ago,
or be grossly non-NPOV etc. As we get more and more articles, this
problem becomes more urgent.

Fortunately, the solution is rather simple. Articles can be certified by
contributors to be high quality. But who is allowed to certify articles?
The system works by allowing groups of people to form certification
teams. Anyone can submit a new team to be created, and anyone can apply
to join an existing team and certify articles in its name. Users can
then decide to view only article revisions certified by members of
selected teams.

So I could decide in my user preferences:
Certification: Approved Teams
Team Nupedia
Team Wiki-Fiction
Team Wiki-Maths

Then there would have to be a way to display certified article
revisions. This could be accomplished by having a "Certified Mode",
showing *only* articles that have received certs, with the most recently
certified revision shown. Somewhat weaker, where an article has been
certified, a link "There is a version of this article certified by Team
X" could be placed above the article, showing the certified revision
when clicked (or a text "This article has been certified by .." if the
current revision is the certified one). This could be the default view,
making users aware of the cert system.

Each team could have its own quality standards, policies, and subject
preferences. I suggest that the creation of new teams would have to be
approved by the Wikipedia cabal to avoid "Team Trolls". New team members
would either be voted on or approved by team members that have a certain
status flag ("can_approve_newcomers"). Teams could get their own
namespace as well.

A decision would have to be made as to which teams to include in the
default view, i.e. the one that anonymous and newly registered users
get. In the short term such decisions may be made by the cabal, in the
long term I would prefer voting.

Implementation:

There needs to be a teams table that at least has a TID and a TDesc, and
a team-member table that links TIDs and UIDs and grants individual users
certain permissions within the team, as well as a "pending" status flag
for newly applied members. A TCert table would link UIDs, TIDs certs and
article IDs (must include timestamp).

I am currently assuming that certifications would be simple "Good
article" binary flags. My reasoning is that a rating of "good, but not
good enough" may not be very helpful. Still, practice may prove that
wrong, and the table should therefore be designed to accomodate possibly
more flexible ratings.

Giving each individual member the power to rule an article certified in
the name of the whole team may be undesirable. Thus, teams should be
configurable to set their own min_number_of_certs, where certifications
would only be valid if so many team numbers agree that the article is
complete and high quality.

Finally, the PHP scripts of course have to be updated to reflect this
functionality. There needs to be some way to apply for team membership,
to approve team membership, and to submit a new team for cabal approval.

A module listing all articles approved by a certain team would be nice
and could substitute the "Brilliant Prose" page.

Results:
--------
If this works as intended, it should solve the quality problem and allow
users to browse Wikipedia as a high quality content only encyclopaedia.
The more teams you would admit to your personal filter, the more content
you would see, but quality standards of individual teams might not be up
to par. By distributing the job of quality approval on several team
leaders, we can get competition of quality standards and social methods,
which is probably a good thing and reduces social problems.

Potential problems:
-------------------
If too many people use highly customized views, caching will get harder.
I don't see this as too big a problem as a) most people typically don't
customize views, b) article retrieval is already very fast with or
without caching.

Too many teams may have undesired effects, such as teams deliberately
inserting POV articles to certify them. This is not a problem with the
team principle per se but with the way teams are approved and moderated.
Generally, teams should have a clear NPOV commitment and respect
Wikipedia policy, otherwise they should be deleted.

Comments on this would be appreciated. This is something I probably
won't have time to implement fully, but I will gladly help with any/all
efforts. I consider it very necessary for Wikipedia in the long term.

Regards,

Erik Moeller
--
FOKUS - Fraunhofer Insitute for Open Communication Systems
Project BerliOS - http://www.berlios.de

Re: Feature Proposal: Certification [ In reply to ]

toby+wikipedia at math

Oct 30, 2002, 11:47 AM

Post #2 of 11 (998 views)

Permalink

[.Note: The post that I'm replying to didn't appear on <wikipedia-l>.
Thus I copy it all, except for the technical aspects.]

Erik Moeller wrote:

>One key problem with a wiki encyclopaedia is that there's no quality
>control whatsoever. An article may have been vandalized 5 seconds ago,
>or be grossly non-NPOV etc. As we get more and more articles, this
>problem becomes more urgent.

>Fortunately, the solution is rather simple. Articles can be certified by
>contributors to be high quality. But who is allowed to certify articles?
>The system works by allowing groups of people to form certification
>teams. Anyone can submit a new team to be created, and anyone can apply
>to join an existing team and certify articles in its name. Users can
>then decide to view only article revisions certified by members of
>selected teams.

>So I could decide in my user preferences:
> Certification: Approved Teams
> Team Nupedia
> Team Wiki-Fiction
> Team Wiki-Maths

>Then there would have to be a way to display certified article
>revisions. This could be accomplished by having a "Certified Mode",
>showing *only* articles that have received certs, with the most recently
>certified revision shown. Somewhat weaker, where an article has been
>certified, a link "There is a version of this article certified by Team
>X" could be placed above the article, showing the certified revision
>when clicked (or a text "This article has been certified by .." if the
>current revision is the certified one). This could be the default view,
>making users aware of the cert system.

>Each team could have its own quality standards, policies, and subject
>preferences. I suggest that the creation of new teams would have to be
>approved by the Wikipedia cabal to avoid "Team Trolls". New team members
>would either be voted on or approved by team members that have a certain
>status flag ("can_approve_newcomers"). Teams could get their own
>namespace as well.

You seem to be using the word "cabal" here in a sense that
is neither derogatory nor ironic. I find that highly disturbing.

>A decision would have to be made as to which teams to include in the
>default view, i.e. the one that anonymous and newly registered users
>get. In the short term such decisions may be made by the cabal, in the
>long term I would prefer voting.

If newcomers see only what is approved by a list of certification teams,
then Wikipedia will no longer be a wiki. There will be a wiki underneath,
which you can get to by registering and then setting your preferences,
but that wiki would be dead without an influx of newcomers.

[technical aspects cut]

>Results:
>--------
>If this works as intended, it should solve the quality problem and allow
>users to browse Wikipedia as a high quality content only encyclopaedia.
>The more teams you would admit to your personal filter, the more content
>you would see, but quality standards of individual teams might not be up
>to par. By distributing the job of quality approval on several team
>leaders, we can get competition of quality standards and social methods,
>which is probably a good thing and reduces social problems.

>Potential problems:
>-------------------
>If too many people use highly customized views, caching will get harder.
>I don't see this as too big a problem as a) most people typically don't
>customize views, b) article retrieval is already very fast with or
>without caching.

>Too many teams may have undesired effects, such as teams deliberately
>inserting POV articles to certify them. This is not a problem with the
>team principle per se but with the way teams are approved and moderated.
>Generally, teams should have a clear NPOV commitment and respect
>Wikipedia policy, otherwise they should be deleted.

>Comments on this would be appreciated. This is something I probably
>won't have time to implement fully, but I will gladly help with any/all
>efforts. I consider it very necessary for Wikipedia in the long term.

-- Toby

Re: Feature Proposal: Certification [ In reply to ]

erik_moeller at gmx

Oct 30, 2002, 2:33 PM

Post #3 of 11 (993 views)

Permalink

> You seem to be using the word "cabal" here in a sense that
> is neither derogatory nor ironic. I find that highly disturbing.

You may find it disturbing, but the truth is that there are people like
Jimbo who can make high level decisions, and there are sysops that have
more power than ordinary users. Many of these decisions already happen in
places that most people don't know about (e.g. the mailing list). Glossing
over the truth doesn't make it go away: there is a Wikipedia cabal.

I have already suggested a voting scheme that would democratize the
decision processes by the inevitable administration. Aside from extreme
opinions like "voting doesn't work", I see few arguments against that. We
need to talk openly about this kind of stuff, or what you find disturbing
will turn into a nightmare eventually.

>> A decision would have to be made as to which teams to include in the
>> default view, i.e. the one that anonymous and newly registered users
>> get. In the short term such decisions may be made by the cabal, in the
>> long term I would prefer voting.

> If newcomers see only what is approved by a list of certification teams,
> then Wikipedia will no longer be a wiki. There will be a wiki underneath,
> which you can get to by registering and then setting your preferences,
> but that wiki would be dead without an influx of newcomers.

You misunderstand me. I am absolutely in favor of creating and keeping a
site that is immediately accessible to the newcomer, where the newbie
quickly notices that WP is editable and joins the process. I love wikis!
My idea centers around the facts that
- there will always be many people who just read and who will never be
contributors,
- even for contributors, it is sometimes desirable to quickly find
trustworthy information,
- some contributors would like there to be some distinction between the
work they have invested much time in and the vandalism of a bored Internet
hooligan.

So what I am suggesting is an alternative viewing mode that would *never*
be the default but optional. It would allow me to browse a Wikipedia where
the article about Mozart *can* not just have been replaced by an image
from goatse.cx. Instead, I would view the last certified version of that
article, which hopefully would be brilliant prose.

Instead of just telling people to use Britannica if they want trustworthy
information, we should substitute this part of Britannica -- quality
control -- as well. That doesn't mean we have to give up any part of the
massively collaborative project that WP is becoming. As a matter of fact,
it would hopefully attract all those skeptics that are afraid that
something like the above happens to their sacro-sanct articles -- it could
still happen, but they could find peace of mind in the fact that the
vandalized version would never be certified.

Last but not least, we should not forget that WP is intended to be a
useful tool, not just for those who like to write, but for those who like
to read, too. Be it interested adults or curious children, rich or poor,
we want Wikipedia to be accessible. We may want to distribute it on CD-
ROMs and on paper. Then how on Earth are the schoolchildren in India going
to wade through megabytes of Middle Earth mythology and stubs, if not if
we supply them with at least the option to filter articles according to
criteria developed collaboratively by various teams, working together to
find the sparkling gems among the ocean we are creating?

"Being a wiki" doesn't mean that we shouldn't extend the original wiki
functionality. Had we stayed with that, Wikipedia would have never become
that big. We would still be using CamelCase.

Regards,

Erik

Re: Feature Proposal: Certification [ In reply to ]

magnus.manske at epost

Oct 30, 2002, 2:47 PM

Post #4 of 11 (997 views)

Permalink

erik_moeller@gmx.de wrote:

>>You seem to be using the word "cabal" here in a sense that
>>is neither derogatory nor ironic. I find that highly disturbing.
>>
>>
>
>You may find it disturbing, but the truth is that there are people like
>Jimbo who can make high level decisions, and there are sysops that have
>more power than ordinary users. Many of these decisions already happen in
>places that most people don't know about (e.g. the mailing list). Glossing
>over the truth doesn't make it go away: there is a Wikipedia cabal.
>
Hush! ;-)

>I have already suggested a voting scheme that would democratize the
>decision processes by the inevitable administration. Aside from extreme
>opinions like "voting doesn't work", I see few arguments against that. We
>need to talk openly about this kind of stuff, or what you find disturbing
>will turn into a nightmare eventually.
>
Which reminds me:
For future issues, we could set up a special "voting" page at the
wikipedia, where proposals can be submitted (with several options to
choose from, including "just leave it as it is"), where every logged-in
wikipedian can vote (once!), with a time limit (say, one or two weeks)
so everyone get a chance to vote, but it won't hang there forever.
Democratic enough?

<snip explanation>

Why not use Larry's proposed read-only 'pedia instead? A new skin and a
few restrictions (that would be set to "unrestricted" for wikipedia, of
course), a nice little server, done :-)

Magnus

Re: Feature Proposal: Certification [ In reply to ]

erik_moeller at gmx

Oct 30, 2002, 3:08 PM

Post #5 of 11 (995 views)

Permalink

> Which reminds me:
> For future issues, we could set up a special "voting" page at the
> wikipedia, where proposals can be submitted (with several options to
> choose from, including "just leave it as it is"), where every logged-in
> wikipedian can vote (once!), with a time limit (say, one or two weeks)
> so everyone get a chance to vote, but it won't hang there forever.
> Democratic enough?

I've been thinking about this for a while, and currently I'm tending
towards a scheme where a user would insert a magic link into a page (talk
or otherwise)

[[=poll=]]

or something like that. Rendered, this would first show a red link

Create a poll

On that page, the user could then set poll questions and voting type (I
would love to allow both preferential voting and normal voting, and more
voting systems could be added later). The poll would then be given an ID
and the magic link would be changed to

[[=poll:id=]]

where id would be the poll's internal ID number. Rendered, this would then
put a nice selection box

option [ ]
option [ ]

on the page. For preferential voting, the [ ] would be an entryfield where
the user enters a number from 1 to n, n being the number of options.
Otherwise the standard radiobutton.

The poll creator and sysops would have permission to modify the poll. On
modification, the poll votes would be reset.

There should be a way to show recent polls, and the poll creation page
should require a textfield like this:

Please describe the purpose of the poll in terms that are
understandable to people not yet involved in the debate
who might be interested.
_______________________________
_______________________________

I'm currently torn as to whether we want to record and possibly show who
voted how. This has benefits in many situations, it makes abuses visible
and creates a better picture of the opinion spectrum. The downside is
possible peer pressure to vote a certain way. If it's visible, it should
probably only be shown once a user has voted. Abuses could become a
serious issue as accounts are trivial to create on WP. IP recording (maybe
as hash? we record them clearly in many places anyway) is essential to
avoid duplicate votes from a single IP.

If we have such a system in place, we can give sysops the permission to
create special polls:

[[=spoll=]]

which would be a poll only usable by sysops but visible to anyone. Such
polls could show up in another list, Recent admin polls, so that admins
could easily see what new issues have come up.

What do you think about such a scheme?

Regards,

Erik

Re: [Wikipedia-l] Re: Feature Proposal: Certification [ In reply to ]

imran at bits

Oct 30, 2002, 4:43 PM

Post #6 of 11 (993 views)

Permalink

On 30 Oct 2002 erik_moeller@gmx.de wrote:

> > You seem to be using the word "cabal" here in a sense that
> > is neither derogatory nor ironic. I find that highly disturbing.
>
> You may find it disturbing, but the truth is that there are people like
> Jimbo who can make high level decisions, and there are sysops that have
> more power than ordinary users. Many of these decisions already happen in
> places that most people don't know about (e.g. the mailing list). Glossing
> over the truth doesn't make it go away: there is a Wikipedia cabal.

TINC.

Cabal implies secrecy, what we have here is a benevolent dictatorship.

Imran
--
http://bits.bris.ac.uk/imran

Re: Feature Proposal: Certification [ In reply to ]

jwales at bomis

Oct 31, 2002, 5:42 AM

Post #7 of 11 (991 views)

Permalink

Summary: I read today that only 6% of the people on the planet have
ever logged into the Internet. Only about half have ever used a
telephone. So Erik's concerns about schoolchildren in India do
resonate with me. I just think it's too early to settle on any
particular certification scheme.

erik_moeller@gmx.de wrote:
> You may find it disturbing, but the truth is that there are people like
> Jimbo who can make high level decisions, and there are sysops that have
> more power than ordinary users. Many of these decisions already happen in
> places that most people don't know about (e.g. the mailing list). Glossing
> over the truth doesn't make it go away: there is a Wikipedia cabal.

To the extent this is true, we should work to make it less true. Keep
in mind that sysop status is open to anyone. Sysop powers should be
kept to just what is absolutely minimally necessary. The mailing list
should be (and is, I think) promoted to such an extent that anyone
with an interest in policy can understand that this is where to come
for most policy discussions, etc.

The problem with getting comfortable with a cabal is that it will tend
to make us lazy about our commitment to openness.

> I have already suggested a voting scheme that would democratize the
> decision processes by the inevitable administration. Aside from extreme
> opinions like "voting doesn't work", I see few arguments against that. We
> need to talk openly about this kind of stuff, or what you find disturbing
> will turn into a nightmare eventually.

Well, voting _doesn't_ work, and I don't think that's an extreme
opinion.

The wiki procedure is to try to get near unanimous consensus. This is
done in the writing of articles, and is inherent to the wiki process.
Authors work together to meet the objections of others, not to
campaign for sufficient numbers to vote down the opposing point of
view. We simulate that process, imperfectly but to a large degree, in
policy decision making.

> So what I am suggesting is an alternative viewing mode that would *never*
> be the default but optional. It would allow me to browse a Wikipedia where
> the article about Mozart *can* not just have been replaced by an image
> from goatse.cx. Instead, I would view the last certified version of that
> article, which hopefully would be brilliant prose.

I think it is inevitable that we, or someone, will someday create such
a thing. One proposal sometimes bandied about is that Nupedia could
become a repository for peer-reviewed articles from wikipedia.

> As a matter of fact, it would hopefully attract all those skeptics
> that are afraid that something like the above happens to their
> sacro-sanct articles -- it could still happen, but they could find
> peace of mind in the fact that the vandalized version would never be
> certified.

I think that the tiny percentage of people who treat their own writing
as sacrosanct will not be comfortable with any free-license writing
project.

> Last but not least, we should not forget that WP is intended to be a
> useful tool, not just for those who like to write, but for those who like
> to read, too. Be it interested adults or curious children, rich or poor,
> we want Wikipedia to be accessible. We may want to distribute it on CD-
> ROMs and on paper. Then how on Earth are the schoolchildren in India going
> to wade through megabytes of Middle Earth mythology and stubs, if not if
> we supply them with at least the option to filter articles according to
> criteria developed collaboratively by various teams, working together to
> find the sparkling gems among the ocean we are creating?
>
> "Being a wiki" doesn't mean that we shouldn't extend the original wiki
> functionality. Had we stayed with that, Wikipedia would have never become
> that big. We would still be using CamelCase.

I totally agree with all of this. But I'm not sure that the time is
ripe just yet. Perhaps in another year. I'm open to alternative
points of view on this, but I think we're not close enough yet to
"well rounded" to start seriously worrying about these issues.

--Jimbo

Re: Feature Proposal: Certification [ In reply to ]

e.moeller at fokus

Oct 31, 2002, 7:59 AM

Post #8 of 11 (987 views)

Permalink

Am Don, 2002-10-31 um 13.42 schrieb Jimmy Wales:

> I totally agree with all of this. But I'm not sure that the time is
> ripe just yet. Perhaps in another year. I'm open to alternative
> points of view on this, but I think we're not close enough yet to
> "well rounded" to start seriously worrying about these issues.

If there's any interest at all in this among other developers, how about
working on it on a separate CVS branch? Then we could try it on, say,
meta.wp and merge it into HEAD + use it on wikipedia.org when we're
happy with the way it works.

Personally, I think that we're reaching the point where we need this
because WP is becoming too big for us to oversee, so we can't really
guarantee that it is crap-free.

Regards,

Erik

--
FOKUS - Fraunhofer Insitute for Open Communication Systems
Project BerliOS - http://www.berlios.de

Re: Feature Proposal: Certification [ In reply to ]

magnus.manske at epost

Oct 31, 2002, 8:12 AM

Post #9 of 11 (989 views)

Permalink

Erik Moeller wrote:

>>I totally agree with all of this. But I'm not sure that the time is
>>ripe just yet. Perhaps in another year. I'm open to alternative
>>points of view on this, but I think we're not close enough yet to
>>"well rounded" to start seriously worrying about these issues.
>>
>>
>
>If there's any interest at all in this among other developers, how about
>working on it on a separate CVS branch? Then we could try it on, say,
>meta.wp and merge it into HEAD + use it on wikipedia.org when we're
>happy with the way it works.
>
I take it you mean the "sysops and truster reviews only" thingy? Can-do,
sure. But, honestly, I don't think I'd go around checking ~90000
articles, including 35000 one-cow-metropoles ;-)
IMHO that experiment would fail not because of missing technology, but
of lack of reviewers.
I'd prefer "Larrypedia" myself, where people dedicated to reviewing can
work their magic.

>Personally, I think that we're reaching the point where we need this
>because WP is becoming too big for us to oversee, so we can't really
>guarantee that it is crap-free.
>
We never did! Our warranty always was and is "good luck"! :-)

Magnus

Re: Feature Proposal: Certification [ In reply to ]

e.moeller at fokus

Oct 31, 2002, 8:49 AM

Post #10 of 11 (992 views)

Permalink

> I take it you mean the "sysops and truster reviews only" thingy?

The idea is that anyone can join one of the teams, if the existing team
members approve him. Elian suggested to use individual ratings instead,
but I think teams are important because they represent aggregated trust.
Ratings themselves can be low-quality, so you need a reputation system
of some type to get the good ones. Nevertheless, the team thing should
be very community-like and open, a bit like SETI@Home, where competition
and cooperation work well together.

I don't think the one cow articles would ever be certified, but when we
import information from a trusted source like the US Bureau of Census,
we could certify all those articles under that name, provided the bot
has not made mistakes.

Regards,

Erik
--
FOKUS - Fraunhofer Insitute for Open Communication Systems
Project BerliOS - http://www.berlios.de

Re: Feature Proposal: Certification [ In reply to ]

saintonge at telus

Oct 31, 2002, 11:00 AM

Post #11 of 11 (994 views)

Permalink

Jimmy Wales wrote:

>Summary: I read today that only 6% of the people on the planet have
>ever logged into the Internet. Only about half have ever used a
>telephone. So Erik's concerns about schoolchildren in India do
>resonate with me. I just think it's too early to settle on any
>particular certification scheme.
>
>erik_moeller@gmx.de wrote:
>
>>Last but not least, we should not forget that WP is intended to be a
>>useful tool, not just for those who like to write, but for those who like
>>to read, too. Be it interested adults or curious children, rich or poor,
>>we want Wikipedia to be accessible. We may want to distribute it on CD-
>>ROMs and on paper. Then how on Earth are the schoolchildren in India going
>>to wade through megabytes of Middle Earth mythology and stubs, if not if
>>we supply them with at least the option to filter articles according to
>>criteria developed collaboratively by various teams, working together to
>>find the sparkling gems among the ocean we are creating?
>>
>I totally agree with all of this. But I'm not sure that the time is
>ripe just yet. Perhaps in another year. I'm open to alternative
>points of view on this, but I think we're not close enough yet to
>"well rounded" to start seriously worrying about these issues.
>
The concept is intriguing, but I agree with Jimbo that we still have a
long way to go before we can have any positive impact on the third
world. Just sending them CDs containing "our" encyclopaedia will likely
not accomplish very much. In some of these places the best use for the
CDs would be as frisbees unless they have adequate hardware.

If, as some of us believe, a project like Wikipedia is to have any
influence in the third world it would likely be practical to concentrate
efforts in some selected community or country. We would likely need to
find some co-operative arrangement with the target country, notably with
its education bureaucracy. Even if there is enough hardware
infrastructure simply sending CDs would not be a useful effort, and we
would still have some expense in producing all those CDs. Encarta or
some other large company that is now producing CD-encyclopaedias could
dump its obsolete stock there much more efficiently than we can.

An interesting possibility would arise if we could transfer the "wiki
way" along with the encyclopaedias. Middle Earth mythology may very
well be alien to these cultures, but the wiki way could be applied in a
study of local literature and mythology. I view the critical challenge
for education is to communicate critical thinking skills; this has
become just as important as the traditional 3 Rs. Third world countries
cannot hope cannot hope to advance without that skill being present in
very broad cross-sections of their populations. I believe that
Wikipedia could eventually help there.

It's just something to think about.
Eclecticology