(moving to the wikitech-l list; see sign-up and archive page at
http://www.wikipedia.org/mailman/listinfo/wikitech-l )
Jonathan Walther wrote:
> I've done some work at converting the Wikipedia to Postgres, but am not
> there yet. So, let's put that aside for now.
Great! I did get postgresql installed on my machine, but got bogged down
in details of converting the table definitions and various interface
behaviors. Someone with prior experience working with postgres would be
a big help there.
> It seems that the wiki "source" is "interpreted" into html every single
> time someone accesses a link. That seems like a lot of overhead.
> Given that for every time a change is made to the wiki source to a page,
> several people "view" it, why not just regenerate the html only when
> changes are made, and store it? It would take more storage space, but
> should be MUCH faster. And if storage is an issue, I can donate some
> hard drives...
We used to cache in the phase II days on the old server. This was
removed for two reasons:
1) Wiki->HTML rendering is still pretty darn fast, particularly with our
new dedicated server; database contention seems to be our main problem
during high-load periods.
2) We had problems keeping the cache consistent with the old code.
On number 2, I would certainly welcome an improved cache subsystem
that's designed right from the ground up. The old one was hacked in as a
"crap! the system's unusably slow, let's hack in some improved code"
On number 1, note that LinkCache::addLink() does a brief query on the
cur table for every link when rendering a wikipage. These could probably
be consolidated somehow or other. (Note that this does not apply to
Recentchanges, which loads everything in a big chunk.)
> The savings on the Recent Changes page alone should work wonders.
On the English wikipedia, Recentchanges is loaded at default options
about 3000 times per day; the number of edits per day is a similar
figure, and every edit means the page has to change to reflect it.
Caching the rendered display wouldn't seem to save significantly over
rerendering it on each view.
-- brion vibber (brion @ pobox.com)
http://www.wikipedia.org/mailman/listinfo/wikitech-l )
Jonathan Walther wrote:
> I've done some work at converting the Wikipedia to Postgres, but am not
> there yet. So, let's put that aside for now.
Great! I did get postgresql installed on my machine, but got bogged down
in details of converting the table definitions and various interface
behaviors. Someone with prior experience working with postgres would be
a big help there.
> It seems that the wiki "source" is "interpreted" into html every single
> time someone accesses a link. That seems like a lot of overhead.
> Given that for every time a change is made to the wiki source to a page,
> several people "view" it, why not just regenerate the html only when
> changes are made, and store it? It would take more storage space, but
> should be MUCH faster. And if storage is an issue, I can donate some
> hard drives...
We used to cache in the phase II days on the old server. This was
removed for two reasons:
1) Wiki->HTML rendering is still pretty darn fast, particularly with our
new dedicated server; database contention seems to be our main problem
during high-load periods.
2) We had problems keeping the cache consistent with the old code.
On number 2, I would certainly welcome an improved cache subsystem
that's designed right from the ground up. The old one was hacked in as a
"crap! the system's unusably slow, let's hack in some improved code"
On number 1, note that LinkCache::addLink() does a brief query on the
cur table for every link when rendering a wikipage. These could probably
be consolidated somehow or other. (Note that this does not apply to
Recentchanges, which loads everything in a big chunk.)
> The savings on the Recent Changes page alone should work wonders.
On the English wikipedia, Recentchanges is loaded at default options
about 3000 times per day; the number of edits per day is a similar
figure, and every edit means the page has to change to reflect it.
Caching the rendered display wouldn't seem to save significantly over
rerendering it on each view.
-- brion vibber (brion @ pobox.com)