Mailing List Archive

Author: jm
Date: Mon Nov 14 17:33:49 2005
New Revision: 344284

URL: http://svn.apache.org/viewcvs?rev=344284&view=rev
Log:
move corpus link-farm rebuilds to a periodic task, instead of attempting to do it every time we do a mass-check; the latter just isn't working out due to race conditions and buildbot timeouts

Added:
spamassassin/trunk/build/automc/freqsd-infrequent (with props)
Modified:
spamassassin/trunk/build/automc/buildbot_ready
spamassassin/trunk/build/automc/freqsd
spamassassin/trunk/build/automc/populate_cor

Modified: spamassassin/trunk/build/automc/buildbot_ready
URL: http://svn.apache.org/viewcvs/spamassassin/trunk/build/automc/buildbot_ready?rev=344284&r1=344283&r2=344284&view=diff
==============================================================================
--- spamassassin/trunk/build/automc/buildbot_ready (original)
+++ spamassassin/trunk/build/automc/buildbot_ready Mon Nov 14 17:33:49 2005
@@ -71,8 +71,9 @@

sub _perform_singleton_actions_unlocked
{
- print "got lock: performing singleton actions\n";
- run "build/automc/populate_cor";
+ # print "got lock: performing singleton actions\n";
+ # run "build/automc/populate_cor";
+ # better off doing this periodically from the qproc
}

# ---------------------------------------------------------------------------

Modified: spamassassin/trunk/build/automc/freqsd
URL: http://svn.apache.org/viewcvs/spamassassin/trunk/build/automc/freqsd?rev=344284&r1=344283&r2=344284&view=diff
==============================================================================
--- spamassassin/trunk/build/automc/freqsd (original)
+++ spamassassin/trunk/build/automc/freqsd Mon Nov 14 17:33:49 2005
@@ -123,9 +123,15 @@
$logdir = "--dir ".$job->{metadata}->{dir};
}

+ # create slow reports
run ("cd masses/rule-qa; ./corpus-hourly ".
"--override='output_classes=$child_reports' ".
"$logdir");
+
+ print "running 'freqsd-infrequent' tasks ($$) at ".(scalar localtime time)."\n";
+
+ # recreate the corpus link-farm
+ run ("build/automc/freqsd-infreqent");

print "completed slow reports ($$) at ".(scalar localtime time)."\n";


Added: spamassassin/trunk/build/automc/freqsd-infrequent
URL: http://svn.apache.org/viewcvs/spamassassin/trunk/build/automc/freqsd-infrequent?rev=344284&view=auto
==============================================================================
--- spamassassin/trunk/build/automc/freqsd-infrequent (added)
+++ spamassassin/trunk/build/automc/freqsd-infrequent Mon Nov 14 17:33:49 2005
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+# recreate the corpus link-farm
+./build/automc/popular_cor
+

Propchange: spamassassin/trunk/build/automc/freqsd-infrequent
------------------------------------------------------------------------------
svn:executable = *

Modified: spamassassin/trunk/build/automc/populate_cor
URL: http://svn.apache.org/viewcvs/spamassassin/trunk/build/automc/populate_cor?rev=344284&r1=344283&r2=344284&view=diff
==============================================================================
--- spamassassin/trunk/build/automc/populate_cor (original)
+++ spamassassin/trunk/build/automc/populate_cor Mon Nov 14 17:33:49 2005
@@ -10,15 +10,26 @@
cd /home/bbmass/rawcor
date

+mkdir /home/bbmass/tmpfs/newcor
+
TMPDIR=/home/bbmass/tmpfs/tmp \
/local/perl586/bin/perl \
$SADIR/masses/corpora/mk-corpus-link-farm \
- -after="4 months ago" \
- -dest /home/bbmass/tmpfs/cor/mc-fast -num 1000 \
- -dest /home/bbmass/tmpfs/cor/mc-med -num 5000 \
- -dest /home/bbmass/tmpfs/cor/mc-slow -num 10000 \
- -dest /home/bbmass/tmpfs/cor/mc-slower \
+ -after="6 months ago" \
+ -dest /home/bbmass/tmpfs/newcor/mc-fast -num 1000 \
+ -dest /home/bbmass/tmpfs/newcor/mc-med -num 5000 \
+ -dest /home/bbmass/tmpfs/newcor/mc-slow -num 10000 \
+ -dest /home/bbmass/tmpfs/newcor/mc-slower \
*
+
+rm -rf /home/bbmass/tmpfs/cor.old # just in case
+
+# and put the new corpus linkfarm in place...
+mv /home/bbmass/tmpfs/cor /home/bbmass/tmpfs/cor.old
+mv /home/bbmass/tmpfs/newcor /home/bbmass/tmpfs/cor
+
+# remove the old one
+rm -rf /home/bbmass/tmpfs/cor.old

for dir in mc-fast mc-med mc-slow mc-slower
do