Hi everybody,
So, you know the code I posted a week or so ago for what I called the
"index trigger"?
It's having a moment, and I'm wondering if anything in the error
messages will ring bells with any of you.
Background:
1. This template gets called from cron every three minutes. Its job is
to look through recently-published stories, figure out what covers need
to be updated, and then publish those covers.
2. Sometimes publishing all those covers takes a while (say, longer than
three minutes). When that happened, a big stack of bric_republish
processes threatened to take down the entire server.
3. So I added a bit to the top of the template, which I hoped would
prevent the nightmare situation described in #2. Here's the start of the
<%init> block:
my @running_copies_of_myself = Bric::Util::Job->list({
executing => 1,
story_instance_id => $story->get_version_id,
failed => 0,
});
if ($running_copies_of_myself[1]) {
use Data::Dumper;
$m->out(Dumper(@running_copies_of_myself));
return 0;
}
About 90% of the time, this works perfectly. If one of these templates
is already going, trying to launch another process just exits the way I
had hoped.
But when the server is under load, the Job->list() frequently hangs for
a long time. Long enough that it's still running three minutes later, so
the processes pile up.
Sometimes the Job->list() gives up. In that case, here's what the Dumper
dumps out:
$VAR1 = bless( { 'priority' => '3',
'_class_id' => '80',
'story_instance_id' => '68277',
'media_instance_id' => undef,
'sched_time' => '2010-03-23 21:45:03',
'id' => 1616337, 'tries' => '2',
'name' => 'Publish "Index trigger"',
'_executing' => '1',
'error_message' => 'Unable to execute SQL statement:
DBD::Pg::st execute failed: ERROR: deadlock detected DETAIL: Process
30142 waits for ShareLock on transaction 187816962; blocked by process
30000. Process 30000 waits for ShareLock on transaction 187816914;
blocked by process 30142. [for Statement " UPDATE resource SET id = ?,
parent_id = ?, path = ?, uri = ?, size = ?, mod_time = ?, is_dir = ?,
media_type__id = (SELECT id FROM media_type WHERE name = ?) WHERE id = ?
" with ParamValues: 9=\'99437\']
at /home/rsn/bricolage_dev/lib/Bric/Util/DBI.pm line 1087, line 19.
[/home/rsn/bricolage_dev/lib/Bric/Util/DBI.pm:1088]
[/home/rsn/bricolage_dev/lib/Bric/Dist/Resource.pm:1676]
[/home/rsn/bricolage_dev/lib/Bric/Util/Burner.pm:2045]
[/home/rsn/bricolage_dev/lib/Bric/Util/Burner/Mason.pm:598]
[/home/rsn/bricolage_dev/lib/Bric/Util/Burner/Mason.pm:297]
[/home/rsn/bricolage_dev/lib/Bric/Util/Burner.pm:1518]
[/home/rsn/bricolage_dev/lib/Bric/Util/Burner.pm:1271]
[/home/rsn/bricolage_dev/lib/Bric/Util/Job/Pub.pm:193]
[/home/rsn/bricolage_dev/lib/Bric/Util/Job.pm:1889]
[/home/rsn/bricolage_dev/lib/Bric/Util/Job.pm:1808]
[/home/rsn/bricolage_dev/lib/Bric/SOAP/Workflow.pm:294]
[/usr/lib/perl5/vendor_perl/5.8.4/SOAP/Lite.pm:2322]
[/usr/lib/perl5/vendor_perl/5.8.4/SOAP/Transport/HTTP.pm:286]
[/usr/lib/perl5/vendor_perl/5.8.4/SOAP/Transport/HTTP.pm:464]
[/home/rsn/bricolage_dev/lib/Bric/SOAP/Handler.pm:184] [PerlHandler
subroutine `Bric::SOAP::Handler::handler\':1] ',
'comp_time' => undef,
'_failed' => '0',
'grp_ids' => [ '30' ],
'_dirty' => undef,
'user_id' => '1168',
'type' => '0'
},
'Bric::Util::Job::Pub' );
$VAR2 = bless( { 'priority' => '3',
'_class_id' => '80',
'story_instance_id' => '68277',
'media_instance_id' => undef,
'sched_time' => '2010-03-23 21:48:00',
'id' => 1616345,
'tries' => '1',
'name' => 'Publish "Index trigger"',
'_executing' => '1',
'error_message' => undef,
'comp_time' => undef,
'_failed' => '0',
'grp_ids' => [ '30' ],
'_dirty' => undef,
'user_id' => '1168',
'type' => '0' },
'Bric::Util::Job::Pub' );
Anyway, I'm feeling a bit lost. Does anybody recognize the deadlock
error, or have suggestions for troubleshooting it?
Thanks,
Bret
--
Bret Dawson
Producer
Pectopah Productions Inc.
(416) 895-7635
bret@pectopah.com
www.pectopah.com
So, you know the code I posted a week or so ago for what I called the
"index trigger"?
It's having a moment, and I'm wondering if anything in the error
messages will ring bells with any of you.
Background:
1. This template gets called from cron every three minutes. Its job is
to look through recently-published stories, figure out what covers need
to be updated, and then publish those covers.
2. Sometimes publishing all those covers takes a while (say, longer than
three minutes). When that happened, a big stack of bric_republish
processes threatened to take down the entire server.
3. So I added a bit to the top of the template, which I hoped would
prevent the nightmare situation described in #2. Here's the start of the
<%init> block:
my @running_copies_of_myself = Bric::Util::Job->list({
executing => 1,
story_instance_id => $story->get_version_id,
failed => 0,
});
if ($running_copies_of_myself[1]) {
use Data::Dumper;
$m->out(Dumper(@running_copies_of_myself));
return 0;
}
About 90% of the time, this works perfectly. If one of these templates
is already going, trying to launch another process just exits the way I
had hoped.
But when the server is under load, the Job->list() frequently hangs for
a long time. Long enough that it's still running three minutes later, so
the processes pile up.
Sometimes the Job->list() gives up. In that case, here's what the Dumper
dumps out:
$VAR1 = bless( { 'priority' => '3',
'_class_id' => '80',
'story_instance_id' => '68277',
'media_instance_id' => undef,
'sched_time' => '2010-03-23 21:45:03',
'id' => 1616337, 'tries' => '2',
'name' => 'Publish "Index trigger"',
'_executing' => '1',
'error_message' => 'Unable to execute SQL statement:
DBD::Pg::st execute failed: ERROR: deadlock detected DETAIL: Process
30142 waits for ShareLock on transaction 187816962; blocked by process
30000. Process 30000 waits for ShareLock on transaction 187816914;
blocked by process 30142. [for Statement " UPDATE resource SET id = ?,
parent_id = ?, path = ?, uri = ?, size = ?, mod_time = ?, is_dir = ?,
media_type__id = (SELECT id FROM media_type WHERE name = ?) WHERE id = ?
" with ParamValues: 9=\'99437\']
at /home/rsn/bricolage_dev/lib/Bric/Util/DBI.pm line 1087, line 19.
[/home/rsn/bricolage_dev/lib/Bric/Util/DBI.pm:1088]
[/home/rsn/bricolage_dev/lib/Bric/Dist/Resource.pm:1676]
[/home/rsn/bricolage_dev/lib/Bric/Util/Burner.pm:2045]
[/home/rsn/bricolage_dev/lib/Bric/Util/Burner/Mason.pm:598]
[/home/rsn/bricolage_dev/lib/Bric/Util/Burner/Mason.pm:297]
[/home/rsn/bricolage_dev/lib/Bric/Util/Burner.pm:1518]
[/home/rsn/bricolage_dev/lib/Bric/Util/Burner.pm:1271]
[/home/rsn/bricolage_dev/lib/Bric/Util/Job/Pub.pm:193]
[/home/rsn/bricolage_dev/lib/Bric/Util/Job.pm:1889]
[/home/rsn/bricolage_dev/lib/Bric/Util/Job.pm:1808]
[/home/rsn/bricolage_dev/lib/Bric/SOAP/Workflow.pm:294]
[/usr/lib/perl5/vendor_perl/5.8.4/SOAP/Lite.pm:2322]
[/usr/lib/perl5/vendor_perl/5.8.4/SOAP/Transport/HTTP.pm:286]
[/usr/lib/perl5/vendor_perl/5.8.4/SOAP/Transport/HTTP.pm:464]
[/home/rsn/bricolage_dev/lib/Bric/SOAP/Handler.pm:184] [PerlHandler
subroutine `Bric::SOAP::Handler::handler\':1] ',
'comp_time' => undef,
'_failed' => '0',
'grp_ids' => [ '30' ],
'_dirty' => undef,
'user_id' => '1168',
'type' => '0'
},
'Bric::Util::Job::Pub' );
$VAR2 = bless( { 'priority' => '3',
'_class_id' => '80',
'story_instance_id' => '68277',
'media_instance_id' => undef,
'sched_time' => '2010-03-23 21:48:00',
'id' => 1616345,
'tries' => '1',
'name' => 'Publish "Index trigger"',
'_executing' => '1',
'error_message' => undef,
'comp_time' => undef,
'_failed' => '0',
'grp_ids' => [ '30' ],
'_dirty' => undef,
'user_id' => '1168',
'type' => '0' },
'Bric::Util::Job::Pub' );
Anyway, I'm feeling a bit lost. Does anybody recognize the deadlock
error, or have suggestions for troubleshooting it?
Thanks,
Bret
--
Bret Dawson
Producer
Pectopah Productions Inc.
(416) 895-7635
bret@pectopah.com
www.pectopah.com