Mailing List Archive

One additional bit of info. I'm running this under mod_perl (latest),
with the searcher held in class data. I suspect it's a concurrency
issue, actually. Any thoughts?

Thanks again,

Matthew



On Jun 11, 2007, at 2:42 PM, Matthew Berk wrote:

> Hi Marvin,
>
> First off, thanks for all your great work with KinoSearch!
>
> I'm running into a little problem that sounds a lot like the once
> posted here:
>
> http://www.rectangular.com/pipermail/kinosearch/2006-May/000157.html
>
> Specifics:
>
> - index has about 15m docs
> - using KinoSearch 0.15
> - running on Debian (Linux oldev4.sea 2.6.18.5-amd64 #2 SMP Fri
> Dec 15 12:21:08 PST 2006 x86_64 GNU/Linux)
> - the error is repeatable; here's the error:
>
> Internal error: field_num 111 > max_field_num 6 at /site/perl/
> perl-5.8.8/site_perl/5.8.8/x86_64-linux/KinoSearch/Index/
> TermInfosReader.pm line 92\n
> \tKinoSearch::Index::TermInfosReader::_scan_enum
> ('KinoSearch::Index::TermInfosReader=HASH(0x93a730)', '\\x{0}\\x{4}
> 23843') called at /site/perl/perl-5.8.8/site_perl/5.8.8/x86_64-
> linux/KinoSearch/Index/TermInfosReader.pm line 64\n
> \tKinoSearch::Index::TermInfosReader::fetch_term_info
> ('KinoSearch::Index::TermInfosReader=HASH(0x93a730)',
> 'KinoSearch::Index::Term=HASH(0x235bc20)') called at /site/perl/
> perl-5.8.8/site_perl/5.8.8/x86_64-linux/KinoSearch/Index/
> SegReader.pm ETC.
>
> - the indexer has 7 spec_fields, 1 of which is not indexed
> - the index file has been optimized, and is about 4.1G
>
> I'm going to set K_DEBUG to 1 and see what I can see. Any thoughts?
>
> Thanks,
>
> Matthew
>



Marchex, Inc.
http://www.marchex.com

This email message and any attachments are solely for intended
recipients, and may contain information that is privileged and
confidential. If you are not the intended recipient, any
dissemination, distribution or copying is strictly prohibited. If you
believe that you may have received this message in error, please
immediately notify the sender by replying to this e-mail message.

On Jun 11, 2007, at 4:32 PM, Matthew Berk wrote:

> One additional bit of info. I'm running this under mod_perl
> (latest), with the searcher held in class data. I suspect it's a
> concurrency issue, actually. Any thoughts?

Yes. KinoSearch isn't thread safe.

The error message you saw suggests that an input stream has gotten
out of sync. It's at the wrong place in the file and reading data
that the module considers insane; there's a sanity check in place to
give you that exception and head off the looming segfault.

The error from a year ago was due to a 32-bit bottleneck I'd missed
truncating a file pointer. Since then, I don't think I've received
any reports of 0.15 i/o sync errors other than yours.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

Would a simple locking mechanism (I know you have one for .2) work to alleviate the problem under mod_perl?

-----Original Message-----
From: kinosearch-bounces+matthew=openlist.com@rectangular.com on behalf of Marvin Humphrey
Sent: Mon 6/11/2007 4:50 PM
To: KinoSearch discussion forum
Subject: Re: [KinoSearch] Re: Internal error: field_num 111 > max_field_num 6


On Jun 11, 2007, at 4:32 PM, Matthew Berk wrote:

> One additional bit of info. I'm running this under mod_perl
> (latest), with the searcher held in class data. I suspect it's a
> concurrency issue, actually. Any thoughts?

Yes. KinoSearch isn't thread safe.

The error message you saw suggests that an input stream has gotten
out of sync. It's at the wrong place in the file and reading data
that the module considers insane; there's a sanity check in place to
give you that exception and head off the looming segfault.

The error from a year ago was due to a 32-bit bottleneck I'd missed
truncating a file pointer. Since then, I don't think I've received
any reports of 0.15 i/o sync errors other than yours.


Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/ms-tnef
Size: 3713 bytes
Desc: not available
Url : http://www.rectangular.com/pipermail/kinosearch/attachments/20070611/b7e64b99/attachment.bin

On Jun 11, 2007, at 9:32 PM, Matthew Berk wrote:

> Would a simple locking mechanism (I know you have one for .2) work
> to alleviate the problem under mod_perl?

Serializing access, say by forcing each complete search into a
subroutine and preventing everybody else from using the shared
resources until the call returns... I think that would solve the
concurrency issue.

However, you're going to get memory errors, because when one of
KinoSearch's C struct objects gets copied to multiple threads, the
internal refcount doesn't get incremented. (KS objects keep their
own refcounts, distinct from Perl's SvREFCNT.) When the first of
several Perl wrapper objects referencing the C object gets DESTROYed,
the KS refcount will fall to 0 and the C object will clean itself
up. From now on, any other Perl wrapper objects referencing that C
struct are pointing at freed memory. That's Bad.

A script which segfaults to illustrate the problem is below.

Solving this is hard. Incrementing the KS refcounts of objects
copied when Perl spawns a thread is a PITA, because CLONE gets
invoked as a *package* method. You have to maintain global
registries keeping track of all live object references, then iterate
over all the items in the registries when CLONE tips you off that a
thread is starting.

Maintaining such a scheme is messy, labor-intensive and error-prone
-- and that's the biggest reason among several why KS doesn't do
threads.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

#----------------------------------------------------------

#!/usr/bin/perl
use strict;
use warnings;

use KinoSearch::Store::FSFolder;
use Carp qw( cluck );
use File::Path qw( rmtree );

sub warn_destroy {
my $self = shift;
cluck "Destroying";
KinoSearch::Util::Obj::DESTROY($self);
}

{
no warnings 'once';
*KinoSearch::Store::FSFolder::DESTROY = *warn_destroy;
}

use threads;

rmtree 'foofolder';
mkdir 'foofolder' or die $!;

my $folder = KinoSearch::Store::FSFolder->new(
path => 'foofolder',
);
warn $folder;

for my $num ( 0 .. 10 ) {
my $thread = threads->create( sub {
$folder->refcount_inc;
my $out = $folder->open_outstream("file_$num");
});
$thread->join;
}

Does this problem also hold under MP, where each use is a separae proc? I take it the answer is yes, because child processes get copies of class data from the parent process. Any other recommendations for safely running this under MP?

Thanks!

Matthew



-----Original Message-----
From: kinosearch-bounces+matthew=openlist.com@rectangular.com on behalf of Marvin Humphrey
Sent: Tue 6/12/2007 7:53 AM
To: KinoSearch discussion forum
Subject: Re: [KinoSearch] Re: Internal error: field_num 111 > max_field_num 6


On Jun 11, 2007, at 9:32 PM, Matthew Berk wrote:

> Would a simple locking mechanism (I know you have one for .2) work
> to alleviate the problem under mod_perl?

Serializing access, say by forcing each complete search into a
subroutine and preventing everybody else from using the shared
resources until the call returns... I think that would solve the
concurrency issue.

However, you're going to get memory errors, because when one of
KinoSearch's C struct objects gets copied to multiple threads, the
internal refcount doesn't get incremented. (KS objects keep their
own refcounts, distinct from Perl's SvREFCNT.) When the first of
several Perl wrapper objects referencing the C object gets DESTROYed,
the KS refcount will fall to 0 and the C object will clean itself
up. From now on, any other Perl wrapper objects referencing that C
struct are pointing at freed memory. That's Bad.

A script which segfaults to illustrate the problem is below.

Solving this is hard. Incrementing the KS refcounts of objects
copied when Perl spawns a thread is a PITA, because CLONE gets
invoked as a *package* method. You have to maintain global
registries keeping track of all live object references, then iterate
over all the items in the registries when CLONE tips you off that a
thread is starting.

Maintaining such a scheme is messy, labor-intensive and error-prone
-- and that's the biggest reason among several why KS doesn't do
threads.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

#----------------------------------------------------------

#!/usr/bin/perl
use strict;
use warnings;

use KinoSearch::Store::FSFolder;
use Carp qw( cluck );
use File::Path qw( rmtree );

sub warn_destroy {
my $self = shift;
cluck "Destroying";
KinoSearch::Util::Obj::DESTROY($self);
}

{
no warnings 'once';
*KinoSearch::Store::FSFolder::DESTROY = *warn_destroy;
}

use threads;

rmtree 'foofolder';
mkdir 'foofolder' or die $!;

my $folder = KinoSearch::Store::FSFolder->new(
path => 'foofolder',
);
warn $folder;

for my $num ( 0 .. 10 ) {
my $thread = threads->create( sub {
$folder->refcount_inc;
my $out = $folder->open_outstream("file_$num");
});
$thread->join;
}





_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/ms-tnef
Size: 4757 bytes
Desc: not available
Url : http://www.rectangular.com/pipermail/kinosearch/attachments/20070612/7244d3de/attachment.bin

Marvin Humphrey writes:
> Maintaining such a [CLONE] scheme is messy, labor-intensive and error-prone
> -- and that's the biggest reason among several why KS doesn't do threads.

That seems reasonable. Am I right in thinking that adding something like
this to a suitable base class would help people avoid shooting themselves in
the foot?

sub CLONE {
croak "KinoSearch cannot currently run in a multi-threaded Perl";
}

--
Aaron Crane

On Jun 12, 2007, at 8:42 AM, Aaron Crane wrote:

> That seems reasonable. Am I right in thinking that adding
> something like
> this to a suitable base class would help people avoid shooting
> themselves in
> the foot?
>
> sub CLONE {
> croak "KinoSearch cannot currently run in a multi-threaded
> Perl";
> }

You're right, Aaron.

That method is actually present in 0.15, in KinoSearch::Util::Class...

sub CLONE {
my $package = shift;
die( "CLONE invoked by package '$package', indicating
that threads "
. "or Win32 fork were initiated, but KinoSearch is not
thread-safe"
);
}

... but I'd taken it out in the 0.20 branch.

I was thinking that it was draconian to kill any threaded app that
loaded any KinoSearch class, even if it was in some distant
dependency somewhere and KS wasn't actually used. Seemed like being
a bad citizen, somehow. But I've been persuaded by this email thread
that more pain results from the failure to throw an appropriate
exception. I'll restore the method as you suggest.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

Marvin Humphrey writes:
> That method is actually present in 0.15, in KinoSearch::Util::Class...
>
> sub CLONE {
> my $package = shift;
> die( "CLONE invoked by package '$package', indicating that threads "
> . "or Win32 fork were initiated, but KinoSearch is not thread-safe"
> );
> }
>
> ... but I'd taken it out in the 0.20 branch.

Ah, I grepped through 0.20_03 for it, but didn't think to check 0.15.

> I was thinking that it was draconian to kill any threaded app that loaded
> any KinoSearch class, even if it was in some distant dependency somewhere
> and KS wasn't actually used. Seemed like being a bad citizen, somehow.
> But I've been persuaded by this email thread that more pain results from
> the failure to throw an appropriate exception.

I know what you mean; it _is_ somewhat draconian. But, as you point out,
it's better than the alternatives, given that KS really can't work in a
multi-threaded environment at the moment.

--
Aaron Crane