Mailing List Archive

I tried again with r3550, it segfaulted after indexing a similar
number of documents, at least 4910000. Here is the backtrace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1210976064 (LWP 5187)]
0xb7dc5197 in memset () from /lib/i686/cmov/libc.so.6
(gdb) bt
#0 0xb7dc5197 in memset () from /lib/i686/cmov/libc.so.6
#1 0xb7ae05c6 in kino_VA_grow (self=0xbfe13290, capacity=94) at
../c_src/KinoSearch/Util/VArray.c:231
#2 0xb7ae26d0 in kino_TokenBatch_append (self=0xbfe13290,
token=0xbfe158b8) at ../c_src/h/KinoSearch/Util/VArray.h:331
#3 0xb7afb174 in kino_Tokenizer_tokenize_str (self=0x83793b8,
string=0xbfe15ae8 "space-time organization in macromolecular
fluids : proceedings of the eleventh taniguchi international
symposium, hakone, japan, november 7-12, 1988 / f. tanaka, m. doi, t.
ohta, eds.", string_len=183, batch=0xbfe13290) at
../c_src/h/KinoSearch/Analysis/TokenBatch.h:342
#4 0xb7ae295c in kino_Tokenizer_transform (self=0x83793b8,
batch=0xbfe15a20) at ../c_src/h/KinoSearch/Analysis/Tokenizer.h:237
#5 0xb7ae22c5 in kino_PolyAnalyzer_transform_text (self=0x831cd68,
text=0x8402430) at ../c_src/h/KinoSearch/Analysis/Analyzer.h:197
#6 0xb7ae961e in kino_Inverter_add_field (self=0x8400d90,
fspec=0x83da938, field_name=0x816f8f8 "all", field_name_len=3,
value=0xbfe15e48 "Space-time organization in macromolecular fluids
: proceedings of the eleventh Taniguchi International Symposium,
Hakone, Japan, November 7-12, 1988 / F. Tanaka, M. Doi, T. Ohta,
eds.", value_len=183) at ../c_src/h/KinoSearch/Analysis/Analyzer.h:207
#7 0xb7afb66c in kino_SegWriter_add_doc (self=0x83e1980,
doc=0x83e01c8) at ../c_src/h/KinoSearch/Index/Inverter.h:277
#8 0xb7a3f221 in XS_KinoSearch__Index__SegWriter_add_doc
(my_perl=0x8150008, cv=0x81dc2dc) at
../c_src/h/KinoSearch/Index/SegWriter.h:249
#9 0x080c0923 in Perl_pp_entersub ()
#10 0x080bf2fb in Perl_runops_standard ()
#11 0x0806721b in perl_run ()
#12 0x08063752 in main ()
(gdb)

--
Edward.

_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

On Jun 27, 2008, at 1:04 AM, Edward Betts wrote:

> [Switching to Thread -1210591040 (LWP 27080)]

This isn't a multi-threaded application, is it? (KS isn't thread-
safe.) I don't immediately see anything else out of the ordinary in
either of the two backtraces.

The first crash happened when manipulating a refcount on a compile-
time-created vtable object which is supposed to be immortal. The
other occurred in a completely different place, when zeroing out some
newly allocated memory after extending an array. There's seemingly no
pattern. Maybe it's the delayed effect of earlier memory corruption.
I'll set the valgrind tests running and see if anything turns up.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

2008/6/27 Marvin Humphrey <marvin@rectangular.com>:
>
> On Jun 27, 2008, at 1:04 AM, Edward Betts wrote:
>
>> [Switching to Thread -1210591040 (LWP 27080)]
>
> This isn't a multi-threaded application, is it? (KS isn't thread-safe.) I
> don't immediately see anything else out of the ordinary in either of the two
> backtraces.

The data to be indexed is being read from a pipe, but there are no
threads involved.

--
Edward.

_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

On Jun 27, 2008, at 2:01 AM, Edward Betts wrote:

>
> The data to be indexed is being read from a pipe, but there are no
> threads involved.


Could we be seeing an out-of-memory error? KS doesn't wrap malloc
calls to verify that NULL wasn't returned. (It should probably
start.) That would explain at least one of the errors, and possibly
the other if in fact it was the assignment to the newly allocated
"self" that failed rather than the refcount increment:

24 self->_ = REFCOUNT_INC(vtable);

Do you see memory growing out of control? I can see at least a couple
leaks as the valgrind tests progress.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

2008/6/27 Marvin Humphrey <marvin@rectangular.com>:
> Could we be seeing an out-of-memory error?

Possibly. The machine has 2GB of RAM. I'll run it again and keep an
eye on memory usage.

--
Edward.

_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

2008/6/27 Edward Betts <edwardbetts@gmail.com>:
> 2008/6/27 Marvin Humphrey <marvin@rectangular.com>:
>> Could we be seeing an out-of-memory error?
>
> Possibly. The machine has 2GB of RAM. I'll run it again and keep an
> eye on memory usage.

It might be an out of memory problem. After running for a few minutes
perl is using 1.5G of memory.

--
Edward.

_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

On Jun 27, 2008, at 2:37 AM, Edward Betts wrote:

> It might be an out of memory problem. After running for a few minutes
> perl is using 1.5G of memory.

Yeah, I just found and fixed a big leak in xs/KinoSearch/Analysis/
LCNormalizer.c, which was draining memory at around the rate of
indexed text throughput. There may be other leaks; I haven't checked
rigorously lately.

It's a bit late for me to nail all of them tonight, but I'll tackle
this tomorrow.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

2008/6/27 Marvin Humphrey <marvin@rectangular.com>:
> Yeah, I just found and fixed a big leak in
> xs/KinoSearch/Analysis/LCNormalizer.c, which was draining memory at around
> the rate of indexed text throughput. There may be other leaks; I haven't
> checked rigorously lately.

Thanks, the index finished without running out of memory.

--
Edward.

_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

On Jun 27, 2008, at 2:52 PM, Edward Betts wrote:

> Thanks, the index finished without running out of memory.


Groovy. :)

Memory leakage has been a recurring, systemic problem for svn trunk
users like yourself, because the ordinary test suite doesn't catch
such problems and refcounting errors are so easy to make. Since the
last discussion on this topic, I've been introduced to the CPAN module
Test::Valgrind. Hopefully we can use that to set up a systemic
solution: nightly automated smokes.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

Marvin Humphrey wrote on 6/27/08 5:04 PM:
>
> On Jun 27, 2008, at 2:52 PM, Edward Betts wrote:
>
>> Thanks, the index finished without running out of memory.
>
>
> Groovy. :)
>
> Memory leakage has been a recurring, systemic problem for svn trunk
> users like yourself, because the ordinary test suite doesn't catch such
> problems and refcounting errors are so easy to make. Since the last
> discussion on this topic, I've been introduced to the CPAN module
> Test::Valgrind. Hopefully we can use that to set up a systemic
> solution: nightly automated smokes.
>

How timely. The last few nights I've been poking at the smoke test script I
posted here several months ago, and think I've got it to a useable place. I just
committed it to trunk as r3552.

I also read the pod for Test::Valgrind. I *think* smoke.pl could be altered to
use it conditionally when it is installed. I committed smoke.pl so others can
play with that idea too.

--
Peter Karman . http://peknet.com/ . peter@peknet.com

_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

On Jun 28, 2008, at 6:13 AM, Peter Karman wrote:

> The last few nights I've been poking at the smoke test script I
> posted here several months ago, and think I've got it to a useable
> place. I just committed it to trunk as r3552.

Sweet! :)

> I also read the pod for Test::Valgrind. I *think* smoke.pl could be
> altered to use it conditionally when it is installed.


It will definitely work. I see that smoke.pl shells out to "./Build
test". Test::Valgrind is designed to wrap an existing test suite, so
we'll set up a new build target: './Build test_valgrind'. From there,
it's just fiddling with command line arguments or config settings to
determine whether smoke.pl should run "test" or "test_valgrind".

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

On Jun 28, 2008, at 6:13 AM, Peter Karman wrote:

> I also read the pod for Test::Valgrind. I *think* smoke.pl could be
> altered to use it conditionally when it is installed.

Hmm, turns out I had some trouble with false passes using
Test::Valgrind 0.04. It happily installed on my system, but there
don't seem to be any tests in Test::Valgrind's test suite which probe
for anything other than a clean run, so it's not clear whether it
really works as intended. When I alter a test from the KS suite which
I *know* to leak right now (patch below), Test::Valgrind gives it a
clean bill of health, but doesn't seem to run the actual test (output
below).

Nevertheless, as of r3554, KS has a test_valgrind build target,
consisting of what was in trunk/devel/bin/valgrind_test.plx augmented
with screen-scraping. That was the lazy way out, since rooting around
in Test::Valgrind looked kind of tricky.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

PATCH:

Index: t/210-deldocs.t
===================================================================
--- t/210-deldocs.t (revision 3553)
+++ t/210-deldocs.t (working copy)
@@ -2,8 +2,11 @@
use warnings;
use lib 'buildlib';

-use Test::More tests => 13;
+use Test::More;
+use Test::Valgrind diag => 1;

use KinoSearch::Index::DelDocs;


Test::Valgrind OUTPUT:

marvin@linlin:~/projects/ks/perl$ /usr/local/debugperl/bin/perl5.10.0 -
Mblib t/210-deldocs.t
1..5
# valgrind --tool=memcheck --leak-check=full --leak-resolution=high --
num-callers=50 --error-limit=yes --suppressions=/usr/local/debugperl/
lib/site_perl/5.10.0/i686-linux/Test/Valgrind/perlTestValgrind.supp --
show-reachable=yes --leak-check=full /usr/local/debugperl/bin/
perl5.10.0 -Ibuildlib -I/home/marvin/projects/ks/perl/blib/arch -I/
home/marvin/projects/ks/perl/blib/lib -I/usr/local/debugperl/lib/
5.10.0/i686-linux -I/usr/local/debugperl/lib/5.10.0 -I/usr/local/
debugperl/lib/site_perl/5.10.0/i686-linux -I/usr/local/debugperl/lib/
site_perl/5.10.0 -I. -MTest::Valgrind=run,1 /usr/local/debugperl/lib/
site_perl/5.10.0/i686-linux/Test/Valgrind.pm
# ==17956== Memcheck, a memory error detector.
# ==17956== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward
et al.
# ==17956== Using LibVEX rev 1732, a library for dynamic binary
translation.
# ==17956== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
# ==17956== Using valgrind-3.2.3-Debian, a dynamic binary
instrumentation framework.
# ==17956== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward
et al.
# ==17956== For more details, rerun with: -v
# ==17956==
# Subroutine import redefined at /usr/local/debugperl/lib/site_perl/
5.10.0/i686-linux/Test/Valgrind.pm line 78.
# ==17956==
# ==17956== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 63
from 3)
# ==17956== malloc/free: in use at exit: 2,771 bytes in 15 blocks.
# ==17956== malloc/free: 43,148 allocs, 43,133 frees, 2,086,050 bytes
allocated.
# ==17956== For counts of detected errors, rerun with: -v
# ==17956== searching for pointers to 15 not-freed blocks.
# ==17956== checked 300,652 bytes.
# ==17956==
# ==17956== LEAK SUMMARY:
# ==17956== definitely lost: 0 bytes in 0 blocks.
# ==17956== possibly lost: 0 bytes in 0 blocks.
# ==17956== still reachable: 0 bytes in 0 blocks.
# ==17956== suppressed: 2,771 bytes in 15 blocks.
ok 1 - valgrind errors
ok 2 - valgrind definitely lost
ok 3 - valgrind indirectly lost
ok 4 - valgrind possibly lost
ok 5 - valgrind still reachable
marvin@linlin:~/projects/ks/perl$


_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch