Mailing List Archive

On Sep 8, 2008, at 5:39 PM, Paul-Kenji Cahier wrote:

> I have been desperatly trying to compile any of the .20_x versions
> under windows, under both activeperl and strawberry perl but without
> success.
> Using activeperl, I was not even able to start any of the compiles
> using nmake(failing very quickly).

While Windows is a target for KS, the dev branch has not worked under
Windows for a long time. I recently did enough work to get KS
compiling again using MSVC 9 and a manually-compiled Perl 5.10.0.
However, with that rig I'm getting a bunch of wacky "free to wrong
pool" memory errors which appear to originate in c_src/KinoSearch/
Index/PostingPool.c or thereabouts.

Perhaps it will be easier to get things going with Strawberry Perl.

What I see in your output is failure at the linking stage, because
Windows is fussy about resolving all symbols at link-time. Unixen
tend to be more forgiving at link-time, throwing runtime errors when a
symbol cannot be resolved, which is why at least one of the problems
remained hidden until now.

> ..\c_src\KinoSearch\Posting\MatchPosting.o:MatchPosting.c:(.text
> +0x15d): undefined reference to `kino_MatchPostScorer_init'

That one's easy, and should be fixed by r3844.

MatchPostingScorer*
-MatchPostingScorer_init(MatchPostingScorer *self, Similarity *sim,
+MatchPostScorer_init(MatchPostingScorer *self, Similarity *sim,
PostingList *plist, Compiler *compiler)

> ..\c_src\KinoSearch\Util\Compat\ProcessID.o:ProcessID.c:(.text
> +0x23): undefined reference to `kill'
> ..\c_src\KinoSearch\Util\Compat\Sleep.o:Sleep.c:(.text+0x5):
> undefined reference to `sleep'

Those two are a little more puzzling. Here's the relevant section of
Sleep.c:


/********************************* UNIXEN
*********************************/
#ifdef CHY_HAS_UNISTD_H

#include <unistd.h>

void
kino_Sleep_sleep(unsigned int seconds)
{
sleep(seconds);
}

/********************************* WINDOWS
********************************/
#elif defined(CHY_HAS_WINDOWS_H)


The WINDOWS section invokes "Sleep" rather than "sleep", so we can
deduce that we're in the UNIXEN section and that unistd.h is present.
The unistd.h header file should define a sleep() function: <http://opengroup.org/onlinepubs/007908799/xsh/sleep.html
>. Is that not the case with this install?

I imagine something similar is happening with the "kill" reference in
ProcessID.c, which ought to be provided by signal.h.

Please update to r3845 and let me know whether we've at least begun to
whittle down our list of obstacles.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

> On Sep 8, 2008, at 5:39 PM, Paul-Kenji Cahier wrote:

>> I have been desperatly trying to compile any of the .20_x versions
>> under windows, under both activeperl and strawberry perl but without
>> success.
>> Using activeperl, I was not even able to start any of the compiles
>> using nmake(failing very quickly).

> While Windows is a target for KS, the dev branch has not worked under
> Windows for a long time. I recently did enough work to get KS
> compiling again using MSVC 9 and a manually-compiled Perl 5.10.0.
> However, with that rig I'm getting a bunch of wacky "free to wrong
> pool" memory errors which appear to originate in c_src/KinoSearch/
> Index/PostingPool.c or thereabouts.

> Perhaps it will be easier to get things going with Strawberry Perl.
Strawberry perl actually uses mingw to provide as much unix
compatibility as possible, so it does go easier than activeperl.


> What I see in your output is failure at the linking stage, because
> Windows is fussy about resolving all symbols at link-time. Unixen
> tend to be more forgiving at link-time, throwing runtime errors when a
> symbol cannot be resolved, which is why at least one of the problems
> remained hidden until now.

>> ..\c_src\KinoSearch\Posting\MatchPosting.o:MatchPosting.c:(.text
>> +0x15d): undefined reference to `kino_MatchPostScorer_init'

> That one's easy, and should be fixed by r3844.

> MatchPostingScorer*
> -MatchPostingScorer_init(MatchPostingScorer *self, Similarity *sim,
> +MatchPostScorer_init(MatchPostingScorer *self, Similarity *sim,
> PostingList *plist, Compiler *compiler)
Thanks. That fixed the error.
I had to actually comment out:
typedef __int64 off64_t;
in compat/windows/lfs.h(since mingw already defines it)

I also had to #define DIR_SEP "/"
since apparently mingw doesnt define it.
(but there are mnor changes)


>> ..\c_src\KinoSearch\Util\Compat\ProcessID.o:ProcessID.c:(.text
>> +0x23): undefined reference to `kill'
>> ..\c_src\KinoSearch\Util\Compat\Sleep.o:Sleep.c:(.text+0x5):
>> undefined reference to `sleep'

> Those two are a little more puzzling. Here's the relevant section of
> Sleep.c:


> /********************************* UNIXEN
> *********************************/
> #ifdef CHY_HAS_UNISTD_H

> #include <unistd.h>

> void
> kino_Sleep_sleep(unsigned int seconds)
> {
> sleep(seconds);
> }

> /********************************* WINDOWS
> ********************************/
> #elif defined(CHY_HAS_WINDOWS_H)


> The WINDOWS section invokes "Sleep" rather than "sleep", so we can
> deduce that we're in the UNIXEN section and that unistd.h is present.
> The unistd.h header file should define a sleep() function: <http://opengroup.org/onlinepubs/007908799/xsh/sleep.html
I checked the mingw headers, and there's no sleep function in them, though
they do tell that one should call the win32 sleep or _sleep, which I changed
and worked for the best as it stopped the sleep error.


>>. Is that not the case with this install?

> I imagine something similar is happening with the "kill" reference in
> ProcessID.c, which ought to be provided by signal.h.

Sadly, for this one, I couldnt find an equivalent to kill.
So I tried making it always return true(and not call kill) which
did indeed let it finish compiling completly.
However it seems to be crashing on many tests.
(according to the debugger, it crashes in kinosearch's dll, but I
cant figure where since I dont have debug information)




> Please update to r3845 and let me know whether we've at least begun to
> whittle down our list of obstacles.

> Marvin Humphrey
> Rectangular Research
> http://www.rectangular.com/


> _______________________________________________
> KinoSearch mailing list
> KinoSearch@rectangular.com
> http://www.rectangular.com/mailman/listinfo/kinosearch


_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

On Sep 8, 2008, at 7:51 PM, Paul-Kenji Cahier wrote:

> Strawberry perl actually uses mingw to provide as much unix
> compatibility as possible, so it does go easier than activeperl.

Yeah. I use MSVC because it's *more* difficult. If code satisfies
both MSVC and GCC, we're in pretty good shape as far as portability.

>> MatchPostingScorer*
>> -MatchPostingScorer_init(MatchPostingScorer *self, Similarity *sim,
>> +MatchPostScorer_init(MatchPostingScorer *self, Similarity *sim,
>> PostingList *plist, Compiler *compiler)
> Thanks. That fixed the error.

Groovy.

> I had to actually comment out:
> typedef __int64 off64_t;
> in compat/windows/lfs.h(since mingw already defines it)

Actually, it's an error to include that file. It's only needed with
older versions of MSVC that don't define _ftelli64. Specifically,
it's needed by the MSVC 6 used to compile ActivePerl8xx.

At this point, I think it's best to drop support for Windows compilers
that don't provide _ftelli64. MSVC is on version 9 now.

r3847 zaps those files.

> I also had to #define DIR_SEP "/"
> since apparently mingw doesnt define it.
> (but there are mnor changes)

DIR_SEP is defined by Charmonizer (a configuration/probing tool). I'm
a little surprised that Charmonizer didn't successfully define DIR_SEP
as anything. The code that does the probing is in trunk/charmonizer/
src/Charmonizer/Probe/DirSep.charm (which is basically C code despite
the suffix). I guess none of the combos succeeded.

> I checked the mingw headers, and there's no sleep function in them,
> though
> they do tell that one should call the win32 sleep or _sleep, which I
> changed
> and worked for the best as it stopped the sleep error.

OK, that code was in there, but it wasn't reached because I assumed
that if unistd.h was available we were all good.

In commit r3846, the order of the #ifdef tests has been reversed so
that "windows.h" is preferred.

Try that.

> Sadly, for this one, I couldnt find an equivalent to kill.
> So I tried making it always return true(and not call kill) which
> did indeed let it finish compiling completly.

The #ifdef test order in ProcessID.c has also been reversed in r3846.
The Windows code doesn't use kill(), it uses OpenProcess(),
GetCurrentProcessId() and other yum yums.

> However it seems to be crashing on many tests.
> (according to the debugger, it crashes in kinosearch's dll, but I
> cant figure where since I dont have debug information)


Yeah, that's probably the same problem that's plaguing my MSVC build. :
( The common thread is that all of the failing tests call $invindexer-
>finish(). I'll take another hack at it.

Too bad there's no Valgrind for Windows -- that would make solving
this one much easier.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

> On Sep 8, 2008, at 7:51 PM, Paul-Kenji Cahier wrote:

>> Strawberry perl actually uses mingw to provide as much unix
>> compatibility as possible, so it does go easier than activeperl.

> Yeah. I use MSVC because it's *more* difficult. If code satisfies
> both MSVC and GCC, we're in pretty good shape as far as portability.

>>> MatchPostingScorer*
>>> -MatchPostingScorer_init(MatchPostingScorer *self, Similarity *sim,
>>> +MatchPostScorer_init(MatchPostingScorer *self, Similarity *sim,
>>> PostingList *plist, Compiler *compiler)
>> Thanks. That fixed the error.

> Groovy.

>> I had to actually comment out:
>> typedef __int64 off64_t;
>> in compat/windows/lfs.h(since mingw already defines it)

> Actually, it's an error to include that file. It's only needed with
> older versions of MSVC that don't define _ftelli64. Specifically,
> it's needed by the MSVC 6 used to compile ActivePerl8xx.

> At this point, I think it's best to drop support for Windows compilers
> that don't provide _ftelli64. MSVC is on version 9 now.

> r3847 zaps those files.

>> I also had to #define DIR_SEP "/"
>> since apparently mingw doesnt define it.
>> (but there are mnor changes)

> DIR_SEP is defined by Charmonizer (a configuration/probing tool). I'm
> a little surprised that Charmonizer didn't successfully define DIR_SEP
> as anything. The code that does the probing is in trunk/charmonizer/
> src/Charmonizer/Probe/DirSep.charm (which is basically C code despite
> the suffix). I guess none of the combos succeeded.

>> I checked the mingw headers, and there's no sleep function in them,
>> though
>> they do tell that one should call the win32 sleep or _sleep, which I
>> changed
>> and worked for the best as it stopped the sleep error.

> OK, that code was in there, but it wasn't reached because I assumed
> that if unistd.h was available we were all good.

> In commit r3846, the order of the #ifdef tests has been reversed so
> that "windows.h" is preferred.

> Try that.

>> Sadly, for this one, I couldnt find an equivalent to kill.
>> So I tried making it always return true(and not call kill) which
>> did indeed let it finish compiling completly.

> The #ifdef test order in ProcessID.c has also been reversed in r3846.
> The Windows code doesn't use kill(), it uses OpenProcess(),
> GetCurrentProcessId() and other yum yums.

With all those fixes it now builds perfectly.
The tests still fail (of course).

>> However it seems to be crashing on many tests.
>> (according to the debugger, it crashes in kinosearch's dll, but I
>> cant figure where since I dont have debug information)


> Yeah, that's probably the same problem that's plaguing my MSVC build. :
> ( The common thread is that all of the failing tests call $invindexer-
>>finish(). I'll take another hack at it.

> Too bad there's no Valgrind for Windows -- that would make solving
> this one much easier.

Wouldn't a debug build allow to trace easily where it crashes?
(I assume you already did that but that the crashing is harder
to figure out)


Another question I had was whether it's possible or not to give
some specific compile options(-g for example with gcc?) without
resorting to direct dirty editing.

If you need any stacktraces I could provide them if everything
is built with debug information.



> Marvin Humphrey
> Rectangular Research
> http://www.rectangular.com/


> _______________________________________________
> KinoSearch mailing list
> KinoSearch@rectangular.com
> http://www.rectangular.com/mailman/listinfo/kinosearch


_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch

On Sep 9, 2008, at 4:27 AM, Paul-Kenji Cahier wrote:

> With all those fixes it now builds perfectly.
> The tests still fail (of course).

Please try revision r3857. I can now get the test suite to pass on my
box using both Strawberry Perl and my custom-compiled MSVC perl.
(MacBook Pro running Windows Vista via VMWare Fusion).

> Another question I had was whether it's possible or not to give
> some specific compile options(-g for example with gcc?) without
> resorting to direct dirty editing.

I looked into it a bit. Such flags seem to exist. However, I don't
have a real interest in learning MSVC in-depth, and there aren't
likely to be that many memory problems left that affect only Windows
after the reasonably thorough Valgrind testing KS gets on a regular
basis.

It turned out that there were two memory-management problems. A const
object wasn't really const, which GCC tolerated but MSVC freaked out
over. And "free" turns out not to mean the system "free" when you
#include the Perl headers -- it's overridden. That caused the
mysterious "free to wrong pool" errors for objects which were
allocated in the core KS module code, but whose destroy methods
happened to be implemented in the bindings because they contain Perl
data structures (Doc and Tokenizer).

Both of those problems were fairly easy to isolate using debug prints
in a binary search pattern.

After that, there was a bunch of file handling stuff and miscellaneous
flotsam to clean up.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


_______________________________________________
KinoSearch mailing list
KinoSearch@rectangular.com
http://www.rectangular.com/mailman/listinfo/kinosearch