Mailing List Archive

Chad Netzer wrote:
>
> Compiling python 1.5.2 on Linux Mandrake 6.0, the "make test" step fails
> because it gets an OverflowError during test_types.py on the following line:
>
> int(long(-sys.maxint-1))

This went wrong with 1.5 and 1.5.1, and it should be gone
for 1.5.2 .

Have a look at the (Tim Peters) implementation.

Especially at the end:
if ((long)x < 0 && (sign > 0 || (x << 1) != 0))
goto overflow;
return (long)x * sign;

where x is unsigned long.
This cannot get wrong, unless you have one of three problems:

1) The mandrake compiler is buggy
2) You don't have 1.5.2 but 1.5.1
3) Oliver Andrich made a mistake with the package

Your comparison to 1.5.1 is invalid since that definately had a bug.

If you can deduce that case 1) must be true, then I'd like
to send you a test program which I just wrote, to simulate
the above without Python. I'd like to see the generated
assembly code, then.

If it is 2), then please stand in the corner for costing my time.

For 3), Oliver is guilty, and he deserves so many hugs about
his RPM stuff that this is just something that happens.

ciao - pirx

________________________________
long
PyLong_AsLong(vv)
PyObject *vv;
{
/* This version by Tim Peters */
register PyLongObject *v;
unsigned long x, prev;
int i, sign;

if (vv == NULL || !PyLong_Check(vv)) {
PyErr_BadInternalCall();
return -1;
}
v = (PyLongObject *)vv;
i = v->ob_size;
sign = 1;
x = 0;
if (i < 0) {
sign = -1;
i = -(i);
}
while (--i >= 0) {
prev = x;
x = (x << SHIFT) + v->ob_digit[i];
if ((x >> SHIFT) != prev)
goto overflow;
}
/* Haven't lost any bits, but if the sign bit is set we're in
* trouble *unless* this is the min negative number. So,
* trouble iff sign bit set && (positive || some bit set other
* than the sign bit).
*/
if ((long)x < 0 && (sign > 0 || (x << 1) != 0))
goto overflow;
return (long)x * sign;

overflow:
PyErr_SetString(PyExc_OverflowError,
"long int too long to convert");
return -1;
}
________________________________

--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
we're tired of banana software - shipped green, ripens at home

Hi,

I compiled the packages on my Mandrake 6.0 system and I am looking into this
myself. ;-)) Concerning the release I have to admin that I am a little bit
late, but a lot of work had to be done. But here is a road map for all
interested:

- I asked for a mailing list on starship python and received it. It is called
python4linux and should be used to discuss issues concerning the packages we
make
- I wrote a announcement and request for participation, cause I like to move
the whole project to a group effort (more work, new girl friend, new side
projects -> less time to maintain the thing myself ;-)
- I finished a new spec file that can build RedHat6, Mandrake6 and RedHat5
packages with just one extra parameter during compilation

All this (after a weekend of testing is ready and available on Monday. The
Announcement will be posted tomorrow to c.l.p.a and the packages will be
uploaded on Monday. The web page has to be reworked, but this is left to a
later date.

Best regards,

Oliver

Chad Netzer <chad@vision.arc.nasa.gov> wrote:
> Compiling python 1.5.2 on Linux Mandrake 6.0, the "make test" step fails
> because it gets an OverflowError during test_types.py on the following line:

> int(long(-sys.maxint-1))

> test_long.py fails also for the same reason. I've tried this on another Linux
> setup, using the default system python 1.5.1 compiled with just the straight
> gcc (not egcs or pgcc) and got the same result. Is this just the expected
> behavior on Linux (in which case maybe the tests should be changed to reflect
> this) Can I ignore it? (This doesn't overflow on my SGI, and I assume the
> Pentium uses 2's-complement. Perhaps this is a glibc issue?) I should look
> at the code, but I'm sure others must have come across this...

> BTW, as a learning experience, I'm building all of Oliver Andrich's Python
> packages for the Mandrake 6.0 architecture, and will make the results available
> when I'm finished, unless Oliver beats me to the punch and obviates the need.
> Our application depends on getting python installed easily, so I took the initiative.

> Chad Netzer
> chad@vision.arc.nasa.gov




--
Oliver Andrich, RZ-Online, Schlossstrasse Str. 42, D-56068 Koblenz
Telefon: +49-261-39210-40 Fax: -33 | http://rhein-zeitung.de
| http://andrich.net
GMC d- s++:++ a- C++ UL++++ P-- L+++ E--- W++ N+ w-- O- M--
V- PS++ PE Y+ PGP+ t 5+ X R+ tv+ b++ DI++ D- G e h- r z+

> 1) The mandrake compiler is buggy

That is the problem. Don't be to aggressive with the compiler options and
don't use the build.pl script. The results are just a little bit better, but a
lot of problems are caused by the architecture options. ;-)))

> For 3), Oliver is guilty, and he deserves so many hugs about
> his RPM stuff that this is just something that happens.

Thanks. I guess the problem above is the reason. This time I am not
responsible. ;-)))

Bye, Oliver Andrich

--
Oliver Andrich, RZ-Online, Schlossstrasse Str. 42, D-56068 Koblenz
Telefon: +49-261-39210-40 Fax: -33 | http://rhein-zeitung.de
| http://andrich.net
GMC d- s++:++ a- C++ UL++++ P-- L+++ E--- W++ N+ w-- O- M--
V- PS++ PE Y+ PGP+ t 5+ X R+ tv+ b++ DI++ D- G e h- r z+

[Chad Netzer]
> Compiling python 1.5.2 on Linux Mandrake 6.0, the "make test" step fails
> because it gets an OverflowError during test_types.py on the
> following line:
>
> int(long(-sys.maxint-1))
>
> test_long.py fails also for the same reason.

Chad, let's break this down to see where the overflow is coming from. Try
this line by line and post your results:

>>> import sys
>>> sys.maxint
2147483647
>>> -sys.maxint
-2147483647
>>> -sys.maxint-1
-2147483648
>>> long(-sys.maxint-1)
-2147483648L
>>> int(long(-sys.maxint-1))
-2147483648
>>>

Neither the OS nor the libraries should have anything to do with this. You
didn't mention the important part, though <wink>: what kind of HW is this
running on? If it has a "funny" wordsize, and limits.h is missing or
foolish, Python may have a bogus idea of what the largest int is (which is
why I want to see what sys.maxint returns for you).

boy-that-linux-is-unstable<wink>-ly y'rs - tim

Tim Peters wrote:

> Chad, let's break this down to see where the overflow is coming from. Try
> this line by line and post your results:
>
> >>> import sys
> >>> sys.maxint
> 2147483647
> >>> -sys.maxint
> -2147483647
> >>> -sys.maxint-1
> -2147483648
> >>> long(-sys.maxint-1)
> -2147483648L
> >>> int(long(-sys.maxint-1))
> -2147483648
> >>>
>

>>> import sys
>>> sys.maxint
2147483647
>>> -sys.maxint
-2147483647
>>> -sys.maxint-1
-2147483648
>>> long(-sys.maxint-1)
-2147483648L
>>> int(long(-sys.maxint-1))
Traceback (innermost last):
File "<stdin>", line 1, in ?
OverflowError: long int too long to convert
>>>

This is on a Pentium III system running straight Mandrake 6.0. I think, as Christian
and Oliver pointed out, it is pgcc related. I tried it on a Mandrake 5.3 system (using
the normal gcc, not pgcc, and it worked correctly)

For kicks, I fired up the debugger, and traced through the PyLongASLong call to see
what was happening. The problem is at the line:

if ((long)x < 0 && (sign > 0 || (x << 1) != 0))

Even though it should evaluate to 0 (and appears to from the debugger's point of view),
it incorrectly proceeds to the "goto overflow" statement. I looked at the assembly language
output from a working version and a non-working version (both 1.5.2 on Pentiums), and
found that the working version tends to use "cmpl", "testl", etc. and the non-working version
doesn't. Here is what I mean, the gist of the code is apparent, even though I'm no wizard at
386 asm (I'll provide some comments):

C code:
if ((long)x < 0 && (sign > 0 || (x << 1) != 0))
goto overflow;
return (long)x * sign;


working asm:
<PyLong_AsLong+140>: cmpl $0x0,0xfffffffc(%ebp) ; if x<0
<PyLong_AsLong+144>: jnl 0x806adbc <PyLong_AsLong+172>
<PyLong_AsLong+146>: cmpl $0x0,0xfffffff0(%ebp) ; if sign > 0
<PyLong_AsLong+150>: jg 0x806adb8 <PyLong_AsLong+168>
<PyLong_AsLong+152>: movl 0xfffffffc(%ebp),%esi
<PyLong_AsLong+155>: leal 0x0(,%esi,2),%eax ; x<<0 (?)
<PyLong_AsLong+162>: testl %eax,%eax
<PyLong_AsLong+164>: jne 0x806adb8 <PyLong_AsLong+168>
<PyLong_AsLong+166>: jmp 0x806adbc <PyLong_AsLong+172>
<PyLong_AsLong+168>: jmp 0x806adc8 <PyLong_AsLong+184> ; goto overflow
<PyLong_AsLong+170>: movl %esi,%esi ; return (long)x * sign;
<PyLong_AsLong+172>: movl 0xfffffffc(%ebp),%edx
<PyLong_AsLong+175>: imull 0xfffffff0(%ebp),%edx
<PyLong_AsLong+179>: movl %edx,%eax
<PyLong_AsLong+181>: jmp 0x806ade4 <PyLong_AsLong+212>

not working:
<PyLong_AsLong+150>: cmpl $0x0,0xfffffffc(%ebp)
<PyLong_AsLong+154>: jge 0x806c3f2 <PyLong_AsLong+178>
<PyLong_AsLong+156>: cmpl $0x0,0xfffffff0(%ebp)
<PyLong_AsLong+160>: jg 0x806c3f0 <PyLong_AsLong+176>
<PyLong_AsLong+162>: mov 0xfffffffc(%ebp),%eax
<PyLong_AsLong+165>: add %eax,%eax ; x<<1
<PyLong_AsLong+167>: test %eax,%eax
<PyLong_AsLong+169>: jne 0x806c3f0 <PyLong_AsLong+176>
<PyLong_AsLong+171>: jmp 0x806c3f2 <PyLong_AsLong+178>
<PyLong_AsLong+173>: lea 0x0(%esi),%esi
<PyLong_AsLong+176>: jmp 0x806c400 <PyLong_AsLong+192> ; goto overflow
<PyLong_AsLong+178>: mov 0xfffffffc(%ebp),%edx ; return (long)x * sign;
<PyLong_AsLong+181>: imul 0xfffffff0(%ebp),%edx
<PyLong_AsLong+185>: mov %edx,%eax
<PyLong_AsLong+187>: jmp 0x806c41a <PyLong_AsLong+218>

If nyone who knows more about assembly, I would mind hearing where the problem
is, and to know if it is worth submitting a bug-report.

It sounds as if Oliver's very-soon-to-be-released versions of the packages may have
fixed the problem w/ compiler flags. I'm anxious to try it out (Oliver, thanks for all
the effort, BTW. I got impatient waiting for your new releases, and have been making
my own based on you specs and packages... LOT of work getting everything just right!
I figured you were busy, as we all are, but I'm happy to see that you'll be releasing soon.
I will definately join the python4linux mailing list, as I would like to help out in the
future)

Thanks for the help everyone.

Chad Netzer
chad@vision.arc.nasa.gov

Here I go, replying to my own posts again...

Chad Netzer wrote:

> C code:
> if ((long)x < 0 && (sign > 0 || (x << 1) != 0))
> goto overflow;
> return (long)x * sign;
>
> not working:

deleted... It was actually working code, assembled with pgcc WITHOUT any optimization.
The bug I've describing only manifests itself when -O is selected (or even -O1 or -O2).
So I apologize if anyone looked at the code and couldn't find a problem; it didn't exist in
that version.

Here is some assembly of the same region, compiled with the -O1 flag, which DOES
exhibit the incorrect behavior. Notice that it is quite different from the previous code,
and I think the bug is that it uses the "LEA" instruction improperly:

<PyLong_AsLong+89>: test %edx,%edx ; x < 0
<PyLong_AsLong+91>: jge 0x806c22a <PyLong_AsLong+106>
<PyLong_AsLong+93>: test %ebx,%ebx ; sign > 0
<PyLong_AsLong+95>: jg 0x806c231 <PyLong_AsLong+113> ; goto overflow
<PyLong_AsLong+97>: lea 0x0(,%edx,2),%eax ; ?? Where is (x << 1)?
<PyLong_AsLong+104>: jne 0x806c231 <PyLong_AsLong+113> ; goto overflow
<PyLong_AsLong+106>: mov %edx,%eax
<PyLong_AsLong+108>: imul %ebx,%eax
<PyLong_AsLong+111>: jmp 0x806c246 <PyLong_AsLong+134>

As you can see, the original code had add %eax, %eax instruction to shift
left by one, the optimized one doesn't. I don't know what the heck it
was trying to do, and I don't know what condition codes "lea" sets, so
I can't know what it intended. In any case, I speculate that line +104
was causing a jump to the overflow handler when it should not.

Anyway, I apologize for cluttering the newsgroup with assembly; clearly the
problem is with pgcc w/ optimization turned on, which is a Mandrake 6.0 issue.

The lesson is NOT to trust pgcc with optimizations, and when compiling
Python 1.5.2, to turn them off (ie. make OPT=-g), at least when compiling
the longobject.c file.

Thanks again for all the help, it was at least a mildly interesting exercise to
track it down. Sorry for the not double-checking the results of my last compilation
without optimizations.

Chad Netzer
chad@vision.arc.nasa.gov

> This is on a Pentium III system running straight Mandrake 6.0. I think, as Christian
> and Oliver pointed out, it is pgcc related. I tried it on a Mandrake 5.3 system (using
> the normal gcc, not pgcc, and it worked correctly)

Yep, that is exactly what happens when you use pgcc. I moved to pgcc some time
ago and caused a lot of problems. The distribution itself worked fine, but as
soon as someone updated a package and recompiled it with egcs or gcc. That
module was not importable by the distribution I created. I moved back to egcs
without this pentium/pentiumII stuff, and everything worked. And if this is
the case again, I have to look into this more then I like to. ;-)))

I reported the ASM code to a (P)GCC/EGCS professional or poweruser and he is
looking into the issue right now. I believes that it is either pgcc or pgcc in
combination with glibc2.1 or so. But I hopefully get some info until the
weekend. I like to have this issue fixed before releasing any new distribution
release. ;-))

> It sounds as if Oliver's very-soon-to-be-released versions of the packages may have
> fixed the problem w/ compiler flags. I'm anxious to try it out (Oliver, thanks for all
> the effort, BTW. I got impatient waiting for your new releases, and have been making
> my own based on you specs and packages... LOT of work getting everything just right!
> I figured you were busy, as we all are, but I'm happy to see that you'll be releasing soon.
> I will definately join the python4linux mailing list, as I would like to help out in the
> future)

That is fine and promising. Do you mind to send me your spec file or SRPM? May
be we can integrate the two files into one and get an even better solution for
some tasks. I am good adt rpming but I am by no means godlike in this field.

Bye, Oliver


--
Oliver Andrich, RZ-Online, Schlossstrasse Str. 42, D-56068 Koblenz
Telefon: +49-261-39210-40 Fax: -33 | http://rhein-zeitung.de
| http://andrich.net
GMC d- s++:++ a- C++ UL++++ P-- L+++ E--- W++ N+ w-- O- M--
V- PS++ PE Y+ PGP+ t 5+ X R+ tv+ b++ DI++ D- G e h- r z+

[Chad Netzer]
> ...
> Here is some assembly of the same region, compiled with the -O1
> flag, which DOES exhibit the incorrect behavior. Notice that it is
> quite different from the previous code,

Yes, but despite being wrong, it's much faster <wink>.

> ... and I don't know what condition codes "lea" sets,

That's the problem: lea doesn't change any flags (it's the "load effective
address" instruction, and when used for its *intended* purpose (loading
effective addresses <wink>), it would get in the way if it did alter any
flags).

So it's as if "x << 1" had never been computed (which the lea does, btw --
it's tricky!) or tested against 0. They (mis)optimized a necessary "test"
away; see the first version of the working code you posted, which does a
very similar thing in this section (but has the crucial "test" after the
lea).

> clearly the problem is with pgcc w/ optimization turned on, which is a
> Mandrake 6.0 issue.

Yup! You should report it to "them", of course -- such things rarely fix
themselves ...

people-also-introduce-bugs-when-optimizing-so-view-it-as-an-example-
of-advanced-humanistic-ai<wink>-ly y'rs - tim

Chad Netzer wrote:
>
> Here I go, replying to my own posts again...
...
> Here is some assembly of the same region, compiled with the -O1 flag, which DOES
> exhibit the incorrect behavior. Notice that it is quite different from the previous code,
> and I think the bug is that it uses the "LEA" instruction improperly:
>
> <PyLong_AsLong+89>: test %edx,%edx ; x < 0
> <PyLong_AsLong+91>: jge 0x806c22a <PyLong_AsLong+106>
> <PyLong_AsLong+93>: test %ebx,%ebx ; sign > 0
> <PyLong_AsLong+95>: jg 0x806c231 <PyLong_AsLong+113> ; goto overflow
> <PyLong_AsLong+97>: lea 0x0(,%edx,2),%eax ; ?? Where is (x << 1)?

lea doesn't change any flags. It optimizes the shift to an add
which is fine. The implied test of the result is just not done
and this is a false optimization.

Somehing similar happened to me when I hand-optimized assembly
code. It is tempting to replace
add ax, 1
by
inc ax

but beware, that doesn't set flags, and it's likely that you forget
about it when changing tests.

> <PyLong_AsLong+104>: jne 0x806c231 <PyLong_AsLong+113> ; goto overflow
> <PyLong_AsLong+106>: mov %edx,%eax
> <PyLong_AsLong+108>: imul %ebx,%eax
> <PyLong_AsLong+111>: jmp 0x806c246 <PyLong_AsLong+134>
>
> As you can see, the original code had add %eax, %eax instruction to shift
> left by one, the optimized one doesn't. I don't know what the heck it
> was trying to do, and I don't know what condition codes "lea" sets, so
> I can't know what it intended. In any case, I speculate that line +104
> was causing a jump to the overflow handler when it should not.
>
> Anyway, I apologize for cluttering the newsgroup with assembly; clearly the
> problem is with pgcc w/ optimization turned on, which is a Mandrake 6.0 issue.

I can't see any bad about assembly. This is just another data type,
and Python can be a fine tool as a generic,
table driven disassembler ;-)

but that's another project which I will continue next year - chris

--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
we're tired of banana software - shipped green, ripens at home

Tim Peters wrote:

> [Chad Netzer]
> > ...
> > Here is some assembly of the same region, compiled with the -O1
> > flag, which DOES exhibit the incorrect behavior. Notice that it is
> > quite different from the previous code,
>
> Yes, but despite being wrong, it's much faster <wink>.

Hey that's what I thought too. :).

>
> So it's as if "x << 1" had never been computed (which the lea does, btw --
> it's tricky!) or tested against 0.

Yeah, I noticed that the test seemed to be missing, and that lea probably shouldn't
set any condition codes (and as you point out, it would be very bad if it did). I
am curious as to how the x<<1 is computed be lea (I don't understand the
0x0(,%edx,2),%eax notation...


> Yup! You should report it to "them", of course -- such things rarely fix
> themselves ...

Naturally; Since Oliver seems to have been able to pass it on to some PGCC personnel
for comment, I'll see that the Mandrake persons know about it (at the very least, I'll
post it to their support lists for others trying to use Python 1.5.2 w/ Mandrake 6)

Chad

[Chad Netzer]
> ...
> Yeah, I noticed that the test seemed to be missing, and that lea
> probably shouldn't set any condition codes (and as you point out,
> it would be very bad if it did). I am curious as to how the x<<1
> is computed be lea (I don't understand the
> 0x0(,%edx,2),%eax notation...

Well, the notation is awful, but about the best you can expect from a
brainless disassembler (the CISC Intel instructions have gazillions (give or
take a bundle) of variations, and a disassembler generally pumps out the
first template that matches the bits it's looking at -- regardless of
whether a *human* would write it that way).

Anyway, the lea is simply using a scaled-index addressing mode, with a 0
base, a 0 offset, and with %edx treated as a "scale by 2" (shift left by 1)
scaled index.

Check out this link:

http://www.comp.it.bton.ac.uk/burks/burks/language/asm/artofasm/artof001.htm

Arguably the best book on assembler programming ever written! The chapters
on x86 addressing modes and LEA will chase the dark clouds away.

to-be-replaced-by-deep-depression-but-them's-the-breaks-ly y'rs - tim

timbot> Check out this link:

timbot> http://www.comp.it.bton.ac.uk/burks/burks/language/asm/artofasm/artof001.htm

timbot> Arguably the best book on assembler programming ever written!
timbot> The chapters on x86 addressing modes and LEA will chase the dark
timbot> clouds away.

Does the timbot have any other interesting URLs squirrelled away in its
large backside cache?

;-)

Skip

[Tim]
> Check out this link:
> [xxx]
> Arguably the best book on assembler programming ever written!
> The chapters on x86 addressing modes and LEA will chase the dark
> clouds away.

[Skip Montanaro]
> Does the timbot have any other interesting URLs squirrelled away in its
> large backside cache?

Yes: http://www.python.org.

Other than that, I don't save URLs -- links break too often for a bot to
waste cycles updating. Search engines are humans' friends too! Although
they're fickle. A better link for that book is the author's own server, at

http://webster.cs.ucr.edu/

or-at-least-likely-there-for-the-next-10-minutes<0.9-wink>-ly y'rs - tim

Tim Peters wrote:

> Anyway, the lea is simply using a scaled-index addressing mode, with a 0
> base, a 0 offset, and with %edx treated as a "scale by 2" (shift left by 1)
> scaled index.
>
> Check out this link:
>
> http://www.comp.it.bton.ac.uk/burks/burks/language/asm/artofasm/artof001.htm
>

Thanks for the reference, Tim. I read it for ~20 minutes, and was enlightened. :-)

Cheers,
Chad