Mailing List Archive: Unittests for 2.7 failing in a unpredictable way

Unittests for 2.7 failing in a unpredictable way

Sep 6, 2004, 12:19 PM

Post #1 of 5 (2727 views)

I have been running the 2.7 unittests today multiple times. Sometimes they
pass, sometimes they fail
when running unchanged inside the same sandbox (Python 2.3.3, Linux):

======================================================================
ERROR: checkConcurrentLargeUpdates
(ZEO.tests.testConnection.FileStorageConnectionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZEO/tests/InvalidationTests.py",
line 552, in checkConcurr
ntLargeUpdates
get_transaction().commit()
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/Transaction.py",
line 241, in commit
ncommitted += self._commit_objects(objects)
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/Transaction.py",
line 356, in _commit_objects
jar.commit(o, self)
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/Connection.py",
line 348, in commit
raise ReadConflictError(object=object)
ReadConflictError: database read conflict error (oid 0x0f, class
BTrees.OOBTree.OOBucket)

======================================================================
FAIL: checkConcurrentUpdates2StoragesMT
(ZEO.tests.testConnection.FileStorageConnectionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZEO/tests/InvalidationTests.py",
line 476, in checkConcurr
ntUpdates2StoragesMT
self._check_threads(tree, t1, t2, t3)
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZEO/tests/InvalidationTests.py",
line 373, in _check_threa
s
self.fail('\n'.join(errormsgs))
File "/opt/python-2.3.4/lib/python2.3/unittest.py", line 270, in fail
raise self.failureException, msg
AssertionError: expected keys != actual keys
key 263 expected but not in tree

======================================================================
FAIL: checkSecondBeginFails (ZEO.tests.testZEO.MappingStorageTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZEO/tests/ThreadTests.py",
line 110, in checkSecondBeginFa
ls
thread2.join()
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZEO/tests/ThreadTests.py",
line 39, in join
assert not self.isAlive()
AssertionError

======================================================================
FAIL: check7ZODBThreads (ZODB.tests.testFileStorage.FileStorageTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/tests/MTStorage.py",
line 217, in check7ZODBThreads
self._checkNThreads(7, ZODBClientThread, db, self)
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/tests/MTStorage.py",
line 206, in _checkNThreads
t.join(60)
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/tests/MTStorage.py",
line 41, in run
self.runtest()
File
"/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/tests/MTStorage.py",
line 70, in runtest
self.test.fail()
File "/opt/python-2.3.4/lib/python2.3/unittest.py", line 270, in fail
raise self.failureException, msg
AssertionError

Any ideas what might go wrong?

Andreas

_______________________________________________
Zope-Coders mailing list
Zope-Coders@zope.org
http://mail.zope.org/mailman/listinfo/zope-coders

Re: Unittests for 2.7 failing in a unpredictable way [ In reply to ]

tim.peters at gmail

Sep 6, 2004, 3:49 PM

Post #2 of 5 (2656 views)

Permalink

[Andreas Jung]
> I have been running the 2.7 unittests today multiple times. Sometimes they
> pass, sometimes they fail when running unchanged inside the same sandbox
> (Python 2.3.3, Linux):

...
> ERROR: checkConcurrentLargeUpdates (ZEO.tests.testConnection.FileStorageConnectionTests)
...
> "/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/Connection.py", line 348, in commit
> raise ReadConflictError(object=object)
> ReadConflictError: database read conflict error (oid 0x0f, class
> BTrees.OOBTree.OOBucket)

> FAIL: checkConcurrentUpdates2StoragesMT
(ZEO.tests.testConnection.FileStorageConnectionTests)
...
> AssertionError: expected keys != actual keys
> key 263 expected but not in tree

> FAIL: checkSecondBeginFails (ZEO.tests.testZEO.MappingStorageTests)
...
> "/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZEO/tests/ThreadTests.py",
> line 39, in join
> assert not self.isAlive()
> AssertionError

> FAIL: check7ZODBThreads (ZODB.tests.testFileStorage.FileStorageTests)
...
> "/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/tests/MTStorage.py",
> line 70, in runtest
> self.test.fail()
> File "/opt/python-2.3.4/lib/python2.3/unittest.py", line 270, in fail
> raise self.failureException, msg
> AssertionError

> Any ideas what might go wrong?

Multi-thread and multi-process tests (ZEO tests in general spawn
multiple processes) are difficult to make work all the time, and
several ZEO tests in particular have always failed if you run them
"often enough", especially when run on a not-otherwise-quiet machine.
For example, since there's no direct way for one process to peer
inside another, the tests too often rely on calling time.sleep(N) in
process A, desperately hoping N is long enough for process B to have
accomplished something necessary for the test to succeed.

The bottom lines:

- It's very time-intensive to truly fix one of these, usually requiring a much
more difficult approach than the probabilistic "sleep and hope";
- I'm the only one who ever tries to fix one of these anymore; and
- Fixing one of these is rarely a top priority anymore.

So that's not good, but I rarely have time to make it better. These
tests are in much better shape than they were in a year ago, and a
year ago were in much better shape than two years ago, and they're in
better shape today in ZODB 3.3 than in 3.2. But they're still not in
good shape, and there's no relief on the immediate horizon.
_______________________________________________
Zope-Coders mailing list
Zope-Coders@zope.org
http://mail.zope.org/mailman/listinfo/zope-coders

Re: Unittests for 2.7 failing in a unpredictable way [ In reply to ]

lists at andreas-jung

Sep 6, 2004, 8:48 PM

Post #3 of 5 (2644 views)

Permalink

--On Montag, 6. September 2004 18:49 Uhr -0400 Tim Peters
<tim.peters@gmail.com> wrote:

> [Andreas Jung]
>> I have been running the 2.7 unittests today multiple times. Sometimes
>> they pass, sometimes they fail when running unchanged inside the same
>> sandbox (Python 2.3.3, Linux):
>
> ...
>> ERROR: checkConcurrentLargeUpdates
>> (ZEO.tests.testConnection.FileStorageConnectionTests)
> ...
>> "/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/Connection.py",
>> line 348, in commit raise ReadConflictError(object=object)
>> ReadConflictError: database read conflict error (oid 0x0f, class
>> BTrees.OOBTree.OOBucket)
>
>> FAIL: checkConcurrentUpdates2StoragesMT
> (ZEO.tests.testConnection.FileStorageConnectionTests)
> ...
>> AssertionError: expected keys != actual keys
>> key 263 expected but not in tree
>
>> FAIL: checkSecondBeginFails (ZEO.tests.testZEO.MappingStorageTests)
> ...
>> "/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZEO/tests/ThreadTest
>> s.py", line 39, in join
>> assert not self.isAlive()
>> AssertionError
>
>> FAIL: check7ZODBThreads (ZODB.tests.testFileStorage.FileStorageTests)
> ...
>> "/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/tests/MTStorage
>> .py", line 70, in runtest
>> self.test.fail()
>> File "/opt/python-2.3.4/lib/python2.3/unittest.py", line 270, in fail
>> raise self.failureException, msg
>> AssertionError
>
>> Any ideas what might go wrong?
>
> Multi-thread and multi-process tests (ZEO tests in general spawn
> multiple processes) are difficult to make work all the time, and
> several ZEO tests in particular have always failed if you run them
> "often enough", especially when run on a not-otherwise-quiet machine.
> For example, since there's no direct way for one process to peer
> inside another, the tests too often rely on calling time.sleep(N) in
> process A, desperately hoping N is long enough for process B to have
> accomplished something necessary for the test to succeed.
>
> The bottom lines:
>
> - It's very time-intensive to truly fix one of these, usually requiring a
> much more difficult approach than the probabilistic "sleep and hope"; -
> I'm the only one who ever tries to fix one of these anymore; and - Fixing
> one of these is rarely a top priority anymore.
>
> So that's not good, but I rarely have time to make it better. These
> tests are in much better shape than they were in a year ago, and a
> year ago were in much better shape than two years ago, and they're in
> better shape today in ZODB 3.3 than in 3.2. But they're still not in
> good shape, and there's no relief on the immediate horizon.

So your message is that we don't have to worry much about the failures?
I just need to know if this is something that could be ignored or not.

Andreas

_______________________________________________
Zope-Coders mailing list
Zope-Coders@zope.org
http://mail.zope.org/mailman/listinfo/zope-coders

Re: Unittests for 2.7 failing in a unpredictable way [ In reply to ]

tim.peters at gmail

Sep 6, 2004, 9:03 PM

Post #4 of 5 (2654 views)

Permalink

[.Andreas Jung, on sporadic failures in 2.7 ZEO and ZODB thread tests]
> So your message is that we don't have to worry much about the failures?
> I just need to know if this is something that could be ignored or not.

It's part of my job to worry about them, but you're welcome to worry
about them too (although I advise against that <wink>). If one of
these tests fails in isolation on an otherwise-quiet machine, then
it's something I'll worry a lot more about, because that could well be
indicative of a real bug. As is, I've spent literally months of my
life making these tests more reliable, and almost all of that turned
out to be plugging timing holes in the tests themselves. Such efforts
pay, but we don't have enough ZODB manpower in-house anymore to afford
to do that in addition to higher-priority ZODB tasks.

So if you see a repeatable failure on an otherwise-quiet box, I'm all
ears. Else feel free to report it anyway, and I'll it add to my file
(there is one!) of "mysterious, sporadic ZEO test failures". That's
pretty much a write-only file, though.
_______________________________________________
Zope-Coders mailing list
Zope-Coders@zope.org
http://mail.zope.org/mailman/listinfo/zope-coders

Re: Unittests for 2.7 failing in a unpredictable way [ In reply to ]

lists at andreas-jung

Sep 6, 2004, 9:11 PM

Post #5 of 5 (2651 views)

Permalink

--On Dienstag, 7. September 2004 0:03 Uhr -0400 Tim Peters
<tim.peters@gmail.com> wrote:

> [.Andreas Jung, on sporadic failures in 2.7 ZEO and ZODB thread tests]
>> So your message is that we don't have to worry much about the failures?
>> I just need to know if this is something that could be ignored or not.
>
> It's part of my job to worry about them, but you're welcome to worry
> about them too (although I advise against that <wink>). If one of
> these tests fails in isolation on an otherwise-quiet machine, then
> it's something I'll worry a lot more about, because that could well be
> indicative of a real bug. As is, I've spent literally months of my
> life making these tests more reliable, and almost all of that turned
> out to be plugging timing holes in the tests themselves. Such efforts
> pay, but we don't have enough ZODB manpower in-house anymore to afford
> to do that in addition to higher-priority ZODB tasks.
>
> So if you see a repeatable failure on an otherwise-quiet box, I'm all
> ears. Else feel free to report it anyway, and I'll it add to my file
> (there is one!) of "mysterious, sporadic ZEO test failures". That's
> pretty much a write-only file, though.

Just because there is only a very small number of people understanding the
ZODB
in detail (I am not counting myself as a ZODB expert) it is important for
me as release
manager to know if there is a serious problem. But I trust in you and I
will try to ignore
the sporadic failures in the future :-)

Thanks,
Andreas
there

_______________________________________________
Zope-Coders mailing list
Zope-Coders@zope.org
http://mail.zope.org/mailman/listinfo/zope-coders