Mailing List Archive

Skip Montanaro wrote:
>
> I've spent amount as much time on generating automated statement and branch
> coverage as I can afford for the next few days. At this point I think I
> have it generating a reasonable facsimile of useful output.

I may have missed email about this during my email troubles yesterday,
but why aren't we doing coverage testing of Python code too? Or am I
missing that link?

--
Paul Prescod - Not encumbered by corporate consensus
The calculus and the rich body of mathematical analysis to which it
gave rise made modern science possible, but it was the algorithm that
made the modern world possible.
- The Advent of the Algorithm (pending), by David Berlinski

paul wrote:


> Skip Montanaro wrote:
> >
> > I've spent amount as much time on generating automated statement and branch
> > coverage as I can afford for the next few days. At this point I think I
> > have it generating a reasonable facsimile of useful output.
>
> I may have missed email about this during my email troubles yesterday,
> but why aren't we doing coverage testing of Python code too? Or am I
> missing that link?

from Skip's original mail:

/.../ I would also like to
adjust the regrtest.py stuff to generate statement coverage for .py files
using my trace module: http://www.musi-cal.com/~skip/python/trace.py.

</F>

>> I've spent amount as much time on generating automated statement and
>> branch coverage as I can afford for the next few days. At this point
>> I think I have it generating a reasonable facsimile of useful output.

Paul> I may have missed email about this during my email troubles
Paul> yesterday, but why aren't we doing coverage testing of Python code
Paul> too? Or am I missing that link?

Perhap you did miss it. That's next on my list. I have a trace module that
will do line counting, but I haven't worked on this thing long enough to get
to it.

If anyone's game, take a look at

http://www.musi-cal.com/~skip/python/trace.py

Shouldn't take an awful lot of effort. I just haven't got the time today.

--
Skip Montanaro, skip@mojam.com, http://www.mojam.com/, http://www.musi-cal.com/
On Phil Jackson's ability to manage multiple Lakers superstars, Shaquille
O'Neal said: "He's got the rings. I listen to the man with the rings."

Skip Montanaro wrote:
>
> ...
> Perhap you did miss it. That's next on my list. I have a trace module that
> will do line counting, but I haven't worked on this thing long enough to get
> to it.
>
> If anyone's game, take a look at
>
> http://www.musi-cal.com/~skip/python/trace.py

Thanks. It's because I've recently been using your trace module that I
originally presumed that the coverage testing would be Python coverage
testing. I don't yet completely understand the counting mechanism that
the tool uses (some numbers look fishy) but overall it is a great boon
for testing.

--
Paul Prescod - Not encumbered by corporate consensus
The calculus and the rich body of mathematical analysis to which it
gave rise made modern science possible, but it was the algorithm that
made the modern world possible.
- The Advent of the Algorithm (pending), by David Berlinski

Fredrik Lundh wrote:
>
> from Skip's original mail:
>
> /.../ I would also like to
> adjust the regrtest.py stuff to generate statement coverage for .py files
> using my trace module: http://www.musi-cal.com/~skip/python/trace.py.

Thanks. If anyone sent me private mail yesterday (perhaps Fred about
xml.*) they should send it again. My ISP's story is that their ISP went
belly up without any warning. Grrr.

--
Paul Prescod - Not encumbered by corporate consensus
The calculus and the rich body of mathematical analysis to which it
gave rise made modern science possible, but it was the algorithm that
made the modern world possible.
- The Advent of the Algorithm (pending), by David Berlinski

> I've spent amount as much time on generating automated statement
> and branch
> coverage as I can afford for the next few days. At this point I think I
> have it generating a reasonable facsimile of useful output.

This is very cool.

It would be great if there was some sort of "multi-platform" support, and a
merging of the results.

My idea here is so modules such as "mmapmodule" can give a better
indication of the true test coverage. Your data shows this test at less
than 50% - probably mainly due to the #ifdef MS_WIN32 blocks. If we merged
a coverage of a Windows run against this same file, we would have an
excellent basis for ensuring that the Python test suite did indeed hit as
much as we could make it.

This is longer term of course.

Along the same vein is the Mozilla "tinderbox". It appears they have a job
that starts a complete rebuild and test suite run every time a checkin is
made to the CVS tree. A Web page shows the progress of each of the
platforms (including some nice flaming images when the build or test suite
is broken.) http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey

Mark.

[Paul Prescod]
> ...If anyone sent me private mail yesterday (perhaps Fred about
> xml.*) they should send it again. My ISP's story is that their ISP went
> belly up without any warning. Grrr.

Ditto! My POP3 server at MSN crapped out a little after midnight, and I
appear to have lost all mail sent from then until about 2pm. The pipermail
archives on python.org stopped archiving around midnight too. My email
.forward from tpeters@beopen.com also seems still to be hosed. Was July 1
some "32-bit day" that slipped thru paranoia's cracks <0.9 wink>?

Mark> It would be great if there was some sort of "multi-platform"
Mark> support, and a merging of the results.

Agreed. We could set up a scheme where people run my scripts on other
systems and then mail me the generated files. Windows would be kind of
tough though, because my stuff currently relies on gcc/gcov. What
facilities does Windows have for coverage testing Mark?

Mark> Along the same vein is the Mozilla "tinderbox". It appears they
Mark> have a job that starts a complete rebuild and test suite run every
Mark> time a checkin is made to the CVS tree.

That seems a bit extreme to me, and I don't know about you, but I don't have
the spare resources available to "burn" on such a project.

skip

[Skip]

> Mark> It would be great if there was some sort of "multi-platform"
> Mark> support, and a merging of the results.

> Agreed. We could set up a scheme where people run my scripts on other
> systems and then mail me the generated files. Windows would be kind of
> tough though, because my stuff currently relies on gcc/gcov. What
> facilities does Windows have for coverage testing Mark?

There is a source-profiler, but Im not sure how deep it lets you into the
data. I'll look into this at some stage.

Or some people are working towards gcc makefiles for windows - maybe that
is a better way to go for this?


...

> That seems a bit extreme to me, and I don't know about you, but
> I don't have
> the spare resources available to "burn" on such a project.

I think that the tinderbox code is open-source too! I agree this is a bit
extreme, but I do believe something like it would be useful, and need not
be too hard. As more and more checkins happen from different people, the
chances of platform-breakage would seem to be quite high.

A Python script that periodically scanned CVS for changes, then executed a
build and did something pretty simple with the log wouldnt be too hard.

However, I do agree we should wait until it is _nearly_ a problem (as
opposed to _is_ a problem :-)

Mark.