Mailing List Archive: Varnish code coverage results online

Varnish code coverage results online

Jan 5, 2017, 8:08 AM

Post #1 of 7 (1229 views)

I have spent the day retooling the code-coverage stuff I have been
running manually since day almost-one of Varnish.

The results are now online:

http://varnish-cache.org/gcov

(There's a link in the navbar at the bottom of the project homepage)

I have also, at the same time raised the official project goal to
"90%+ of *all* executable code tested automatically".

Your assistance reaching this goal will be appreciated.

Right now none of the curses(3) code is tested automatically.
Doing so is non-trivial, but doable.

Maybe varnishtest::process should simply (ha!) grow a -pty option ?

Another big chunk of non-tested code is in libvgz, and just
on general principles I might go in there with some lobotomizing
#ifdefs and WRONG()'s, since that is code that we don't want
to ever execute in the first place.

Poul-Henning

PS: Needless to say: The instant we hit the 90% goal I'll raise it again.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Varnish code coverage results online [ In reply to ]

dridi at varni

Jan 5, 2017, 9:31 AM

Post #2 of 7 (1228 views)

Permalink

> I have also, at the same time raised the official project goal to
> "90%+ of *all* executable code tested automatically".

Only line coverage? Nothing for branches?

90% branch coverage is even a bigger challenge ;)

Another thing: if we want proper code coverage reports, we need to
aggregate from different platforms if we hope to cover platform-specific
code. Does lcov(1) work on FreeBSD? I know the "l" stands for Linux
but at least with lcov I _know_ we can merge reports.

The problem would then be: how to collect the reports?

I personally use codecov.io which can receive and aggregate coverage
reports and pin them to commits. There are other similar services, I'm
not trying to promote a particular solution.

Dridi

_______________________________________________
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Varnish code coverage results online [ In reply to ]

phk at phk

Jan 6, 2017, 3:05 AM

Post #3 of 7 (1226 views)

Permalink

--------
In message <CABoVN9Aym1aQj+=7k93eYf1jyZmCq9Ys0KmG3rCTKwTSKXHypg@mail.gmail.com>, Dridi Boukelmoune
writes:

>> I have also, at the same time raised the official project goal to
>> "90%+ of *all* executable code tested automatically".
>
>Only line coverage? Nothing for branches?
>
>90% branch coverage is even a bigger challenge ;)

That's probably impossible, given the number of asserts we have.

>Another thing: if we want proper code coverage reports, [...]

Are my my l33t HTML skills not proper ?!

Seriously: This is good enough to tell us which bits of our
code isn't covered by "make check", which is the important part.

Merging in data from other runs would not be a problem, my scripts
already has support for merging because some sourcefiles drop .gcda
in multiple directories already. We'd just need to set up a client
to run "make check" and submit a condensed version of the raw data.

If you feel like running a Linux (or other) client, let me know and
we can get it set up, I think my current plan is to run this only
once per night.

Poul-Henning

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Varnish code coverage results online [ In reply to ]

dridi at varni

Jan 6, 2017, 3:21 AM

Post #4 of 7 (1226 views)

Permalink

On Fri, Jan 6, 2017 at 12:05 PM, Poul-Henning Kamp <phk@phk.freebsd.dk> wrote:
> --------
> In message <CABoVN9Aym1aQj+=7k93eYf1jyZmCq9Ys0KmG3rCTKwTSKXHypg@mail.gmail.com>, Dridi Boukelmoune
> writes:
>
>>> I have also, at the same time raised the official project goal to
>>> "90%+ of *all* executable code tested automatically".
>>
>>Only line coverage? Nothing for branches?
>>
>>90% branch coverage is even a bigger challenge ;)
>
> That's probably impossible, given the number of asserts we have.

Yes, my approach is to run the test suite with assertions, and disable
them for coverage.

>>Another thing: if we want proper code coverage reports, [...]
>
> Are my my l33t HTML skills not proper ?!
>
> Seriously: This is good enough to tell us which bits of our
> code isn't covered by "make check", which is the important part.
>
> Merging in data from other runs would not be a problem, my scripts
> already has support for merging because some sourcefiles drop .gcda

Cool, I had no idea!

> in multiple directories already. We'd just need to set up a client
> to run "make check" and submit a condensed version of the raw data.

And here comes tools/vcov.sh :p

> If you feel like running a Linux (or other) client, let me know and
> we can get it set up, I think my current plan is to run this only
> once per night.

If we come up with something, Travis CI could do that for us for each
push (so some work needs to be done to match the nightly scheduling).
We'd just need to set up a new job, and I think it would even run as
root and skip less tests.

Dridi

_______________________________________________
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Varnish code coverage results online [ In reply to ]

phk at phk

Jan 6, 2017, 3:36 AM

Post #5 of 7 (1226 views)

Permalink

--------
In message <CABoVN9BUXbWO3mN34HXDzN_PdL9v757N=PQN309ab++KNHJLvg@mail.gmail.com>, Dridi Boukelmoune
writes:
>On Fri, Jan 6, 2017 at 12:05 PM, Poul-Henning Kamp <phk@phk.freebsd.dk> wrote:

>> If you feel like running a Linux (or other) client, let me know and
>> we can get it set up, I think my current plan is to run this only
>> once per night.
>
>If we come up with something, Travis CI could do that for us for each
>push (so some work needs to be done to match the nightly scheduling).

Doing it on each push would be silly.

In particular because it takes one and a half error to run "make
check" with gcov: Gcov isn't parallel-safe, so all the tests has
to run serially, and the implementation sucks, so it takes forever
to update the .g??? files.

I'll look at it, I hadn't considered multiplatform when I did this.

The simplest is probably to (ab)use vtest to do this.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Varnish code coverage results online [ In reply to ]

dridi at varni

Jan 6, 2017, 4:47 AM

Post #6 of 7 (1226 views)

Permalink

On Fri, Jan 6, 2017 at 12:36 PM, Poul-Henning Kamp <phk@phk.freebsd.dk> wrote:
> --------
> In message <CABoVN9BUXbWO3mN34HXDzN_PdL9v757N=PQN309ab++KNHJLvg@mail.gmail.com>, Dridi Boukelmoune
> writes:
>>On Fri, Jan 6, 2017 at 12:05 PM, Poul-Henning Kamp <phk@phk.freebsd.dk> wrote:
>
>>> If you feel like running a Linux (or other) client, let me know and
>>> we can get it set up, I think my current plan is to run this only
>>> once per night.
>>
>>If we come up with something, Travis CI could do that for us for each
>>push (so some work needs to be done to match the nightly scheduling).
>
> Doing it on each push would be silly.

We may be able to run the last build "with coverage enabled" using
some API. I don't know, I'd need to look at the docs, but this way it
could be done from the Varnish infra.

> In particular because it takes one and a half error to run "make
> check" with gcov: Gcov isn't parallel-safe, so all the tests has
> to run serially, and the implementation sucks, so it takes forever
> to update the .g??? files.

I thought gcov'ed binaries could run concurrently if the underlying
file-system allowed it. I never had any issue with gcov and parallel
builds and regardless, if varnishtest spawns a varnishd, won't they
compete on coverage collection for libvarnish for instance?

> I'll look at it, I hadn't considered multiplatform when I did this.
>
> The simplest is probably to (ab)use vtest to do this.

I don't know about simplest, running nightly builds from a script that
runs pseudo-continuously doesn't sound straightforward.

Dridi

_______________________________________________
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Varnish code coverage results online [ In reply to ]

phk at phk

Jan 6, 2017, 5:28 AM

Post #7 of 7 (1226 views)

Permalink

--------
In message <CABoVN9Ad=R+qyAGz1+_XYXbzOvr6krHUYVoBi44PyEGUeNm7ig@mail.gmail.com>, Dridi Boukelmoune
writes:

>I thought gcov'ed binaries could run concurrently if the underlying
>file-system allowed it. I never had any issue with gcov and parallel
>builds and regardless, if varnishtest spawns a varnishd, won't they
>compete on coverage collection for libvarnish for instance?

Last I tried it, I got totally screwed up numbers.

(And no, because varnishtest waits for varnishd to exit before it
exits.)

>I don't know about simplest, running nightly builds from a script that
>runs pseudo-continuously doesn't sound straightforward.

if `date +%Y%m%d` changed do gcov ?

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

_______________________________________________
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev