Mailing List Archive

Hi there,

On Mon, 27 Feb 2023, demerphq wrote:

> ... Odds are *very* likely we get CPAN breakage from the very
> moment we deprecate something ...
> ...
> It would be nice if people in the community would help out with this and
> contribute patches and bug reports, etc, for these issues and take on some
> of the "blead breaks CPAN" burden of these changes.

I'm the maintainer of exactly one CPAN module, but since it's used by
both Sendmail and Postfix users I'm very keen to keep it working. It
would help me greatly in this effort if for example a row or two could
be added to this table:

http://matrix.cpantesters.org/?dist=Sendmail%3A%3APMilter

to show development versions of Perl, or maybe create another table at
another URI if it wouldn't be desirable to extend the existing table -
it might break existing automated tooling for example.

I have no clue what might be involved.

Observations?

--

73,
Ged.

On Mon, 27 Feb 2023 at 10:56, G.W. Haywood via perl5-porters <
perl5-porters@perl.org> wrote:

>
> I'm the maintainer of exactly one CPAN module, but since it's used by
> both Sendmail and Postfix users I'm very keen to keep it working. It
> would help me greatly in this effort if for example a row or two could
> be added to this table:
>
> http://matrix.cpantesters.org/?dist=Sendmail%3A%3APMilter
>
> to show development versions of Perl, or maybe create another table at
> another URI if it wouldn't be desirable to extend the existing table -
> it might break existing automated tooling for example.
>
> I have no clue what might be involved.
>
> Observations?
>

The top seven rows on that page already show the most recent development
versions of Perl. So I'm not really sure what more information you're
asking for.

Dave...

Hi there,

On Mon, 27 Feb 2023, Dave Cross wrote:

> On Mon, 27 Feb 2023 at 10:56, G.W. Haywood via perl5-porters <
> perl5-porters@perl.org> wrote:
>
>>
>> I'm the maintainer of exactly one CPAN module, but since it's used by
>> both Sendmail and Postfix users I'm very keen to keep it working. It
>> would help me greatly in this effort if for example a row or two could
>> be added to this table:
>>
>> http://matrix.cpantesters.org/?dist=Sendmail%3A%3APMilter
>>
>> to show development versions of Perl ...
>
> The top seven rows on that page already show the most recent development
> versions of Perl. So I'm not really sure what more information you're
> asking for.

Sorry, maybe I misunderstand. Is one of those what you'd call 'blead'?
If so, I'll crawl back into my hole.

--

73,
Ged.

On Mon, 27 Feb 2023 at 12:19, G.W. Haywood via perl5-porters <
perl5-porters@perl.org> wrote:

> Hi there,
>
> On Mon, 27 Feb 2023, Dave Cross wrote:
>
> > On Mon, 27 Feb 2023 at 10:56, G.W. Haywood via perl5-porters <
> > perl5-porters@perl.org> wrote:
> >
> >>
> >> I'm the maintainer of exactly one CPAN module, but since it's used by
> >> both Sendmail and Postfix users I'm very keen to keep it working. It
> >> would help me greatly in this effort if for example a row or two could
> >> be added to this table:
> >>
> >> http://matrix.cpantesters.org/?dist=Sendmail%3A%3APMilter
> >>
> >> to show development versions of Perl ...
> >
> > The top seven rows on that page already show the most recent development
> > versions of Perl. So I'm not really sure what more information you're
> > asking for.
>
> Sorry, maybe I misunderstand.


I think you did. That tables lists released versions. Such tables would not
list every patch in blead.


> Is one of those what you'd call 'blead'?
>

No. You would need to build perl from a checkout of the repo at
https://github.com/Perl/perl5

However 5.37.9 contains the apostrophe patch. Unfortunately it does not
contain the smartmatch patches.

If your module does not use smartmatch or related features you are good.
That wasn't my point however, my point was it would be helpful if folks
could take the time to build perl against the latest commit in the blead
branch on github and then test it against their favourite modules. If
anything doesn't build, file a patch with the author of the module and or
maybe with us as an issue.

You can do this with 5.37.9 if you wish, but it will only validate the
apostrophe as a package separator deprecation, not the smartmatch
deprecation.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

Hi there,

On Mon, 27 Feb 2023, demerphq wrote:

> On Mon, 27 Feb 2023 at 12:19, G.W. Haywood via perl5-porters wrote:
>> On Mon, 27 Feb 2023, Dave Cross wrote:
>>> On Mon, 27 Feb 2023 at 10:56, G.W. Haywood via perl5-porters wrote:
>>>
>>>>
>>>> I'm the maintainer of exactly one CPAN module, but since it's used by
>>>> both Sendmail and Postfix users I'm very keen to keep it working. It
>>>> would help me greatly in this effort if for example a row or two could
>>>> be added to this table:
>>>>
>>>> http://matrix.cpantesters.org/?dist=Sendmail%3A%3APMilter
>>>>
>>>> to show development versions of Perl ...
>>>
>>> The top seven rows on that page already show the most recent development
>>> versions of Perl. So I'm not really sure what more information you're
>>> asking for.
>>
>> Sorry, maybe I misunderstand.
>
> I think you did. That tables lists released versions. Such tables would not
> list every patch in blead.
>
>> Is one of those what you'd call 'blead'?
>
> No.

In that case I don't I misunderstood - I think you did. But that's
probably because I'm unfamiliar with the terminology and don't know
how to express my ideas so people on this list can understand them.

> You would need to build perl from a checkout of the repo at
> https://github.com/Perl/perl5

My point exactly. And in my case, because of my pathological aversion
to github (and github's aversion to my browser), I simply won't do it.

> That wasn't my point however, my point was it would be helpful if folks
> could take the time to build perl against the latest commit in the blead
> branch on github and then test it against their favourite modules. If
> anything doesn't build, file a patch with the author of the module and or
> maybe with us as an issue.

That's incredibly onerous. Can't you just get the patched version(s)
of Perl into cpan-testers and get everything tested automatically by
the next morning, together with a nice page of idiot lights which show
Where Things Failed?

--

73,
Ged.

On Mon, 27 Feb 2023 at 13:01, G.W. Haywood via perl5-porters <
perl5-porters@perl.org> wrote:

> Hi there,
>
> On Mon, 27 Feb 2023, demerphq wrote:
>
> > On Mon, 27 Feb 2023 at 12:19, G.W. Haywood via perl5-porters wrote:
> >> On Mon, 27 Feb 2023, Dave Cross wrote:
> >>> On Mon, 27 Feb 2023 at 10:56, G.W. Haywood via perl5-porters wrote:
> >>>
> >>>>
> >>>> I'm the maintainer of exactly one CPAN module, but since it's used by
> >>>> both Sendmail and Postfix users I'm very keen to keep it working. It
> >>>> would help me greatly in this effort if for example a row or two could
> >>>> be added to this table:
> >>>>
> >>>> http://matrix.cpantesters.org/?dist=Sendmail%3A%3APMilter
> >>>>
> >>>> to show development versions of Perl ...
> >>>
> >>> The top seven rows on that page already show the most recent
> development
> >>> versions of Perl. So I'm not really sure what more information you're
> >>> asking for.
> >>
> >> Sorry, maybe I misunderstand.
> >
> > I think you did. That tables lists released versions. Such tables would
> not
> > list every patch in blead.
> >
> >> Is one of those what you'd call 'blead'?
> >
> > No.
>
> In that case I don't I misunderstood - I think you did. But that's
> probably because I'm unfamiliar with the terminology and don't know
> how to express my ideas so people on this list can understand them.
>
> > You would need to build perl from a checkout of the repo at
> > https://github.com/Perl/perl5
>
> My point exactly. And in my case, because of my pathological aversion
> to github (and github's aversion to my browser), I simply won't do it.
>

I dont understand what your browser has to do with it.

git clone https://github.com/Perl/perl5.git
./Configure -Dusethreads -Doptimize=-O3 -d -Dusedevel
make -j16

Or maybe:

git clone https://github.com/Perl/perl5.git
cd perl5
git pull
perlbrew uninstall latest_blead
perlbrew install --notest --noman --64all --as latest_blead --debug
--thread --multi -Doptimize=-g -Dusedevel -j 16 ./
cpanm Your::Favorite::Module


>
> > That wasn't my point however, my point was it would be helpful if folks
> > could take the time to build perl against the latest commit in the blead
> > branch on github and then test it against their favourite modules. If
> > anything doesn't build, file a patch with the author of the module and or
> > maybe with us as an issue.
>
> That's incredibly onerous.


I'm sorry you feel that way. It's what all the people who work hard to
maintain perl do all the time.


> Can't you just get the patched version(s)
> of Perl into cpan-testers and get everything tested automatically by
> the next morning, together with a nice page of idiot lights which show
> Where Things Failed?


IMO no. There are tens of thousands of modules on CPAN, and we can merge
dozens of commits a day sometimes. It's a lot bigger of an undertaking than
you seem to realize.

On the other hand, pulling the latest code, and then using perlbrew to
build it and install it locally and then using cpanm to install your
favorite dependencies is not that difficult.

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Mon, 27 Feb 2023 13:46:46 +0100, demerphq <demerphq@gmail.com> wrote:

> On the other hand, pulling the latest code, and then using perlbrew to build it and install it locally and then using >cpanm to install your favorite dependencies is not that difficult.

I'd like to note that phrasing the actionable like this, as opposed to the original email, is much more likely to actually get someone to do something.

In future if asking someone to do something, highly recommend including step instructions, even if seemingly obvious. Also in this case, probably best to add a small email reply at the top level with such steps.

--
With regards,
Christian Walde

On Mon, Feb 27, 2023 at 7:47?AM demerphq <demerphq@gmail.com> wrote:
>
>>
>> > That wasn't my point however, my point was it would be helpful if folks
>> > could take the time to build perl against the latest commit in the blead
>> > branch on github and then test it against their favourite modules. If
>> > anything doesn't build, file a patch with the author of the module and or
>> > maybe with us as an issue.
>>
>> That's incredibly onerous.
>
>
> I'm sorry you feel that way. It's what all the people who work hard to maintain perl do all the time.
>
>>
>> Can't you just get the patched version(s)
>> of Perl into cpan-testers and get everything tested automatically by
>> the next morning, together with a nice page of idiot lights which show
>> Where Things Failed?
>
>
> IMO no. There are tens of thousands of modules on CPAN, and we can merge dozens of commits a day sometimes. It's a lot bigger of an undertaking than you seem to realize.
>
> On the other hand, pulling the latest code, and then using perlbrew to build it and install it locally and then using cpanm to install your favorite dependencies is not that difficult.
>

I wonder, is there any way to surface any of the existing
blead-breaks-CPAN information for module authors to use?

@demerphq, it seems like you have some way of knowing which CPAN
modules are broken by blead. Where does that information come from?
What would be the easiest way to just blat it onto a webpage that
module authors could access?

--
Thanks,
Jeremy Hetzler
(he/him)

Hi there,

On Mon, 27 Feb 2023, demerphq wrote:
> On Mon, 27 Feb 2023 at 13:01, G.W. Haywood via perl5-porters wrote:
>
>> ... That's incredibly onerous.
>
> I'm sorry you feel that way.

Please understand that I don't feel that way because it causes me any
extra work. It doesn't. I feel that way because it seems to me that
you're asking people to do work which isn't necessary. Wasteful, even.

> It's what all the people who work hard to maintain perl do all the time.

I'm sure you're right. I'm trying to eliminate some of the burden.

>> Can't you just get the patched version(s) of Perl into cpan-testers
>> and get everything tested automatically by the next morning,
>> together with a nice page of idiot lights which show Where Things
>> Failed?
>
> IMO no.

Why not?

> There are tens of thousands of modules on CPAN ...

That was kinda my point. You either get a few people (if you're very
lucky, apparently most people won't bother) to do a Perl build on some
random machine/architecture, over which you have no control, then test
it by building a few random modules (which seems to me like dropping
bombs from an Avro Manchester in WWII[*]), or you get a single person
to do pretty much the same job once, except CPAN test machines can now
use the new Perl to build tens of thousands of modules on a bunch of
different architectures, automatically, within a few hours, more or
less for free. Maybe they'll even mail you a report.

Isn't that going to be a much more useful exercise? It doesn't have
to be after every patch, every day, of course. Just pick some version
ad hoc which seems reasonable to the people doing the development.

> On the other hand, pulling the latest code, and then using perlbrew to
> build it and install it locally and then using cpanm to install your
> favorite dependencies is not that difficult.

Isn't it just as easy to do that on the machine that runs CPAN testers?

Explain to my satisfaction why it isn't and I'll go away. I just want
for you to give the blead Perls some really good exercise within hours
of a commit instead of everybody waiting for (your estimated) couple
of years for it to hit the fan.

[*] About 5% of the bombs landed within five miles of the target.

--

73,
Ged.

On 2/27/23 08:46, Jeremy Hetzler wrote:
> On Mon, Feb 27, 2023 at 7:47?AM demerphq <demerphq@gmail.com> wrote:
>>
>>>
>>>> That wasn't my point however, my point was it would be helpful if
folks
>>>> could take the time to build perl against the latest commit in the
blead
>>>> branch on github and then test it against their favourite modules. If
>>>> anything doesn't build, file a patch with the author of the module
and or
>>>> maybe with us as an issue.
>>>
>>> That's incredibly onerous.
>>
>>
>> I'm sorry you feel that way. It's what all the people who work hard
to maintain perl do all the time.
>>
>>>
>>> Can't you just get the patched version(s)
>>> of Perl into cpan-testers and get everything tested automatically by
>>> the next morning, together with a nice page of idiot lights which show
>>> Where Things Failed?
>>
>>
>> IMO no. There are tens of thousands of modules on CPAN, and we can
merge dozens of commits a day sometimes. It's a lot bigger of an
undertaking than you seem to realize.
>>
>> On the other hand, pulling the latest code, and then using perlbrew
to build it and install it locally and then using cpanm to install your
favorite dependencies is not that difficult.
>>
>
> I wonder, is there any way to surface any of the existing
> blead-breaks-CPAN information for module authors to use?

What we refer to as "blead-breaks-CPAN" (or "BBC") is in the first
instance, just a subset of data generated by individual contributors to
the CPANtesters projects. If you are a CPAN author/maintainer and
someone tests your distribution against *any* version of Perl, on *any*
operating system, with *any* configuration of Perl, and your
distribution fails to build or experiences test failures, then an email
is generated to you and sent within 24 hours. The breakage is also
displayed on the web at matrix.cpantesters.org (or
fast-matrix.cpantesters.org) which is searchable by CPAN distribution.

That report tells you, the distribution author/maintainer, and anyone
else *what* broke but not *why* it broke or what must be done to get the
distribution to build and test properly again. Determining the 'why'
and the 'what is to be done' is a process that must be done by humans
and cannot be automated.

There are a number of people who, over what is now *decades* of
CPANtesting work, have evolved their own procedures for periodically
testing a large number of CPAN distributions against Perl's monthly
development releases. Most of what we refer to as "BBC failures" are
really failures against the most recent monthly dev release. We depend
on those people, as well as individual CPAN maintainers whose code
appears to have been broken by recent dev releases, to report BBC
failures to our GitHub issues queue. We periodically review these
issues; I try to slap a "BBC" label on them in GitHub, which makes them
selectable in the interface. We try to triage these bug reports along
these lines:

* Is the problem a clear, unambiguous mistake in Perl's development
branch ("blead"), such that the onus is on Perl 5 Porters to correct,
with no action needed on the part of the CPAN maintainer?

* Is the problem sub-optimal code in the CPAN distribution whose
problems were there all along but only exposed via a change in Perl
blead, such that the onus is on the CPAN maintainer to fix? (If we have
the time available, we -- P5P -- will try to prepare a patch for the
CPAN maintainer's consideration, but this cannot be guaranteed.)

* Is it some combination of the first two bullet points, such that P5P
and the CPAN maintainers must negotiate a solution?

>
> @demerphq, it seems like you have some way of knowing which CPAN
> modules are broken by blead. Where does that information come from?
> What would be the easiest way to just blat it onto a webpage that
> module authors could access?
>

As mentioned above, CPAN author/maintainers are notified of failures on
CPANtesters by email. Should those failures have occurred against
recent dev releases, the maintainers should contact P5P immediately. We
try our best, but, as @demerphq noted, the size of CPAN means that even
the automatable parts of the process have a formidable scope -- and the
analytical parts of the process cannot be automated.

On Mon, 27 Feb 2023 at 14:47, Jeremy Hetzler <jeremyhetzler@gmail.com>
wrote:

> I wonder, is there any way to surface any of the existing
> blead-breaks-CPAN information for module authors to use?
>

Not sure i follow about "for module authors to use". We have a group of
volunteers, Andreas Koenig, Slaven Rezik, Carlos Guevara being notable
examples, who try to build modules with blead perl as often as they can,
and when they do they file BBC tickets. Some of them (Andreas and Slaven in
particular) are kind enough to bisect the cause and include the commit that
broke the BBC module. I know that from time to time they file tickets with
module owners, and from time to time they file tickets with us. Depending I
guess on whether they think the module is at fault or perl is.

We call these tickets "BBC" tickets. You can search the open tickets on our
issue tracker on github:

https://github.com/Perl/perl5/issues?q=is%3Aissue+is%3Aopen+BBC

Other people do the same but they build with the latest release, eg, perl
5.37.9. I am not sure if they use specific infra for this. Hopefully some
of them will speak up.


>
> @demerphq, it seems like you have some way of knowing which CPAN
> modules are broken by blead. Where does that information come from?
>

It is either via bug reports in our issue tracker, or by me manually trying
to install things with cpanm and the latest perl. Often the latter happens
as part of responding to the former. For instance, installing Dist::Zilla
with the latest blead pulls in a lot of modules so it is a pretty good way
to test a lot of stuff at once.


> What would be the easiest way to just blat it onto a webpage that
> module authors could access?
>

Not sure. I just came back from a 13 hour flight, and my jetlag is really
kicking in now, so I might have better ideas tomorrow when I am more
rested.

cheers,
Yves



--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Mon, Feb 27, 2023 at 9:17 PM demerphq <demerphq@gmail.com> wrote:

> There seems to be this idea that if we deprecate something that we have
> two release cycles, IOW, two *years*, before we have to get stuff fixed,
> and that the only code which might be broken by the deprecation is code
> that use fatal warnings, which we generally advise people not to use.
>

IMO, in this case a suitable fix is to have the fatality of the warnings
disabled.


> But that isn't correct. Odds are *very* likely we get CPAN breakage from
> the very moment we deprecate something, as many modules test to see if they
> generate warnings when they run. This can be transitive breakage with
> "action at a distance", where distribution A uses B uses C used D which
> uses a deprecated feature, and only A actually fails test because it tests
> for warnings but B and C and D do not.
>

IMO, in this case a suitable fix is to have distribution A's test suite
modified to accept or ignore the fact that module D presented a deprecation
warning. There's no need to modify module D.

Is that about it ?
Are there any other scenarios we need to consider at this stage?

The first thing to do would be to file bug reports against the affected
modules - ie either because they fatalize warnings, or because their test
suite overlooked the possibility of "action at a distance".
After that, what we do depends upon whether the registered maintainer
co-operates, or responds but won't co-operate, or doesn't respond.

Have the changes that trigger these issues already been merged into blead ?
If so, I'll start checking the distros listed at
https://github.com/Perl/perl5/issues/20863 in the next day or so, and file
bug reports about the impending changes as appropriate.
Is that a sane way to start ?

Cheers,
Rob

On Tue, 28 Feb 2023 at 12:22, sisyphus <sisyphus359@gmail.com> wrote:

> On Mon, Feb 27, 2023 at 9:17 PM demerphq <demerphq@gmail.com> wrote:
>
>> There seems to be this idea that if we deprecate something that we have
>> two release cycles, IOW, two *years*, before we have to get stuff fixed,
>> and that the only code which might be broken by the deprecation is code
>> that use fatal warnings, which we generally advise people not to use.
>>
>
> IMO, in this case a suitable fix is to have the fatality of the warnings
> disabled.
>

That is still a change we or the author have to make to the distribution.
If the author is AWOL, then the toolchain is broken for that release.


>
>
>> But that isn't correct. Odds are *very* likely we get CPAN breakage from
>> the very moment we deprecate something, as many modules test to see if they
>> generate warnings when they run. This can be transitive breakage with
>> "action at a distance", where distribution A uses B uses C used D which
>> uses a deprecated feature, and only A actually fails test because it tests
>> for warnings but B and C and D do not.
>>
>
> IMO, in this case a suitable fix is to have distribution A's test suite
> modified to accept or ignore the fact that module D presented a deprecation
> warning. There's no need to modify module D.
>

That is MUCH easier said than done. It is generally much easier to fix D
than to fix A to ignore that specific warning. (IMO)


>
> Is that about it ?
> Are there any other scenarios we need to consider at this stage?
>

The main point is that we need to do something, immediately. When I brought
this up on #p5p irc I got a lot of pushback that we have two years. The
point of this mail is that we don't. From the moment we release the
deprecation warnings the CPAN river is impacted. If amodule is high up in
the river and doing what i think many would consider "responsible testing"
and validating their code runs warning free then the distribution will be
broken, even if strictly speaking the code is just fine albeit noisy with
deprecation warnings.


>
>
The first thing to do would be to file bug reports against the affected
> modules - ie either because they fatalize warnings, or because their test
> suite overlooked the possibility of "action at a distance".
> After that, what we do depends upon whether the registered maintainer
> co-operates, or responds but won't co-operate, or doesn't respond.
>

Sure, but it is still a BBC. We still have to do something. We can't just
pretend we have two years to go before we can do something. Until it is
actually fixed upstream the dependency is broken from a CPAN perspective.


> Have the changes that trigger these issues already been merged into blead ?
>

Yes. One is already released in 5.37.9, and one will be released with
5.37.10 and is present in blead.


> If so, I'll start checking the distros listed at
> https://github.com/Perl/perl5/issues/20863 in the next day or so, and
> file bug reports about the impending changes as appropriate.
>
Is that a sane way to start ?
>

Yes indeed. Although I have already filed bug reports (and patches) for a
lot of Father Chrysotomos'es modules (aka SPROUT aka @cpansprout), he was a
proud (ab)user of apostrophe as a package separator. I have not fixed all
of them however, the ones lower down in the river rating on metacpan I have
not checked. It would be nice if we could get that list regenerated by
author.

cheers,
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"

Regarding the thing about the automated CPAN testers infrastructure providing an
automated "blead breaks cpan" report anyone can see, I feel that this idea has
merit, and in theory it should be doable in a way that isn't too onerous.

I take this as an extension of practices I see in some companies where there is
an automated process that regularly runs the full build and test pipeline on
trunk so its more clear if something broke as the result of a recent merge,
often by devs not running all the tests etc before merging.

What I suggest is that on a reasonably frequent but not too frequent schedule,
such as once per 24 hours, each participating CPAN testers machine pulls the
latest Perl blead, its trunk, and builds that, and then tries to build a
selection of CPAN modules of its choice.

I say "selection" under the assumption that a tester machine wouldn't do all of
CPAN in one day, but if say it could normally get through all of CPAN in 5 days
for an individual version of Perl, it would do 1/5 against one day's blead, the
next 1/5 against the next day's blead, and so on round robin, so on the 6th day
it does the first 1/5 again with a version of blead 5 days newer.

In this scenario, the blead results are special, in that whenever a module is
tested against blead "again", that result replaces the one from several days
earlier, so that at any given time it isn't remembering more than one per module
for blead, or blead is a rolling target.

In this case, where one needs to know "when" blead broke CPAN, one can get
enough granularity for this purpose in the CPAN testers testing of the monthly
released dev versions. Mainly the interest is "does blead break this module
NOW", and if it stops breaking then we're fine, and otherwise we compare that
result to the most immediate dev version, and if it broke after that, we know it
was as recent as since that release, and we have heads up that if the issue
isn't addressed then the next dev release will also break the module, and
possibly this can be prevented.

Presumably part of the required logic changes to make this work is that the
official Perl version in the report will need to say something like "blead"
rather than whatever the most recently released normally numbered version is,
since AFAIK trunk/blead continues to have that version formally until the next
release is pumpkined, and I assume the normal CPAN testers logic simply uses the
normally self-reported Perl version number. But the self-reported Perl version
number would have to be left unchanged by this new logic so that any
version-number-sensitive logic in any tests or modules doesn't have to know.

So the end result is that the regular CPAN testers reports have an extra Perl
pseudo-version that they report on called "blead" say but that rather than being
a maybe-immutable log entry like normal test results, it regularly gets replaced
any time the same module is tested against "blead" again, so it has a rolling
meaning; if a module stops being tested then it just reflects the last time that
module was tested.

I realize this would take some work to setup, but hopefully not too much work,
and I feel it could be very useful to the community and possibly cut down on
some manual work to know if blead breaks cpan, while having a granularity of
more than once a month. Although those doing the existing way can still keep
doing that, especially if they want more immediate results than maybe once every
few days or whatever.

-- Darren Duncan

On Tue, 28 Feb 2023 at 12:56, Darren Duncan <darren@darrenduncan.net> wrote:

> Regarding the thing about the automated CPAN testers infrastructure
> providing an
> automated "blead breaks cpan" report anyone can see, I feel that this idea
> has
> merit, and in theory it should be doable in a way that isn't too onerous.
>

One thing to consider is that you want to have a comparison corpus. So for
instance if we introduce deprecation warnings in a given commit, what you
want to do is more than just look for test failures, you want to look for
any output changes between the most recent build and some "last known good"
state. It is not just as simple as run test and find what is broken, as
that doesnt address the transitive breakage very well. It also doesn't help
with a module that has been long broken. Sometimes when I look into
something it now fails for something we recently added, but was failing
before that for something we not-so-recently changed. For instance i looked
at a module (the name eludes me just now) that /was/ failing because scalar
keys was removed in a previous release, but is now failing because
smartmatch is deprecated as well.

Anyway, dont let that discourage you from coming up with something to
automate this more than it is and helping to improve the infra overall. I
feel like our existing cpantesters infra is in a less than ideal state. I
cant get reports from the db from reports we *do* already receive for
instance, the DB is just too slow and overloaded most times. (Not sure what
we can do about that really, except throw money at the problem and get
more/bigger databases. Maybe a company with deep pockets like Booking.com
would like to help out.)

cheers,
Yves

On 2023-02-28 3:56 a.m., Darren Duncan wrote:
> I say "selection" under the assumption that a tester machine wouldn't do all of
> CPAN in one day, but if say it could normally get through all of CPAN in 5 days
> for an individual version of Perl, it would do 1/5 against one day's blead, the
> next 1/5 against the next day's blead, and so on round robin, so on the 6th day
> it does the first 1/5 again with a version of blead 5 days newer.

To clarify, the 1/5ths over all CPAN thing is just an example.

Just as any given tester presumably doesn't test every module against every
regular Perl release, especially every versioned dev release, it doesn't have to
test every module against blead either.

So "selection" is however a tester wants to define it.

If this isn't already done by CPAN testers, something that would be logical to
prioritize for more frequent testing are the modules highest up the CPAN river,
the knowledge of which we AFAIK already have reliable automated gathering of,
and then after those, the most popular modules, which may be a bit trickier to
automate knowledge of but a reasonable approximation should be possible.

So then those prioritized ones, if say there are just on the order of a few
hundred say, they could be retested every time a new blead is built, eg once
every 24 hours, and then the rest of CPAN arbitrarily less often, or the rest
could be skipped and be it considered that just testing on the monthly dev
releases is frequent enough.

But there can be a lot of flexibility here and no need to be prescriptive on
what gets tested or how often, only prescribe that when something is tested
against blead it is done and reported in a prescribed consistent format.

-- Darren Duncan

On 2023-02-28 4:11 a.m., demerphq wrote:
> On Tue, 28 Feb 2023 at 12:56, Darren Duncan wrote:
>
> Regarding the thing about the automated CPAN testers infrastructure
> providing an
> automated "blead breaks cpan" report anyone can see, I feel that this idea has
> merit, and in theory it should be doable in a way that isn't too onerous.
>
> One thing to consider is that you want to have a comparison corpus. So for
> instance if we introduce deprecation warnings in a given commit, what you want
> to do is more than just look for test failures, you want to look for any output
> changes between the most recent build and some "last known good" state. It is
> not just as simple as run test and find what is broken, as that doesnt address
> the transitive breakage very well. It also doesn't help with a module that has
> been long broken. Sometimes when I look into something it now fails for
> something we recently added, but was failing before that for something we
> not-so-recently changed. For instance i looked at a module (the name eludes me
> just now) that /was/ failing because scalar keys was removed in a previous
> release, but is now failing because smartmatch is deprecated as well.
>
> Anyway, dont let that discourage you from coming up with something to automate
> this more than it is and helping to improve the infra overall. I feel like our
> existing cpantesters infra is in a less than ideal state. I cant get reports
> from the db from reports we *do*  already receive for instance, the DB is just
> too slow and overloaded most times. (Not sure what we can do about that really,
> except throw money at the problem and get more/bigger databases. Maybe a company
> with deep pockets like Booking.com would like to help out.)

I consider the minimally useful version of this idea to just flag that there is
a problem in blead, and the comparison corpus is the indicators from the
otherwise most recent tested versioned Perl release.

The main point of the idea is that there is a large number of CPAN modules and
its useful to find out quickly which ones have a problem, and then performing
the manual work of gathering more details only needs to be done on those modules
and not the rest.

As far as a corpus of other output useful to figuring out the problem, I would
assume that this rolling "blead" report would have all of the same details
normally associated with a CPAN testers report. So if the normal ones already
have the required details then we're good just keep doing that. If they don't,
then this issue you raised is orthogaonal to the one I'm talking about and can
be implemented independently for its own benefits.

-- Darren Duncan

On Mon, Feb 27, 2023 at 11:01 PM G.W. Haywood via perl5-porters <
perl5-porters@perl.org> wrote:

>
>
> That's incredibly onerous. Can't you just get the patched version(s)
> of Perl into cpan-testers and get everything tested automatically by
> the next morning, together with a nice page of idiot lights which show
> Where Things Failed?
>

Unfortunately, www.cpantesters.org is incredibly flakey.
For more than 4 days now, cpantesters test reports have been unobtainable.
Relying on them for anything useful often requires far more patience than I
have.

Cheers,
Rob

On Tue, 28 Feb 2023 at 13:27, sisyphus <sisyphus359@gmail.com> wrote:

>
>
> On Mon, Feb 27, 2023 at 11:01 PM G.W. Haywood via perl5-porters <
> perl5-porters@perl.org> wrote:
>
>>
>>
>> That's incredibly onerous. Can't you just get the patched version(s)
>> of Perl into cpan-testers and get everything tested automatically by
>> the next morning, together with a nice page of idiot lights which show
>> Where Things Failed?
>>
>
> Unfortunately, www.cpantesters.org is incredibly flakey.
> For more than 4 days now, cpantesters test reports have been unobtainable.
> Relying on them for anything useful often requires far more patience than
> I have.
>

+1

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Tue, Feb 28, 2023 at 10:38 PM demerphq <demerphq@gmail.com> wrote:

> On Tue, 28 Feb 2023 at 12:22, sisyphus <sisyphus359@gmail.com> wrote:
>
.
>
>> IMO, in this case a suitable fix is to have distribution A's test suite
>> modified to accept or ignore the fact that module D presented a deprecation
>> warning. There's no need to modify module D.
>>
>
> That is MUCH easier said than done. It is generally much easier to fix D
> than to fix A to ignore that specific warning. (IMO)
>
>

Ok - file bug reports against both A and D.
Fixing it in D will suffice, but A is still doing the wrong thing (IMO) by
failing the tests because a warning was given. (Essentially, A's test suite
has made warnings fatal.)


>
> The main point is that we need to do something, immediately.
>


Then let's get started filing bug reports.
The sooner we start, the sooner we find out whether a module has a
responsive maintainer or not, ... and the sooner we find out exactly what
we need to do.

For mine, if the module's metacpan page specifies a github "Repository",
I'll check whether an issue has been raised there - and raise one if it
hasn't.
If there's no github repo, I'll check rt.cpan.org and file a report there
(unless one has already been filed).
Of course, prior to that, I will check that the issue does in fact exist ;-)

Is that how to start ?

Cheers,
Rob

On Tue, 28 Feb 2023 at 14:06, sisyphus <sisyphus359@gmail.com> wrote:

> On Tue, Feb 28, 2023 at 10:38 PM demerphq <demerphq@gmail.com> wrote:
>
>> On Tue, 28 Feb 2023 at 12:22, sisyphus <sisyphus359@gmail.com> wrote:
>>
> .
>>
>>> IMO, in this case a suitable fix is to have distribution A's test suite
>>> modified to accept or ignore the fact that module D presented a deprecation
>>> warning. There's no need to modify module D.
>>>
>>
>> That is MUCH easier said than done. It is generally much easier to fix D
>> than to fix A to ignore that specific warning. (IMO)
>>
>>
>
> Ok - file bug reports against both A and D.
> Fixing it in D will suffice, but A is still doing the wrong thing (IMO) by
> failing the tests because a warning was given. (Essentially, A's test suite
> has made warnings fatal.
>

I can see the argument, but i can also see the counter argument; that
people want to test that their code functions correctly and is warning
free. Modules like Test::Warn wouldn't exist if this weren't a common
practice. I don't think we will get a lot of traction telling people not to
do it. Maybe Test::Warn should know to ignore deprecation warnings except
when AUTHOR_TESTING is true.

See
https://github.com/Perl-Toolchain-Gang/toolchain-site/blob/master/lancaster-consensus.md

Even then i can see people arguing that one either way. Regardless for
perl5-porters, i believe our position is we take responsibility if
something we do breaks something on CPAN, even if really it shouldnt have.
Hyrums law in a nutshell i guess.

https://www.hyrumslaw.com/


>
>>
>> The main point is that we need to do something, immediately.
>>
>
>
> Then let's get started filing bug reports.
>

I have, and I have been filing patches too.


> The sooner we start, the sooner we find out whether a module has a
> responsive maintainer or not, ... and the sooner we find out exactly what
> we need to do.
>

Agreed. It is also something that you get faster at the more you do.


> For mine, if the module's metacpan page specifies a github "Repository",
> I'll check whether an issue has been raised there - and raise one if it
> hasn't.
> If there's no github repo, I'll check rt.cpan.org and file a report there
> (unless one has already been filed).
> Of course, prior to that, I will check that the issue does in fact exist
> ;-)
>
> Is that how to start ?
>

Yes, I think so. I generally check the "repository" link on metacpan.org,
and then if its not there, the "issues" link. Some things on CPAN seem to
point at RT even though they have a github repo with an Issues tab. If they
have a repository link i tend to clone and fix and then push PR instead of
filing a bug report. But maybe that isnt the right thing to do. James
might have some additional thoughts on that, historically he has done a lot
of the BBC coordination and cat-herding.

Cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Tue, Feb 28, 2023 at 5:42?AM demerphq <demerphq@gmail.com> wrote:

> I can see the argument, but i can also see the counter argument; that
> people want to test that their code functions correctly and is warning
> free. Modules like Test::Warn wouldn't exist if this weren't a common
> practice. I don't think we will get a lot of traction telling people not to
> do it. Maybe Test::Warn should know to ignore deprecation warnings except
> when AUTHOR_TESTING is true.
>
> See
> https://github.com/Perl-Toolchain-Gang/toolchain-site/blob/master/lancaster-consensus.md
>
> Even then i can see people arguing that one either way. Regardless for
> perl5-porters, i believe our position is we take responsibility if
> something we do breaks something on CPAN, even if really it shouldnt have.
> Hyrums law in a nutshell i guess.
>
> https://www.hyrumslaw.com/
>

Yes, it's been my practice for many years to only enable fail-on-warnings
when AUTHOR_TESTING is true. All test files in my distributions do this:

use if $ENV{AUTHOR_TESTING}, 'Test::Warnings';

On Mon, Feb 27, 2023 at 4:01?AM G.W. Haywood via perl5-porters <
perl5-porters@perl.org> wrote:

> > You would need to build perl from a checkout of the repo at
> > https://github.com/Perl/perl5
>
> My point exactly. And in my case, because of my pathological aversion
> to github (and github's aversion to my browser), I simply won't do it.
>

Why not? Why can't you do:

git clone https://github.com/Perl/perl5.git

What does your browser have to do with anything?

Hi there,

On Tue, 28 Feb 2023, Karen Etheridge wrote:

> On Mon, Feb 27, 2023 at 4:01?AM G.W. Haywood via perl5-porters wrote:
>
>>
>> ... because of my pathological aversion
>> to github (and github's aversion to my browser), I simply won't do it.
>
> Why not? Why can't you do:
>
> git clone https://github.com/Perl/perl5.git

Can you remember ADSL?

I *can* do it, but I won't. Last time I supplied a ten-character
patch to the documentation for an open source project (ClamAV) for
example, to do it I had to transfer about 20 megabytes of data. I
could have done the same thing with a 500 byte email in a fraction of
the time, but there's no facility to do that. It's insane.

> What does your browser have to do with anything?

Just that if I want to use it to look at anything that's been going on
in a project, either I get a message telling me to, er, update to a
supported browser (a euphemism, it means junk the one I'm used to and
get another one) or at best a little twirly thing. It's like a whole
planet has completely forgotten the point of HTML. And don't get me
started on the bowdlerization of patches by text-to-HTML conversion.
</rant:you did ask>

--

73,
Ged.