Mailing List Archive

> On Tuesday, 23 November 2021, 18:55:55 CET, Neil Bowers <neilb@neilb.org>
> wrote:
>
> When you install Perl, although it comes with HTTP::Tiny, it doesn't support 
> https because HTTP::Tiny requires IO::Socket::SSL (which isn’t bundled with
> Perl). This means that CPAN.pm and other tools can't make https mandatory,
> as when bootstrapping a new box you'll typically install perl then run cpan to
> first install Net::SSLeay and IO::Socket::SSL.

I love this idea and as a standalone, I support it as is, if feasible. However, if we do this, can we improve the overall experience of Perl?

One of the issue that I think Perl has is that we our presence is almost entirely in sysadmin tasks and web development. I can't comment on the former, but the latter also requires a few more modules to make Perl more natively useful. If nothing else, a fast JSON tool would be welcome (not the JSON::PP that is in core. That's very slow)

Also, some proper native readline support would be nice for a useful debugger. I assume Term::ReadLine::Gnu isn't bundled because of the licence, but my debugger is always broken when I first use it.

Best,
Ovid

Op 23-11-2021 om 19:18 schreef Ovid via perl5-porters:
>> On Tuesday, 23 November 2021, 18:55:55 CET, Neil Bowers <neilb@neilb.org>
>> wrote:
>>
>> When you install Perl, although it comes with HTTP::Tiny, it doesn't support
>> https because HTTP::Tiny requires IO::Socket::SSL (which isn’t bundled with
>> Perl). This means that CPAN.pm and other tools can't make https mandatory,
>> as when bootstrapping a new box you'll typically install perl then run cpan to
>> first install Net::SSLeay and IO::Socket::SSL.
> I love this idea and as a standalone, I support it as is, if feasible.


+1


> However, if we do this, can we improve the overall experience of Perl?
>
> One of the issue that I think Perl has is that we our presence is almost entirely in sysadmin tasks and web development. I can't comment on the former, but the latter also requires a few more modules to make Perl more natively useful. If nothing else, a fast JSON tool would be welcome (not the JSON::PP that is in core. That's very slow)


Standard CSV parsing is another one. I use a lot of Python only because
that does support CSV parsing out of the box.


>
> Also, some proper native readline support would be nice for a useful debugger. I assume Term::ReadLine::Gnu isn't bundled because of the licence, but my debugger is always broken when I first use it.


+10 ;)


HTH,

M4

* Neil Bowers <neilb@neilb.org> [2021-11-23 17:55:28 +0000]:

> When you install Perl, although it comes with HTTP::Tiny, it doesn't support https because HTTP::Tiny requires IO::Socket::SSL (which isn?t bundled with Perl). This means that CPAN.pm and other tools can't make https mandatory, as when bootstrapping a new box you'll typically install perl then run cpan to first install Net::SSLeay and IO::Socket::SSL.
>
> I think we should make Perl support https "out of the box", so that CPAN.pm could make https mandatory, and other modules could then support https while relying only on core modules.
>
> One approach is to bundle Net::SSLeay and IO::Socket::SSL, but I don't know if that's the best / right solution?
>
> One problem with bundling these modules is that we are then committed to ensuring that they work on all supported platforms / configurations. Is it ok to support https out-of-the-box on some supported platforms but not all of them? I think so.

That last point is the rub. I've on several occasions run into issues with
Net::SSLeay not building on upgrade Ubuntu because of a lag in supporting
some breaking ABI change in libssl. In addition to that, there might be some
issues with OSes that distribute perl even including the necessary development
libraries/headers out of the box. Otherwise, I think this is a nice idea -
just might require a large coordinated effort - particularly from "downstream"
packages, ports, and base OS maintainers. I'm none of those, but I do feel your
pain..

Brett

>
> Neil

--
--
oodler@cpan.org
oodler577@sdf-eu.org
SDF-EU Public Access UNIX System - http://sdfeu.org
irc.perl.org #openmp #pdl #native

Awesome as SSL is a modern standard.

23 noy 2021, Ç.a. 22:30 tarixind? Oodler 577 via perl5-porters <
perl5-porters@perl.org> yazd?:

> * Neil Bowers <neilb@neilb.org> [2021-11-23 17:55:28 +0000]:
>
> > When you install Perl, although it comes with HTTP::Tiny, it doesn't
> support https because HTTP::Tiny requires IO::Socket::SSL (which isn?t
> bundled with Perl). This means that CPAN.pm and other tools can't make
> https mandatory, as when bootstrapping a new box you'll typically install
> perl then run cpan to first install Net::SSLeay and IO::Socket::SSL.
> >
> > I think we should make Perl support https "out of the box", so that
> CPAN.pm could make https mandatory, and other modules could then support
> https while relying only on core modules.
> >
> > One approach is to bundle Net::SSLeay and IO::Socket::SSL, but I don't
> know if that's the best / right solution?
> >
> > One problem with bundling these modules is that we are then committed to
> ensuring that they work on all supported platforms / configurations. Is it
> ok to support https out-of-the-box on some supported platforms but not all
> of them? I think so.
>
> That last point is the rub. I've on several occasions run into issues with
> Net::SSLeay not building on upgrade Ubuntu because of a lag in supporting
> some breaking ABI change in libssl. In addition to that, there might be
> some
> issues with OSes that distribute perl even including the necessary
> development
> libraries/headers out of the box. Otherwise, I think this is a nice idea -
> just might require a large coordinated effort - particularly from
> "downstream"
> packages, ports, and base OS maintainers. I'm none of those, but I do feel
> your
> pain..
>
> Brett
>
> >
> > Neil
>
> --
> --
> oodler@cpan.org
> oodler577@sdf-eu.org
> SDF-EU Public Access UNIX System - http://sdfeu.org
> irc.perl.org #openmp #pdl #native
>

??, 23 ????. 2021 ?. ? 20:55, Neil Bowers <neilb@neilb.org>:
> One approach is to bundle Net::SSLeay and IO::Socket::SSL, but I don't know if that's the best / right solution?
>
> One problem with bundling these modules is that we are then committed to ensuring that they work on all supported platforms / configurations. Is it ok to support https out-of-the-box on some supported platforms but not all of them? I think so.

There exists (at least) one huge gotcha. openssl major api bump &
rolling distros like arch/void. There will be situations when the
distro already has an api-incompatible library, but even bleadperl
still doesn't have an updated bindings, not starting to speak about
x-1 release. So for a successful experience, both in-core tests and
http layer over it should be ready for a complete compilation failures
and subsequent module absence.

I'd say that it should be it's own separate level of guarantees for
those modules and their availability despite being bundled with core.

Best regards,
Sergey Aleynikov

On Tue, 23 Nov 2021 17:55:28 +0000
Neil Bowers <neilb@neilb.org> wrote:

> I think we should make Perl support https "out of the box", so that
> CPAN.pm could make https mandatory, and other modules could then
> support https while relying only on core modules.

I don't think anyone objects to the idea of better (i.e. any at all)
support of SSL/TLS out-of-the-box, but in case it needed being said:

+1 from me too.

> One approach is to bundle Net::SSLeay and IO::Socket::SSL, but I
> don't know if that's the best / right solution?
>
> One problem with bundling these modules is that we are then committed
> to ensuring that they work on all supported platforms /
> configurations. Is it ok to support https out-of-the-box on some
> supported platforms but not all of them? I think so.

I feel surely this isn't a problem that Perl uniquely is trying to
solve. It feels like this should be easily solved by "doing what the
other languages do" - i.e. could someone more familiar with some of the
others report back on how e.g. Python or Ruby handle this kind of
thing? If it works for them then surely we can do the same?

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

On Tue, Nov 23, 2021 at 1:19 PM Ovid via perl5-porters <
perl5-porters@perl.org> wrote:

> > On Tuesday, 23 November 2021, 18:55:55 CET, Neil Bowers <neilb@neilb.org
> >
> > wrote:
> >
> > When you install Perl, although it comes with HTTP::Tiny, it doesn't
> support
> > https because HTTP::Tiny requires IO::Socket::SSL (which isn’t bundled
> with
> > Perl). This means that CPAN.pm and other tools can't make https
> mandatory,
> > as when bootstrapping a new box you'll typically install perl then run
> cpan to
> > first install Net::SSLeay and IO::Socket::SSL.
>

I agree that this is a huge problem and we should make significant effort
to try to solve it. However it's not yet clear how we could best solve it.
Bundling Net::SSLeay and IO::Socket::SSL is of course the easiest in terms
of immediate effort

An alternative is to link against SSL libs within Perl's own build process
like some other languages do, which would have the potential for greater
flexibility/portability. But it means some duplication of effort, and that
it can't be upgraded from CPAN like Net::SSLeay (important for any SSL
functionality - though it could be implemented in a blead-first dual-life
module to make this possible), and that IO::Socket::SSL (probably) would
need to be updated to make use of the new functionality. And foremost, this
would need a volunteer well versed in SSL libraries.


> I love this idea and as a standalone, I support it as is, if feasible.
> However, if we do this, can we improve the overall experience of Perl?
>
> One of the issue that I think Perl has is that we our presence is almost
> entirely in sysadmin tasks and web development. I can't comment on the
> former, but the latter also requires a few more modules to make Perl more
> natively useful. If nothing else, a fast JSON tool would be welcome (not
> the JSON::PP that is in core. That's very slow)
>
> Also, some proper native readline support would be nice for a useful
> debugger. I assume Term::ReadLine::Gnu isn't bundled because of the
> licence, but my debugger is always broken when I first use it.
>

Not sure what either of these have to do with supporting HTTPS out of the
box. Perhaps suitable for other threads?

-Dan

On Tue, 23 Nov 2021 17:55:28 +0000
Neil Bowers <neilb@neilb.org> wrote:

> When you install Perl, although it comes with HTTP::Tiny, it doesn't support https because HTTP::Tiny requires IO::Socket::SSL (which isn?t bundled with Perl). This means that CPAN.pm and other tools can't make https mandatory, as when bootstrapping a new box you'll typically install perl then run cpan to first install Net::SSLeay and IO::Socket::SSL.
>
> I think we should make Perl support https "out of the box", so that CPAN.pm could make https mandatory, and other modules could then support https while relying only on core modules.
>
> One approach is to bundle Net::SSLeay and IO::Socket::SSL, but I don't know if that's the best / right solution?
>
> One problem with bundling these modules is that we are then committed to ensuring that they work on all supported platforms / configurations. Is it ok to support https out-of-the-box on some supported platforms but not all of them? I think so.

We have a few optional core modules with external dependencies. For
example, we don't build DB_File when BerkeleyDB isn't installed. I think
it would be acceptable to do the same with the SSL modules.

In the future we might want to extend IO::Socket::SSL with additional
non-OpenSSL backends (e.g. pure-perl, SChannel for Windows), but that
shouldn't block its inclusion in core.

2021-11-24 2:55 Neil Bowers <neilb@neilb.org> wrote:

> When you install Perl, although it comes with HTTP::Tiny, it doesn't
> support https because HTTP::Tiny requires IO::Socket::SSL (which isn’t
> bundled with Perl). This means that CPAN.pm and other tools can't make
> https mandatory, as when bootstrapping a new box you'll typically install
> perl then run cpan to first install Net::SSLeay and IO::Socket::SSL.
>
> I think we should make Perl support https "out of the box", so that
> CPAN.pm could make https mandatory, and other modules could then support
> https while relying only on core modules.
>
> One approach is to bundle Net::SSLeay and IO::Socket::SSL, but I don't
> know if that's the best / right solution?
>
> One problem with bundling these modules is that we are then committed to
> ensuring that they work on all supported platforms / configurations. Is it
> ok to support https out-of-the-box on some supported platforms but not all
> of them? I think so.
>
> Neil
>

I have a question about technical matters.

Net::SSLeay is a Perl binding of the OpenSSL library.

Normally, we can install Net::SSLeay only if the OpenSSL library is already
installed and the tests of Net::SSLeay are successful.

Suppose the Perl core contains Net::SSLeay.

In this case, Net::SSLeay is already installed without the OpenSSL library
and the tests of Net::SSLeay are not yet done.

How do we test for correctness of behavior of Net::SSLeay?

Do we need a conversation with the OpenSSL developers?

On Wed, Dec 1, 2021, at 8:08 PM, Yuki Kimoto wrote:
> Suppose the Perl core contains Net::SSLeay.
>
> In this case, Net::SSLeay is already installed without the OpenSSL library and the tests of Net::SSLeay are not yet done.
>
> How do we test for correctness of behavior of Net::SSLeay?

We would only build, test, and install Net::SSLeay if openssl was available.

--
rjbs

2021-12-2 10:55 Ricardo Signes <perl.p5p@rjbs.manxome.org> wrote:

> On Wed, Dec 1, 2021, at 8:08 PM, Yuki Kimoto wrote:
>
> Suppose the Perl core contains Net::SSLeay.
>
> In this case, Net::SSLeay is already installed without the OpenSSL library
> and the tests of Net::SSLeay are not yet done.
>
> How do we test for correctness of behavior of Net::SSLeay?
>
>
> We would only build, test, and install Net::SSLeay if openssl was
> available.
>
> --
> rjbs
>

I've read the article that Perl is required to build OpenSSL.

If this is true, can OpenSSL exist before Perl is installed?

On Wed, Dec 1, 2021 at 7:55 PM Ricardo Signes <perl.p5p@rjbs.manxome.org> wrote:
>
> On Wed, Dec 1, 2021, at 8:08 PM, Yuki Kimoto wrote:
>
> Suppose the Perl core contains Net::SSLeay.
>
> In this case, Net::SSLeay is already installed without the OpenSSL library and the tests of Net::SSLeay are not yet done.
>
> How do we test for correctness of behavior of Net::SSLeay?
>
>
> We would only build, test, and install Net::SSLeay if openssl was available.

I agree this is what we would want to do, with the "extensions"
Configure variable just not including Net::SSLeay if we couldn't
locate the SSL library at Configure time. It might be a trickier case
than any we've had before, though. The Makefile.PL for Net::SSLeay:

https://github.com/radiator-software/p5-net-ssleay/blob/master/Makefile.PL

goes through all sorts of gyrations to locate the OpenSSL or LibreSSL
libraries and headers in their platform-specific hiding places, which
tend to move around with library and platform releases. It would be a
shame to have to copy and then maintain separately all of that logic
in Configure, but offhand I'm not sure how else we would do it.

Also note that Net::SSLeay requires Perl to be built with the same
compiler and options as (Open|Libre)SSL. Which would probably be
uneventful in many cases but could lead to a new flavor of bug report
in others.

Not trying to throw sand in the gears, but just trying to identify the
sand that will get in anyway.

On Thu, Dec 2, 2021 at 12:28 AM Yuki Kimoto <kimoto.yuki@gmail.com> wrote:

> I've read the article that Perl is required to build OpenSSL.
>
> If this is true, can OpenSSL exist before Perl is installed?

A previous version of Perl should be adequate to build OpenSSL, but if
you are really building everything from scratch, you would need to
build a Perl without SSL support, build OpenSSL, and then rebuild
Perl.

2021-12-2 22:41 Craig A. Berry <craig.a.berry@gmail.com> wrote:

> On Thu, Dec 2, 2021 at 12:28 AM Yuki Kimoto <kimoto.yuki@gmail.com> wrote:
>
> > I've read the article that Perl is required to build OpenSSL.
> >
> > If this is true, can OpenSSL exist before Perl is installed?
>
> A previous version of Perl should be adequate to build OpenSSL, but if
> you are really building everything from scratch, you would need to
> build a Perl without SSL support, build OpenSSL, and then rebuild
> Perl.
>

Craig

OK. thanks!

I've tried to re-read all the different threads related to this,
but Spark mail gets confused, and reading this all via nntp.perl.org
ain't user friendly either.

The motives are, at least, the following:
 - almost all APIs are https these days
 - and increasingly all websites
 - it will enable CPAN.pm to do https on a fresh install
 (it has other options for https, but those might not be available)

options:
 1. bundle modules, look for openssl (and poss other libs)
 2. bundle an SSL lib and modules such as Mbed TLS, WolfSSL
 3. Go with Curl, as it can work with a range of SSL libraries
 4. work with various SSL programs (wget, curl, etc)
 5. anything else?

I suspect that the path of least resistance right now is to take
the first option:
 - dual-life Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
 - Configure would look for OpenSSL or LibreSSL
 - presumably it would have to make sure it's a recent enough version
 - possibly refuse to proceed with certain versions that are now
 known to have serious security issues
 - cope with a later version of OpenSSL that's incompatible

We'd have to approach the current maintainers of those modules to
see if they're happy for them to be bundle, explaining the implications.

Net-SSLeay's Makefile.PL has stuff to to look for OpenSSL and work
out compatibility. One possible path is that someone familiar with
that stuff works with someone familiar with metaconfig, so that
when installed with Perl, Configure is doing the heavy lifting,
but it can still be installed from CPAN with Makefile.PL doing its thing.

This doesn't preclude people starting parallel efforts to experiment
with other approaches, but my sense is that more effort has been put
into Net::SSLeay + OpenSSL, so we should start off on those shoulders.

There are a number of pieces to this pie, so I'm guessing this will
probably need a small group of people to work on it.

I wonder if a good first step would be a proof-of-concept:
a. hack together a metaconfig unit based on Net::SSLeay's Makefile.PL
b. if Net::SSLeay builds and tests successfully, then install
    Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
c. Contact the maintainers of those modules to sound them out on
    openness to dual-lifing.
d. See if this can be made to work on Linux, *BSD, MacOS, and Windows.

Going with Curl is appealing, but unless we've got someone prepared
to take that on, it feels more complex and less well-known than the
approach outlined above.

Neil

Awesome, totally agree on the first approach.

I hope to see these three distributions in core for Perl 5.38.

These three modules are already heavily used on almost all Perl
applications as de-facto SSL standard, so why not dual-life them finally?

Usually installing the IO-Socket-SSL package pulls the two others as
dependencies on Unix systems used for HTTPS support in Perl.

Curl is PHP's approach to HTTP(S) from the beginning. It's already too late
to consider Curl as an option I believe. Because it's an alien in the Perl
world, and not as mature and integrated as the previous three.

These are my thoughts stemming from my personal experience with Perl on
Unix for many years.

Thanks.



14 iyn 2022, Ç.a. 01:11 tarixind? Neil Bowers <neilb@neilb.org> yazd?:

> I've tried to re-read all the different threads related to this,
> but Spark mail gets confused, and reading this all via nntp.perl.org
> ain't user friendly either.
>
> The motives are, at least, the following:
> - almost all APIs are https these days
> - and increasingly all websites
> - it will enable CPAN.pm to do https on a fresh install
> (it has other options for https, but those might not be available)
>
> options:
> 1. bundle modules, look for openssl (and poss other libs)
> 2. bundle an SSL lib and modules such as Mbed TLS, WolfSSL
> 3. Go with Curl, as it can work with a range of SSL libraries
> 4. work with various SSL programs (wget, curl, etc)
> 5. anything else?
>
> I suspect that the path of least resistance right now is to take
> the first option:
> - dual-life Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
> - Configure would look for OpenSSL or LibreSSL
> - presumably it would have to make sure it's a recent enough version
> - possibly refuse to proceed with certain versions that are now
> known to have serious security issues
> - cope with a later version of OpenSSL that's incompatible
>
> We'd have to approach the current maintainers of those modules to
> see if they're happy for them to be bundle, explaining the implications.
>
> Net-SSLeay's Makefile.PL has stuff to to look for OpenSSL and work
> out compatibility. One possible path is that someone familiar with
> that stuff works with someone familiar with metaconfig, so that
> when installed with Perl, Configure is doing the heavy lifting,
> but it can still be installed from CPAN with Makefile.PL doing its thing.
>
> This doesn't preclude people starting parallel efforts to experiment
> with other approaches, but my sense is that more effort has been put
> into Net::SSLeay + OpenSSL, so we should start off on those shoulders.
>
> There are a number of pieces to this pie, so I'm guessing this will
> probably need a small group of people to work on it.
>
> I wonder if a good first step would be a proof-of-concept:
> a. hack together a metaconfig unit based on Net::SSLeay's Makefile.PL
> b. if Net::SSLeay builds and tests successfully, then install
> Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
> c. Contact the maintainers of those modules to sound them out on
> openness to dual-lifing.
> d. See if this can be made to work on Linux, *BSD, MacOS, and Windows.
>
> Going with Curl is appealing, but unless we've got someone prepared
> to take that on, it feels more complex and less well-known than the
> approach outlined above.
>
> Neil
>

On 6/13/22 17:11, Neil Bowers wrote:
> options:
> 1. bundle modules, look for openssl (and poss other libs)
> 2. bundle an SSL lib and modules such as Mbed TLS, WolfSSL
> 3. Go with Curl, as it can work with a range of SSL libraries
> 4. work with various SSL programs (wget, curl, etc)
> 5. anything else?
>
> I suspect that the path of least resistance right now is to take
> the first option:
> - dual-life Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
> - Configure would look for OpenSSL or LibreSSL
> - presumably it would have to make sure it's a recent enough version
> - possibly refuse to proceed with certain versions that are now
> known to have serious security issues
> - cope with a later version of OpenSSL that's incompatible
>
> We'd have to approach the current maintainers of those modules to
> see if they're happy for them to be bundle, explaining the implications.

+1 for this approach, it keeps the ball rolling.


I would like to add that the Docker perl image now supports HTTPS by
default and also ships with cpm.

Cheers,
Wesley


--
Wesley Schwengle

Dual-life of Mozilla::CA seems problematic for the same reason as bundling a TLS library: the need for updates.

Most OpenSSL installs seem to include a root bundle; could we defer to that instead?

A short C snippet that shows how to get that directory:
https://stackoverflow.com/a/4141173/586723

-FG


> On Jun 13, 2022, at 19:56, Elvin Aslanov <rwp.primary@gmail.com> wrote:
>
> Awesome, totally agree on the first approach.
>
> I hope to see these three distributions in core for Perl 5.38.
>
> These three modules are already heavily used on almost all Perl applications as de-facto SSL standard, so why not dual-life them finally?
>
> Usually installing the IO-Socket-SSL package pulls the two others as dependencies on Unix systems used for HTTPS support in Perl.
>
> Curl is PHP's approach to HTTP(S) from the beginning. It's already too late to consider Curl as an option I believe. Because it's an alien in the Perl world, and not as mature and integrated as the previous three.
>
> These are my thoughts stemming from my personal experience with Perl on Unix for many years.
>
> Thanks.
>
>
>
> 14 iyn 2022, Ç.a. 01:11 tarixind? Neil Bowers <neilb@neilb.org> yazd?:
> I've tried to re-read all the different threads related to this,
> but Spark mail gets confused, and reading this all via nntp.perl.org
> ain't user friendly either.
>
> The motives are, at least, the following:
> - almost all APIs are https these days
> - and increasingly all websites
> - it will enable CPAN.pm to do https on a fresh install
> (it has other options for https, but those might not be available)
>
> options:
> 1. bundle modules, look for openssl (and poss other libs)
> 2. bundle an SSL lib and modules such as Mbed TLS, WolfSSL
> 3. Go with Curl, as it can work with a range of SSL libraries
> 4. work with various SSL programs (wget, curl, etc)
> 5. anything else?
>
> I suspect that the path of least resistance right now is to take
> the first option:
> - dual-life Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
> - Configure would look for OpenSSL or LibreSSL
> - presumably it would have to make sure it's a recent enough version
> - possibly refuse to proceed with certain versions that are now
> known to have serious security issues
> - cope with a later version of OpenSSL that's incompatible
>
> We'd have to approach the current maintainers of those modules to
> see if they're happy for them to be bundle, explaining the implications.
>
> Net-SSLeay's Makefile.PL has stuff to to look for OpenSSL and work
> out compatibility. One possible path is that someone familiar with
> that stuff works with someone familiar with metaconfig, so that
> when installed with Perl, Configure is doing the heavy lifting,
> but it can still be installed from CPAN with Makefile.PL doing its thing.
>
> This doesn't preclude people starting parallel efforts to experiment
> with other approaches, but my sense is that more effort has been put
> into Net::SSLeay + OpenSSL, so we should start off on those shoulders.
>
> There are a number of pieces to this pie, so I'm guessing this will
> probably need a small group of people to work on it.
>
> I wonder if a good first step would be a proof-of-concept:
> a. hack together a metaconfig unit based on Net::SSLeay's Makefile.PL
> b. if Net::SSLeay builds and tests successfully, then install
> Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
> c. Contact the maintainers of those modules to sound them out on
> openness to dual-lifing.
> d. See if this can be made to work on Linux, *BSD, MacOS, and Windows.
>
> Going with Curl is appealing, but unless we've got someone prepared
> to take that on, it feels more complex and less well-known than the
> approach outlined above.
>
> Neil

yeah but `cpan Mozilla::CA` isn't hard to do to update the module and it
won't break with newer Perl versions as well since it's just plaintext
non-code certificates bundle

On Tue, Jun 14, 2022 at 7:52 PM Felipe Gasper <felipe@felipegasper.com>
wrote:

> Dual-life of Mozilla::CA seems problematic for the same reason as bundling
> a TLS library: the need for updates.
>
> Most OpenSSL installs seem to include a root bundle; could we defer to
> that instead?
>
> A short C snippet that shows how to get that directory:
> https://stackoverflow.com/a/4141173/586723
>
> -FG
>
>
> > On Jun 13, 2022, at 19:56, Elvin Aslanov <rwp.primary@gmail.com> wrote:
> >
> > Awesome, totally agree on the first approach.
> >
> > I hope to see these three distributions in core for Perl 5.38.
> >
> > These three modules are already heavily used on almost all Perl
> applications as de-facto SSL standard, so why not dual-life them finally?
> >
> > Usually installing the IO-Socket-SSL package pulls the two others as
> dependencies on Unix systems used for HTTPS support in Perl.
> >
> > Curl is PHP's approach to HTTP(S) from the beginning. It's already too
> late to consider Curl as an option I believe. Because it's an alien in the
> Perl world, and not as mature and integrated as the previous three.
> >
> > These are my thoughts stemming from my personal experience with Perl on
> Unix for many years.
> >
> > Thanks.
> >
> >
> >
> > 14 iyn 2022, Ç.a. 01:11 tarixind? Neil Bowers <neilb@neilb.org> yazd?:
> > I've tried to re-read all the different threads related to this,
> > but Spark mail gets confused, and reading this all via nntp.perl.org
> > ain't user friendly either.
> >
> > The motives are, at least, the following:
> > - almost all APIs are https these days
> > - and increasingly all websites
> > - it will enable CPAN.pm to do https on a fresh install
> > (it has other options for https, but those might not be available)
> >
> > options:
> > 1. bundle modules, look for openssl (and poss other libs)
> > 2. bundle an SSL lib and modules such as Mbed TLS, WolfSSL
> > 3. Go with Curl, as it can work with a range of SSL libraries
> > 4. work with various SSL programs (wget, curl, etc)
> > 5. anything else?
> >
> > I suspect that the path of least resistance right now is to take
> > the first option:
> > - dual-life Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
> > - Configure would look for OpenSSL or LibreSSL
> > - presumably it would have to make sure it's a recent enough version
> > - possibly refuse to proceed with certain versions that are now
> > known to have serious security issues
> > - cope with a later version of OpenSSL that's incompatible
> >
> > We'd have to approach the current maintainers of those modules to
> > see if they're happy for them to be bundle, explaining the implications.
> >
> > Net-SSLeay's Makefile.PL has stuff to to look for OpenSSL and work
> > out compatibility. One possible path is that someone familiar with
> > that stuff works with someone familiar with metaconfig, so that
> > when installed with Perl, Configure is doing the heavy lifting,
> > but it can still be installed from CPAN with Makefile.PL doing its thing.
> >
> > This doesn't preclude people starting parallel efforts to experiment
> > with other approaches, but my sense is that more effort has been put
> > into Net::SSLeay + OpenSSL, so we should start off on those shoulders.
> >
> > There are a number of pieces to this pie, so I'm guessing this will
> > probably need a small group of people to work on it.
> >
> > I wonder if a good first step would be a proof-of-concept:
> > a. hack together a metaconfig unit based on Net::SSLeay's Makefile.PL
> > b. if Net::SSLeay builds and tests successfully, then install
> > Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
> > c. Contact the maintainers of those modules to sound them out on
> > openness to dual-lifing.
> > d. See if this can be made to work on Linux, *BSD, MacOS, and Windows.
> >
> > Going with Curl is appealing, but unless we've got someone prepared
> > to take that on, it feels more complex and less well-known than the
> > approach outlined above.
> >
> > Neil
>
>
>

If we’re OK with that then we could ship mbedTLS and have TLS always “just work”.

I’ve been under the impression that’s unacceptable to depend on users to update TLS stuff manually, be it the root store or the code.

-FG

> On Jun 14, 2022, at 12:18, Elvin Aslanov <rwp.primary@gmail.com> wrote:
>
> yeah but `cpan Mozilla::CA` isn't hard to do to update the module and it won't break with newer Perl versions as well since it's just plaintext non-code certificates bundle
>
> On Tue, Jun 14, 2022 at 7:52 PM Felipe Gasper <felipe@felipegasper.com> wrote:
> Dual-life of Mozilla::CA seems problematic for the same reason as bundling a TLS library: the need for updates.
>
> Most OpenSSL installs seem to include a root bundle; could we defer to that instead?
>
> A short C snippet that shows how to get that directory:
> https://stackoverflow.com/a/4141173/586723
>
> -FG
>
>
> > On Jun 13, 2022, at 19:56, Elvin Aslanov <rwp.primary@gmail.com> wrote:
> >
> > Awesome, totally agree on the first approach.
> >
> > I hope to see these three distributions in core for Perl 5.38.
> >
> > These three modules are already heavily used on almost all Perl applications as de-facto SSL standard, so why not dual-life them finally?
> >
> > Usually installing the IO-Socket-SSL package pulls the two others as dependencies on Unix systems used for HTTPS support in Perl.
> >
> > Curl is PHP's approach to HTTP(S) from the beginning. It's already too late to consider Curl as an option I believe. Because it's an alien in the Perl world, and not as mature and integrated as the previous three.
> >
> > These are my thoughts stemming from my personal experience with Perl on Unix for many years.
> >
> > Thanks.
> >
> >
> >
> > 14 iyn 2022, Ç.a. 01:11 tarixind? Neil Bowers <neilb@neilb.org> yazd?:
> > I've tried to re-read all the different threads related to this,
> > but Spark mail gets confused, and reading this all via nntp.perl.org
> > ain't user friendly either.
> >
> > The motives are, at least, the following:
> > - almost all APIs are https these days
> > - and increasingly all websites
> > - it will enable CPAN.pm to do https on a fresh install
> > (it has other options for https, but those might not be available)
> >
> > options:
> > 1. bundle modules, look for openssl (and poss other libs)
> > 2. bundle an SSL lib and modules such as Mbed TLS, WolfSSL
> > 3. Go with Curl, as it can work with a range of SSL libraries
> > 4. work with various SSL programs (wget, curl, etc)
> > 5. anything else?
> >
> > I suspect that the path of least resistance right now is to take
> > the first option:
> > - dual-life Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
> > - Configure would look for OpenSSL or LibreSSL
> > - presumably it would have to make sure it's a recent enough version
> > - possibly refuse to proceed with certain versions that are now
> > known to have serious security issues
> > - cope with a later version of OpenSSL that's incompatible
> >
> > We'd have to approach the current maintainers of those modules to
> > see if they're happy for them to be bundle, explaining the implications.
> >
> > Net-SSLeay's Makefile.PL has stuff to to look for OpenSSL and work
> > out compatibility. One possible path is that someone familiar with
> > that stuff works with someone familiar with metaconfig, so that
> > when installed with Perl, Configure is doing the heavy lifting,
> > but it can still be installed from CPAN with Makefile.PL doing its thing.
> >
> > This doesn't preclude people starting parallel efforts to experiment
> > with other approaches, but my sense is that more effort has been put
> > into Net::SSLeay + OpenSSL, so we should start off on those shoulders.
> >
> > There are a number of pieces to this pie, so I'm guessing this will
> > probably need a small group of people to work on it.
> >
> > I wonder if a good first step would be a proof-of-concept:
> > a. hack together a metaconfig unit based on Net::SSLeay's Makefile.PL
> > b. if Net::SSLeay builds and tests successfully, then install
> > Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
> > c. Contact the maintainers of those modules to sound them out on
> > openness to dual-lifing.
> > d. See if this can be made to work on Linux, *BSD, MacOS, and Windows.
> >
> > Going with Curl is appealing, but unless we've got someone prepared
> > to take that on, it feels more complex and less well-known than the
> > approach outlined above.
> >
> > Neil
>
>

* Neil Bowers <neilb@neilb.org> [2022-06-13 22:11:11 +0100]:

> I've tried to re-read all the different threads related to this,
> but Spark mail gets confused, and reading this all via?nntp.perl.org
> ain't user friendly either.
>
> The motives are, at least, the following:
> ?- almost all APIs are https these days
> ?- and increasingly all websites
> ?- it will enable CPAN.pm to do https on a fresh install
> ?(it has other options for https, but those might not be available)
>
> options:
> ?1. bundle modules, look for openssl (and poss other libs)
> ?2. bundle an SSL lib and modules such as Mbed TLS, WolfSSL
> ?3. Go with Curl, as it can work with a range of SSL libraries
> ?4. work with various SSL programs (wget, curl, etc)
> ?5. anything else?

As a user, concur that #1 seems to be the most reasonable, with the caveat
that IO::Socket::SSL provided by MAJOR packages be factored in somehow. I suspect
distros like Debian/Ubuntu would necessarily want to strip this out somehow
given they strip out things like perldoc into separate packages. But, knowing
that my opinion means nothing, I'd place much more emphasis on opinions pkg folks
from Debian, OpenBSD, FreeBSD, and pkgsrc (NetBSD), etc (I just can't name them
all out of ignorance other wise I would have). So my only request is to please
approach them somehow and get their opinions. I am thankful for the many and robust
distribution channels perl has, and I just think their opinions matter more
than (e.g.,) mine. And I think a lot of these problems are well understood,
if not solved outright.

Cheers,
Brett

>
> I suspect that the path of least resistance right now is to take
> the first option:
> ?- dual-life Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
> ?- Configure would look for OpenSSL or LibreSSL
> ?- presumably it would have to make sure it's a recent enough version
> ?- possibly refuse to proceed with certain versions that are now
> ?known to have serious security issues
> ?- cope with a later version of OpenSSL that's incompatible
>
> We'd have to approach the current maintainers of those modules to
> see if they're happy for them to be bundle, explaining the implications.
>
> Net-SSLeay's Makefile.PL has stuff to to look for OpenSSL and work
> out compatibility. One possible path is that someone familiar with
> that stuff works with someone familiar with metaconfig, so that
> when installed with Perl, Configure is doing the heavy lifting,
> but it can still be installed from CPAN with Makefile.PL doing its thing.
>
> This doesn't preclude people starting parallel efforts to experiment
> with other approaches, but my sense is that more effort has been put
> into Net::SSLeay + OpenSSL, so we should start off on those shoulders.
>
> There are a number of pieces to this pie, so I'm guessing this will
> probably need a small group of people to work on it.
>
> I wonder if a good first step would be a proof-of-concept:
> a. hack together a metaconfig unit based on Net::SSLeay's Makefile.PL
> b. if Net::SSLeay builds and tests successfully, then install
> ?? ?Net::SSLeay, IO::Socket::SSL, and Mozilla::CA
> c. Contact the maintainers of those modules to sound them out on
> ?? ?openness to dual-lifing.
> d. See if this can be made to work on Linux, *BSD, MacOS, and Windows.
>
> Going with Curl is appealing, but unless we've got someone prepared
> to take that on, it feels more complex and less well-known than the
> approach outlined above.
>
> Neil

--
--
oodler@cpan.org
oodler577@sdf-eu.org
SDF-EU Public Access UNIX System - http://sdfeu.org
irc.perl.org #openmp #pdl #native

On Tue, Jun 14, 2022 at 6:18 PM Elvin Aslanov <rwp.primary@gmail.com> wrote:

> yeah but `cpan Mozilla::CA` isn't hard to do to update the module and it
> won't break with newer Perl versions as well since it's just plaintext
> non-code certificates bundle
>

I'd prefer if the stack used the OS trusted CAs by default instead of
having its own list.
This should only be the default and overrideable for private CA use-cases.

On Wed, Jun 15, 2022 at 2:23 AM Alexander Hartmaier
<alex.hartmaier@gmail.com> wrote:
>
> On Tue, Jun 14, 2022 at 6:18 PM Elvin Aslanov <rwp.primary@gmail.com> wrote:
>>
>> yeah but `cpan Mozilla::CA` isn't hard to do to update the module and it won't break with newer Perl versions as well since it's just plaintext non-code certificates bundle
>
>
> I'd prefer if the stack used the OS trusted CAs by default instead of having its own list.
> This should only be the default and overrideable for private CA use-cases.

But that is a *massively* more difficult portability problem than just
"where do I find OpenSSL or LibreSSL?". Do you know where the OS
trusted CAs are for every platform and distribution on which Perl
runs? Or if there even is such a thing as an "OS trusted CA" on all of
them? Or what format they are in? Or whether they even exist on the
filesystem or are in some proprietary data store?

> On Jun 15, 2022, at 18:22, Craig A. Berry <craig.a.berry@gmail.com> wrote:
>
> On Wed, Jun 15, 2022 at 2:23 AM Alexander Hartmaier
> <alex.hartmaier@gmail.com> wrote:
>>
>> On Tue, Jun 14, 2022 at 6:18 PM Elvin Aslanov <rwp.primary@gmail.com> wrote:
>>>
>>> yeah but `cpan Mozilla::CA` isn't hard to do to update the module and it won't break with newer Perl versions as well since it's just plaintext non-code certificates bundle
>>
>>
>> I'd prefer if the stack used the OS trusted CAs by default instead of having its own list.
>> This should only be the default and overrideable for private CA use-cases.
>
> But that is a *massively* more difficult portability problem than just
> "where do I find OpenSSL or LibreSSL?". Do you know where the OS
> trusted CAs are for every platform and distribution on which Perl
> runs? Or if there even is such a thing as an "OS trusted CA" on all of
> them? Or what format they are in? Or whether they even exist on the
> filesystem or are in some proprietary data store?

Clarification: by “OS-trusted CAs” I believe Alexander refers specifically to OpenSSL’s roots, which are easily discoverable via the mechanism I mentioned earlier in this thread.

The idea is: if there’s OpenSSL, use it; if not, no out-of-the-box TLS.

-FG

2021-11-24 2:55 Neil Bowers <neilb@neilb.org> wrote:

> When you install Perl, although it comes with HTTP::Tiny, it doesn't
> support https because HTTP::Tiny requires IO::Socket::SSL (which isn’t
> bundled with Perl). This means that CPAN.pm and other tools can't make
> https mandatory, as when bootstrapping a new box you'll typically install
> perl then run cpan to first install Net::SSLeay and IO::Socket::SSL.
>
> I think we should make Perl support https "out of the box", so that
> CPAN.pm could make https mandatory, and other modules could then support
> https while relying only on core modules.
>
> One approach is to bundle Net::SSLeay and IO::Socket::SSL, but I don't
> know if that's the best / right solution?
>
> One problem with bundling these modules is that we are then committed to
> ensuring that they work on all supported platforms / configurations. Is it
> ok to support https out-of-the-box on some supported platforms but not all
> of them? I think so.
>
> Neil
>

My easy idea is to ask OS-distributors for including Net::SSLeay,
Mozilla::CA, and IO::Socket::SSL to the standard Perl package.

For example, ubuntu

# Current perl package doesn't contain Net::SSLeay, Mozilla::CA, and
IO::Socket::SSL
apt install perl

This is changed to:

# perl package contains Net::SSLeay, Mozilla::CA, and IO::Socket::SSL
apt install perl

# perl-nossl package doesn't contain Net::SSLeay, Mozilla::CA, and
IO::Socket::SSL
apt install perl-nossl

On Wed, Jun 15, 2022 at 7:38 PM Felipe Gasper <felipe@felipegasper.com> wrote:
>
>
>
> > On Jun 15, 2022, at 18:22, Craig A. Berry <craig.a.berry@gmail.com> wrote:
> >
> > On Wed, Jun 15, 2022 at 2:23 AM Alexander Hartmaier
> > <alex.hartmaier@gmail.com> wrote:
> >>
> >> On Tue, Jun 14, 2022 at 6:18 PM Elvin Aslanov <rwp.primary@gmail.com> wrote:
> >>>
> >>> yeah but `cpan Mozilla::CA` isn't hard to do to update the module and it won't break with newer Perl versions as well since it's just plaintext non-code certificates bundle
> >>
> >>
> >> I'd prefer if the stack used the OS trusted CAs by default instead of having its own list.
> >> This should only be the default and overrideable for private CA use-cases.
> >
> > But that is a *massively* more difficult portability problem than just
> > "where do I find OpenSSL or LibreSSL?".

> Clarification: by “OS-trusted CAs” I believe Alexander refers specifically to OpenSSL’s roots, which are easily discoverable via the mechanism I mentioned earlier in this thread.
>
> The idea is: if there’s OpenSSL, use it; if not, no out-of-the-box TLS.

As the StackOverflow article you cited makes clear, it is easy to
discover the directory where OpenSSL will look for certificates, but
what certificates are stored there has nothing to do with OpenSSL. If
the OpenSSL packager includes authoritative certificates and updates
them with package updates, that's certainly a convenient way to stay
current. But how does one tell that is the case? Someone may have
installed OpenSSL from source or from a packager that does not include
certs and have no certificates all or only self-signed or example
certificates.

> On Jun 16, 2022, at 09:46, Craig A. Berry <craig.a.berry@gmail.com> wrote:
>
> On Wed, Jun 15, 2022 at 7:38 PM Felipe Gasper <felipe@felipegasper.com> wrote:
>>
>>
>>
>>> On Jun 15, 2022, at 18:22, Craig A. Berry <craig.a.berry@gmail.com> wrote:
>>>
>>> On Wed, Jun 15, 2022 at 2:23 AM Alexander Hartmaier
>>> <alex.hartmaier@gmail.com> wrote:
>>>>
>>>> On Tue, Jun 14, 2022 at 6:18 PM Elvin Aslanov <rwp.primary@gmail.com> wrote:
>>>>>
>>>>> yeah but `cpan Mozilla::CA` isn't hard to do to update the module and it won't break with newer Perl versions as well since it's just plaintext non-code certificates bundle
>>>>
>>>>
>>>> I'd prefer if the stack used the OS trusted CAs by default instead of having its own list.
>>>> This should only be the default and overrideable for private CA use-cases.
>>>
>>> But that is a *massively* more difficult portability problem than just
>>> "where do I find OpenSSL or LibreSSL?".
>
>> Clarification: by “OS-trusted CAs” I believe Alexander refers specifically to OpenSSL’s roots, which are easily discoverable via the mechanism I mentioned earlier in this thread.
>>
>> The idea is: if there’s OpenSSL, use it; if not, no out-of-the-box TLS.
>
> As the StackOverflow article you cited makes clear, it is easy to
> discover the directory where OpenSSL will look for certificates, but
> what certificates are stored there has nothing to do with OpenSSL. If
> the OpenSSL packager includes authoritative certificates and updates
> them with package updates, that's certainly a convenient way to stay
> current. But how does one tell that is the case? Someone may have
> installed OpenSSL from source or from a packager that does not include
> certs and have no certificates all or only self-signed or example
> certificates.

If OpenSSL’s roots are old or invalid, then anything that uses that OpenSSL is buggy. Thus, I would think Perl doesn’t really need to care about how well the root store is maintained.

It might be good to assemble a list of how other languages solve this problem. I checked a few and started a gist here:
https://gist.github.com/FGasper/9ea9432409a4acf89fc206083cbae278

Given that node.js, PHP, and Python all seem to rely on OpenSSL’s root store, that path doesn’t seem like it would be unduly problematic for Perl.

-F

THIS IS A DUPLICATE RESPONSE. I HAD PREVIOUSLY SENT THE SAME THING UNDER AN
ALTERED SUBJECT LINE. REPEATED TO KEEP IT TOGETHER WITH THE MAIN THREAD.


I propose an alternate solution, summarized as bootstrapping.

Perl bundles an implementation of HTTPS support including certs or whatever, so
there is zero dependency on any HTTPS library or files being with the operating
system.

This bundled implementation is purposefully limited to only being able to invoke
CPAN or similar, that is, it has an inclusion list which is limited to CPAN
only, though users can add more to the inclusion list as a configuration if they
have to, and connections to anything not on the list is refused.

All CPAN clients bundled with Perl are set to use this bundled HTTPS support
instead of using HTTP.

Anyone wanting to use HTTPS in Perl for any purpose other than getting stuff
from CPAN has to download a regular traditional HTTPS support module from CPAN,
and then they would use that as they always have. These can then rely on the
system-provided HTTPS support and certs or whatever as they always have.

Given that the only thing the bundled HTTPS support has to do is talk to CPAN,
it can be a lot more minimalist than a typical HTTPS support, not needing extra
features than those cooperatively decided with the CPAN mirrors, and having no
dependencies besides Perl itself.

This bootstrapping would also prevent circular dependencies in the case of any
traditional HTTPS tools using Perl to build or install themselves.

I'm not advocating any specific HTTPS implementation to bundle a copy of, and
I'm not suggesting building a new one from scratch (bad idea for security
things), but we can pick some existing good implementation and strip it down
either physically or by way of configuration, and keep the clone in some other
namespace somehow so that its presence doesn't interfere with installing or
using the regular version of the same tool.

So do others agree or not that this is better?

-- Darren Duncan

On Thu, Jun 16, 2022 at 9:18 AM Felipe Gasper <felipe@felipegasper.com> wrote:
>
>
> > On Jun 16, 2022, at 09:46, Craig A. Berry <craig.a.berry@gmail.com> wrote:
> >
> > On Wed, Jun 15, 2022 at 7:38 PM Felipe Gasper <felipe@felipegasper.com> wrote:
> >>
> >>
> >>
> >>> On Jun 15, 2022, at 18:22, Craig A. Berry <craig.a.berry@gmail.com> wrote:
> >>>
> >>> On Wed, Jun 15, 2022 at 2:23 AM Alexander Hartmaier
> >>> <alex.hartmaier@gmail.com> wrote:
> >>>>
> >>>> On Tue, Jun 14, 2022 at 6:18 PM Elvin Aslanov <rwp.primary@gmail.com> wrote:
> >>>>>
> >>>>> yeah but `cpan Mozilla::CA` isn't hard to do to update the module and it won't break with newer Perl versions as well since it's just plaintext non-code certificates bundle
> >>>>
> >>>>
> >>>> I'd prefer if the stack used the OS trusted CAs by default instead of having its own list.
> >>>> This should only be the default and overrideable for private CA use-cases.
> >>>
> >>> But that is a *massively* more difficult portability problem than just
> >>> "where do I find OpenSSL or LibreSSL?".
> >
> >> Clarification: by “OS-trusted CAs” I believe Alexander refers specifically to OpenSSL’s roots, which are easily discoverable via the mechanism I mentioned earlier in this thread.
> >>
> >> The idea is: if there’s OpenSSL, use it; if not, no out-of-the-box TLS.
> >
> > As the StackOverflow article you cited makes clear, it is easy to
> > discover the directory where OpenSSL will look for certificates, but
> > what certificates are stored there has nothing to do with OpenSSL. If
> > the OpenSSL packager includes authoritative certificates and updates
> > them with package updates, that's certainly a convenient way to stay
> > current. But how does one tell that is the case? Someone may have
> > installed OpenSSL from source or from a packager that does not include
> > certs and have no certificates all or only self-signed or example
> > certificates.
>
> If OpenSSL’s roots are old or invalid, then anything that uses that OpenSSL is buggy. Thus, I would think Perl doesn’t really need to care about how well the root store is maintained.
>
> It might be good to assemble a list of how other languages solve this problem. I checked a few and started a gist here:
> https://gist.github.com/FGasper/9ea9432409a4acf89fc206083cbae278
>
> Given that node.js, PHP, and Python all seem to rely on OpenSSL’s root store, that path doesn’t seem like it would be unduly problematic for Perl.

Again, OpenSSL does not come with any certificates in its root store.
From the OpenSSL docs here:

https://www.openssl.org/docs/manmaster/man1/openssl-verification-options.html

>>> quote <<<
Note that OpenSSL does not provide a default set of trust anchors.
Many Linux distributions include a system default and configure
OpenSSL to point to that. Mozilla maintains an influential trust store
that can be found at
https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/.
>>> end quote <<<

So let's please not go with a Linux-only solution and just use
Mozilla::CA as already planned.

> On Jun 16, 2022, at 14:42, Craig A. Berry <craig.a.berry@gmail.com> wrote:
>
> On Thu, Jun 16, 2022 at 9:18 AM Felipe Gasper <felipe@felipegasper.com> wrote:
>>
>> If OpenSSL’s roots are old or invalid, then anything that uses that OpenSSL is buggy. Thus, I would think Perl doesn’t really need to care about how well the root store is maintained.
>>
>> It might be good to assemble a list of how other languages solve this problem. I checked a few and started a gist here:
>> https://gist.github.com/FGasper/9ea9432409a4acf89fc206083cbae278
>>
>> Given that node.js, PHP, and Python all seem to rely on OpenSSL’s root store, that path doesn’t seem like it would be unduly problematic for Perl.
>
> Again, OpenSSL does not come with any certificates in its root store.

I this is “mostly false”, actually.

The OpenSSL project doesn’t *provide* a root store, sure, but many (most?) production OpenSSL deployments do, in fact, “come with” a root store--it just happens not to be something that OpenSSL itself provides.

> So let's please not go with a Linux-only solution and just use
> Mozilla::CA as already planned.

How would using OpenSSL’s root certs be less “Linux-only” than using Net::SSLeay?

If we want a fully cross-platform solution, then bundling mbedTLS seems the only reasonable path. That doesn’t seem likely.

If, though, we’re going to use OpenSSL exclusively, we might as well prefer whatever roots it may include. It seems to work well enough for node.js, Python, and PHP--all of which are popular outside Linux.

A “hybrid” approach might be to ship a new module that implements a fallback mechanism:

1) Check for OpenSSL’s roots.
2) Check for Mozilla::CA.
3) Check list of hard-coded paths (steal from golang’s lists).

… or some other logic like checking mtimes or what not to determine a best root bundle.

-F

* Darren Duncan <darren@darrenduncan.net> [2022-06-16 07:58:18 -0700]:

> THIS IS A DUPLICATE RESPONSE. I HAD PREVIOUSLY SENT THE SAME THING UNDER AN
> ALTERED SUBJECT LINE. REPEATED TO KEEP IT TOGETHER WITH THE MAIN THREAD.
>
>
> I propose an alternate solution, summarized as bootstrapping.
>
> Perl bundles an implementation of HTTPS support including certs or whatever,
> so there is zero dependency on any HTTPS library or files being with the
> operating system.

The problem with this is one of expertise and available manpower. Which is
why I believe leveraging, if possible, what MAJOR distros already provide
might be the wiser option; again, if possible. This really has less to do
with *should* perl by default support SSL outside of distribution channel
support; I submit it does not. Not to mention SSL just makes things more
fragile no matter how you paint it, often for zero benefit.
>
> This bundled implementation is purposefully limited to only being able to
> invoke CPAN or similar, that is, it has an inclusion list which is limited
> to CPAN only, though users can add more to the inclusion list as a
> configuration if they have to, and connections to anything not on the list
> is refused.

I don't see how that'd be useful, frankly. This is just coming from my hat
as an application developer.

Now, when one builds perl from source on a platform (even if via perlbrew),
the build process subjects the system to a litany of "configure" checks. And
I think this is where the middle road may be. If, for example, a proper curl
or wget is available on the system, this would probably be sufficient to assume
some minimal support that is then useful for boot strapping. But if this is
not possible, skip anything related to SSL. This is already in place with the
necessary condition of there being gcc or autoconf (or dev/build tool stack)
available. No one is trying to figure out how to build perl fundamentally if
there is no proper C compiler available.

>
> All CPAN clients bundled with Perl are set to use this bundled HTTPS support
> instead of using HTTP.
>
> Anyone wanting to use HTTPS in Perl for any purpose other than getting stuff
> from CPAN has to download a regular traditional HTTPS support module from
> CPAN, and then they would use that as they always have. These can then rely
> on the system-provided HTTPS support and certs or whatever as they always
> have.

I find it to be a huge mistake to try to roll HTTPS support in isolation, and
the problems as mentioned before has to do with manpower. This is also not
a unique problem, as demonstrated with the fact that a C compiler (usually
gcc) is required to build perl from source at step 0.

>
> Given that the only thing the bundled HTTPS support has to do is talk to
> CPAN, it can be a lot more minimalist than a typical HTTPS support, not
> needing extra features than those cooperatively decided with the CPAN
> mirrors, and having no dependencies besides Perl itself.
>
> This bootstrapping would also prevent circular dependencies in the case of
> any traditional HTTPS tools using Perl to build or install themselves.
>
> I'm not advocating any specific HTTPS implementation to bundle a copy of,
> and I'm not suggesting building a new one from scratch (bad idea for
> security things), but we can pick some existing good implementation and
> strip it down either physically or by way of configuration, and keep the
> clone in some other namespace somehow so that its presence doesn't interfere
> with installing or using the regular version of the same tool.
>
> So do others agree or not that this is better?

I disagree for the reasons above. We should not be "rolling our own" here. Minimally
there should be a canary indicating "system supports SSL" checked during the
configuration check the predicates any source build.

For system supplied perl or perl installed via a package system (e.g., FreeBSD
provides a sliding window of perl version), then https support should also be
supported also through the package system.

What I don't know is, given a "curl" or "wget" that supports SSL present on the
system, but are the steps then the perl build could utilize to pull itself up
from ye old bootstraps.

Keep in mind, both curl and wget were greatly affected when letsencrypt yolo'd
(or didn't) this CA update; which really only revealed how fragile the whole
set up is, particularly when relying on a "free" CA provider that of course
everyone is going to use. I understand this has to do more with the system's CA
store, but again this just speaks more to how fragile the whole sham is rather
than how sound the potential to just using curl or wget would be.

If curl or wget is not installed? You get no https "out of the box" - just like
you get no perl if there is no gcc or working C compiler+dev tools.

Cheers,
Brett

>
> -- Darren Duncan
>

--
--
oodler@cpan.org
oodler577@sdf-eu.org
SDF-EU Public Access UNIX System - http://sdfeu.org
irc.perl.org #openmp #pdl #native

Op 16-06-2022 om 16:58 schreef Darren Duncan:
> THIS IS A DUPLICATE RESPONSE.  I HAD PREVIOUSLY SENT THE SAME THING
> UNDER AN ALTERED SUBJECT LINE.  REPEATED TO KEEP IT TOGETHER WITH THE
> MAIN THREAD.
>
>
> I propose an alternate solution, summarized as bootstrapping.
>
> Perl bundles an implementation of HTTPS support including certs or
> whatever, so there is zero dependency on any HTTPS library or files
> being with the operating system.
>

Including certs is problematic, certs change over time. I assume we want
a static tarball for any perl release, not one that changes over time.
So I think this is not a feasable idea. You could include a long lived
root cert and try to always use that on root cert, but I would not go
that way. If you would like to use a different root cert later, you
painted yourself in a corner.


HTH,

M4

On Thu, Jun 16, 2022 at 2:11 PM Felipe Gasper <felipe@felipegasper.com> wrote:
>
> > On Jun 16, 2022, at 14:42, Craig A. Berry <craig.a.berry@gmail.com> wrote:
> > So let's please not go with a Linux-only solution and just use
> > Mozilla::CA as already planned.
>
> How would using OpenSSL’s root certs be less “Linux-only” than using Net::SSLeay?

Both OpenSSL and Net::SSLeay are very portable. The Linuxy assumption
is that an OpenSSL package includes (or symlinks to) authoritative
certificates provided by the OS distributor. That is unlikely to be
the case on non-Linux. I don't know, but the BSDs very likely do
something similar with LibreSSL, though as far as I can find, the
authoritative certs included with BSD distros are just the same
Mozilla certs you'd get with Mozilla::CA. If the Mozilla certificates
are good enough for the BSDs and for curl, why wouldn't they be good
enough for us, especially since there is already a Perl-friendly way
to maintain them?

On Wed, 15 Jun 2022 02:52:01 +0000, Oodler 577 via perl5-porters wrote:

> > options:
> > ?1. bundle modules, look for openssl (and poss other libs)
> > ?2. bundle an SSL lib and modules such as Mbed TLS, WolfSSL
> > ?3. Go with Curl, as it can work with a range of SSL libraries
> > ?4. work with various SSL programs (wget, curl, etc)
> > ?5. anything else?
> As a user, concur that #1 seems to be the most reasonable, with the caveat
> that IO::Socket::SSL provided by MAJOR packages be factored in somehow. I suspect
> distros like Debian/Ubuntu would necessarily want to strip this out somehow
> given they strip out things like perldoc into separate packages. But, knowing
> that my opinion means nothing, I'd place much more emphasis on opinions pkg folks
> from Debian, OpenBSD, FreeBSD, and pkgsrc (NetBSD), etc

I can't speak for Niko and Dom, who are the maintainers of perl
itself in Debian (but they're subscribed here, so can add to my
comments), still some thought from a CPAN->Debian packager:

* Having TLS/https in perl sounds nice.
* We are not packaging Mozilla::CA, because we have the very same
certs in the ca-certificates package (and duplication, skew,
duplicate efforts, security updates, you name it).
When a CPAN dist requires Mozilla::CA we patch it, and I'm pretty
sure the same would happen if perl included Mozilla::CA
unconditionally.
So for us any solution which probes for CA certs or has a configure
option or whatever would be nice.
* This whole discussion about HTTPS is completely in vain, as long as
HTTP::Tiny doesn't verify the certificates.
Cf. https://github.com/chansen/p5-http-tiny/issues/134

Cheers,
gregor


--
.''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org
: :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06
`. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe

------- Original Message -------
On Thursday, June 16th, 2022 at 15:58, Darren Duncan <darren@darrenduncan.net> wrote:


> THIS IS A DUPLICATE RESPONSE. I HAD PREVIOUSLY SENT THE SAME THING UNDER AN
> ALTERED SUBJECT LINE. REPEATED TO KEEP IT TOGETHER WITH THE MAIN THREAD.

ALSO A DUPLICATED ANSWER :-)
>

> I propose an alternate solution, summarized as bootstrapping.
>
> Perl bundles an implementation of HTTPS support including certs or whatever, so
> there is zero dependency on any HTTPS library or files being with the operating
> system.


While I understand Darren proposal's goal, and at some point it makes sense, I do not think it is the solution more interesting to Perl developers and Perl itself.

This suggestion is almost like including an HTTP client that can just connect to CPAN, and if you want to connect to anywhere else, then download a module from there.

But this is not the case. Perl ships with a fairy good, although tiny, implementation of a HTTP Client. And nowadays, you almost do not need anything else to perform HTTP connections. Nevertheless, not having HTTPS support makes it, day by day, more useless without extra modules.

Thus, I do not think it is worth the effort of streamlining an SSL library to ship with code to just connect to one specific host.

Of course, just my 5 cents for the discussion :-)
Kindest regards,
Alberto (aka ambs)

> On Jun 16, 2022, at 18:19, Craig A. Berry <craig.a.berry@gmail.com> wrote:
>
> On Thu, Jun 16, 2022 at 2:11 PM Felipe Gasper <felipe@felipegasper.com> wrote:
>>
>>> On Jun 16, 2022, at 14:42, Craig A. Berry <craig.a.berry@gmail.com> wrote:
>>> So let's please not go with a Linux-only solution and just use
>>> Mozilla::CA as already planned.
>>
>> How would using OpenSSL’s root certs be less “Linux-only” than using Net::SSLeay?
>
> Both OpenSSL and Net::SSLeay are very portable. The Linuxy assumption
> is that an OpenSSL package includes (or symlinks to) authoritative
> certificates provided by the OS distributor. That is unlikely to be
> the case on non-Linux. I don't know, but the BSDs very likely do
> something similar with LibreSSL, though as far as I can find, the
> authoritative certs included with BSD distros are just the same
> Mozilla certs you'd get with Mozilla::CA. If the Mozilla certificates
> are good enough for the BSDs and for curl, why wouldn't they be good
> enough for us, especially since there is already a Perl-friendly way
> to maintain them?

My macOS, FreeBSD, and Cygwin installs all have roots at $OPENSSLDIR/cert.pem. That seems like an awfully wide swath of potential Perl users.

The problem with Mozilla::CA is that there’s too little motivation to keep the root store up-to-date. It’s a good solution for when we can’t find *any* root store, but if OpenSSL *tells* us where it keeps its roots, we might as well just latch onto whatever mechanism keeps OpenSSL updated. This appears to be the same mechanism that PHP, Python, and node.js all use … is it a problem for those folks, do we know?

Alternatively, ship a very-simple CPAN module that duplicates golang’s root-finding logic. There are separate paths per OS; look at “root_*.go” files under https://go.dev/src/crypto/x509/.

-FG

Felipe Gasperwrote:
>>> On Jun 16, 2022, at 14:42, Craig A. Berry <craig.a.berry@gmail.com> wrote:
>>>> So let's please not go with a Linux-only solution and just use
>>>> Mozilla::CA as already planned.
>
> My macOS, FreeBSD, and Cygwin installs all have roots at $OPENSSLDIR/cert.pem. [...]
>
> The problem with Mozilla::CA is [...]
>
> Alternatively, ship a very-simple CPAN module that duplicates golang’s root-finding logic.


Neil's proposal is to go with option 1 aka IO::Socket::SSL, which already has its own logic. It checks several paths, including the one you mentioned above, and when it fails to find a root store, it falls back to Mozilla::CA.

https://metacpan.org/pod/IO::Socket::SSL#Basic-SSL-Client
https://metacpan.org/pod/IO::Socket::SSL#IO::Socket::SSL::default_ca([-path|dir|-SSL_ca_file-=-...,-SSL_ca_path-=%3E-...-])%3E

What, exactly, would be the problem with this part of IO::Socket::SSL?


By the way, I haven't seen HTTP::Tinyish discussed yet. Michiel Beijen pointed this out last time around.

This small module seems to solve Neil's option 4 (work with wget, curl, etc) and might perhaps be the simplest solution here. HTTP::Tinyish is basically how cpanm does HTTPS.

Among other things, it might side-step the lack of a configure stage and it avoids having to deal with finding a CA root store because it just uses whatever logic wget/curl has. However, it depends on other non-core modules, namely File::Which and IPC::Run3. Additionally, we would need to consider systems that have neither wget nor curl.

https://www.nntp.perl.org/group/perl.perl5.porters/2021/12/msg262117.html
https://metacpan.org/pod/HTTP::Tinyish


--
Arne Johannessen
<https://arne.johannessen.de/>

> On Jun 17, 2022, at 10:48, Arne Johannessen <aj22@thaw.de> wrote:
>
> Felipe Gasperwrote:
>>>> On Jun 16, 2022, at 14:42, Craig A. Berry <craig.a.berry@gmail.com> wrote:
>>>>> So let's please not go with a Linux-only solution and just use
>>>>> Mozilla::CA as already planned.
>>
>> My macOS, FreeBSD, and Cygwin installs all have roots at $OPENSSLDIR/cert.pem. [...]
>>
>> The problem with Mozilla::CA is [...]
>>
>> Alternatively, ship a very-simple CPAN module that duplicates golang’s root-finding logic.
>
>
> Neil's proposal is to go with option 1 aka IO::Socket::SSL, which already has its own logic. It checks several paths, including the one you mentioned above, and when it fails to find a root store, it falls back to Mozilla::CA.
>
> https://metacpan.org/pod/IO::Socket::SSL#Basic-SSL-Client
> https://metacpan.org/pod/IO::Socket::SSL#IO::Socket::SSL::default_ca([-path|dir|-SSL_ca_file-=-...,-SSL_ca_path-=%3E-...-])%3E
>
> What, exactly, would be the problem with this part of IO::Socket::SSL?

Nothing at all; this is basically what I was proposing. I didn’t realize that IO::Socket::SSL does it already. Bravo.

-F

On Thu, Jun 16, 2022 at 2:38 AM Felipe Gasper <felipe@felipegasper.com>
wrote:

>
>
> > On Jun 15, 2022, at 18:22, Craig A. Berry <craig.a.berry@gmail.com>
> wrote:
> >
> > On Wed, Jun 15, 2022 at 2:23 AM Alexander Hartmaier
> > <alex.hartmaier@gmail.com> wrote:
> >>
> >> On Tue, Jun 14, 2022 at 6:18 PM Elvin Aslanov <rwp.primary@gmail.com>
> wrote:
> >>>
> >>> yeah but `cpan Mozilla::CA` isn't hard to do to update the module and
> it won't break with newer Perl versions as well since it's just plaintext
> non-code certificates bundle
> >>
> >>
> >> I'd prefer if the stack used the OS trusted CAs by default instead of
> having its own list.
> >> This should only be the default and overrideable for private CA
> use-cases.
> >
> > But that is a *massively* more difficult portability problem than just
> > "where do I find OpenSSL or LibreSSL?". Do you know where the OS
> > trusted CAs are for every platform and distribution on which Perl
> > runs? Or if there even is such a thing as an "OS trusted CA" on all of
> > them? Or what format they are in? Or whether they even exist on the
> > filesystem or are in some proprietary data store?
>
> Clarification: by “OS-trusted CAs” I believe Alexander refers specifically
> to OpenSSL’s roots, which are easily discoverable via the mechanism I
> mentioned earlier in this thread.
>
> The idea is: if there’s OpenSSL, use it; if not, no out-of-the-box TLS.
>
> -FG
>

I wasn't really talking about OpenSSL but Debian/Ubuntu's ca-certificates
package which puts the trusted root CA cert files in PEM format in
/etc/ssl/certs. I'm sure RHEL and derivates have such a directory as well.
For Windows I don't know how and where they are stored nor how to access
them, but the UI for managing them is launching mmc.exe and adding the
'Certificates' snapin.

That's what a regular user expects, just as usage of a system set proxy,
which is also a topic to make the HTTPS Internet access work.
That's with my 'large enterprise security engineer/architect' hat on.
FYI our current international client network can't even resolve Internet
DNS any more, just the proxy can.
There are also multiple internal root CAs that you have to trust, either
for internal https resource access or even man-in-the-middle TLS (please
don't call it SSL, that long dead) inspection.

I'd split the required parts up into (in order of importance, can also be
shortcut using configure switches if autodetection fails or isn't
implemented for a platform):
- TLS library setup
- trusted root CA certificate detection (different per OS, maybe even dist
specific)
- system proxy detection

I just stumbled over this site and think it's helpful when deciding version
requirements: https://repology.org

Cheers, Alex