Mailing List Archive

> On Dec 1, 2021, at 22:16, Darren Duncan <darren@darrenduncan.net> wrote:
>
> On 2021-12-01 5:54 p.m., Ricardo Signes wrote:
>> On Wed, Dec 1, 2021, at 8:08 PM, Yuki Kimoto wrote:
>>> Suppose the Perl core contains Net::SSLeay.
>>>
>>> In this case, Net::SSLeay is already installed without the OpenSSL library and the tests of Net::SSLeay are not yet done.
>>>
>>> How do we test for correctness of behavior of Net::SSLeay?
>> We would only build, test, and install Net::SSLeay if openssl was available.
>
> There is another important thing to consider in this discussion, which is the ability for core to use alternatives to OpenSSL.
>
> I have understood for years now that OpenSSL, unless I'm confusing it with something else, while widely used, also has serious quality issues.
>
> There are alternatives to OpenSSL which ostensibly are much better in quality control and are a fair bit tighter in focus and so are much less likely to suffer from enabling major security issues like HeartBleed and such.
>
> Historically installing modules to use SSL from CPAN meant users had a choice between OpenSSL and others.
>
> When adding SSL support to core, we should be doing it in such a way that better alternatives to OpenSSL can be used from core without OpenSSL or plain HTTP ever being needed to bootstrap that usage, as far as Perl is concerned.
>
> So the same logic for Perl core which tests the system and uses OpenSSL conditionally, it should also test the system for OpenSSL alternatives, and use them preferentially when we know they are better, or make it easy for the user to decide which to use.
>
> I am speaking primarily about what gets built-in at the start which is used when bootstrapping the toolchain, to enable downloading anything else from CPAN etc.

Toward this end: how feasible would it be to bundle something like mbedTLS--optionally?--with Perl in order to have out-of-the-box HTTPS?

-FG

* Felipe Gasper <felipe@felipegasper.com> [2021-12-01 23:52:23 -0500]:

>
> > On Dec 1, 2021, at 22:16, Darren Duncan <darren@darrenduncan.net> wrote:
> >
> > On 2021-12-01 5:54 p.m., Ricardo Signes wrote:
> >> On Wed, Dec 1, 2021, at 8:08 PM, Yuki Kimoto wrote:
> >>> Suppose the Perl core contains Net::SSLeay.
> >>>
> >>> In this case, Net::SSLeay is already installed without the OpenSSL library and the tests of Net::SSLeay are not yet done.
> >>>
> >>> How do we test for correctness of behavior of Net::SSLeay?
> >> We would only build, test, and install Net::SSLeay if openssl was available.
> >
> > There is another important thing to consider in this discussion, which is the ability for core to use alternatives to OpenSSL.
> >
> > I have understood for years now that OpenSSL, unless I'm confusing it with something else, while widely used, also has serious quality issues.
> >
> > There are alternatives to OpenSSL which ostensibly are much better in quality control and are a fair bit tighter in focus and so are much less likely to suffer from enabling major security issues like HeartBleed and such.
> >
> > Historically installing modules to use SSL from CPAN meant users had a choice between OpenSSL and others.
> >
> > When adding SSL support to core, we should be doing it in such a way that better alternatives to OpenSSL can be used from core without OpenSSL or plain HTTP ever being needed to bootstrap that usage, as far as Perl is concerned.
> >
> > So the same logic for Perl core which tests the system and uses OpenSSL conditionally, it should also test the system for OpenSSL alternatives, and use them preferentially when we know they are better, or make it easy for the user to decide which to use.
> >
> > I am speaking primarily about what gets built-in at the start which is used when bootstrapping the toolchain, to enable downloading anything else from CPAN etc.
>
> Toward this end: how feasible would it be to bundle something like mbedTLS--optionally?--with Perl in order to have out-of-the-box HTTPS?
>
> -FG

This seems to be all tending towards the problem native package
managers solve. And in this case, supporting "other" SSL should
probably fall something that Net::SSLeay would then be able to
use. Far be it from me to damper the conversation, but I have to
point out that this is just shifting the "problem" around and
even creating new ones. Not that these ideas do not contain merit;
but the origin pre-RFC of "supporting https" out of the box is
predicated on the practical fact that OpenSSL support on each
platform is very specific to the maintainers of each platform.

And we should be weary of adopting the same problems of making
sure this works on all the platforms supported by perl - even if
there is some kind of check of if "openssl is installed". I've
had fairly recent experiences of Net::SSeay/IO::Socket::SSL falling
behind new Ubuntus, as I have mentioned.

I also don't see much difference between "checking for openssl",
keeping in mind you don't need the binary, you need the development
libraries/headers; and checking to see if for a base or available
perl, the package for Net::SSeay exists (and if so, install it).

This really only works if using a base perl or a perl formally
supported in the platform's native package manager. Which is not
always straightforward since FreeBSD supports the different perls;
and as I type this I can't say which one, say the p5-Net-SSeay
port wants. They tend to be pretty good about using whichever one
is the "designated" perl (on FreeBSD), so it may be a moot point.

FWIW, outside of Debian/Ubuntu, OpenBSD is the other "major" platform
shipping with a base perl. Then there's NetBSD's pkgsrc, which
does support a lot of platforms (and is used as the native package
manager for some of them). But I don't think we want to be including
our own pkgsrc based dependency manager thingy either. Maybe I am
over thinking this, hopefully I am.

Cheers,
Brett

--
--
oodler@cpan.org
oodler577@sdf-eu.org
SDF-EU Public Access UNIX System - http://sdfeu.org
irc.perl.org #openmp #pdl #native

2021-12-2 12:17 Darren Duncan <darren@darrenduncan.net> wrote:

> On 2021-12-01 5:54 p.m., Ricardo Signes wrote:
> > On Wed, Dec 1, 2021, at 8:08 PM, Yuki Kimoto wrote:
> >> Suppose the Perl core contains Net::SSLeay.
> >>
> >> In this case, Net::SSLeay is already installed without the OpenSSL
> library and
> >> the tests of Net::SSLeay are not yet done.
> >>
> >> How do we test for correctness of behavior of Net::SSLeay?
> >
> > We would only build, test, and install Net::SSLeay if openssl was
> available.
>
> There is another important thing to consider in this discussion, which is
> the
> ability for core to use alternatives to OpenSSL.
>
>
------
https://metacpan.org/pod/Net::SSLeay

Net::SSLeay supports the following libssl implementations:

Any stable release of OpenSSL in the 0.9.8 - 1.1.1 branches, except for
OpenSSL 0.9.8 - 0.9.8b.

Any stable release of LibreSSL in the 2.0 - 3.1 series.
------

Current Net::SSLeay supports LibreSSL as an OpenSSL alternative .

I'm going to reïterate my question from earlier as I don't see anyone
having tackled it:

> I feel surely this isn't a problem that Perl uniquely is trying to
> solve. It feels like this should be easily solved by "doing what the
> other languages do" - i.e. could someone more familiar with some of
> the others report back on how e.g. Python or Ruby handle this kind of
> thing? If it works for them then surely we can do the same?

For example, for Python you can begin here:

https://github.com/python/cpython/blob/main/Lib/ssl.py

which soon leads to:

https://github.com/python/cpython/tree/main/Modules/_ssl.c

and suggests they're just using openssl:

/* Include OpenSSL header files */
#include "openssl/rsa.h"
...

There's also the "installation instructions" which point out that
libssl-dev is an (optional) build-time dependency:

https://devguide.python.org/setup/#install-dependencies

I suspect if that works for them, we can probably do similar.


Ruby, on the other hand, appears after a 30second search not to come
with SSL baked into the core, but can be added with an add-on module
("gem") in similar style to our current Perl attempt;

https://github.com/ruby/openssl


But theabove is all based on a mere 5 minutes of research from me - I'm
sure there'll be folks around who know more about the situation so can
chime in with extra detail. ;)

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

Hi,

On Fri, Dec 3, 2021 at 7:39 PM Paul "LeoNerd" Evans
<leonerd@leonerd.org.uk> wrote:
>
> But theabove is all based on a mere 5 minutes of research from me - I'm
> sure there'll be folks around who know more about the situation so can
> chime in with extra detail. ;)

In my opinion there are two scenarios:

- you use the system perl. that way you can use the system provided
tooling for bringing you ssl, so there is no problem!
- you use a perl you compiled yourself. that's great! this means you
had _some_ way to securely get perl on your system. and a C compiler.
you can use that same mechanism to get ssl there, so there is no
problem!

Let me also state that with just SSL libraries you're _not_ done. You
also need to get a trusted CA root in order to validate the
certificates! This needs to be updated frequently. That is something
really best to be left to the system Perl runs on to provide.

cpan libraries are on HTTPS mirrors, the cpan client uses curl if
IO::Socket::SSL etc are not available. I think that there is nothing
for perl itself to improve herel

--
Michiel

On Fri, Dec 3, 2021 at 11:05 PM Michiel Beijen <michiel.beijen@gmail.com> wrote:
> I think that there is nothing
> for perl itself to improve herel

Let me clarify myself: I think in the current situation, the
installation of CPAN modules using a new Perl is as good as it can be:
it defaults to https using the system curl if no other options are
available.

That said, there _are_ things to improve related to ssl/https support
in Perl in general:

Issuing HTTPS requests is common for any code. There is no core module
that can provide this for you. But then again, Perl was never the
'batteries included' language in the first place!
There is HTTP::Tiny but it has external dependencies for HTTPS.
Installation of IO::Socket::SSL with 'modern' tools will sometimes
fail, will not always very clearly explain the problem (or at least:
to me, and maybe mostly in the past). There is the Alien:: class of
modules which makes it easy to get & install C libraries with your
CPAN client. There is Alien::OpenSSL. Maybe this could be promoted
more widely as a solution for some people?

Alternatively, Python has binary modules called 'wheels' that can be
installed, that include pre-compiled libraries for the most commonly
used platforms, and their package manager pip will prefer using a
wheel if available, in the default configuration. Having such a thing
is great! But that would be a lot of work to achieve.

Please also note that HTTP::Tinyish exists, it's a pure-perl module
used by cpanm to enable HTTPS by falling back to curl or wget if
needed.

--
Michiel

Two other facts which seem relevant, that haven't been mentioned so far:

1) Perl is currently supported on many more platforms than OpenSSL is

2) OpenSSL requires perl during its build process

For both of these reasons, core perl *cannot* require OpenSSL.

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

2021-12-9 4:12 Paul "LeoNerd" Evans <leonerd@leonerd.org.uk> wrote:

> Two other facts which seem relevant, that haven't been mentioned so far:
>
> 1) Perl is currently supported on many more platforms than OpenSSL is
>
> 2) OpenSSL requires perl during its build process
>
> For both of these reasons, core perl *cannot* require OpenSSL.
>
>
For these reasons, in many cases what Perl users need to do for
SSL/TLS support is install IO::Socket::SSL(this depends Net::SSLeay).

This needs one action. However, it's easy.

If the distribution contains IO::Socket::SSL, Net::SSLeay and OpenSSL
libraries, users use HTTPS from the beginning(For example, Strawberry Perl)

On the other hand, I feel managing the SSL library and CA cert(like
Mozilla::CA provides) by ourselves is too risky and costly(although I want
to access HTTPS by default)

On Wed, Dec 8, 2021 at 1:12 PM Paul "LeoNerd" Evans
<leonerd@leonerd.org.uk> wrote:
>
> Two other facts which seem relevant, that haven't been mentioned so far:
>
> 1) Perl is currently supported on many more platforms than OpenSSL is
>
> 2) OpenSSL requires perl during its build process
>
> For both of these reasons, core perl *cannot* require OpenSSL.

It cannot require it but it can provide Perl-level support for TLS if
the underlying capability is available. This has all been covered in
the original thread Neil started. Obviously it must still be possible
to build Perl without SSL support via Configure-time mechanisms that
exclude Net::SSLeay and IO::Socket::SSL (and Mozilla::CA) when the
underlying library support (LibreSSL or OpenSSL) is not available. As
I've previously mentioned, that's more difficult than similar
Configure-time questions such as "does the GDBM library exist?" but I
don't think it's impossible.

If you can build a Perl without TLS support, then you can build
OpenSSL on any platform it supports, because OpenSSL does not require
an SSL-enabled Perl to build. Pretty much any old Perl will do. So
your #2 just goes away and has nothing to do with what core Perl can
or should do.

As far as your #1, platforms that do not support OpenSSL in the sense
of being able to build OpenSSL on the platform are few and far
between. What platforms does Perl support that do not support
OpenSSL? If by "support" you mean installed by default, that's a
different matter. Notably Windows and macOS do not have OpenSSL nor
LibreSSL present by default, so it cannot be relied on to be available
and the OS vendor does not supply a standard mechanism to get it.
There are other ways to get it, and people who build Perl installers
for those platforms may want to ensure that it's available.

As others have mentioned, CPAN.pm will use TLS support via curl or
wget if Perl module support is not available and those utilities have
the relevant capabilities. That will be completely adequate in many
cases, such as some hypothetical Linux distro that has a curl enabled
with gnuTLS but no OpenSSL nor LibreSSL included.

I don't think it is possible near-term to guarantee that every Perl
installation has TLS 1.2+ support with certificate verification. But
it seems feasible to greatly narrow the impact of the CVEs by ensuring
that support via Perl modules is available if the libraries upon which
those modules depend are available.

There are plenty of challenges and no perfect solutions, such as the
same chicken-and-egg problem everyone has of needing to update your
certificate store using a certificate that may have been revoked but
you won't know until after you've updated your certificate store. But
is there any reason other than it's hard work that Perl shouldn't
include Net::SSLeay, IO::Socket::SSL, and Mozilla::CA in the core and
build them whenever the underlying support is available?

> On Dec 8, 2021, at 22:42, Craig A. Berry <craig.a.berry@gmail.com> wrote:
>
> But
> is there any reason other than it's hard work that Perl shouldn't
> include Net::SSLeay, IO::Socket::SSL, and Mozilla::CA in the core and
> build them whenever the underlying support is available?

It occurs to me (as I’ve been learning mbedTLS) is that IO::Socket::SSL explicitly wraps Net::SSLeay/OpenSSL/LibreSSL.

ISTM a more ideal pattern would be some library-agnostic mechanism that doesn’t privilege one TLS library over another. Curl, for example, can use OpenSSL, GnuTLS, wolfSSL, NSS, mbedTLS, Secure Transport, or any of several others.

I know of no such abstraction on CPAN now, in December 2021, but I thought it still worth mentioning.

-FG

On Wed, Dec 8, 2021 at 10:12 PM Felipe Gasper <felipe@felipegasper.com> wrote:
>
> > On Dec 8, 2021, at 22:42, Craig A. Berry <craig.a.berry@gmail.com> wrote:
> >
> > But
> > is there any reason other than it's hard work that Perl shouldn't
> > include Net::SSLeay, IO::Socket::SSL, and Mozilla::CA in the core and
> > build them whenever the underlying support is available?
>
> It occurs to me (as I’ve been learning mbedTLS) is that IO::Socket::SSL explicitly wraps Net::SSLeay/OpenSSL/LibreSSL.
>
> ISTM a more ideal pattern would be some library-agnostic mechanism that doesn’t privilege one TLS library over another. Curl, for example, can use OpenSSL, GnuTLS, wolfSSL, NSS, mbedTLS, Secure Transport, or any of several others.
>
> I know of no such abstraction on CPAN now, in December 2021, but I thought it still worth mentioning.

Broader support would be ideal, but hard to justify as in scope for
Perl core development. The good news is that anyone interested is
free to explore such support on CPAN, possibly via new modules or
possibly via enhancements to existing ones. Can Net::SSLeay be taught
about other TLS libraries? I don't know; I suspect the only reason
LibreSSL works is that it's just a fork of OpenSSL with all the
non-Linux support gleefully chopped out, but that doesn't mean it's
impossible. Or maybe IO::Socket::SSL is the more appropriate place to
plug in different lower-level modules that support different TLS
libraries. Or maybe HTTP::Tiny could have support for Net::Curl
shoehorned into it, but then it wouldn't be that tiny and would add
libcurl as a dependency.

There are numerous non-trivial development efforts that could broaden
TLS support to different implementations. But if Perl core does
anything near-term, I think it needs to focus on bundling what's
already available. Just my $0.02.

On 12/9/21 9:40 AM, Craig A. Berry wrote:
> [snip] I don't know; I suspect the only reason
> LibreSSL works is that it's just a fork of OpenSSL with all the
> non-Linux support gleefully chopped out, but that doesn't mean it's
> impossible.

Small factual points:

LibreSSL originated in, and is developed by, OpenBSD. See:
http://www.libressl.org/

It is available in FreeBSD. See:
https://wiki.freebsd.org/LibreSSL

jimk

On Thu, Dec 09, 2021 at 08:40:21AM -0600, Craig A. Berry wrote:
<SNIP>
> Or maybe HTTP::Tiny could have support for Net::Curl shoehorned into
> it, but then it wouldn't be that tiny and would add libcurl as a
> dependency.
>
> There are numerous non-trivial development efforts that could broaden
> TLS support to different implementations. But if Perl core does
> anything near-term, I think it needs to focus on bundling what's
> already available. Just my $0.02.

HTTP::Tinyish[1] already wraps the HTTP::Tiny interface over several
implementations like LWP, curl, and wget. I have a hack[2] to make
HTTP::Tiny use the LibreSSL nc that has tls support[3] (some features
already supported) that could be added to Tinyish.

[1] https://metacpan.org/pod/HTTP::Tinyish
[2] https://gist.github.com/afresh1/c4ac35cdd29e6b2cd15eef38246c4c71
[3] https://man.openbsd.org/nc.1#c

l8rZ,
--
andrew

Beta. Software undergoes beta testing shortly before it's released.
Beta is Latin for "still doesn't work."

* Craig A. Berry [09/12/2021 08:40] :
>
> I don't know; I suspect the only reason
> LibreSSL works is that it's just a fork of OpenSSL with all the
> non-Linux support gleefully chopped out, but that doesn't mean it's
> impossible.

As James has already said, LibreSSL was created by the BSD community
and has, in fact, not seen much support in Linux-based distributions:
https://lwn.net/Articles/841664/

Emmanuel

On Thu, Dec 9, 2021 at 2:25 PM Emmanuel Seyman <emmanuel@seyman.fr> wrote:
>
> * Craig A. Berry [09/12/2021 08:40] :
> >
> > I don't know; I suspect the only reason
> > LibreSSL works is that it's just a fork of OpenSSL with all the
> > non-Linux support gleefully chopped out, but that doesn't mean it's
> > impossible.
>
> As James has already said, LibreSSL was created by the BSD community
> and has, in fact, not seen much support in Linux-based distributions:
> https://lwn.net/Articles/841664/

Sorry, I remembered the commit messages about taking a chainsaw to
OpenSSL's portability features when the work started but had forgotten
who wrote them :-).

On Wed, Dec 08, 2021 at 09:42:47PM -0600, Craig A. Berry wrote:
> On Wed, Dec 8, 2021 at 1:12 PM Paul "LeoNerd" Evans
> <leonerd@leonerd.org.uk> wrote:
> >
> > Two other facts which seem relevant, that haven't been mentioned so far:
> >
> > 1) Perl is currently supported on many more platforms than OpenSSL is
> >
> > 2) OpenSSL requires perl during its build process
> >
> > For both of these reasons, core perl *cannot* require OpenSSL.
>
> It cannot require it but it can provide Perl-level support for TLS if
> the underlying capability is available. This has all been covered in
> the original thread Neil started. Obviously it must still be possible

...

this, because this:

> I don't think it is possible near-term to guarantee that every Perl
> installation has TLS 1.2+ support with certificate verification. But
> it seems feasible to greatly narrow the impact of the CVEs by ensuring
> that support via Perl modules is available if the libraries upon which
> those modules depend are available.
>
> There are plenty of challenges and no perfect solutions, such as the
> same chicken-and-egg problem everyone has of needing to update your
> certificate store using a certificate that may have been revoked but
> you won't know until after you've updated your certificate store. But
> is there any reason other than it's hard work that Perl shouldn't
> include Net::SSLeay, IO::Socket::SSL, and Mozilla::CA in the core and
> build them whenever the underlying support is available?

"other than hard work" - no (other) reasons are obvious to me.

*but*

both that list and the other mentions of "but use curl or wget" seem to
forget Windows. Which I admit that I don't know that much about, but my
list of "known unknowns" is already:

1) What is the OS default certificate store on Windows (that we can assume
the OS handles updates and revocations), and what module is needed to
read it?

2) Are curl and wget likely to be found on Win32? Is there any command line
tool that a HTTP::TIMTOWTDI module can shell out to, to deal with https?


other question was

*) for bootstrapping how many (well, few) certificates do we need to ship?
and how often do these change?


In that if the minimal bootstrap only defaults to https://www.cpan.org/
(where that's a CDN which is up, and going down is a newsworthy catastrophe)
then surely it's sufficient to locally know only the root certificate that
signed www.cpan.org? And shipping and tracking that is a much smaller
problem than all the crap that web browsers currently accept as trustworthy.

We don't need our bootstrapping https to be able to verify the identity of
arbitrary sites that might happen to us some unusual root certificate. Just
one sane site that we also (roughly) control. (Or at least talk to)

Nicholas Clark

On Fri, 10 Dec 2021 at 18:37, Nicholas Clark <nick@ccl4.org> wrote:

> both that list and the other mentions of "but use curl or wget" seem to
> forget Windows. Which I admit that I don't know that much about, but my
> list of "known unknowns" is already:
>

Given the existence of Strawberry Perl and the Linux subsystem, building
Perl from scratch on Windows seems more like an advanced-user, "you know
what you're doing already" situation?

Also, if CPAN.pm already use wget/curl if available, and the target
objective is more-secure-by-default CPAN installation, seems the only
necessary step for core is perhaps to add some CPAN.pm tweaks: refuse
HTTP-only downloads by default (needs an override otherwise we make life
harder for situations where caching is desirable).

On Fri, Dec 10, 2021 at 07:02:15PM +0800, Tom Molesworth via perl5-porters wrote:
> On Fri, 10 Dec 2021 at 18:37, Nicholas Clark <nick@ccl4.org> wrote:
>
> > both that list and the other mentions of "but use curl or wget" seem to
> > forget Windows. Which I admit that I don't know that much about, but my
> > list of "known unknowns" is already:
> >
>
> Given the existence of Strawberry Perl and the Linux subsystem, building
> Perl from scratch on Windows seems more like an advanced-user, "you know
> what you're doing already" situation?

Possibly. But it feels wrong to immediately treat Windows for this as
"Somebody Else's Problem" by effectively saying "we solved this for OS/X and
most *nix, but for Win32 you need to get your source from Strawberry".
If Strawberry can find a solution for this, why is it downstream, when we
think it's worth being upstream for "real" operating systems?

> Also, if CPAN.pm already use wget/curl if available, and the target
> objective is more-secure-by-default CPAN installation, seems the only
> necessary step for core is perhaps to add some CPAN.pm tweaks: refuse
> HTTP-only downloads by default (needs an override otherwise we make life
> harder for situations where caching is desirable).

This seems sensible.


But having seen how OS distributions take the hack-of-least-resistance
combined with *how* stripped down Docker base images are, I wonder if
enough OSes will opt to flip this flag, rather than have perl depend on
more packages - ie they want to ship a "working" perl, and that's easier
by their packaged perl enabling http downloads, than by adding a dependency
from perl to curl or wget binaries.

Or, they don't do either. And then their base image install seems just to
just be broken when end users try to use it to follow instructions published
on the Internet. They might do this because they decide that the system is
insecure if command line tools such as wget are shipped in the "base" image,
but have to ship some sort of "perl" in their "base" image (because other
tooling depends on it), so ship with our defaults and theirs, and CPAN is
broken as standard.


We've had "fun" in the past with OS distributions shipping core perl
packages with some modules removed (notably CGI.pm, *before* *we* removed
it), and even adding new command line flags to the perl binary. That's stuff
that should have been obviously wrong - so obvious that we didn't document
that as "please, no, just don't". So I'm wary that designing something with
a low effort escape hatch intended for individuals who *have* read some
documentation and made a choice, makes it easy for OS package maintainers to
make that (risky) choice for users but then hide it from them.

Nicholas Clark

On 10/12/21 11:37, Nicholas Clark wrote:

> 2) Are curl and wget likely to be found on Win32? Is there any command line
> tool that a HTTP::TIMTOWTDI module can shell out to, to deal with https?
>

This works for me on Windows 10:

bitsadmin /rawreturn /transfer perl /download ^
https://www.cpan.org/src/5.0/perl-5.34.0.tar.gz ^
c:\users\salva\perl2.tgz


Also, urlmon.dll provides URLDownloadToFile:

$d = Win32::API->new(urlmon => URLDownloadToFileA => 'NPPNN', 'N');
$d->Call(0, "https://www.cpan.org/src/5.0/perl-5.34.0.tar.gz",
'c:\users\salva\perl3.tgz', 0,0);

On Fri, Dec 10, 2021 at 6:44 AM Salvador Fandiño <sfandino@gmail.com> wrote:
>
> On 10/12/21 11:37, Nicholas Clark wrote:
>
> > 2) Are curl and wget likely to be found on Win32? Is there any command line
> > tool that a HTTP::TIMTOWTDI module can shell out to, to deal with https?
> >
>
> This works for me on Windows 10:
>
> bitsadmin /rawreturn /transfer perl /download ^
> https://www.cpan.org/src/5.0/perl-5.34.0.tar.gz ^
> c:\users\salva\perl2.tgz
>
>
> Also, urlmon.dll provides URLDownloadToFile:
>
> $d = Win32::API->new(urlmon => URLDownloadToFileA => 'NPPNN', 'N');
> $d->Call(0, "https://www.cpan.org/src/5.0/perl-5.34.0.tar.gz",
> 'c:\users\salva\perl3.tgz', 0,0);
>

Those look promising. It appears that PowerShell's Invoke-WebRequest,
Invoke-RestMethod, and Start-BitsTransfer could all do the job as
well.

Very recent versions of Windows include OpenSSH, but I don't think
it's enabled by default, so sftp is probably not a practical
alternative (if that's even enabled on the server side).

I like the urlmon solution since it doesn't even involve spawning an
external command, but Win32::API is not in core, is it? So bitsadmin
might be the simplest and most reliable solution.

On Fri, 10 Dec 2021 10:37:19 +0000
Nicholas Clark <nick@ccl4.org> wrote:

> 1) What is the OS default certificate store on Windows (that we can assume
> the OS handles updates and revocations), and what module is needed to
> read it?

OpenSSL can't make use of it, so users rely on the certificates provided
by Mozilla::CA.

> 2) Are curl and wget likely to be found on Win32? Is there any command line
> tool that a HTTP::TIMTOWTDI module can shell out to, to deal with https?

Windows ships with curl.exe since Windows 10 1803.

>
> other question was
>
> *) for bootstrapping how many (well, few) certificates do we need to ship?
> and how often do these change?

Mozilla::CA is the module that provides SSL certificates. Unfortunately
it's almost always outdated so ideally we should get co-maint on it and
automate its updates.

On Fri, 10 Dec 2021 17:45:29 +0100
Tomasz Konojacki <me@xenu.pl> wrote:

> On Fri, 10 Dec 2021 10:37:19 +0000
> Nicholas Clark <nick@ccl4.org> wrote:
>
> > 2) Are curl and wget likely to be found on Win32? Is there any command line
> > tool that a HTTP::TIMTOWTDI module can shell out to, to deal with https?
>
> Windows ships with curl.exe since Windows 10 1803.

PS. BSDs are a bigger problem that Windows. While CPAN.pm can fall back
to curl/wget, neither of them are available there OOTB, at least on
FreeBSD and DragonFly BSD. They provide "fetch" command instead.

On Fri, Dec 10, 2021 at 05:54:16PM +0100, Tomasz Konojacki wrote:
> On Fri, 10 Dec 2021 17:45:29 +0100
> Tomasz Konojacki <me@xenu.pl> wrote:
>
> > On Fri, 10 Dec 2021 10:37:19 +0000
> > Nicholas Clark <nick@ccl4.org> wrote:
> >
> > > 2) Are curl and wget likely to be found on Win32? Is there any command line
> > > tool that a HTTP::TIMTOWTDI module can shell out to, to deal with https?
> >
> > Windows ships with curl.exe since Windows 10 1803.
>
> PS. BSDs are a bigger problem that Windows. While CPAN.pm can fall back
> to curl/wget, neither of them are available there OOTB, at least on
> FreeBSD and DragonFly BSD. They provide "fetch" command instead.

OpenBSD has ftp(1)[1] that grew https GET support as well as shipping
(obviously?) with LibreSSL and its nc(1) with -c (TLS)[2] support. As I
said in another post, I have a hack that uses that for https in
HTTP::Tiny as a "socket" interface so supports any method.

I think if there is a module that provides a front-end with support for
using different backends you will find folks willing to provide patches
to make it work with them.

(AFAIK FreeBSD's fetch(1) only supports GET as well, which is likely
fine but the solution should have a way for backends to say which http
methods they support)

[1] https://man.openbsd.org/ftp.1
[2] https://man.openbsd.org/nc.1#c
[3] https://gist.github.com/afresh1/c4ac35cdd29e6b2cd15eef38246c4c71

l8rZ,
--
andrew

Mountain Dew and doughnuts...
because breakfast is the most important meal of the day.

On Fri, Dec 10, 2021 at 10:45 AM Tomasz Konojacki <me@xenu.pl> wrote:
>
> On Fri, 10 Dec 2021 10:37:19 +0000
> Nicholas Clark <nick@ccl4.org> wrote:

> > other question was
> >
> > *) for bootstrapping how many (well, few) certificates do we need to ship?

Is it any easier to ship a certificate file with one certificate
rather than 100? I'm no expert but I believe root certs typically
have many-year expiration lifecycles, much longer than the specific
domain certs created from them, so there could be advantages to using
a standard bundle of root certs even for bootstrapping purposes.
Specifically, if we ship Perl with one and only one cert for cpan.org
and it happens to expire two weeks after a Perl release ships, how
does that one cert get updated before expiration? Root certs do
eventually expire and revocations are always possible, but
unpleasantness that is rare is preferable to unpleasantness that is
more frequent.

Also, if we include a cert file that only has one certificate in it
because that's all we need for bootstrapping CPAN.pm, doesn't that
potentially break other uses of the same toolchain
(Net::SSLeay/IO::Socket::SSL), such as folks scraping random parts of
the web or whatever? Or we have to use a differently-named cert file
for bootstrapping and make the tools find it.

> > and how often do these change?

Looks like every month or two here:

<https://curl.se/docs/caextract.html>

although that may just be how often curl keeps up and not how often
they actually change.

> Mozilla::CA is the module that provides SSL certificates. Unfortunately
> it's almost always outdated so ideally we should get co-maint on it and
> automate its updates.

It certainly looks automatable as it already has a script to produce a
new release when the cacert.pem file changes. How does the end user
get the updated version, though? Would CPAN.pm have to always update
Mozilla::CA even if it's not in the dependency chain of whatever is
being installed?