Mailing List Archive

time to purge microperl
So microperl doesn't build on blead:

cc -c -o uav.o -DPERL_CORE -DPERL_MICRO -DPERL_USE_SAFE_PUTENV -DNO_MATHOMS av.c
In file included from iperlsys.h:50,
from perl.h:4047,
from av.c:20:
perlio.h:32:4: error: #error "stdio is no longer supported as the default base layer -- use perlio."
# error "stdio is no longer supported as the default base layer -- use perlio."
^~~~~

...

That #error was added in July 2015 by commit 786296d4896f80d4:

Evict PERLIO_IS_STDIO from top-level core files.

Configuring with perlio has been the only option since dd35fa16610
in 2011, first released in 5.16.0. Yet we have still have had all
this dead code for stdio support cluttering up the sources and
dulling the machetes of anyone who wants to do further work on
perlio or someone (like me) who just stumbles on some code that
looks like it needs fixing but isn't worth the time since the
compiler will never see it.

Leave a vestigial perlsdio.h since its presence is hard-coded in
a number of places.


OK, so it worked until then?

No. Turns out not. I did a bisect. Is was broken in 2013:
commit e2051532106d99a4928bf7d13194edd2ac1f7b1a (refs/bisect/bad)
Date: Fri Nov 1 20:12:53 2013 -0400

Move the function to set $^X to its own file

This also moves the indirect dependency on stdbool.h to its
own file, rather than being pulled in for all of perl.c, for
those cases where one may want to test using other definitions
of bool.

MANIFEST | 1 +
Makefile.SH | 4 +-
caretx.c | 128 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
embed.fnc | 2 +
embed.h | 1 +
perl.c | 87 +----------------------------------------
perl.h | 7 ----
proto.h | 1 +
8 files changed, 137 insertions(+), 94 deletions(-)


So microperl has been broken in every stable release since v5.20.0

This was actually reported in 2014, but it seems that the ticket stalled:

https://github.com/Perl/perl5/issues/14076


There are 48 references to PERL_MICRO, 43 in C code:

$ git grep PERL_MICRO *.c *.h | wc -l
43


Note that microperl can't actually deliver on the promise of a "configure
free perl" because you already have to pick a config for 32 vs 64 bit
pointers, and I suspect it will break on 64 bit Windows
(IIRC it's LLP64 - is sizeof(long) == 4 but sizeof(void *) == 8)


It's clear that no-one is using this, as it would have been fixed sometime
in the past 7 stable releases, instead of being abandoned.

The current approach is just leaving clutter that is getting in the way.
It should go.


If there are specific features that we should be able to build without,
they should go into their own #defines. We already have some like this -
see unixish.h and its functional closes:

/*
* The following symbols are defined if your operating system supports
* functions by that name. All Unixes I know of support them, thus they
* are not checked by the configuration script, but are directly defined
* here.
*/


Nicholas Clark
Re: time to purge microperl [ In reply to ]
On 9/14/21 8:55 AM, Nicholas Clark wrote:
> So microperl doesn't build on blead:
>
> [snip]

> It's clear that no-one is using this, as it would have been fixed sometime
> in the past 7 stable releases, instead of being abandoned.
>
> The current approach is just leaving clutter that is getting in the way.
> It should go.
>

I agree with you ... but there was an argument for partially fixing
microperl implemented in https://github.com/Perl/perl5/pull/18596

There was also a posting about microperl on this list in August of last
year:
https://www.nntp.perl.org/group/perl.perl5.porters/2020/08/msg258242.html

Do you have comments on those ideas?

Thank you very much.
Jim Keenan
Re: time to purge microperl [ In reply to ]
On Tue, Sep 14, 2021 at 09:11:30AM -0400, James E Keenan wrote:
> On 9/14/21 8:55 AM, Nicholas Clark wrote:
> > So microperl doesn't build on blead:
> >
> > [snip]
>
> > It's clear that no-one is using this, as it would have been fixed sometime
> > in the past 7 stable releases, instead of being abandoned.
> >
> > The current approach is just leaving clutter that is getting in the way.
> > It should go.
> >
>
> I agree with you ... but there was an argument for partially fixing
> microperl implemented in https://github.com/Perl/perl5/pull/18596
>
> There was also a posting about microperl on this list in August of last
> year:
> https://www.nntp.perl.org/group/perl.perl5.porters/2020/08/msg258242.html
>
> Do you have comments on those ideas?

The sensible question asked there:

"perhaps there might be some community interest"

got zero replies, so that seems pretty firmly zero interest.

We would not simplify the bootstrap process by trying to rewrite working
code from shell into perl. We'd just spend a lot of time adding bugs.

(And that's before we try to transcribe the existing hints files from shell
to Perl, or convert the VMS, Win32 etc build systems to Perl, and try to
unify what we can.)

The message that message references is more interesting:

The main reason I looked at it is it was there already and seemed like
an interesting base, but in hindsight it's a bit of a red herring, it
provides a very simple way to get something resembling perl running
(especially when cross-compiling) but as soon as you want to use
signals or various other features that aren't available (sometimes
failing in unobvious ways) you either have to customise it or switch
to a "real" perl.

https://www.nntp.perl.org/group/perl.perl5.porters/2012/08/msg191038.html

microperl isn't a solution for a better build system.



I'd missed that recent PR about it and the branch it references.

I also wasn't aware that the work has been done to get it working again.
Although the comment was:

I'm not a user of microperl, so I don't know what good it is, but it
compiles blazingly fast ( 6 seconds ), so I see a use when
modifying/testing regex changes for example.

In which case, it *might* be useful. However, with ccache I can rebuild
miniperl on a Rasbperry Pi in 8 seconds (after deleting all object files),
so we're pretty close to "blazingly fast" already.
(2 seconds on serious server hardware)

And then I can go on from there in that same source tree to build the
rest of perl and run the full regression tests.

I've never found the need for something smaller than miniperl.
Rebuilding miniperl (keeping config.h) solves most problems.

And for configuration problems, one is interested in getting Configure
to generate the correct config.sh and config.h for the system

(eg my recent fun of "dammit, how do I coax it to give the correct gcc flags
for C99 features for this ancient gcc?")


So I'm still not sure that we want to keep it. But if we do, we ought to

1) have the branch up to date and then merged
2) have a CI test for it working

and really

3) replace all the PERL_MICRO #defines with more granular HAS_FOO/HAS_BAR

(which really is part of the maintenance pain here - there's a bunch of
#if !defined THIS || defined PERL_MICRO
that really should be simplified to something like a microish.h that just
undefined/defines THIS THAT etc correctly.)


But all that seems to end up with is a "thing that builds" that no-one
really has a good use for. It's more of a pet than a beast of burden.

Nicholas Clark
Re: time to purge microperl [ In reply to ]
2021-9-14 21:56 Nicholas Clark <nick@ccl4.org> wrote:

> So microperl doesn't build on blead:
>
> cc -c -o uav.o -DPERL_CORE -DPERL_MICRO -DPERL_USE_SAFE_PUTENV
> -DNO_MATHOMS av.c
> In file included from iperlsys.h:50,
> from perl.h:4047,
> from av.c:20:
> perlio.h:32:4: error: #error "stdio is no longer supported as the default
> base layer -- use perlio."
> # error "stdio is no longer supported as the default base layer -- use
> perlio."
> ^~~~~
>
> ...
>
> That #error was added in July 2015 by commit 786296d4896f80d4:
>
> Evict PERLIO_IS_STDIO from top-level core files.
>
> Configuring with perlio has been the only option since dd35fa16610
> in 2011, first released in 5.16.0. Yet we have still have had all
> this dead code for stdio support cluttering up the sources and
> dulling the machetes of anyone who wants to do further work on
> perlio or someone (like me) who just stumbles on some code that
> looks like it needs fixing but isn't worth the time since the
> compiler will never see it.
>
> Leave a vestigial perlsdio.h since its presence is hard-coded in
> a number of places.
>
>
Is it possible for miniperl to use standard I/O after this purging?

Is only the Perl IO layer(built on standard I/O) affected?
Re: time to purge microperl [ In reply to ]
On Tue, Sep 14, 2021 at 7:12 PM Yuki Kimoto <kimoto.yuki@gmail.com> wrote:
>
> 2021-9-14 21:56 Nicholas Clark <nick@ccl4.org> wrote:
>>
>> So microperl doesn't build on blead:
>>
>> cc -c -o uav.o -DPERL_CORE -DPERL_MICRO -DPERL_USE_SAFE_PUTENV -DNO_MATHOMS av.c
>> In file included from iperlsys.h:50,
>> from perl.h:4047,
>> from av.c:20:
>> perlio.h:32:4: error: #error "stdio is no longer supported as the default base layer -- use perlio."
>> # error "stdio is no longer supported as the default base layer -- use perlio."
>> ^~~~~
>>
>> ...
>>
>> That #error was added in July 2015 by commit 786296d4896f80d4:
>>
>> Evict PERLIO_IS_STDIO from top-level core files.
>>
>> Configuring with perlio has been the only option since dd35fa16610
>> in 2011, first released in 5.16.0. Yet we have still have had all
>> this dead code for stdio support cluttering up the sources and
>> dulling the machetes of anyone who wants to do further work on
>> perlio or someone (like me) who just stumbles on some code that
>> looks like it needs fixing but isn't worth the time since the
>> compiler will never see it.
>>
>> Leave a vestigial perlsdio.h since its presence is hard-coded in
>> a number of places.
>>
>
> Is it possible for miniperl to use standard I/O after this purging?

No, and has not been for about ten years. And was not a good idea
since before that. And miniperl was no different from perl in this
regard.

> Is only the Perl IO layer(built on standard I/O) affected?

Yes. perlio layered on unix I/O (file descriptors, read()/write())
had been the preferred mechanism for many years but perlio layered on
stdio (FILE*, fread(), fwrite()) was still available as a
configuration option despite its numerous problems. It was removed as
a configuration option in 2011, and I removed some of the code made
redundant by that in 2015.

This thread, though, has nothing to do with miniperl. It is about
microperl, which apparently still depends on PERLIO_IS_STDIO, which I
somehow missed in 2015, having entirely forgotten about the existence
of microperl and having never used it. As Nicholas explained, there
does not seem to be much of a use case for microperl. Getting it to
work with unix I/O would probably not be that hard, but it was already
broken before I removed stdio as a bottom I/O layer, and why spend
time fixing something no one is using?

On a superficially-related note, the weather report in the U.S.
currently says, "the outer rain bands of Nicholas may produce a
tornado or two." So, if you've got any cruft you want to keep, nail
it down if you see Nicholas coming :-).
Re: time to purge microperl [ In reply to ]
2021-9-15 10:26 Craig A. Berry <craig.a.berry@gmail.com> wrote:

>
> >
> > Is it possible for miniperl to use standard I/O after this purging?
>
> No, and has not been for about ten years. And was not a good idea
> since before that. And miniperl was no different from perl in this
> regard.
>
>
Craig

Sorry, I heard microperl at first. I had thought about miniperl.
Re: time to purge microperl [ In reply to ]
On Tue, Sep 14, 2021 at 08:26:19PM -0500, Craig A. Berry wrote:

> does not seem to be much of a use case for microperl. Getting it to
> work with unix I/O would probably not be that hard, but it was already

so...


On Tue, Sep 14, 2021 at 01:41:04PM +0000, Nicholas Clark wrote:

> We would not simplify the bootstrap process by trying to rewrite working
> code from shell into perl. We'd just spend a lot of time adding bugs.

Correct me if I'm wrong, but I *think* that I'm the most recent person
to refactor chunks of the build system across all of *nix, VMS and Win32
Twice, actually:

* merging all the extension building code into make_ext.pl
Previously there was a shell version for *nix, a Perl script for Win32
and IIRC a third thing for VMS.
* changing miniperl so that it enabled a variant of sitecusomize.pl, and
using that to set @INC to the build tree

although this was all related to re-arranging ext (which in turn required
writing DCL for VMS...)

It's not as easy as it looks...

> I'd missed that recent PR about it and the branch it references.

... so

the branch is https://github.com/Corion/perl5/tree/corion/microperl


I think that this diff hunk from that branch sums up the problem:

@@ -37,6 +37,10 @@
#endif
#endif

+#ifdef PERL_MICRO
+# include <fcntl.h>
+#endif
+
#include <math.h>
#include <stdlib.h>


This is the rational way to get "unix I/O" working.
(Same change is also present in perlio.c)

I would have done the same - get it working, then refine it.

Without this include, the code won't compile due to about 6 different
constants needed for open() etc.

But with this include, it means that microperl now has a dependency on a
Unix-like environment, which (historically) wasn't the "plan" - if there
*was* a plan, it was as much working Perl as possible with just what is
available from ANSI C APIs.

To be clear - with this include, gcc is spitting out warnings about implicit
declarations of functions such as open() - basically everything common in
<unistd.h>. Functions which are present by default in the Unix C libraries.


(Aside - if anyone wants to replicate this, I had to hack the build with
ln -s uconfig.h config.h` as perl_langinfo.h includes "config.h", and the
branch didn't spot that one)


There's a microtest target:

$ make -f Makefile.micro microtest
perly.h is uptodate
perly.c is uptodate
cd t && (rm -f perl; ln -s ../microperl perl) \
&& ./perl TEST base/*.t cmd/*.t
t/base/cond ........ Unable to flush stdout: Bad file descriptor
Segmentation fault (core dumped)
make: [Makefile.micro:195: microtest] Error 139 (ignored)


(gdb) where
#0 Perl_my_popen (cmd=0x55c3a42aeae0 "./perl base/cond.t",
mode=0x7fff702b47fb "r") at util.c:2906
#1 0x000055c3a3c222d2 in Perl_do_open6 (gv=0x55c3a42ab0c8,
oname=0x55c3a42b63f0 "./perl base/cond.t |", len=18, supplied_fp=0x0,
svp=0x55c3a420dde8, num_svs=0) at doio.c:852
#2 0x000055c3a3d495b5 in Perl_pp_open () at pp_sys.c:639
#3 0x000055c3a3e01063 in Perl_runops_standard () at run.c:41
#4 0x000055c3a3cd88f1 in S_run_body (oldscope=1) at perl.c:2743
#5 0x000055c3a3cd84e7 in perl_run (my_perl=0x55c3a42082a0) at perl.c:2666
#6 0x000055c3a3c8ee7b in main (argc=16, argv=0x7fff702b4bf8,
env=0x7fff702b4c80) at miniperlmain.c:116

which valgrind reports like this:

==2184802== at 0x3CAD4F: Perl_my_popen (util.c:2898)
==2184802== by 0x14A2D1: Perl_do_open6 (doio.c:852)
==2184802== by 0x2715B4: Perl_pp_open (pp_sys.c:639)
==2184802== by 0x329062: Perl_runops_standard (run.c:41)
==2184802== by 0x2008F0: S_run_body (perl.c:2743)
==2184802== by 0x2004E6: perl_run (perl.c:2666)
==2184802== by 0x1B6E7A: main (miniperlmain.c:116)
==2184802==
==2184802== Syscall param close(fd) contains uninitialised byte(s)
==2184802== at 0x4A916C3: close (close.c:27)
==2184802== by 0x3CADA9: Perl_my_popen (util.c:2904)
==2184802== by 0x14A2D1: Perl_do_open6 (doio.c:852)
==2184802== by 0x2715B4: Perl_pp_open (pp_sys.c:639)
==2184802== by 0x329062: Perl_runops_standard (run.c:41)
==2184802== by 0x2008F0: S_run_body (perl.c:2743)
==2184802== by 0x2004E6: perl_run (perl.c:2666)
==2184802== by 0x1B6E7A: main (miniperlmain.c:116)
==2184802==
==2184802== Warning: invalid file descriptor 79101248 in syscall close()
==2184802== Conditional jump or move depends on uninitialised value(s)
==2184802== at 0x13B9B4: Perl_av_fetch (av.c:285)
==2184802== by 0x3CADCC: Perl_my_popen (util.c:2906)
==2184802== by 0x14A2D1: Perl_do_open6 (doio.c:852)
==2184802== by 0x2715B4: Perl_pp_open (pp_sys.c:639)
==2184802== by 0x329062: Perl_runops_standard (run.c:41)
==2184802== by 0x2008F0: S_run_body (perl.c:2743)
==2184802== by 0x2004E6: perl_run (perl.c:2666)
==2184802== by 0x1B6E7A: main (miniperlmain.c:116)
==2184802==
==2184802== Conditional jump or move depends on uninitialised value(s)
==2184802== at 0x13B9BB: Perl_av_fetch (av.c:286)
==2184802== by 0x3CADCC: Perl_my_popen (util.c:2906)
==2184802== by 0x14A2D1: Perl_do_open6 (doio.c:852)
==2184802== by 0x2715B4: Perl_pp_open (pp_sys.c:639)
==2184802== by 0x329062: Perl_runops_standard (run.c:41)
==2184802== by 0x2008F0: S_run_body (perl.c:2743)
==2184802== by 0x2004E6: perl_run (perl.c:2666)
==2184802== by 0x1B6E7A: main (miniperlmain.c:116)

(etc)

which (I think) is because there's no implementation of some of the pipe
open code (to get it to compile at all), but that C replacement doesn't
cleanly report that it failed to do anything, with the result that the
rest of the code attempts "keep calm and carry on" with ensuing chaos.


I don't think that there's any viable way to get (back to) the idea of
taking the existing working Perl source code and being able to compile
a portable, minimally functional, configure-free variant.

Nicholas Clark