Mailing List Archive

On Fri, Aug 13, 2021 at 12:19 AM Neil Bowers <neilb@neilb.org> wrote:

> Back in 2012, Steffen Mueller did some experimenting and found that taint
> adds somewhere between 10% and 20% runtime overhead:
> https://www.nntp.perl.org/group/perl.perl5.porters/2012/10/msg193822.html
> As you can see, there was some discussion at that time, but it didn’t seem
> to go anywhere. This has come up again as a result of the Quirks document,
> and we discussed it in last week’s PSC meeting.
>
> Anecdotally, very few people use taint (these days), yet we’re all paying
> the price. Furthermore, taint causes problems on Windows. For example,
> File::Spec is broken on Windows when used with Taint mode on, because Taint
> mode restricts use of environment variables (which doesn't protect
> anything). See also this reddit discussion[1], on the problems with taint.
>
> We’d like to consider adding a Configure option for disabling taint mode.
> We see this as a potential first step to having this disabled by default,
> and then possibly removing support for taint entirely.
>
> We’re interested in hearing thoughts on this.
>
> Neil
>
> [1]
> https://www.reddit.com/r/perl6/comments/718z4o/taint_mode_for_perl_6/dnmu83i/
>

Using -DNO_TAINT_SUPPORT as Steffen mentioned you can already build such a
perl. All we really need to do is add a Configure option so that it can
easily be disabled and so that code (especially tests) can keep it into
account.

One open question is if SILENT_NO_TAINT_SUPPORT should be enabled along
with it. I would argue it should, without it many tests will refuse to even
start running.

Leon

On Fri, 13 Aug 2021 00:27:47 +0200
Leon Timmermans <fawaka@gmail.com> wrote:

> Using -DNO_TAINT_SUPPORT as Steffen mentioned you can already build such a
> perl. All we really need to do is add a Configure option so that it can
> easily be disabled and so that code (especially tests) can keep it into
> account.
>
> One open question is if SILENT_NO_TAINT_SUPPORT should be enabled along
> with it. I would argue it should, without it many tests will refuse to even
> start running.

SILENT_NO_TAINT_SUPPORT is much better, there's way too much code with
-T in the wild.

Anyway, I really wish we completely removed that misfeature, but if we
can't, adding it as an option to Configure is the next best thing.

On Fri, Aug 13, 2021 at 12:27 AM Leon Timmermans <fawaka@gmail.com> wrote:

> On Fri, Aug 13, 2021 at 12:19 AM Neil Bowers <neilb@neilb.org> wrote:
>
>> Back in 2012, Steffen Mueller did some experimenting and found that taint
>> adds somewhere between 10% and 20% runtime overhead:
>> https://www.nntp.perl.org/group/perl.perl5.porters/2012/10/msg193822.html
>> As you can see, there was some discussion at that time, but it didn’t
>> seem to go anywhere. This has come up again as a result of the Quirks
>> document, and we discussed it in last week’s PSC meeting.
>>
>> Anecdotally, very few people use taint (these days), yet we’re all paying
>> the price. Furthermore, taint causes problems on Windows. For example,
>> File::Spec is broken on Windows when used with Taint mode on, because Taint
>> mode restricts use of environment variables (which doesn't protect
>> anything). See also this reddit discussion[1], on the problems with taint.
>>
>> We’d like to consider adding a Configure option for disabling taint mode.
>> We see this as a potential first step to having this disabled by default,
>> and then possibly removing support for taint entirely.
>>
>> We’re interested in hearing thoughts on this.
>>
>> Neil
>>
>> [1]
>> https://www.reddit.com/r/perl6/comments/718z4o/taint_mode_for_perl_6/dnmu83i/
>>
>
> Using -DNO_TAINT_SUPPORT as Steffen mentioned you can already build such a
> perl. All we really need to do is add a Configure option so that it can
> easily be disabled and so that code (especially tests) can keep it into
> account.
>
> One open question is if SILENT_NO_TAINT_SUPPORT should be enabled along
> with it. I would argue it should, without it many tests will refuse to even
> start running.
>

Probably most of the work is making the test suite use that %Config option
to skip any tests depending on it.

Leon

> Anyway, I really wish we completely removed that misfeature, but if wecan't, adding it as an option to Configure is the next best thing.

I think/hope that adding a Configure option is the first step on a path that looks something like the following:

1. Add Configure option, with taint included by default
2. Toggle so that taint isn’t included by default
3. Remove taint, so there isn’t a Configure option

Once step 1 is done, we can find out how many things on CPAN break, and do something about those. We’d need to give people time to adjust, so I guess we’d have to stay on step 1 for at least two annual releases.

If we move to step 2, distributions would presumably start shipping a perl that doesn’t have taint support, which I think inexorably moves us towards step 3.

Depending on the discussion we get here, and the response to a Configure option, of course we may decide that we should stick at step 1. But I suspect that giving people a "go 10% faster" option, where the trade off is a feature they never use, is going to nudge us along.

Neil

On Fri, Aug 13, 2021 at 11:31 AM Neil Bowers <neilb@neilb.org> wrote:

> Anyway, I really wish we completely removed that misfeature, but if wecan't,
> adding it as an option to Configure is the next best thing.
>
>
> I think/hope that adding a Configure option is the first step on a path
> that looks something like the following:
>
> 1. Add Configure option, with taint included by default
> 2. Toggle so that taint isn’t included by default
> 3. Remove taint, so there isn’t a Configure option
>
> Once step 1 is done, we can find out how many things on CPAN break, and do
> something about those. We’d need to give people time to adjust, so I guess
> we’d have to stay on step 1 for at least two annual releases.
>
> If we move to step 2, distributions would presumably start shipping a perl
> that doesn’t have taint support, which I think inexorably moves us towards
> step 3.
>
> Depending on the discussion we get here, and the response to a Configure
> option, of course we may decide that we should stick at step 1. But I
> suspect that giving people a "go 10% faster" option, where the trade off is
> a feature they never use, is going to nudge us along.
>
>
> Neil
>

Thanks, Neil. That sounds like a good plan to me! :+1:

>

On Fri, 13 Aug 2021 at 11:31, Neil Bowers <neilb@neilb.org> wrote:

> Anyway, I really wish we completely removed that misfeature, but if wecan't,
> adding it as an option to Configure is the next best thing.
>
>
> I think/hope that adding a Configure option is the first step on a path
> that looks something like the following:
>
> 1. Add Configure option, with taint included by default
> 2. Toggle so that taint isn’t included by default
> 3. Remove taint, so there isn’t a Configure option
>
> Once step 1 is done, we can find out how many things on CPAN break, and do
> something about those. We’d need to give people time to adjust, so I guess
> we’d have to stay on step 1 for at least two annual releases.
>

I can double check, but we have been running with this in all our builds
for ages. When we rolled it out I think we had some turbulence learning
which test files could be ignored/skipped, I think we might have pushed
some patches upstream so they would test that taint was available and skip
the test, and i think we also just built something into our build process
so we know what test files not to run. But other than build issues I am not
aware it caused any issues.

cheers
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Fri, 13 Aug 2021 at 03:53, Leon Timmermans <fawaka@gmail.com> wrote:

> On Fri, Aug 13, 2021 at 12:27 AM Leon Timmermans <fawaka@gmail.com> wrote:
>
>> On Fri, Aug 13, 2021 at 12:19 AM Neil Bowers <neilb@neilb.org> wrote:
>>
>>> Back in 2012, Steffen Mueller did some experimenting and found that
>>> taint adds somewhere between 10% and 20% runtime overhead:
>>> https://www.nntp.perl.org/group/perl.perl5.porters/2012/10/msg193822.html
>>> As you can see, there was some discussion at that time, but it didn’t
>>> seem to go anywhere. This has come up again as a result of the Quirks
>>> document, and we discussed it in last week’s PSC meeting.
>>>
>>> Anecdotally, very few people use taint (these days), yet we’re all
>>> paying the price. Furthermore, taint causes problems on Windows. For
>>> example, File::Spec is broken on Windows when used with Taint mode on,
>>> because Taint mode restricts use of environment variables (which doesn't
>>> protect anything). See also this reddit discussion[1], on the problems with
>>> taint.
>>>
>>> We’d like to consider adding a Configure option for disabling taint
>>> mode. We see this as a potential first step to having this disabled by
>>> default, and then possibly removing support for taint entirely.
>>>
>>> We’re interested in hearing thoughts on this.
>>>
>>> Neil
>>>
>>> [1]
>>> https://www.reddit.com/r/perl6/comments/718z4o/taint_mode_for_perl_6/dnmu83i/
>>>
>>
>> Using -DNO_TAINT_SUPPORT as Steffen mentioned you can already build such
>> a perl. All we really need to do is add a Configure option so that it can
>> easily be disabled and so that code (especially tests) can keep it into
>> account.
>>
>> One open question is if SILENT_NO_TAINT_SUPPORT should be enabled along
>> with it. I would argue it should, without it many tests will refuse to even
>> start running.
>>
>
> Probably most of the work is making the test suite use that %Config option
> to skip any tests depending on it.
>
>
I have a vague recollection we pushed some patches like this when Steffen
introduced the flag. We have been building our perls with this ever since.

cheers,
Yves


--
perl -Mre=debug -e "/just|another|perl|hacker/"

The key driver for adding a Configure option to control taint support is that we think this brings a performance improvement, but we’re not sure exactly what that improvement will be.

To that end, we asked Steffen Schwigon if he’d be interested in helping out, by benchmarking perl with and without taint support, to give us a better feeling on the improvement. Steffen has run various benchmarks on perl over the years, to see whether new releases perform against older ones.

Steffen will outline his plans, giving people a chance to ask questions and suggest things. Please be realistic on your suggestions  ;-)

Neil

Neil Bowers <neilb@neilb.org> writes:
> The key driver for adding a Configure option to control taint support is that we think this brings a performance improvement, but we’re not sure exactly what that improvement will be.
>
> To that end, we asked Steffen Schwigon if he’d be interested in helping out, by benchmarking perl with and without taint support, to give us a better feeling on the improvement. Steffen has run various
> benchmarks on perl over the years, to see whether new releases perform against older ones.
>
> Steffen will outline his plans, giving people a chance to ask questions and suggest things. Please be realistic on your suggestions ;-)

I hate to promise things in advance but anyway. :-) The plan is to
simply do what I did in the past, without inventing new stuff. If you
want to get a code snippet benchmarked, see "Contributions" below.

Here is the plan:

***

Benchmarking taintmode (-DNO_TAINT_SUPPORT)


I will try to benchmark the impact of the compilation option
-DNO_TAINT_SUPPORT to perl's performance.

For that I will use my Perl::Formance framework, which takes care of
generating reliable numbers with meta information. Past results and
presentations can be found here:

* http://renormalist.github.io/Benchmark-Perl-Formance/

The latest short summary is from PerlCon 2019:

* http://renormalist.github.io/Benchmark-Perl-Formance/res/2019-08-08-perlcon-riga-perlformance-status.pdf


Overview:

The focus of the Perl::Formance project is on real world Perl with
CPAN dependencies. However, there are also some micro benchmarks
contained.

In contrast to earlier years where I measured Perl versions over time,
for the taintmode exercise I will only build one blead version from git
with two configurations: with and without taintmode.

To ensure dependency stability I use a static local CPAN mirror, maybe
tweaked with distroprefs. It gets a latest sync and then no updates
anymore.


Hardware:

I will do it on an Intel i7-10610U from 2020 (4 core, 8 threads, maybe
with threading switched off). When I have time I can maybe do the same
on an AMD Phenom II X6 from 2009, and an older i7 from 2012.


Benchmarks:

I will run as many as possible plugins from the Perl::Formance suite,
starting with the easiest (PerlStone2015), later trying the harder
ones, like SpamAssassin. Each benchmark runs multiple iterations.

* https://metacpan.org/pod/Benchmark::Perl::Formance::Plugin::PerlStone2015


Results:

I should provide simple "charts", probably looking a bit silly, with
just two data points per benchmark representing an average, plus raw
values about the standard deviation and confidence intervall (ci_95
lower+upper value), like here:

* http://renormalist.github.io/Benchmark-Perl-Formance/charts/perlformance/index.html
* http://renormalist.github.io/Benchmark-Perl-Formance/charts/perlformance/raw-numbers.txt

(Remember, it will look more silly with only 2 data points for the two
variants of one common perl version.)

If I can draw a single-sentence conclusion, I will do.


Bonus results:

Only when I'm feeling lucky I might generate different charts types from
the raw data to better present the result distribution (histogram,
whisker plot, you name it), but I can not promise that.


Risks:

1) Having no taintmode might fail some test suites of CPAN
dependencies. I will have to skip the tests, interfere, or
"force". Broken or missing deps can theoretically lead to different
code paths behind the scenes.

2) I haven't done any Perl after v5.30 yet with latest CPAN
lately. Everything can go wrong.


Contributions:

If you want to contribute code snippets to get benchmarked, then
copy/paste/extend a module for the PerlStone2015 plugin and send a pull
request:

* https://metacpan.org/pod/Benchmark::Perl::Formance::Plugin::PerlStone2015

Have a look at fib.pm for an easy start.


Timeframe:

I have a block of spare time available in the week from 2021-09-13.

EOF.

Kind regards,
Steffen Schwigon ("renormalist")
--
Steffen Schwigon <ss5@renormalist.net>
Dresden Perl Mongers

Steffen Schwigon <ss5@renormalist.net> writes:
> Neil Bowers <neilb@neilb.org> writes:
>> To that end, we asked Steffen Schwigon if he’d be interested in
>> helping out, by benchmarking perl with and without taint support,
>
> I will try to benchmark the impact of the compilation option
> -DNO_TAINT_SUPPORT to perl's performance.

Can I find the code snippets and maybe even the results that Steffen
Mueller used 10 years ago when he benchmarked NO_PERL_SUPPORT?

Kind regards,
Steffen
--
Steffen Schwigon <ss5@renormalist.net>
Perl benchmarks <http://perlformance.net>
Dresden Perl Mongers <http://dresden-pm.org/>

Steffen Schwigon <ss5@renormalist.net> writes:
> I will try to benchmark the impact of the compilation option
> -DNO_TAINT_SUPPORT to perl's performance.

Executive Summary:

With very few exceptions or outliers Perl with -DNO_TAINT_SUPPORT is
generally faster than with taint support by about 1% to 5% for average
real world code. There are some more extreme results where the
benchmarks are special:

- Some algorithmic micro benchmarks are nearly not affected or even
slower.

- Some particular Perl feature micro benchmarks are up to 20% faster.

- The complex text processing benchmark SpamAssassin is 14% faster ([1]).


[1] Obviously, in real life SpamAssassin would use taintmode for good
reason.


More details with boxplots and percentiles are here:

* http://renormalist.github.io/Benchmark-Perl-Formance/res/2021-09-17-perl-benchmark-notaintsupport-v1.html


Here is documented what I did:

* https://github.com/renormalist/perl-benchmarking-taintmode


I hope that helps.

Kind regards,
Steffen
--
Steffen Schwigon <ss5@renormalist.net>
Perl benchmarks <http://perlformance.net>
Dresden Perl Mongers <http://dresden-pm.org/>

On Fri, 17 Sept 2021 at 22:30, Steffen Schwigon <ss5@renormalist.net> wrote:

> Steffen Schwigon <ss5@renormalist.net> writes:
> > I will try to benchmark the impact of the compilation option
> > -DNO_TAINT_SUPPORT to perl's performance.
>
> Executive Summary:
>
> With very few exceptions or outliers Perl with -DNO_TAINT_SUPPORT is
> generally faster than with taint support by about 1% to 5% for average
> real world code. There are some more extreme results where the
> benchmarks are special:
>
> - Some algorithmic micro benchmarks are nearly not affected or even
> slower.
>
> - Some particular Perl feature micro benchmarks are up to 20% faster.
>
> - The complex text processing benchmark SpamAssassin is 14% faster ([1]).
>
>
> [1] Obviously, in real life SpamAssassin would use taintmode for good
> reason.
>
>
> More details with boxplots and percentiles are here:
>
> *
> http://renormalist.github.io/Benchmark-Perl-Formance/res/2021-09-17-perl-benchmark-notaintsupport-v1.html
>
>
> Here is documented what I did:
>
> * https://github.com/renormalist/perl-benchmarking-taintmode
>
>
> I hope that helps.
>

That lines up with what we saw, although i think ours were closer to the 5
than the 1% change.

cheers,
Yves