Present: Neil, Paul, Rik plus Karl, Nick, Yves
Background: The overall problem of trying to track "programmer intent"
on things like strings-vs-numbers; useful for situations like encoders
for JSON/MsgPack/etc, Sereal, as well as general in-language use in
places that Perl does not currently support.
We spent the first part of the meeting on a fairly informal discussion
trying to drill down into what the real problem is that we're trying to
address. The consensus overall was that the existing SV flags (the 3
"public" POK, NOK, IOK, plus 3 "private" pPOK, pNOK, pIOK) are a poor
interface for XS code (and by extension perl code) to ask as to the
intended nature of Perl values.
The discussion quickly agreed that the meaning of the private flags is
fairly clear - private flags are set when the corresponding `struct SV`
field is set to a value, and thus that field is meaningful to be read
again.
What is far less clear is what the public flags are meant to mean. We
spent a while each discussing what our personal "mental model" of these
flags is, to eventually discover that there isn't a clear consensus on
what they really mean in precise detail; there's edge-cases all round.
For instance, currently PV->IV conversion will turn on IOK and pIOK,
but PV->NV conversion will not turn on NOK while it does pNOK. Looking
historically there have been a number of changes to the behaviours of
public flags over the years. Sometime around the 5.18 era Chip made a
big change.
Nick writes:
The big Chip patch was in 2012:
https://github.com/Perl/perl5/commit/4bac9ae47b5ad7845a24e26b0e95609805de688a
Effectively in its comment about "This scheme did not cover ROK,
however" it's referencing this commit:
https://github.com/Perl/perl5/commit/463ee0b2acbd047c27e8b5393cdd8398881824c5
(I'd love to say "just follow that link to be surprised" but you'd all
think that it was a Rickroll, so I'll observe that it's "perl 5.0
alpha 4")
which added the flags SVp_POK ...IOK and ..NOK and added references
for the first time, but didn't add a private flag for references. I
assume because it didn't occur to Larry that magic get might return a
reference, not just values.
In summary: it's a mess.
Having spent enough time on that we decided it might be better to
attack the problem from a different route, by carefully defining what
sort of API shape we want, in terms of what operations should be
supported and how they should behave and interact with one another.
A central theme of the problem is that while we have many API-defined
functions/macros for setting SV value fields (e.g. `sv_setiv()`) and
querying them again (e.g. `SvIV()`), we don't actually have anything to
ask type-intent of values (e.g. the "is this a string?" question). All
we have is large bodies of XS code that tries to do this by testing the
various public (or at times private) xOK flags, but without clear
guidance from core they often cope poorly in weird cornercases. Core's
behaviour of smudging additional flags on also doesn't help here - it's
far too easy to end up with SVs that are both POK and IOK, having
forgotten if they were originally strings or numbers. Nick's PR
(https://github.com/Perl/perl5/pull/18958) will certainly help this
situation but that isn't sufficient.
The suggested next steps here involve creating a long list of "test
cases"; situations involving performing various kinds of operations on
values/variables, and specifying what are the properties of results,
and side-effects on variables within it. Likely many of these
properties will take the form of "appears to be a string" or "appears
to be a number" or similar.
These test cases would suggest the form of an `XS::APItest`-like `.t`
file which can live in Perl core, containing all the tests. Most
critically it would require defining a new API to ask questions of the
form "appears to be (some sort of data shape)" about the SVs. For
instance, while the recent "stable-bools" branch added `SvIsBOOL()`
there aren't similar test functions for other value shapes like strings
and numbers. So that would need thinking about somehow.
Once an API shape is settled on, we can then look at how to implement
the predicate tests. Likely many of them will take the form of SV flags
tests, in combination with maybe some amount of behavioural change in
how core perl sets these flags. But it is important to stress that the
SV flags are very much the "core internals" of how the tests would
work. The test functions themselves would be the expected API that XS
authors would use (and likely pureperl authors via some wrapper
functions in maybe Scalar::Util or more likely whatever our new
std/builtins/functions namespace ends up becoming; see #PSC 034).
Acknowledging the fact that there is currently a large body of existing
XS code which tests particular flag combinations, it would still be
useful if the new behaviour of core internals was largely compatible
with what decisions this existing code makes, at least in the common
unambiguous cases.
--
Paul "LeoNerd" Evans
leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/
Background: The overall problem of trying to track "programmer intent"
on things like strings-vs-numbers; useful for situations like encoders
for JSON/MsgPack/etc, Sereal, as well as general in-language use in
places that Perl does not currently support.
We spent the first part of the meeting on a fairly informal discussion
trying to drill down into what the real problem is that we're trying to
address. The consensus overall was that the existing SV flags (the 3
"public" POK, NOK, IOK, plus 3 "private" pPOK, pNOK, pIOK) are a poor
interface for XS code (and by extension perl code) to ask as to the
intended nature of Perl values.
The discussion quickly agreed that the meaning of the private flags is
fairly clear - private flags are set when the corresponding `struct SV`
field is set to a value, and thus that field is meaningful to be read
again.
What is far less clear is what the public flags are meant to mean. We
spent a while each discussing what our personal "mental model" of these
flags is, to eventually discover that there isn't a clear consensus on
what they really mean in precise detail; there's edge-cases all round.
For instance, currently PV->IV conversion will turn on IOK and pIOK,
but PV->NV conversion will not turn on NOK while it does pNOK. Looking
historically there have been a number of changes to the behaviours of
public flags over the years. Sometime around the 5.18 era Chip made a
big change.
Nick writes:
The big Chip patch was in 2012:
https://github.com/Perl/perl5/commit/4bac9ae47b5ad7845a24e26b0e95609805de688a
Effectively in its comment about "This scheme did not cover ROK,
however" it's referencing this commit:
https://github.com/Perl/perl5/commit/463ee0b2acbd047c27e8b5393cdd8398881824c5
(I'd love to say "just follow that link to be surprised" but you'd all
think that it was a Rickroll, so I'll observe that it's "perl 5.0
alpha 4")
which added the flags SVp_POK ...IOK and ..NOK and added references
for the first time, but didn't add a private flag for references. I
assume because it didn't occur to Larry that magic get might return a
reference, not just values.
In summary: it's a mess.
Having spent enough time on that we decided it might be better to
attack the problem from a different route, by carefully defining what
sort of API shape we want, in terms of what operations should be
supported and how they should behave and interact with one another.
A central theme of the problem is that while we have many API-defined
functions/macros for setting SV value fields (e.g. `sv_setiv()`) and
querying them again (e.g. `SvIV()`), we don't actually have anything to
ask type-intent of values (e.g. the "is this a string?" question). All
we have is large bodies of XS code that tries to do this by testing the
various public (or at times private) xOK flags, but without clear
guidance from core they often cope poorly in weird cornercases. Core's
behaviour of smudging additional flags on also doesn't help here - it's
far too easy to end up with SVs that are both POK and IOK, having
forgotten if they were originally strings or numbers. Nick's PR
(https://github.com/Perl/perl5/pull/18958) will certainly help this
situation but that isn't sufficient.
The suggested next steps here involve creating a long list of "test
cases"; situations involving performing various kinds of operations on
values/variables, and specifying what are the properties of results,
and side-effects on variables within it. Likely many of these
properties will take the form of "appears to be a string" or "appears
to be a number" or similar.
These test cases would suggest the form of an `XS::APItest`-like `.t`
file which can live in Perl core, containing all the tests. Most
critically it would require defining a new API to ask questions of the
form "appears to be (some sort of data shape)" about the SVs. For
instance, while the recent "stable-bools" branch added `SvIsBOOL()`
there aren't similar test functions for other value shapes like strings
and numbers. So that would need thinking about somehow.
Once an API shape is settled on, we can then look at how to implement
the predicate tests. Likely many of them will take the form of SV flags
tests, in combination with maybe some amount of behavioural change in
how core perl sets these flags. But it is important to stress that the
SV flags are very much the "core internals" of how the tests would
work. The test functions themselves would be the expected API that XS
authors would use (and likely pureperl authors via some wrapper
functions in maybe Scalar::Util or more likely whatever our new
std/builtins/functions namespace ends up becoming; see #PSC 034).
Acknowledging the fact that there is currently a large body of existing
XS code which tests particular flag combinations, it would still be
useful if the new behaviour of core internals was largely compatible
with what decisions this existing code makes, at least in the common
unambiguous cases.
--
Paul "LeoNerd" Evans
leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/