Mailing List Archive

On 8/5/20 8:54 AM, Felipe Gasper wrote:
> This interface appears to be undocumented, but per IRC discussion it’s useful and widely used.
>
> Would it be appropriate at this point to document? I’m happy to contribute if so.
>
> Thank you!
>
> -FG
>


Even if it isn't API, it should be documented, like nearly everything.
If it isn't API, it can go into perlintern. So there is no reason
anyone can legitimately object to this proposal.

On Wed, 5 Aug 2020 10:54:50 -0400
Felipe Gasper <felipe@felipegasper.com> wrote:

> This interface appears to be undocumented, but per IRC discussion it?s useful and widely used.
>
> Would it be appropriate at this point to document? I?m happy to contribute if so.
>
> Thank you!
>
> -FG

My guess is it wasn't documented because making it a public API would
make changing the underlying datastructure of AVs to anything else than
a plain C array impossible (e.g. to some kind of sparse array or perhaps
an indexable skiplist).

I think that ship has sailed decades ago because, as you said, it's
being widely used, and even if it wasn't, I think it's unlikely we would
accept the performance trade-offs of the other datastructures.

On Wed, Aug 05, 2020 at 11:29:12PM +0200, Tomasz Konojacki wrote:
> On Wed, 5 Aug 2020 10:54:50 -0400
> Felipe Gasper <felipe@felipegasper.com> wrote:
>
> > This interface appears to be undocumented, but per IRC discussion it?s useful and widely used.
> >
> > Would it be appropriate at this point to document? I?m happy to contribute if so.
> >
> > Thank you!
> >
> > -FG
>
> My guess is it wasn't documented because making it a public API would
> make changing the underlying datastructure of AVs to anything else than
> a plain C array impossible (e.g. to some kind of sparse array or perhaps
> an indexable skiplist).
>
> I think that ship has sailed decades ago because, as you said, it's
> being widely used, and even if it wasn't, I think it's unlikely we would
> accept the performance trade-offs of the other datastructures.

Note that there is an important difference between AvARRAY(av)
and AvARRAY(av)[n]. The former can be API without the latter being. In
particular, a few years ago we changed unused elements of an array
from being 'AvARRAY(av)[n] = &PL_sv_undef' to 'AvARRAY(av)[n] = NULL'.
This caused a certain very vocal XS author to declare that once again we
had deliberately sabotaged perl.

I personally would leave AvARRAY not part of the API unless we are going
to start guaranteeing all aspects of its behaviour.

--
You live and learn (although usually you just live).

> On Aug 10, 2020, at 7:40 AM, Dave Mitchell <davem@iabyn.com> wrote:
>
> On Wed, Aug 05, 2020 at 11:29:12PM +0200, Tomasz Konojacki wrote:
>> On Wed, 5 Aug 2020 10:54:50 -0400
>> Felipe Gasper <felipe@felipegasper.com> wrote:
>>
>>> This interface appears to be undocumented, but per IRC discussion it? useful and widely used.
>>>
>>> Would it be appropriate at this point to document? I? happy to contribute if so.
>>>
>>> Thank you!
>>>
>>> -FG
>>
>> My guess is it wasn't documented because making it a public API would
>> make changing the underlying datastructure of AVs to anything else than
>> a plain C array impossible (e.g. to some kind of sparse array or perhaps
>> an indexable skiplist).
>>
>> I think that ship has sailed decades ago because, as you said, it's
>> being widely used, and even if it wasn't, I think it's unlikely we would
>> accept the performance trade-offs of the other datastructures.
>
> Note that there is an important difference between AvARRAY(av)
> and AvARRAY(av)[n]. The former can be API without the latter being. In
> particular, a few years ago we changed unused elements of an array
> from being 'AvARRAY(av)[n] = &PL_sv_undef' to 'AvARRAY(av)[n] = NULL'.
> This caused a certain very vocal XS author to declare that once again we
> had deliberately sabotaged perl.
>
> I personally would leave AvARRAY not part of the API unless we are going
> to start guaranteeing all aspects of its behaviour.

Hm.

My use case is to implement splice(@array, 0, 1) in XS. After initially thinking I’d have to iterate through the AV, it was pointed out in IRC that I could simply Move() the AvARRAY contents. For now that’s what I’m doing, but if AvARRAY is to be omitted from the API by design then I may need to revisit.

What if I created an av_remove() macro that did this? Something like:

void av_remove(AV *av, SSize_t offset, SSize_t len)

??

-FG

On Mon, Aug 10, 2020 at 09:15:29AM -0400, Felipe Gasper wrote:
> What if I created an av_remove() macro that did this? Something like:
>
> void av_remove(AV *av, SSize_t offset, SSize_t len)

How about a more general av_slice() API function?

void Perl_av_slice(AV *av, SSize_t offset, SSize_t len, AV* ret, AV*fill);

If ret is non-null, it must be an AV, which will then be assigned any
deleted elements; and fill, if non-null, is an AV whose elements are
inserted. So equivalent to

@ret = splice @av, offset, len, @fill);

--
No man treats a motor car as foolishly as he treats another human being.
When the car will not go, he does not attribute its annoying behaviour to
sin, he does not say, You are a wicked motorcar, and I shall not give you
any more petrol until you go. He attempts to find out what is wrong and
set it right.
-- Bertrand Russell,
Has Religion Made Useful Contributions to Civilization?

> On Aug 10, 2020, at 10:09 AM, Dave Mitchell <davem@iabyn.com> wrote:
>
> On Mon, Aug 10, 2020 at 09:15:29AM -0400, Felipe Gasper wrote:
>> What if I created an av_remove() macro that did this? Something like:
>>
>> void av_remove(AV *av, SSize_t offset, SSize_t len)
>
> How about a more general av_slice() API function?
>
> void Perl_av_slice(AV *av, SSize_t offset, SSize_t len, AV* ret, AV*fill);
>
> If ret is non-null, it must be an AV, which will then be assigned any
> deleted elements; and fill, if non-null, is an AV whose elements are
> inserted. So equivalent to
>
> @ret = splice @av, offset, len, @fill);

Hi Dave,

This (or something similar) sounds like the way to go.

I’m assuming this should go in av.c … but why Perl_av_slice() rather than just av_slice()?

-F

On Mon, Aug 10, 2020 at 10:19:27AM -0400, Felipe Gasper wrote:
> This (or something similar) sounds like the way to go.
>
> I’m assuming this should go in av.c … but why Perl_av_slice() rather than just av_slice()?

All API functions are of the form Perl_foo() with a foo() macro defined to
handle context correctly. Don't worry, I wasn't suggesting anything
unusual.

--
The Enterprise is captured by a vastly superior alien intelligence which
does not put them on trial.
-- Things That Never Happen in "Star Trek" #10

On 10/8/20 13:40, Dave Mitchell wrote:
> On Wed, Aug 05, 2020 at 11:29:12PM +0200, Tomasz Konojacki wrote:
>> On Wed, 5 Aug 2020 10:54:50 -0400
>> Felipe Gasper <felipe@felipegasper.com> wrote:
>>
>>> This interface appears to be undocumented, but per IRC discussion it’s useful and widely used.
>>>
>>> Would it be appropriate at this point to document? I’m happy to contribute if so.
>>>
>>> Thank you!
>>>
>>> -FG
>>
>> My guess is it wasn't documented because making it a public API would
>> make changing the underlying datastructure of AVs to anything else than
>> a plain C array impossible (e.g. to some kind of sparse array or perhaps
>> an indexable skiplist).
>>
>> I think that ship has sailed decades ago because, as you said, it's
>> being widely used, and even if it wasn't, I think it's unlikely we would
>> accept the performance trade-offs of the other datastructures.
>
> Note that there is an important difference between AvARRAY(av)
> and AvARRAY(av)[n]. The former can be API without the latter being. In
> particular, a few years ago we changed unused elements of an array
> from being 'AvARRAY(av)[n] = &PL_sv_undef' to 'AvARRAY(av)[n] = NULL'.
> This caused a certain very vocal XS author to declare that once again we
> had deliberately sabotaged perl.
>
> I personally would leave AvARRAY not part of the API unless we are going
> to start guaranteeing all aspects of its behaviour.
>

"sortsv" documentation used to say:

> sortsv
>
> Sort an array. Here is an example:
>
> sortsv(AvARRAY(av), av_len(av)+1, Perl_sv_cmp_locale);


And so, that idiom is relatively common (and also, there is no other
simple and/or efficient way to call sortsv in order to sort an array).

On Mon, 10 Aug 2020 at 15:15, Felipe Gasper <felipe@felipegasper.com> wrote:
>
>
>
> > On Aug 10, 2020, at 7:40 AM, Dave Mitchell <davem@iabyn.com> wrote:
> >
> > On Wed, Aug 05, 2020 at 11:29:12PM +0200, Tomasz Konojacki wrote:
> >> On Wed, 5 Aug 2020 10:54:50 -0400
> >> Felipe Gasper <felipe@felipegasper.com> wrote:
> >>
> >>> This interface appears to be undocumented, but per IRC discussion it? useful and widely used.
> >>>
> >>> Would it be appropriate at this point to document? I? happy to contribute if so.
> >>>
> >>> Thank you!
> >>>
> >>> -FG
> >>
> >> My guess is it wasn't documented because making it a public API would
> >> make changing the underlying datastructure of AVs to anything else than
> >> a plain C array impossible (e.g. to some kind of sparse array or perhaps
> >> an indexable skiplist).
> >>
> >> I think that ship has sailed decades ago because, as you said, it's
> >> being widely used, and even if it wasn't, I think it's unlikely we would
> >> accept the performance trade-offs of the other datastructures.
> >
> > Note that there is an important difference between AvARRAY(av)
> > and AvARRAY(av)[n]. The former can be API without the latter being. In
> > particular, a few years ago we changed unused elements of an array
> > from being 'AvARRAY(av)[n] = &PL_sv_undef' to 'AvARRAY(av)[n] = NULL'.
> > This caused a certain very vocal XS author to declare that once again we
> > had deliberately sabotaged perl.
> >
> > I personally would leave AvARRAY not part of the API unless we are going
> > to start guaranteeing all aspects of its behaviour.
>
> Hm.
>
> My use case is to implement splice(@array, 0, 1) in XS. After initially thinking I’d have to iterate through the AV, it was pointed out in IRC that I could simply Move() the AvARRAY contents. For now that’s what I’m doing, but if AvARRAY is to be omitted from the API by design then I may need to revisit.

This is not what Perl would do. You should not move anything when you
remove items from the front of an array. You increment the array
pointer, and then fiddle with a flag so that perl knows you have
hidden items at the start of the array. I believe it stores the number
of elements hidden in the IV part of the struct, but i havent looked
closely.

perl -MDevel::Peek -e'my @array= (1..10); Dump(\@array); shift @array;
Dump(\@array)'
SV = IV(0x1dec468) at 0x1dec478
REFCNT = 1
FLAGS = (TEMP,ROK)
RV = 0x1df7438
SV = PVAV(0x1dd8160) at 0x1df7438
REFCNT = 2
FLAGS = (PADMY)
ARRAY = 0x1e01780
FILL = 9
MAX = 9
ARYLEN = 0x0
FLAGS = (REAL)
[elements trimmed]

SV = IV(0x1dd6208) at 0x1dd6218
REFCNT = 1
FLAGS = (TEMP,ROK)
RV = 0x1df7438
SV = PVAV(0x1dd8160) at 0x1df7438
REFCNT = 2
FLAGS = (PADMY)
ARRAY = 0x1e01788 (offset=1) <<<<<<<<<<<<<<<<<<< NOTICE THIS
ALLOC = 0x1e01780
FILL = 8
MAX = 8
ARYLEN = 0x0
FLAGS = (REAL)
[elements trimmed]


>
> What if I created an av_remove() macro that did this? Something like:
>
> void av_remove(AV *av, SSize_t offset, SSize_t len)

So long as it as optimal as perl would be it makes sense to me. I have
a touch of uneasiness of using SSize_t, but i can see why you want the
function to resolve negative indexes. Although how could the *length*
be negative? Shouldnt that be STRLEN?

Yves


--
perl -Mre=debug -e "/just|another|perl|hacker/"

On Tue, 11 Aug 2020 at 12:13, demerphq <demerphq@gmail.com> wrote:
>
> On Mon, 10 Aug 2020 at 15:15, Felipe Gasper <felipe@felipegasper.com> wrote:
> >
> >
> >
> > > On Aug 10, 2020, at 7:40 AM, Dave Mitchell <davem@iabyn.com> wrote:
> > >
> > > On Wed, Aug 05, 2020 at 11:29:12PM +0200, Tomasz Konojacki wrote:
> > >> On Wed, 5 Aug 2020 10:54:50 -0400
> > >> Felipe Gasper <felipe@felipegasper.com> wrote:
> > >>
> > >>> This interface appears to be undocumented, but per IRC discussion it? useful and widely used.
> > >>>
> > >>> Would it be appropriate at this point to document? I? happy to contribute if so.
> > >>>
> > >>> Thank you!
> > >>>
> > >>> -FG
> > >>
> > >> My guess is it wasn't documented because making it a public API would
> > >> make changing the underlying datastructure of AVs to anything else than
> > >> a plain C array impossible (e.g. to some kind of sparse array or perhaps
> > >> an indexable skiplist).
> > >>
> > >> I think that ship has sailed decades ago because, as you said, it's
> > >> being widely used, and even if it wasn't, I think it's unlikely we would
> > >> accept the performance trade-offs of the other datastructures.
> > >
> > > Note that there is an important difference between AvARRAY(av)
> > > and AvARRAY(av)[n]. The former can be API without the latter being. In
> > > particular, a few years ago we changed unused elements of an array
> > > from being 'AvARRAY(av)[n] = &PL_sv_undef' to 'AvARRAY(av)[n] = NULL'.
> > > This caused a certain very vocal XS author to declare that once again we
> > > had deliberately sabotaged perl.
> > >
> > > I personally would leave AvARRAY not part of the API unless we are going
> > > to start guaranteeing all aspects of its behaviour.
> >
> > Hm.
> >
> > My use case is to implement splice(@array, 0, 1) in XS. After initially thinking I’d have to iterate through the AV, it was pointed out in IRC that I could simply Move() the AvARRAY contents. For now that’s what I’m doing, but if AvARRAY is to be omitted from the API by design then I may need to revisit.
>
> This is not what Perl would do. You should not move anything when you
> remove items from the front of an array. You increment the array
> pointer, and then fiddle with a flag so that perl knows you have
> hidden items at the start of the array. I believe it stores the number
> of elements hidden in the IV part of the struct, but i havent looked
> closely.

More generically the only time you should move data in an AV with a
splice/remove operation is when you are removing items from the
middle, and then you have no choice but to move items (and should
probably take care to move the least items). But when you are removing
things from the front or end of an array the operation should be O(1)
and be only bookkeeping not data moves.

Yves

On Tue, 11 Aug 2020 12:18:19 +0200
demerphq <demerphq@gmail.com> wrote:

> On Tue, 11 Aug 2020 at 12:13, demerphq <demerphq@gmail.com> wrote:
> >
> > On Mon, 10 Aug 2020 at 15:15, Felipe Gasper
> > <felipe@felipegasper.com> wrote:
> > >
> > >
> > >
> > > > On Aug 10, 2020, at 7:40 AM, Dave Mitchell <davem@iabyn.com>
> > > > wrote:
> > > >
> > > > On Wed, Aug 05, 2020 at 11:29:12PM +0200, Tomasz Konojacki
> > > > wrote:
> > > >> On Wed, 5 Aug 2020 10:54:50 -0400
> > > >> Felipe Gasper <felipe@felipegasper.com> wrote:
> > > >>
> > > >>> This interface appears to be undocumented, but per IRC
> > > >>> discussion it? useful and widely used.
> > > >>>
> > > >>> Would it be appropriate at this point to document? I? happy
> > > >>> to contribute if so.
> > > >>>
> > > >>> Thank you!
> > > >>>
> > > >>> -FG
> > > >>
> > > >> My guess is it wasn't documented because making it a public
> > > >> API would make changing the underlying datastructure of AVs to
> > > >> anything else than a plain C array impossible (e.g. to some
> > > >> kind of sparse array or perhaps an indexable skiplist).
> > > >>
> > > >> I think that ship has sailed decades ago because, as you said,
> > > >> it's being widely used, and even if it wasn't, I think it's
> > > >> unlikely we would accept the performance trade-offs of the
> > > >> other datastructures.
> > > >
> > > > Note that there is an important difference between AvARRAY(av)
> > > > and AvARRAY(av)[n]. The former can be API without the latter
> > > > being. In particular, a few years ago we changed unused
> > > > elements of an array from being 'AvARRAY(av)[n] = &PL_sv_undef'
> > > > to 'AvARRAY(av)[n] = NULL'. This caused a certain very vocal XS
> > > > author to declare that once again we had deliberately sabotaged
> > > > perl.
> > > >
> > > > I personally would leave AvARRAY not part of the API unless we
> > > > are going to start guaranteeing all aspects of its behaviour.
> > >
> > > Hm.
> > >
> > > My use case is to implement splice(@array, 0, 1) in XS. After
> > > initially thinking I’d have to iterate through the AV, it was
> > > pointed out in IRC that I could simply Move() the AvARRAY
> > > contents. For now that’s what I’m doing, but if AvARRAY is to be
> > > omitted from the API by design then I may need to revisit.
> >
> > This is not what Perl would do. You should not move anything when
> > you remove items from the front of an array. You increment the array
> > pointer, and then fiddle with a flag so that perl knows you have
> > hidden items at the start of the array. I believe it stores the
> > number of elements hidden in the IV part of the struct, but i
> > havent looked closely.
>
> More generically the only time you should move data in an AV with a
> splice/remove operation is when you are removing items from the
> middle, and then you have no choice but to move items (and should
> probably take care to move the least items).

But that was the entire point of the original question - in effect, an
API for doing splice() operations.

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

On Mon, 10 Aug 2020 15:09:36 +0100
Dave Mitchell <davem@iabyn.com> wrote:

> On Mon, Aug 10, 2020 at 09:15:29AM -0400, Felipe Gasper wrote:
> > What if I created an av_remove() macro that did this? Something
> > like:
> >
> > void av_remove(AV *av, SSize_t offset, SSize_t len)
>
> How about a more general av_slice() API function?
>
> void Perl_av_slice(AV *av, SSize_t offset, SSize_t len, AV* ret,
> AV*fill);
>
> If ret is non-null, it must be an AV, which will then be assigned any
> deleted elements; and fill, if non-null, is an AV whose elements are
> inserted. So equivalent to
>
> @ret = splice @av, offset, len, @fill);

Question: Why av_slice() and not av_splice(), since it is an XS/API
version of the splice() perl operator...?

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

On Tue, 11 Aug 2020 at 13:10, Paul "LeoNerd" Evans
<leonerd@leonerd.org.uk> wrote:
> On Tue, 11 Aug 2020 12:18:19 +0200
> demerphq <demerphq@gmail.com> wrote:
> > On Tue, 11 Aug 2020 at 12:13, demerphq <demerphq@gmail.com> wrote:
> > > On Mon, 10 Aug 2020 at 15:15, Felipe Gasper
> > > > My use case is to implement splice(@array, 0, 1) in XS. After
> > > > initially thinking I’d have to iterate through the AV, it was
> > > > pointed out in IRC that I could simply Move() the AvARRAY
> > > > contents. For now that’s what I’m doing, but if AvARRAY is to be
> > > > omitted from the API by design then I may need to revisit.

> But that was the entire point of the original question - in effect, an
> API for doing splice() operations.

Notice the paragraph I was responding to talks about using Move()
AvARRAY to handle splice(@array,0,1), which is not how that particular
statement should be handled. Sure a hypothetical function that
implements splice should use Move from time to time, but not in the
stated case.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

> On Aug 11, 2020, at 09:01, demerphq <demerphq@gmail.com> wrote:
>
> ?On Tue, 11 Aug 2020 at 13:10, Paul "LeoNerd" Evans
> <leonerd@leonerd.org.uk> wrote:
>>> On Tue, 11 Aug 2020 12:18:19 +0200
>>> demerphq <demerphq@gmail.com> wrote:
>>> On Tue, 11 Aug 2020 at 12:13, demerphq <demerphq@gmail.com> wrote:
>>>> On Mon, 10 Aug 2020 at 15:15, Felipe Gasper
>>>>> My use case is to implement splice(@array, 0, 1) in XS. After
>>>>> initially thinking I’d have to iterate through the AV, it was
>>>>> pointed out in IRC that I could simply Move() the AvARRAY
>>>>> contents. For now that’s what I’m doing, but if AvARRAY is to be
>>>>> omitted from the API by design then I may need to revisit.
>
>> But that was the entire point of the original question - in effect, an
>> API for doing splice() operations.
>
> Notice the paragraph I was responding to talks about using Move()
> AvARRAY to handle splice(@array,0,1), which is not how that particular
> statement should be handled. Sure a hypothetical function that
> implements splice should use Move from time to time, but not in the
> stated case.

I’m sorry for the unclear initial statement of the problem, but yeah, I need to remove an element from an arbitrary place in the array.

-F

On Tue, 11 Aug 2020 at 15:23, Felipe Gasper <felipe@felipegasper.com> wrote:
>
>
>
> > On Aug 11, 2020, at 09:01, demerphq <demerphq@gmail.com> wrote:
> >
> > ?On Tue, 11 Aug 2020 at 13:10, Paul "LeoNerd" Evans
> > <leonerd@leonerd.org.uk> wrote:
> >>> On Tue, 11 Aug 2020 12:18:19 +0200
> >>> demerphq <demerphq@gmail.com> wrote:
> >>> On Tue, 11 Aug 2020 at 12:13, demerphq <demerphq@gmail.com> wrote:
> >>>> On Mon, 10 Aug 2020 at 15:15, Felipe Gasper
> >>>>> My use case is to implement splice(@array, 0, 1) in XS. After
> >>>>> initially thinking I’d have to iterate through the AV, it was
> >>>>> pointed out in IRC that I could simply Move() the AvARRAY
> >>>>> contents. For now that’s what I’m doing, but if AvARRAY is to be
> >>>>> omitted from the API by design then I may need to revisit.
> >
> >> But that was the entire point of the original question - in effect, an
> >> API for doing splice() operations.
> >
> > Notice the paragraph I was responding to talks about using Move()
> > AvARRAY to handle splice(@array,0,1), which is not how that particular
> > statement should be handled. Sure a hypothetical function that
> > implements splice should use Move from time to time, but not in the
> > stated case.
>
> I’m sorry for the unclear initial statement of the problem, but yeah, I need to remove an element from an arbitrary place in the array.

Ok. FWIW, the typical approach to dealing with this kind of thing is
to take the pp_ sub and then extract its guts into a generic api that
that the pp_ sub calls.

See pp_splice in pp.c. I didnt see any use of Move in it, I suspect
because Move() only helps one case when you want to insert a bunch of
items in the middle and you need to move the stuff after it to make
room, whereas the actual pp_splice code is written to handle more
cases with one loop.

cheers,
Yves


--
perl -Mre=debug -e "/just|another|perl|hacker/"