Mailing List Archive

Well, Tim, I not sure enough to comment on your description of how a tie()
works in SFIO. Let's try reversing things. Here is the immediate use I
see (and would like to have available) for tying a handle:

$term = new Term::Control; #Use curses, terminfo, termcap, whatever

select($term);

print "Hello, world!\n";

print "Please type in some text: ";
$text = <>;

print "\n\nThank you very much. Now press a key: ';

$key = getc $term;

print "You pressed '$key'\n";

print "\n\n";

ioctl $term, T_ATTRON, "blink";
print "Goodbye!";
ioctl $term, T_ATTROFF, "blink";

close( $term );
__END__


Now, how would you see this tying (sorry) in with the SFIO discipline
paradigm?

> Tim.

--
Kenneth Albanowski (kjahds@kjahds.com, CIS: 70705,126)

Tim Bunce writes:
>
> I've not seen this come back so I'm reposting it.

I certainly haven't seen this before.

> >From timbo Mon Oct 23 20:19:28 1995
>
> Whilst thinking of tying file handles it seems to me that the number of
> perl builtins which use filehandles is so large that it may take
> significant effort for the developer to effectively derive a new class.
>
> It may also require considerable coding effort for perl
> internals to do-the-right-thing for -f $fh_obj etc.
>
> Pondering the impact of SFIO and its concept of stackable disciplines
> it occurred to me that a more narrow behind-the-scenes form of
> tying which more closely matched SFIO's discipline interface may
> be useful.
>
> An SFIO discipline is defined by a structure containing four function
> pointers for read, write, seek and expections. Read write and seek
> should be obvious but expections needs more explanation. Here's an
> extract from the manual:
[...]
> Food for thought.

Indeed. I've been meaning to look at sfio but haven't had a chance.
The description you include above confirms to me that it really
does deserve consideration. I'd hate for there to be even more
mucking around with filehandle internals and API and only find out
too late that it doesn't interoperate nicely with sfio. I'll grab
sfio-oct92.tar.gz from CPAN, which archie now tells me is around,
and see if I can make any useful comments. It's probably going to
make the filehandle thread expand and drag on, but I suspect that's
a necessary nettle to grasp.

--Malcolm

--
Malcolm Beattie <mbeattie@sable.ox.ac.uk>
Unix Systems Programmer
Oxford University Computing Services

Kenneth Albanowski writes:
>
>
> Well, Tim, I not sure enough to comment on your description of how a tie()
> works in SFIO. Let's try reversing things. Here is the immediate use I
> see (and would like to have available) for tying a handle:
>
> $term = new Term::Control; #Use curses, terminfo, termcap, whatever
>
> select($term);
>
> print "Hello, world!\n";
>
> print "Please type in some text: ";
> $text = <>;
>
> print "\n\nThank you very much. Now press a key: ';
>
> $key = getc $term;
>
> print "You pressed '$key'\n";
>
> print "\n\n";
>
> ioctl $term, T_ATTRON, "blink";
> print "Goodbye!";
> ioctl $term, T_ATTROFF, "blink";
>
> close( $term );
> __END__
>
>
> Now, how would you see this tying (sorry) in with the SFIO discipline
> paradigm?

If we're lucky, maybe something like
$term = STDOUT->clone; # or whatever the syntax would be
$term->pushmod("Curses");
select($term);
print "Hello, world!\n";
[...]
$term->ioctl(attron => "blink");
print "Goodbye!";
$term->ioctl(attroff => "blink");
Curses could be a stackable discipline on top of ordinary filehandles.
Those ioctls would be passed along the stack of discipline modules
until it reached one that understood it (like STREAMS). I initially
was going to suggest
$term->attron("blink");
print "Goodbye!";
$term->attroff("blink");
with an AUTOLOAD to pass the messages down the discipline stack
but maybe the method caching would interact badly with pushing/popping
disciplines. If method inheritance is "vertical" then looking
through the discipline stack for a module that understood is really
a "horizontal" search. Maybe multiple inheritance does that, though.
Push the class that implements the new discipline onto the @ISA of
the class (*) that represents the current discipline stack. Note that
(*) would more normally be an object than class, but maybe an
anonymous class would be a sort of "static" representation of a
discipline stack. I haven't really thought this through yet so this
may all be rubbish.

--Malcolm

--
Malcolm Beattie <mbeattie@sable.ox.ac.uk>
Unix Systems Programmer
Oxford University Computing Services

> From: Malcolm Beattie <mbeattie@sable.ox.ac.uk>
>
> Tim Bunce writes:
> >
> > Whilst thinking of tying file handles it seems to me that the number of
> > perl builtins which use filehandles is so large that it may take
> > significant effort for the developer to effectively derive a new class.
> >
> > It may also require considerable coding effort for perl
> > internals to do-the-right-thing for -f $fh_obj etc.
> >
> > Pondering the impact of SFIO and its concept of stackable disciplines
> > it occurred to me that a more narrow behind-the-scenes form of
> > tying which more closely matched SFIO's discipline interface may
> > be useful.
> >
> > An SFIO discipline is defined by a structure containing four function
> > pointers for read, write, seek and expections. Read write and seek
> > should be obvious but expections needs more explanation. Here's an
> > extract from the manual:
> [...]
> > Food for thought.
>
> Indeed. I've been meaning to look at sfio but haven't had a chance.
> The description you include above confirms to me that it really
> does deserve consideration. I'd hate for there to be even more
> mucking around with filehandle internals and API and only find out
> too late that it doesn't interoperate nicely with sfio. I'll grab
> sfio-oct92.tar.gz from CPAN, which archie now tells me is around,
> and see if I can make any useful comments. It's probably going to
> make the filehandle thread expand and drag on, but I suspect that's
> a necessary nettle to grasp.

I agree.

Remember to have a word with Paul Marquess <pmarquess@bfsec.bt.co.uk>
and take a look at his SFIO module.

> From: Malcolm Beattie <mbeattie@sable.ox.ac.uk>
>
> Kenneth Albanowski writes:
> >
> > Well, Tim, I not sure enough to comment on your description of how a tie()
> > works in SFIO.

Well, I wasn't really pushing tie as a mechanism related to sfio. I was
simply pointing out that sfio and its stackable disciplines should be
included in our discussions.

> > Let's try reversing things. Here is the immediate use I
> > see (and would like to have available) for tying a handle:
> >
> > $term = new Term::Control; #Use curses, terminfo, termcap, whatever
> > ioctl $term, T_ATTRON, "blink";

Umm, personally I think ioctl's aren't the right model for termcap etc.

> > Now, how would you see this tying (sorry) in with the SFIO discipline
> > paradigm?

Well, basically, I don't. Sfio disciplines only encompass read, write, seek
and 'exceptions'. It doesn't include ioctl's.

> Curses could be a stackable discipline on top of ordinary filehandles.
> Those ioctls would be passed along the stack of discipline modules
> until it reached one that understood it (like STREAMS).

Remember that sfio is a replacement for stdio and stdio doesn't include ioctl's.

> I initially was going to suggest
> $term->attron("blink");
> print "Goodbye!";
> $term->attroff("blink");
> with an AUTOLOAD to pass the messages down the discipline stack
> but maybe the method caching would interact badly with pushing/popping
> disciplines. If method inheritance is "vertical" then looking
> through the discipline stack for a module that understood is really
> a "horizontal" search. Maybe multiple inheritance does that, though.
> Push the class that implements the new discipline onto the @ISA of
> the class (*) that represents the current discipline stack. Note that
> (*) would more normally be an object than class, but maybe an
> anonymous class would be a sort of "static" representation of a
> discipline stack. I haven't really thought this through yet so this
> may all be rubbish.

I think there is a misunderstanding about what sfio is and is not.

I'll try to outline my perspective here (I've CC'd this to Phong @ AT&T
in case my understanding is incorrect).

Firstly basic SFIO (ignoring disciplines): A safe/fast replacement for
STDIO with known behaviour. Good error handling. Read/write to memory
strings. Using mmap() for speed where available. Stream pools for
auto-synchronization or data serialization. Stackable streams (not
disciplines, think of transparent #include handling in a preprocessor).
Polling (select) on streams. etc etc. All good stuff.

SFIO disciplines: The sfio manual says "A normal file stream makes use
of the system calls read(), write and lseek() to read, write and
position in the under-lying file. Certain applications may implement
alternative IO methods or may further process the data before handling
it to the application or before writing out to the file. This can be
done by defining an IO discipline."

Disciplines are quite 'narrow' things. The read/write/seek/exception
scope mandates that. They are not a replacement for treating files as
objects. ioctl $myfh @args should be a method call on $myfh and would
have nothing directly to do with sfio.

A good example of both the utility and narrow scope of disciplines is
the list of disciplines supplied with sfio:

Sfdisc_t* dcnewskable(Sfio_t* f);
int dcdelskable(Sfdisc_t* disc);
dcnewskable() creates a discipline that when inserted on the
stream f will ensure that f is seekable. If f is originally
unseekable, data will be shadowed in a temporary file stream
to allow seekability. .

Sfdisc_t* dcnewtee(Sfio_t* tee);
int dcdeltee(Sfdisc_t* disc);
dcnewtee() creates a discipline that when inserted into a
stream f will duplicate to the stream tee any data written to f.

Sfdisc_t* dcnewfilter(char* cmd);
int dcdelfilter(Sfdisc_t* disc);
dcnewfilter() creates a discipline that when inserted into a
stream f will run the command cmd to process any input data
before making it available to the application. For example,
dcnewfilter("uncompress") is an equivalent but slower alter-
native to the lzw discipline below. .

Sfdisc_t* dcnewsubstream(Sfio_t* base, long offset, long
extent);
int dcdelsubstream(Sfdisc_t* disc);
dcnewsubstream() creates a discipline that reserves a por-
tion of the stream base starting at offset with length
extent and makes this portion appear as if it is a stream.
When this discipline is inserted into a stream, it will make
cause all IO operations on this stream to take place in the
reserved portion of the base stream. .

Sfdisc_t* dcnewlzw(void);
int dcdellzw(Sfdisc_t* disc);
dcnewlzw() creates a discipline that when inserted into a
stream f will run the uncompress algorithm on input data
from f before making it available to the application. This
is useful to allow applications to process data from a file
packed with the UNIX compress utility as if the data is in
plain text.

Sfdisc_t* dcnewunion(Sfio_t** list, int n);
int dcdelunion(Sfdisc_t* disc);
dcnewunion() creates a discipline that concatenates input
data from all n streams in list. When inserted into a
stream f, this discipline will cause all input operations on
f to come from the merged data stream. .

[Wishful thinking alert :-]

Sfdisc_t* dcnewperl(SV *object);
dcnewperl() creates a proxy discipline that converts all sfio
discipline function calls into perl method calls on object.
The methods call are READ, WRITE, SEEK and each of the named
exception types, SF_NEW, SF_CLOSE, SF_DPOLL etc.

So, yes, we need to think about SFIO but no, it's not going to replace
file-handles-as-objects. SFIO would work behind the scenes, especially
given Larry's point that tie for filehandles might actually just be a
bless (and hence that method calls should act as the 'user interface'
to file handles).

I suspect we'll just have a core SFIO functionality transparently
replacing the STDIO functionality perl currently uses and then have a
separate SFIO module as an interface to the more whizz-bang features.

More food for thought.

Tim.

Tim Bunce writes:
> > From: Malcolm Beattie <mbeattie@sable.ox.ac.uk>
> >
> > Tim Bunce writes:
> > >
> > > Whilst thinking of tying file handles it seems to me that the number of
> > > perl builtins which use filehandles is so large that it may take
> > > significant effort for the developer to effectively derive a new class.
> > >
> > > It may also require considerable coding effort for perl
> > > internals to do-the-right-thing for -f $fh_obj etc.
> > >
> > > Pondering the impact of SFIO and its concept of stackable disciplines
> > > it occurred to me that a more narrow behind-the-scenes form of
> > > tying which more closely matched SFIO's discipline interface may
> > > be useful.
> > >
> > > An SFIO discipline is defined by a structure containing four function
> > > pointers for read, write, seek and expections. Read write and seek
> > > should be obvious but expections needs more explanation. Here's an
> > > extract from the manual:
> > [...]
> > > Food for thought.
> >
> > Indeed. I've been meaning to look at sfio but haven't had a chance.
> > The description you include above confirms to me that it really
> > does deserve consideration. I'd hate for there to be even more
> > mucking around with filehandle internals and API and only find out
> > too late that it doesn't interoperate nicely with sfio. I'll grab
> > sfio-oct92.tar.gz from CPAN, which archie now tells me is around,
> > and see if I can make any useful comments. It's probably going to
> > make the filehandle thread expand and drag on, but I suspect that's
> > a necessary nettle to grasp.
>
> I agree.
>
> Remember to have a word with Paul Marquess <pmarquess@bfsec.bt.co.uk>
> and take a look at his SFIO module.

Will do.

> > From: Malcolm Beattie <mbeattie@sable.ox.ac.uk>
> >
> > Kenneth Albanowski writes:
> > >
> > > Well, Tim, I not sure enough to comment on your description of how a tie()
> > > works in SFIO.
>
> Well, I wasn't really pushing tie as a mechanism related to sfio. I was
> simply pointing out that sfio and its stackable disciplines should be
> included in our discussions.
>
> > > Let's try reversing things. Here is the immediate use I
> > > see (and would like to have available) for tying a handle:
> > >
> > > $term = new Term::Control; #Use curses, terminfo, termcap, whatever
> > > ioctl $term, T_ATTRON, "blink";
>
> Umm, personally I think ioctl's aren't the right model for termcap etc.

Perhaps ioctl has the wrong connotations. I was thinking of ioctl
in this context of meaning a fairly free-form out-of-band hook for
disciplines to implement non-standard methods in disciplines. Users
just need to know a name (e.g. "attron") and the sort of data it
wants and the internals would pass the message along to the right
discipline in the stack.

> > > Now, how would you see this tying (sorry) in with the SFIO discipline
> > > paradigm?
>
> Well, basically, I don't. Sfio disciplines only encompass read, write, seek
> and 'exceptions'. It doesn't include ioctl's.

But, since perl has a fairly dynamic class API (just define a
sub attron { } and suddenly it's available as a method call for
anyone that wants it) there's nothing to stop discipline classes
from making out-of-band methods available. It's in this sense that
I was thinking of the name ioctl, and perhaps Kenneth was too.
Anything that isn't a read/write/seek gets implemented in its own
(shared) namespace and lumped under the name "ioctl". STREAMS turns
the user call ioctl(fd, I_FOO, somedata) into a standard format
message and passes it down the stack of modules until it finds one
which understands it. Analogously, perl could take a method call
$stream->attron($something) (or $stream->ioctl($something)) and
walk down the stack of disciplines (either explicitly or via some
previously set-up inheritance) until it comes across a discipline
class that understands it.

> > Curses could be a stackable discipline on top of ordinary filehandles.
> > Those ioctls would be passed along the stack of discipline modules
> > until it reached one that understood it (like STREAMS).
>
> Remember that sfio is a replacement for stdio and stdio doesn't include ioctl's.

I realise that (I've been playing with sfio this afternoon) but the
API that sfio exposes looks as though it could be perlified quite
nicely and beefed up to be more general-purpose, as with the Curses
example above.

> I think there is a misunderstanding about what sfio is and is not.
>
> I'll try to outline my perspective here (I've CC'd this to Phong @ AT&T
> in case my understanding is incorrect).
>
> Firstly basic SFIO (ignoring disciplines): A safe/fast replacement for
> STDIO with known behaviour. Good error handling. Read/write to memory
> strings. Using mmap() for speed where available. Stream pools for
> auto-synchronization or data serialization. Stackable streams (not
> disciplines, think of transparent #include handling in a preprocessor).
> Polling (select) on streams. etc etc. All good stuff.

Agreed.

> SFIO disciplines: The sfio manual says "A normal file stream makes use
> of the system calls read(), write and lseek() to read, write and
> position in the under-lying file. Certain applications may implement
> alternative IO methods or may further process the data before handling
> it to the application or before writing out to the file. This can be
> done by defining an IO discipline."
>
> Disciplines are quite 'narrow' things. The read/write/seek/exception
> scope mandates that. They are not a replacement for treating files as
> objects. ioctl $myfh @args should be a method call on $myfh and would
> have nothing directly to do with sfio.

Agreed, except that that the user API could make use of the existence
of the discipline stack to make
ioctl $stream "attron";
do "the right thing" in terms of inheritance to allow the discipline
classes that are pushed at the time to intercept messages intended
for them.

> A good example of both the utility and narrow scope of disciplines is
> the list of disciplines supplied with sfio:

[...]
> [Wishful thinking alert :-]
>
> Sfdisc_t* dcnewperl(SV *object);
> dcnewperl() creates a proxy discipline that converts all sfio
> discipline function calls into perl method calls on object.
> The methods call are READ, WRITE, SEEK and each of the named
> exception types, SF_NEW, SF_CLOSE, SF_DPOLL etc.
>
> So, yes, we need to think about SFIO but no, it's not going to replace
> file-handles-as-objects.

I don't see why SFIO shouldn't be used as a basis for a powerful
streams(lowercase)/discipline API for perl. I also think such a
thing would graft in quite nicely to the current perl internals,
which is your next point.

> SFIO would work behind the scenes, especially
> given Larry's point that tie for filehandles might actually just be a
> bless (and hence that method calls should act as the 'user interface'
> to file handles).
>
> I suspect we'll just have a core SFIO functionality transparently
> replacing the STDIO functionality perl currently uses and then have a
> separate SFIO module as an interface to the more whizz-bang features.

This last paragraph is sort of what I've been talking about.
If sfio disciplines (plus a little bit more) can be implemented
easily as classes written in perl, then there should be less pressure
for those who want to add more and more fields to SVs/XPVIOs and so
on to store state. The state can be offloaded to the discipline
object and pushed onto the stream.

How about this, for example?
# Sfio_t *stream;
# Sfdisc_t *uncdisc, *grepdisc;
$stream = new Sfio "foo"; # stream = sfopen(NULL, "foo", "r");
$uncompress = new Disc::Filter "uncompress";
# uncdisc = sfdcnewfilter("uncompress");
$stream->disc($uncompress); # sfdisc(stream, uncdisc);
$grep = new Disc::Filter "grep bar";
# grepdisc = sfdcnewfilter("grep bar");
$stream->disc($grep); # sfdisc(stream, grepdisc);

or just

$stream = new Sfio "foo";
$stream->disc(new Disc::Filter "uncompress");
$stream->disc(new Disc::Filter "grep bar");


To begin with, the Sfio module could be solely an ordinary XSUB module
with its own methods for I/O following the C names: $stream->sfprintf
and so on. The two places where changes to the perl internals would be
necessary to integrate it seamlessly would be (1) native perl I/O
operators (print, printf and so on) which would have to map to the
Sfio versions and (2) the XPVIO would have to be able to distinguish
between stdio and sfio streams unless stdio was either dropped all
in one go from the internals or else the compatibility-mode of sfio
was used for the stdio in perl. I suspect that they would have to run
in parallel for a while in order to be able to get good benchmarks of
sfio performance compared to the perl built-in stdio speedups.

--Malcolm

--
Malcolm Beattie <mbeattie@sable.ox.ac.uk>
Unix Systems Programmer
Oxford University Computing Services

> From: Malcolm Beattie <mbeattie@sable.ox.ac.uk>
>
> Tim Bunce writes:
> >
> > > From: Malcolm Beattie <mbeattie@sable.ox.ac.uk>
> > >
> > > Kenneth Albanowski writes:
> > > >
> > > > Well, Tim, I not sure enough to comment on your description of how a tie()
> > > > works in SFIO.
> >
> > Well, I wasn't really pushing tie as a mechanism related to sfio. I was
> > simply pointing out that sfio and its stackable disciplines should be
> > included in our discussions.
> >
> > > > Let's try reversing things. Here is the immediate use I
> > > > see (and would like to have available) for tying a handle:
> > > >
> > > > $term = new Term::Control; #Use curses, terminfo, termcap, whatever
> > > > ioctl $term, T_ATTRON, "blink";
> >
> > Umm, personally I think ioctl's aren't the right model for termcap etc.
>
> Perhaps ioctl has the wrong connotations. I was thinking of ioctl
> in this context of meaning a fairly free-form out-of-band hook for
> disciplines to implement non-standard methods in disciplines. Users
> just need to know a name (e.g. "attron") and the sort of data it
> wants and the internals would pass the message along to the right
> discipline in the stack.

An out-of-band hook is a tempting idea but not part of sfio. If you're
thinking of changing sfio then you're into a whole new set of issues.

Basically it sounds like you're looking to merge sfio with a STREAMS-like
messaging system. I suspect we're trying to run with sfio before we can walk.

> > > > Now, how would you see this tying (sorry) in with the SFIO discipline
> > > > paradigm?
> >
> > Well, basically, I don't. Sfio disciplines only encompass read, write, seek
> > and 'exceptions'. It doesn't include ioctl's.
>
> But, since perl has a fairly dynamic class API (just define a
> sub attron { } and suddenly it's available as a method call for
> anyone that wants it) there's nothing to stop discipline classes
> from making out-of-band methods available. It's in this sense that
> I was thinking of the name ioctl, and perhaps Kenneth was too.
> Anything that isn't a read/write/seek gets implemented in its own
> (shared) namespace and lumped under the name "ioctl". STREAMS turns
> the user call ioctl(fd, I_FOO, somedata) into a standard format
> message and passes it down the stack of modules until it finds one
> which understands it. Analogously, perl could take a method call
> $stream->attron($something) (or $stream->ioctl($something)) and
> walk down the stack of disciplines (either explicitly or via some
> previously set-up inheritance) until it comes across a discipline
> class that understands it.

It would have to be explicitly since a stream stack would have no
relationship with an inheritance hierarchy (and faking one would incur
the heavy cost of invalidating all the ISA caches).


> > > Curses could be a stackable discipline on top of ordinary filehandles.
> > > Those ioctls would be passed along the stack of discipline modules
> > > until it reached one that understood it (like STREAMS).
> >
> > Remember that sfio is a replacement for stdio and stdio doesn't include ioctl's.
>
> I realise that (I've been playing with sfio this afternoon) but the
> API that sfio exposes looks as though it could be perlified quite
> nicely and beefed up to be more general-purpose, as with the Curses
> example above.

I tend to take the view that sfio disciplines are best suited to what
you might use a | series | of | pipes for in unix, only this way they're
all in the same process. Curses don't seem to fit that model well.
I think it's a case of "just because you could doesn't make it a good idea".

> I don't see why SFIO shouldn't be used as a basis for a powerful
> streams(lowercase)/discipline API for perl. I also think such a
> thing would graft in quite nicely to the current perl internals,
> which is your next point.

I have to agree that it's an appealing idea in general.

> > SFIO would work behind the scenes, especially
> > given Larry's point that tie for filehandles might actually just be a
> > bless (and hence that method calls should act as the 'user interface'
> > to file handles).
> >
> > I suspect we'll just have a core SFIO functionality transparently
> > replacing the STDIO functionality perl currently uses and then have a
> > separate SFIO module as an interface to the more whizz-bang features.
>
> This last paragraph is sort of what I've been talking about.
> If sfio disciplines (plus a little bit more) can be implemented
> easily as classes written in perl, then there should be less pressure
> for those who want to add more and more fields to SVs/XPVIOs and so
> on to store state. The state can be offloaded to the discipline
> object and pushed onto the stream.

Interesting point.

> To begin with, the Sfio module could be solely an ordinary XSUB module
> with its own methods for I/O following the C names: $stream->sfprintf
> and so on. The two places where changes to the perl internals would be
> necessary to integrate it seamlessly would be (1) native perl I/O
> operators (print, printf and so on) which would have to map to the
> Sfio versions and (2) the XPVIO would have to be able to distinguish
> between stdio and sfio streams unless stdio was either dropped all
> in one go from the internals or else the compatibility-mode of sfio
> was used for the stdio in perl. I suspect that they would have to run
> in parallel for a while in order to be able to get good benchmarks of
> sfio performance compared to the perl built-in stdio speedups.

A key aim of the original sfio module work by Paul was to prove the
portability of sfio itself.

I'm glad you're thinking about binary compatibility since that likely
to prove to be a sticky issue. Many extensions link to system and
third-party libraries which have been linked against stdio. Even
dynamic libraries may have dependencies on the size of the FILE struct.

Since I've not seen Paul post recently I'll append a couple of his
last messages on the Sfio module (Feb 95). They're a handy reminder
that the path ahead may not be all that smooth.

Tim.


----- Begin Included Message -----

Date: Tue, 21 Feb 95 13:10:55 GMT
From: Paul Marquess <pmarquess@bfsec.bt.co.uk>
To: mbeattie@sable.ox.ac.uk
Subject: Re: Sfio: opinions wanted.
Cc: perl5-porters@africa.nicoh.com, pmarquess@bfsec.bt.co.uk

From: Malcolm Beattie <mbeattie@sable.ox.ac.uk>
>
> Sfio sounds like another good reason to implement SVs whose strings
> aren't allocated with malloc. The first good reason that I came across
> was mmap. It was nice to be able to do
> $foo = mmap(HUGEFILE, 0, $hugefilesize); # or some such syntax
> while ($foo =~ /some interesting pattern/g) { do_something() }
> The idea was to have a flag SVf_NOTMALLOC so that when the SV went away,
> the pointer wasn't freed. I didn't address the problem of reallocating
> the string (sv_grow) since that was called from all over the place and
> I seem to remember one or two other gotchas. If the patches aren't
> archived in perl5-porters, I'll try to dig them out: they weren't large.
> It might be as easy to do from scratch. I think that with such a flag and
> some extra magic types (for a generalised reallocate and free), both sfio
> and mmap can be done elegantly. There may well be other uses too.

I had forgotten that discussion till you mentioned it Malcolm.

ok, I've found your original posting on mmap. From the quick look at
what you have done it looks like you have only implemented a read only
buffer. Where there any later patches?

It certainly would make life simpler for Sfio to have an SV which
corresponded to a buffer not under the control of Perl. I *do* think
that this would be a useful feature for module writers in general, so I
think it needs to be given a bit of thought and done properly. Comments?

Things that Sfio (at least) needs

1. Given a pointer & a size, the ability to have an SV 'adopt' it.

2. As Malcolm says, internal support to prevent and any calls to
sv_grow or calling free. I did a quick grep on the Perl source and
there are over 30 places where SvGROW gets called - not too bad.

3. Internal support to make sure that a user cannot write off the end
of the buffer. I'm not sure if this would be caught by any changes
needed for sv_grow.

4. Optionally set the SV read-only. This should be straightforward.
My recollection is that although there is no support for read only
variables in the language (yet), the internal support is all there.

5. The ability to make an 'ordinary' SV have the same behaviour as
above. sfio allows you to specify the buffer to be used for i/o, so
it would be nice to have this type of SV exibit the same behaviour
as one created by 1.

Looking at your reply, Malcolm, I guess 2 & 3 could be dealt with in a
more general way by having magic support for realloc & free. Must have
another look at the magic section in perlguts :)

Can anyone think of any other features that would be required?

Paul

----- End Included Message -----


----- Begin Included Message -----

From: Paul Marquess <pmarquess@bfsec.bt.co.uk>
Subject: Sfio: opinions wanted.
To: perl5-porters@africa.nicoh.com
Date: Sun, 19 Feb 1995 15:00:22 +0000 (GMT)
Cc: Paul Marquess <pmarquess@bfsec.bt.co.uk>
Reply-To: pmarquess@bfsec.bt.co.uk

I didn't get much time to have a look at Sfio (i.e. the Perl module, as
opposed to sfio the package it interfaces to) this week, but things have
settled down enough for me to have another look at it.

I'm playing with disciplines just now & I'm trying to decide the best way
to represent the memory buffers that sfio uses all over the place.

Basically there are a lot of occasions where sfio expects or presents the
user with a pointer to an area of memory along with an associated size.
These areas of memory correspond to the buffers used by the package for
i/o. The user of the Sfio package will have to be able to

1. Create an area of memory which sfio can use for a buffer.
2. Access an area of memory that sfio has allocated itself.
3. Read from either types of buffer
4. Write to either types of buffer

First I just used the good old 'copy the complete buffer into a Perl scalar'
approach. Whilst this does work, it means a *lot* of copying goes on.

Next I thought about presenting the user with the pointer & size as is. They
can use unpack to get to the buffer, *but* as far as I can see pack cannot
write back to the same place. Is this correct? I might very well have been
looking too hard at the code, so if I have completely missed the obvious
perhaps one of you could point it out to me.
I promise not to scream too loud :)

My third idea was to implement a Memory class. This would allow the creation
of areas of memory in Perl via a constructor and would also allow the
pointer/size pairs which sfio presents the user with to be dealt with.
The things I like most about this approach are that

1. The memory can be presented to the user as the pointer/size pair and
either as a Perl scalar or as an array - given the appropriate ties.

2. The memory will only be copied if the user actually needs to.

3. The class can prevent writing off the end of the memory area. Given the
nature of the memory, it is very important not too stray out of bounds.

Possible downsides

1. Am I using a sledehammer to crack a nut?

2. The tie mechanism is a bit slow at the moment. This may or may not be
addressed in the future when Perl gets optimiseded. Currently then, it
remains to be seen if the Memory class is any faster than simply copying
buffers around.

Opinions?

Paul

----- End Included Message -----

> From: Malcolm Beattie <mbeattie@sable.ox.ac.uk>
>
> Tim Bunce writes:
> >
> > > From: Malcolm Beattie <mbeattie@sable.ox.ac.uk>
> > >
> > > Kenneth Albanowski writes:
> > > >
> > > > Well, Tim, I not sure enough to comment on your description of how a tie()
> > > > works in SFIO.
> >
> > Well, I wasn't really pushing tie as a mechanism related to sfio. I was
> > simply pointing out that sfio and its stackable disciplines should be
> > included in our discussions.
> >
> > > > Let's try reversing things. Here is the immediate use I
> > > > see (and would like to have available) for tying a handle:
> > > >
> > > > $term = new Term::Control; #Use curses, terminfo, termcap, whatever
> > > > ioctl $term, T_ATTRON, "blink";
> >
> > Umm, personally I think ioctl's aren't the right model for termcap etc.
>
> Perhaps ioctl has the wrong connotations. I was thinking of ioctl
> in this context of meaning a fairly free-form out-of-band hook for
> disciplines to implement non-standard methods in disciplines. Users
> just need to know a name (e.g. "attron") and the sort of data it
> wants and the internals would pass the message along to the right
> discipline in the stack.

An out-of-band hook is a tempting idea but not part of sfio. If you're
thinking of changing sfio then you're into a whole new set of issues.

Basically it sounds like you're looking to merge sfio with a STREAMS-like
messaging system. I suspect we're trying to run with sfio before we can walk.

> > > > Now, how would you see this tying (sorry) in with the SFIO discipline
> > > > paradigm?
> >
> > Well, basically, I don't. Sfio disciplines only encompass read, write, seek
> > and 'exceptions'. It doesn't include ioctl's.
>
> But, since perl has a fairly dynamic class API (just define a
> sub attron { } and suddenly it's available as a method call for
> anyone that wants it) there's nothing to stop discipline classes
> from making out-of-band methods available. It's in this sense that
> I was thinking of the name ioctl, and perhaps Kenneth was too.
> Anything that isn't a read/write/seek gets implemented in its own
> (shared) namespace and lumped under the name "ioctl". STREAMS turns
> the user call ioctl(fd, I_FOO, somedata) into a standard format
> message and passes it down the stack of modules until it finds one
> which understands it. Analogously, perl could take a method call
> $stream->attron($something) (or $stream->ioctl($something)) and
> walk down the stack of disciplines (either explicitly or via some
> previously set-up inheritance) until it comes across a discipline
> class that understands it.

It would have to be explicitly since a stream stack would have no
relationship with an inheritance hierarchy (and faking one would incur
the heavy cost of invalidating all the ISA caches).


> > > Curses could be a stackable discipline on top of ordinary filehandles.
> > > Those ioctls would be passed along the stack of discipline modules
> > > until it reached one that understood it (like STREAMS).
> >
> > Remember that sfio is a replacement for stdio and stdio doesn't include ioctl's.
>
> I realise that (I've been playing with sfio this afternoon) but the
> API that sfio exposes looks as though it could be perlified quite
> nicely and beefed up to be more general-purpose, as with the Curses
> example above.

I tend to take the view that sfio disciplines are best suited to what
you might use a | series | of | pipes for in unix, only this way they're
all in the same process. Curses don't seem to fit that model well.
I think it's a case of "just because you could doesn't make it a good idea".

> I don't see why SFIO shouldn't be used as a basis for a powerful
> streams(lowercase)/discipline API for perl. I also think such a
> thing would graft in quite nicely to the current perl internals,
> which is your next point.

I have to agree that it's an appealing idea in general.

> > SFIO would work behind the scenes, especially
> > given Larry's point that tie for filehandles might actually just be a
> > bless (and hence that method calls should act as the 'user interface'
> > to file handles).
> >
> > I suspect we'll just have a core SFIO functionality transparently
> > replacing the STDIO functionality perl currently uses and then have a
> > separate SFIO module as an interface to the more whizz-bang features.
>
> This last paragraph is sort of what I've been talking about.
> If sfio disciplines (plus a little bit more) can be implemented
> easily as classes written in perl, then there should be less pressure
> for those who want to add more and more fields to SVs/XPVIOs and so
> on to store state. The state can be offloaded to the discipline
> object and pushed onto the stream.

Interesting point.

> To begin with, the Sfio module could be solely an ordinary XSUB module
> with its own methods for I/O following the C names: $stream->sfprintf
> and so on. The two places where changes to the perl internals would be
> necessary to integrate it seamlessly would be (1) native perl I/O
> operators (print, printf and so on) which would have to map to the
> Sfio versions and (2) the XPVIO would have to be able to distinguish
> between stdio and sfio streams unless stdio was either dropped all
> in one go from the internals or else the compatibility-mode of sfio
> was used for the stdio in perl. I suspect that they would have to run
> in parallel for a while in order to be able to get good benchmarks of
> sfio performance compared to the perl built-in stdio speedups.

A key aim of the original sfio module work by Paul was to prove the
portability of sfio itself.

I'm glad you're thinking about binary compatibility since that likely
to prove to be a sticky issue. Many extensions link to system and
third-party libraries which have been linked against stdio. Even
dynamic libraries may have dependencies on the size of the FILE struct.

Since I've not seen Paul post recently I'll append a couple of his
last messages on the Sfio module (Feb 95). They're a handy reminder
that the path ahead may not be all that smooth.

Tim.


----- Begin Included Message -----

Date: Tue, 21 Feb 95 13:10:55 GMT
From: Paul Marquess <pmarquess@bfsec.bt.co.uk>
To: mbeattie@sable.ox.ac.uk
Subject: Re: Sfio: opinions wanted.
Cc: perl5-porters@africa.nicoh.com, pmarquess@bfsec.bt.co.uk

From: Malcolm Beattie <mbeattie@sable.ox.ac.uk>
>
> Sfio sounds like another good reason to implement SVs whose strings
> aren't allocated with malloc. The first good reason that I came across
> was mmap. It was nice to be able to do
> $foo = mmap(HUGEFILE, 0, $hugefilesize); # or some such syntax
> while ($foo =~ /some interesting pattern/g) { do_something() }
> The idea was to have a flag SVf_NOTMALLOC so that when the SV went away,
> the pointer wasn't freed. I didn't address the problem of reallocating
> the string (sv_grow) since that was called from all over the place and
> I seem to remember one or two other gotchas. If the patches aren't
> archived in perl5-porters, I'll try to dig them out: they weren't large.
> It might be as easy to do from scratch. I think that with such a flag and
> some extra magic types (for a generalised reallocate and free), both sfio
> and mmap can be done elegantly. There may well be other uses too.

I had forgotten that discussion till you mentioned it Malcolm.

ok, I've found your original posting on mmap. From the quick look at
what you have done it looks like you have only implemented a read only
buffer. Where there any later patches?

It certainly would make life simpler for Sfio to have an SV which
corresponded to a buffer not under the control of Perl. I *do* think
that this would be a useful feature for module writers in general, so I
think it needs to be given a bit of thought and done properly. Comments?

Things that Sfio (at least) needs

1. Given a pointer & a size, the ability to have an SV 'adopt' it.

2. As Malcolm says, internal support to prevent and any calls to
sv_grow or calling free. I did a quick grep on the Perl source and
there are over 30 places where SvGROW gets called - not too bad.

3. Internal support to make sure that a user cannot write off the end
of the buffer. I'm not sure if this would be caught by any changes
needed for sv_grow.

4. Optionally set the SV read-only. This should be straightforward.
My recollection is that although there is no support for read only
variables in the language (yet), the internal support is all there.

5. The ability to make an 'ordinary' SV have the same behaviour as
above. sfio allows you to specify the buffer to be used for i/o, so
it would be nice to have this type of SV exibit the same behaviour
as one created by 1.

Looking at your reply, Malcolm, I guess 2 & 3 could be dealt with in a
more general way by having magic support for realloc & free. Must have
another look at the magic section in perlguts :)

Can anyone think of any other features that would be required?

Paul

----- End Included Message -----


----- Begin Included Message -----

From: Paul Marquess <pmarquess@bfsec.bt.co.uk>
Subject: Sfio: opinions wanted.
To: perl5-porters@africa.nicoh.com
Date: Sun, 19 Feb 1995 15:00:22 +0000 (GMT)
Cc: Paul Marquess <pmarquess@bfsec.bt.co.uk>
Reply-To: pmarquess@bfsec.bt.co.uk

I didn't get much time to have a look at Sfio (i.e. the Perl module, as
opposed to sfio the package it interfaces to) this week, but things have
settled down enough for me to have another look at it.

I'm playing with disciplines just now & I'm trying to decide the best way
to represent the memory buffers that sfio uses all over the place.

Basically there are a lot of occasions where sfio expects or presents the
user with a pointer to an area of memory along with an associated size.
These areas of memory correspond to the buffers used by the package for
i/o. The user of the Sfio package will have to be able to

1. Create an area of memory which sfio can use for a buffer.
2. Access an area of memory that sfio has allocated itself.
3. Read from either types of buffer
4. Write to either types of buffer

First I just used the good old 'copy the complete buffer into a Perl scalar'
approach. Whilst this does work, it means a *lot* of copying goes on.

Next I thought about presenting the user with the pointer & size as is. They
can use unpack to get to the buffer, *but* as far as I can see pack cannot
write back to the same place. Is this correct? I might very well have been
looking too hard at the code, so if I have completely missed the obvious
perhaps one of you could point it out to me.
I promise not to scream too loud :)

My third idea was to implement a Memory class. This would allow the creation
of areas of memory in Perl via a constructor and would also allow the
pointer/size pairs which sfio presents the user with to be dealt with.
The things I like most about this approach are that

1. The memory can be presented to the user as the pointer/size pair and
either as a Perl scalar or as an array - given the appropriate ties.

2. The memory will only be copied if the user actually needs to.

3. The class can prevent writing off the end of the memory area. Given the
nature of the memory, it is very important not too stray out of bounds.

Possible downsides

1. Am I using a sledehammer to crack a nut?

2. The tie mechanism is a bit slow at the moment. This may or may not be
addressed in the future when Perl gets optimiseded. Currently then, it
remains to be seen if the Memory class is any faster than simply copying
buffers around.

Opinions?

Paul

----- End Included Message -----

On Mon, 30 Oct 1995, Malcolm Beattie wrote:

> > > > Let's try reversing things. Here is the immediate use I
> > > > see (and would like to have available) for tying a handle:
> > > >
> > > > $term = new Term::Control; #Use curses, terminfo, termcap, whatever
> > > > ioctl $term, T_ATTRON, "blink";
> >
> > Umm, personally I think ioctl's aren't the right model for termcap etc.
>
> Perhaps ioctl has the wrong connotations. I was thinking of ioctl
> in this context of meaning a fairly free-form out-of-band hook for
> disciplines to implement non-standard methods in disciplines. Users
> just need to know a name (e.g. "attron") and the sort of data it
> wants and the internals would pass the message along to the right
> discipline in the stack.

That was my thought exactly. This _isn't_ part of SFIO, but there seems
to be some need of passing OOB data to a tie()'d filehandle. Since these
are _real_ tied handles, we can't assume custom methods. The only halfway
elegant communications method left is ioctl().

> > > > Now, how would you see this tying (sorry) in with the SFIO discipline
> > > > paradigm?
> >
> > Well, basically, I don't. Sfio disciplines only encompass read, write, seek
> > and 'exceptions'. It doesn't include ioctl's.
>
> But, since perl has a fairly dynamic class API (just define a
> sub attron { } and suddenly it's available as a method call for
> anyone that wants it) there's nothing to stop discipline classes
> from making out-of-band methods available. It's in this sense that
> I was thinking of the name ioctl, and perhaps Kenneth was too.

Just about.

> Anything that isn't a read/write/seek gets implemented in its own
> (shared) namespace and lumped under the name "ioctl". STREAMS turns
> the user call ioctl(fd, I_FOO, somedata) into a standard format
> message and passes it down the stack of modules until it finds one
> which understands it. Analogously, perl could take a method call
> $stream->attron($something) (or $stream->ioctl($something)) and
> walk down the stack of disciplines (either explicitly or via some
> previously set-up inheritance) until it comes across a discipline
> class that understands it.

An interesting proposition. This really _would_ be merging SFIO and tie'd
filehandles.

> > > Curses could be a stackable discipline on top of ordinary filehandles.
> > > Those ioctls would be passed along the stack of discipline modules
> > > until it reached one that understood it (like STREAMS).
> >
> > Remember that sfio is a replacement for stdio and stdio doesn't include ioctl's.
>
> I realise that (I've been playing with sfio this afternoon) but the
> API that sfio exposes looks as though it could be perlified quite
> nicely and beefed up to be more general-purpose, as with the Curses
> example above.

Note: I'm not saying that SFIO disicplines would be the best way to
implement the Curses example. For example, don't "pushed" diciplines have
to have a real filehandle underneath them? But the "Curses" filehandle
wouldn't really be talking to STDOUT, but to the Curses library. SFIO
seems like overkill in this instance.

> > Firstly basic SFIO (ignoring disciplines): A safe/fast replacement for
> > STDIO with known behaviour. Good error handling. Read/write to memory
> > strings. Using mmap() for speed where available. Stream pools for
> > auto-synchronization or data serialization. Stackable streams (not
> > disciplines, think of transparent #include handling in a preprocessor).
> > Polling (select) on streams. etc etc. All good stuff.
>
> Agreed.

Ditto.

> [...]
>
> --Malcolm
>
> --
> Malcolm Beattie <mbeattie@sable.ox.ac.uk>
> Unix Systems Programmer
> Oxford University Computing Services
>

--
Kenneth Albanowski (kjahds@kjahds.com, CIS: 70705,126)

On Mon, 30 Oct 1995, Malcolm Beattie wrote:

> If we're lucky, maybe something like
> $term = STDOUT->clone; # or whatever the syntax would be
> $term->pushmod("Curses");
> select($term);
> print "Hello, world!\n";
> [...]
> $term->ioctl(attron => "blink");
> print "Goodbye!";
> $term->ioctl(attroff => "blink");
> Curses could be a stackable discipline on top of ordinary filehandles.
> Those ioctls would be passed along the stack of discipline modules
> until it reached one that understood it (like STREAMS). I initially
> was going to suggest
> $term->attron("blink");
> print "Goodbye!";
> $term->attroff("blink");
> with an AUTOLOAD to pass the messages down the discipline stack
> but maybe the method caching would interact badly with pushing/popping
> disciplines. If method inheritance is "vertical" then looking
> through the discipline stack for a module that understood is really
> a "horizontal" search. Maybe multiple inheritance does that, though.
> Push the class that implements the new discipline onto the @ISA of
> the class (*) that represents the current discipline stack. Note that
> (*) would more normally be an object than class, but maybe an
> anonymous class would be a sort of "static" representation of a
> discipline stack. I haven't really thought this through yet so this
> may all be rubbish.

All of this is interesting, but I do wonder whether it's overkill. When
done this way, SFIO is required, and print() (for example) would have to
go through all the SFIO and Perl layers just to come back to a Perl
callback. Whereas the "sub PRINT { ... }" overloading style puts the
burden on Perl, not SFIO.

There is an interesting point here: in the example, you apply the
"Curses" dicipline to the STDOUT filehandle. But in reality, Curses is
used instead of a filehandle, not on top of one. (The fact that Curses
itself might use the equivalent of STDOUT internally is irrelevant, as
it's an implementation detail. You might just as well consider
Term::Control::VIO which would interface directly with the OS/2 console
driver, and not use a filehandle at all.) In the Perl tie() style, there
doesn't actually have to be an underlying handle, while with the
"dicipline" SFIO style there is such a need.

> --Malcolm
>
> --
> Malcolm Beattie <mbeattie@sable.ox.ac.uk>
> Unix Systems Programmer
> Oxford University Computing Services
>

--
Kenneth Albanowski (kjahds@kjahds.com, CIS: 70705,126)

From ig.co.uk!Tim.Bunce Mon Oct 30 15:09:41 +0000 1995
I'll try to outline my perspective here (I've CC'd this to Phong @ AT&T
in case my understanding is incorrect).
I am a little late in this discussion so I am not sure I understand
all the threads. But, for me, disciplines are just abstractions that
capture certain dependencies that a software package requires from its
environment - see the discipline section in Practical Reusable Unix Software,
ed. Bala Krisnamurthy, Wiley. This paper also discusses the idea of
categorizing operations in two different types: discipline for resource
acquisition and method for resource manipulation (after acquisition).
OO people tend to glomp both of these into one (called method) and
I think that is a source of confusion for library designers.

Sfio's basic functionality is to provide a reasonable stream model for
data processing. In order to do this, it requires the fundamental operations
of read/write/lseek and some mechanism to handle error states. So operations
that look at the data and do something to it fit the discipline model.
Others may or may not fit. Let's take for example the translation of
\n to/from \r\n. This is a perfect candidate for a discipline that
would look at the output data, find the appropriate pattern, then replace
it by the other. In the same way, embedded video attribute commands in the
stream such as "\BEnter password:\0" (\B for bold, \0 for off all attributes)
can be handle easily. However, terminal->attron(A_BOLD) is probably an operation
that wouldn't fit well with the discipline model because it is unclear how
you would access the internal discipline states. At the application (perl) level,
it may be easier to just think of this as a method function that writes "\B"
to the underlying stream then have the underlying discipline only know about
\pattern translation. The danger of doing this is that users may themselves
write \patterns into the data but maybe that isn't bad.

A possibility is to extend sfio to allow applications to directly raise
application/system-defined exceptions via a call like sfexcept(f,event,data).
If this is in place then you can implement terminal->attron(A_BOLD)
as raising some sort of events like sfexcept(sfstdout,E_ATTR,A_BOLD)
and that event would then be handle by whatever discipline on the discipline
stack that first knows about it. If you think this is useful, let me know.

Phong