Mailing List Archive

M.-A. Lemburg wrote:
>
> Fred L. Drake, Jr. wrote:
> >
> > M.-A. Lemburg writes:
> > > Aside: Is the buffer interface reachable in any way from within
> > > Python ? Why isn't the interface exposed via __XXX__ methods
> > > on normal Python instances (could be implemented by returning a
> > > buffer object) ?
> >
> > Would it even make sense? I though a large part of the intent was
> > to for performance, avoiding memory copies. Perhaps there should be
> > an .__as_buffer__() which returned an object that supports the C
> > buffer interface. I'm not sure how useful it would be; perhaps for
> > classes that represent image data? They could return a buffer object
> > created from a string/array/NumPy array.

There is no way to do this. The buffer interface only returns pointers
to memory. There would be no place to return an intermediary object, nor
a way to retain the reference for it.

For example, your class instance quickly sets up a PyBufferObject with
the relevant data and returns that. The underlying C code must now hold
that reference *and* return a pointer to the calling code. Impossible.

Fredrik's open/close concept for buffer accesses would make this
possible, as long as clients are aware that any returned pointer is
valid only until the buffer_close call. The context argument he proposes
would hold the object reference.

Having class instances respond to the buffer interface is interesting,
but until more code attempts to *use* the interface, I'm not quite sure
of the utility...

>...
> Hmm, how about adding a writeable binary object to the core ?
> This would be useful for the __getwritebbuffer__() API because
> currently, I think, only mmap'ed files are useable as write
> buffers -- no other in-memory type. Perhaps buffer objects
> could be used for this purpose too, e.g. by having them
> allocate the needed memory chunk in case you pass None as
> object.

Yes, this would be very good. I would recommend that you pass an
integer, however, rather than None. You need to tell it the size of the
buffer to allocate. Since buffer(5) has no meaning at the moment,
altering the semantics to include this form would not be a problem.

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

On Sat, 14 Aug 1999, Greg Stein wrote:

> Having class instances respond to the buffer interface is interesting,
> but until more code attempts to *use* the interface, I'm not quite sure
> of the utility...

Well, here's an example from my work today. Maybe someone can suggest an
alternative that I haven't seen.

I'm using buffer objects to pass pointers to structs back and forth
between Python and Windows (Win32's GUI scheme involves sending messages
to functions with, oftentimes, addresses of structs as arguments, and
expect the called function to modify the struct directly -- similarly, I
must call Win32 functions w/ pointers to memory that Windows will modify,
and be able to read the modified memory). With 'raw' buffer object
manipulation (after exposing the PyBuffer_FromMemoryReadWrite call to
Python), this works fine [*]. So far, no instances.

I also have a class which allows the user to describe the buffer memory
layout in a natural way given the C struct, and manipulate the buffer
layout w/ getattr/setattr. For example:

class Win32MenuItemStruct(AutoStruct):
#
# for each slot, specify type (maps to a struct.pack specifier),
# name (for setattr/getattr behavior) and optional defaults.
#
table = [.(UINT, 'cbSize', AutoStruct.sizeOfStruct),
(UINT, 'fMask', MIIM_STRING | MIIM_TYPE | MIIM_ID),
(UINT, 'fType', MFT_STRING),
(UINT, 'fState', MFS_ENABLED),
(UINT, 'wID', None),
(HANDLE, 'hSubMenu', 0),
(HANDLE, 'hbmpChecked', 0),
(HANDLE, 'hbmpUnchecked', 0),
(DWORD, 'dwItemData', 0),
(LPSTR, 'name', None),
(UINT, 'cch', 0)]

AutoStruct has machinery which allows setting of buffer slices by slot
name, conversion of numeric types, etc. This is working well.

The only hitch is that to send the buffer to the SWIG'ed function call, I
have three options, none ideal:

1) define a __str__ method which makes a string of the buffer and pass
that to the function which expects an "s#" argument. This send
a copy of the data, not the address. As a result, this works
well for structs which I create from scratch as long as I don't need
to see any changes that Windows might have performed on the memory.

2) send the instance but make up my own 'get-the-instance-as-buffer'
API -- complicates extension module code.

3) send the buffer attribute of the instance instead of the instance --
complicates Python code, and the C code isn't trivial because there
is no 'buffer' typecode for PyArg_ParseTuple().

If I could define an

def __aswritebuffer__

and if there was a PyArg_ParseTuple() typecode associated with read/write
buffers (I nominate 'w'!), I believe things would be simpler -- I could
then send the instance, specify in the PyArgParse_Tuple that I want a
pointer to memory, and I'd be golden.

What did I miss?

--david

[*] I feel naughty modifying random bits of memory from Python, but Bill
Gates made me do it!

Greg Stein wrote:
>
> [.me suggesting new __XXX__ methods on Python instances to provide
> the buffer slots to Python programmers]
>
> Having class instances respond to the buffer interface is interesting,
> but until more code attempts to *use* the interface, I'm not quite sure
> of the utility...

Well, there already is lots of code supporting the interface,
e.g. fp.write(), socket.write() etc. Basically all streaming
interfaces I guess. So these APIs could be used to "write"
the object directly into a file.

> >...
> > Hmm, how about adding a writeable binary object to the core ?
> > This would be useful for the __getwritebbuffer__() API because
> > currently, I think, only mmap'ed files are useable as write
> > buffers -- no other in-memory type. Perhaps buffer objects
> > could be used for this purpose too, e.g. by having them
> > allocate the needed memory chunk in case you pass None as
> > object.
>
> Yes, this would be very good. I would recommend that you pass an
> integer, however, rather than None. You need to tell it the size of the
> buffer to allocate. Since buffer(5) has no meaning at the moment,
> altering the semantics to include this form would not be a problem.

I was thinking of using the existing buffer(object,offset,size)
constructor... that's why I took None as object. offset would
then always be 0 and size gives the size of the memory chunk
to allocate. Of course, buffer(size) would look nicer, but it seems
a rather peculiar interface definition to say: ok, if you pass
a real Python integer, we'll take that as size. Who knows, maybe
at some in the future, you want to "write" integers via the
buffer interface too... then you'd probably also want to write
None... so how about a new builtin writebuffer(size) ?

Also, I think it would make sense to extend buffers to have
methods and attributes:

.writeable - attribute that tells whether the buffer is writeable
.chardata - true iff the getcharbuffer slot is available
.asstring() - return the buffer as Python string object

--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: 138 days left
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/

David Ascher wrote:
>
> On Sat, 14 Aug 1999, Greg Stein wrote:
>
> > Having class instances respond to the buffer interface is interesting,
> > but until more code attempts to *use* the interface, I'm not quite sure
> > of the utility...
>
> Well, here's an example from my work today. Maybe someone can suggest an
> alternative that I haven't seen.
>
> I'm using buffer objects to pass pointers to structs back and forth
> between Python and Windows (Win32's GUI scheme involves sending messages
> to functions with, oftentimes, addresses of structs as arguments, and
> expect the called function to modify the struct directly -- similarly, I
> must call Win32 functions w/ pointers to memory that Windows will modify,
> and be able to read the modified memory). With 'raw' buffer object
> manipulation (after exposing the PyBuffer_FromMemoryReadWrite call to
> Python), this works fine [*]. So far, no instances.

So that's why you were suggesting that struct.pack returns a buffer
rather than a string ;-)

Actually, I think you could use arrays to do the trick right now,
because they are writeable (unlike strings). Until creating
writeable buffer objects becomes possible that is...

> I also have a class which allows the user to describe the buffer memory
> layout in a natural way given the C struct, and manipulate the buffer
> layout w/ getattr/setattr. For example:
>
> class Win32MenuItemStruct(AutoStruct):
> #
> # for each slot, specify type (maps to a struct.pack specifier),
> # name (for setattr/getattr behavior) and optional defaults.
> #
> table = [.(UINT, 'cbSize', AutoStruct.sizeOfStruct),
> (UINT, 'fMask', MIIM_STRING | MIIM_TYPE | MIIM_ID),
> (UINT, 'fType', MFT_STRING),
> (UINT, 'fState', MFS_ENABLED),
> (UINT, 'wID', None),
> (HANDLE, 'hSubMenu', 0),
> (HANDLE, 'hbmpChecked', 0),
> (HANDLE, 'hbmpUnchecked', 0),
> (DWORD, 'dwItemData', 0),
> (LPSTR, 'name', None),
> (UINT, 'cch', 0)]
>
> AutoStruct has machinery which allows setting of buffer slices by slot
> name, conversion of numeric types, etc. This is working well.
>
> The only hitch is that to send the buffer to the SWIG'ed function call, I
> have three options, none ideal:
>
> 1) define a __str__ method which makes a string of the buffer and pass
> that to the function which expects an "s#" argument. This send
> a copy of the data, not the address. As a result, this works
> well for structs which I create from scratch as long as I don't need
> to see any changes that Windows might have performed on the memory.
>
> 2) send the instance but make up my own 'get-the-instance-as-buffer'
> API -- complicates extension module code.
>
> 3) send the buffer attribute of the instance instead of the instance --
> complicates Python code, and the C code isn't trivial because there
> is no 'buffer' typecode for PyArg_ParseTuple().
>
> If I could define an
>
> def __aswritebuffer__
>
> and if there was a PyArg_ParseTuple() typecode associated with read/write
> buffers (I nominate 'w'!), I believe things would be simpler -- I could
> then send the instance, specify in the PyArgParse_Tuple that I want a
> pointer to memory, and I'd be golden.
>
> What did I miss?

Just a naming thingie: __getwritebuffer__ et al. would map to the
C interfaces more directly.

The new typecode "w#" for writeable buffer style objects is a good idea
(it should only work on single segment buffers).

--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: 138 days left
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/

On Sun, 15 Aug 1999, M.-A. Lemburg wrote:

> Actually, I think you could use arrays to do the trick right now,
> because they are writeable (unlike strings). Until creating
> writeable buffer objects becomes possible that is...

No, because I can't make an array around existing memory which Win32
allocates before I get to it.

> Just a naming thingie: __getwritebuffer__ et al. would map to the
> C interfaces more directly.

Whatever.

> The new typecode "w#" for writeable buffer style objects is a good idea
> (it should only work on single segment buffers).

Indeed.

--david

> On Sun, 15 Aug 1999, M.-A. Lemburg wrote:
>
> > Actually, I think you could use arrays to do the trick right now,
> > because they are writeable (unlike strings). Until creating
> > writeable buffer objects becomes possible that is...
>
> No, because I can't make an array around existing memory which Win32
> allocates before I get to it.

Would adding a buffer interface to cobject solve your problem? Cobject is
described as being used for passing C objects between Python modules, but I've
always thought of it as passing C objects from one C routine to another C
routine through Python, which doesn't necessarily understand what the object
is all about.

That latter description seems to fit your bill quite nicely.
--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

On Mon, 16 Aug 1999, Jack Jansen wrote:

> Would adding a buffer interface to cobject solve your problem? Cobject is
> described as being used for passing C objects between Python modules, but I've
> always thought of it as passing C objects from one C routine to another C
> routine through Python, which doesn't necessarily understand what the object
> is all about.
>
> That latter description seems to fit your bill quite nicely.

It's an interesting idea, but it wouldn't do as it is, as I'd need the
ability to create a CObject given a memory location and a size. Also, I
am not expected to free() the memory, which would happen when the CObject
got GC'ed.

(BTW: I am *not* arguing that PyBuffer_FromReadWriteMemory() should be
exposed by default. I'm happy with exposing it in my little extension
module for my exotic needs.)

--david