Mailing List Archive

Mark Hammond wrote:
> ...
> Therefore, I would like to propose these functions to be added to
> abstract.c:
>
> int PyObject_GetBufferSize();
> void *PyObject_GetReadWriteBuffer(); /* or "char *"? */
> const void *PyObject_GetReadOnlyBuffer();
>
> Although equivalent functions exist for the buffer object, I can't see the
> equivalent abstract implementations - ie, that work with any object
> supporting the protocol.
>
> Im willing to provide a patch if there is agreement a) the general idea is
> good, and b) my specific spelling of the idea is OK (less likely -
> PyBuffer_* seems better, but loses any implication of being abstract?).

Marc-Andre proposed exactly the same thing back at the end of March (to
me and Guido). The two of us hashed out some of the stuff and M.A. came
up with a full patch for the stuff. Guido was relatively non-committal
at the point one way or another, but said they seemed fine. It appears
the stuff never made it into source control.

If Marc-Andre can resurface the final proposal/patch, then we'd be set.

Until then: use the bufferprocs :-)

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

Greg Stein wrote:
>
> Mark Hammond wrote:
> > ...
> > Therefore, I would like to propose these functions to be added to
> > abstract.c:
> >
> > int PyObject_GetBufferSize();
> > void *PyObject_GetReadWriteBuffer(); /* or "char *"? */
> > const void *PyObject_GetReadOnlyBuffer();
> >
> > Although equivalent functions exist for the buffer object, I can't see the
> > equivalent abstract implementations - ie, that work with any object
> > supporting the protocol.
> >
> > Im willing to provide a patch if there is agreement a) the general idea is
> > good, and b) my specific spelling of the idea is OK (less likely -
> > PyBuffer_* seems better, but loses any implication of being abstract?).
>
> Marc-Andre proposed exactly the same thing back at the end of March (to
> me and Guido). The two of us hashed out some of the stuff and M.A. came
> up with a full patch for the stuff. Guido was relatively non-committal
> at the point one way or another, but said they seemed fine. It appears
> the stuff never made it into source control.
>
> If Marc-Andre can resurface the final proposal/patch, then we'd be set.

Below is the code I currently use. I don't really remember if this
is what Greg and I discussed a while back, but I'm sure he'll
correct me ;-) Note that you the buffer length is implicitly
returned by these APIs.

/* Takes an arbitrary object which must support the character (single
segment) buffer interface and returns a pointer to a read-only
memory location useable as character based input for subsequent
processing.

buffer and buffer_len are only set in case no error
occurrs. Otherwise, -1 is returned and an exception set.

*/

static
int PyObject_AsCharBuffer(PyObject *obj,
const char **buffer,
int *buffer_len)
{
PyBufferProcs *pb = obj->ob_type->tp_as_buffer;
const char *pp;
int len;

if ( pb == NULL ||
pb->bf_getcharbuffer == NULL ||
pb->bf_getsegcount == NULL ) {
PyErr_SetString(PyExc_TypeError,
"expected a character buffer object");
goto onError;
}
if ( (*pb->bf_getsegcount)(obj,NULL) != 1 ) {
PyErr_SetString(PyExc_TypeError,
"expected a single-segment buffer object");
goto onError;
}
len = (*pb->bf_getcharbuffer)(obj,0,&pp);
if (len < 0)
goto onError;
*buffer = pp;
*buffer_len = len;
return 0;

onError:
return -1;
}

/* Same as PyObject_AsCharBuffer() except that this API expects a
readable (single segment) buffer interface and returns a pointer
to a read-only memory location which can contain arbitrary data.

buffer and buffer_len are only set in case no error
occurrs. Otherwise, -1 is returned and an exception set.

*/

static
int PyObject_AsReadBuffer(PyObject *obj,
const void **buffer,
int *buffer_len)
{
PyBufferProcs *pb = obj->ob_type->tp_as_buffer;
void *pp;
int len;

if ( pb == NULL ||
pb->bf_getreadbuffer == NULL ||
pb->bf_getsegcount == NULL ) {
PyErr_SetString(PyExc_TypeError,
"expected a readable buffer object");
goto onError;
}
if ( (*pb->bf_getsegcount)(obj,NULL) != 1 ) {
PyErr_SetString(PyExc_TypeError,
"expected a single-segment buffer object");
goto onError;
}
len = (*pb->bf_getreadbuffer)(obj,0,&pp);
if (len < 0)
goto onError;
*buffer = pp;
*buffer_len = len;
return 0;

onError:
return -1;
}

/* Takes an arbitrary object which must support the writeable (single
segment) buffer interface and returns a pointer to a writeable
memory location in buffer of size buffer_len.

buffer and buffer_len are only set in case no error
occurrs. Otherwise, -1 is returned and an exception set.

*/

static
int PyObject_AsWriteBuffer(PyObject *obj,
void **buffer,
int *buffer_len)
{
PyBufferProcs *pb = obj->ob_type->tp_as_buffer;
void*pp;
int len;

if ( pb == NULL ||
pb->bf_getwritebuffer == NULL ||
pb->bf_getsegcount == NULL ) {
PyErr_SetString(PyExc_TypeError,
"expected a writeable buffer object");
goto onError;
}
if ( (*pb->bf_getsegcount)(obj,NULL) != 1 ) {
PyErr_SetString(PyExc_TypeError,
"expected a single-segment buffer object");
goto onError;
}
len = (*pb->bf_getwritebuffer)(obj,0,&pp);
if (len < 0)
goto onError;
*buffer = pp;
*buffer_len = len;
return 0;

onError:
return -1;
}


--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: 150 days left
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/

Why not pass the index to the As*Buffer routines as well and make getsegcount
available too? Then you could code things like
for(i=0; i<PyObject_GetBufferCount(obj); i++) {
if ( PyObject_AsCharBuffer(obj, &buf, &count, i) < 0 )
return -1;
write(fp, buf, count);
}

--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

Jack Jansen wrote:
>
> Why not pass the index to the As*Buffer routines as well and make getsegcount
> available too? Then you could code things like
> for(i=0; i<PyObject_GetBufferCount(obj); i++) {
> if ( PyObject_AsCharBuffer(obj, &buf, &count, i) < 0 )
> return -1;
> write(fp, buf, count);
> }

Simply because multiple segments hasn't been seen. All objects
supporting the buffer interface have a single segment. IMO, it is best
to drop the argument to make typical usage easier. For handling multiple
segments, a caller can use the raw interface rather than the handy
functions.

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

Mark Hammond wrote:
>
> Hi all,
> Im trying to slowly wean myself over to the buffer interfaces.

OK, I'll bite. Where is the buffer interface documented? I found references
to it in various places (e.g. built-in buffer()) but didn't find the interface
itself.

Jim

--
Jim Fulton mailto:jim@digicool.com Python Powered!
Technical Director (888) 344-4332 http://www.python.org
Digital Creations http://www.digicool.com http://www.zope.org

Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission. Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.

Jack Jansen wrote:
>
> Why not pass the index to the As*Buffer routines as well and make getsegcount
> available too? Then you could code things like
> for(i=0; i<PyObject_GetBufferCount(obj); i++) {
> if ( PyObject_AsCharBuffer(obj, &buf, &count, i) < 0 )
> return -1;
> write(fp, buf, count);
> }

Well, just like Greg said, this is not much different than using the
buffer interface directly. While the above would be a handy
PyObject_WriteAsBuffer() kind of helper, I don't think that this
is really used all that much. E.g. in mxODBC I use the APIs
for accessing the raw char data in a buffer: the pointer is passed
directly to the ODBC APIs without copying, which makes things
quite fast. IMHO, this is the greatest advantage of the buffer
interface.

--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: 150 days left
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/

Jim Fulton wrote:
>
> Mark Hammond wrote:
> >
> > Hi all,
> > Im trying to slowly wean myself over to the buffer interfaces.
>
> OK, I'll bite. Where is the buffer interface documented? I found references
> to it in various places (e.g. built-in buffer()) but didn't find the interface
> itself.

I guess it's a read-the-source feature :-) Objects/bufferobject.c
and Include/object.h provide a start. Objects/stringobject.c has
a "sample" implementation.

--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: 150 days left
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/

Greg Stein writes:
> Until then: use the bufferprocs :-)

Greg,
On the topic of the buffer interface: Have you written documentation
for this that I can include in the API reference? Bugging you about
this is on my to-do list. ;-)


-Fred

--
Fred L. Drake, Jr. <fdrake@acm.org>
Corporation for National Research Initiatives

> > Why not pass the index to the As*Buffer routines as well and make getsegcount
> > available too?
>
> Simply because multiple segments hasn't been seen. All objects
> supporting the buffer interface have a single segment.

Hmm. And I went out of my way to include this stupid multi-buffer stuff
because the NumPy folks said they couldn't live without it (and one of the
reasons for the buffer stuff was to allow NumPy arrays, which may be
discontiguous, to be written efficiently).

Can someone confirm that the Numeric stuff indeed doesn't use this?
--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

On Tue, 3 Aug 1999, Greg Stein wrote:

> Simply because multiple segments hasn't been seen. All objects
> supporting the buffer interface have a single segment. IMO, it is best

FYI, if/when NumPy objects support the buffer API, they will require
multiple-segments.

On Tue, 3 Aug 1999, Jack Jansen wrote:

> > > Why not pass the index to the As*Buffer routines as well and make getsegcount
> > > available too?
> >
> > Simply because multiple segments hasn't been seen. All objects
> > supporting the buffer interface have a single segment.
>
> Hmm. And I went out of my way to include this stupid multi-buffer stuff
> because the NumPy folks said they couldn't live without it (and one of the
> reasons for the buffer stuff was to allow NumPy arrays, which may be
> discontiguous, to be written efficiently).
>
> Can someone confirm that the Numeric stuff indeed doesn't use this?

/usr/LLNLDistribution/Numerical/Include$ grep buffer *.h
/usr/LLNLDistribution/Numerical/Include$

Yes. =)

See the other thread on low-overhead pickling.

But again, *if* multiarrays supported the buffer interface, they'd have to
use the multi-segment feature (repeating myself).

--david

> > Simply because multiple segments hasn't been seen. All objects
> > supporting the buffer interface have a single segment. IMO, it is best
>
> FYI, if/when NumPy objects support the buffer API, they will require
> multiple-segments.

same goes for PIL. in the worst case, there's
one segment per line.

...

on the other hand, I think something is missing from
the buffer design; I definitely don't like that people
can write and marshal objects that happen to
implement the buffer interface, only to find that
Python didn't do what they expected...

>>> import unicode
>>> import marshal
>>> u = unicode.unicode
>>> s = u("foo")
>>> data = marshal.dumps(s)
>>> marshal.loads(data)
'f\000o\000o\000'
>>> type(marshal.loads(data))
<type 'string'>

as for PIL, I would also prefer if the exported buffer
corresponded to what you get from im.tostring(). iirc,
that cannot be done -- I cannot export via a temporary
memory buffer, since there's no way to know when to
get rid of it...

</F>