Mailing List Archive

Questions concerning PERLAPI/PERLGUTS
Dear Perl-creators, -fans and -supporters:

Recently I tried (and eventually succeeded) to include a C library of my own
(for all sorts of date calculations, in agreement with the ISO/R 2015-1971
and DIN 1355 norms) as a (dynamically linked) package into Perl 5.

Unfortunately, it was rather difficult, because it was my first time and
because the PERLAPI/PERLGUTS manual doesn't mention or explain a couple
of important items.

I would be very glad if somebody could answer the questions I still have!
(See further below)

(The answers will - no doubt - be also very valuable for anyone trying the
same kind of endeavour)

I'll also include a list of traps I fell into:

1) Users should be warned that if they are trying to build a package
"package.pm" (along with "package.xs"), any file named "package.c"
gets overwritten.

So your C library should have a different name.

The simplest way to include the C library file into the "package.c"
file (that eventually gets compiled into the dynamically linkable
shared library), is to include it in the header of the "package.xs"
file, like this:

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "my_c_lib.c"

The file "my_c_lib.c" should in turn include everything it needs to
get itself compiled.

2) You'll need to supply a file named ".packlist". Use the ".packlist"
files of other extensions as a model.

3) You might need to supply a "typemap" file, for instance in case you
used "typedef"s in your code. You'll find the main typemap file at
the following location:

/..../yourperl5distribution/lib/ExtUtils/typemap

Use this as a model for your own; you will find Perl equivalents for
most of your basic data types.

Now the questions:

1) Suppose the Perl interface of a C function is to be as follows:

($cc,$yy,$mm,$dd) = decode($date);

By how many elements do I need to extend the stack using the EXTEND(sp,num)
macro in this case, if the return values are to be pushed onto the stack
with the PUSHs(SV*) macro? (In detail: "PUSHs(sv_2mortal(newSVnv(cc)));")

By the number of values returned (i.e., EXTEND(sp,4)), or by the number of
values returned MINUS the number of input values, in this case 4 - 1 = 3?

The examples in PERLAPI (for instance, pages 168 middle and 169 bottom)
are not consistent concerning EXTEND(sp,num)! (See also following pages)

2) The PERLGUTS manual fails to specify what has to be supplied as the second
argument to the function

SV* newSVpv(char*, int);

From examples found in the Perl C sources, it seems to me that the length
of the string pointed to by the first argument has to be given, but in
many examples, this value was zero!

Also, it is unclear to me wether the STRING pointed to by the first argu-
ment is COPIED into newly allocated memory (allowing to free() the origi-
nal string), or if the newly created variable only contains a copy of the
POINTER (free()ing the original string would be a bad idea in this case!).

3) I didn't understand the real difference between EXPORT and EXPORT_OK.

Is it possible to call functions of a package without explicitly referen-
cing them, if they are exported by the package by one of the two state-
ments above, and imported by a "use package qw(function1 function2);"
statement in your application?

Like this, for instance:

#!/opt/bin/perl5
use datecalc qw( decode(date) );

...

($cc,$yy,$mm,$dd) = decode($date);

instead of:

($cc,$yy,$mm,$dd) = datecalc::decode($date);

Also, I discovered that I didn't need to declare the functions in the "use"
statement in order to use them (they were exported by the package via the
"EXPORT_OK" statement).

Is this a bug or a feature?

If it's a feature, what do I need the declaration of functions in the "use"
statement for?

My "EXPORT_OK" statement in "datecalc.pm" is:

@EXPORT = qw();

@EXPORT_OK = qw( leap(year) encode(yy,mm,dd) decode(date) valid_date(date)
date_string(date) check_date(year,mm,dd) calc_days(year,mm,dd)
dates_difference(year1,mm1,dd1,year2,mm2,dd2) day_of_week(year,mm,dd)
calc_new_date(year,mm,dd,offset) date_to_short(year,mm,dd)
date_to_string(year,mm,dd) week_number(year,mm,dd) first_in_week(week,year)
weeks_in_year(year) decode_date(buffer) day_short_tab(dd) day_name_tab(dd)
month_short_tab(mm) month_name_tab(mm) );

Maybe there is something wrong with this? (Am I missing something obvious?)

Any help is greatly appreciated (as always ;-) )
and will be highly praised! ;-)

Many thanks in advance,
have fun!

Yours,
--
Steffen Beyer
mailto:sb@sdm.de |s |d &|m | software design & management GmbH&Co.KG
phone: +49 89 63812-244 | | | | Thomas-Dehler-Str. 27
fax: +49 89 63812-150 | | | | 81737 Munich, Germany.
Re: Questions concerning PERLAPI/PERLGUTS [ In reply to ]
> From: sb@sdm.de (Steffen Beyer)


> Unfortunately, it was rather difficult, because it was my first time and
> because the PERLAPI/PERLGUTS manual doesn't mention or explain a couple
> of important items.

Steffen, thanks for the note. We're hoping to have more of this stuff
cleared up for 5.002.


> I'll also include a list of traps I fell into:
>
> 1) Users should be warned that if they are trying to build a package
> "package.pm" (along with "package.xs"), any file named "package.c"
> gets overwritten.

thanks.


> 2) You'll need to supply a file named ".packlist". Use the ".packlist"
> files of other extensions as a model.

Anytime you feel like you're messing with some nitpicking, annoying,
obscure, "what the heck is this", goofy, unnatural gizmos -- there's
probably a better way. You should forget you ever discovered .packlist.
(I didn't know about it until you made me go grepping for it.)

We'll have to teach you how to use MakeMaker. I'll attach a simplified
recipe at the end. We're already taking steps to correct this deficiency
for 5.002.


> 3) You might need to supply a "typemap" file, for instance in case you
> used "typedef"s in your code. You'll find the main typemap file at
> the following location:
>
> /..../yourperl5distribution/lib/ExtUtils/typemap
>
> Use this as a model for your own; you will find Perl equivalents for
> most of your basic data types.

See "The Typemap" in perlapi, especially the second and third paragraphs.



> 1) Suppose the Perl interface of a C function is to be as follows:
>
> ($cc,$yy,$mm,$dd) = decode($date);
>
> By how many elements do I need to extend the stack using the EXTEND(sp,num)
> macro in this case, if the return values are to be pushed onto the stack
> with the PUSHs(SV*) macro? (In detail: "PUSHs(sv_2mortal(newSVnv(cc)));")
>
> By the number of values returned (i.e., EXTEND(sp,4)), or by the number of
> values returned MINUS the number of input values, in this case 4 - 1 = 3?
>
> The examples in PERLAPI (for instance, pages 168 middle and 169 bottom)
> are not consistent concerning EXTEND(sp,num)! (See also following pages)

That manpage has only 20 pages. I think you're talking about the example in
the section "The PPCODE: Keyword".

"The EXTEND() macro is used to make room on the argument stack for 2
return values."

I don't see that EXTEND is being used inconsistent with its description, but
I'll admit that I don't know if the example in perlapi needed an
EXTEND(sp,2) or if EXTEND(sp,1) would have been sufficient. Hopefully
someone else on this list can see through the haze and enlighten us.


> 2) The PERLGUTS manual fails to specify what has to be supplied as the second
> argument to the function
>
> SV* newSVpv(char*, int);
>
> From examples found in the Perl C sources, it seems to me that the length
> of the string pointed to by the first argument has to be given, but in
> many examples, this value was zero!

In perlguts, under "Working with SV's", after that function is mentioned:

"Notice that you can choose to specify the length of the string to
be assigned by using sv_setpvn or newSVpv, or you may allow Perl to
calculate the length by using sv_setpv or specifying 0 as the second
argument to newSVpv."


> Also, it is unclear to me wether the STRING pointed to by the first argu-
> ment is COPIED into newly allocated memory (allowing to free() the origi-
> nal string), or if the newly created variable only contains a copy of the
> POINTER (free()ing the original string would be a bad idea in this case!).

I think when Jeff wrote "An SV can be created and loaded" he was trying to
imply "copy". In most cases, the string is copied into the SV.


> 3) I didn't understand the real difference between EXPORT and EXPORT_OK.

Your questions and examples for the Exporter are way off. I think Tim and
Tom need to handle this one. (I do remember once mentioning on this list
that we documented the complex stuff but skipped the basics.)


Dean


-------
>Newsgroups: comp.lang.perl.misc
>Subject: Re: linking 'c' libraries with perl
>Summary:
>Expires:
>References: <41lj5q$b9q@news1.halcyon.com>
>From: Dean Roehrich
>Sender:
>Followup-To:
>Distribution:
>Organization: Cray Research, Inc.
>Keywords:
>Cc:
>
>In article <41lj5q$b9q@news1.halcyon.com>, Scott Moore <slm> wrote:
>>I am looking for any information about linking c libraries with perl5
>>I have tried a few different ways but nothing seems to work. Has anyone
>>out there done this and can give me some help?????
>
>To link a C library to Perl5 you'll need to build a Perl extension. The
>extension's job is to let Perl5 access the C library. The perlapi manpage
>addresses the syntax of the XS language used when writing an extension.
>There are some gaps in the Perl documentation because the perlapi manpage
>was written well before the entire extension-building process was well
>understood and defined.
>
>Here's a version of something I sent to someone else a few weeks ago. This
>isn't comprehensive, but kinda fills most gaps...
>
>Always begin a new extension by running the h2xs tool. This will generate
>templates for the PM, XS, MANIFEST, and PL files. The Makefile.PL file is a
>Perl script and when run it will generate a Makefile. This makefile knows
>how to find and run xsubpp for your extension. When you type "make" your XS
>file will be run through xsubpp and a C file will be produced. Then the C
>file will be compiled. A simple example looks like this:
>
> $ h2xs -Afn Foo
> $ cd ext/Foo
> $ ls
> Foo.pm Foo.xs MANIFEST Makefile.PL
> $ perl5 Makefile.PL
> $ ls
> Foo.pm Foo.xs MANIFEST Makefile.PL Makefile
> $ <edit Foo.pm and Foo.xs to add your stuff>
> $ make
> <you will see xsubpp run on Foo.xs and you'll see the C compiler
> <run on Foo.c, and a bunch of other less-interesting junk
> <will happen.
>
>If your Perl5 was built with dynamic loading then the makefile will build a
>dynamically loadable extension. If you don't have dynamic loading then the
>makefile will build a static extension and should create a new Perl binary.
>The default behavior depends on what is available.
>
>For more information about h2xs consult its manpage, embedded in the
>source. Also consult the perlapi manpage for information about XS, xsubpp,
>and h2xs. If you have the simple things nailed and want to get fancy then
>consult the perlguts and perlcall manpages. For information about tweeking
>the Makefile.PL and Makefile consult the MakeMaker manpage, embedded in
>$PERL5LIB/ExtUtils/MakeMaker.pm.
>
>Dean
>roehrich@cray.com
>
Re: Questions concerning PERLAPI/PERLGUTS [ In reply to ]
: I don't see that EXTEND is being used inconsistent with its description, but
: I'll admit that I don't know if the example in perlapi needed an
: EXTEND(sp,2) or if EXTEND(sp,1) would have been sufficient. Hopefully
: someone else on this list can see through the haze and enlighten us.

That depends on where sp is pointing at the time. If it's pointing after
the arguments, you can subtract off the arguments. If it's pointing before
the arguments, you have to count them in. For your typical PPCODE, sp
has already been decremented, so you add in the arguments. For other
XS code, the sp has not been decremented yet. Alternately, don't base
it off of sp, but off of mark, which is guaranteed to point after
the arguments for either kind of code.

On the other hand, I'll say, "When in doubt, pick the higher number."
One unnecessary stack extension isn't going to kill you. And your
stack extension may make someone else not extend the stack later, in
which case it matters not a whit, for certain values of whit.

Note, though, that you don't have to call EXTEND at all if you know the
stack is long enough already. And you may know a little more than you
think you know. If you look at the code for POSIX::frexp, it has one
argument, and pushes two back on the stack. Yet it doesn't do an
EXTEND because we know that the stack also had the subroutine reference
on it (removed by pp_entersub) before this subroutine was called, so it
comes out even. So if you see EXTEND(sp, 1), it's probably useless.

Larry
Re: Questions concerning PERLAPI/PERLGUTS [ In reply to ]
Dean Roehrich writes:
> > 2) The PERLGUTS manual fails to specify what has to be supplied as the second
> > argument to the function
> >
> > SV* newSVpv(char*, int);
> >
> > From examples found in the Perl C sources, it seems to me that the length
> > of the string pointed to by the first argument has to be given, but in
> > many examples, this value was zero!
>
> In perlguts, under "Working with SV's", after that function is mentioned:
>
> "Notice that you can choose to specify the length of the string to
> be assigned by using sv_setpvn or newSVpv, or you may allow Perl to
> calculate the length by using sv_setpv or specifying 0 as the second
> argument to newSVpv."
>
>
> > Also, it is unclear to me wether the STRING pointed to by the first argu-
> > ment is COPIED into newly allocated memory (allowing to free() the origi-
> > nal string), or if the newly created variable only contains a copy of the
> > POINTER (free()ing the original string would be a bad idea in this case!).
>
> I think when Jeff wrote "An SV can be created and loaded" he was trying to
> imply "copy". In most cases, the string is copied into the SV.
>

The description of API is far from complete (50% at the best). setpv
will copy, but you may sv_usepvn to "steal" the malloced region.

Ilya
Re: Questions concerning PERLAPI/PERLGUTS [ In reply to ]
Re: Questions concerning PERLAPI/PERLGUTS [ In reply to ]
Re: Questions concerning PERLAPI/PERLGUTS [ In reply to ]
> From: sb@sdm.de (Steffen Beyer)
>
> Hello Ilya Zakharevich, in a previous mail you wrote:
> >
> > The description of API is far from complete (50% at the best). setpv
>
> What a pity! Including C libraries is most useful, and the effort invested
> already (h2xs, xsubpp etc.) is substantial!
>
> > will copy, but you may sv_usepvn to "steal" the malloced region.
>
> How can that work? Isn't there additional space required to store the length
> information? Or is this information stored in the SV itself, along with the
> pointer to the string?
>
> Is this documented somewhere?

In the source.

Seriously, if you want to work with perl internals, even just the notional
'api' then studying the major runtime (non-compiler) modules is a must.

It may be tough going at first but it will reap great rewards.

Online documentation is unlikely to ever convey all the finer points.

Tim.
Re: Questions concerning PERLAPI/PERLGUTS [ In reply to ]
Re: Questions concerning PERLAPI/PERLGUTS [ In reply to ]
> From: sb@sdm.de (Steffen Beyer)
>
> Hello Tim Bunce, in a previous mail you wrote:
> >
> > > Is this documented somewhere?
> >
> > In the source.
> >
> > Seriously, if you want to work with perl internals, even just the notional
> > 'api' then studying the major runtime (non-compiler) modules is a must.
> >
> > It may be tough going at first but it will reap great rewards.
> >
> > Online documentation is unlikely to ever convey all the finer points.
>
> That's good to know! Thanks for the hint!
>
> Could you just give me another little hint: The paths and names of the runtime
> sources (or where to start)?

Find the source tree somewhere (local or from a CPAn archive) and take a
look at (in a very loose kind of order):

sv.c
av.c
sv.h
perl.h

You should reread the perlapi and perlguts man pages before, during and after :-)
After that just follow you nose...

Don't expect to understand much to start with. Don't expect to understand
much more after several days. It's not _that_ easy.

Good luck.

Tim.
Re: Questions concerning PERLAPI/PERLGUTS [ In reply to ]
> From: sb@sdm.de (Steffen Beyer)

> > will copy, but you may sv_usepvn to "steal" the malloced region.
>
> How can that work? Isn't there additional space required to store the length
> information? Or is this information stored in the SV itself, along with the
> pointer to the string?

You get an SV. An SV always knows the length of the data it holds.
Sv_usepvn() uses the Renew() macro, which uses saferealloc, so your string
may get moved and your own copy of the pointer will become invalid. There's
a good chance that right about now we've gone way beyond what you need.

Just use newSVpv() and sv_setpv() and remember that they want to copy the
string into the SV.

Anytime you need to go beyond perlguts (um, sometimes even while using
perlguts) you should have the Perl source handy. In the case of figuring
out the requirements and effects of sv_usepvn, newSVpv, and sv_setpv, you'll
want the sv.c module.

> I am using the PerlDoc.ps documentation, maybe it differs from the online
> man-pages?

That PerlDoc thingy was created back in the 5.000 days, wasn't it?

Dean