Mailing List Archive

Re: perl5 and files > 2gig
[.courtesy cc of this posting sent to cited author via email]

In comp.lang.perl.misc,
garyf@chimera-f.convex.com (Gary Faulkner) writes:
:Hello all,
:
:Since I didn't seem to see this in a FAQ or any of the multitudes of postings
:in this newsgroup, I figured I'd just ask the group.
:
:Has anyone done any work to get perl5 to handle (from all aspects) files
:greater than 2 gig (i.e. 64 bit off_t and stat->st_size)?? I have managed to
:modify the source so that the appropriate calls and stat structures/off_t types
:are used internally; however, I cannot seem to get perl to recognize that the
:arguements to seek(), the return from tell(), or the st_size member of the
:array returned by stat()/fstat() are 64 bit quantities.
:
:(yes, the O/S's I am porting to support these values)...
:
:For example, if I do a :
:
:@statstr = stat("/largefiles/3gigfile");
:print ("@statstr\n");
:
:The size member is invalid (I'm sure because I'm only picking up the low order
:word of the stat->st_size member).
:
:I would appreciate pointers from anyone (even to a missed piece of
:documentation) on what to do to externalize these values.

Fascinating.

I'll bet that the problem is that in your sys/stat.h, you have st_size has
an off_t, and that an off_t is acutally a 64 bit quantity, like a long
long, not just a 32 one. And yet the perl source, in pp_sys.c around line
1741 says:

PUSHs(sv_2mortal(newSViv((I32)statcache.st_size)));

which is going to be a problem. I don't know of a good solution, either.
I suppose tell and seek and stuff take I64 off_t's as well for you.

CC'd to perl-porters, as this really does seem a porting issue.

--tom

--
Tom Christiansen Perl Consultant, Gamer, Hiker tchrist@mox.perl.com

if (shm == (char *)-1) /* I hate System V IPC, I really do */
--Larry Wall, from doio.c in the v5.0 perl distribution
Re: perl5 and files > 2gig [ In reply to ]
> I'll bet that the problem is that in your sys/stat.h, you have st_size has
> an off_t, and that an off_t is acutally a 64 bit quantity, like a long
> long, not just a 32 one. And yet the perl source, in pp_sys.c around line
> 1741 says:
>
> PUSHs(sv_2mortal(newSViv((I32)statcache.st_size)));
>
> which is going to be a problem. I don't know of a good solution, either.
> I suppose tell and seek and stuff take I64 off_t's as well for you.

actually, no. Both the st_size of args/returns from seek/tell/fposset/etc. are
special (actually, fposset takes a pos_t which is a bogus 2-long structure on
one of the platforms). Anyway, they are all off64_t, or fpos64_t (w/ the
exception noted above).

After making the changes to use the correct types (cvxstat and/or struct
stat64), and use the correct calls (cvxstat and stat64) I was able to at least
get the basic functionatlity to work correctly for large files. For example,
the file test operators (-f, -d, -b, etc.) now work correctly, and a stat()
call will actually work (although the size is bogus). I was just hoping for a
pointer on how to get the i64 values in and out of perl.

--
Gary Faulkner e-mail: garyf@convex.com
Data Management Group Phone: (214) 497-4715
Convex Computer Corporation Fax: (214) 497-4500
3000 Waterview Parkway
Richardson, Texas 75083
Re: perl5 and files > 2gig [ In reply to ]
> From: Gary Faulkner <garyf@chimera-f.convex.com>
>
> > I'll bet that the problem is that in your sys/stat.h, you have st_size has
> > an off_t, and that an off_t is acutally a 64 bit quantity, like a long
> > long, not just a 32 one. And yet the perl source, in pp_sys.c around line
> > 1741 says:
> >
> > PUSHs(sv_2mortal(newSViv((I32)statcache.st_size)));
> >
> > which is going to be a problem. I don't know of a good solution, either.
> > I suppose tell and seek and stuff take I64 off_t's as well for you.
>
> actually, no. Both the st_size of args/returns from seek/tell/fposset/etc. are
> special (actually, fposset takes a pos_t which is a bogus 2-long structure on
> one of the platforms). Anyway, they are all off64_t, or fpos64_t (w/ the
> exception noted above).
>
> After making the changes to use the correct types (cvxstat and/or struct
> stat64), and use the correct calls (cvxstat and stat64) I was able to at least
> get the basic functionatlity to work correctly for large files. For example,
> the file test operators (-f, -d, -b, etc.) now work correctly, and a stat()
> call will actually work (although the size is bogus). I was just hoping for a
> pointer on how to get the i64 values in and out of perl.

Maybe making sure IV is typed as a 64 bit int and changing code fragments like:

- PUSHs(sv_2mortal(newSViv((I32)statcache.st_size)));
+ PUSHs(sv_2mortal(newSViv((IV)statcache.st_size)));

That I32 cast was wrong anyway.

Tim.

(reading p5p by string and mirrors - I hope you get this sometime soon ;-)
Re: perl5 and files > 2gig [ In reply to ]
Maybe making sure IV is typed as a 64 bit int and changing code fragments like:

- PUSHs(sv_2mortal(newSViv((I32)statcache.st_size)));
+ PUSHs(sv_2mortal(newSViv((IV)statcache.st_size)));

That I32 cast was wrong anyway.

i don't want IV to be 64 bits on my netbsd/sparc machine but i do
want 64 bit off_t's available. this isn't the correct answer :-(
Re: perl5 and files > 2gig [ In reply to ]
> From: matthew green <Matthew.Green@fulcrum.com.au>
>
> Maybe making sure IV is typed as a 64 bit int and changing code fragments like:
>
> - PUSHs(sv_2mortal(newSViv((I32)statcache.st_size)));
> + PUSHs(sv_2mortal(newSViv((IV)statcache.st_size)));
>
> That I32 cast was wrong anyway.
>
> i don't want IV to be 64 bits on my netbsd/sparc machine but i do
> want 64 bit off_t's available. this isn't the correct answer :-(

I think IV is your only option but I'd be happy to be corrected.

Tim.
Re: perl5 and files > 2gig [ In reply to ]
: > i don't want IV to be 64 bits on my netbsd/sparc machine but i do
: > want 64 bit off_t's available. this isn't the correct answer :-(
:
: I think IV is your only option but I'd be happy to be corrected.

An NV gives about 5 more orders of magnitude over I32 before you start
clobbering integers, and should work for the next 10 years or so.

Larry
Re: perl5 and files > 2gig [ In reply to ]
> From: Larry Wall <lwall@scalpel.netlabs.com>
>
> : > i don't want IV to be 64 bits on my netbsd/sparc machine but i do
> : > want 64 bit off_t's available. this isn't the correct answer :-(
> :
> : I think IV is your only option but I'd be happy to be corrected.
>
> An NV gives about 5 more orders of magnitude over I32 before you start
> clobbering integers, and should work for the next 10 years or so.
>
> Larry

For some reason I had integers on my mind (and this is the guy who
hacked some code to pack strings into doubles in a safe way as a
workaround on one of our apps ;-)

Thanks.

Tim.