Mailing List Archive

>2GB Data.fs files on FreeBSD
[.Cc'ed to python-dev from the zope-dev mailing list; trim your
follow-ups appropriately]

R. David Murray writes:
>So it looks like there is a problem using Zope with a large database
>no matter what the platform. Has anyone figured out how to fix this?
...
>But given the number of people who have said "use FreeBSD if you want
>big files", I'm really wondering about this. What if later I
>have an application where I really need a >2GB database?

Different system calls are used for large files, because you can no
longer use 32-bit ints to store file position. There's a
HAVE_LARGEFILE_SUPPORT #define that turns on the use of these
alternate system calls; see Python's configure.in for the test used to
detect when it should be turned on. You could just hack the generated
config.h to turn on large file support and recompile your copy of
Python, but if the configure.in test is incorrect, that should be
fixed.

The test is:
AC_MSG_CHECKING(whether to enable large file support)
if test "$have_long_long" = yes -a \
"$ac_cv_sizeof_off_t" -gt "$ac_cv_sizeof_long" -a \
"$ac_cv_sizeof_long_long" -ge "$ac_cv_sizeof_off_t"; then
AC_DEFINE(HAVE_LARGEFILE_SUPPORT)
AC_MSG_RESULT(yes)
else
AC_MSG_RESULT(no)
fi

I thought you have to use the loff_t type instead of off_t; maybe this
test should check for it instead? Anyone know anything about large
file support?

--
A.M. Kuchling http://starship.python.net/crew/amk/
When I dream, sometimes I remember how to fly. You just lift one leg, then you
lift the other leg, and you're not standing on anything, and you can fly.
-- Chloe Russell, in SANDMAN #43: "Brief Lives:3"
Re: [Zope-dev] >2GB Data.fs files on FreeBSD [ In reply to ]
On Thu, 13 Apr 2000, Andrew M. Kuchling wrote:
> longer use 32-bit ints to store file position. There's a
> HAVE_LARGEFILE_SUPPORT #define that turns on the use of these
> alternate system calls; see Python's configure.in for the test used to

I just looked in my python config.h on my FreeBSD system, and I see:

#define HAVE_LARGEFILE_SUPPORT 1

So it looks like it is on, and it seems to me the problem could be
in either Python or FileStorage.py in Zope.

This is a Zope 2.1.2 system (but I diffed filestorage.py against
the 2.1.6 version and didn't see any relevant changes) running
on a FreeBSD 3.1 system. A make test in Python passed all tests,
but I don't know if large file support is tested by the tests.

--RDM

PS: anyone from the python list replying to this please CC me as
I am not on that list.
Re: [Zope-dev] >2GB Data.fs files on FreeBSD [ In reply to ]
OK, some more info. The code in FileStorage.py looks like this:

-------------------
def read_index(file, name, index, vindex, tindex, stop='\377'*8,
ltid=z64, start=4, maxoid=z64):

read=file.read
seek=file.seek
seek(0,2)
file_size=file.tell()

print file_size, start

if file_size:
if file_size < start: raise FileStorageFormatError, file.name
[etc]
-------------------

I stuck that print statement in there. The results of the print are:

-2147248811L 4

So it looks to my uneducated eye like file.tell() is broken. The actual
on-disk size of the file, by the way, is indeed 2147718485, so it looks
like somebody's not using the right size data structure somewhere.

So, can anyone tell me what to look for, or am I stuck for the moment?

--RDM

PS: anyone on pthon-dev replying please CC me as I am only on the zope list.
Re: Re: [Zope-dev] >2GB Data.fs files on FreeBSD [ In reply to ]
R. David Murray writes:
> So it looks to my uneducated eye like file.tell() is broken. The actual
> on-disk size of the file, by the way, is indeed 2147718485, so it looks
> like somebody's not using the right size data structure somewhere.
>
> So, can anyone tell me what to look for, or am I stuck for the moment?

Hmm. What is off_t defined to be on your platform? In config.h, is
HAVE_FTELLO or HAVE_FTELL64 defined?


-Fred

--
Fred L. Drake, Jr. <fdrake at acm.org>
Corporation for National Research Initiatives