Mailing List Archive: Large file upload problems

Large file upload problems

warren at etr-usa

Apr 30, 2004, 8:07 PM

Post #1 of 9 (7358 views)

I have an ASP page that needs to accept files which are typically rather
large. (Average 2GB +/- 1GB.) These files ultimately need to be stored
in a particular place in the web server's file system. I've set it up
so that the temp file is kept around, so that I can simply do a
system("mv $tempfile $target") within the ASP code to move the uploaded
file into place.

That works, but there are a number of weaknesses I've run into:

1. Since $tempfile is on the system disk and $target is on a big RAID
array, the move incurs a lot of extra disk I/O. Is there a way to tell
CGI.pm or Apache::ASP to use a directory on the RAID array instead of
/usr/tmp?

2. There seems to be no upload progress indicator, at least with
Mozilla. Is there a way that I can insert some code that gets run right
after the first HTTP header on the upload gets processed, so I can pick
off the expected file size? If so, I could pop up a window with my own
progress bar.

3. Something in the chain is barfing if the file is too large. The
limit seems to be just under 2GB. Here's the spewage I get in the log
for a 1.99GB file:

> [Fri Apr 30 20:47:31 2004] [error] [client 172.16.0.42] Malformed
> multipart
> POST!!MultipartBuffer::read('MultipartBuffer=HASH(0x8b214ac)',0)
> called at (eval 109) line
> 55!!MultipartBuffer::new('MultipartBuffer','CGI=HASH(0x8ba6b58)','---------------------------9040894219264',-2147483254,'undef')
> called at (eval 107) line
> 4!!CGI::new_MultipartBuffer('CGI=HASH(0x8ba6b58)','---------------------------9040894219264',-2147483254,'undef')
> called at (eval 106) line
> 3!!CGI::read_multipart('CGI=HASH(0x8ba6b58)','---------------------------9040894219264',-2147483254)
> called at /usr/lib/perl5/5.8.0/CGI.pm line
> 415!!CGI::init('CGI=HASH(0x8ba6b58)','undef') called at
> /usr/lib/perl5/5.8.0/CGI.pm line 286!!CGI::new('CGI') called at
> /usr/lib/perl5/site_perl/5.8.0/Apache/ASP/Request.pm line
> 81!!Apache::ASP::Request::new('Apache::ASP=HASH(0x824ce84)') called
> at /usr/lib/perl5/site_perl/5.8.0/Apache/ASP.pm line
> 387!!Apache::ASP::new('Apache::ASP','Apache::RequestRec=SCALAR(0x8b41f64)','/home/tangent/mms5/cli/mma-edit-title.asp')
> called at /usr/lib/perl5/site_perl/5.8.0/Apache/ASP.pm line
> 181!!Apache::ASP::handler('Apache::RequestRec=SCALAR(0x8b41f64)')
> called at -e line 0!!eval {...} called at -e line 0!, referer:
> http://frank/mma-edit-title.asp?tid=97&sid=1

I'd guess some bit of code along the path is treating the Content-Length
header as a signed 32-bit value. This is a Linux 2.4 system, so it can
handle large files. What code is to blame here?

---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org

Re: Large file upload problems [ In reply to ]

warren at etr-usa

May 4, 2004, 4:48 AM

Post #2 of 9 (7148 views)

Stuart Johnston wrote:
>
> Try setting this at the top of your script:
>
> $TempFile::TMPDIRECTORY='/target/location';

What is $TempFile? Do you mean something under $Request->{FileUpload}?
Perhaps $Request->{FileUpload}{file}->{TempFile}?

> There is a mod_perl module you might find helpful:
>
> http://search.cpan.org/search?query=UploadMeter&mode=module

Hmmm...too many server configuration requirements. I need to do this on
multiple servers that are already set up.

> Also, we use a java applet for upload which give a progress meter as
> well as some other features:
>
> http://jupload.biz/

Interesting, but its focus on multiple file uploads makes it suboptimal
for our simple needs.

> Sorry, I don't have any suggestions on your other question.

That third one is the killer. If I can't get past the 2GB limit, it
doesn't matter how easy or efficient the solution is.

I think what I'm going to do is step outside HTTP/ASP altogether. I've
got this hammer, so all my problems are looking like nails. HTTP is
primarily a file download protocol; I'm swimming against the stream to
use this protocol for uploads. I think I'll write a native program that
will use a more appropriate protocol to upload the file. This won't be
onerous for our purposes.

---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org

Re: Large file upload problems [ In reply to ]

sjohnston at vaultranet

May 4, 2004, 7:40 AM

Post #3 of 9 (7154 views)

Warren Young wrote:
> Stuart Johnston wrote:
>
>>
>> Try setting this at the top of your script:
>>
>> $TempFile::TMPDIRECTORY='/target/location';
>
>
> What is $TempFile? Do you mean something under $Request->{FileUpload}?
> Perhaps $Request->{FileUpload}{file}->{TempFile}?

Apache::ASP uses the CGI module for file uploads. That is where this
setting comes from:

http://stein.cshl.org/WWW/software/CGI/#upload_caveats

BTW, I have never tested this setting myself. I just pulled it from
that page.

Stuart Johnston

---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org

Re: Large file upload problems [ In reply to ]

warren at etr-usa

May 5, 2004, 2:42 PM

Post #4 of 9 (7145 views)

Warren Young wrote:
>
> What code is to blame here?

Just to follow up, the source of this limit appears to be with Perl
itself. Perl won't use 64-bit ints unless it is compiled to do so, even
if the platform will support 64-bit ints. Since CGI.pm uses a scalar to
hold the expected length of the file, the number looks negative when
it's over 2**31-1. CGI.pm has a check in it for this, and it dies when
the file length is "negative".

I could recompile Perl, but this would apparently break the ABI, so I'd
have to rebuild all my CPAN modules (and more?) as well. Ugh.

Thanks to all who helped. I don't have a solution yet, but I do at
least know where solutions may be had, and where to stop looking.

---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org

Re: Large file upload problems [ In reply to ]

ian.cass at medwaywireless

May 6, 2004, 2:23 AM

Post #5 of 9 (7147 views)

Warren Young wrote:

> Just to follow up, the source of this limit appears to be with Perl
> itself. Perl won't use 64-bit ints unless it is compiled to do so,
> even if the platform will support 64-bit ints. Since CGI.pm uses a
> scalar to hold the expected length of the file, the number looks
> negative when it's over 2**31-1. CGI.pm has a check in it for this,
> and it dies when the file length is "negative".

I guess you could play with this...

http://search.cpan.org/~tels/bignum-0.15/lib/bigint.pm

... and see if you can get CGI.pm using it.

--
Ian Cass

---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org

Re: Large file upload problems [ In reply to ]

warren at etr-usa

May 6, 2004, 4:48 PM

Post #6 of 9 (7146 views)

Ian Cass wrote:

> I guess you could play with this...
>
> http://search.cpan.org/~tels/bignum-0.15/lib/bigint.pm
>
> ... and see if you can get CGI.pm using it.

Wouldn't you know it, but Lincoln Stein proposed something similar when
I reported the problem to him. He says he'll be making that section of
the code use Math::BigInt, which appears to be a related module.

---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org

Re: Large file upload problems [ In reply to ]

May 6, 2004, 10:28 PM

Post #7 of 9 (7153 views)

Warren Young wrote:
> Ian Cass wrote:
>
>> I guess you could play with this...
>>
>> http://search.cpan.org/~tels/bignum-0.15/lib/bigint.pm
>>
>> ... and see if you can get CGI.pm using it.
>
>
> Wouldn't you know it, but Lincoln Stein proposed something similar when
> I reported the problem to him. He says he'll be making that section of
> the code use Math::BigInt, which appears to be a related module.
>

Sorry for chiming in so late on this, but you might consider using
the mod_perl interface directly for the file uploads, by first disabling
Apache::ASP's file upload processing:

PerlSetVar RequestBinaryRead Off

And then using the Apache::Request interface to get the $apr->upload data.

Regards,

Josh
________________________________________________________________________
Josh Chamas, Founder | NodeWorks - http://www.nodeworks.com
Chamas Enterprises Inc. | NodeWorks Directory - http://dir.nodeworks.com
http://www.chamas.com | Apache::ASP - http://www.apache-asp.org

---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org

Re: Large file upload problems [ In reply to ]

warren at etr-usa

May 7, 2004, 10:56 PM

Post #8 of 9 (7143 views)

Josh Chamas wrote:
>
> you might consider using
> the mod_perl interface directly for the file uploads, by first disabling
> Apache::ASP's file upload processing:

What is the benefit from doing this?

> And then using the Apache::Request interface to get the $apr->upload data.

I tried to install Apache::Request, but it complains that it doesn't
work with mod_perl2 yet.

---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org

Re: Large file upload problems [ In reply to ]

May 8, 2004, 11:59 AM

Post #9 of 9 (7143 views)

Warren Young wrote:
> Josh Chamas wrote:
>
>>
>> you might consider using
>> the mod_perl interface directly for the file uploads, by first disabling
>> Apache::ASP's file upload processing:
>
>
> What is the benefit from doing this?
>
>> And then using the Apache::Request interface to get the $apr->upload
>> data.
>
>
> I tried to install Apache::Request, but it complains that it doesn't
> work with mod_perl2 yet.
>

If it did work, then using Apache::Request upload API might not have
had the 2G file limit you work working on. Further, it seems that you
could actually call it after sending something of a header or partial
status wait page, but I am not positive on this last one, whether
you can keep reading the input after using the output handle
for the Apache request.

On a more general note, I think you were right on in considering other
mechnanisms for moving large files around. I would consider rsync especially
for this, but ftp is an old time favorite. Note that if you are in a mixed
windows/unix environment, there is rsync/ssh support available through
the Cygwin project at http://www.cygwin.com/

Regards,

Josh

________________________________________________________________________
Josh Chamas, Founder | NodeWorks - http://www.nodeworks.com
Chamas Enterprises Inc. | NodeWorks Directory - http://dir.nodeworks.com
http://www.chamas.com | Apache::ASP - http://www.apache-asp.org

---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org