Mailing List Archive: bric_soap media export fails

bric_soap media export fails

Mar 24, 2010, 12:10 PM

Post #1 of 8 (2504 views)

Hello,

bric_soap media export is failing with "Apache::SizeLimit httpd process too big" in the log file.

My export looks something like this...

/usr/local/bricolage/bin/bric_soap media list_ids --with-related-media --username xxx --password xxx --server http://localhost/ --timeout 0 --continue-on-errors --search cover_date_start=2010-02-01T00:01:00Z --search cover_date_end=2010-02-01T23:59:00Z --timeout 0 | /usr/local/bricolage/bin/bric_soap media export -> media_2010_02_01.xml --username xxx --password xxx --server http://localhost/ --timeout 0 --continue-on-errors -

I also tried modifying /usr/local/bricolage/lib/Bric/SOAPMedia.pm file to skip reading and writing the actual image asset by commenting that logic out and writing a blank dataElement in it's place like this...

sub serialize_asset {
...
# read in file data
# my $fh = $media->get_file;
# my $data = join('',<$fh>);
# $writer->dataElement(data => MIME::Base64::encode_base64($data,''));
$writer->dataElement(data => '');
# close $fh;
...
}

We only do about 20 photos per day, so I cannot understand why a)it's taking over an hour to process and b)it eventually fails. This is the message when it fails:
TRANSPORT ERROR: 500 Server closed connection without sending any data back
Check the Apache error log for more information.

My pertinent bricolage.conf settings are:
CHECK_PROCESS_SIZE = Yes
MAX_PROCESS_SIZE = 64000
CHECK_FREQUENCY = 10
MIN_SHARE_SIZE = 0
MAX_UNSHARED_SIZE = 0

Any thoughts and help is greatly appreciated.

Thank you,
Mike

Re: bric_soap media export fails [ In reply to ]

david at kineticode

Mar 24, 2010, 6:59 PM

Post #2 of 8 (2455 views)

Permalink

On Mar 24, 2010, at 3:10 PM, Fletcher, Michael wrote:

> Hello,
>
> bric_soap media export is failing with "Apache::SizeLimit httpd process too big" in the log file.

That's not an error. That's just Apache::SizeLimit killing a process that got too big. It only kills processes when they're not serving requests.

> My export looks something like this...
>
> /usr/local/bricolage/bin/bric_soap media list_ids --with-related-media --username xxx --password xxx --server http://localhost/ --timeout 0 --continue-on-errors --search cover_date_start=2010-02-01T00:01:00Z --search cover_date_end=2010-02-01T23:59:00Z --timeout 0 | /usr/local/bricolage/bin/bric_soap media export -> media_2010_02_01.xml --username xxx --password xxx --server http://localhost/ --timeout 0 --continue-on-errors -

Are you using bric_queued or bric_dist_mon? The latter can take a while. Also, the --with-related-media is useless for list_ids; that's only for export.

> I also tried modifying /usr/local/bricolage/lib/Bric/SOAPMedia.pm file to skip reading and writing the actual image asset by commenting that logic out and writing a blank dataElement in it's place like this...
>
> sub serialize_asset {
> ...
> # read in file data
> # my $fh = $media->get_file;
> # my $data = join('',<$fh>);
> # $writer->dataElement(data => MIME::Base64::encode_base64($data,''));
> $writer->dataElement(data => '');
> # close $fh;
> ...
> }

Okay.

> We only do about 20 photos per day, so I cannot understand why a)it's taking over an hour to process and b)it eventually fails. This is the message when it fails:
> TRANSPORT ERROR: 500 Server closed connection without sending any data back
> Check the Apache error log for more information.

Hrm. What happens if you just run list_ids without piping it to `media export`?

Best,

David

RE: bric_soap media export fails [ In reply to ]

mike.fletcher at cbs

Mar 25, 2010, 11:29 AM

Post #3 of 8 (2442 views)

Permalink

On Wednesday, March 24, 2010 7:00 PM David E. Wheeler wrote:

> That's not an error. That's just Apache::SizeLimit killing
> a process that got too big. It only kills processes when
> they're not serving requests.

Right - I didn't say it was an error, I just said it was failing. :)

> Are you using bric_queued or bric_dist_mon? The latter can take
> a while. Also, the --with-related-media is useless for list_ids;
> that's only for export.

I'm using bric_queued
Great point about --with-related-media!!! Thanks!!!

> Hrm. What happens if you just run list_ids without piping it to
> `media export`?

I get a list of media_[ids]

Since writing the first email, I restarted bric_apachectl and now it seems to be recognizing my change to Media.pm and producing an xml output with <data></data>, but it still fails if I do more than a single day at a time. Argh.

My plan is to export the media with an empty <data> element, and then manually FTP all the actual jpg files from one server to another. I don't think I'm the first person to wish there was an option to omit <data> from the export. Maybe that should be an option. Seems like a very simple change to Media.pm.

Thanks,
Mike

Re: bric_soap media export fails [ In reply to ]

david at kineticode

Mar 25, 2010, 7:07 PM

Post #4 of 8 (2456 views)

Permalink

On Mar 25, 2010, at 2:29 PM, Fletcher, Michael wrote:

>> Hrm. What happens if you just run list_ids without piping it to
>> `media export`?
>
> I get a list of media_[ids]

Well that's good.

> Since writing the first email, I restarted bric_apachectl and now it seems to be recognizing my change to Media.pm and producing an xml output with <data></data>, but it still fails if I do more than a single day at a time. Argh.

I thought you didn't want the <data></data> blocks. What kind of RAM does this box has? How is PostgreSQL doing as it runs? Can you do one day at a time?

> My plan is to export the media with an empty <data> element, and then manually FTP all the actual jpg files from one server to another. I don't think I'm the first person to wish there was an option to omit <data> from the export. Maybe that should be an option. Seems like a very simple change to Media.pm.

It won't know the media are there, because if you upload without a data section, it won't have a file to point at. See http://github.com/bricoleurs/bricolage/commit/954e1fcd78c7769bea2d291a0cc74212e163718b for everything I did to allow no data block recently for 1.11.3.

Best,

David

RE: bric_soap media export fails [ In reply to ]

mike.fletcher at cbs

Mar 29, 2010, 6:58 AM

Post #5 of 8 (2433 views)

Permalink

On Thursday, March 25, 2010 7:07 PM David E. Wheeler wrote:

> It won't know the media are there, because if you upload without
> a data section, it won't have a file to point at.

I don't understand. If I change the XML to have an empty <data></data> element, and then manually FTP all the of images to the new server in the same folder structure as they are in the old server, why wouldn't the new server find them? All I'm excluding from the XML is the actual image file, and putting them in their same location myself.

Thanks,
Mike

Re: bric_soap media export fails [ In reply to ]

david at kineticode

Mar 29, 2010, 8:25 AM

Post #6 of 8 (2422 views)

Permalink

On Mar 29, 2010, at 6:58 AM, Fletcher, Michael wrote:

> I don't understand. If I change the XML to have an empty <data></data> element, and then manually FTP all the of images to the new server in the same folder structure as they are in the old server, why wouldn't the new server find them? All I'm excluding from the XML is the actual image file, and putting them in their same location myself.

Oh, if it's empty that might work. I was still thinking of updating media, not creating new ones on the server. But you'll need to make sure you FTP them to the right place. The location is based on the media ID, and is something like comp/data/media/10/24/v.0/filename.png, where "10/24/v.0" is for media ID 1024 version 0.

Best,

David

RE: bric_soap media export fails [ In reply to ]

mike.fletcher at cbs

Apr 6, 2010, 6:53 AM

Post #7 of 8 (2391 views)

Permalink

On Monday, March 29, 2010 8:25 AM David E. Wheeler wrote:

> Oh, if it's empty that might work. I was still thinking
> of updating media, not creating new ones on the server.
> But you'll need to make sure you FTP them to the right place.
> The location is based on the media ID, and is something like
> comp/data/media/10/24/v.0/filename.png, where "10/24/v.0"
> is for media ID 1024 version 0.

"bric_soap media create" creates a new media ID and UUID, it does not use the one from the XML. Is there a way to override that and force it to use the one from the XML? If not, then none of my media is going to match up with the actual media folder locations on the disk or the stories I'm importing.

Thanks,
Mike

Re: bric_soap media export fails [ In reply to ]

david at kineticode

Apr 6, 2010, 9:04 AM

Post #8 of 8 (2390 views)

Permalink

On Apr 6, 2010, at 6:53 AM, Fletcher, Michael wrote:

> "bric_soap media create" creates a new media ID and UUID, it does not use the one from the XML. Is there a way to override that and force it to use the one from the XML?

No.

> If not, then none of my media is going to match up with the actual media folder locations on the disk or the stories I'm importing.

If you have them in a certain order in the XML, you can look at what gets output by bric_soap

media_1024
media_1025
media_1026

And figure it out from there.

But in truth, you're best off sending the actual media files along in the XML. Slow yes, but least headache for you.

Best,

David