Mailing List Archive

Hi Alexander,
On 2015年11月24日 at 9:52:37, Alexander B. Zubkov (info@zubkov.info) wrote:

Hi!

Sorry for bothering you. I installed ExternalStorage plugin for RT but it does not help much, it takes out of DB less than 15% (996Mb in DB, 143Mb on disk). I really expected every attachment should be moved. I took a look at source code and I don't understand a few things:
I’m happy to report that in 4.4, we moved the ExternalStorage extension into core as a standard RT feature. As part of that I cleaned up many of the issues you ran into:



1)
            $attach->Limit(
                FIELD     => 'ContentType',
                OPERATOR  => 'NOT STARTSWITH',
                VALUE     => $_,
                SUBCLAUSE => 'applies',
                ENTRYAGGREGATOR => "AND",
            ) for "text/", "message/", "image/", "multipart/";

Are you intentionally skipping images? Why? Documentation say images should be moved.
The code has been refactored in core. Image attachments are not skipped.



2)
    } elsif ($type =~ m{^image/}) {
        # Ditto images, which may be displayed inline
        return 1 if $length > 10 * 1024 * 1024;
        return 0;

It means any file less than 10Mb will not be moved? Does it make sense?
Correct. However, in the cored version of ExternalStorage, this is now configuration (ExternalStorageCutoffSize) so you can tune it down to 500kb or whatever threshold makes sense for you.

I tried to patch to 10Kb (10 * 1024) in 3 places. After that a few more files (about 20) were moved but still so much data is in DB. But I have a 26k tickets archive, it looks like it processes only tickets from last day but I would like it to process ALL attachments from very beginning.
This is because we maintain a high water mark for avoiding processing the same attachments over and over and over again.

So is it possible somehow to force this plugin to move all images and other files out of DB? Thanks.
If you *temporarily* change the line in the script (extract-attachments in the extension, rt-externalize-attachments in 4.4):

$last = $last ? $last->Content : {};

to just

$last = {};

it should re-process all attachments. It won’t re-externalize attachments that were already externalized.



Hope this helps,

Shawn

Am 24.11.2015 um 22:01 schrieb Shawn Moore:
>> I tried to patch to 10Kb (10 * 1024) in 3 places. After that a few
>> more files (about 20) were moved but still so much data is in DB. But
>> I have a 26k tickets archive, it looks like it processes only tickets
>> from last day but I would like it to process ALL attachments from very
>> beginning.
>
> This is because we maintain a high water mark for avoiding processing
> the same attachments over and over and over again.
>
>> So is it possible somehow to force this plugin to move all images and
>> other files out of DB? Thanks.
>
> If you *temporarily* change the line in the script (extract-attachments
> in the extension, rt-externalize-attachments in 4.4):
>
> $last = $last ? $last->Content : {};
>
> to just
>
> $last = {};
>
> it should re-process all attachments. It won’t re-externalize
> attachments that were already externalized.
>

A better solution to force an extraction run on all attachments is to
run this inside your Database:
delete from Attributes where Name = 'ExternalStorage';

This actually removes the "high water mark".

@shawn
May the script should become an "force-re-run" option which removes the
Attribute?

Chris

On 25/11/15 10:37, Christian Loos wrote:
> A better solution to force an extraction run on all attachments is to
> run this inside your Database: delete from Attributes where Name =
> 'ExternalStorage'; This actually removes the "high water mark". @shawn
> May the script should become an "force-re-run" option which removes
> the Attribute? Chris
Thanks, this helped. I patched the plugin to extract images always and
text never and got exactly what I wanted, great!

BTW maybe nobody ever had a 10Mb text, but if text/html attachment is
extracted somehow, RT is unable to open it, error was: "unknown encoding
external". If some implementation is missing, maybe you should not
extract text?

A forced rerun only make sense if image size is configurable, otherwise
the plugin always works correct with hardcoded settings.

On 2015年11月25日 at 2:37:07, Christian Loos (cloos@netcologne.de) wrote:
> @shawn
> May the script should become an "force-re-run" option which removes the
> Attribute?

Sure, that’d be great. :) I’ve made a ticket for this: https://issues.bestpractical.com/Ticket/Display.html?id=31517

> Chris

Thanks!
Shawn

Hi Alexander,

On 2015年11月25日 at 5:04:35, Alexander B. Zubkov (info@zubkov.info) wrote:
> BTW maybe nobody ever had a 10Mb text, but if text/html attachment is
> extracted somehow, RT is unable to open it, error was: "unknown encoding
> external". If some implementation is missing, maybe you should not
> extract text?

I’ve made a ticket for tracking this: https://issues.bestpractical.com/Ticket/Display.html?id=31518

> A forced rerun only make sense if image size is configurable, otherwise
> the plugin always works correct with hardcoded settings.

The image size is indeed configurable in RT 4.4, so it does make sense to have the flag. The flag is also useful if you make local customizations to the logic that decides whether an object should move to external storage.

Thanks!
Shawn