Mailing List Archive: ripmime making tons of files??

ripmime making tons of files??

Dec 8, 2004, 3:10 PM

Post #1 of 7 (8374 views)

I'm having an issue with ripmime and a relatively large email.. it's creating
26 text files and 15 of them are almost the same size as the original email,
and including most of the content of the original email. This is causing
major problems as this is being used in conjunction with clamav for scanning
inbound emails for viruses.

Here's an example:
[root@mail tmp2]# ls -l msg.1102542794.736261.1027
-rw-r--r-- 1 simscan root 10228587 Dec 8 16:55 msg.1102542794.736261.1027

[root@mail tmp2]# ripmime -i msg.1102542794.736261.1027

[root@mail tmp2]# ls -l
total 163324
-rw-r--r-- 1 root root 1944183 Dec 8 17:57 00537_RJ.JPG
-rw-r--r-- 2 root root 53208 Dec 8 17:57 applefile
-rw-r--r-- 1 root root 1221858 Dec 8 17:57 DSC_0509 copy_1.JPG
-rw-r--r-- 2 root root 53208 Dec 8 17:57 DSC_0509 copy.JPG
-rw-r--r-- 1 simscan root 10228587 Dec 8 16:55 msg.1102542794.736261.1027
-rw-r--r-- 1 root root 1 Dec 8 17:57 textfile0
-rw-r--r-- 1 root root 198 Dec 8 17:57 textfile1
-rw-r--r-- 1 root root 235 Dec 8 17:57 textfile10
-rw-r--r-- 1 root root 10224479 Dec 8 17:57 textfile11
-rw-r--r-- 1 root root 10224397 Dec 8 17:57 textfile12
-rw-r--r-- 1 root root 10224262 Dec 8 17:57 textfile13
-rw-r--r-- 1 root root 10224105 Dec 8 17:57 textfile14
-rw-r--r-- 1 root root 10224007 Dec 8 17:57 textfile15
-rw-r--r-- 1 root root 10223826 Dec 8 17:57 textfile16
-rw-r--r-- 1 root root 10223699 Dec 8 17:57 textfile17
-rw-r--r-- 1 root root 10223668 Dec 8 17:57 textfile18
-rw-r--r-- 1 root root 10223660 Dec 8 17:57 textfile19
-rw-r--r-- 1 root root 235 Dec 8 17:57 textfile2
-rw-r--r-- 1 root root 10223602 Dec 8 17:57 textfile20
-rw-r--r-- 1 root root 1 Dec 8 17:57 textfile21
-rw-r--r-- 1 root root 198 Dec 8 17:57 textfile22
-rw-r--r-- 1 root root 235 Dec 8 17:57 textfile23
-rw-r--r-- 1 root root 139 Dec 8 17:57 textfile24
-rw-r--r-- 1 root root 1041 Dec 8 17:57 textfile25
-rw-r--r-- 1 root root 0 Dec 8 17:57 textfile26
-rw-r--r-- 1 root root 10226899 Dec 8 17:57 textfile3
-rw-r--r-- 1 root root 10226817 Dec 8 17:57 textfile4
-rw-r--r-- 1 root root 10226653 Dec 8 17:57 textfile5
-rw-r--r-- 1 root root 10226367 Dec 8 17:57 textfile6
-rw-r--r-- 1 root root 10226308 Dec 8 17:57 textfile7
-rw-r--r-- 1 root root 1 Dec 8 17:57 textfile8
-rw-r--r-- 1 root root 198 Dec 8 17:57 textfile9

Is this normal? This is ending up using 160MB of storage space, as well as
hogging up a lot of cpu, and clamav is also eating up lots of cpu trying to
scan these files...

If you need the email causing the problem, I will have to contact the customer
to see if I can release it (but it would only be privately) for debugging
purposes, just let me know.

Thanks in advance.

-Jeremy

--
Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc.
jeremy@inter7.com ++ www.inter7.com ++ 866.528.3530 ++ 815.776.9465 int'l
kitchen @ #qmail #gentoo on EFnet IRC ++ scriptkitchen.com/qmail
GnuPG Key ID: 481BF7E2 ++ jabber:kitchen@scriptkitchen.com

RE: ripmime making tons of files?? [ In reply to ]

dallase at nmgi

Dec 8, 2004, 3:19 PM

Post #2 of 7 (8110 views)

Permalink

>
> I'm having an issue with ripmime and a relatively large
> email.. it's creating
> 26 text files and 15 of them are almost the same size as the
> original email, and including most of the content of the
> original email. This is causing major problems as this is
> being used in conjunction with clamav for scanning inbound
> emails for viruses.
>

I had the exact same problem with an almost identical message. Although
it only expanded to like 15 files and took up about 50MB, but it damn
near kills the system when that message comes through... Ripmime didn't
seem to have any problems decoding them, but clamav runs forever... The
120 second alarm is the only thing that saves it.

If you decode that same message with reformime or something similar you
get the same results right? I believe I did. I cant remember if it was
something with the JPG compression or the mime structure, but I
diffinitely remember it was picture files from a camera, just like you
are seeing here.

And the other thing I remember is I could never get a resolution as to
why clamav took so damn long when uvscan scanned it in about 13 seconds.
I finally put a block on that users email addres until they called in :)
haha

Dallas

_______________________________________________
Ripmime-general mailing list
Ripmime-general@pldaniels.com
http://www.pldaniels.com/mailman/listinfo/ripmime-general

Re: ripmime making tons of files?? [ In reply to ]

pldaniels at pldaniels

Dec 8, 2004, 5:40 PM

Post #3 of 7 (8137 views)

Permalink

Hello all,

The problem lies with the oinion-shelling nature of ripMIME.

If you turn off things like --disable-qmail-bounce you'll notice that a lot of the repeated files go away.

I'm trying to find an effective way eliminate these repeats.

Regards.

--
PLDaniels - Software - Xamime
Unix systems Internet Development A.B.N. 19 500 721 806
PGP Public Key at http://www.pldaniels.com/gpg-keys.pld

RE: ripmime making tons of files?? [ In reply to ]

dallase at nmgi

Dec 9, 2004, 6:16 AM

Post #4 of 7 (8141 views)

Permalink

>
> The problem lies with the oinion-shelling nature of ripMIME.
>
> If you turn off things like --disable-qmail-bounce
> you'll notice that a lot of the repeated files go away.
>

I don't believe that was the case in my situation. I have been running
like this for over a year now,

-q --no-ole --disable-qmail-bounce --unique_names -i - -d
$ENV{'TMPDIR'}/

and this many large files issue was only a fews months ago. Jeremy,
are you running with --disable-qmail-bounce? If not, when you run that
file through with that option does the output change?

Thanks,

Dallas Engelken

_______________________________________________
Ripmime-general mailing list
Ripmime-general@pldaniels.com
http://www.pldaniels.com/mailman/listinfo/ripmime-general

Re: ripmime making tons of files?? [ In reply to ]

pldaniels at pldaniels

Dec 9, 2004, 6:59 AM

Post #5 of 7 (8112 views)

Permalink

Dallas,

> I don't believe that was the case in my situation. I have been running
> like this for over a year now,

Okay, would you like to forward to me a mailpack that exhibits these large/long files lists?

Ultimately it still tends to come down to the onion-peel nature of how ripMIME decodes the files. It's currently
"sometimes" an issue which can be controlled by the --recursion-max parameter (set it to 5 perhaps), however, because
MIME is a format capable of being nested these files do have a bad habit of multiplying.

Paul.

--
PLDaniels - Software - Xamime
Unix systems Internet Development A.B.N. 19 500 721 806
PGP Public Key at http://www.pldaniels.com/gpg-keys.pld

Re: ripmime making tons of files?? [ In reply to ]

pldaniels at pldaniels

Dec 10, 2004, 9:55 PM

Post #6 of 7 (8139 views)

Permalink

Have you ever noticed how you can forget even the simplest of things about a program that you've written yourself?

Try using the --no-nameless flag for removing the 'textfile*' files - while they'll still be generated during the
process, they will be culled at the end leaving you with just the derived attachments.

Paul.

--
PLDaniels - Software - Xamime
Unix systems Internet Development A.B.N. 19 500 721 806
PGP Public Key at http://www.pldaniels.com/gpg-keys.pld

Re: ripmime making tons of files?? [ In reply to ]

pldaniels at pldaniels

Dec 16, 2004, 11:50 PM

Post #7 of 7 (8118 views)

Permalink

Okay everybody,

I've not yet solved the problems that you guys are having, with the masses of files all different slightly (I'm working
on it).

The solution I'm working on is basically one where an output file will be flagged according to how it was generated.
In short, if it required no special translation (ie, not B64/QP/UU) it won't be specifically preserved, rather it'll
just leave the parent file to be the holder of the information. The trick is, I have to work out how to implement this
;-)

Paul.

--
PLDaniels - Software - Xamime
Unix systems Internet Development A.B.N. 19 500 721 806
PGP Public Key at http://www.pldaniels.com/gpg-keys.pld