Mailing List Archive

Exim header size limit error causes big problem...
Sorry to beat a dead horse, but what I thought was a harmless error took down
Exim last night. I've posted previously (as have others) about this error in
my Exim paniclog:

2004-02-10 01:06:04 1AqTqC-00083u-Rd string_sprintf expansion was longer than
8192

It happened about once per minute. It was caused by installing all these great
new rulesets (Tripwire, Chickenpox, etc.). These sets are similar in that
they are a large collection of small rules which combine to make an effective
set. The problem is that each small rule outputs its own entry in the SA
report. After hitting a bazillion little rules, the report swells to a huge
size, and when inserted into email headers, generates the above error.

I was just ignoring the errors for the time being, figuring those emails were
probably spam anyway. What I didn't know is that every time that error hit,
Exiscan didn't clean up that message's directory in the scan dir
(/var/spool/exim/scan on my system). Eventually the directory hit the max
filesystem limit for number of files/dirs and Exim started rejecting incoming
mail with "temporary local error".

I realize that's Exiscan's fault, but I think it drives home my contention
that reporting every little hit on these huge rulesets is a Bad Thing(tm). As
Chris Santerre said today in another thread, this has been a period of many
new rules and and heavy rule development. As these new rules show up, I'm
sure we'll see reports get longer still. Surely there is a point at which a
reasonable limit must be placed, especially for the
"Tripwire"/"Chickenpox"-type rules. Additionally, the output from these types
of rules is very verbose, and while interesting, most times I don't really
care _exactly_ which strings of random crap it's hit on this particular
message. :) I'd just like to know it's doing the job and probably how many
total rules and points have been hit for each set.

I'm not sure if SA includes that kind of functionality, but IMHO, it would be
a good thing to have. Another nice feature would be a configurable cap on
report size.

In the mean time I've altered my report to be smaller. However, I consider
that a work-around, not a fix. The error could still happen with my more
terse report, if enough rules hit.

PS:
In creating my new report, I'm trying to use _TESTSSCORES()_, but I'd like a
newline as the delimiter. I've tried \n and ^M, but \n is just printed and ^M
doesn't seem to have an effect at all. Any ideas?
--
Matt
Systems Administrator
Local Access Communications
360.330.5535
Re: Exim header size limit error causes big problem... [ In reply to ]
On Tuesday 10 February 2004 11:17 am, Matthew Trent wrote:
> Sorry to beat a dead horse, but what I thought was a harmless error took
> down Exim last night. I've posted previously (as have others) about this
> error in my Exim paniclog:
>
> 2004-02-10 01:06:04 1AqTqC-00083u-Rd string_sprintf expansion was longer
> than 8192

Oh yeah, and, as an addendum, I realize it's easy enough to raise the header
size limit in the Exim source, but I still think there is a
practical/reasonable limit to how big a report should be and how verbose any
one ruleset should be. Changing the size limit is just a workaround...
--
Matt
Systems Administrator
Local Access Communications
360.330.5535
Re: Exim header size limit error causes big problem... [ In reply to ]
Not sure if it has been mentioned before, but perhaps it would be more
sane to restructure the rulesets with alot of small rules so that they
have "__" in front of them (and I assume won't show up in the
X-Spam-Report?) and use meta tests on them.

Ryan Moore
----------
Perigee.net Corporation
704-849-8355 (sales)
704-849-8017 (tech)
www.perigee.net



Matthew Trent wrote:
> Sorry to beat a dead horse, but what I thought was a harmless error took down
> Exim last night. I've posted previously (as have others) about this error in
> my Exim paniclog:
>
> 2004-02-10 01:06:04 1AqTqC-00083u-Rd string_sprintf expansion was longer than
> 8192
>
> It happened about once per minute. It was caused by installing all these great
> new rulesets (Tripwire, Chickenpox, etc.). These sets are similar in that
> they are a large collection of small rules which combine to make an effective
> set. The problem is that each small rule outputs its own entry in the SA
> report. After hitting a bazillion little rules, the report swells to a huge
> size, and when inserted into email headers, generates the above error.
>
> I was just ignoring the errors for the time being, figuring those emails were
> probably spam anyway. What I didn't know is that every time that error hit,
> Exiscan didn't clean up that message's directory in the scan dir
> (/var/spool/exim/scan on my system). Eventually the directory hit the max
> filesystem limit for number of files/dirs and Exim started rejecting incoming
> mail with "temporary local error".
>
> I realize that's Exiscan's fault, but I think it drives home my contention
> that reporting every little hit on these huge rulesets is a Bad Thing(tm). As
> Chris Santerre said today in another thread, this has been a period of many
> new rules and and heavy rule development. As these new rules show up, I'm
> sure we'll see reports get longer still. Surely there is a point at which a
> reasonable limit must be placed, especially for the
> "Tripwire"/"Chickenpox"-type rules. Additionally, the output from these types
> of rules is very verbose, and while interesting, most times I don't really
> care _exactly_ which strings of random crap it's hit on this particular
> message. :) I'd just like to know it's doing the job and probably how many
> total rules and points have been hit for each set.
>
> I'm not sure if SA includes that kind of functionality, but IMHO, it would be
> a good thing to have. Another nice feature would be a configurable cap on
> report size.
>
> In the mean time I've altered my report to be smaller. However, I consider
> that a work-around, not a fix. The error could still happen with my more
> terse report, if enough rules hit.
>
> PS:
> In creating my new report, I'm trying to use _TESTSSCORES()_, but I'd like a
> newline as the delimiter. I've tried \n and ^M, but \n is just printed and ^M
> doesn't seem to have an effect at all. Any ideas?
Re: Exim header size limit error causes big problem... [ In reply to ]
On Tuesday 10 February 2004 11:45 am, Raquel Rice wrote:
> > Oh yeah, and, as an addendum, I realize it's easy enough to raise
> > the header size limit in the Exim source, but I still think there
> > is a practical/reasonable limit to how big a report should be and
> > how verbose any one ruleset should be. Changing the size limit is
> > just a workaround...--
> > Matt
>
> I don't know if you're looking for input, so I'll offer my point of
> view. I like to keep tabs on what rules are working and which are
> not. I like having the ability to make adjustments if I feel it's
> necessary. Without the headers, especially "X--Spam-Report:", it's
> difficult to do that. I like having the headers there.

I think you're missing my point...

I like the headers... I can't live without 'em, they're great. My complaint is
only about extraneous information in the report. If the Tripwire set hits 30
times, I don't want 30 lines just for Tripwire. I think it would be more
appropriate to have 1 line saying something like "Tripwire: 30 hits, 7
points". In addition, with all these custom sets floating around in addition
to the stock rules, it would be nice to have a configurable limit on report
size.
--
Matt
Systems Administrator
Local Access Communications
360.330.5535
Re: Exim header size limit error causes big problem... [ In reply to ]
On Tuesday 10 February 2004 12:00 pm, Ryan Moore wrote:
> Not sure if it has been mentioned before, but perhaps it would be more
> sane to restructure the rulesets with alot of small rules so that they
> have "__" in front of them (and I assume won't show up in the
> X-Spam-Report?) and use meta tests on them.
>
> Ryan Moore
> ----------
> Perigee.net Corporation
> 704-849-8355 (sales)
> 704-849-8017 (tech)
> www.perigee.net

Exactly. And just output the totals for the whole set.
--
Matt
Systems Administrator
Local Access Communications
360.330.5535
Re: Exim header size limit error causes big problem... [ In reply to ]
Matthew Trent wrote:
> Sorry to beat a dead horse, but what I thought was a harmless error took down
> Exim last night. I've posted previously (as have others) about this error in
> my Exim paniclog:
>
> 2004-02-10 01:06:04 1AqTqC-00083u-Rd string_sprintf expansion was longer than
> 8192

It looks to me like this is very easily changed (and I wonder why it's
hard coded). At the top of header.c in the src directory of the Exim
source, line 34 (exim-4.30):

uschar buffer[8192];

Better yet, some context:

void
header_add(int type, char *format, ...)
{
header_line *new;
va_list ap;
uschar buffer[8192];


Of course you'd want to change the error test and message:

if (!string_vformat(buffer, sizeof(buffer), format, ap))
log_write(0, LOG_MAIN|LOG_PANIC_DIE, "string too long in header_add:
%.100s ...",
buffer);


Cheers,
bob
--
Bob Amen
O'Reilly Media, Inc.
http://www.ora.com/
http://www.oreilly.com/
Re: Exim header size limit error causes big problem... [ In reply to ]
On Tuesday 10 February 2004 12:19 pm, Bob Amen wrote:
> It looks to me like this is very easily changed (and I wonder why it's
> hard coded). At the top of header.c in the src directory of the Exim
> source, line 34 (exim-4.30):
>
> uschar buffer[8192];

Yes, I realize that. However, 8k seems like plenty to me, and I'd rather see a
real solution than a workaround. The way it's going right now, that report
can be DARN long, and eventually it may hit the limit again, even if you up
it...
--
Matt
Systems Administrator
Local Access Communications
360.330.5535
Re: Exim header size limit error causes big problem... [ In reply to ]
On Tuesday 10 February 2004 12:36 pm, Raquel Rice wrote:
> Don't trivialize what I said ....

I didn't mean to, I didn't think we were on the same page...

> I like knowing which of the rules in Tripwire have scored.

Me too, it's great info. However, it constitutes a huge amount of data, and I
don't think all of that is necessary in every report.

> If changing the header size limit for Exim is a workaround, isn't it
> also a workaround to want the same ability in SpamAssassin?

Well, Spamassassin is where the data comes from, and it should have reasonable
limits on what it spits out. Exim already has a reasonable limit. SA doesn't
have anything.
--
Matt
Systems Administrator
Local Access Communications
360.330.5535
Re: Exim header size limit error causes big problem... [ In reply to ]
On Tuesday 10 February 2004 12:40 pm, Raquel Rice wrote:
> > Yes, I realize that. However, 8k seems like plenty to me, and I'd
> > rather see a real solution than a workaround. The way it's going
> > right now, that report can be DARN long, and eventually it may hit
> > the limit again, even if you up it...
> > --
> > Matt
>
> And, I ask again. Why is changing SpamAssassin a "fix" and changing
> a setting in Exim a "workaround"?

Because if SA has absolutely no limit, no matter how high I set Exim, that
limit can always be reached.

It seems like these new rules constitute a new "genre" of large rule sets with
many small and similar rules all designed to work together. Perhaps there are
special considerations that need to be taken for them. They do seem to work
well (props to the creators/maintainers), so I'm sure many people are or will
be using them.
--
Matt
Systems Administrator
Local Access Communications
360.330.5535
RE: Exim header size limit error causes big problem... [ In reply to ]
> -----Original Message-----
> From: Matthew Trent [mailto:mtrent@localaccess.com]
> Sent: Tuesday, February 10, 2004 3:53 PM
> To: spamassassin-users@incubator.apache.org
> Subject: Re: Exim header size limit error causes big problem...
>
>
> On Tuesday 10 February 2004 12:40 pm, Raquel Rice wrote:
> > > Yes, I realize that. However, 8k seems like plenty to me, and I'd
> > > rather see a real solution than a workaround. The way it's going
> > > right now, that report can be DARN long, and eventually it may hit
> > > the limit again, even if you up it...
> > > --
> > > Matt
> >
> > And, I ask again. Why is changing SpamAssassin a "fix" and changing
> > a setting in Exim a "workaround"?
>
> Because if SA has absolutely no limit, no matter how high I
> set Exim, that
> limit can always be reached.
>
> It seems like these new rules constitute a new "genre" of
> large rule sets with
> many small and similar rules all designed to work together.
> Perhaps there are
> special considerations that need to be taken for them. They
> do seem to work
> well (props to the creators/maintainers), so I'm sure many
> people are or will
> be using them.
> --
> Matt
> Systems Administrator
> Local Access Communications
> 360.330.5535


I actually though Fred had made them into meta rules. Or maybe it was
another list goer? I know someone did.

--Chris (I don't sleep much, and I like it.)
Re: Exim header size limit error causes big problem... [ In reply to ]
On Tuesday 10 February 2004 01:20 pm, Chris Santerre wrote:
> I actually though Fred had made them into meta rules. Or maybe it was
> another list goer? I know someone did.
>
> --Chris (I don't sleep much, and I like it.)

Doesn't look like stock has changed at least. I just checked.
--
Matt
Systems Administrator
Local Access Communications
360.330.5535
Re: Exim header size limit error causes big problem... [ In reply to ]
On Tue, 10 Feb 2004, Matthew Trent wrote:

> > If changing the header size limit for Exim is a workaround, isn't it
> > also a workaround to want the same ability in SpamAssassin?
>
> Well, Spamassassin is where the data comes from, and it should have reasonable
> limits on what it spits out. Exim already has a reasonable limit. SA doesn't
> have anything.

OK, and what happens when some spammer/hacker decides to DoS you with
messages with horribly long headers? Or Mr. Executive/Boss/... creates
a maillist or alias that expands to 1000 recipients, etc...

That's why industrial strength MTAs use dynamically allocated buffers,
they've learned from bad past experience that any "reasonable limit"
sooner or later becomes an unreasonable choke-point/attack-point.

You may have no need to see all those match headers listed, but
what about developers trying to see where their rules hit, or
an admin trying to debug a particular message that went wrong.

If you limit yourself to Exim, that's your choice but please don't
expect the rest of the SA world to cripple our tool to fit your
limitations.
Maybe your time would be better spent over in the Exim community
encouraging its developers to improve their program to the level
that other MTAs have already achieved.

--
Dave Funk University of Iowa
<dbfunk (at) engineering.uiowa.edu> College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_admin Iowa City, IA 52242-1527
#include <std_disclaimer.h>
Better is not better, 'standard' is better. B{
Re: Exim header size limit error causes big problem... [ In reply to ]
From: "David B Funk" <dbfunk@engineering.uiowa.edu>

> On Tue, 10 Feb 2004, Matthew Trent wrote:
>
> > > If changing the header size limit for Exim is a workaround, isn't it
> > > also a workaround to want the same ability in SpamAssassin?
> >
> > Well, Spamassassin is where the data comes from, and it should have
reasonable
> > limits on what it spits out. Exim already has a reasonable limit. SA
doesn't
> > have anything.
>
> OK, and what happens when some spammer/hacker decides to DoS you with
> messages with horribly long headers? Or Mr. Executive/Boss/... creates
> a maillist or alias that expands to 1000 recipients, etc...
>
> That's why industrial strength MTAs use dynamically allocated buffers,
> they've learned from bad past experience that any "reasonable limit"
> sooner or later becomes an unreasonable choke-point/attack-point.
>
> You may have no need to see all those match headers listed, but
> what about developers trying to see where their rules hit, or
> an admin trying to debug a particular message that went wrong.
>
> If you limit yourself to Exim, that's your choice but please don't
> expect the rest of the SA world to cripple our tool to fit your
> limitations.
> Maybe your time would be better spent over in the Exim community
> encouraging its developers to improve their program to the level
> that other MTAs have already achieved.

On another paw, Dave, you could rewrite say the chickenpox.cf rules
as __rulename and then collate at the end by adding them all up and
multiplying by 0.6 to get the results. If that result could be issued
as a spam score directly then you get the summation. I have hacked
spamassassin 2.55 when I "got made at it." (I repaired the foreground
vs background matching colors test and added another similar test that
seems to already be in 2.63.) But I have not gotten far enough to hack
that kind of meta value directly into a score. I'm not sure how I would
do it. But I believe it would have value for the chickenpox rules. (I'd
even use the square of the number of chickenpox hits with a much smaller
incremental value if I had the chance.)

{^_^} Joanne
Re: Exim header size limit error causes big problem... [ In reply to ]
On Tue, 10 Feb 2004 12:36:54 -0800, Raquel Rice wrote:

>  If changing the header size limit for Exim is a workaround, isn't
>  it also a workaround to want the same ability in SpamAssassin?

One possible reason:

Because changing Exim only changes Exim while giving the possibility (note: giving, not enforcing) to limit the size of reports in SA can fix this problem for other configurations as well.

Truncating the report in SA would be a workaround, but the possibility to combine multiple actual scores into one reported score would be feature that actually fixes the problem.

Example of a non Exim-related issue:

MIMEDefang (a sendmail milter that can call SA) currently uses a 8192 bytes buffer for the SA scores. If enough scores are hit, problems may occur.

David (author of MIMEDefang) tried upping the limit to 16384 bytes (in order to delay the problems), but this made MIMEDefang crash on Freebsd 4.7 (he thinks that *may* be because of stack limitations on that platform).

On Tue, 10 Feb 2004, Matthew Trent wrote:

> OK, and what happens when some spammer/hacker decides to DoS you with
> messages with horribly long headers? Or Mr. Executive/Boss/... creates
> a maillist or alias that expands to 1000 recipients, etc...

The limit on what size of headers/report/whatever SA can insert does not necessarily correlate with the SMTP servers handling of headers for incoming mail.

In my example above, this has nothing to do with the maximum size of headers in sendmail or milters (SA doesn't insert the headers itself when called from MD), it is only about the communication between MD and SA.

I also wouldn't be suprised if sendmail and other smtp server capable of calling filters has completely different limitations in their filter API than they have for incoming mail. Does anyone know what (if any) limits sendmails milter API puts on headers inserted by milters?

Regards
/Jonas
Re: Exim header size limit error causes big problem... [ In reply to ]
On Tuesday 10 February 2004 10:05 pm, you wrote:
> > Well, Spamassassin is where the data comes from, and it should have
> > reasonable limits on what it spits out. Exim already has a reasonable
> > limit. SA doesn't have anything.
>
> OK, and what happens when some spammer/hacker decides to DoS you with
> messages with horribly long headers? Or Mr. Executive/Boss/... creates
> a maillist or alias that expands to 1000 recipients, etc...
>
> That's why industrial strength MTAs use dynamically allocated buffers,
> they've learned from bad past experience that any "reasonable limit"
> sooner or later becomes an unreasonable choke-point/attack-point.

I would hesitate to accuse Exim of not being an industrial-strength MTA. We've
been using it in high-load production environments for years now and and I
know many other people are also. We've been very pleased with its
performance, reliability, and flexibility.

It should be noted that the problem I ran into was not Exim's, but Exiscan's.
True, Exim does have a hard limit on header size, which may not be the
greatest thing, but Exiscan is what wasn't doing its housekeeping and causing
trouble. We're we running vanilla Exim, this never would have come up, even
if somebody was purposefully trying to exceed the size limit.

And I would also like to note that the _only_ instances I could find of this
header size limit error are caused by Spamassassin. This issue came up in
pre-2.40 SA versions, and a patch was included in future SA versions to fix
it. See:
http://www.exim.org/pipermail/exim-users/Week-of-Mon-20020923/044137.html
- and -
http://bugzilla.spamassassin.org/show_bug.cgi?id=444

Why not patch it again for a new breed of rules?

> You may have no need to see all those match headers listed, but
> what about developers trying to see where their rules hit, or
> an admin trying to debug a particular message that went wrong.

So it should have a granularity setting or something. We don't need every
possible bit of information on every message. However, I still don't think
what I'm saying means loosing all that much info. With these types of rules,
it's only the difference between knowing if it matched gibberish of 'bhj' or
'ehx' or a bogus html tag of <asdf> versus <jkl;>. Not stuff that usually has
a whole lot of use for anybody but the developer. I say a summary of the
whole ruleset would be better.

> If you limit yourself to Exim, that's your choice but please don't
> expect the rest of the SA world to cripple our tool to fit your
> limitations.

Regardless of limitations, I don't particularly WANT more than 8k or so of
headers. How much is too much for SA to spit out? 8k? 32k? 128k? Shall there
be no limit at all? I would like to see every significant rule that hits, but
I don't necessarily want the full report of every single rule in a huge
Tripwire/Chickenpox-style set.

> Maybe your time would be better spent over in the Exim community
> encouraging its developers to improve their program to the level
> that other MTAs have already achieved.

I did post a bug report to the Exiscan list. Again, this isn't an error in
Exim itself.
--
Matt
Systems Administrator
Local Access Communications
360.330.5535