Mailing List Archive: Spamassassin and transport failure

Spamassassin and transport failure

May 17, 2002, 9:10 AM

Post #1 of 7 (3118 views)

Hello,

I have been looking to use spamassassin (version 2.20) on our central
mailhubs. I have run it successfully on my linux PC with Exim 4.04. So I
thought I'd move it onto one of the mailhubs - a Sun Solaris 8 box with Exim
3.36.

It all seems to work okay for messages submitted from the mailhub itself
(i.e. I log onto the mailhub and invoke:
exim jhorne@tracy.csd.plymouth.ac.uk
('tracy' is the name of the system). The message headers show the
received protocol as 'spam-scanned'. However, if I return to my work PC and
send a message so that it goes via 'tracy', then I get an error returned:

The following address(es) failed:

jhorne@tracy.csd.plymouth.ac.uk
Child process of spamcheck transport returned 2 from command:
/usr/local/exim/bin/exim

The following text was generated during the delivery attempt:

------ jhorne@tracy.csd.plymouth.ac.uk ------

An error was detected while processing a file of BSMTP input.
The error message was:

554 Unexpected end of file

The SMTP transaction started in line 0.
The error was detected in line 4.
0 previous messages were successfully processed.
The rest of the batch was abandoned.
554 Unexpected end of file
Transaction started in line 0
Error detected in line 4

==========================================

It then gives a copy of the original message. I installed spamassassin
according to the web site:
http://dman.ddts.net/~dman/config_docs/exim3_spamassassin.html

I am using spamd which was started as root.

The relevant bits of the Exim configure file are:

# Spam Assassin (transport)
spamcheck:
driver = pipe
command = /usr/local/exim/bin/exim -oMr spam-scanned -bS
transport_filter = /usr/local/bin/spamc
bsmtp = all
home_directory = /tmp
current_directory = /tmp
user = exim
group = mail
no_return_path_add
log_output
return_fail_output
prefix =
suffix =

# Spam Assassin (director)
spamcheck_router:
no_expn
no_verify
domains = tracy.csd.plymouth.ac.uk : localhost
local_parts = jhorne
condition = "${if and { {!def:h_X-Spam-Flag:} \
{!eq {$received_protocol}{spam-scanned}}} {1}{0}}"
driver = smartuser
transport = spamcheck

-------------------

I have searched the mailing list archives and can find similar problems when
using the spamcheck.pl script (?), but no real solution. I can find no-one
having this problem when using spamd. I should add that for the trusted
users 'exim' is included, so I can see no reason to have a problem with the
pipe. The error seems to indicate that it is a problem with running the exim
binary, but I have no idea what it is - why does it get an unexpected end of
file? Because local messages work okay I can't run exim with '-d9' since it
won't show any problem (actually I have just done that and I'll check the
output, but it looks okay and no indication of what may cause a problem
remotely).

Very strange. Anyway, as usual these things happen on a Friday afternoon :-)

If anyone has any ideas about this then I'd be grateful.

John.

------------------------------------------------------------------------
John Horne, University of Plymouth, UK Tel: +44 (0)1752 233914
E-mail: jhorne@plymouth.ac.uk
PGP key available from public key servers

Re: Spamassassin and transport failure [ In reply to ]

jpdalbec at cc

May 17, 2002, 9:52 AM

Post #2 of 7 (3058 views)

Permalink

John Horne wrote:
>
> Hello,
>
> I have been looking to use spamassassin (version 2.20) on our central
> mailhubs. I have run it successfully on my linux PC with Exim 4.04. So I
> thought I'd move it onto one of the mailhubs - a Sun Solaris 8 box with Exim
> 3.36.
>
> It all seems to work okay for messages submitted from the mailhub itself
> (i.e. I log onto the mailhub and invoke:
> exim jhorne@tracy.csd.plymouth.ac.uk
> ('tracy' is the name of the system). The message headers show the
> received protocol as 'spam-scanned'. However, if I return to my work PC and
> send a message so that it goes via 'tracy', then I get an error returned:
>
> The following address(es) failed:
>
> jhorne@tracy.csd.plymouth.ac.uk
> Child process of spamcheck transport returned 2 from command:
> /usr/local/exim/bin/exim
>
> The following text was generated during the delivery attempt:
>
> ------ jhorne@tracy.csd.plymouth.ac.uk ------
>
> An error was detected while processing a file of BSMTP input.
> The error message was:
>
> 554 Unexpected end of file
>
> The SMTP transaction started in line 0.
> The error was detected in line 4.
> 0 previous messages were successfully processed.
> The rest of the batch was abandoned.
> 554 Unexpected end of file
> Transaction started in line 0
> Error detected in line 4
>
> ==========================================
>
> It then gives a copy of the original message. I installed spamassassin
> according to the web site:
> http://dman.ddts.net/~dman/config_docs/exim3_spamassassin.html
>
> I am using spamd which was started as root.

What flags did you start spamd with? I think you need -F 0 at a
minimum. Otherwise spamd will add an mbox-style "From " line.
John
>
> The relevant bits of the Exim configure file are:
>
> # Spam Assassin (transport)
> spamcheck:
> driver = pipe
> command = /usr/local/exim/bin/exim -oMr spam-scanned -bS
> transport_filter = /usr/local/bin/spamc
> bsmtp = all
> home_directory = /tmp
> current_directory = /tmp
> user = exim
> group = mail
> no_return_path_add
> log_output
> return_fail_output
> prefix =
> suffix =
>
> # Spam Assassin (director)
> spamcheck_router:
> no_expn
> no_verify
> domains = tracy.csd.plymouth.ac.uk : localhost
> local_parts = jhorne
> condition = "${if and { {!def:h_X-Spam-Flag:} \
> {!eq {$received_protocol}{spam-scanned}}} {1}{0}}"
> driver = smartuser
> transport = spamcheck
>
> -------------------
>
> I have searched the mailing list archives and can find similar problems when
> using the spamcheck.pl script (?), but no real solution. I can find no-one
> having this problem when using spamd. I should add that for the trusted
> users 'exim' is included, so I can see no reason to have a problem with the
> pipe. The error seems to indicate that it is a problem with running the exim
> binary, but I have no idea what it is - why does it get an unexpected end of
> file? Because local messages work okay I can't run exim with '-d9' since it
> won't show any problem (actually I have just done that and I'll check the
> output, but it looks okay and no indication of what may cause a problem
> remotely).
>
> Very strange. Anyway, as usual these things happen on a Friday afternoon :-)
>
> If anyone has any ideas about this then I'd be grateful.
>
> John.
>
> ------------------------------------------------------------------------
> John Horne, University of Plymouth, UK Tel: +44 (0)1752 233914
> E-mail: jhorne@plymouth.ac.uk
> PGP key available from public key servers
>
> --
>
> ## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##

Re: Spamassassin and transport failure [ In reply to ]

tengel at fluid

May 17, 2002, 11:15 AM

Post #3 of 7 (3057 views)

Permalink

Hi John,

I had the same exact problem, and there was no real answer ever posted
to the problem. I saw a few posts on SAtalk list that there were/are
"known problems" with perl < 5.6.0 (I am using redhat 6.2, perl 5.005),
so my *theory* is that it's some obscure dumb perl problem.

For our installation, I solved it with the following steps (I also
started with the same configs you did, on dman's webpage):

1) upgrade to Exim4.04 -- the same *type* of error still happened, but
it was much more informative and gentler to deal with. The upgrade was
a little rocky, but the guys here on the list helped me through it. yay.

2) after watching the logs, I noticed that only *local* mails caused
this weird BSMTP error 2 problem. So (again with help fromt he lists) I
altered the smapcheck_router 'condition' line to ignore all local,
inter-domain email:

condition = "${if and { {!def:h_X-Spam-Flag:} {!eq
{$received_protocol}{spam-scanned}} {!eq {$received_protocol}{local}}
{!eq {$sender_address_domain}{$domain}} } {1}{0}}"

3) move spamcheck_router: to *above* system_aliases int he exim.conf
file. This, coupled with #2, causes things to be scanned or skipped
before the aliases are expanded, which cuts down on re-scanning the same
email sent inbound to multiple people (like staff and project group
aliases, etc).

We've now been running sucessfully for 3 days now without a single BSMTP
error 2 anymore, and spamassassin is doing it's thing. Performance is
also damn good now that local mails are skipped.

hope this helps,
-te

John Horne wrote:
> Hello,
>
> I have been looking to use spamassassin (version 2.20) on our central
> mailhubs. I have run it successfully on my linux PC with Exim 4.04. So I
> thought I'd move it onto one of the mailhubs - a Sun Solaris 8 box with Exim
> 3.36.
>
> It all seems to work okay for messages submitted from the mailhub itself
> (i.e. I log onto the mailhub and invoke:
> exim jhorne@tracy.csd.plymouth.ac.uk
> ('tracy' is the name of the system). The message headers show the
> received protocol as 'spam-scanned'. However, if I return to my work PC and
> send a message so that it goes via 'tracy', then I get an error returned:
>
> The following address(es) failed:
>
> jhorne@tracy.csd.plymouth.ac.uk
> Child process of spamcheck transport returned 2 from command:
> /usr/local/exim/bin/exim
>
> The following text was generated during the delivery attempt:
>
> ------ jhorne@tracy.csd.plymouth.ac.uk ------
>
> An error was detected while processing a file of BSMTP input.
> The error message was:
>
> 554 Unexpected end of file
>
> The SMTP transaction started in line 0.
> The error was detected in line 4.
> 0 previous messages were successfully processed.
> The rest of the batch was abandoned.
> 554 Unexpected end of file
> Transaction started in line 0
> Error detected in line 4
>
> ==========================================
>
> It then gives a copy of the original message. I installed spamassassin
> according to the web site:
> http://dman.ddts.net/~dman/config_docs/exim3_spamassassin.html
>
> I am using spamd which was started as root.
>
> The relevant bits of the Exim configure file are:
>
> # Spam Assassin (transport)
> spamcheck:
> driver = pipe
> command = /usr/local/exim/bin/exim -oMr spam-scanned -bS
> transport_filter = /usr/local/bin/spamc
> bsmtp = all
> home_directory = /tmp
> current_directory = /tmp
> user = exim
> group = mail
> no_return_path_add
> log_output
> return_fail_output
> prefix =
> suffix =
>
> # Spam Assassin (director)
> spamcheck_router:
> no_expn
> no_verify
> domains = tracy.csd.plymouth.ac.uk : localhost
> local_parts = jhorne
> condition = "${if and { {!def:h_X-Spam-Flag:} \
> {!eq {$received_protocol}{spam-scanned}}} {1}{0}}"
> driver = smartuser
> transport = spamcheck
>
> -------------------
>
> I have searched the mailing list archives and can find similar problems when
> using the spamcheck.pl script (?), but no real solution. I can find no-one
> having this problem when using spamd. I should add that for the trusted
> users 'exim' is included, so I can see no reason to have a problem with the
> pipe. The error seems to indicate that it is a problem with running the exim
> binary, but I have no idea what it is - why does it get an unexpected end of
> file? Because local messages work okay I can't run exim with '-d9' since it
> won't show any problem (actually I have just done that and I'll check the
> output, but it looks okay and no indication of what may cause a problem
> remotely).
>
> Very strange. Anyway, as usual these things happen on a Friday afternoon :-)
>
> If anyone has any ideas about this then I'd be grateful.
>
>
> John.
>
> ------------------------------------------------------------------------
> John Horne, University of Plymouth, UK Tel: +44 (0)1752 233914
> E-mail: jhorne@plymouth.ac.uk
> PGP key available from public key servers
>
> --
>
> ## List details at http://www.exim.org/mailman/listinfo/exim-users Exim details at http://www.exim.org/ ##
>

--
Troy Engel, Systems Engineer
Hockey. Kinda like Figure Skating in a War Zone.

Re: Spamassassin and transport failure [ In reply to ]

J.Horne at plymouth

May 17, 2002, 1:19 PM

Post #4 of 7 (3061 views)

Permalink

On Friday 17 May 2002 19:15, Troy Engel wrote:
> I had the same exact problem, and there was no real answer ever posted
> to the problem. I saw a few posts on SAtalk list that there were/are
> "known problems" with perl < 5.6.0 (I am using redhat 6.2, perl 5.005),
> so my *theory* is that it's some obscure dumb perl problem.
>
The mailhub has perl version 5.6 installed:
perl -V
Summary of my perl5 (revision 5.0 version 6 subversion 1) configuration:

> For our installation, I solved it with the following steps (I also
> started with the same configs you did, on dman's webpage):
>
> 1) upgrade to Exim4.04 -- the same *type* of error still happened, but
> it was much more informative and gentler to deal with. The upgrade was
> a little rocky, but the guys here on the list helped me through it. yay.
>
Can't do that at the moment. Probably a couple of months when all the
students have gone home.

> 2) after watching the logs, I noticed that only *local* mails caused
> this weird BSMTP error 2 problem. So (again with help fromt he lists) I
> altered the smapcheck_router 'condition' line to ignore all local,
> inter-domain email:
>
Nope, I get the opposite. The local mail is fine, it's the non-local mail
that's the problem.

> 3) move spamcheck_router: to *above* system_aliases int he exim.conf
> file. This, coupled with #2, causes things to be scanned or skipped
> before the aliases are expanded, which cuts down on re-scanning the same
> email sent inbound to multiple people (like staff and project group
> aliases, etc).
>
Already done.

Also From John Dalbec:
> What flags did you start spamd with? I think you need -F 0 at a
> minimum. Otherwise spamd will add an mbox-style "From " line.
>
Yup, already done that. spamd starts with '-d -x -F 0'.

I'll keep looking at it, but anymore suggestions would be welcome.

John.
--
John Horne, University of Plymouth, UK Tel: +44 (0)1752 233914
E-mail: jhorne@plymouth.ac.uk
PGP key available from public key servers

Re: Spamassassin and transport failure [ In reply to ]

J.Horne at plymouth

May 17, 2002, 4:04 PM

Post #5 of 7 (3069 views)

Permalink

On Friday 17 May 2002 17:10, John Horne wrote:
> The following address(es) failed:
>
> jhorne@tracy.csd.plymouth.ac.uk
> Child process of spamcheck transport returned 2 from command:
> /usr/local/exim/bin/exim
>
> The following text was generated during the delivery attempt:
>
> ------ jhorne@tracy.csd.plymouth.ac.uk ------
>
> An error was detected while processing a file of BSMTP input.
> The error message was:
>
> 554 Unexpected end of file
>
Okay, I've got this solved :-) I can't debug exim remotely, but of course the
transport is invoking exim for me, so I changed my exim configure to say:

command = /usr/local/exim/bin/exim -oMr spam-scanned -bS -d9

Restart exim. I then sent a message from my work PC and all the debugging was
returned as part of the error message. I received (the relevant bit):

SMTP<< MAIL FROM:<jhorne@plymouth.ac.uk>
SMTP<< RCPT TO:<jhorne@tracy.csd.plymouth.ac.uk>
SMTP<< DATA
search_tidyup called
>>Original headers (size=90):
ld.so.1: /usr/local/bin/spamc: fatal: libgdbm.so.2: open failed: No such
file or directory

LOG: 4 MAIN
unexpected EOF while reading SMTP data (after header) from exim
Handling error in batched SMTP input
An error was detected while processing a file of BSMTP input.

The problem was that spamc wasn't finding the libgdbm libraries. The
libraries were installed into /usr/local/lib, and I have that path in the
/etc/profile file, so I'm a little lost as to why it had a problem (must be
the way the process is forked? I'll have to read a bit more about pipe
transports). Anyway, I created soft links of the libraries in /usr/lib, and
it then started to work okay :-)

John.
--
John Horne, University of Plymouth, UK Tel: +44 (0)1752 233914
E-mail: jhorne@plymouth.ac.uk
PGP key available from public key servers

Re: Spamassassin and transport failure [ In reply to ]

sys044 at abdn

May 22, 2002, 6:37 AM

Post #6 of 7 (3060 views)

Permalink

My problems is when the system runs out of resources, the time to scan
takes longer and longer. Yesterday is was failing on a few 100 messages
but I think I needed more tday. In the spamd log it shows by the time to
scan going from 0 or 1 sec to 20 to 100 to .... and finally it breaks the
timeout on [B]SMTP.

The error mesage is givben below with d9 as you suggested.

I use Exiscan v1 patched and modified so I can force it to limit the
number of exims it spawns (and then spamd's). It now fails at 50 but works
fine at 5.

I have also included the message size in the transport condition so as not
to even call spamd if the message is too large. I expect that the SA
whitelist would be better in the EXIM condition.

However the fact remains that it can fail and start bouncing messages is
what concerns me. I cannot find anything about pipes failing but that is
the crux of the problem now.

QUESTION: does anyone know how to trap errors in PIPES and hold the mail
for a retry?

John Linn

Re: Spamassassin and transport failure [ In reply to ]

dsh8290 at rit

May 22, 2002, 6:59 AM

Post #7 of 7 (3069 views)

Permalink

--
On Wed, May 22, 2002 at 02:37:50PM +0100, j.linn wrote:
| My problems is when the system runs out of resources, the time to scan
| takes longer and longer.

How about looking at the deliver_queue_load_max option?

| Yesterday is was failing on a few 100 messages but I think I needed
| more tday. In the spamd log it shows by the time to scan going from
| 0 or 1 sec to 20 to 100 to .... and finally it breaks the timeout on
| [B]SMTP.

I managed to do that once :-). I had just brought my system up and
was trying to transfer the mails that had landed in my inbox at school
to my own machine. With ~900 messages arriving as fast as formail and
sendmail could transfer them my system load average neared 30. Oops.
After a couple of minutes spamd got through all the messages and the
load went down. (Then I realized I messed up the formail command and
hosed the messages. I started the process over again, but after
setting the load_max option.)

-D

--

"640K ought to be enough for anybody" -Bill Gates, 1981

GnuPG key : http://dman.ddts.net/~dman/public_key.gpg

--
[ Content of type application/pgp-signature deleted ]
--