Mailing List Archive

On Wednesday, Oct 15, 2003, at 14:57 Europe/Amsterdam, Chris Mason
wrote:

> Hello everyone,
>
> I'm using dbmail 1.2 from cvs (Oct 12), and also reproduced on
> 2.0-alpha1. I'm delivering to dbmail-smtp through procmail.
>
> When I use dbmail-smtp -d alias to deliver, things work fine.
>
> When I use dbmail-smtp -m mailbox -u mason to deliver, I get some
> corrupted entries in the database. A look through the tables show the
> corrupted entries have no headers, and watching the logs from
> dbmail-smtp confirms that read_headers() found nothing but a few spaces
> before hitting eof.
>
> This happens to about 1 out of 20 of the messages.
> Does this sound like a known issue?
Does not sound famaliar to me..
If it's both in dbmail 1.2 and 2.0 it's probably some "old" bug.

could you please add some log entries where things go wrong? That
could shed some light on the matter :)

thanks,
Ilja

On Wed, 2003-10-15 at 09:17, Ilja Booij wrote:
> On Wednesday, Oct 15, 2003, at 14:57 Europe/Amsterdam, Chris Mason
> wrote:
>
> > Hello everyone,
> >
> > I'm using dbmail 1.2 from cvs (Oct 12), and also reproduced on
> > 2.0-alpha1. I'm delivering to dbmail-smtp through procmail.
> >
> > When I use dbmail-smtp -d alias to deliver, things work fine.
> >
> > When I use dbmail-smtp -m mailbox -u mason to deliver, I get some
> > corrupted entries in the database. A look through the tables show the
> > corrupted entries have no headers, and watching the logs from
> > dbmail-smtp confirms that read_headers() found nothing but a few spaces
> > before hitting eof.
> >
> > This happens to about 1 out of 20 of the messages.
> > Does this sound like a known issue?
> Does not sound famaliar to me..
> If it's both in dbmail 1.2 and 2.0 it's probably some "old" bug.
>
> could you please add some log entries where things go wrong? That
> could shed some light on the matter :)

It seems to have something to do with procmail, I can't reproduce when
inserting the messages manually. Looking harder...

-chris

On Wed, 2003-10-15 at 09:39, Chris Mason wrote:

[ empty messages in database ]

> > > This happens to about 1 out of 20 of the messages.
> > > Does this sound like a known issue?
> > Does not sound famaliar to me..
> > If it's both in dbmail 1.2 and 2.0 it's probably some "old" bug.
> >
> > could you please add some log entries where things go wrong? That
> > could shed some light on the matter :)
>
> It seems to have something to do with procmail, I can't reproduce when
> inserting the messages manually. Looking harder...

Sorry for the noise, it was a procmail problem. fixed now.

-chris

Chris Mason wrote:

>
> It seems to have something to do with procmail, I can't reproduce when
> inserting the messages manually. Looking harder...

I noticed that you insert messages via "dbmail-smtp -m mailbox -u mason",
I've got approx 20 users using procmail to insert their mail with
dbmail-smtp, but I do it via "dbmail-smtp -u username" without specifying a
mailbox and haven't found any problems yet. I'm not sure if this has got
anything to do with it?

On a second note, I've been thinking that with the procmail delivery support
calling dbmail-smtp -u/-m, by the time it comes round to inserting the mail,
if the database is not contactable, then the mail is lost (bounced or
whatever) rather than being inserted into the database.

My idea round this was, that when dbmail-smtp finds that it cannot contact
the database, rather than just exiting, it should write a file in a
configurable directory. Each file should be the mail it cannot delivery
because of this database problem - one file per mail. The filename could be
something useful, since we know that the process is called with a username,
the filename could be something like
"$username-$mailbox-$date/time-dbmail-deffered.mail". Re-insertion should be
easy, since each file is a mailbox in effect, and raw-convert.c does this
function for us.

Matt

On Wed, 2003-10-15 at 09:55, Matt Dickinson wrote:
> Chris Mason wrote:
>
> >
> > It seems to have something to do with procmail, I can't reproduce when
> > inserting the messages manually. Looking harder...
>
> I noticed that you insert messages via "dbmail-smtp -m mailbox -u mason",
> I've got approx 20 users using procmail to insert their mail with
> dbmail-smtp, but I do it via "dbmail-smtp -u username" without specifying a
> mailbox and haven't found any problems yet. I'm not sure if this has got
> anything to do with it?
>

The docs made me think so (saying -u was less tested), which was my
.procmailrc error went unnoticed for so long ;-)

> On a second note, I've been thinking that with the procmail delivery support
> calling dbmail-smtp -u/-m, by the time it comes round to inserting the mail,
> if the database is not contactable, then the mail is lost (bounced or
> whatever) rather than being inserted into the database.
>
> My idea round this was, that when dbmail-smtp finds that it cannot contact
> the database, rather than just exiting, it should write a file in a
> configurable directory. Each file should be the mail it cannot delivery
> because of this database problem - one file per mail. The filename could be
> something useful, since we know that the process is called with a username,
> the filename could be something like
> "$username-$mailbox-$date/time-dbmail-deffered.mail". Re-insertion should be
> easy, since each file is a mailbox in effect, and raw-convert.c does this
> function for us.

procmail can do much of this for you, when the dbmail-smtp fails it
falls back to deliver into the default spool file. This gives you
/var/spool/mail/$user

When the database comes back up, run formail -s procmail <
/var/spool/mail/$user as each user and things will get delivered. It
won't be super fast since you're forking a procmail for each message but
it'll work.

-chris

Chris Mason wrote:
>
> procmail can do much of this for you, when the dbmail-smtp fails it
> falls back to deliver into the default spool file. This gives you
> /var/spool/mail/$user
>
> When the database comes back up, run formail -s procmail <
> /var/spool/mail/$user as each user and things will get delivered. It
> won't be super fast since you're forking a procmail for each message
> but it'll work.

Then maybe my .procmailrc is wrong :-(

Currently it's:

:0fw: spamassassin.lock
| spamc

:0fw
| /usr/local/sbin/dbmail-smtp -u username

Each time the mail gets inserted into the database, the users' mailbox in
/var/spool/mail/username gets increased by 1 byte in size - this in fact is
just a new line added to the end of the mailbox for each mail delivered,
which in turn leads to a mangled mailbox. Is there a way round this?

Matt

PS: sorry slightly OT...

On Wed, 2003-10-15 at 10:19, Matt Dickinson wrote:
> Chris Mason wrote:
> >
> > procmail can do much of this for you, when the dbmail-smtp fails it
> > falls back to deliver into the default spool file. This gives you
> > /var/spool/mail/$user
> >
> > When the database comes back up, run formail -s procmail <
> > /var/spool/mail/$user as each user and things will get delivered. It
> > won't be super fast since you're forking a procmail for each message
> > but it'll work.
>
> Then maybe my .procmailrc is wrong :-(
>
> Currently it's:
>
> :0fw: spamassassin.lock
> | spamc
>
> :0fw
> | /usr/local/sbin/dbmail-smtp -u username
>
> Each time the mail gets inserted into the database, the users' mailbox in
> /var/spool/mail/username gets increased by 1 byte in size - this in fact is
> just a new line added to the end of the mailbox for each mail delivered,
> which in turn leads to a mangled mailbox. Is there a way round this?

The f in :0fw tells procmail the pipe is a filter, so it expects to put
the resulting output somewhere, instead of thinking the pipe is a
delivery mechanism. Try it without the f.

Also try setting
LOGFILE=$MAILDIR/log and then look at the log file to see where procmail
is sending things.

-chris

Chris Mason wrote:
> The f in :0fw tells procmail the pipe is a filter, so it expects to
> put the resulting output somewhere, instead of thinking the pipe is a
> delivery mechanism. Try it without the f.
>
> Also try setting
> LOGFILE=$MAILDIR/log and then look at the log file to see where
> procmail is sending things.

Doh! Thanks. I feel quite silly now, there was I starting to write a patch
and procmail does it already :-(

Thanks for the help,

Matt