Mailing List Archive

Slow LMTP Inserts
Howdy folks,

I'm running dbmail 3.2.3 on a postgresql db and am having slow inserts.
About 2 seconds each. This is not really ok and it tends to get behind
during peak usage or when someone emails every mailbox. I read that 2
seconds a message is not abnormal, can anyone confirm this?

My solution is to run balance (https://www.inlab.de/balance.html) to round
robin postfix to 4 instances of dbmail-lmtp. I've been running it this way
for a couple days and it looks like it is going to work. Does anyone else
do it this way or see a reason for me not to run it this way?

I did try to run dbmail-lmtp in stdin/stdout mode and call it from xinetd
but the processes weren't ending like I expected and I ended up with a lot
of them running and having file pointer errors eventually. Has anyone made
it work with xinetd or inetd and can share the config with me?

Not really related but my pg_dump backup is a little over twice the size of
my db on disk. Is this normal or should I start worrying about that?

Thanks
Rich
Re: Slow LMTP Inserts [ In reply to ]
Let's got in order:

* Slow insert. It's not "normal". You'd better check what you have in
the dbmail_headername tables, clean it up, and probabily disable the
dbmail_headername auto-insert. Can you give a few information on the DB
size as well? I'm not using pgsql but probably checking the indexes
would help a lot.
* LMTPD over INETD. There's a bug in the code (there is an open issue
for that) that causes lmtpd to get stuck in an infinite loop when used
thought inetd and the remote session get closed while receiving a mail.
I don't think there will be a fix in a short time for that BUT you can
overcome that with an utility called "timelimit". It just sits in the
pipe of the commands and if a subsequent command runs for more than X
seconds it kills it.

In my inetd.conf I'm running:

lmtpd stream tcp nowait/15 root /usr/local/bin/timelimit timelimit -t120
-T120 /usr/local/sbin/dbmail-lmtpd -n

And it works like a charm. It clearly doesn't fix the problem but at
least it gives you a working situation.

I can confirm that having multiple lmtpd dispatched by inetd is much
faster than the daemon. Furthermore it helps you a lot with the DB
Connection pool and avoid memory leaks.

---

Andrea Brancatelli
Schema31 S.p.a.
Chief Technology Officier

ROMA - FI - PA
ITALY
Tel: +39.06.98.358.472
Cell: +39.331.2488468
Fax: +39.055.71.880.466
Società del Gruppo OVIDIO TECH S.R.L.

On 2018-02-26 01:44, rich carroll wrote:

> Howdy folks,
>
> I'm running dbmail 3.2.3 on a postgresql db and am having slow inserts. About 2 seconds each. This is not really ok and it tends to get behind during peak usage or when someone emails every mailbox. I read that 2 seconds a message is not abnormal, can anyone confirm this?
>
> My solution is to run balance (https://www.inlab.de/balance.html) to round robin postfix to 4 instances of dbmail-lmtp. I've been running it this way for a couple days and it looks like it is going to work. Does anyone else do it this way or see a reason for me not to run it this way?
>
> I did try to run dbmail-lmtp in stdin/stdout mode and call it from xinetd but the processes weren't ending like I expected and I ended up with a lot of them running and having file pointer errors eventually. Has anyone made it work with xinetd or inetd and can share the config with me?
>
> Not really related but my pg_dump backup is a little over twice the size of my db on disk. Is this normal or should I start worrying about that?
>
> Thanks
> Rich
>
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> http://lists.nfg.nl/mailman/listinfo/dbmail
Re: Slow LMTP Inserts [ In reply to ]
Andrea,

My postfix lmtp usually has a delay of 2.2 (delays=0.01/0.01/0/2.2) but
when it gets busy they can be higher, but they are never lower then 2.

table rows
dbmail_users 2246
dbmail_header 135684624
dbmail_phymessage 6315243

I don't know if those are big numbers or not. It seems like I have alot of
indexes. I upgraded from 2.2 this year. I could get a dump of the structure
if that would help anyone. I don't see any triggers anywhere but I am not
sure if I have looked in all the right spots.





I am using xinetd so I got it working like this:

service lmtp-xinetd
{
port = 9998
disable = no
socket_type = stream
protocol = tcp
wait = no
user = root
server = /usr/bin/timelimit
server_args = -t120 -T120 /usr/local/sbin/dbmail-lmtpd -n

}

but shortly after it starting up I get 4 dbmail-lmtpd processes sitting at
100% (lmtp_destination_concurrency_limit = 4)

root 36834 98.0 0.0 154388 12400 ? Rl 20:24 5:00
/usr/local/sbin/dbmail-lmtpd -n
root 37238 96.9 0.0 154572 13060 ? Rl 20:25 3:31
/usr/local/sbin/dbmail-lmtpd -n
root 37248 96.8 0.0 154388 12548 ? Rl 20:25 3:27
/usr/local/sbin/dbmail-lmtpd -n
root 37253 96.8 0.0 154388 12636 ? Rl 20:25 3:27
/usr/local/sbin/dbmail-lmtpd -n

I have switched it back to running 4 dbmail-lmtpd daemons running on ports
24 26 27 and 28 with with postfix feeding them to balance on port 9999 and
balance switching between the 4 dbmail-lmtpd processes.

/usr/bin/balance 9999 localhost:24 localhost:26 localhost:27 localhost:28

And they do not have the same issues as running in -n mode.

dbmail 38406 0.0 0.0 164252 14420 ? Sl 20:31 0:00
/usr/local/sbin/dbmail-lmtpd -f /etc/dbmail/dbmail.conf
dbmail 38416 0.0 0.0 165224 14996 ? Sl 20:31 0:00
/usr/local/sbin/dbmail-lmtpd -f /etc/dbmail/dbmail2.conf -p
/var/run/dbmail/dbmail-lmtpd2.pid
dbmail 38426 0.0 0.0 162528 12452 ? Sl 20:31 0:00
/usr/local/sbin/dbmail-lmtpd -f /etc/dbmail/dbmail3.conf -p
/var/run/dbmail/dbmail-lmtpd3.pid
dbmail 38436 0.0 0.0 162320 12376 ? Sl 20:31 0:00
/usr/local/sbin/dbmail-lmtpd -f /etc/dbmail/dbmail4.conf -p
/var/run/dbmail/dbmail-lmtpd4.pid

However when I was running only 1 lmtpd I had all sorts of problems, too
many files and maxing out db connections.. but for the last 5 days have
been doing it this way with no issues. I have several 5 minute crons
checking services and syslog for new errors and when it finds one I restart
all mail services. which hasn't triggered since I moved to 4 dbmail-lmtpds.
I will update after running it this way for a few more weeks. I still get
an active queue when spammers get through or they send notifications to all
the mailboxes but it chews through them pretty fast. I don't see a need for
running more then 4 but I bet 10 would work just as good.

Mostly I am wanting to know if there is any reason not to run it this way.
I'd also like more DB performance but I am not sure even where to begin
with the db. I am afraid if I jack with indexes I may mess something up.

Thanks for looking into this
rich



On Tue, Feb 27, 2018 at 11:40 AM, Andrea Brancatelli <
abrancatelli@schema31.it> wrote:

> Let's got in order:
>
> 1. Slow insert. It's not "normal". You'd better check what you have in
> the dbmail_headername tables, clean it up, and probabily disable the
> dbmail_headername auto-insert. Can you give a few information on the DB
> size as well? I'm not using pgsql but probably checking the indexes would
> help a lot.
> 2. LMTPD over INETD. There's a bug in the code (there is an open issue
> for that) that causes lmtpd to get stuck in an infinite loop when used
> thought inetd and the remote session get closed while receiving a mail. I
> don't think there will be a fix in a short time for that *BUT* you can
> overcome that with an utility called "timelimit". It just sits in the pipe
> of the commands and if a subsequent command runs for more than X seconds it
> kills it.
>
> In my inetd.conf I'm running:
>
> lmtpd stream tcp nowait/15 root /usr/local/bin/timelimit timelimit -t120
> -T120 /usr/local/sbin/dbmail-lmtpd -n
>
> And it works like a charm. It clearly doesn't fix the problem but at least
> it gives you a working situation.
>
> I can confirm that having multiple lmtpd dispatched by inetd is much
> faster than the daemon. Furthermore it helps you a lot with the DB
> Connection pool and avoid memory leaks.
>
>
> ---
>
>
> *Andrea Brancatelli
> Schema31 S.p.a.
> Chief Technology Officier*
> ROMA - FI - PA
> ITALY
> Tel: +39.06.98.358.472
> Cell: +39.331.2488468 <+39%20331%20248%208468>
> Fax: +39.055.71.880.466
> Società del Gruppo *OVIDIO TECH S.R.L.*
>
>
> On 2018-02-26 01:44, rich carroll wrote:
>
> Howdy folks,
>
> I'm running dbmail 3.2.3 on a postgresql db and am having slow inserts.
> About 2 seconds each. This is not really ok and it tends to get behind
> during peak usage or when someone emails every mailbox. I read that 2
> seconds a message is not abnormal, can anyone confirm this?
>
> My solution is to run balance (https://www.inlab.de/balance.html) to
> round robin postfix to 4 instances of dbmail-lmtp. I've been running it
> this way for a couple days and it looks like it is going to work. Does
> anyone else do it this way or see a reason for me not to run it this way?
>
> I did try to run dbmail-lmtp in stdin/stdout mode and call it from xinetd
> but the processes weren't ending like I expected and I ended up with a lot
> of them running and having file pointer errors eventually. Has anyone made
> it work with xinetd or inetd and can share the config with me?
>
> Not really related but my pg_dump backup is a little over twice the size
> of my db on disk. Is this normal or should I start worrying about that?
>
> Thanks
> Rich
>
>
>
>
>
>
>
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> http://lists.nfg.nl/mailman/listinfo/dbmail
>
>
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> http://lists.nfg.nl/mailman/listinfo/dbmail
>
>


--
Richard Carroll
richcarroll@gmail.com
785-288-1144
Re: Slow LMTP Inserts [ In reply to ]
Somewhere in the documentation there's some note about LMTPD not being
designed to be run in multiple instances at the same time.

In my experience that is not exactly true. It *CAN* create some unusual
but pretty innocent situations, caused by multiple transactions running
at the same time.

Your tables are pretty small, insertion time should be somewhere around
1 second but actually if the DB machine is busy it can get higher.

My previous mail was too vague about the headers.

Please check the header_cache_readonly setting in dbmail.conf. It should
be set to "yes" or your dbmail_header* will grow to an huge number or
rows slowing down everything.

Your dbmail_headername should be around 200 rows.

If that's not the case (and I think it will probably be not) I can give
you some hints on how to trim it down.

---

Andrea Brancatelli
Schema31 S.p.a.
Chief Technology Officier

ROMA - FI - PA
ITALY
Tel: +39.06.98.358.472
Cell: +39.331.2488468
Fax: +39.055.71.880.466
Società del Gruppo OVIDIO TECH S.R.L.

On 2018-03-01 04:15, rich carroll wrote:

> Andrea,
>
> My postfix lmtp usually has a delay of 2.2 (delays=0.01/0.01/0/2.2) but when it gets busy they can be higher, but they are never lower then 2.
>
> table rows
> dbmail_users 2246
> dbmail_header 135684624
> dbmail_phymessage 6315243
>
> I don't know if those are big numbers or not. It seems like I have alot of indexes. I upgraded from 2.2 this year. I could get a dump of the structure if that would help anyone. I don't see any triggers anywhere but I am not sure if I have looked in all the right spots.
>
> I am using xinetd so I got it working like this:
>
> service lmtp-xinetd
> {
> port = 9998
> disable = no
> socket_type = stream
> protocol = tcp
> wait = no
> user = root
> server = /usr/bin/timelimit
> server_args = -t120 -T120 /usr/local/sbin/dbmail-lmtpd -n
>
> }
>
> but shortly after it starting up I get 4 dbmail-lmtpd processes sitting at 100% (lmtp_destination_concurrency_limit = 4)
>
> root 36834 98.0 0.0 154388 12400 ? Rl 20:24 5:00 /usr/local/sbin/dbmail-lmtpd -n
> root 37238 96.9 0.0 154572 13060 ? Rl 20:25 3:31 /usr/local/sbin/dbmail-lmtpd -n
> root 37248 96.8 0.0 154388 12548 ? Rl 20:25 3:27 /usr/local/sbin/dbmail-lmtpd -n
> root 37253 96.8 0.0 154388 12636 ? Rl 20:25 3:27 /usr/local/sbin/dbmail-lmtpd -n
>
> I have switched it back to running 4 dbmail-lmtpd daemons running on ports 24 26 27 and 28 with with postfix feeding them to balance on port 9999 and balance switching between the 4 dbmail-lmtpd processes.
>
> /usr/bin/balance 9999 localhost:24 localhost:26 localhost:27 localhost:28
>
> And they do not have the same issues as running in -n mode.
>
> dbmail 38406 0.0 0.0 164252 14420 ? Sl 20:31 0:00 /usr/local/sbin/dbmail-lmtpd -f /etc/dbmail/dbmail.conf
>
> dbmail 38416 0.0 0.0 165224 14996 ? Sl 20:31 0:00 /usr/local/sbin/dbmail-lmtpd -f /etc/dbmail/dbmail2.conf -p /var/run/dbmail/dbmail-lmtpd2.pid
> dbmail 38426 0.0 0.0 162528 12452 ? Sl 20:31 0:00 /usr/local/sbin/dbmail-lmtpd -f /etc/dbmail/dbmail3.conf -p /var/run/dbmail/dbmail-lmtpd3.pid
> dbmail 38436 0.0 0.0 162320 12376 ? Sl 20:31 0:00 /usr/local/sbin/dbmail-lmtpd -f /etc/dbmail/dbmail4.conf -p /var/run/dbmail/dbmail-lmtpd4.pid
>
> However when I was running only 1 lmtpd I had all sorts of problems, too many files and maxing out db connections.. but for the last 5 days have been doing it this way with no issues. I have several 5 minute crons checking services and syslog for new errors and when it finds one I restart all mail services. which hasn't triggered since I moved to 4 dbmail-lmtpds. I will update after running it this way for a few more weeks. I still get an active queue when spammers get through or they send notifications to all the mailboxes but it chews through them pretty fast. I don't see a need for running more then 4 but I bet 10 would work just as good.
>
> Mostly I am wanting to know if there is any reason not to run it this way. I'd also like more DB performance but I am not sure even where to begin with the db. I am afraid if I jack with indexes I may mess something up.
>
> Thanks for looking into this
> rich
>
> On Tue, Feb 27, 2018 at 11:40 AM, Andrea Brancatelli <abrancatelli@schema31.it> wrote:
>
> Let's got in order:
>
> * Slow insert. It's not "normal". You'd better check what you have in the dbmail_headername tables, clean it up, and probabily disable the dbmail_headername auto-insert. Can you give a few information on the DB size as well? I'm not using pgsql but probably checking the indexes would help a lot.
> * LMTPD over INETD. There's a bug in the code (there is an open issue for that) that causes lmtpd to get stuck in an infinite loop when used thought inetd and the remote session get closed while receiving a mail. I don't think there will be a fix in a short time for that BUT you can overcome that with an utility called "timelimit". It just sits in the pipe of the commands and if a subsequent command runs for more than X seconds it kills it.
>
> In my inetd.conf I'm running:
>
> lmtpd stream tcp nowait/15 root /usr/local/bin/timelimit timelimit -t120 -T120 /usr/local/sbin/dbmail-lmtpd -n
>
> And it works like a charm. It clearly doesn't fix the problem but at least it gives you a working situation.
>
> I can confirm that having multiple lmtpd dispatched by inetd is much faster than the daemon. Furthermore it helps you a lot with the DB Connection pool and avoid memory leaks.
>
> ---
>
> Andrea Brancatelli
> Schema31 S.p.a.
> Chief Technology Officier
>
> ROMA - FI - PA
> ITALY
> Tel: +39.06.98.358.472
> Cell: +39.331.2488468 [1]
> Fax: +39.055.71.880.466
> Società del Gruppo OVIDIO TECH S.R.L.
>
> On 2018-02-26 01:44, rich carroll wrote:
>
> Howdy folks,
>
> I'm running dbmail 3.2.3 on a postgresql db and am having slow inserts. About 2 seconds each. This is not really ok and it tends to get behind during peak usage or when someone emails every mailbox. I read that 2 seconds a message is not abnormal, can anyone confirm this?
>
> My solution is to run balance (https://www.inlab.de/balance.html [2]) to round robin postfix to 4 instances of dbmail-lmtp. I've been running it this way for a couple days and it looks like it is going to work. Does anyone else do it this way or see a reason for me not to run it this way?
>
> I did try to run dbmail-lmtp in stdin/stdout mode and call it from xinetd but the processes weren't ending like I expected and I ended up with a lot of them running and having file pointer errors eventually. Has anyone made it work with xinetd or inetd and can share the config with me?
>
> Not really related but my pg_dump backup is a little over twice the size of my db on disk. Is this normal or should I start worrying about that?
>
> Thanks
> Rich
>
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> http://lists.nfg.nl/mailman/listinfo/dbmail [3]
>
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> http://lists.nfg.nl/mailman/listinfo/dbmail [3]

--
Richard Carroll
richcarroll@gmail.com
785-288-1144

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://lists.nfg.nl/mailman/listinfo/dbmail



Links:
------
[1] tel:+39%20331%20248%208468
[2] https://www.inlab.de/balance.html
[3] http://lists.nfg.nl/mailman/listinfo/dbmail