Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. DBMail>
  3. users
dbmail hanging
rogerio at porta80
Oct 29, 2014, 7:11 AM
Post #1 of 10 (2732 views)
Permalink
Hi,

I have detected a failure in the dbmail process. Sometimes it hangs and can't recover to normal operations. A restart is needed.

The server logs shows that rsyslog is rate limiting dbmail message logs generation when it hangs.

The /var/log/message displays the following: Oct 29 10:49:26 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting Oct 29 10:49:32 dbmail01 rsyslogd-2177: imuxsock lost 92936 messages from pid 27550 due to rate-limiting Oct 29 10:49:32 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting Oct 29 10:49:38 dbmail01 rsyslogd-2177: imuxsock lost 114578 messages from pid 27550 due to rate-limiting Oct 29 10:49:38 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting Oct 29 10:49:44 dbmail01 rsyslogd-2177: imuxsock lost 119995 messages from pid 27550 due to rate-limiting Oct 29 10:49:44 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting Oct 29 10:49:50 dbmail01 rsyslogd-2177: imuxsock lost 133795 messages from pid 27550 due to rate-limiting Oct 29 10:49:50 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting Oct 29 10:49:56 dbmail01 rsyslogd-2177: imuxsock lost 101392 messages from pid 27550 due to rate-limiting Oct 29 10:49:56 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting Oct 29 10:50:02 dbmail01 rsyslogd-2177: imuxsock lost 131723 messages from pid 27550 due to rate-limiting Oct 29 10:50:02 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting Oct 29 10:50:08 dbmail01 rsyslogd-2177: imuxsock lost 128597 messages from pid 27550 due to rate-limiting Oct 29 10:50:08 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting Oct 29 10:50:15 dbmail01 rsyslogd-2177: imuxsock lost 90241 messages from pid 27550 due to rate-limiting Oct 29 10:50:16 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting Oct 29 10:50:22 dbmail01 rsyslogd-2177: imuxsock lost 86302 messages from pid 27550 due to rate-limiting Oct 29 10:50:22 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting Oct 29 10:50:28 dbmail01 rsyslogd-2177: imuxsock lost 97177 messages from pid 27550 due to rate-limiting Oct 29 10:50:28 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting Oct 29 10:50:34 dbmail01 rsyslogd-2177: imuxsock lost 67308 messages from pid 27550 due to rate-limiting Oct 29 10:50:34 dbmail01 rsyslogd-2177: imuxsock begins to drop messages from pid 27550 due to rate-limiting

Dbmail.conf: file_logging_levels = 7 syslog_logging_levels = 15 Is it possible that Dbmail hangs because of rsyslog throttling? How to prevent it? I am running Dbmail 3.1.17, Debian 7, 200 email accounts, 100 GB PostgreSQL database in size. Sincerely. Rogerio
Re: dbmail hanging [ In reply to ]
h.reindl at thelounge
Oct 29, 2014, 7:30 AM
Post #2 of 10 (2699 views)
Permalink
Am 29.10.2014 um 15:11 schrieb Rogerio Pereira:
> I have detected a failure in the dbmail process. Sometimes it hangs and
> can't recover to normal operations. A restart is needed.
>
> The server logs shows that rsyslog is rate limiting dbmail message logs
> generation when it hangs.
>
> The /var/log/message displays the following:
>
> Oct 29 10:49:26 dbmail01 rsyslogd-2177: imuxsock begins to drop messages
> from pid 27550 due to rate-limiting
> Oct 29 10:49:32 dbmail01 rsyslogd-2177: imuxsock lost 92936 messages
> from pid 27550 due to rate-limiting

no, at the begin it displays something else *before* get throtteled

> Dbmail.conf:
>
> file_logging_levels = 7
> syslog_logging_levels = 15
> Is it possible that Dbmail hangs because of rsyslog throttling? How to
> prevent it?

no, that's just the messenger avoiding to flood logs
what is in the logs *before* it get throttled?

Attached Files:

Re: dbmail hanging [ In reply to ]
rogerio at porta80
Oct 29, 2014, 7:54 AM
Post #3 of 10 (2703 views)
Permalink
Hi,

The /var/log/mail.err file has thousands of this message:

Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server] _sock_cb(+582): 24:Too many open files
Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server] _sock_cb(+582): 24:Too many open files
Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server] _sock_cb(+582): 24:Too many open files
Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server] _sock_cb(+582): 24:Too many open files
Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server] _sock_cb(+582): 24:Too many open files

Sincerely.

Rogerio
Re: dbmail hanging [ In reply to ]
h.reindl at thelounge
Oct 29, 2014, 8:00 AM
Post #4 of 10 (2706 views)
Permalink
Am 29.10.2014 um 15:48 schrieb rogerio@porta80.com.br:
> The /var/log/mail.err file has thousands of this message:
> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
> _sock_cb(+582): 24:Too many open files
> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
> _sock_cb(+582): 24:Too many open files
> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
> _sock_cb(+582): 24:Too many open files
> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
> _sock_cb(+582): 24:Too many open files
> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
> _sock_cb(+582): 24:Too many open files

please stay on the list!

you likely have too many connections for your OS setup
on systemd-systems "LimitNOFILE=50000" may fix this

in case of sysvinit that needs to happen directly in the sysvunit as far
as i remember with ulimit (not touched sysvinit for some years)

but in any case you need to ask yourself if your machine can handle that
much connections or something else goes down

in another post (pleast stay ina thread and try to quote some context)
you talk about DOS attacks - just limit that properly, see below, you
can even limit the number for connections for subnet sizes

0 0 REJECT tcp -- eth0 * !192.168.196.0/24
0.0.0.0/0 multiport dports 110,143,993,995 ctstate NEW
recent: UPDATE seconds: 1800 hit_count: 100 name: dovecot2 side: source
mask:
255.255.255.255 reject-with icmp-port-unreachable
0 0 REJECT tcp -- eth0 * !192.168.196.0/24
0.0.0.0/0 multiport dports 110,143,993,995 ctstate NEW
recent: UPDATE seconds: 300 hit_count: 50 name: dovecot1 side: source
mask: 2
55.255.255.255 reject-with icmp-port-unreachable
0 0 REJECT tcp -- eth0 * !192.168.196.0/24
0.0.0.0/0 multiport dports 25,465,587 ctstate NEW recent:
UPDATE seconds: 1800 hit_count: 75 name: postfix2 side: source mask: 255.2
55.255.255 reject-with icmp-port-unreachable
3 144 REJECT tcp -- eth0 * !192.168.196.0/24
0.0.0.0/0 multiport dports 25,465,587 ctstate NEW recent:
UPDATE seconds: 300 hit_count: 40 name: postfix1 side: source mask: 255.25
5.255.255 reject-with icmp-port-unreachable
0 0 DROP udp -- eth0 * !192.168.196.0/24
0.0.0.0/0 ctstate NEW recent: UPDATE seconds: 2 hit_count:
75 name: udpflood side: source mask: 255.255.255.255
0 0 DROP tcp -- eth0 * !192.168.196.0/24
0.0.0.0/0 ctstate NEW recent: UPDATE seconds: 2 hit_count:
75 name: DEFAULT side: source mask: 255.255.255.255
0 0 DROP tcp -- eth0 * !192.168.196.0/24
0.0.0.0/0 multiport dports 25,80,443,465,587 tcp
flags:0x17/0x02 #conn src/32 > 75
0 0 DROP tcp -- eth0 * !192.168.196.0/24
0.0.0.0/0 multiport dports 25,80,443,465,587 tcp
flags:0x17/0x02 #conn src/24 > 150
0 0 DROP tcp -- eth0 * !192.168.196.0/24
0.0.0.0/0 multiport dports 25,80,443,465,587 tcp
flags:0x17/0x02 #conn src/16 > 175
0 0 DROP tcp -- eth0 * !192.168.196.0/24
0.0.0.0/0 multiport dports 25,80,443,465,587 tcp
flags:0x17/0x02 #conn src/8 > 200

Attached Files:

Re: dbmail hanging [ In reply to ]
rogerio at porta80
Oct 29, 2014, 11:54 AM
Post #5 of 10 (2693 views)
Permalink
Hi,

I am not a Debian expert, but I guess the following config should solve the “too many open files” issue. I don’t have systemd installed, so I need to change it in Debian config files.

--> /etc/security/limits.conf

* soft nofile 4096
* hard nofile 32768
root soft nofile 4096
root hard nofile 32768

Should I add these lines to /etc/init.d/dbmail as well?

ulimit -Hn 32768
ulimit -Sn 32768

Thanks in advance.

Rogerio
Re: dbmail hanging [ In reply to ]
lordvan at lordvan
Oct 30, 2014, 1:31 AM
Post #6 of 10 (2701 views)
Permalink
Am 29.10.2014 um 15:54 schrieb rogerio@porta80.com.br:
> Hi,
>
> The /var/log/mail.err file has thousands of this message:
>
> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
> _sock_cb(+582): 24:Too many open files
> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
> _sock_cb(+582): 24:Too many open files
> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
> _sock_cb(+582): 24:Too many open files
> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
> _sock_cb(+582): 24:Too many open files
> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
> _sock_cb(+582): 24:Too many open files
>
Hi

I have a similar problem, but with timsieved -- it seems that my problem
is caused by monit checking the port (in case of timsieved i do not
really care too much though and just restart it once per hour for good
measure).

Do you have anything that checks if your services are alive? If so:
is it just a "connection open" test or a "proper" imap test ?

I think that sometimes dbmail leaves sockets "open" when there is no
proper commands sent -- still got to debug that at some point though -
since I do not have the problem with imapd but monit has a proper imap
check (and no timsieved check )

Regards,
Thomas R


_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
Re: dbmail hanging [ In reply to ]
ahicks at p-o
Oct 30, 2014, 2:12 AM
Post #7 of 10 (2700 views)
Permalink
Hi,

On 30/10/2014 08:31, Thomas Raschbacher wrote:
> Am 29.10.2014 um 15:54 schrieb rogerio@porta80.com.br:
>> Hi,
>>
>> The /var/log/mail.err file has thousands of this message:
>>
>> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
>> _sock_cb(+582): 24:Too many open files
>> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
>> _sock_cb(+582): 24:Too many open files
>> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
>> _sock_cb(+582): 24:Too many open files
>> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
>> _sock_cb(+582): 24:Too many open files
>> Oct 29 10:49:26 dbmail01 dbmail/imap4d[27550]: Error:[server]
>> _sock_cb(+582): 24:Too many open files
>>
> Hi
>
> I have a similar problem, but with timsieved -- it seems that my problem
> is caused by monit checking the port (in case of timsieved i do not
> really care too much though and just restart it once per hour for good
> measure).

[shameless plug]
It's easy to check that timsieve is running with Nagios or Icinga,
perhaps it's easy to adapt to Monit?
http://p-o.co.uk/tech-articles/nagios-plugin-check-sieve/
[/shameless plug]

Regards,
Alan
>
> Do you have anything that checks if your services are alive? If so:
> is it just a "connection open" test or a "proper" imap test ?
>
> I think that sometimes dbmail leaves sockets "open" when there is no
> proper commands sent -- still got to debug that at some point though -
> since I do not have the problem with imapd but monit has a proper imap
> check (and no timsieved check )
>
> Regards,
> Thomas R
>
>
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
>

--
Persistent Objects Ltd
128 Lilleshall Road
Morden SM4 6DR

The Home of Lasting Solutions

Mobile: 079 3030 5004
Tel: 020 8544 5292
Web: p-o.co.uk
Skype: alan-hicks-london
Personal blog https://plus.google.com/+AlanHicksLondon
Company blog https://plus.google.com/+PoCoUkLondon/posts
LinkedIn https://uk.linkedin.com/in/alanhickslondon/
GitHub https://github.com/alan-hicks
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
Re: dbmail hanging [ In reply to ]
rogerio at porta80
Oct 30, 2014, 6:23 AM
Post #8 of 10 (2696 views)
Permalink
>>Do you have anything that checks if your services are alive? If so:
>> is it just a "connection open" test or a "proper" imap test ?

Yes, I have. HostMonitor is installed in my Android Phone and queries the imap, pop ports every 2 minutes to see if it’s alive. It’s a not “proper” imap test. It’s just a “connection open” test as you mentioned.

I am monitoring the production server because of the DBmail stability issues. Service uptime is a huge deal for me right now.

I am monitoring the “opened files” by dbmail imapd process and I can see it grows steadily every minute (ls -l /proc/<pid>/fd | wc –l). I changed the “open files” limit to 4096 in the server. It may fix the imapd issue, but it does not fix the pop3d issues. Pop3d hangs every day with “general protection” errors.

dbmail01 kernel: [21981989.961192] dbmail-pop3d[9821] general protection ip:7f92daba880a sp:7fff2cb2ae38 error:0 in libdbmail.so.0.0.0[7f92dab64000+6b000]

Is Pop3d failure related to this monitoring as well?

What changes in dbmail.conf could fix this “connection leak” created by monitoring?

I am running dbmail 3.1.17, Pgsql 9.3.

Thanks.

Rogerio
Re: dbmail hanging [ In reply to ]
paul at nfg
Nov 2, 2014, 5:29 AM
Post #9 of 10 (2674 views)
Permalink
I wonder why you need to fiddle with those limits to begin with. Haven't
had to do that in ages.

Are you actually trying to handle 10k+ concurrent connections?

Is there some program that is trying to connect to dbmail without
closing the connections? Such connections are cleaned up by dbmail, but
that may require some timeout to kick in first (login_timeout).

try: lsof -p `pidof dbmail-imapd`
and see of that tells you something.



On 29-10-14 19:54, Rogerio Pereira wrote:
> Hi,
>
> I am not a Debian expert, but I guess the following config should solve
> the “too many open files” issue. I don’t have systemd installed, so I
> need to change it in Debian config files.
>
> --> /etc/security/limits.conf
>
> * soft nofile 4096
> * hard nofile 32768
> root soft nofile 4096
> root hard nofile 32768
>
> Should I add these lines to /etc/init.d/dbmail as well?
>
> ulimit -Hn 32768
> ulimit -Sn 32768
>
> Thanks in advance.
>
> Rogerio
>
>
>
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
>

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, github, linkedin
www.nfg.nl/info@nfg.nl/+31.85.877.99.97
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
Re: dbmail hanging [ In reply to ]
paul at nfg
Nov 2, 2014, 5:48 AM
Post #10 of 10 (2677 views)
Permalink
On 30-10-14 14:23, Rogerio Pereira wrote:
>
>>>Do you have anything that checks if your services are alive? If so:
>>> is it just a "connection open" test or a "proper" imap test ?
>
> Yes, I have. HostMonitor is installed in my Android Phone and queries
> the imap, pop ports every 2 minutes to see if it’s alive. It’s a not
> “proper” imap test. It’s just a “connection open” test as you mentioned.

Every 2 minutes should not be a problem, if you have a login_timeout=60
or lower.

> I am monitoring the production server because of the DBmail stability
> issues. Service uptime is a huge deal for me right now.

Normally, if you kill an existing connection (RST) this is detected by
the network layer and the connection is cleaned up. Sometimes however,
if dbmail doesn't receive the RST packet, the connection will have to
timeout. If you are using a monitoring app that doesn't send a RST to
terminate a TCP session, I wonder if the app is not the problem.

I've worked quite a bit on this last year, and the only way I could test
and reproduce this behaviour was by setting up firewall rules that would
drop RST packets.

>
> I am monitoring the “opened files” by dbmail imapd process and I can see
> it grows steadily every minute (ls -l /proc/<pid>/fd | wc –l). I changed
> the “open files” limit to 4096 in the server. It may fix the imapd
> issue, but it does not fix the pop3d issues. Pop3d hangs every day with
> “general protection” errors.
>
> dbmail01 kernel: [21981989.961192] dbmail-pop3d[9821] general protection
> ip:7f92daba880a sp:7fff2cb2ae38 error:0 in
> libdbmail.so.0.0.0[7f92dab64000+6b000]
>
> Is Pop3d failure related to this monitoring as well?

If you find out, please let me know! But I don't think so.

> What changes in dbmail.conf could fix this “connection leak” created by
> monitoring?

Try setting a lower login_timeout in dbmail.conf


--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, github, linkedin
www.nfg.nl/info@nfg.nl/+31.85.877.99.97
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal