Mailing List Archive: Re: Re dbmail hanging

Hi Paul,

The problem with "too many opened files" is fixed after the change.

The server has almost 100 imapd simultaneous connections. It may have 400 as soon as I move all accounts from an old email server.

command: lsof -p `pidof dbmail-imapd`
Results: dbmail-im 18804 dbmail cwd DIR 254,0 4096 2 / dbmail-im 18804 dbmail rtd DIR 254,0 4096 2 / dbmail-im 18804 dbmail txt REG 254,0 455652 2234524 /usr/local/sbin/dbmail-imapd dbmail-im 18804 dbmail mem REG 254,0 10272 2097807 /usr/lib/x86_64-linux-gnu/gconv/CP1251.so dbmail-im 18804 dbmail mem REG 254,0 10272 2099827 /usr/lib/x86_64-linux-gnu/gconv/ISO8859-1.so dbmail-im 18804 dbmail mem REG 254,0 47616 396852 /lib/x86_64-linux-gnu/libnss_files-2.13.so dbmail-im 18804 dbmail mem REG 254,0 43560 396854 /lib/x86_64-linux-gnu/libnss_nis-2.13.so dbmail-im 18804 dbmail mem REG 254,0 31584 396850 /lib/x86_64-linux-gnu/libnss_compat-2.13.so dbmail-im 18804 dbmail mem REG 254,0 112776 2493664 /root/dbmail-3.1.17/src/modules/.libs/libauth_sql.so.0.0.0 dbmail-im 18804 dbmail mem REG 254,0 72808 2101461 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0 dbmail-im 18804 dbmail mem REG 254,0 68064 2101464 /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16 dbmail-im 18804 dbmail mem REG 254,0 516128 393368 /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0 dbmail-im 18804 dbmail mem REG 254,0 785552 2101479 /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4 dbmail-im 18804 dbmail mem REG 254,0 109360 2100708 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25 dbmail-im 18804 dbmail mem REG 254,0 59640 2100713 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3 dbmail-im 18804 dbmail mem REG 254,0 14320 393717 /lib/x86_64-linux-gnu/libkeyutils.so.1.4 dbmail-im 18804 dbmail mem REG 254,0 35400 2100688 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1 dbmail-im 18804 dbmail mem REG 254,0 162632 2100694 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1 dbmail-im 18804 dbmail mem REG 254,0 89056 393220 /lib/x86_64-linux-gnu/libgcc_s.so.1 dbmail-im 18804 dbmail mem REG 254,0 991600 2101103 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17 dbmail-im 18804 dbmail mem REG 254,0 323088 2100714 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3 dbmail-im 18804 dbmail mem REG 254,0 257288 2100706 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2 dbmail-im 18804 dbmail mem REG 254,0 14672 393283 /lib/x86_64-linux-gnu/libcom_err.so.2.1 dbmail-im 18804 dbmail mem REG 254,0 868096 2100704 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3 dbmail-im 18804 dbmail mem REG 254,0 697120 2101494 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6 dbmail-im 18804 dbmail mem REG 254,0 3424672 2110180 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0 dbmail-im 18804 dbmail mem REG 254,0 182784 2103614 /usr/lib/libpq.so.5.4 dbmail-im 18804 dbmail mem REG 254,0 174152 2110860 /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5.1.7 dbmail-im 18804 dbmail mem REG 254,0 248968 393727 /lib/x86_64-linux-gnu/libpcre.so.3.13.1 dbmail-im 18804 dbmail mem REG 254,0 52192 2110174 /usr/lib/x86_64-linux-gnu/libffi.so.5.0.10 dbmail-im 18804 dbmail mem REG 254,0 80712 396857 /lib/x86_64-linux-gnu/libresolv-2.13.so dbmail-im 18804 dbmail mem REG 254,0 126232 393262 /lib/x86_64-linux-gnu/libselinux.so.1 dbmail-im 18804 dbmail mem REG 254,0 14112 393366 /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0 dbmail-im 18804 dbmail mem REG 254,0 214456 2104090 /usr/lib/libgpgme-pthread.so.11.7.0 dbmail-im 18804 dbmail mem REG 254,0 89056 396849 /lib/x86_64-linux-gnu/libnsl-2.13.so dbmail-im 18804 dbmail mem REG 254,0 92752 393306 /lib/x86_64-linux-gnu/libz.so.1.2.7 dbmail-im 18804 dbmail mem REG 254,0 14768 396846 /lib/x86_64-linux-gnu/libdl-2.13.so dbmail-im 18804 dbmail mem REG 254,0 1603600 396843 /lib/x86_64-linux-gnu/libc-2.13.so dbmail-im 18804 dbmail mem REG 254,0 131107 396839 /lib/x86_64-linux-gnu/libpthread-2.13.so dbmail-im 18804 dbmail mem REG 254,0 1637705 2368117 /usr/local/lib/dbmail/libdbmail.so.0.0.0 dbmail-im 18804 dbmail mem REG 254,0 114168 2110838 /usr/lib/x86_64-linux-gnu/libzdb.so.9.0.2 dbmail-im 18804 dbmail mem REG 254,0 2048480 2099587 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 dbmail-im 18804 dbmail mem REG 254,0 392072 2099588 /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 dbmail-im 18804 dbmail mem REG 254,0 289224 2100674 /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.7 dbmail-im 18804 dbmail mem REG 254,0 10064 2110866 /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5.1.7 dbmail-im 18804 dbmail mem REG 254,0 176696 2110886 /usr/lib/libmhash.so.2.0.1 dbmail-im 18804 dbmail mem REG 254,0 530736 396847 /lib/x86_64-linux-gnu/libm-2.13.so dbmail-im 18804 dbmail mem REG 254,0 1012792 393745 /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4 dbmail-im 18804 dbmail mem REG 254,0 326816 2110902 /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.3200.4 dbmail-im 18804 dbmail mem REG 254,0 1381360 2110907 /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.3200.4 dbmail-im 18804 dbmail mem REG 254,0 463040 2111133 /usr/lib/libgmime-2.6.so.0.600.10 dbmail-im 18804 dbmail mem REG 254,0 31744 396858 /lib/x86_64-linux-gnu/librt-2.13.so dbmail-im 18804 dbmail mem REG 254,0 6120 2110903 /usr/lib/x86_64-linux-gnu/libgthread-2.0.so.0.3200.4 dbmail-im 18804 dbmail mem REG 254,0 14600 2110904 /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0.3200.4 dbmail-im 18804 dbmail mem REG 254,0 35104 396845 /lib/x86_64-linux-gnu/libcrypt-2.13.so dbmail-im 18804 dbmail mem REG 254,0 136936 396840 /lib/x86_64-linux-gnu/ld-2.13.so dbmail-im 18804 dbmail mem REG 254,0 26066 2099894 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache dbmail-im 18804 dbmail 0w REG 254,0 0 1323158 /var/log/dbmail.log dbmail-im 18804 dbmail 1w REG 254,0 1354134131 1323316 /var/log/dbmail.err dbmail-im 18804 dbmail 2r CHR 1,3 0t0 3075 /dev/null dbmail-im 18804 dbmail 3u IPv4 300169763 0t0 TCP 172.16.0.132:57511->172.16.0.129:postgresql (ESTABLISHED) dbmail-im 18804 dbmail 4u IPv4 300169765 0t0 TCP 172.16.0.132:57512->172.16.0.129:postgresql (ESTABLISHED) dbmail-im 18804 dbmail 5u IPv4 300230960 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->177.65.62.54:59451 (ESTABLISHED) dbmail-im 18804 dbmail 6u IPv4 300230989 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->bd6501c6.virtua.com.br:43237 (ESTABLISHED) dbmail-im 18804 dbmail 7u IPv4 300169771 0t0 TCP 172.16.0.132:57515->172.16.0.129:postgresql (ESTABLISHED) dbmail-im 18804 dbmail 8u unix 0xffff8801f3c21540 0t0 300169204 socket dbmail-im 18804 dbmail 9u unix 0xffff8801f3c20500 0t0 300169205 socket dbmail-im 18804 dbmail 10u 0000 0,9 0 4274 anon_inode dbmail-im 18804 dbmail 11u IPv4 300169207 0t0 TCP *:imap2 (LISTEN) dbmail-im 18804 dbmail 12w REG 0,14 6 300169212 /run/dbmail-imapd.pid dbmail-im 18804 dbmail 13r FIFO 0,8 0t0 300169213 pipe dbmail-im 18804 dbmail 14w FIFO 0,8 0t0 300169213 pipe dbmail-im 18804 dbmail 15u IPv4 300229772 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->187-24-146-39.3g.claro.net.br:49775 (ESTABLISHED) dbmail-im 18804 dbmail 16u IPv4 300235058 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->b12063f8.virtua.com.br:45687 (ESTABLISHED) dbmail-im 18804 dbmail 17u IPv4 300231105 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->bd21df1c.virtua.com.br:51545 (ESTABLISHED) dbmail-im 18804 dbmail 18u IPv4 300235893 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->191.173.175.60:54331 (ESTABLISHED) dbmail-im 18804 dbmail 19u IPv4 300235093 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->177-114-183-121.user.vivozap.com.br:50788 (ESTABLISHED) dbmail-im 18804 dbmail 20u IPv4 300229717 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->187-24-146-39.3g.claro.net.br:49774 (ESTABLISHED) dbmail-im 18804 dbmail 21u IPv4 300224279 0t0 TCP 172.16.0.132:60278->172.16.0.129:postgresql (ESTABLISHED) dbmail-im 18804 dbmail 22u IPv4 300235894 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->191.173.175.60:54332 (ESTABLISHED) dbmail-im 18804 dbmail 24u IPv4 300174008 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->bd6501c6.virtua.com.br:48690 (ESTABLISHED) dbmail-im 18804 dbmail 26u IPv4 300173336 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->instant6.my.com:43600 (ESTABLISHED) dbmail-im 18804 dbmail 30u IPv4 300214842 0t0 TCP 172.16.0.132:59758->172.16.0.129:postgresql (ESTABLISHED) dbmail-im 18804 dbmail 33u IPv4 300235037 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->177-114-183-121.user.vivozap.com.br:50786 (ESTABLISHED) dbmail-im 18804 dbmail 34u sock 0,7 0t0 300195365 can't identify protocol dbmail-im 18804 dbmail 35u IPv4 300234880 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->177-114-183-121.user.vivozap.com.br:50770 (ESTABLISHED) dbmail-im 18804 dbmail 36u IPv4 300234881 0t0 TCP dbmail01.dbmail.porta80.com.br:imap2->177-114-183-121.user.vivozap.com.br:50771 (ESTABLISHED)

dbmail-im 18804 dbmail 37u sock 0,7 0t0 300195371 can't identify protocol Regards.

Rogerio

On 30-10-14 14:23, Rogerio Pereira wrote:

Every 2 minutes should not be a problem, if you have a login_timeout=60
or lower.

Just changed login_timeout=30

Normally, if you kill an existing connection (RST) this is detected by
the network layer and the connection is cleaned up. Sometimes however,
if dbmail doesn't receive the RST packet, the connection will have to
timeout. If you are using a monitoring app that doesn't send a RST to
terminate a TCP session, I wonder if the app is not the problem.

I've worked quite a bit on this last year, and the only way I could test
and reproduce this behaviour was by setting up firewall rules that would
drop RST packets.

I am running APF but I am not sure it will drop RST packets. Do you think Fail2ban is a better solution?

If you find out, please let me know! But I don't think so.

Today I noticed a new "pool segfault" with the following message. I have seen this problem affecting pop3d and imapd as well in different moments.

Nov 2 16:22:46 dbmail01 kernel: [22261910.564522] pool[4764]: segfault at 49 ip 00007f655b63fa15 sp 00007f654d8b6d48 error 4 in libglib-2.0.so.0.3200.4[7f655b5cf000+f5000]

Dmail.err displays the following errors when the kernel pool crashes:

Nov 02 16:20:26 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[clientbase] client_error_cb(+127): [0x7f653c5528e0] fd [6] Connection timed out[110], 0x7f653c5528e0 Nov 02 16:20:27 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[clientbase] client_error_cb(+127): [0x7f6544213000] fd [29] Connection reset by peer[104], 0x7f6544213000 Nov 02 16:20:30 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[clientbase] client_error_cb(+127): [0xec43b0] fd [50] Broken pipe[32], 0xec43b0 Nov 02 16:20:30 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[imap] imap_session_printf(+316): ci_write failed [Unknown error 18446744073709551615] Nov 02 16:20:33 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[clientbase] client_error_cb(+127): [0x14f0070] fd [51] Broken pipe[32], 0x14f0070 Nov 02 16:20:33 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[imap] imap_session_printf(+316): ci_write failed [Unknown error 18446744073709551615] Nov 02 16:20:52 dbmail01 dbmail-imapd[4741]: [0x84d590] Error:[MailboxState] MailboxState_info(+707): SQLException: unknown error Nov 02 16:20:52 dbmail01 dbmail-imapd[4741]: [0x84d590] Alert:[libzdb] TabortHandler(+43): SQLException: unknown error Nov 02 16:20:52 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[clientbase] client_error_cb(+127): [0x105cfc0] fd [45] Broken pipe[32], 0x105cfc0 Nov 02 16:20:55 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[clientbase] client_error_cb(+127): [0x1530d80] fd [51] Broken pipe[32], 0x1530d80 Nov 02 16:20:55 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[imap] imap_session_printf(+316): ci_write failed [Unknown error 18446744073709551615] Nov 02 16:20:56 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[clientbase] client_error_cb(+127): [0x7f654439a4b0] fd [40] Connection reset by peer[104], 0x7f654439a4b0Nov 02 16:20:56 dbmail01 dbmail-imapd[4741]: [0x84d630] Debug:[server] dm_thread_dispatch(+217): data[0xa86ec0], user_data[(nil)] Nov 02 16:20:57 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[clientbase] client_error_cb(+127): [0xe836a0] fd [48] Broken pipe[32], 0xe836a0 Nov 02 16:20:57 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[imap] imap_session_printf(+316): ci_write failed [Unknown error 18446744073709551615] Nov 02 16:21:03 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[clientbase] client_error_cb(+127): [0x1530d80] fd [39] Broken pipe[32], 0x1530d80 Nov 02 16:21:03 dbmail01 dbmail-imapd[4741]: [0x84d590] Database:[db] db_con_close(+291): [0xada110] connection to poolNov 02 16:21:03 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[imap] imap_session_printf(+316): ci_write failed [Unknown error 18446744073709551615] Nov 02 16:21:13 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[clientbase] client_error_cb(+127): [0x1530d80] fd [39] Broken pipe[32], 0x1530d80 Nov 02 16:21:13 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[imap] imap_session_printf(+316): ci_write failed [Unknown error 18446744073709551615] Nov 02 16:21:17 dbmail01 dbmail-imapd[4741]: [0x857a00] Debug:[imap] _ic_cb_leave(+686): handling imap session [0x14ad000] client_state [ error] Nov 02 16:22:46 dbmail01 dbmail-imapd[4741]: [0x84d540] Error:[MailboxState] MailboxState_new(+212): SQLException: unknown error Nov 02 16:22:46 dbmail01 dbmail-imapd[4741]: [0x84d540] Alert:[libzdb] TabortHandler(+43): SQLException: unknown error And PostgreSQL displays the following: < 2014-11-02 16:19:40.003 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:19:40.004 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:19:40.004 BRST >LOG: unexpected EOF on client connection with an open transaction < 2014-11-02 16:19:40.004 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:19:40.005 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:19:40.005 BRST >LOG: unexpected EOF on client connection with an open transaction < 2014-11-02 16:19:40.006 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:19:40.018 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:19:40.034 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:19:40.034 BRST >LOG: unexpected EOF on client connection with an open transaction < 2014-11-02 16:19:44.994 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:20:00.154 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:20:00.478 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:21:37.662 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:21:37.680 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:21:37.680 BRST >LOG: unexpected EOF on client connection with an open transaction < 2014-11-02 16:21:38.554 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:21:54.054 BRST >LOG: could not receive data from client: Connection reset by peer < 2014-11-02 16:22:10.149 BRST >LOG: could not receive data from client: Connection reset by peer
> What changes in dbmail.conf could fix this connection leak created by
> monitoring?

Try setting a lower login_timeout in dbmail.conf

Just did it to login_timeout = 30
--