A NOTE has been added to this issue.
======================================================================
http://www.dbmail.org/mantis/view.php?id=1024 ======================================================================
Reported By: rmoesbergen
Assigned To:
======================================================================
Project: DBMail
Issue ID: 1024
Category: IMAP daemon
Reproducibility: always
Severity: major
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 09-Sep-13 10:15 CEST
Last Modified: 27-Sep-13 09:04 CEST
======================================================================
Summary: Imap daemon leaks extensive amounts of memory
Description:
During load tests we discovered that the dbmail-imap daemon leaks memory
eventually causing the machine(s) that it runs on to OOM and start killing
processes. Memory usage seems dependant on the amount of messages in the
database, but only ever increases when more messages are sent and received,
even though the messages are deleted after retrieval.
This bug is probably masked in a default installation due to the fact that
the default logrotate script restarts all dbmail daemons every day,
resetting memory usage. We've modified this script because this restart
would cause unwanted downtime (we have very strict SLA's with our
customers)
======================================================================
----------------------------------------------------------------------
(0003576) paul (administrator) - 09-Sep-13 17:07
http://www.dbmail.org/mantis/view.php?id=1024#c3576 ----------------------------------------------------------------------
I've passed along the valgrind results to the libzdb author, and he urges
you to upgrade libzdb:
"""
Please use the last version of libzdb (2.12). I see you use
libzdb.so.8.0.3 in the log you sent and there has indeed been memory
problems related to MySQL, but those should have been fixed back in 2.11.2
"""
----------------------------------------------------------------------
(0003577) rmoesbergen (reporter) - 10-Sep-13 13:56
http://www.dbmail.org/mantis/view.php?id=1024#c3577 ----------------------------------------------------------------------
I've built a new libzdb package (2.12), rebuilt dbmail against it and
re-tested with valgrind. The zdb leaks are indeed gone, see attached
vg-2.log. However, the unlimited growth of the imapd process is still
there, unfortunately.
----------------------------------------------------------------------
(0003578) rmoesbergen (reporter) - 20-Sep-13 15:26
http://www.dbmail.org/mantis/view.php?id=1024#c3578 ----------------------------------------------------------------------
Should I try the 'master' branch to see if this is fixed there? I see some
memory related commits that aren't in 3_1..?
----------------------------------------------------------------------
(0003579) paul (administrator) - 21-Sep-13 15:47
http://www.dbmail.org/mantis/view.php?id=1024#c3579 ----------------------------------------------------------------------
No point in testing master.
My assumption is that there is no real leakage, but some severe memory
fragmentation. Real leakage would have been exposed by valgrind.
You might try:
1)
DM_POOL=yes dbmail-imapd
to see of using the memory pools helps. However, I expect it will decrease
fragmentation, but will actually increase the memory footprint if you serve
many concurrent clients since each client will get it's own memory arena.
2)
build with jemalloc support. Maybe that will reduce fragmentation.
3)
try a more recent kernel and glibc version if possible since I've seen
drastic improvements on linux when I moved from a 2.6 kernel to a 3.2
kernel.
4)
setup a load-balancing proxy in front of two or more imapd instances.
This
will allow you to recycle imapd when it becomes too big without suffering
downtime. Nginx and ha-proxy are two possible solutions here.
----------------------------------------------------------------------
(0003588) rmoesbergen (reporter) - 23-Sep-13 17:26
http://www.dbmail.org/mantis/view.php?id=1024#c3588 ----------------------------------------------------------------------
1) Tested. With this setting on, memory increases even faster, but only
increases, mem usage still never goes down.
2) Tested, no difference with 'standard' dbmail.
3) I'm already on 3.2.0 (ubuntu 12.04)
Are you sure it's fragmentation? If find that kind of hard to believe...
We're talking hundreds om megabytes of memory in a few minutes here...
Wouldn't there be much more software with this problem if that was the
issue? I suspect something is being cached/referenced and kept in memory
when it shouldn't. I guess the only option I have left now is option 4,
restart the processes (we already do load balancing over multiple servers).
----------------------------------------------------------------------
(0003590) paul (administrator) - 23-Sep-13 17:39
http://www.dbmail.org/mantis/view.php?id=1024#c3590 ----------------------------------------------------------------------
I'm not sure of anything without a proper valgrind proof of leakage.
Hundreds of mbs of leakage is hard to believe, though I will take your
word for it. No-one else has reported anything like it though, despite
extensive load testing.
Perhaps you can generate the exact IMAP commands used, and give me an idea
of the size and complexity of the typical test message used.
Are the imapd connections using IDLE? Small leaks can quickly explode if
they occur often enough, so maybe if I'll be able to replay your imap
commands I can find something.... anything.
----------------------------------------------------------------------
(0003597) rmoesbergen (reporter) - 24-Sep-13 10:14
http://www.dbmail.org/mantis/view.php?id=1024#c3597 ----------------------------------------------------------------------
Ok, here's a transcript of a typical session. The messages used are small,
around 400 - 500 bytes, but there's a lot of messages... (6250 per run).
The transcript below is for 1 message.
Possible "issue" is that the software uses long-running IMAP connections
that repeat NOOP and SEARCH commands in a loop. However: if there are no
messages sent/received, no memory increase occurs, only during actual
message retrieval. Disconnecting all IMAP sessions after doing a testrun
does not lead to imapd freeing memory.
A001 SELECT "INBOX"
A001 OK [READ-WRITE] SELECT completed[CR][LF]
A002 NOOP
A002 OK NOOP completed[CR][LF]
.... every 5 seconds
.... After 25 NOOP's with no mailbox status info:
A105 STATUS INBOX (MESSAGES)
* STATUS "INBOX" (MESSAGES 0)[CR][LF]
A105 OK STATUS completed[CR][LF]
.... A message arrives
A152 NOOP
* 2 EXISTS[CR][LF]
* 2 RECENT[CR][LF]
A152 OK NOOP completed[CR][LF]
A153 SEARCH NOT Deleted
* SEARCH 1 2[CR][LF]
A153 OK SEARCH completed[CR][LF]
A154 FETCH 1 (FLAGS BODY[HEADER])
* 1 FETCH (FLAGS (\Recent) BODY[HEADER] {483}[CR][LF]
Return-Path: <mr01@intercommit.nl>[CR][LF]
Delivery-date: Tue, 24 Sep 2013 08:18:33 +0100[CR][LF]
Received: id 1VOMtF-0005fa-Az for <>; Tue, 24 Sep 2013 08:18:33
+0100[CR][LF]
From: <mr01@intercommit.nl>[CR][LF]>
To: <mr01@intercommit.nl>[CR][LF]>
Subject: Bericht 4038122[CR][LF]>
Message-ID: 138000737100001@edicon.go-secure.nl[CR][LF]>
X-IcitMsgID: ThuZeqwTQj2RFo4YfEZjTg[CR][LF]>
X-mtype: [CR][LF]>
X-Mailer: InterCommIT Edicon+ - 1.8.30 GNU/Linux-AMD64 DX[CR][LF]>
MIME-Version: 1.0[CR][LF]>
Content-Type: multipart/mixed;[CR][LF]>
boundary="=_7yIZwROu3P2Gg1dbRjkU"[CR][LF]>
[CR][LF]
* 1 FETCH (FLAGS (\Seen \Recent))[CR][LF]
A154 OK FETCH completed[CR][LF]
A155 FETCH 1 BODY[1.MIME]
* 1 FETCH (BODY[1.MIME] {157}[CR][LF]
Content-Type: application/octet-stream[CR][LF]
Content-Transfer-Encoding: base64;conversions=gzip[CR][LF]
Content-Disposition: attachment;[CR][LF]
filename="msg4038122.dat"[CR][LF]
[CR][LF]
A155 OK FETCH completed[CR][LF]
A156 FETCH 1 (BODY[1])
* 1 FETCH (BODY[1] {428}[CR][LF]
.... Message body is received (428 bytes, base64 encoded)
A156 OK FETCH completed[CR][LF]
A157 FETCH 1 BODY[2.MIME]
* 1 FETCH (BODY[2.MIME] NIL)[CR][LF]
A157 OK FETCH completed[CR][LF]
.... No more attachments ...
A158 NOOP
* 2 RECENT[CR][LF]
A158 OK NOOP completed[CR][LF]
A159 STORE 1 +FLAGS.SILENT (\Deleted)
A159 OK STORE completed[CR][LF]
.... Same sequence for message nr 2 ...
.... No more messages found, so expunge:
A164 EXPUNGE
* 2 EXPUNGE[CR][LF]
* 1 EXPUNGE[CR][LF]
A164 OK EXPUNGE completed[CR][LF]
.... Then it returns to polling every 5 seconds with the following loop:
A702 NOOP
A702 OK NOOP completed[CR][LF]
A703 SEARCH NOT Deleted
* SEARCH[CR][LF]
A703 OK SEARCH completed[CR][LF]
A704 EXPUNGE
A704 OK EXPUNGE completed[CR][LF]
Done.
----------------------------------------------------------------------
(0003598) paul (administrator) - 25-Sep-13 14:22
http://www.dbmail.org/mantis/view.php?id=1024#c3598 ----------------------------------------------------------------------
I've added a test-script that emulates your behaviour. It did uncover a
leak that had been fixed in the master branch already.
Please test bfd0ac138a4780da8f5683c2aa7194b001510e0f
----------------------------------------------------------------------
(0003599) rmoesbergen (reporter) - 27-Sep-13 09:04
http://www.dbmail.org/mantis/view.php?id=1024#c3599 ----------------------------------------------------------------------
I've tested with e7b0b93780edca0280c6df7937f0ac5c91d21c4f, so that's with
the second 'fix leak' included. First tested with 3.1.6, then restarted and
re-tested with 3.1.7+fixes. I've attached a zabbix graph with the results.
As you can see, no real difference in memory usage :(. The scale is a bit
off, max memory is 31 Mb, after 3 x 6250 messages. I left 3.1.7 running
without any message traffic and memory usage stayed constant.
Issue History
Date Modified Username Field Change
======================================================================
09-Sep-13 10:15 rmoesbergen New Issue
09-Sep-13 10:15 rmoesbergen File Added: vg.log
09-Sep-13 17:07 paul Note Added: 0003576
10-Sep-13 13:54 rmoesbergen File Added: vg-2.log
10-Sep-13 13:56 rmoesbergen Note Added: 0003577
20-Sep-13 15:26 rmoesbergen Note Added: 0003578
21-Sep-13 15:47 paul Note Added: 0003579
23-Sep-13 17:26 rmoesbergen Note Added: 0003588
23-Sep-13 17:39 paul Note Added: 0003590
24-Sep-13 10:14 rmoesbergen Note Added: 0003597
25-Sep-13 14:22 paul Note Added: 0003598
27-Sep-13 08:57 rmoesbergen File Added: zabbix.png
27-Sep-13 09:04 rmoesbergen Note Added: 0003599
======================================================================
_______________________________________________
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev