Mailing List Archive

DBMail Evaluation Questions
Hello,

I am seriously considering using DBMail in our company for the
scalability and redundancy potential. We are currently using UW-IMAP
with mbx folders and our custom WebMail application.

I have the following questions and concerns that do not seem to be
covered by the documentation or a quick perusal of the past several
months of the list archives. I would appreciate any feedback that you
could give:

1. DBMail doesn't support "shared" mailboxes well -- where multiple
users can access the same mailbox. However I would assume it allows
multiple IMAP clients owned by the same user to access the same folder
at the same time. This isn't explicitly stated, and I just want to make
sure.

2. Like SquirrelMail, our WebMail uses the "SORT" IMAP feature to speed
mailbox access. This is VERY important for large mailboxes as it saves
reading ALL headers on every page view. I don't think most people would
want to throw out their existing IMAP server and replace it with
"dbmail" until you get SORT support. My users would be yelling about
the slowdown.

3. Once person on Google said that DBMail is not fully RFC compliant.
He gave the example of problems with message numbers as one example, see:

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=40272cbb%241%40news.inveigle.net&rnum=40&prev=/groups%3Fq%3Ddbmail%26start%3D30%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26selm%3D40272cbb%25241%2540news.inveigle.net%26rnum%3D40

Have these problems been fixed, or are they non-issues?

4. With MySQL/InnoDB, I assume that the database size is really only
limited by the file system size. If we were to setup a SCSI RAID 5
array for the MySQL server, are there any considerations that would
seriously affect performance once we start getting 40Gb to 80Gb+ of
email data in one server?


5. It doesn't look like the programs such as "dbmail-smtp" have a
command line parameter that allow you to specify which "dbmail.conf"
file to use. This would be VERY helpful, for example, on an inbound
email server that may be filtering email for delivery on more than one
dbmail database.


6. How stable is the 2.0 branch and is there any significant reason to
use it over the 1.X branch? Is there a change list somewhere? Will
the upgrade from 1.X to 2.0 be seamless? When is 2.X expected to be
released?


7. With all the separate IMAP and POP daemons that run on the dbmail
server, each one presumably has its own database handle. Once you start
getting a lot of users using a cluster of IMAP/POP servers, you get a
high number of MySQL collections. What are the practical limits that
have been seen in terms of the maximum number of connections? Any
case-histories that can be used for comparison and specification design?


8. Has the "dbmail-smtp" branch been code reviewed by 3rd parties? I am
concerned about the possibility of SQL Injection.



Thank you.

-Erik Kangas


PS: Another interesting thing that could also be done in the future is:

* A separate database for user authentication. This way, you could have
multiple separate databases for email storage and the user
authentication could tell you which database to use for which user.
Moving users from one machine to another would then be seamless as the
front-end machine would be the same, only the database entries would
change. This becomes a concern once your big database server runs out
of room.


--

Erik Kangas, Ph.D. --- President of Lux Scientiae, Incorporated

Lux Scientiae: 1-800-441-6612 46 Central Street
FAX: 1-413-332-0598 Somerville, Massachusetts
Cell: 1-617-596-9558 02143, United States of America
AIM/ICQ/Yahoo Chat: Screen Name "luxsci"

kangas@luxsci.com --- http://luxsci.com
RE: DBMail Evaluation Questions [ In reply to ]
> 1. DBMail doesn't support "shared" mailboxes well -- where multiple
> users can access the same mailbox. However I would assume it allows
> multiple IMAP clients owned by the same user to access the
> same folder
> at the same time. This isn't explicitly stated, and I just
> want to make
> sure.

1.0/2.0 - multiple access to same imap folder work
2.0 - (currently in final beta) supports shared mailboxes, and a lot more
features.

> 4. With MySQL/InnoDB, I assume that the database size is really only
> limited by the file system size. If we were to setup a SCSI RAID 5
> array for the MySQL server, are there any considerations that would
> seriously affect performance once we start getting 40Gb to 80Gb+ of
> email data in one server?

We're currently running at around 130Gb of mail data, on a dual Xeon 2.4G
dedicated database server with load running between 30 and 80% average
(bursts above but only for brief periods). Scales very well. I would
recommend putting your smtp/pop servers on different machines if you're
looking at a large install. We're running 6 smtp servers (also doing
spam/virus scanning) and 3 pop/imap servers. The pop/imap servers don't have
high load at all - mainly there for redundancy and scalability.

> 7. With all the separate IMAP and POP daemons that run on the dbmail
> server, each one presumably has its own database handle.
> Once you start
> getting a lot of users using a cluster of IMAP/POP servers, you get a
> high number of MySQL collections. What are the practical limits that
> have been seen in terms of the maximum number of connections? Any
> case-histories that can be used for comparison and
> specification design?

I'm starting to have a few issues with that - but the daemons do persistent
connections so not to intense. The dbmail-smtp forks so connects and
disconnects, which can be a bit hard on the database and harder to predict.

Every now and then we get thread memory problems reported and a failed
connect on the database. Trying to work through this now -- I'm sure it's a
config problem at our end. If any one can help out here (happy to pay
reasonable rates) [nudge nudge Eelco or Roel]. We're currently running at
about 200 connections from daemons/smtp processes.

The lmtp daemon in 2.0 should mean persistent connects and more predictable
perfomance. Haven't tried this yet.

> 8. Has the "dbmail-smtp" branch been code reviewed by 3rd
> parties? I am concerned about the possibility of SQL Injection.

Peer review. There was a bug in early 1.x train, but was fixed I believe.

> * A separate database for user authentication. This way, you
> could have
> multiple separate databases for email storage and the user
> authentication could tell you which database to use for which user.
> Moving users from one machine to another would then be
> seamless as the
> front-end machine would be the same, only the database entries would
> change. This becomes a concern once your big database server
> runs out
> of room.

They had this in an early 1.x release -- but unfortunately removed it. There
is discussion for later in 2.x train to provide more configurable
authentication support (e.g. ldap there now, other plugins can be written).
I looked at the code and it wouldn't take too much work to write this again.
The focus I think is on core-needed features rather than the big-end
scalability needs at the moment. I'm not certain but I get the impression
there aren't many people who are running servers with over 20G of data. But
for my part - I love dbmail.

/Mark
RE: DBMail Evaluation Questions [ In reply to ]
Hello,

> > 8. Has the "dbmail-smtp" branch been code reviewed by 3rd
> > parties? I am concerned about the possibility of SQL Injection.
>
> Peer review. There was a bug in early 1.x train, but was fixed I
believe.


FWIW, there is an sql-injection bug in dbmail-smtp, though I
don't remember the specifics of how you trigger it. From memory
it seems like you had to call dbmail-smtp with a username that
had imbedded sql, which under most setups would be hard to get
through the mta. There's a fix for it (in function auth_user_exists)
in a patch I made last November-ish. That patch was too big
(mainly because of I ran automake myself instead of having Ilja
do it), and never got applied; I've been wanting to go break it
up into individual patches (eg. one for security stuff, one for
misc. fixes, and one for usermap support), but haven't had the
time. Look for something like "dbmail-20031108-usermap.patch"
and pull the pertinent parts out, if you want it. :)

Of much greater concern is a similar sql-injection bug for the
APOP command, fixed in that same patch. Iirc, that one took
nothing special, just telnet to the box and submit an APOP command
with sql embedded in the username part. I don't have any un-
patched servers around to test on, but it should be something
like "APOP user'; select something from somewhere where 1=1 and'
<32char md5 digest>".
I don't know exactly what the query looks like you're exploiting,
so that sql probably doesn't work, but that's the general idea.
Search that patch for "escuser" to find the fix (which was almost
identical to the fix for the USER command a while back).

Jn

--
Jesse Norell

administrator@kci.net is not my email address;
change "administrator" to my first name.
--
Re: DBMail Evaluation Questions [ In reply to ]
I'm just picking out two of your several questions. See inline...

Erik Kangas <kangas@luxsci.com> said:

> 2. Like SquirrelMail, our WebMail uses the "SORT" IMAP feature to speed
> mailbox access. This is VERY important for large mailboxes as it saves
> reading ALL headers on every page view. I don't think most people would
> want to throw out their existing IMAP server and replace it with
> "dbmail" until you get SORT support. My users would be yelling about
> the slowdown.

I would also like to see the SORT extension supported. It's in the 2.x
Development list under the "needs more discussion" section. IMHO, the results
of the discussions are that SORT is definitely an important feature to support!

> 6. How stable is the 2.0 branch and is there any significant reason to
> use it over the 1.X branch? Is there a change list somewhere? Will
> the upgrade from 1.X to 2.0 be seamless? When is 2.X expected to be
> released?

The 2.0 branch has a number of new features including the ACL and NAMESPACE
extensions and an LMTP server to reduce the thrashing of starting a new
dbmail-smtp process for each message received.

The upgrade isn't seamless at all, although there are now migration scripts in
CVS, so the headaches should be minimal. Most or all of the configuration is
the same as the 1.2 series, so it's really just a matter of getting the
database schemas up to date using the database scripts.

We're currently working towards 2.0rc3. I have a patch that I'm hoping will go
in very soon so that error conditions are properly reported to the MTA during
delivery. It's a fairly invasive patch, but I prefer the route of fixing the
problem by (re-)design rather than hacking it to work well enough for 2.0.
Yes, you are correctly reading the implication that the current 2.0rc2 and CVS
do silently lose mail in some situations... it's my fault, really, because of
my new delivery chain not having had a good way to pass errors up the call
chain. The patch fixes that.

In short, 2.0rc2 and CVS are not yet ready for production use.

Aaron

--
Re: DBMail Evaluation Questions [ In reply to ]
Hi,

(going inline..)

Aaron Stone wrote:

> I'm just picking out two of your several questions. See inline...
>
> Erik Kangas <kangas@luxsci.com> said:
>
>
>>2. Like SquirrelMail, our WebMail uses the "SORT" IMAP feature to speed
>>mailbox access. This is VERY important for large mailboxes as it saves
>>reading ALL headers on every page view. I don't think most people would
>>want to throw out their existing IMAP server and replace it with
>>"dbmail" until you get SORT support. My users would be yelling about
>>the slowdown.
>
>
> I would also like to see the SORT extension supported. It's in the 2.x
> Development list under the "needs more discussion" section. IMHO, the results
> of the discussions are that SORT is definitely an important feature to support!
After reading through the SORT and THREAD proposal (am I right that
there's no RFC yet?), I see the importance of this feature. However,
it's not a trivial thing to implement I guess. I'll add it to the
wishlist as something we will need to implement in the (near) future.

Ilja