Mailing List Archive

Great!

I'm happy for you guys!!

will now svn 2.3x will be open?


Jorge


----- Original Message -----
From: "Paul J Stevens" <paul@nfg.nl>
To: "DBMail mailinglist" <dbmail@dbmail.org>
Cc: "DBMAIL Developers Mailinglist" <dbmail-dev@dbmail.org>
Sent: Wednesday, January 31, 2007 10:31 AM
Subject: [Dbmail] ANNOUNCE: DBMail 2.2.2 released


> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> DBMail 2.2.2 released.
>
> We're happy to announce the availability of DBMail 2.2.2.
>
> This is a bugfix maintenance release in the current stable
> production series.
>
> For more information about the changes included in this version please
> check the news item at
>
> http://www.dbmail.org/index.php?page=news&id=36
>
> kind regards,
>
> - --
> ________________________________________________________________
> Paul Stevens paul at nfg.nl
> NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
> The Netherlands________________________________http://www.nfg.nl
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFFwG/m8iITvBH4zTERAkG9AJ48LE3MIz2eNkqjzz3y1LgQkSXe2QCg4gNp
> ndhXsQBaRpzmyjhqsM6Lpu8=
> =Ouqw
> -----END PGP SIGNATURE-----
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> https://mailman.fastxs.nl/mailman/listinfo/dbmail
>
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Wed, 2007-01-31 at 10:46 +0000, Jorge Bastos wrote:
> Great!
>
> I'm happy for you guys!!
>
> will now svn 2.3x will be open?

Please don't go anywhere near SVN trunk; it actually reflects a dev
state from late DBMail 2.1... before we branched and put a lot of work
into code cleanup and stability on dbmail_2_2_branch.

Looking at the release history of DBMail, it's been about two years
between major stable releases. Expect that to be the 2.4 timeframe, too.

Aaron

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

"Aaron Stone" <aaron@serendipity.cx> schrieb:
> On Wed, 2007-01-31 at 10:46 +0000, Jorge Bastos wrote:
>> Great!
>>
>> I'm happy for you guys!!
>>
>> will now svn 2.3x will be open?
>
> Please don't go anywhere near SVN trunk; it actually reflects a dev
> state from late DBMail 2.1... before we branched and put a lot of work
> into code cleanup and stability on dbmail_2_2_branch.
>
> Looking at the release history of DBMail, it's been about two years
> between major stable releases. Expect that to be the 2.4 timeframe, too.

Hopefully it will not take about 2 years until the next major release.
That's too long. Please try to keep small steps and try to release more
often.

That's my point of view as maintainer of another opensource project.


--
Lars Kneschke
Metaways Infosystems GmbH
Pickhuben 4
20457
Hamburg
Germany

eGroupWare Training & Support ==>
http://www.egroupware-support.net

E-Mail:
mailto:l.kneschke@metaways.de
Web: http://www.metaways.de
Tel: +49 (0)40
317031-21
Fax: +49 (0)40 317031-81
Mobile: +49 (0)175 9304324


_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Wed, 2007-01-31 at 18:42 +0100, Lars Kneschke wrote:
> "Aaron Stone" <aaron@serendipity.cx> schrieb:
>
> > Looking at the release history of DBMail, it's been about two years
> > between major stable releases. Expect that to be the 2.4 timeframe, too.
>
> Hopefully it will not take about 2 years until the next major release.
> That's too long. Please try to keep small steps and try to release more
> often.
>
> That's my point of view as maintainer of another opensource project.

The next big project is multithreading the daemons. Even if that's the
only change that is made, I think it will take a while to get right. Of
course I'm open to different ideas, and I'm researching threading
frameworks that we can leverage to reduce our development time. Anyways,
getting into those details we should move onto the dbmail-dev list.

My main point off this thread is that anybody who uses SVN trunk is
asking for serious trouble at this time.

Aaron

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Aaron Stone wrote:
> On Wed, 2007-01-31 at 10:46 +0000, Jorge Bastos wrote:
>> Great!
>>
>> I'm happy for you guys!!
>>
>> will now svn 2.3x will be open?
>
> Please don't go anywhere near SVN trunk; it actually reflects a dev
> state from late DBMail 2.1... before we branched and put a lot of work
> into code cleanup and stability on dbmail_2_2_branch.

Well actually, I've merged branch_2_2 into trunk a couple of times since
2.2.0.



--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Aaron Stone wrote:
> The next big project is multithreading the daemons. Even if that's the
> only change that is made, I think it will take a while to get right. Of
> course I'm open to different ideas, and I'm researching threading
> frameworks that we can leverage to reduce our development time. Anyways,
> getting into those details we should move onto the dbmail-dev list.


May I ask why this is important? Threads aren't the end-all-be-all that
lots people think they are. I believe a multi-process system can be
every bit as fast as a threaded server, and it's a lot less complicated.
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Matthew T. O'Connor wrote:
> Aaron Stone wrote:
>> The next big project is multithreading the daemons. Even if that's the
>> only change that is made, I think it will take a while to get right. Of
>> course I'm open to different ideas, and I'm researching threading
>> frameworks that we can leverage to reduce our development time. Anyways,
>> getting into those details we should move onto the dbmail-dev list.
>
>
> May I ask why this is important? Threads aren't the end-all-be-all that
> lots people think they are. I believe a multi-process system can be
> every bit as fast as a threaded server, and it's a lot less complicated.

Using threading is not a goal in itself, but a means.

One of the goals for 2.3+ is adding new imap capabilities like NOTIFY
that require a lot of IPC between running imapd processes. Using threads
will make that quite straightforward.

Another goal is scaling up the number of concurrent connected clients
without depleting the database connections. Using connection pools is a
best practice there, and even though that could be done in a
multi-process architecture, this is generally done using threads.

So yes, imo also there's a very solid case for threads in dbmail. But
like Aaron says, it will take some time to do this right. Neither I nor
Aaron (afaik) have any experience with threads in C so we'll have to
learn how to do it as we go along :-)

--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

It may be worth looking at something like APR (http://apr.apache.org/)
for dealing with things like threads and shared memory on disparate
platforms. No doubt this would bloat dbmail somewhat, but it's always
nice to not have to do it all yourself, as we've seen with gmime.

Paul J Stevens wrote:
> Matthew T. O'Connor wrote:
>> Aaron Stone wrote:
>>> The next big project is multithreading the daemons. Even if that's the
>>> only change that is made, I think it will take a while to get right. Of
>>> course I'm open to different ideas, and I'm researching threading
>>> frameworks that we can leverage to reduce our development time. Anyways,
>>> getting into those details we should move onto the dbmail-dev list.
>>
>> May I ask why this is important? Threads aren't the end-all-be-all that
>> lots people think they are. I believe a multi-process system can be
>> every bit as fast as a threaded server, and it's a lot less complicated.
>
> Using threading is not a goal in itself, but a means.
>
> One of the goals for 2.3+ is adding new imap capabilities like NOTIFY
> that require a lot of IPC between running imapd processes. Using threads
> will make that quite straightforward.
>
> Another goal is scaling up the number of concurrent connected clients
> without depleting the database connections. Using connection pools is a
> best practice there, and even though that could be done in a
> multi-process architecture, this is generally done using threads.
>
> So yes, imo also there's a very solid case for threads in dbmail. But
> like Aaron says, it will take some time to do this right. Neither I nor
> Aaron (afaik) have any experience with threads in C so we'll have to
> learn how to do it as we go along :-)
>
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Besides dbmail specific goals,

threading is better suited for memory management and limitation.

threading is more controllable during a DoS attack.

threading if done right, will provide better scaling abilities (see
roxen versus apache).

threading will provide us eventually with a more sleak and clean design,
because it does not allow for the apache programming model in the sense of:
"We have this child, it leaks memory so lets kill it after x connects en
start a new one."

Kind regards,

Marc
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Marc Dirix wrote:
> threading is better suited for memory management and limitation.

How's that exactly? Using unix fork() which is copy on write, forking
many processes tends to be very memory efficient.

> threading is more controllable during a DoS attack.

I've never heard this before, please expound.

> threading if done right, will provide better scaling abilities (see
> roxen versus apache).

I don't know much about roxen, but I think lots of things will scale
better than Apache since Apache by design is a kitchen sink type program.

> threading will provide us eventually with a more sleak and clean design,
> because it does not allow for the apache programming model in the sense of:
> "We have this child, it leaks memory so lets kill it after x connects en
> start a new one."

This sounds like a threads are better because they are argument. The
benifit of being able to kill off a process with memory leaks is that
the server still runs, if you have the same memory leak in a treaded
app, you have to restart the whole server.

I'm not against threading, I'm just concerned that we are going to add a
ton of complexity for very little gain.
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Paul J Stevens wrote:
> Using threading is not a goal in itself, but a means.
>
> One of the goals for 2.3+ is adding new imap capabilities like NOTIFY
> that require a lot of IPC between running imapd processes. Using threads
> will make that quite straightforward.

OK, though IPC is possible in a multi-process model, but I agree,
communication is probably the one place where a threaded system is more
simple than multi-process.

> Another goal is scaling up the number of concurrent connected clients
> without depleting the database connections. Using connection pools is a
> best practice there, and even though that could be done in a
> multi-process architecture, this is generally done using threads.

I don't see how this makes things better, what it sounds like you are
suggesting is that DBMail will become in addition to an IMAP server, a
database connection pooling engine. There are tools out there that do
this already.

> So yes, imo also there's a very solid case for threads in dbmail. But
> like Aaron says, it will take some time to do this right. Neither I nor
> Aaron (afaik) have any experience with threads in C so we'll have to
> learn how to do it as we go along :-)

This sounds scary to me. Obviously it's not my project and I'm not the
one coding, I'm just worried as an admin that depends on DBMail that we
are going to open up a large can of worms for dubious gain.

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Wed, 2007-01-31 at 17:13 -0500, Matthew T. O'Connor wrote:
> Paul J Stevens wrote:

> > Another goal is scaling up the number of concurrent connected clients
> > without depleting the database connections. Using connection pools is a
> > best practice there, and even though that could be done in a
> > multi-process architecture, this is generally done using threads.
>
> I don't see how this makes things better, what it sounds like you are
> suggesting is that DBMail will become in addition to an IMAP server, a
> database connection pooling engine. There are tools out there that do
> this already.

http://monkey.org/~provos/libevent/

Memcache has incredible performance based on this library, with all O(1)
operations behind it. I would like to use a similar architecture for
handling incoming commands, then farming out the command responses to
threads. Parsing the first 10 - 20 bytes of a command should allow us to
whittle down to a single function that responds to that command. The
function will be called in its own thread and the main thread will go on
to process the next 10 - 20 incoming bytes on another connection.

This is the reactor/proactor pattern (they are subtly different, and we
might want to blend them a little bit) and after lots of reading, its
what I think we should be doing next.

Also, if each thread only asks for a database handle if it needs one, we
should be able to economize on those connections. Further, if we
leverage memcache, certain operations should become extremely fast. For
example, if each time a message is delivered, dbmail-lmtpd or
dbmail-smtp updates the user's mailbox status in memcache, the next time
they poll (or, better yet, IDLE) for status updates, no database handle
will be needed and the information will come directly from memory.

Aaron

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Aaron Stone wrote:
> http://monkey.org/~provos/libevent/
>
> Memcache has incredible performance based on this library, with all O(1)
> operations behind it. I would like to use a similar architecture for
> handling incoming commands, then farming out the command responses to
> threads. Parsing the first 10 - 20 bytes of a command should allow us to
> whittle down to a single function that responds to that command. The
> function will be called in its own thread and the main thread will go on
> to process the next 10 - 20 incoming bytes on another connection.

This all sounds interesting, however I should point out that from a
quick read of the page you reference above, libevent doesn't require
threads.

As for the design, are you planning on making memcached a requirement
for DBMail? Or just an optional way to make things faster if need be?

> Also, if each thread only asks for a database handle if it needs one, we
> should be able to economize on those connections. Further, if we
> leverage memcache, certain operations should become extremely fast. For
> example, if each time a message is delivered, dbmail-lmtpd or
> dbmail-smtp updates the user's mailbox status in memcache, the next time
> they poll (or, better yet, IDLE) for status updates, no database handle
> will be needed and the information will come directly from memory.

The memcached stuff sounds very interesting, and like a very good idea
to help DBMail scale up, but again, this isn't a reason for threading,
this looks like it will work fine in a multi-process environment, and it
looks straightforward enough that we can probably gain a lot without
re-engineering most of DBMail.



_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Bottom line, if you want a thousand open connections on a single mail
server with current DBMail, you're going to need a thousand processes
and a thousand database handles. This don't scale.

I really would prefer to keep most of this discussion on the dbmail-dev
list, for the sake of the sanity of those who are on the dbmail list for
setup and configuration help and not for core architectural issues!

Aaron

On Wed, 2007-01-31 at 22:16 -0500, Matthew T. O'Connor wrote:

> This all sounds interesting, however I should point out that from a
> quick read of the page you reference above, libevent doesn't require
> threads.
[etc, snip]

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Wed, Jan 31, 2007 at 05:08:18PM -0500, Matthew T. O'Connor wrote:
> Marc Dirix wrote:
> >threading is better suited for memory management and limitation.
>
> How's that exactly? Using unix fork() which is copy on write, forking
> many processes tends to be very memory efficient.

Processes tend to do lot's of memory writes, certainly in an use environment.
I meant here that a seperate thread can be instituted to do garbage
collection of used memory.


>
> >threading is more controllable during a DoS attack.
>
> I've never heard this before, please expound.

When using a monolithic threading model, the socket is only opened to
the main thread. This main thread will create handler threads if someone
sends data to the socket. In this model all new connection are handled
sequentially, however the return values can be asynchron from each
thread.
The main thread can easily manage both traffic shaping/limiting and
drop multiple connects from the same client.

This agains a multi-process model where there are x default handlers
trying to beat eachother to the socket when a client connects. There is
no control whatsoever about the socket.

There are thread library's specially designed for being used in a
service environment.
>
> >threading if done right, will provide better scaling abilities (see
> >roxen versus apache).
>
> I don't know much about roxen, but I think lots of things will scale
> better than Apache since Apache by design is a kitchen sink type program.
>

That is actually the other way around. Apache is a webserver. If
scripting or whatever is needed one has to look for external compilers
like php or perl. However Roxen is a webapplication server, it has a
internal (RXML) parser which can do just about everything you need for
dynamic pages. (And also if needed interface to php/perl).

The problem with apache is in the fact that to be able to handle N
connects it has to create N childs which eventually will limit to a
maximum. This doesn't scale well. A model where only one socket exists
and connections are handled sequential will scale very well until system limits
that is.


> >threading will provide us eventually with a more sleak and clean design,
> >because it does not allow for the apache programming model in the sense of:
> >"We have this child, it leaks memory so lets kill it after x connects en
> >start a new one."
>
> This sounds like a threads are better because they are argument. The
> benifit of being able to kill off a process with memory leaks is that
> the server still runs, if you have the same memory leak in a treaded
> app, you have to restart the whole server.
>

That's the sloppy programmers kind of view. "We know we leak, but
instead of fixing it we will kill our children".

> I'm not against threading, I'm just concerned that we are going to add a
> ton of complexity for very little gain.

That is true, it will take a steep programming curve to do right.
And sure it will take a long time before dbmail is back to the current
level, however now is the time to make such a decission not after
putting another year of effort into 2.3. The next chance would only be
after 2.4 release.

Marc
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

good news! way to go, guys
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFwbhH4DJ/LCkjclMRAhsFAJ41GO8HLPusbIj7j/ACrP4aTB47AgCfbXG3
OuKRo/2kwLoGk4o7S2zz5Go=
=mgap
-----END PGP SIGNATURE-----
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Donnerstag, 1. Februar 2007 05:24 Aaron Stone wrote:
> I really would prefer to keep most of this discussion on the
> dbmail-dev list, for the sake of the sanity of those who are on the
> dbmail list for setup and configuration help and not for core
> architectural issues!

A bit of general discussion on this topic is OK, as it's not for very
tech details but more for design.

As for threads: they are faster, nicer, scale better, but require more
advanced programmers than for processes, as synchronisation of the
threads is more complex. But I believe only the devs can decide how
they do something, we "stupid users" are just allowed to say "hey, I
want more performance and feature x". It's nice that they do this for
us already.

I just hope that switching to threads does not bind such a lot of
manpower that other features do not enhance. I don't know if it's
possible that multiple clients connect from different PCs via IMAP to
the same account, as cyrus can do. If it wouldn't, I'd see this as a
much more urgent feature than threads - for example.

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0676/846 914 666 .network.your.ideas.
// PGP Key: "curl -s http://zmi.at/zmi4.asc | gpg --import"
// Fingerprint: EA39 8918 EDFF 0A68 ACFB 11B7 BA2D 060F 1C6F E6B0
// Keyserver: www.keyserver.net Key-ID: 1C6FE6B0

Michael Monnerie wrote:
> I just hope that switching to threads does not bind such a lot of
> manpower that other features do not enhance. I don't know if it's
> possible that multiple clients connect from different PCs via IMAP to
> the same account, as cyrus can do.

We've been doing this since dbmail 1.1.x

Just letting you know,
Alex
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Donnerstag, 1. Februar 2007 11:37 Aleksander wrote:
> We've been doing this since dbmail 1.1.x

That's what I hoped and expected, thanks :-)

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0676/846 914 666 .network.your.ideas.
// PGP Key: "curl -s http://zmi.at/zmi4.asc | gpg --import"
// Fingerprint: EA39 8918 EDFF 0A68 ACFB 11B7 BA2D 060F 1C6F E6B0
// Keyserver: www.keyserver.net Key-ID: 1C6FE6B0

Aleksander wrote:
> Michael Monnerie wrote:
>> I just hope that switching to threads does not bind such a lot of
>> manpower that other features do not enhance. I don't know if it's
>> possible that multiple clients connect from different PCs via IMAP to
>> the same account, as cyrus can do.
>
> We've been doing this since dbmail 1.1.x

You can do it (I do) but it does create problems. If you connect two
thunderbirds (A and B) to the same mailbox, and A deletes messages, this
is not communicated to B, which leads B to issue illegal message id
ranges in its fetch command.

--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Paul J Stevens wrote:
> If you connect two
> thunderbirds (A and B) to the same mailbox, and A deletes messages, this
> is not communicated to B, which leads B to issue illegal message id
> ranges in its fetch command.

Oh, thanks. That explains this very rare problem I have seen. I
sometimes check mail from home via ssh tunnel. And then only rarely
manage/delete/move mail. So I've seen this problem at work a few times
(I don't log out for the night, only lock the session). A TB restart helps.

Btw, TB in that case politely displays the IMAP servers error "invalid
range specified ...".

Does this only apply to Thunderbird and the specific folder where the
messages have been deleted?

Alex
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

dbmail 2.2.2. gentoo ebuild just added to gentoo sunrise.
http://www.gentoo-sunrise.org/sunrise/browser/sunrise/net-mail/dbmail

regards Martin

On Thu, Feb 1, 2007, Martin Hierling <martin@mh-itc.de> said:

> dbmail 2.2.2. gentoo ebuild just added to gentoo sunrise.
> http://www.gentoo-sunrise.org/sunrise/browser/sunrise/net-mail/dbmail
>
> regards Martin

Awesome. Have you had any luck getting in touch with someone in net-mail?
Half the people I've emailed haven't even bothered responding, and the
other half say that their plates are full and since they don't use DBMail
personally, they don't care to include it into Gentoo Portage.

Other Gentooers, see here: http://bugs.gentoo.org/show_bug.cgi?id=22331

Aaron
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Aron,

Awesome. Have you had any luck getting in touch with someone in net-mail?
> Half the people I've emailed haven't even bothered responding, and the


i have tried to get g developer (for another project) but without luck.
Personally i use a bunch of overlays so it doesnt bother me to get dbmail
out of sunrise. I actually have never tried to get in touch with the
net-mail guys. But i will try next days. dbmail is not that intensive in
maintaining so perhaps someone will bring it into portage if i write the
ebuilds .... will ask.

regards Martin

Paul J Stevens wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> DBMail 2.2.2 released.
>
> We're happy to announce the availability of DBMail 2.2.2.
>

Excellent :) Any word on the memory footprint of dbmail-imapd processes
increasing over time? I checked the changelog but didn't see any mention
of it. I had a script that restarted the dbmail-imapd every 2 hours, but
unfortunately thunderbird doesn't handle the imap server going down
during some operations ( ie it crashes ), and I've had a user complain
about loosing an email they were working on - they had just hit 'save
draft' when the imapd server restarted.

--
Daniel Kasak
IT Developer
NUS Consulting Group
Level 5, 77 Pacific Highway
North Sydney, NSW, Australia 2060
T: (+61) 2 9922-7676 / F: (+61) 2 9922 7989
email: dkasak@nusconsulting.com.au
website: http://www.nusconsulting.com.au
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Thu, Feb 1, 2007, Daniel Kasak <dkasak@nusconsulting.com.au> said:

> Paul J Stevens wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>>
>> DBMail 2.2.2 released.
>>
>> We're happy to announce the availability of DBMail 2.2.2.
>>
>
> Excellent :) Any word on the memory footprint of dbmail-imapd processes
> increasing over time? I checked the changelog but didn't see any mention
> of it. I had a script that restarted the dbmail-imapd every 2 hours, but
> unfortunately thunderbird doesn't handle the imap server going down
> during some operations ( ie it crashes ), and I've had a user complain
> about loosing an email they were working on - they had just hit 'save
> draft' when the imapd server restarted.

It's better, but not perfect (yet!). You should definitely be able to get
more than two hours of life out of each process; you've kinda taken the
baseball bat approach to freeing memory there :-P

Aaron
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

I maintain my own dbmail ebuild since 2002, tried to get that in the
portage since 2002
through the sparc herds but no avail :(

I certainly whish the ebuild goes in

xming

Martin Hierling wrote:
> Aron,
>
> Awesome. Have you had any luck getting in touch with someone in
> net-mail?
> Half the people I've emailed haven't even bothered responding, and the
>
>
> i have tried to get g developer (for another project) but without
> luck. Personally i use a bunch of overlays so it doesnt bother me to
> get dbmail out of sunrise. I actually have never tried to get in touch
> with the net-mail guys. But i will try next days. dbmail is not that
> intensive in maintaining so perhaps someone will bring it into portage
> if i write the ebuilds .... will ask.
>
> regards Martin
> ------------------------------------------------------------------------
>
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> https://mailman.fastxs.nl/mailman/listinfo/dbmail
>

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

xming,

is your ebuild public available? URL? If not can you send it to me? Perhaps
i can improve the one in sunrise.

regards Martin

On 1/31/2007, "Matthew T. O'Connor" <matthew@zeut.net> wrote:
>> So yes, imo also there's a very solid case for threads in dbmail. But
>> like Aaron says, it will take some time to do this right. Neither I nor
>> Aaron (afaik) have any experience with threads in C so we'll have to
>> learn how to do it as we go along :-)
>
>This sounds scary to me. Obviously it's not my project and I'm not the
>one coding, I'm just worried as an admin that depends on DBMail that we
>are going to open up a large can of worms for dubious gain.
>

I think the prudent thing to do in this case is to make sure that the
2.2.2 release is something that works very well and is quite stable.
The goal here is to give the dbmail community a IMAP/POP server that
will be suitable for their needs for some time to come in the future.

This will give the developers the time necessary to do everything that
they need to do in order to get threads working right. And without an
immediate crunch to fix some bugs or get a release out.

And then those of us who are more willing to take the newer releases and
test them out can pick up the new 2.3x version as willing.

The damaging thing would be to start working on a threaded version with a
large list of known defects in the existing release.
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On 2/1/2007, "Aaron Stone" <aaron@serendipity.cx> wrote:

>Bottom line, if you want a thousand open connections on a single mail
>server with current DBMail, you're going to need a thousand processes
>and a thousand database handles. This don't scale.
>
>I really would prefer to keep most of this discussion on the dbmail-dev
>list, for the sake of the sanity of those who are on the dbmail list for
>setup and configuration help and not for core architectural issues!
>

Yes, but I personally am finding this discussion educational even if you
have decided for other reasons to pursue a given direction.
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Fri, 2007-02-02 at 12:24 -0500, tom@tacocat.net wrote:
> On 1/31/2007, "Matthew T. O'Connor" <matthew@zeut.net> wrote:
> >> So yes, imo also there's a very solid case for threads in dbmail. But
> >> like Aaron says, it will take some time to do this right. Neither I nor
> >> Aaron (afaik) have any experience with threads in C so we'll have to
> >> learn how to do it as we go along :-)
> >
> >This sounds scary to me. Obviously it's not my project and I'm not the
> >one coding, I'm just worried as an admin that depends on DBMail that we
> >are going to open up a large can of worms for dubious gain.
> >
>
> I think the prudent thing to do in this case is to make sure that the
> 2.2.2 release is something that works very well and is quite stable.
> The goal here is to give the dbmail community a IMAP/POP server that
> will be suitable for their needs for some time to come in the future.
>
> This will give the developers the time necessary to do everything that
> they need to do in order to get threads working right. And without an
> immediate crunch to fix some bugs or get a release out.
>
> And then those of us who are more willing to take the newer releases and
> test them out can pick up the new 2.3x version as willing.
>
> The damaging thing would be to start working on a threaded version with a
> large list of known defects in the existing release.

I'm going to save this one in my mental file under "smart project
management" -- you'd think it's common sense, but I think we've all been
under that time crunch situation before that makes us think stupid, not
sleep, and try to finish new and broken code instead of fixing old,
broken, but well understood code.

It is also damaging to have the next generation code remain broken for
too long before it really starts to work and show its value -- if there
even is value, as some NG projects tank miserably -- so it's definitely
a balancing act as to how many new things you can do at once.

Aaron

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On 2-feb-2007, at 18:48, Aaron Stone wrote:

> On Fri, 2007-02-02 at 12:24 -0500, tom@tacocat.net wrote:
>> On 1/31/2007, "Matthew T. O'Connor" <matthew@zeut.net> wrote:
>>>> So yes, imo also there's a very solid case for threads in
>>>> dbmail. But
>>>> like Aaron says, it will take some time to do this right.
>>>> Neither I nor
>>>> Aaron (afaik) have any experience with threads in C so we'll
>>>> have to
>>>> learn how to do it as we go along :-)
>>>
>>> This sounds scary to me. Obviously it's not my project and I'm
>>> not the
>>> one coding, I'm just worried as an admin that depends on DBMail
>>> that we
>>> are going to open up a large can of worms for dubious gain.
>>>
>>
>> I think the prudent thing to do in this case is to make sure that the
>> 2.2.2 release is something that works very well and is quite stable.
>> The goal here is to give the dbmail community a IMAP/POP server that
>> will be suitable for their needs for some time to come in the future.
>>
>> This will give the developers the time necessary to do everything
>> that
>> they need to do in order to get threads working right. And
>> without an
>> immediate crunch to fix some bugs or get a release out.
>>
>> And then those of us who are more willing to take the newer
>> releases and
>> test them out can pick up the new 2.3x version as willing.
>>
>> The damaging thing would be to start working on a threaded version
>> with a
>> large list of known defects in the existing release.
>
> I'm going to save this one in my mental file under "smart project
> management" -- you'd think it's common sense, but I think we've all
> been
> under that time crunch situation before that makes us think stupid,
> not
> sleep, and try to finish new and broken code instead of fixing old,
> broken, but well understood code.
>
> It is also damaging to have the next generation code remain broken for
> too long before it really starts to work and show its value -- if
> there
> even is value, as some NG projects tank miserably -- so it's
> definitely
> a balancing act as to how many new things you can do at once.

Perhaps this would be a good time to call upon the dbmail constituency
and let the users democratically decide what needs to be done next.

We're using the Bacula backup software and I keep an eye on their
mailing
list. Recently, Bacula 2.0 was released. Subsequently, users were
allowed
to submit feature requests and/or other Bacula-related (sub-)projects
that
they would like to see developed in the next version of Bacula.
After a set time, a list of these proposals was compiled and the
users could
then cast a vote on their favorite projects, which lead to the
current TODO
list for the coming months

See also:

http://www.bacula.org/?page=vote
http://www.bacula.org/?page=projects

I think this a smart way to interact with the userbase of an (open
source)
development project and assures that the work the developers are doing
is both welcome and needed.

Of course, this implies that the developers should not be too proud
to "take orders"
from the community :-)

Just a thought...

Leander
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

That'd be one approach, and would be interesting / maybe worthwhile.
What's already been done, ie. requesting people to submit feature
requests/suggestions for 2.3, is largely intended to do the same thing,
but the popularity numbers would be good to have. I don't think the
features in this particular thread are in question, but the best means
to achieve them, which I'd think is probably better left to
discussion/experience than popular vote (which might tend towards voting
for the latest buzzwords).


On Fri, 2007-02-02 at 19:22 +0100, Leander Koornneef wrote:
> Perhaps this would be a good time to call upon the dbmail constituency
> and let the users democratically decide what needs to be done next.
--
Jesse Norell - jesse@kci.net
Kentec Communications, Inc.

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Fri, Feb 2, 2007, Leander Koornneef <leander@koornneef.net> said:

[snip]
> Perhaps this would be a good time to call upon the dbmail constituency
> and let the users democratically decide what needs to be done next.

For some incidental features, I would be more than happy to take votes on
how those features should play out. For core features, I'm going to write
code that I think I can maintain and that I think has the best performance
characteristics -- within reason, of course.

[snip]
> Of course, this implies that the developers should not be too proud
> to "take orders" from the community :-)

Purchase orders are good, too ;-) But let's consider taking that
seriously: Paul and Lars are currently selling consulting services at
dbmail.eu, and I have done consulting work in the past for specific fixes
that were needed on short deadlines and for Sieve features.

Many of the major features that we've talked about putting into DBMail 2.3
will take a lot of work to code; in particular, those looking for reliable
clustering, massive scalability, better searching and even faster IMAP
should consider that these enterprise features cost megabucks from
commercial vendors, and even with open source vendors, entail significant
support contracts.

So, those using DBMail commercially, make a reasonable offer on features
that are key to your businesses, and we'll see what we can do!

Aaron
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Aaron Stone wrote:
> On Fri, Feb 2, 2007, Leander Koornneef <leander@koornneef.net> said:
>
> [snip]
>
>> Perhaps this would be a good time to call upon the dbmail constituency
>> and let the users democratically decide what needs to be done next.
>>
>
> For some incidental features, I would be more than happy to take votes on
> how those features should play out. For core features, I'm going to write
> code that I think I can maintain and that I think has the best performance
> characteristics -- within reason, of course.
>
> [snip]
>
>> Of course, this implies that the developers should not be too proud
>> to "take orders" from the community :-)
>>
>
> Purchase orders are good, too ;-) But let's consider taking that
> seriously: Paul and Lars are currently selling consulting services at
> dbmail.eu, and I have done consulting work in the past for specific fixes
> that were needed on short deadlines and for Sieve features.
>
> Many of the major features that we've talked about putting into DBMail 2.3
> will take a lot of work to code; in particular, those looking for reliable
> clustering, massive scalability, better searching and even faster IMAP
> should consider that these enterprise features cost megabucks from
> commercial vendors, and even with open source vendors, entail significant
> support contracts.
>
> So, those using DBMail commercially, make a reasonable offer on features
> that are key to your businesses, and we'll see what we can do!
>
> Aaron
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> https://mailman.fastxs.nl/mailman/listinfo/dbmail
>
I wonder if clustering and scalability enhancements might be pushed to
dbmail V3.
2.3/4 could be more related to feature enhancements and such like.

The move to a truly threaded, scalable and HA architecture is a big
change. I don't think its going to be a standard upgrade for most
people. If people want true HA then the level of funkyness is going to
go up pretty drastically. Heck its almost a fork(spoon). Dbmail 2 for
nice small instillations, dbmail 3 for big things. Otherwise you might
be trying to cover too many bases. Something big and scalable probably
wont be easy peasy to install and configure for your average home user.

Personally I see some "top level daemon" managing the whole thing and
talking via IP to the various front ends and databases. It also being
responsible for managing redundancy of data amongst a pool of databases
and the like. Basically meaning scaling is copy over the Xen image, boot
it and tell the controller that its allowed to use that.

On Sat, 2007-02-03 at 16:57 +1100, Jake Anderson wrote:

> I wonder if clustering and scalability enhancements might be pushed to
> dbmail V3. 2.3/4 could be more related to feature enhancements and
> such like.

Ok, yeah, there's a lot of IMAP work that can be done.

> The move to a truly threaded, scalable and HA architecture is a big
> change. I don't think its going to be a standard upgrade for most
> people. If people want true HA then the level of funkyness is going to
> go up pretty drastically. Heck its almost a fork(spoon). Dbmail 2 for
> nice small instillations, dbmail 3 for big things. Otherwise you might
> be trying to cover too many bases. Something big and scalable probably
> wont be easy peasy to install and configure for your average home
> user.

What funkyness are you thinking there might be?

I challenge your statement that DBMail is currently for "nice small
inst[a]llations" because there are people using it for very large
systems already. I want to design for and target that level of use
rather than just happen to be able to handle it by accident.

> Personally I see some "top level daemon" managing the whole thing and
> talking via IP to the various front ends and databases. It also being
> responsible for managing redundancy of data amongst a pool of
> databases and the like.

Yes, no. Some kind of cluster manager may be an inevitable design
decision. Built-in redundancy not so much - I'd rather rely on the
database to do this for us. With respect to pooling databases (no
relation whatsoever to pooling database connections, btw), I don't
presume that I know enough to partition what data goes to which database
server better than the experts in database design.

If someone hits the hard limits of how much data can go into one of the
database servers we use, then they're definitely doing something where
they can afford to spend the time and money needed to beef up the
software to work around the rather huge size limits in MySQL and
PostgreSQL.

> Basically meaning scaling is copy over the Xen image, boot it and tell
> the controller that its allowed to use that.

Neat idea! I think it runs exactly contrary to your assertion that it
would make everything very complex. If the cluster manager directed
other cluster members configurations, the tough part would be setting up
cluster membership. Probably involving some public key encryption to
make sure rogue nodes don't join the cluster. That's on the hard side,
but would be fun to work on.

Aaron

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

*caution* Long rambling post ahead best taken with an ice cold ginger
beer. (and possibly some salt)
>
>
>> The move to a truly threaded, scalable and HA architecture is a big
>> change. I don't think its going to be a standard upgrade for most
>> people. If people want true HA then the level of funkyness is going to
>> go up pretty drastically. Heck its almost a fork(spoon). Dbmail 2 for
>> nice small instillations, dbmail 3 for big things. Otherwise you might
>> be trying to cover too many bases. Something big and scalable probably
>> wont be easy peasy to install and configure for your average home
>> user.
>>
>
> What funkyness are you thinking there might be?
>
> I challenge your statement that DBMail is currently for "nice small
> inst[a]llations" because there are people using it for very large
> systems already. I want to design for and target that level of use
> rather than just happen to be able to handle it by accident.
>
>
I mean dbmail is (on debian/ubuntu at least) really easy to install and
setup.
unless you start packaging up the stack of applications you depend on,
configured as you depend on them you can run into the realm of
difficulty. Asterisk is a decent example of this i think. Many things
have to be "just so" for it to really work well. Asterisknow however (a
distro for asterisk) packages it all up nicley, drop the image onto the
server boot it and away you go. I am having problems getting its Xen
version running but VMware version worked well.
>> Personally I see some "top level daemon" managing the whole thing and
>> talking via IP to the various front ends and databases. It also being
>> responsible for managing redundancy of data amongst a pool of
>> databases and the like.
>>
>
> Yes, no. Some kind of cluster manager may be an inevitable design
> decision. Built-in redundancy not so much - I'd rather rely on the
> database to do this for us. With respect to pooling databases (no
> relation whatsoever to pooling database connections, btw), I don't
> presume that I know enough to partition what data goes to which database
> server better than the experts in database design.
>
>
I was refering to a list of things like "User A is on servers X and Y".
If for some reason you wish to remove server Y from the system the
manager can move the data off that server and into another. Relying on
the database for the clustering can lead to issues with scalability and
reliability. If your application is "cluster aware" in itself then you
can do crazy things like have half your servers mysql and the other half
postgress.
That in it self would have HA weenies drooling i'd think ;->

> If someone hits the hard limits of how much data can go into one of the
> database servers we use, then they're definitely doing something where
> they can afford to spend the time and money needed to beef up the
> software to work around the rather huge size limits in MySQL and
> PostgreSQL.
>
>
You can scale those out a fair bit but mysql cluster (at the moment) is
in ram tables only, which sucks donkey balls. (i run mysql for my db btw
so.... yeah... sucks to be me in that regard).
I feel (from the armchair, or in this case plastic outdoor dining chair)
that its best if the app knows whats happening. You can still use mysql
and postgres and cluster things up the wazoo if you feel like doing
that. But its simpler (i think) from the end user POV to treat the
databases as the "ideal raid hard drives", want more storage? add
another box. Want more performance? add another box. Want more X ? add
another box. Without having to muck about with setting up cluster stuff
in a database.

>> Basically meaning scaling is copy over the Xen image, boot it and tell
>> the controller that its allowed to use that.
>>
>
> Neat idea! I think it runs exactly contrary to your assertion that it
> would make everything very complex. If the cluster manager directed
> other cluster members configurations, the tough part would be setting up
> cluster membership. Probably involving some public key encryption to
> make sure rogue nodes don't join the cluster. That's on the hard side,
> but would be fun to work on.
>
> Aaron
>
The hard part is in making the complex stuff simple ;->
If the system can be made to install and setup easily with lots of
managment goodness (IE I don't have to do anything) then super ;->
Thinking for the managment app C might not be the best thing for it,
perhaps python or some such, as each individual message won't need to
hit it or will only need to do so in a trivial way. And the logic is
likley to become scary ;->.

I have been pondering ways of achieving true high availability, where a
server failure causes 0 disruption to service, even if you are half way
through recieving an email. Though that requires some additional
funkyness (all traffic into the cluster must be broadcast/multicast and
a whole bunch of other dren)


In my "ideal" system the setup is something like this.
Email recieved by a front end server (perhaps by broadcast traffic? each
server can pick which "conversations" its a part of and ignore the
others, it will scale well to a point and much farther than any current
system before you need stuff like dns round robins and proxies (though
proxy would be my 2nd choice)) that server checks its list (in memory)
of all users, if we are accepting the email then it can be passed on to
spam checks and the like.
Email then hits our "stuff it into db" section, that looks at where that
users information is stored. That app sticks the email into the
databases that need it. (so your A grade customers have 3 copies, your B
customers 2 copies and your C customers just the 1). The exact same
entry, so all ID numbers right through the db are the same.

The "Stuff it in the database" app will then notify (directally) any
imap servers that have that user registered to them that a new email has
arrived. At this point the "stuff it in the db" app is finished with the
email.

IMAP servers, are pretty similar to whats around now, difference being
that when a connection comes in, it checks with the manager which
database it should connect to for that user as a part of their
authentication. (manager picks servers based on load). If the server
tries to do a query that fails then it will try again on any other
servers that have that users data (keeping in mind that all ID's are
unique). So if a db server dies or goes offline the user doesn't even
notice. What would be nice is if the managment node could direct the
incoming imap connections to the least loaded server (again i would like
to achieve this with all machines in the pool having the same IP and
just ignoring connections they don't need)

The managment node is responsible for load balancing the servers it has
in its pool. Attach a 386, and it'll get 4 users in its database. The
managment node dynamically manages the users. So as users are added and
their usage patterns become established the load can be moved around the
servers. eg

New User johnny.
The system is pretty busy so he gets put on the least loaded server.
Johnny turns out to be a super power user with assloads of searches and
the like.
Managment node moves some of the less intensive users off that server
and onto others.

All this moving happens live as there is (always) 2 copies of the users
data in the databases.

I can see a system like that scaling as far as you could want it to,
Without the need for funkyness in terms of admin. While at the same time
not *requiring* loads of hardware. On an embedded tiny system though it
is going to run slower than dbmail does now. Theres a bunch of stuff
there your average joe isn't going to use.
A "Corperate" mail system though could be setup to be pretty HA and high
performance with just 2 boxes. The scaling being pretty linear and all.

The hardest thing is for all that stuff to work out of the box, The best
way to get people to install it at work is make it easy to install at
home. Thats why I use ubuntu.
apt-get install dbmail-hardcore

BTW wrt threads and the like, I prefer threads = processors * 2 type of
approach. Event driven state machines seem to be the most efficient way
of doing things when you get really loaded. You don't have all that
switching between threads and the overhead of hanging on to them all. To
my mind it should make the coding simpler because once you have a state
machine which will run the imap protocol, it should scale pretty much
linearly without needing to worry too much about IPC and the like.
(dbmail 7.9 perhaps?)