Mailing List Archive

Paul J Stevens wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> DBMail 2.2.2 released.
>
> We're happy to announce the availability of DBMail 2.2.2.
>

Excellent :) Any word on the memory footprint of dbmail-imapd processes
increasing over time? I checked the changelog but didn't see any mention
of it. I had a script that restarted the dbmail-imapd every 2 hours, but
unfortunately thunderbird doesn't handle the imap server going down
during some operations ( ie it crashes ), and I've had a user complain
about loosing an email they were working on - they had just hit 'save
draft' when the imapd server restarted.

--
Daniel Kasak
IT Developer
NUS Consulting Group
Level 5, 77 Pacific Highway
North Sydney, NSW, Australia 2060
T: (+61) 2 9922-7676 / F: (+61) 2 9922 7989
email: dkasak@nusconsulting.com.au
website: http://www.nusconsulting.com.au
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Thu, Feb 1, 2007, Daniel Kasak <dkasak@nusconsulting.com.au> said:

> Paul J Stevens wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>>
>> DBMail 2.2.2 released.
>>
>> We're happy to announce the availability of DBMail 2.2.2.
>>
>
> Excellent :) Any word on the memory footprint of dbmail-imapd processes
> increasing over time? I checked the changelog but didn't see any mention
> of it. I had a script that restarted the dbmail-imapd every 2 hours, but
> unfortunately thunderbird doesn't handle the imap server going down
> during some operations ( ie it crashes ), and I've had a user complain
> about loosing an email they were working on - they had just hit 'save
> draft' when the imapd server restarted.

It's better, but not perfect (yet!). You should definitely be able to get
more than two hours of life out of each process; you've kinda taken the
baseball bat approach to freeing memory there :-P

Aaron
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

I maintain my own dbmail ebuild since 2002, tried to get that in the
portage since 2002
through the sparc herds but no avail :(

I certainly whish the ebuild goes in

xming

Martin Hierling wrote:
> Aron,
>
> Awesome. Have you had any luck getting in touch with someone in
> net-mail?
> Half the people I've emailed haven't even bothered responding, and the
>
>
> i have tried to get g developer (for another project) but without
> luck. Personally i use a bunch of overlays so it doesnt bother me to
> get dbmail out of sunrise. I actually have never tried to get in touch
> with the net-mail guys. But i will try next days. dbmail is not that
> intensive in maintaining so perhaps someone will bring it into portage
> if i write the ebuilds .... will ask.
>
> regards Martin
> ------------------------------------------------------------------------
>
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> https://mailman.fastxs.nl/mailman/listinfo/dbmail
>

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

xming,

is your ebuild public available? URL? If not can you send it to me? Perhaps
i can improve the one in sunrise.

regards Martin

On 1/31/2007, "Matthew T. O'Connor" <matthew@zeut.net> wrote:
>> So yes, imo also there's a very solid case for threads in dbmail. But
>> like Aaron says, it will take some time to do this right. Neither I nor
>> Aaron (afaik) have any experience with threads in C so we'll have to
>> learn how to do it as we go along :-)
>
>This sounds scary to me. Obviously it's not my project and I'm not the
>one coding, I'm just worried as an admin that depends on DBMail that we
>are going to open up a large can of worms for dubious gain.
>

I think the prudent thing to do in this case is to make sure that the
2.2.2 release is something that works very well and is quite stable.
The goal here is to give the dbmail community a IMAP/POP server that
will be suitable for their needs for some time to come in the future.

This will give the developers the time necessary to do everything that
they need to do in order to get threads working right. And without an
immediate crunch to fix some bugs or get a release out.

And then those of us who are more willing to take the newer releases and
test them out can pick up the new 2.3x version as willing.

The damaging thing would be to start working on a threaded version with a
large list of known defects in the existing release.
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On 2/1/2007, "Aaron Stone" <aaron@serendipity.cx> wrote:

>Bottom line, if you want a thousand open connections on a single mail
>server with current DBMail, you're going to need a thousand processes
>and a thousand database handles. This don't scale.
>
>I really would prefer to keep most of this discussion on the dbmail-dev
>list, for the sake of the sanity of those who are on the dbmail list for
>setup and configuration help and not for core architectural issues!
>

Yes, but I personally am finding this discussion educational even if you
have decided for other reasons to pursue a given direction.
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Fri, 2007-02-02 at 12:24 -0500, tom@tacocat.net wrote:
> On 1/31/2007, "Matthew T. O'Connor" <matthew@zeut.net> wrote:
> >> So yes, imo also there's a very solid case for threads in dbmail. But
> >> like Aaron says, it will take some time to do this right. Neither I nor
> >> Aaron (afaik) have any experience with threads in C so we'll have to
> >> learn how to do it as we go along :-)
> >
> >This sounds scary to me. Obviously it's not my project and I'm not the
> >one coding, I'm just worried as an admin that depends on DBMail that we
> >are going to open up a large can of worms for dubious gain.
> >
>
> I think the prudent thing to do in this case is to make sure that the
> 2.2.2 release is something that works very well and is quite stable.
> The goal here is to give the dbmail community a IMAP/POP server that
> will be suitable for their needs for some time to come in the future.
>
> This will give the developers the time necessary to do everything that
> they need to do in order to get threads working right. And without an
> immediate crunch to fix some bugs or get a release out.
>
> And then those of us who are more willing to take the newer releases and
> test them out can pick up the new 2.3x version as willing.
>
> The damaging thing would be to start working on a threaded version with a
> large list of known defects in the existing release.

I'm going to save this one in my mental file under "smart project
management" -- you'd think it's common sense, but I think we've all been
under that time crunch situation before that makes us think stupid, not
sleep, and try to finish new and broken code instead of fixing old,
broken, but well understood code.

It is also damaging to have the next generation code remain broken for
too long before it really starts to work and show its value -- if there
even is value, as some NG projects tank miserably -- so it's definitely
a balancing act as to how many new things you can do at once.

Aaron

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On 2-feb-2007, at 18:48, Aaron Stone wrote:

> On Fri, 2007-02-02 at 12:24 -0500, tom@tacocat.net wrote:
>> On 1/31/2007, "Matthew T. O'Connor" <matthew@zeut.net> wrote:
>>>> So yes, imo also there's a very solid case for threads in
>>>> dbmail. But
>>>> like Aaron says, it will take some time to do this right.
>>>> Neither I nor
>>>> Aaron (afaik) have any experience with threads in C so we'll
>>>> have to
>>>> learn how to do it as we go along :-)
>>>
>>> This sounds scary to me. Obviously it's not my project and I'm
>>> not the
>>> one coding, I'm just worried as an admin that depends on DBMail
>>> that we
>>> are going to open up a large can of worms for dubious gain.
>>>
>>
>> I think the prudent thing to do in this case is to make sure that the
>> 2.2.2 release is something that works very well and is quite stable.
>> The goal here is to give the dbmail community a IMAP/POP server that
>> will be suitable for their needs for some time to come in the future.
>>
>> This will give the developers the time necessary to do everything
>> that
>> they need to do in order to get threads working right. And
>> without an
>> immediate crunch to fix some bugs or get a release out.
>>
>> And then those of us who are more willing to take the newer
>> releases and
>> test them out can pick up the new 2.3x version as willing.
>>
>> The damaging thing would be to start working on a threaded version
>> with a
>> large list of known defects in the existing release.
>
> I'm going to save this one in my mental file under "smart project
> management" -- you'd think it's common sense, but I think we've all
> been
> under that time crunch situation before that makes us think stupid,
> not
> sleep, and try to finish new and broken code instead of fixing old,
> broken, but well understood code.
>
> It is also damaging to have the next generation code remain broken for
> too long before it really starts to work and show its value -- if
> there
> even is value, as some NG projects tank miserably -- so it's
> definitely
> a balancing act as to how many new things you can do at once.

Perhaps this would be a good time to call upon the dbmail constituency
and let the users democratically decide what needs to be done next.

We're using the Bacula backup software and I keep an eye on their
mailing
list. Recently, Bacula 2.0 was released. Subsequently, users were
allowed
to submit feature requests and/or other Bacula-related (sub-)projects
that
they would like to see developed in the next version of Bacula.
After a set time, a list of these proposals was compiled and the
users could
then cast a vote on their favorite projects, which lead to the
current TODO
list for the coming months

See also:

http://www.bacula.org/?page=vote
http://www.bacula.org/?page=projects

I think this a smart way to interact with the userbase of an (open
source)
development project and assures that the work the developers are doing
is both welcome and needed.

Of course, this implies that the developers should not be too proud
to "take orders"
from the community :-)

Just a thought...

Leander
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

That'd be one approach, and would be interesting / maybe worthwhile.
What's already been done, ie. requesting people to submit feature
requests/suggestions for 2.3, is largely intended to do the same thing,
but the popularity numbers would be good to have. I don't think the
features in this particular thread are in question, but the best means
to achieve them, which I'd think is probably better left to
discussion/experience than popular vote (which might tend towards voting
for the latest buzzwords).


On Fri, 2007-02-02 at 19:22 +0100, Leander Koornneef wrote:
> Perhaps this would be a good time to call upon the dbmail constituency
> and let the users democratically decide what needs to be done next.
--
Jesse Norell - jesse@kci.net
Kentec Communications, Inc.

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

On Fri, Feb 2, 2007, Leander Koornneef <leander@koornneef.net> said:

[snip]
> Perhaps this would be a good time to call upon the dbmail constituency
> and let the users democratically decide what needs to be done next.

For some incidental features, I would be more than happy to take votes on
how those features should play out. For core features, I'm going to write
code that I think I can maintain and that I think has the best performance
characteristics -- within reason, of course.

[snip]
> Of course, this implies that the developers should not be too proud
> to "take orders" from the community :-)

Purchase orders are good, too ;-) But let's consider taking that
seriously: Paul and Lars are currently selling consulting services at
dbmail.eu, and I have done consulting work in the past for specific fixes
that were needed on short deadlines and for Sieve features.

Many of the major features that we've talked about putting into DBMail 2.3
will take a lot of work to code; in particular, those looking for reliable
clustering, massive scalability, better searching and even faster IMAP
should consider that these enterprise features cost megabucks from
commercial vendors, and even with open source vendors, entail significant
support contracts.

So, those using DBMail commercially, make a reasonable offer on features
that are key to your businesses, and we'll see what we can do!

Aaron
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

Aaron Stone wrote:
> On Fri, Feb 2, 2007, Leander Koornneef <leander@koornneef.net> said:
>
> [snip]
>
>> Perhaps this would be a good time to call upon the dbmail constituency
>> and let the users democratically decide what needs to be done next.
>>
>
> For some incidental features, I would be more than happy to take votes on
> how those features should play out. For core features, I'm going to write
> code that I think I can maintain and that I think has the best performance
> characteristics -- within reason, of course.
>
> [snip]
>
>> Of course, this implies that the developers should not be too proud
>> to "take orders" from the community :-)
>>
>
> Purchase orders are good, too ;-) But let's consider taking that
> seriously: Paul and Lars are currently selling consulting services at
> dbmail.eu, and I have done consulting work in the past for specific fixes
> that were needed on short deadlines and for Sieve features.
>
> Many of the major features that we've talked about putting into DBMail 2.3
> will take a lot of work to code; in particular, those looking for reliable
> clustering, massive scalability, better searching and even faster IMAP
> should consider that these enterprise features cost megabucks from
> commercial vendors, and even with open source vendors, entail significant
> support contracts.
>
> So, those using DBMail commercially, make a reasonable offer on features
> that are key to your businesses, and we'll see what we can do!
>
> Aaron
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> https://mailman.fastxs.nl/mailman/listinfo/dbmail
>
I wonder if clustering and scalability enhancements might be pushed to
dbmail V3.
2.3/4 could be more related to feature enhancements and such like.

The move to a truly threaded, scalable and HA architecture is a big
change. I don't think its going to be a standard upgrade for most
people. If people want true HA then the level of funkyness is going to
go up pretty drastically. Heck its almost a fork(spoon). Dbmail 2 for
nice small instillations, dbmail 3 for big things. Otherwise you might
be trying to cover too many bases. Something big and scalable probably
wont be easy peasy to install and configure for your average home user.

Personally I see some "top level daemon" managing the whole thing and
talking via IP to the various front ends and databases. It also being
responsible for managing redundancy of data amongst a pool of databases
and the like. Basically meaning scaling is copy over the Xen image, boot
it and tell the controller that its allowed to use that.

On Sat, 2007-02-03 at 16:57 +1100, Jake Anderson wrote:

> I wonder if clustering and scalability enhancements might be pushed to
> dbmail V3. 2.3/4 could be more related to feature enhancements and
> such like.

Ok, yeah, there's a lot of IMAP work that can be done.

> The move to a truly threaded, scalable and HA architecture is a big
> change. I don't think its going to be a standard upgrade for most
> people. If people want true HA then the level of funkyness is going to
> go up pretty drastically. Heck its almost a fork(spoon). Dbmail 2 for
> nice small instillations, dbmail 3 for big things. Otherwise you might
> be trying to cover too many bases. Something big and scalable probably
> wont be easy peasy to install and configure for your average home
> user.

What funkyness are you thinking there might be?

I challenge your statement that DBMail is currently for "nice small
inst[a]llations" because there are people using it for very large
systems already. I want to design for and target that level of use
rather than just happen to be able to handle it by accident.

> Personally I see some "top level daemon" managing the whole thing and
> talking via IP to the various front ends and databases. It also being
> responsible for managing redundancy of data amongst a pool of
> databases and the like.

Yes, no. Some kind of cluster manager may be an inevitable design
decision. Built-in redundancy not so much - I'd rather rely on the
database to do this for us. With respect to pooling databases (no
relation whatsoever to pooling database connections, btw), I don't
presume that I know enough to partition what data goes to which database
server better than the experts in database design.

If someone hits the hard limits of how much data can go into one of the
database servers we use, then they're definitely doing something where
they can afford to spend the time and money needed to beef up the
software to work around the rather huge size limits in MySQL and
PostgreSQL.

> Basically meaning scaling is copy over the Xen image, boot it and tell
> the controller that its allowed to use that.

Neat idea! I think it runs exactly contrary to your assertion that it
would make everything very complex. If the cluster manager directed
other cluster members configurations, the tough part would be setting up
cluster membership. Probably involving some public key encryption to
make sure rogue nodes don't join the cluster. That's on the hard side,
but would be fun to work on.

Aaron

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail

*caution* Long rambling post ahead best taken with an ice cold ginger
beer. (and possibly some salt)
>
>
>> The move to a truly threaded, scalable and HA architecture is a big
>> change. I don't think its going to be a standard upgrade for most
>> people. If people want true HA then the level of funkyness is going to
>> go up pretty drastically. Heck its almost a fork(spoon). Dbmail 2 for
>> nice small instillations, dbmail 3 for big things. Otherwise you might
>> be trying to cover too many bases. Something big and scalable probably
>> wont be easy peasy to install and configure for your average home
>> user.
>>
>
> What funkyness are you thinking there might be?
>
> I challenge your statement that DBMail is currently for "nice small
> inst[a]llations" because there are people using it for very large
> systems already. I want to design for and target that level of use
> rather than just happen to be able to handle it by accident.
>
>
I mean dbmail is (on debian/ubuntu at least) really easy to install and
setup.
unless you start packaging up the stack of applications you depend on,
configured as you depend on them you can run into the realm of
difficulty. Asterisk is a decent example of this i think. Many things
have to be "just so" for it to really work well. Asterisknow however (a
distro for asterisk) packages it all up nicley, drop the image onto the
server boot it and away you go. I am having problems getting its Xen
version running but VMware version worked well.
>> Personally I see some "top level daemon" managing the whole thing and
>> talking via IP to the various front ends and databases. It also being
>> responsible for managing redundancy of data amongst a pool of
>> databases and the like.
>>
>
> Yes, no. Some kind of cluster manager may be an inevitable design
> decision. Built-in redundancy not so much - I'd rather rely on the
> database to do this for us. With respect to pooling databases (no
> relation whatsoever to pooling database connections, btw), I don't
> presume that I know enough to partition what data goes to which database
> server better than the experts in database design.
>
>
I was refering to a list of things like "User A is on servers X and Y".
If for some reason you wish to remove server Y from the system the
manager can move the data off that server and into another. Relying on
the database for the clustering can lead to issues with scalability and
reliability. If your application is "cluster aware" in itself then you
can do crazy things like have half your servers mysql and the other half
postgress.
That in it self would have HA weenies drooling i'd think ;->

> If someone hits the hard limits of how much data can go into one of the
> database servers we use, then they're definitely doing something where
> they can afford to spend the time and money needed to beef up the
> software to work around the rather huge size limits in MySQL and
> PostgreSQL.
>
>
You can scale those out a fair bit but mysql cluster (at the moment) is
in ram tables only, which sucks donkey balls. (i run mysql for my db btw
so.... yeah... sucks to be me in that regard).
I feel (from the armchair, or in this case plastic outdoor dining chair)
that its best if the app knows whats happening. You can still use mysql
and postgres and cluster things up the wazoo if you feel like doing
that. But its simpler (i think) from the end user POV to treat the
databases as the "ideal raid hard drives", want more storage? add
another box. Want more performance? add another box. Want more X ? add
another box. Without having to muck about with setting up cluster stuff
in a database.

>> Basically meaning scaling is copy over the Xen image, boot it and tell
>> the controller that its allowed to use that.
>>
>
> Neat idea! I think it runs exactly contrary to your assertion that it
> would make everything very complex. If the cluster manager directed
> other cluster members configurations, the tough part would be setting up
> cluster membership. Probably involving some public key encryption to
> make sure rogue nodes don't join the cluster. That's on the hard side,
> but would be fun to work on.
>
> Aaron
>
The hard part is in making the complex stuff simple ;->
If the system can be made to install and setup easily with lots of
managment goodness (IE I don't have to do anything) then super ;->
Thinking for the managment app C might not be the best thing for it,
perhaps python or some such, as each individual message won't need to
hit it or will only need to do so in a trivial way. And the logic is
likley to become scary ;->.

I have been pondering ways of achieving true high availability, where a
server failure causes 0 disruption to service, even if you are half way
through recieving an email. Though that requires some additional
funkyness (all traffic into the cluster must be broadcast/multicast and
a whole bunch of other dren)


In my "ideal" system the setup is something like this.
Email recieved by a front end server (perhaps by broadcast traffic? each
server can pick which "conversations" its a part of and ignore the
others, it will scale well to a point and much farther than any current
system before you need stuff like dns round robins and proxies (though
proxy would be my 2nd choice)) that server checks its list (in memory)
of all users, if we are accepting the email then it can be passed on to
spam checks and the like.
Email then hits our "stuff it into db" section, that looks at where that
users information is stored. That app sticks the email into the
databases that need it. (so your A grade customers have 3 copies, your B
customers 2 copies and your C customers just the 1). The exact same
entry, so all ID numbers right through the db are the same.

The "Stuff it in the database" app will then notify (directally) any
imap servers that have that user registered to them that a new email has
arrived. At this point the "stuff it in the db" app is finished with the
email.

IMAP servers, are pretty similar to whats around now, difference being
that when a connection comes in, it checks with the manager which
database it should connect to for that user as a part of their
authentication. (manager picks servers based on load). If the server
tries to do a query that fails then it will try again on any other
servers that have that users data (keeping in mind that all ID's are
unique). So if a db server dies or goes offline the user doesn't even
notice. What would be nice is if the managment node could direct the
incoming imap connections to the least loaded server (again i would like
to achieve this with all machines in the pool having the same IP and
just ignoring connections they don't need)

The managment node is responsible for load balancing the servers it has
in its pool. Attach a 386, and it'll get 4 users in its database. The
managment node dynamically manages the users. So as users are added and
their usage patterns become established the load can be moved around the
servers. eg

New User johnny.
The system is pretty busy so he gets put on the least loaded server.
Johnny turns out to be a super power user with assloads of searches and
the like.
Managment node moves some of the less intensive users off that server
and onto others.

All this moving happens live as there is (always) 2 copies of the users
data in the databases.

I can see a system like that scaling as far as you could want it to,
Without the need for funkyness in terms of admin. While at the same time
not *requiring* loads of hardware. On an embedded tiny system though it
is going to run slower than dbmail does now. Theres a bunch of stuff
there your average joe isn't going to use.
A "Corperate" mail system though could be setup to be pretty HA and high
performance with just 2 boxes. The scaling being pretty linear and all.

The hardest thing is for all that stuff to work out of the box, The best
way to get people to install it at work is make it easy to install at
home. Thats why I use ubuntu.
apt-get install dbmail-hardcore

BTW wrt threads and the like, I prefer threads = processors * 2 type of
approach. Event driven state machines seem to be the most efficient way
of doing things when you get really loaded. You don't have all that
switching between threads and the overhead of hanging on to them all. To
my mind it should make the coding simpler because once you have a state
machine which will run the imap protocol, it should scale pretty much
linearly without needing to worry too much about IPC and the like.
(dbmail 7.9 perhaps?)