Mailing List Archive

Hi lou,

The problem here is that the database consistency is not guaranteed -
the databases are not synchronized so behaviour seems pretty undefined
when for example the imap daemon connects to another database in the
mid of a session. The unique-id's and message_idnr's are no longer
unique nor will the message_idnr guarantee the correct order of
delivery; some messages/folders will suddenly be no longer available
when a system fails and some others again will no longer be available
as the first system is up again.

We are still looking for some good replication funcionality but it
seems that the logics for such failover system should be a database
issue and not a dbmail one - the ultimate system would allow dbmail to
connect to some front-end (preferrably local so network failure is
shielded from dbmail) SQL interface which would implement all the
failover functionality we desire: different groups of replicating
clusters spread out over the world :)

regards roel


lou heeft op woensdag, 26 maa 2003 om 20:57 (Europe/Amsterdam) het
volgende geschreven:

> Ello group{s},
>
> some time ago i mentioned something about having a fallback database in
> case the first one doesnt respond. I found it really usefull in the
> following scenario.
> I have domain.com and two MX records for it mx1.domain.com(5) and
> mx2.domain.com(10),
> and i`m using dbmail, let say the db on mx1 is gone, what happens, mx2
> wont help me, but
> with this patch if dbmail service on mx1 cant connect to it`s primary
> db it`ll to the
> secondary at mx2, where db1 and db2 are quite aware with it`s data in
> sense of
> replication.
>
> kinda
> if(conn1 == fails){ tellus; conn2; if(conn2 == fails) { tellus; return
> _err; } }
> of course with each connect N it`ll try to connect to db1 before
> falling back to db2;
>
> ligthly tested with pgsql/mysql agains dbmail-1.1(from
> http://www.dbmai.org), it`s quite
> simple, though i cant say how it`ll work on your mailservers.
>
>
> let me know if i did something wrong. sometime (when i find it) i`ll
> try to change the
> stuff to use more than 2 dbs and not to be so static. Hope Eelco, Roel
> would be keen on
> impl something like this permanently?
> (patch attached)
>
>
> cheers
> -lou
>
> --
>
> Lou Kamenov AEYE R&D lou.kamenov@aeye.net
> FreeBSD BGUG http://www.freebsd-bg.org lou@FreeBSD-bg.org
> Secureroot UK http://secureroot.org.uk phayze@secureroot.org.uk
> Key Fingerprint - 936F F64A AD50 2D27 07E7 6629 F493 95AE A297 084A
> One advantage of talking to yourself is that you know at least
> somebody's listening. - Franklin P. Jones
> <dbmail-fallback.patch.gz>

_________________________
R.A. Rozendaal
ICT Manager
IC&S
T: +31 30 2322878
F: +31 30 2322305
www.ic-s.nl

> Hi lou,
Ello Roel,

> The problem here is that the database consistency is not guaranteed -
> the databases are not synchronized so behaviour seems pretty undefined
> when for example the imap daemon connects to another database in the
> mid of a session.

True, no database consistency = problems, but.
I still dont get something, let say if we have good replication (not perfect anyway,
let say multimaster replication - no locks). If the replication keeps the two db
servers replicated at any time, i dont see a reason why it wont work.

> The unique-id's and message_idnr's are no longer
> unique nor will the message_idnr guarantee the correct order of
> delivery; some messages/folders will suddenly be no longer available
> when a system fails and some others again will no longer be available
> as the first system is up again.

Let say two parallel inserts in the very same time of the replication i.e. server1 will
have uID2 and server2 uID2 what happens after that? How the servers are going to deal with
it, i`m sure someone at mysql.com and pgsql-r thought of that and found the right
algorithm for it. Anaway, what happens when we replace the uIDs with randomly generated
unique IDs.

Anyway, we expect really quick replication from server2 to server1 to make the messages
available *again* on server1.

>
> We are still looking for some good replication funcionality but it
> seems that the logics for such failover system should be a database
> issue and not a dbmail one - the ultimate system would allow dbmail to
> connect to some front-end (preferrably local so network failure is
> shielded from dbmail) SQL interface which would implement all the
> failover functionality we desire: different groups of replicating
> clusters spread out over the world :)

It`s possible with code but the performance will be decresed and the whole thing would be
much more complex than it`s now.

I personally dont strife for clustering but for grid redundancy and shared responsibility
regarding different jobs and objects.

That will be really neat algorithm, since a machine is not able to analyze the objects
before applying/inserting/updating them. Even a one with rules in it, will face a new
stuff, and wont be able to decide what to do with it, since that problem is not only
present in dbs but in everything. Sometime ago i was having the same discussion about fs
synchronization and consistency, but trust me Databases are far more consistent in any way
than a filesystem, so that`s why i`m using dbmail.

Anyway if you`re happy discussing this issue or i misunderstood you in any way, please let
me know.

If it doesnt work, i`ll be the first one to find out, since i installed it recently on two
productional systems.

cheers,
-lou


--
Lou Kamenov AEYE R&D lou.kamenov@aeye.net
FreeBSD BGUG http://www.freebsd-bg.org lou@FreeBSD-bg.org
Secureroot UK http://secureroot.org.uk phayze@secureroot.org.uk
Key Fingerprint - 936F F64A AD50 2D27 07E7 6629 F493 95AE A297 084A
One advantage of talking to yourself is that you know at least
somebody's listening. - Franklin P. Jones

Hello,

A few thoughts... I think Roel is probably on a more feasible
track for dbmail - it seems the current implimentation would need
some substantial redesign if dbmail itsself is going to manage
redundancy and consistency, and would probably have to also provide
some utilities to syncronize the db servers after that; right now
it needs to be done transparently, handled completely by the
database.

Consider the following scenario: I have 2 sites in 2
geographically dispersed locations, and setup one dbmail server and
one database server at each location. The site allows anonymous
user signups, and there are users connecting to the local ISP hosting
the servers using the system. Now one site looses internet (and
intranet) connectivity. Users continue to be made and maybe delete,
mail is sent locally and one site continues to receive mail from the
Internet. Now comparing the 2 databases, you have multiple users with
the same user_idnr, possibly duplicate userid's, messages with the
same message_idnr's, mailboxes created with duplicate mailbox_idnr's,
and generally one big mess. There is no right way to solve that - it
would take redesigning almost all the dbmail database tables and a
fair chunk of how it deals with databases.

Don't get me wrong, I would *love* to have duplicate database
servers too, and I think it's definitely worth looking at all the
options, but I don't think a quick "try alternate database
connections" approach will work now.

Jesse


---- Original Message ----
From: lou <dbmail-dev@dbmail.org>
To: dbmail-dev@dbmail.org
Subject: [Dbmail-dev] re: redundancy patch
Sent: Thu, 27 Mar 2003 18:59:34 +0000

> > Hi lou,
> Ello Roel,
>
> > The problem here is that the database consistency is not guaranteed -
> > the databases are not synchronized so behaviour seems pretty undefined
> > when for example the imap daemon connects to another database in the
> > mid of a session.
>
> True, no database consistency = problems, but.
> I still dont get something, let say if we have good replication (not perfect anyway,
> let say multimaster replication - no locks). If the replication keeps the two db
> servers replicated at any time, i dont see a reason why it wont work.
>
> > The unique-id's and message_idnr's are no longer
> > unique nor will the message_idnr guarantee the correct order of
> > delivery; some messages/folders will suddenly be no longer available
> > when a system fails and some others again will no longer be available
> > as the first system is up again.
>
> Let say two parallel inserts in the very same time of the replication i.e. server1 will
> have uID2 and server2 uID2 what happens after that? How the servers are going to deal with
> it, i`m sure someone at mysql.com and pgsql-r thought of that and found the right
> algorithm for it. Anaway, what happens when we replace the uIDs with randomly generated
> unique IDs.
>
> Anyway, we expect really quick replication from server2 to server1 to make the messages
> available *again* on server1.
>
> >
> > We are still looking for some good replication funcionality but it
> > seems that the logics for such failover system should be a database
> > issue and not a dbmail one - the ultimate system would allow dbmail to
> > connect to some front-end (preferrably local so network failure is
> > shielded from dbmail) SQL interface which would implement all the
> > failover functionality we desire: different groups of replicating
> > clusters spread out over the world :)
>
> It`s possible with code but the performance will be decresed and the whole thing would be
> much more complex than it`s now.
>
> I personally dont strife for clustering but for grid redundancy and shared responsibility
> regarding different jobs and objects.
>
> That will be really neat algorithm, since a machine is not able to analyze the objects
> before applying/inserting/updating them. Even a one with rules in it, will face a new
> stuff, and wont be able to decide what to do with it, since that problem is not only
> present in dbs but in everything. Sometime ago i was having the same discussion about fs
> synchronization and consistency, but trust me Databases are far more consistent in any way
> than a filesystem, so that`s why i`m using dbmail.
>
> Anyway if you`re happy discussing this issue or i misunderstood you in any way, please let
> me know.
>
> If it doesnt work, i`ll be the first one to find out, since i installed it recently on two
> productional systems.
>
> cheers,
> -lou
>
>
> --
> Lou Kamenov AEYE R&D lou.kamenov@aeye.net
> FreeBSD BGUG http://www.freebsd-bg.org lou@FreeBSD-bg.org
> Secureroot UK http://secureroot.org.uk phayze@secureroot.org.uk
> Key Fingerprint - 936F F64A AD50 2D27 07E7 6629 F493 95AE A297 084A
> One advantage of talking to yourself is that you know at least
> somebody's listening. - Franklin P. Jones
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>
-- End Original Message --


--
Jesse Norell
jesse (at) kci.net

> Hello,
>
> A few thoughts... I think Roel is probably on a more feasible
> track for dbmail - it seems the current implimentation would need
> some substantial redesign if dbmail itsself is going to manage
> redundancy and consistency, and would probably have to also provide
> some utilities to syncronize the db servers after that; right now
> it needs to be done transparently, handled completely by the
> database.

true.

> Consider the following scenario: I have 2 sites in 2
> geographically dispersed locations, and setup one dbmail server and
> one database server at each location. The site allows anonymous
> user signups, and there are users connecting to the local ISP hosting
> the servers using the system. Now one site looses internet (and
> intranet) connectivity. Users continue to be made and maybe delete,
> mail is sent locally and one site continues to receive mail from the
> Internet. Now comparing the 2 databases, you have multiple users with
> the same user_idnr, possibly duplicate userid's, messages with the
> same message_idnr's, mailboxes created with duplicate mailbox_idnr's,
> and generally one big mess. There is no right way to solve that - it
> would take redesigning almost all the dbmail database tables and a
> fair chunk of how it deals with databases.
>

i`m quite aware of that, and it has been already tested, and yes it`s failing.

> Don't get me wrong, I would *love* to have duplicate database
> servers too, and I think it's definitely worth looking at all the
> options, but I don't think a quick "try alternate database
> connections" approach will work now.

How many options do we have? i say *quite* a few, db replication in a consistent manner or
let the code synchronize it in a more smart manner, change the store procedures? If anyone
can think of something else i`ll be more than delighted to hear it.


for these reasons all of us using dbmail are statically linked to 1db per server without
being related in any way to any other db server regarding the problems that you and roel
mentioned in your emails. Since i`m not aware of db smart enuf to cope with it.
However I personally find the task implemeting a mechanism and
redesigning the backend much more easier. Consider the fact that some dbs will let us
replicate in our dream way, what about the others? that means the users will be urged to
use only N specific dbs when they want to use M let say.


I`m about 80% sure that you guys are aware of this:
why this wont happen on a filesystem, when you sync 2 different maildirs, let say using
qmail. Cause basically the identifiers indexing this objects are not the same never and
simple rsync would synchronize it.


think about that, cause i think that is the case where we should look for a workaround for
now until something more robust and sofisticated comes on the scene. And not in the
application itself but the way it stores all the data, to be exceptional from any
other. I`ll be happy to help with anything i can if anyone is keen on moving forward
regarding this impl.

all you guys from IC&C did something amazing in the form of dbmail, thanks a lot!

just my two bits.

cheers,
-lou



--

Lou Kamenov AEYE R&D lou.kamenov@aeye.net
FreeBSD BGUG http://www.freebsd-bg.org lou@FreeBSD-bg.org
Secureroot UK http://secureroot.org.uk phayze@secureroot.org.uk
Key Fingerprint - 936F F64A AD50 2D27 07E7 6629 F493 95AE A297 084A
One advantage of talking to yourself is that you know at least
somebody's listening. - Franklin P. Jones

An interesting front end is SQL Relay, http://firstworks.com/

My thinking on the database redundancy issue is a compromise with the
full-failover and the write-master+read-slaves ideas; There's a main
server with read/write (the master) and a slave with read only. The
primary and secondary MX idea is a very good one, and when the primary
goes down, the secondary will receive the mail but not be allowed to
insert it. Incoming messages will queue up between postfix and dbmail
(must be tuned not to attempt immediate retries, though) and once the
master is alive, will be delivered. In the mean time, everyone can still
read their mail from the slave (but no flagging, deleting or moving).

Aaron


On Thu, 27 Mar 2003, Roel Rozendaal - IC&S wrote:

> Hi lou,
>
> The problem here is that the database consistency is not guaranteed -
> the databases are not synchronized so behaviour seems pretty undefined
> when for example the imap daemon connects to another database in the
> mid of a session. The unique-id's and message_idnr's are no longer
> unique nor will the message_idnr guarantee the correct order of
> delivery; some messages/folders will suddenly be no longer available
> when a system fails and some others again will no longer be available
> as the first system is up again.
>
> We are still looking for some good replication funcionality but it
> seems that the logics for such failover system should be a database
> issue and not a dbmail one - the ultimate system would allow dbmail to
> connect to some front-end (preferrably local so network failure is
> shielded from dbmail) SQL interface which would implement all the
> failover functionality we desire: different groups of replicating
> clusters spread out over the world :)
>
> regards roel
>
>
> lou heeft op woensdag, 26 maa 2003 om 20:57 (Europe/Amsterdam) het
> volgende geschreven:
>
> > Ello group{s},
> >
> > some time ago i mentioned something about having a fallback database in
> > case the first one doesnt respond. I found it really usefull in the
> > following scenario.
> > I have domain.com and two MX records for it mx1.domain.com(5) and
> > mx2.domain.com(10),
> > and i`m using dbmail, let say the db on mx1 is gone, what happens, mx2
> > wont help me, but
> > with this patch if dbmail service on mx1 cant connect to it`s primary
> > db it`ll to the
> > secondary at mx2, where db1 and db2 are quite aware with it`s data in
> > sense of
> > replication.
> >
> > kinda
> > if(conn1 == fails){ tellus; conn2; if(conn2 == fails) { tellus; return
> > _err; } }
> > of course with each connect N it`ll try to connect to db1 before
> > falling back to db2;
> >
> > ligthly tested with pgsql/mysql agains dbmail-1.1(from
> > http://www.dbmai.org), it`s quite
> > simple, though i cant say how it`ll work on your mailservers.
> >
> >
> > let me know if i did something wrong. sometime (when i find it) i`ll
> > try to change the
> > stuff to use more than 2 dbs and not to be so static. Hope Eelco, Roel
> > would be keen on
> > impl something like this permanently?
> > (patch attached)
> >
> >
> > cheers
> > -lou
> >
> > --
> >
> > Lou Kamenov AEYE R&D lou.kamenov@aeye.net
> > FreeBSD BGUG http://www.freebsd-bg.org lou@FreeBSD-bg.org
> > Secureroot UK http://secureroot.org.uk phayze@secureroot.org.uk
> > Key Fingerprint - 936F F64A AD50 2D27 07E7 6629 F493 95AE A297 084A
> > One advantage of talking to yourself is that you know at least
> > somebody's listening. - Franklin P. Jones
> > <dbmail-fallback.patch.gz>
>
> _________________________
> R.A. Rozendaal
> ICT Manager
> IC&S
> T: +31 30 2322878
> F: +31 30 2322305
> www.ic-s.nl
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>