Mailing List Archive: dbmail + geo-replication

dbmail + geo-replication

Sep 20, 2018, 8:26 PM

Post #1 of 5 (2140 views)

I am currently operating a small mail server (postfix + dovecot) but I
have experienced a couple of ISP issues recently and would like to
improve the availability of my server by operating a second server at a
separate physical site (the sites I have available are separated by a
60ms ping).

I am new to the world of high availability, but given my resources and
constraints, an "active/passive" configuration seems to be my best
option, where I have a primary server that is up most of the time and I
can use pacemaker to switch from one to the other for failovers.

I think I could get away with not using a distributed file system if I
were to switch from dovecot to dbmail. I definitely need the
distributed database, so avoid also using a distributed file system
seems like a way to keep my architecture as simple as possible (but
please correct me if I am wrong).

For doing geo-replication, I have read that galera supports
georeplication. But my gut sense is that galera's synchronous protocol
in combination with my ping time would make anything that needed to
write to the database a lot (like dbmail) prohibitively slow. Does
anybody know if that is the case or not?

Also, I read that auto_increment columns with the galera plugin are
guaranteed to be unique, but not guaranteed to be sequential. Would
that break dbmail?

My other option would be a master-master synchronization between the two
databases. I know people on this mailing list have been doing that for
a while, because I read about that setup on this mailing list as far
back as 2006. But (noob question alert), since the master-master
database sync is asynchronous, doesn't that run the risk of data loss or
corruption from the very latest data if a server crashes? Has that ever
happened to any of you with dbmail, and what is the recover process
like?

Ryan
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://lists.nfg.nl/mailman/listinfo/dbmail

Re: dbmail + geo-replication [ In reply to ]

mauro at ezplanet

Sep 21, 2018, 8:28 AM

Post #2 of 5 (2140 views)

Permalink

Hello Ryan,

The way I achieve this through dbmail is by using a replica mysql
cluster as a back-end database

In practical terms you set up the 2 sites each with a mysql DB and
configure them for mutual replication. On top you have 1 dbmail instance
deployed at each site that points to its respective database.

In my case I have also a Linux Virtual Server load balancer in front of
the cluster with a Virtual IP used by my clients to connect to dbmail.

In this manner you can also periodically shutdown one of the 2 replicas.
Once the replica is brought back on-line mysql automatically resumes
replication.

I hope this helps,

Mauro

On 21/09/18 04:26, Ryan Beethe wrote:
> I am currently operating a small mail server (postfix + dovecot) but I
> have experienced a couple of ISP issues recently and would like to
> improve the availability of my server by operating a second server at a
> separate physical site (the sites I have available are separated by a
> 60ms ping).
>
> I am new to the world of high availability, but given my resources and
> constraints, an "active/passive" configuration seems to be my best
> option, where I have a primary server that is up most of the time and I
> can use pacemaker to switch from one to the other for failovers.
>
> I think I could get away with not using a distributed file system if I
> were to switch from dovecot to dbmail. I definitely need the
> distributed database, so avoid also using a distributed file system
> seems like a way to keep my architecture as simple as possible (but
> please correct me if I am wrong).
>
> For doing geo-replication, I have read that galera supports
> georeplication. But my gut sense is that galera's synchronous protocol
> in combination with my ping time would make anything that needed to
> write to the database a lot (like dbmail) prohibitively slow. Does
> anybody know if that is the case or not?
>
> Also, I read that auto_increment columns with the galera plugin are
> guaranteed to be unique, but not guaranteed to be sequential. Would
> that break dbmail?
>
> My other option would be a master-master synchronization between the two
> databases. I know people on this mailing list have been doing that for
> a while, because I read about that setup on this mailing list as far
> back as 2006. But (noob question alert), since the master-master
> database sync is asynchronous, doesn't that run the risk of data loss or
> corruption from the very latest data if a server crashes? Has that ever
> happened to any of you with dbmail, and what is the recover process
> like?
>
> Ryan
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> http://lists.nfg.nl/mailman/listinfo/dbmail

_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://lists.nfg.nl/mailman/listinfo/dbmail

Re: dbmail + geo-replication [ In reply to ]

abrancatelli at schema31

Sep 22, 2018, 6:14 AM

Post #3 of 5 (2138 views)

Permalink

We've been having a MySQL multi master replication with three
geographically distant server for years without any particular issue.

You don't need the galera-specific "distributed acknowledge" stuff that,
as you guessed, slows down every insert.

Basically everything in dbmail's database has a native unique primary
key (auto incrementing), PLUS an hash column that is used for
deduplication. That means that the only really clumsy problem, even in
case of a split brain situation is that if the very same email arrives
in two different location while the database is not in sync, is that the
mail doesn't get deduped. Definitely not a big issue.

One tricky parts, on other hand, is the IMAP side as you have to make
sure that any IMAP session goes to the same Backend or, always in case
of a split brain situation, the user may see mail appearing and
disappearing if two different IMAP session go to two different,
unaligned, sites.

But there are very easy workaround for that too - here we usually NGINX
as a frontend IMAP and we have a script that does some hashing of the
login name to map it to a specific geographic location - unless that
location is down.

If you need some more details in all of this just ask.

---

Andrea Brancatelli

On 2018-09-21 05:26, Ryan Beethe wrote:

> I am currently operating a small mail server (postfix + dovecot) but I
> have experienced a couple of ISP issues recently and would like to
> improve the availability of my server by operating a second server at a
> separate physical site (the sites I have available are separated by a
> 60ms ping).
>
> I am new to the world of high availability, but given my resources and
> constraints, an "active/passive" configuration seems to be my best
> option, where I have a primary server that is up most of the time and I
> can use pacemaker to switch from one to the other for failovers.
>
> I think I could get away with not using a distributed file system if I
> were to switch from dovecot to dbmail. I definitely need the
> distributed database, so avoid also using a distributed file system
> seems like a way to keep my architecture as simple as possible (but
> please correct me if I am wrong).
>
> For doing geo-replication, I have read that galera supports
> georeplication. But my gut sense is that galera's synchronous protocol
> in combination with my ping time would make anything that needed to
> write to the database a lot (like dbmail) prohibitively slow. Does
> anybody know if that is the case or not?
>
> Also, I read that auto_increment columns with the galera plugin are
> guaranteed to be unique, but not guaranteed to be sequential. Would
> that break dbmail?
>
> My other option would be a master-master synchronization between the two
> databases. I know people on this mailing list have been doing that for
> a while, because I read about that setup on this mailing list as far
> back as 2006. But (noob question alert), since the master-master
> database sync is asynchronous, doesn't that run the risk of data loss or
> corruption from the very latest data if a server crashes? Has that ever
> happened to any of you with dbmail, and what is the recover process
> like?
>
> Ryan
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> http://lists.nfg.nl/mailman/listinfo/dbmail

Re: dbmail + geo-replication [ In reply to ]

ryan at splintermail

Sep 22, 2018, 12:45 PM

Post #4 of 5 (2138 views)

Permalink

Thank you, that is all really useful information.

The more I work through my configuration the more I think my best option
at this time is going to be to put a cloud-based HAProxy in front of my
two servers so if one goes down I can easily just switch to the other.
It is good to hear that you have had a positive experience with the
multi-master syncing. It sounds like a solid solution, and in my case a
simple one because for now I won't even need IMAP from either server.

The other thing that worries me is that dbmail is going to fall into a
state of disrepair without any developers backing it (as was recently
mentioned on this list). What do other dbmail users say about that?
Are you thinking of switching?

Ryan

On Sat, Sep 22, 2018 at 03:14:50PM +0200, Andrea Brancatelli wrote:
> We've been having a MySQL multi master replication with three geographically
> distant server for years without any particular issue.
>
> You don't need the galera-specific "distributed acknowledge" stuff that, as you
> guessed, slows down every insert.
>
> Basically everything in dbmail's database has a native unique primary key (auto
> incrementing), PLUS an hash column that is used for deduplication. That means
> that the only really clumsy problem, even in case of a split brain situation is
> that if the very same email arrives in two different location while the
> database is not in sync, is that the mail doesn't get deduped. Definitely not a
> big issue.
>
> One tricky parts, on other hand, is the IMAP side as you have to make sure that
> any IMAP session goes to the same Backend or, always in case of a split brain
> situation, the user may see mail appearing and disappearing if two different
> IMAP session go to two different, unaligned, sites.
>
> But there are very easy workaround for that too - here we usually NGINX as a
> frontend IMAP and we have a script that does some hashing of the login name to
> map it to a specific geographic location - unless that location is down.
>
> If you need some more details in all of this just ask.
>
> ---
>
> Andrea Brancatelli
>
>
>
> On 2018-09-21 05:26, Ryan Beethe wrote:
>
> I am currently operating a small mail server (postfix + dovecot) but I
> have experienced a couple of ISP issues recently and would like to
> improve the availability of my server by operating a second server at a
> separate physical site (the sites I have available are separated by a
> 60ms ping).
>
> I am new to the world of high availability, but given my resources and
> constraints, an "active/passive" configuration seems to be my best
> option, where I have a primary server that is up most of the time and I
> can use pacemaker to switch from one to the other for failovers.
>
> I think I could get away with not using a distributed file system if I
> were to switch from dovecot to dbmail. I definitely need the
> distributed database, so avoid also using a distributed file system
> seems like a way to keep my architecture as simple as possible (but
> please correct me if I am wrong).
>
> For doing geo-replication, I have read that galera supports
> georeplication. But my gut sense is that galera's synchronous protocol
> in combination with my ping time would make anything that needed to
> write to the database a lot (like dbmail) prohibitively slow. Does
> anybody know if that is the case or not?
>
> Also, I read that auto_increment columns with the galera plugin are
> guaranteed to be unique, but not guaranteed to be sequential. Would
> that break dbmail?
>
> My other option would be a master-master synchronization between the two
> databases. I know people on this mailing list have been doing that for
> a while, because I read about that setup on this mailing list as far
> back as 2006. But (noob question alert), since the master-master
> database sync is asynchronous, doesn't that run the risk of data loss or
> corruption from the very latest data if a server crashes? Has that ever
> happened to any of you with dbmail, and what is the recover process
> like?
>
> Ryan
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> http://lists.nfg.nl/mailman/listinfo/dbmail
>
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://lists.nfg.nl/mailman/listinfo/dbmail

Re: dbmail + geo-replication [ In reply to ]

abrancatelli at schema31

Sep 24, 2018, 1:11 AM

Post #5 of 5 (2138 views)

Permalink

While it's true that dbmail didn't receive any commit for new code
later, emails are a pretty old and not moving target.

In general there a couple of open points and a couple of bugs here and
there that would need fixing, but frankly I don't feel worried about
using it.

Personally I'd love to see the product restart and I'd love to
personally do it, but time is very low.

---

Andrea Brancatelli

On 2018-09-22 21:45, Ryan Beethe wrote:

> Thank you, that is all really useful information.
>
> The more I work through my configuration the more I think my best option
> at this time is going to be to put a cloud-based HAProxy in front of my
> two servers so if one goes down I can easily just switch to the other.
> It is good to hear that you have had a positive experience with the
> multi-master syncing. It sounds like a solid solution, and in my case a
> simple one because for now I won't even need IMAP from either server.
>
> The other thing that worries me is that dbmail is going to fall into a
> state of disrepair without any developers backing it (as was recently
> mentioned on this list). What do other dbmail users say about that?
> Are you thinking of switching?
>
> Ryan
>
> On Sat, Sep 22, 2018 at 03:14:50PM +0200, Andrea Brancatelli wrote:
>
>> We've been having a MySQL multi master replication with three geographically
>> distant server for years without any particular issue.
>>
>> You don't need the galera-specific "distributed acknowledge" stuff that, as you
>> guessed, slows down every insert.
>>
>> Basically everything in dbmail's database has a native unique primary key (auto
>> incrementing), PLUS an hash column that is used for deduplication. That means
>> that the only really clumsy problem, even in case of a split brain situation is
>> that if the very same email arrives in two different location while the
>> database is not in sync, is that the mail doesn't get deduped. Definitely not a
>> big issue.
>>
>> One tricky parts, on other hand, is the IMAP side as you have to make sure that
>> any IMAP session goes to the same Backend or, always in case of a split brain
>> situation, the user may see mail appearing and disappearing if two different
>> IMAP session go to two different, unaligned, sites.
>>
>> But there are very easy workaround for that too - here we usually NGINX as a
>> frontend IMAP and we have a script that does some hashing of the login name to
>> map it to a specific geographic location - unless that location is down.
>>
>> If you need some more details in all of this just ask.
>>
>> ---
>>
>> Andrea Brancatelli
>>
>> On 2018-09-21 05:26, Ryan Beethe wrote:
>>
>> I am currently operating a small mail server (postfix + dovecot) but I
>> have experienced a couple of ISP issues recently and would like to
>> improve the availability of my server by operating a second server at a
>> separate physical site (the sites I have available are separated by a
>> 60ms ping).
>>
>> I am new to the world of high availability, but given my resources and
>> constraints, an "active/passive" configuration seems to be my best
>> option, where I have a primary server that is up most of the time and I
>> can use pacemaker to switch from one to the other for failovers.
>>
>> I think I could get away with not using a distributed file system if I
>> were to switch from dovecot to dbmail. I definitely need the
>> distributed database, so avoid also using a distributed file system
>> seems like a way to keep my architecture as simple as possible (but
>> please correct me if I am wrong).
>>
>> For doing geo-replication, I have read that galera supports
>> georeplication. But my gut sense is that galera's synchronous protocol
>> in combination with my ping time would make anything that needed to
>> write to the database a lot (like dbmail) prohibitively slow. Does
>> anybody know if that is the case or not?
>>
>> Also, I read that auto_increment columns with the galera plugin are
>> guaranteed to be unique, but not guaranteed to be sequential. Would
>> that break dbmail?
>>
>> My other option would be a master-master synchronization between the two
>> databases. I know people on this mailing list have been doing that for
>> a while, because I read about that setup on this mailing list as far
>> back as 2006. But (noob question alert), since the master-master
>> database sync is asynchronous, doesn't that run the risk of data loss or
>> corruption from the very latest data if a server crashes? Has that ever
>> happened to any of you with dbmail, and what is the recover process
>> like?
>>
>> Ryan
>> _______________________________________________
>> DBmail mailing list
>> DBmail@dbmail.org
>> http://lists.nfg.nl/mailman/listinfo/dbmail