Mailing List Archive

On Wed, Mar 27, 2024 at 11:59?AM J. Roeleveld <joost@antarean.org> wrote:
>
> Hi all,
>
> I am looking for a way to synchronise a filesystem between 2 servers.
Changes
> can occur on both sides which means I need to have it synchronise in both
> directions.
>
> Does anyone have any thoughts on this?
>
> Also, both servers are connected using a slow VPN link, which is why I
can't
> simply access files on the remote server.
>
> --
> Joost
>
>

How synchronized? For instance, does it need to handle identicals where
a file is on both sides but has been moved?

- Mark

Il 27/03/24 19:58, J. Roeleveld ha scritto:
> Hi all,
>
> I am looking for a way to synchronise a filesystem between 2 servers. Changes
> can occur on both sides which means I need to have it synchronise in both
> directions.
>
> Does anyone have any thoughts on this?
>
> Also, both servers are connected using a slow VPN link, which is why I can't
> simply access files on the remote server.
>
> --
> Joost
>
Unison. I use it just for that but can't say anything about the VPN bit,
my servers are on local network.

raffaele

Am Wed, Mar 27, 2024 at 08:18:14PM +0100 schrieb ralfconn:
> Il 27/03/24 19:58, J. Roeleveld ha scritto:
> > Hi all,
> >
> > I am looking for a way to synchronise a filesystem between 2 servers. Changes
> > can occur on both sides which means I need to have it synchronise in both
> > directions.
> >
> > Does anyone have any thoughts on this?
> >
> > Also, both servers are connected using a slow VPN link, which is why I can't
> > simply access files on the remote server.


+1 for Unison. I’ve been using it for many years now to synchronise between
the four PC systems in my household.

> I use it just for that but can't say anything about the VPN bit, my
> servers are on local network.

Unison creates a local index of all files it syncronised. So when you move a
file around on one end, Unison will notice that because the file at the new
location has the same hash as the file at the old location. As a result, it
does not transmit the file anew to the remote host, but instead copies it
locally on the remote host.

Since Unison uses ssh underneath, you can use ssh’s transparent compression
to speed up the transfer.

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

If all people were the same, one would basically suffice.

On Wed, 2024-03-27 at 19:58 +0100, J. Roeleveld wrote:
> Hi all,
>
> I am looking for a way to synchronise a filesystem between 2 servers.
> Changes can occur on both sides which means I need to have it
> synchronise in both directions.
>
> Does anyone have any thoughts on this?
>
> Also, both servers are connected using a slow VPN link, which is why
> I can't simply access files on the remote server.

I've been using syncthing for years and am extremely pleased with it.
It works so well that I sometimes forget that its there, truly in the
It Just Works category of software.

Am Wed, Mar 27, 2024 at 03:42:07PM -0400 schrieb Matt Connell:
> On Wed, 2024-03-27 at 19:58 +0100, J. Roeleveld wrote:
> > Hi all,
> >
> > I am looking for a way to synchronise a filesystem between 2 servers.
> > Changes can occur on both sides which means I need to have it
> > synchronise in both directions.
> >
> > Does anyone have any thoughts on this?
> >
> > Also, both servers are connected using a slow VPN link, which is why
> > I can't simply access files on the remote server.
>
> I've been using syncthing for years and am extremely pleased with it.
> It works so well that I sometimes forget that its there, truly in the
> It Just Works category of software.

Syncthing is also a good idea. The major difference: syncthing is a
permanently running daemon, so changes are synced very fast (the interval is
configurable, IIRC). OTOH, Unison is run individually by you. That’s why I
prefer the latter: in case I broke some file on my machine, I can get it
back from another machine without having to break out the backup disk (which
may not even have what I need because my backup interval is too big).

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

I have seen things you lusers would not believe.
I've seen Sun monitors on fire off the side of the multimedia lab.
I've seen NTU lights glitter in the dark near the Mail Gate.
All these things will be lost in time, like the root partition last week.
Time to die.

On Wed, 2024-03-27 at 20:54 +0100, Frank Steinmetzger wrote:
> Am Wed, Mar 27, 2024 at 03:42:07PM -0400 schrieb Matt Connell:
> > On Wed, 2024-03-27 at 19:58 +0100, J. Roeleveld wrote:
> > > Hi all,
> > >
> > > I am looking for a way to synchronise a filesystem between 2
> > > servers. Changes can occur on both sides which means I need to
> > > have it synchronise in both directions.
> > >
> > > Does anyone have any thoughts on this?
> > >
> > > Also, both servers are connected using a slow VPN link, which is
> > > why I can't simply access files on the remote server.
> >
> > I've been using syncthing for years and am extremely pleased with
> > it. It works so well that I sometimes forget that its there, truly
> > in the It Just Works category of software.
>
> Syncthing is also a good idea. The major difference: syncthing is a
> permanently running daemon, so changes are synced very fast (the
> interval is configurable, IIRC). OTOH, Unison is run individually by
> you. That’s why I prefer the latter: in case I broke some file on my
> machine, I can get it back from another machine without having to
> break out the backup disk (which may not even have what I need
> because my backup interval is too big).

Good point. I mainly use syncthing as a "stuff I need on multiple
machines" bucket, rather than a big directory of active working files.
Think more along the lines of "I saved this PDF on my PC but I'll need
it on my phone tomorrow so it goes into the Special Folder" kind of use
case. As such it might not be the right fit for "synchronise a file
system" use case.

On 2024-03-27 20:54+0100 Frank Steinmetzger <Warp_7@gmx.de> wrote:

> Am Wed, Mar 27, 2024 at 03:42:07PM -0400 schrieb Matt Connell:
> > On Wed, 2024-03-27 at 19:58 +0100, J. Roeleveld wrote:
> > > Hi all,
> > >
> > > I am looking for a way to synchronise a filesystem between 2
> > > servers. Changes can occur on both sides which means I need to
> > > have it synchronise in both directions.
> > >
> > > Does anyone have any thoughts on this?
> > >
> > > Also, both servers are connected using a slow VPN link, which is
> > > why I can't simply access files on the remote server.
> >
> > I've been using syncthing for years and am extremely pleased with
> > it. It works so well that I sometimes forget that its there, truly
> > in the It Just Works category of software.
>
> Syncthing is also a good idea. The major difference: syncthing is a
> permanently running daemon, so changes are synced very fast (the
> interval is configurable, IIRC). OTOH, Unison is run individually by
> you. That’s why I prefer the latter: in case I broke some file on my
> machine, I can get it back from another machine without having to
> break out the backup disk (which may not even have what I need
> because my backup interval is too big).

syncthing has inbuilt backups if you enable it ????

Am Wed, Mar 27, 2024 at 03:59:18PM -0400 schrieb Matt Connell:

> > Syncthing is also a good idea. The major difference: syncthing is a
> > permanently running daemon, so changes are synced very fast (the
> > interval is configurable, IIRC). OTOH, Unison is run individually by
> > you. That’s why I prefer the latter: in case I broke some file on my
> > machine, I can get it back from another machine without having to
> > break out the backup disk (which may not even have what I need
> > because my backup interval is too big).
>
> Good point. I mainly use syncthing as a "stuff I need on multiple
> machines" bucket, rather than a big directory of active working files.

I sync most of my files on the home and media partition. So all my
documents, photos and music library. I do use syncthing -- between my
android phone and PC. Because I don’t like to fiddle with the filesystem on
the very constrained UI of touch devices.

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

Can you give me a cigarette? Mine are still in the vending machine.

On Wed 27 Mar 2024 20:37:27 GMT, Frank Steinmetzger wrote:
> +1 for Unison. I’ve been using it for many years now to synchronise between
> the four PC systems in my household.
>
> Unison creates a local index of all files it syncronised. So when you move a
> file around on one end, Unison will notice that because the file at the new
> location has the same hash as the file at the old location. As a result, it
> does not transmit the file anew to the remote host, but instead copies it
> locally on the remote host.
>
> Since Unison uses ssh underneath, you can use ssh’s transparent compression
> to speed up the transfer.

I’ve been thinking about using it to synchronise dovecot maildir folders,
since dsync is now deprecated. But I’m not sure about it as I never used
it under “heavy” loads. Do you have any thoughts about it?

--
Alarig

On Wednesday, 27 March 2024 20:08:00 CET Mark Knecht wrote:
> On Wed, Mar 27, 2024 at 11:59?AM J. Roeleveld <joost@antarean.org> wrote:
> > Hi all,
> >
> > I am looking for a way to synchronise a filesystem between 2 servers.
>
> Changes
>
> > can occur on both sides which means I need to have it synchronise in both
> > directions.
> >
> > Does anyone have any thoughts on this?
> >
> > Also, both servers are connected using a slow VPN link, which is why I
>
> can't
>
> > simply access files on the remote server.
> >
> > --
> > Joost
>
> How synchronized? For instance, does it need to handle identicals where
> a file is on both sides but has been moved?

Yes, it needs to handle changes to files/movements/deletions that occur on both
sides. Most of the time, there will only be changes on 1 side, but this is not
guaranteed.
I also need it to identify conflicts and handle this in a way that is
managable. (creating a copy with a different name would be sufficient)

--
Joost

On Wednesday, 27 March 2024 20:37:27 CET Frank Steinmetzger wrote:
> Am Wed, Mar 27, 2024 at 08:18:14PM +0100 schrieb ralfconn:
> > Il 27/03/24 19:58, J. Roeleveld ha scritto:
> > > Hi all,
> > >
> > > I am looking for a way to synchronise a filesystem between 2 servers.
> > > Changes can occur on both sides which means I need to have it
> > > synchronise in both directions.
> > >
> > > Does anyone have any thoughts on this?
> > >
> > > Also, both servers are connected using a slow VPN link, which is why I
> > > can't simply access files on the remote server.
>
> +1 for Unison. I’ve been using it for many years now to synchronise between
> the four PC systems in my household.
>
> > I use it just for that but can't say anything about the VPN bit, my
> > servers are on local network.
>
> Unison creates a local index of all files it syncronised. So when you move a
> file around on one end, Unison will notice that because the file at the new
> location has the same hash as the file at the old location. As a result, it
> does not transmit the file anew to the remote host, but instead copies it
> locally on the remote host.
>
> Since Unison uses ssh underneath, you can use ssh’s transparent compression
> to speed up the transfer.

Unison sounds interesting. How does it handle conflicts (eg, file is changed on
both sides?)

--
Joost

On Wednesday, 27 March 2024 20:54:14 CET Frank Steinmetzger wrote:
> Am Wed, Mar 27, 2024 at 03:42:07PM -0400 schrieb Matt Connell:
> > On Wed, 2024-03-27 at 19:58 +0100, J. Roeleveld wrote:
> > > Hi all,
> > >
> > > I am looking for a way to synchronise a filesystem between 2 servers.
> > > Changes can occur on both sides which means I need to have it
> > > synchronise in both directions.
> > >
> > > Does anyone have any thoughts on this?
> > >
> > > Also, both servers are connected using a slow VPN link, which is why
> > > I can't simply access files on the remote server.
> >
> > I've been using syncthing for years and am extremely pleased with it.
> > It works so well that I sometimes forget that its there, truly in the
> > It Just Works category of software.
>
> Syncthing is also a good idea. The major difference: syncthing is a
> permanently running daemon, so changes are synced very fast (the interval is
> configurable, IIRC). OTOH, Unison is run individually by you. That’s why I
> prefer the latter: in case I broke some file on my machine, I can get it
> back from another machine without having to break out the backup disk
> (which may not even have what I need because my backup interval is too
> big).

I had a quick look and it seems to depend on systems running outside of my
network. Can it be used without any link to a "centralised" server?

--
Joost

On 2024-03-28 07:32+0100 "J. Roeleveld" <joost@antarean.org> wrote:

> On Wednesday, 27 March 2024 20:54:14 CET Frank Steinmetzger wrote:
> > Am Wed, Mar 27, 2024 at 03:42:07PM -0400 schrieb Matt Connell:
> > > On Wed, 2024-03-27 at 19:58 +0100, J. Roeleveld wrote:
> > > > Hi all,
> > > >
> > > > I am looking for a way to synchronise a filesystem between 2
> > > > servers. Changes can occur on both sides which means I need to
> > > > have it synchronise in both directions.
> > > >
> > > > Does anyone have any thoughts on this?
> > > >
> > > > Also, both servers are connected using a slow VPN link, which
> > > > is why I can't simply access files on the remote server.
> > >
> > > I've been using syncthing for years and am extremely pleased with
> > > it. It works so well that I sometimes forget that its there,
> > > truly in the It Just Works category of software.
> >
> > Syncthing is also a good idea. The major difference: syncthing is a
> > permanently running daemon, so changes are synced very fast (the
> > interval is configurable, IIRC). OTOH, Unison is run individually
> > by you. That’s why I prefer the latter: in case I broke some file
> > on my machine, I can get it back from another machine without
> > having to break out the backup disk (which may not even have what I
> > need because my backup interval is too big).
>
> I had a quick look and it seems to depend on systems running outside
> of my network. Can it be used without any link to a "centralised"
> server?

yes, you can disable the external servers. if the systems are running on
the same LAN (same VPN could work too?) they should find each other
nevertheless. or you could host your own discovery/relay server
somewhere.

On 2024-03-27, Mark Knecht <markknecht@gmail.com> wrote:
> On Wed, Mar 27, 2024 at 11:59?AM J. Roeleveld <joost@antarean.org> wrote:

>> I am looking for a way to synchronise a filesystem between 2
>> servers. Changes can occur on both sides which means I need to
>> have it synchronise in both directions.
>
> How synchronized? For instance, does it need to handle identicals where
> a file is on both sides but has been moved?

Does it need to handle the case where the same file is modified
independently on both sides?

On Thursday, 28 March 2024 14:51:42 CET Grant Edwards wrote:
> On 2024-03-27, Mark Knecht <markknecht@gmail.com> wrote:
> > On Wed, Mar 27, 2024 at 11:59?AM J. Roeleveld <joost@antarean.org> wrote:
> >> I am looking for a way to synchronise a filesystem between 2
> >> servers. Changes can occur on both sides which means I need to
> >> have it synchronise in both directions.
> >
> > How synchronized? For instance, does it need to handle identicals where
> > a file is on both sides but has been moved?
>
> Does it need to handle the case where the same file is modified
> independently on both sides?

Yes, as this is something that could happen

Il 28/03/24 07:30, J. Roeleveld ha scritto:
>> Unison creates a local index of all files it syncronised. So when you move a
>> file around on one end, Unison will notice that because the file at the new
>> location has the same hash as the file at the old location. As a result, it
>> does not transmit the file anew to the remote host, but instead copies it
>> locally on the remote host.
>>
>> Since Unison uses ssh underneath, you can use ssh’s transparent compression
>> to speed up the transfer.
> Unison sounds interesting. How does it handle conflicts (eg, file is changed on
> both sides?)
>
I use Unison GUI on one of the two machines (on the other peer it's just
a program invoked from the ssh). When the analysis is complete, the GUI
shows what it would do to sync the machines, indicating the conflicts
and giving you the chance to choose what to do.

I believe it can be used from the command line or maybe even in batch
mode instead of GUI but I never did it that way.

raffaele

On 3/27/24 13:58, J. Roeleveld wrote:
> Hi all,

Hi,

> I am looking for a way to synchronise a filesystem between 2 servers. Changes
> can occur on both sides which means I need to have it synchronise in both
> directions.

What sort of turn around time are you looking for? seconds, minus,
hours, longer?

> Does anyone have any thoughts on this?

I would wonder about using rsync.

host1 -> host2 at the top of the hour
host2 -> host1 at the bottom of the hour

Or if you wanted to get fancy

host1 pushes to host2 at the top of the hour
host2 pushes to host1 at a quarter past
host2 pulls from host1 at the bottom of the hour
host1 pulls from host2 at a quarter till

I'm thinking like if one of them was a road warrior and only one side
could initiate because of a stateful firewall.

> Also, both servers are connected using a slow VPN link, which is why I can't
> simply access files on the remote server.

ACK



--
Grant. . . .

Am Thu, Mar 28, 2024 at 01:08:03AM +0100 schrieb Alarig Le Lay:
> On Wed 27 Mar 2024 20:37:27 GMT, Frank Steinmetzger wrote:
> > +1 for Unison. I’ve been using it for many years now to synchronise between
> > the four PC systems in my household.
> >
> > Unison creates a local index of all files it syncronised. So when you move a
> > file around on one end, Unison will notice that because the file at the new
> > location has the same hash as the file at the old location. As a result, it
> > does not transmit the file anew to the remote host, but instead copies it
> > locally on the remote host.
> >
> > Since Unison uses ssh underneath, you can use ssh’s transparent compression
> > to speed up the transfer.
>
> I’ve been thinking about using it to synchronise dovecot maildir folders,
> since dsync is now deprecated. But I’m not sure about it as I never used
> it under “heavy” loads. Do you have any thoughts about it?

Among the many files I sync is my Mail folder. It currently contains around
106k files. My music collection is another 50k files. If there are no
changes and I did a recent run, then another run to look for changes takes
only very few seconds.

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

A computer only crashes if your haven’t saved your text for a long time.

Am Thu, Mar 28, 2024 at 05:33:43PM +0100 schrieb ralfconn:
> Il 28/03/24 07:30, J. Roeleveld ha scritto:
> > > Unison creates a local index of all files it syncronised. So when you move a
> > > file around on one end, Unison will notice that because the file at the new
> > > location has the same hash as the file at the old location. As a result, it
> > > does not transmit the file anew to the remote host, but instead copies it
> > > locally on the remote host.
> > >
> > > Since Unison uses ssh underneath, you can use ssh’s transparent compression
> > > to speed up the transfer.
> > Unison sounds interesting. How does it handle conflicts (eg, file is changed on
> > both sides?)
> >
> I use Unison GUI on one of the two machines (on the other peer it's just a
> program invoked from the ssh). When the analysis is complete, the GUI shows
> what it would do to sync the machines, indicating the conflicts and giving
> you the chance to choose what to do.
>
> I believe it can be used from the command line or maybe even in batch mode
> instead of GUI but I never did it that way.

You can set up a merge command to solve conflicts on the cmdline, such as
vimdiff. But when I set that, it blocks the GUI. Maybe I did something wrong
with the setup. Anyways, when I get a conflict, I make a backup of the file
locally, overwrite it with the remote and then do a conflict resolution with
vim.

In my every-day workflow, I usually only get conflicts in text files (logs,
notes, and so on). Binary conflicts are rare and usually due to recent
actions, such as editing an image or music file. In that case I can decide
on a per-case-basis which version to keep.

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

How does the Heisenberg compensator work? – Thank you, fine.

Am Thu, Mar 28, 2024 at 09:26:33PM -0500 schrieb Grant Taylor:
> On 3/27/24 13:58, J. Roeleveld wrote:
> > Hi all,
>
> Hi,
>
> > I am looking for a way to synchronise a filesystem between 2 servers. Changes
> > can occur on both sides which means I need to have it synchronise in both
> > directions.
>
> What sort of turn around time are you looking for? seconds, minus, hours,
> longer?
>
> > Does anyone have any thoughts on this?
>
> I would wonder about using rsync.

Rsync can’t handle file moves. Given:

> > Also, both servers are connected using a slow VPN link, which is why I can't
> > simply access files on the remote server.

it would be beneficial to conserve traffic as much as possible.

While you can use the -u flag to only overwrite if the source is newer than
the destination, AFAIK rsync can’t detect if the destination has also been
altered since the last sync, so it might clobber important changes. That’s
why sync tools use a metadata cache to remember last edit timestamps.

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

In the 60's people took acid to make the world weird.
Now the world is weird and people take Prozac to make it normal.

On Wednesday, 27 March 2024 19:58:47 CET J. Roeleveld wrote:
> Hi all,
>
> I am looking for a way to synchronise a filesystem between 2 servers.
> Changes can occur on both sides which means I need to have it synchronise
> in both directions.
>
> Does anyone have any thoughts on this?
>
> Also, both servers are connected using a slow VPN link, which is why I can't
> simply access files on the remote server.
>
> --
> Joost

All,

Many thanks to everyone. I will be investigating and testing "net-misc/unison"
and "net-p2p/syncthing" to see which fits my requirement best.

Thanks again,

Joost