Mailing List Archive

Howto log multiple sftpd instances with their chroot shared via NFS
OpenSSH 5.9p1 + 7.6p1

syslog-ng 3.3.4 + 3.13.2

Hello, having an Ubuntu server with sftpd running where /var/data/chroot/ is an NFS mount from a remote central NFS server,
and each sftpd user's chroot home is /var/data/chroot/<username>/
and every user has a log device /var/data/chroot/<username>/dev/log which I read in successfully with syslog-ng:

source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
destination d_sftp_<username> { file("/var/log/sftp/<username>.log"); };
log { source(s_chroot_<username>); destination(d_sftp_<username>); };

Now I have a second sftpd server in parallel, with the same user database and also mounts /var/data/chroot/ via NFS, and has the same syslog-ng config,
so every user can login on the one server or on the other. This is for high availability. This works so far.

What is not working now is the sftpd logging: The sftp user's log is only available on one sftp server exclusively, and that is the one where syslog-ng was started least,
because as I understand it takes the exclusive unix socket file lock for each user's /dev/log.

So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.

If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.

How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?

Thanks

Crossposting (because unanswered)
https://unix.stackexchange.com/questions/666641/howto-log-multiple-sftpd-servers-activity-which-users-chrooted-home-is-on-shar
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On Tue, 21 Sep 2021, Hildegard Meier wrote:

> OpenSSH 5.9p1 + 7.6p1
>
> syslog-ng 3.3.4 + 3.13.2
>
> Hello, having an Ubuntu server with sftpd running where /var/data/chroot/ is an NFS mount from a remote central NFS server,
> and each sftpd user's chroot home is /var/data/chroot/<username>/
> and every user has a log device /var/data/chroot/<username>/dev/log which I read in successfully with syslog-ng:
>
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> destination d_sftp_<username> { file("/var/log/sftp/<username>.log"); };
> log { source(s_chroot_<username>); destination(d_sftp_<username>); };
>
> Now I have a second sftpd server in parallel, with the same user database and also mounts /var/data/chroot/ via NFS, and has the same syslog-ng config,
> so every user can login on the one server or on the other. This is for high availability. This works so far.
>
> What is not working now is the sftpd logging: The sftp user's log is only available on one sftp server exclusively, and that is the one where syslog-ng was started least,
> because as I understand it takes the exclusive unix socket file lock for each user's /dev/log.
>
> So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
>
> If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.
>
> How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?

Right now there is no solution for this inside OpenSSH. There have been
some proposals for post-auth logging to be proxied via the priviledged
sshd monitor process but we haven't pursued them yet.

Maybe someone with more Linux/NFS wit could suggest an OS-side solution
for you?

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
Hi Hildegard,

On Tue, 21 Sep 2021, Hildegard Meier wrote:
> Now I have a second sftpd server in parallel, with the same user
> database and also mounts /var/data/chroot/ via NFS, and has the same
> syslog-ng config,
> so every user can login on the one server or on the other. This is for
> high availability. This works so far.
>
> What is not working now is the sftpd logging: The sftp user's log is
> only available on one sftp server exclusively, and that is the one
> where syslog-ng was started least, because as I understand it takes
> the exclusive unix socket file lock for each user's /dev/log.
>
> So, if a user logs in on the first server, where syslog-ng was started
> least, the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is
> not logged, neither on the second nor on the first server.

Forward the log entries on both machines to a log host.? E.g.

destination d_tcp {
network("log_host" port(1999));
};

Regards,

David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On Tue, 21 Sept 2021 at 19:58, Hildegard Meier <daku8938@gmx.de> wrote:
[...]

> Now I have a second sftpd server in parallel, with the same user database
> and also mounts /var/data/chroot/ via NFS

[...]

> How can I get the sftp user's activity be logged on each sftp server, when
> a user logs in to that server, while the user's home is shared on both
> servers via NFS?
>

Mount a tiny local ramfs or tmpfs over /var/data/chroot/dev?

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On 22.09.21 11:18, David Newall wrote:
> On Tue, 21 Sep 2021, Hildegard Meier wrote:
>> So, if a user logs in on the first server, where syslog-ng was started
>> least, the user's sftp activity is logged on the first server.
>> But if the user logs in on the second server, it's sftp activity is
>> not logged, neither on the second nor on the first server.
>
> Forward the log entries on both machines to a log host.

Considering that server B is not logging *at all* right now, I doubt
that it'll have anything to forward to a log host, either.

The problem *presumably* is that the syslogd on server A has put some
sort of file lock on the device that propagates through the NFS server
and interferes with syslogd on server B using it.

One solution might be to reconfigure the syslogd's to use a method of
locking that does *not* propagate through NFS. I'm afraid I don't know
syslog-ng well enough to advise on that.

Then there's the possibility of reconfiguring *NFS* to stop the
forwarding, but "breaking" file locking on NFS is, of course, a can of
worms of possible side effects ...

(Bind) mounting a local .../dev over the NFS-shared chroot dirtree ...
ought to work, but complicates unmounting/remounting, which was already
enough of a hair-puller in failure scenarios when I last worked with NFS.

What do the chrooted users have for a homedir *within* the chroot? Would
it be possible to have /var/data/chroot be a local FS and mount only
/var/data/chroot/home from the NFS server? (If there are files that you
need to keep identical on both servers, e.g., under
/var/data/chroot/etc, you can still symlink those to some special subdir
like /var/data/chroot/home/ETC to put the actual data onto the NFS share.)

Regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
Aw: Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On 2021/09/22 13:06, Jochen Bern wrote:
> What do the chrooted users have for a homedir *within* the chroot? Would it
> be possible to have /var/data/chroot be a local FS and mount only
> /var/data/chroot/home from the NFS server? (If there are files that you need
> to keep identical on both servers, e.g., under /var/data/chroot/etc, you can
> still symlink those to some special subdir like /var/data/chroot/home/ETC to
> put the actual data onto the NFS share.)

The description was for /var/data/chroot/<username>/dev/log i.e. each
user has their own separate chroot. So this type of approach would
require mounting a local fs of some sort over the top of each user's dir
which soon gets messy.

This is amongst the reasons why OpenBSD has the sendsyslog(2) syscall,
https://man.openbsd.org/sendsyslog.2 - the syslog daemon opens a
kernel socket to receive those messages, and processes which want to
write a log entry just call the standard syslog functions which use
sendsyslog(2) rather than /dev/log, so it works even through FD
exhaustion, in chroot, and with syscall filters that prohibit
filesystem access.

Not entirely pleasant but I suppose it could alternatively be done by
using a LD_PRELOAD wrapper to override syslog functions (I think just
syslog_r is probably enough for openssh) and have them send over a
network socket instead.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On Fri, 24 Sep 2021, Stuart Henderson wrote:

> This is amongst the reasons why OpenBSD has the sendsyslog(2) syscall,
> https://man.openbsd.org/sendsyslog.2 - the syslog daemon opens a
> kernel socket to receive those messages, and processes which want to
> write a log entry just call the standard syslog functions which use

Oh, nice.

> The description was for /var/data/chroot/<username>/dev/log i.e. each
> user has their own separate chroot. So this type of approach would
> require mounting a local fs of some sort over the top of each user's dir

This made me curious, and I tried¹ this. It is possible to bind-mount
sockets on Linux iff the target exists as regular file.

sudo touch /var/data/chroot/<username>/dev/log # but beware of
# filesystem-based
# attacks here!
sudo mount --bind /dev/log /var/data/chroot/<username>/dev/log

? I went and began using this technology here:
https://github.com/mirabilos/shellsnippets/blob/master/posix/debchroot.sh

bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

****************************************************
/?\ The UTF-8 Ribbon
? ? Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
? HTML eMail! Also, https://www.tarent.de/newsletter
? ? header encryption!
****************************************************
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On 9/21/2021 4:49 AM, Hildegard Meier wrote:

>
> How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?

based on:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/88

This might work for a few sftp servers.
On first server, hostname1, use /dev/<hostname1>/log for all its default logging and its normal syslog-ng.conf and for the special
syslog-ng.conf:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
(May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)

Do the above for second server, hostname2, use /dev/<hostname2>/log

Then add these to the chroot directories in NFS:
/var/data/chroot/<username>/dev/<hostname1>/log
/var/data/chroot/<username>/dev/<hostname2>/log

This should cause hostname1 to lock on /var/data/chroot/<username>/dev/<hostname1>/log
and hostname 2 to lock on /var/data/chroot/<username>/dev/<hostname2>/log
because the syslog-ng does not really look at the syslog-ng.conf in the chroot.


>
> Thanks
>
> Crossposting (because unanswered)
> https://unix.stackexchange.com/questions/666641/howto-log-multiple-sftpd-servers-activity-which-users-chrooted-home-is-on-shar
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Aw: Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
Many thanks for your answers!
 
More ideas appreciated, since it would be relly important for us to have a solution for that.
 
 
>Date: Wed, 22 Sep 2021 13:06:43 +0200
>From: Jochen Bern <Jochen.Bern@binect.de>
>To: openssh-unix-dev@mindrot.org
>Subject: Re: Howto log multiple sftpd instances with their chroot
>shared via NFS
On 22.09.21 11:18, David Newall wrote:
> On Tue, 21 Sep 2021, Hildegard Meier wrote:
>> So, if a user logs in on the first server, where syslog-ng was started
>> least, the user's sftp activity is logged on the first server.
>> But if the user logs in on the second server, it's sftp activity is
>> not logged, neither on the second nor on the first server.
>
> Forward the log entries on both machines to a log host.
>Considering that server B is not logging *at all* right now, I doubt
>that it'll have anything to forward to a log host, either.
 
Yes the log destination is not the problem, the empty log source is.


>The problem *presumably* is that the syslogd on server A has put some
>sort of file lock on the device that propagates through the NFS server
>and interferes with syslogd on server B using it.
 
As I understand, syslog-ng creates /dev/log in the user's chroot directories as Unix stream socket (see https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/28#TOPIC-1209171).
 
This seems also to be called IPC socket (inter-process communication socket)   or AF_UNIX socket.
 
"It is used in POSIX operating systems for inter-process communication. The correct standard POSIX term is POSIX Local IPC Sockets. Unix domain connections appear as byte streams, much like network connections, but all data _remains within the local computer_."
 
"It means that if you create a AF_UNIX socket on a NFS disk which is shared between two machines A and B, you cannot have a process on A writing data to the unix socket and a process on B reading data from that socket.
The communication happens at kernel level, and you can only transfer data among processes sitting in the same kernel."
(see source: https://stackoverflow.com/questions/783134/af-unix-domain-why-use-local-file-names-only )


>One solution might be to reconfigure the syslogd's to use a method of
>locking that does *not* propagate through NFS. I'm afraid I don't know
>syslog-ng well enough to advise on that.
 
I think the problem is that the chrooted sftp users only have write access (for logging) under their chroot home dir (and there /devlog) , and since that is on NFS, how could one escape that. A possibility would be a symlink to a local unix stream device, but chroot does not allow to access out of it, that's the whole point of being chrooted (jailed).


>Then there's the possibility of reconfiguring *NFS* to stop the
>forwarding, but "breaking" file locking on NFS is, of course, a can of
>worms of possible side effects ...
Do not know what you exactly mean, but I think we cannot fiddle with NFS file locking since we have several hundrets of sftp customer accounts on the NFS chroot, which transfer many many files.


>(Bind) mounting a local .../dev over the NFS-shared chroot dirtree ...
>ought to work, but complicates unmounting/remounting, which was already
>enough of a hair-puller in failure scenarios when I last worked with NFS.
man bind says: "mount --bind olddir newdir"
/dev/log is not a directory, but created by syslog-ng as a unix stream socket (see above). I guess if it would a directory, the sftpd process could not log to it?
# mkdir -p /my/local/dir/
# mount -v --bind /var/data/chroot/test/dev/log /my/local/dir
mount: /my/local/dir: mount(2) system call failed: Not a directory.


>What do the chrooted users have for a homedir *within* the chroot? Would
>it be possible to have /var/data/chroot be a local FS and mount only
>/var/data/chroot/home from the NFS server? (If there are files that you
>need to keep identical on both servers, e.g., under
>/var/data/chroot/etc, you can still symlink those to some special subdir
>like /var/data/chroot/home/ETC to put the actual data onto the NFS share.)
 
Strict technically, we really _need_ to share every sftp user's chrooted upload directory (/var/data/chroot/<username>/in/) and download directory (/var/data/chroot/<username>/out/), since we have a single backend process that reads all files that the customers have uploaded into their chroot /in/ directory (be it that they have logged in to the one sftp server or the other, both sftp servers are TCP load balanced, but the time order of the uploads needs to be honoured), and writes files to the customers chroot /out/ dir (and the customer can download the files from it's chrooted /out/ dir whether having sftp logged in to the one sftp server or the other).
But since we have 800 sftp customers, so 800 /var/data/chroot/<username>/in/ and 800 /var/data/chroot/<username>/out/ it is not feasable to have 1600 single NFS mounts, to achieve that /var/data/chroot/<username>/dev/log is local. That is the problem.
 
 
>Mount a tiny local ramfs or tmpfs over /var/data/chroot/dev?
 
For the same reason, since we have so many /var/data/chroot/<username>/dev/log , this would not be feasable (apart of the question if this would technically be a solution).
 
We need also a single log source for each every sftp user, because we need a single log file for every single sftp user, to be able to view each sftp user's sftp activity.
 
Best regards
Hildegard
 
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Aw: Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
Thanks, this sounds interesting and maybe in this direction could be the solution? But I do not understand you precisely.

> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
You mean there should be
/var/data/chroot/<username>/dev/hostname1/log
/var/data/chroot/<username>/dev/hostname2/log

?

> (May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
You mean there should be the Symlink
/var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname1/log

?

> Do the above for second server, hostname2, use /dev/<hostname2>/log

It would not be possible to have a Symlink
/var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname2/log

since /var/data/chroot/<username>/dev/log is already a Symlink to /var/data/chroot/<username>/dev/hostname1/log
(see first step), and it is the same NFS filesystem.



> Gesendet: Freitag, 24. September 2021 um 16:08 Uhr
> Von: "Douglas E Engert" <deengert@gmail.com>
> An: openssh-unix-dev@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> On 9/21/2021 4:49 AM, Hildegard Meier wrote:
>
> >
> > How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
>
> based on:
> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/88
>
> This might work for a few sftp servers.
> On first server, hostname1, use /dev/<hostname1>/log for all its default logging and its normal syslog-ng.conf and for the special
> syslog-ng.conf:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
> (May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
>
> Do the above for second server, hostname2, use /dev/<hostname2>/log
>
> Then add these to the chroot directories in NFS:
> /var/data/chroot/<username>/dev/<hostname1>/log
> /var/data/chroot/<username>/dev/<hostname2>/log
>
> This should cause hostname1 to lock on /var/data/chroot/<username>/dev/<hostname1>/log
> and hostname 2 to lock on /var/data/chroot/<username>/dev/<hostname2>/log
> because the syslog-ng does not really look at the syslog-ng.conf in the chroot.
>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Aw: Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On Fri, 24 Sep 2021, Hildegard Meier wrote:

> It would not be possible to have a Symlink
> /var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname2/log
>
> since /var/data/chroot/<username>/dev/log is already a Symlink to
> /var/data/chroot/<username>/dev/hostname1/log (see first step), and it
> is the same NFS filesystem.

Not sure I understand the entire situation, but can’t you make
/var/data/chroot/<username>/dev/log a regular file (on the NFS
mount) then bind-mount (see my earlier mail) /dev/log from the
host over it?

bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

****************************************************
/?\ The UTF-8 Ribbon
? ? Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
? HTML eMail! Also, https://www.tarent.de/newsletter
? ? header encryption!
****************************************************
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Aw: Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
Thanks, this sounds like the solution could be in this direction.
I think, the sftpd process should just not write to the /dev/log unix socket (because this leads to the problem here), but to the local kernel directly, something like what you describe here.

But how could I do this concrete with Ubuntu Linux? What you write is rather abstract and I am not so expert that I understand what you mean with LD_PRELOAD wrapper.

Unfortunately, I could not change our sftp server to OpenBSD operating system since we would not have the capacity to maintain this one special operating system. We maintain our 350 Ubuntu Linux servers with already established processes.



> This is amongst the reasons why OpenBSD has the sendsyslog(2) syscall,
> https://man.openbsd.org/sendsyslog.2 - the syslog daemon opens a
> kernel socket to receive those messages, and processes which want to
> write a log entry just call the standard syslog functions which use
> sendsyslog(2) rather than /dev/log, so it works even through FD
> exhaustion, in chroot, and with syscall filters that prohibit
> filesystem access.
>
> Not entirely pleasant but I suppose it could alternatively be done by
> using a LD_PRELOAD wrapper to override syslog functions (I think just
> syslog_r is probably enough for openssh) and have them send over a
> network socket instead.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Aw: Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On 9/24/2021 10:03 AM, Hildegard Meier wrote:
> Thanks, this sounds interesting and maybe in this direction could be the solution? But I do not understand you precisely.
>
>> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
> You mean there should be
> /var/data/chroot/<username>/dev/hostname1/log
> /var/data/chroot/<username>/dev/hostname2/log
>

The point is each application's syslog library will write to /dev/hostnameX/log or if running in chroot
var/data/chroot/<username>/dev/hostnameX/log

Thus each sever has it own lock for each user.


So any lock done while application is running in chroot will lock will lock on different file.


>
>> (May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
> You mean there should be the Symlink
> /var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname1/log
>

Symlink may not be needed. I was refereing to a symlink when not using the chroot.

>
>> Do the above for second server, hostname2, use /dev/<hostname2>/log
>
> It would not be possible to have a Symlink
> /var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname2/log
>
> since /var/data/chroot/<username>/dev/log is already a Symlink to /var/data/chroot/<username>/dev/hostname1/log
> (see first step), and it is the same NFS filesystem.
>
>
>
>> Gesendet: Freitag, 24. September 2021 um 16:08 Uhr
>> Von: "Douglas E Engert" <deengert@gmail.com>
>> An: openssh-unix-dev@mindrot.org
>> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>>
>> On 9/21/2021 4:49 AM, Hildegard Meier wrote:
>>
>>>
>>> How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
>>
>> based on:
>> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/88
>>
>> This might work for a few sftp servers.
>> On first server, hostname1, use /dev/<hostname1>/log for all its default logging and its normal syslog-ng.conf and for the special
>> syslog-ng.conf:
>> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
>> (May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
>>
>> Do the above for second server, hostname2, use /dev/<hostname2>/log
>>
>> Then add these to the chroot directories in NFS:
>> /var/data/chroot/<username>/dev/<hostname1>/log
>> /var/data/chroot/<username>/dev/<hostname2>/log
>>
>> This should cause hostname1 to lock on /var/data/chroot/<username>/dev/<hostname1>/log
>> and hostname 2 to lock on /var/data/chroot/<username>/dev/<hostname2>/log
>> because the syslog-ng does not really look at the syslog-ng.conf in the chroot.
>>
>
> .
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Aw: Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On 9/24/2021 10:11 AM, Thorsten Glaser wrote:
> On Fri, 24 Sep 2021, Hildegard Meier wrote:
>
>> It would not be possible to have a Symlink
>> /var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname2/log

Correct that is not needed.
It looks like the server has a line:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };

It is not clear if this is created ahead of time or on the fly when the NFS volume is mounted
or for each possible user to create a unix-stream socket in NFS.


What I was proposing is each of the two servers has it own /var/data/chroot/<username>/dev/hostnameX.log
What is not clear is when running in chroot, when an application does
an openlog, syslog or vsyslog are these system calls, or library calls.
i.e. do the read any configuration from the chroot directory?

I think it expects to find /dev/log in the chroot directory and this name
matches what is in the real root directory. So I was proposing
that each sftpd server has a different /dev/hostname.log so a user
could be on both of the servers at the same time.

Hildegard said these are Ubuntu servers, and Ubuntu uses rsyslogd that looks
similar to syslog-ng.

Google'ing for: rsyslog chroot /dev/log

These look interesting.

https://www.the-art-of-web.com/system/sftp-logging-chroot/
https://serverfault.com/questions/878627/logging-chrooted-sftp-user-activity



>>
>> since /var/data/chroot/<username>/dev/log is already a Symlink to
>> /var/data/chroot/<username>/dev/hostname1/log (see first step), and it
>> is the same NFS filesystem.
>
> Not sure I understand the entire situation, but can’t you make
> /var/data/chroot/<username>/dev/log a regular file (on the NFS
> mount) then bind-mount (see my earlier mail) /dev/log from the
> host over it?

Is this what you are trying to do?
https://serverfault.com/questions/878627/logging-chrooted-sftp-user-activity

Instead, use input(type="imuxsock" Socket="/opt/bindmount/dev/log") to create an additional
socket (which you can then bind-mount such that all chroots see it at /dev/log).

>
> bye,
> //mirabilos
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Aw: Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On Fri, 24 Sep 2021, Douglas E Engert wrote:

> Instead, use input(type="imuxsock" Socket="/opt/bindmount/dev/log") to create
> an additional
> socket (which you can then bind-mount such that all chroots see it at
> /dev/log).

Wasn’t it said earlier in the thread that this doesn’t scale across
multiple chroots, especially when they may be added and removed dy?
namically? Restarting the syslog daemon all the time isn’t going to
fly… and this can avoid that.

bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

****************************************************
/?\ The UTF-8 Ribbon
? ? Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
? HTML eMail! Also, https://www.tarent.de/newsletter
? ? header encryption!
****************************************************
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
Hi Hildegard,

Hildegard Meier wrote:
> I think, the sftpd process should just not write to the /dev/log
> unix socket (because this leads to the problem here), but to the
> local kernel directly, something like what you describe here.

Exactly, but unfortunately the code that sets the socket path is in
the system C library, without any interface to control the setting.


> But how could I do this concrete with Ubuntu Linux? What you write
> is rather abstract and I am not so expert that I understand what
> you mean with LD_PRELOAD wrapper.

Setting the LD_PRELOAD environment variable to a library filename when
starting a program loads that library into memory before the system C
library and the program itself. Symbols (ie. functions) loaded from
the preloaded library are not replaced by later (system) libraries,
making it possible to override and/or wrap libc functionality in this
fairly brutish way, hence a bit unpleasant.

Attached is source code for a simple such wrapper overriding openlog(),
syslog() and closelog(). It might work only on the older system, or on
both. You only need this hack on one server, the other can use /dev/log.

The wrapper directs logging to /dev/log_<hostname> instead of /dev/log.
Since libc has no interface to manipulate the socket path these functions
override the normal functions with a very limited implementation.
Adjust the template on line 49 if you want to change the destinateion.

Although functionality is limited it could work well for sftp-server.

One simplification is that this openlog() ignores all option flags
except LOG_NDELAY and that syslog() always behaves as if openlog() was
called with the LOG_PID option. Since this is exactly what OpenSSH log.c
does all log messages should look the same as with the libc implementation.

Note that log messages are silently lost on any error, e.g. syslog-ng
not running. Also note that if log messages, the openlog() ident
parameter or the username (when ident == NULL) are very long then
messages are silently truncated to 2048 chars. OpenSSH sends max 500
chars.


You didn't mention if your two sshd are configured to use the in-process
sftp-server (Subsystem sftp internal-sftp in sshd_config) - if so then
this is probably not really the best solution, since LD_PRELOAD would
have to be set for the entire sshd process, which I don't recommend
because this code is so simplistic compared to libc.

Anyway, try it if you like.


The comment near the top of the file shows how to compile. sshd can't
be configured to set LD_PRELOAD so to use the compiled .so with
sftp-server you have to create a simple script:

--8<-- /usr/local/libexec/sftp-server-hostlog
#!/bin/sh
export LD_PRELOAD=/usr/local/libexec/hostlog.so
exec /usr/lib/sftp-server "$*"
-->8--

and then configure sshd to call that instead of sftp-server:

--8<-- /etc/ssh/sshd_config
Subsystem sftp /usr/local/libexec/sftp-server-hostlog
-->8--

Adjust paths as needed, also in the compile command.


For anyone else interested please go ahead but note that I haven't
put huge effort into portability; it might work outside Linux but
I haven't tested anywhere else so YMMV.


Kind regards

//Peter
Re: Aw: Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On 24.09.21 16:37, Hildegard Meier wrote:
> More ideas appreciated, since it would be relly important for us
> to have a solution for that.

OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
have no idea how portable it is to other distribs, so just give it a try:

> # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
> Subsystem sftp internal-sftp
> Match group mandanten
> ForceCommand internal-sftp -l INFO -u 0077


> # ls -al ~lvinq4/dev
> insgesamt 0
> drwx--x---. 2 root mandanten 6 20. Mai 17:25 .
> drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..

As you can see, the chroots have an *empty* /dev subdir, but logging
*still works*, apparently because the chrooted process just keeps using
the system-central /dev/log it opened *before* chroot()ing.

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
Jochen Bern wrote:
> OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
> have no idea how portable it is to other distribs, so just give it a try:
>
> > # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
> > Subsystem sftp internal-sftp
> > Match group mandanten
> > ForceCommand internal-sftp -l INFO -u 0077
>
>
> > # ls -al ~lvinq4/dev
> > insgesamt 0
> > drwx--x---. 2 root mandanten 6 20. Mai 17:25 .
> > drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
>
> As you can see, the chroots have an *empty* /dev subdir, but logging
> *still works*, apparently because the chrooted process just keeps using
> the system-central /dev/log it opened *before* chroot()ing.

I think that works specifically because *no* new process is created
when using internal-sftp as opposed to executing the sftp-server binary.

If syslog-ng can be made to reliably route internal-sftp messages to
user-specific log files then this approach would work well in Hildegard's
setup and would be a lot more pleasant than messing with LD_PRELOAD.


//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On 9/26/2021 3:20 PM, Peter Stuge wrote:
> Jochen Bern wrote:
>> OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
>> have no idea how portable it is to other distribs, so just give it a try:
>>
>>> # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
>>> Subsystem sftp internal-sftp
>>> Match group mandanten
>>> ForceCommand internal-sftp -l INFO -u 0077
>>
>>
>>> # ls -al ~lvinq4/dev
>>> insgesamt 0
>>> drwx--x---. 2 root mandanten 6 20. Mai 17:25 .
>>> drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
>>
>> As you can see, the chroots have an *empty* /dev subdir, but logging
>> *still works*, apparently because the chrooted process just keeps using
>> the system-central /dev/log it opened *before* chroot()ing.
>
> I think that works specifically because *no* new process is created
> when using internal-sftp as opposed to executing the sftp-server binary.
>

It is not clear from Hildegard's email what is in the user's home directory
and if the sftp server can run in interactive mode. i.e. if a shell is started,
and if it can run any commands that may require logging.

OpenSSH sshd_config man page discusses ChrootDirectory says: "For an interactive session this requires at least a shell"
and also says: "sessions which use logging may require /dev/log inside the chroot directory on some operating systems."

So I would say the /dev/log or logs are required.

Based on the problem in original note:

> So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
>
> If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.

Logging works on the server that started syslog-ng. It sounds like syslog-ng opens a "unix-stream" socket.
She also says: "for every user has a log device /var/data/chroot/<username>/dev/log"
(not sure what it changes, it may save the socket handle/index (or whatever it is) to be used in that server's kernel.)

Each server is overwriting the /dev/log in the chroot directories.

So if interactive is not required, then no chroot /dev/log would be needed and sftp-initeral could work.
But it sounds like there is a need to share these NFS volumes

So we are back to using bind-mounting as it is done in the kernel and NFS version of /dev/log is not touched.

OR

Each server has its own /dev/log_<hostname> and the syslog-ng files on each server do the same
and use LD_PRELOAD= as proposed by Peter Stuge. I have used the LD_PRELOAD before when running
valgrind and/or gdb to not unload modules so as the debug tables stay available,


> If syslog-ng can be made to reliably route internal-sftp messages to
> user-specific log files then this approach would work well in Hildegard's
> setup and would be a lot more pleasant than messing with LD_PRELOAD.

As noted above, depends if any other process is stared that does logging is started after the
change to chroot.

>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
Hallo all, thank You all very much for your answers and suggestions.
Since the discussion has been shared between several mails now, I would like to try to summarize all up in one mail again here
and fill in the informations that where missing before.

We have 800 (eight hundred) sftp customers, each sftp customer has the same simple local Linux account on both sftp servers (simple entry in /etc/passwd and /etc/shadow etc.).
(Before we had only one sftp server, but for higher availability we want now to run two or more sftp servers parallel, accessed via TCP (sftp) load balancer)

Each customer account is in the group "sftp-cust", and only members of that group are allowed to login via sftp.
Each customer has it's chrooted home dir in /var/data/chroot/<username>/.

Here the (relevant part) of the sftpd config:

---------------------------------------------------------------
AllowGroups sftp-cust
Subsystem sftp internal-sftp -f LOCAL5 -l INFO
Match Group sftp-cust
ChrootDirectory %h
ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
AllowTcpForwarding no
---------------------------------------------------------------

Each sftp customer's sftp activity needs to be available in a dedicated log file for each sftp customer, for our support to be able to look into it. Here is an example of this log file:

Sep 28 15:54:25 myhostname internal-sftp[1618]: session opened for local user <username> from [1.2.3.4]
Sep 28 15:55:52 myhostname internal-sftp[27918]: remove name "/in/file.dat"
Sep 28 15:55:52 myhostname internal-sftp[27918]: sent status No such file
Sep 28 15:55:52 myhostname internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE mode 0666
Sep 28 15:55:52 myhostname internal-sftp[27918]: close "/in/file.dat" bytes read 0 written 2966
Sep 28 15:55:52 myhostname internal-sftp[27918]: set "/in/file.dat" modtime 20210928-15:55:46

If one does not use the /dev/log in the chroot environment (that is /var/data/chroot/<username>/dev/log absolute), you have a global sftpd log (I think in /var/log/messages on the server or something like that).
So a solution for the log problem would be, to not use the chroot logging, but to parse the global sftpd log (which is available on both sftpd servers locally for all local logins separated).
But I think this is not trivial to make this reliable and robust, since we need to parse the process id (sftpd session), check which username that session belongs to and write that log lines in the username specific log file.
Of course, process IDs are reused for different sessions, we have overlapping sessions etc. But if there is no other solution, than we need to try that. Maybe someone has already written such a session tracking parser.
Does somebody know of a log analyzer that can do this? A problem would be e.g. if the user never logs out, and the log file rotates daily, so after one day, the parser could not find any "session opened for local user xxx" log lines anymore.

If we want to try to keep using the available session chroot logging, man (8) sftp-server says:

"For logging to work, sftp-server must be able to access /dev/log. Use of sftp-server in a chroot configuration therefore requires that
syslogd(8) establish a logging socket inside the chroot directory."

As Peter has written here
https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html

as I understand it is hard coded in the system C library that the log devices name is /dev/log, so this cannot be changed (e.g. to log to chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2 on hostname2).

With the syslog-ng we use, we only need to create the directory /var/data/chroot/<username>/dev for each user once we create a new account.

We have the following syslog-ng config snippet file for each sftp user:

source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
destination d_sftp_<username> { file("/var/log/app/sftp/<username>.log"); };
log { source(s_chroot_<username>); destination(d_sftp_<username>); };

When starting syslog-ng the unix stream file /var/data/chroot/<username>/dev/log is created automatically by syslog-ng. If you delete it it gets recreated upon syslog-ng restart.

But the problem is that the last started syslog-ng aquires the lock for the NFS shared /var/data/chroot/<username>/dev/log so the other server cannot read it anymore and so there is no sftp log for the sessions on the other server.

I guess this is because syslog-ng creates /dev/log in the user's chroot directories as Unix stream socket (see https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/28#TOPIC-1209171).
This seems also to be called IPC socket (inter-process communication socket) or AF_UNIX socket.
"It is used in POSIX operating systems for inter-process communication. The correct standard POSIX term is POSIX Local IPC Sockets. Unix domain connections appear as byte streams, much like network connections, but all data _remains within the local computer_."

"It means that if you create a AF_UNIX socket on a NFS disk which is shared between two machines A and B, you cannot have a process on A writing data to the unix socket and a process on B reading data from that socket.
The communication happens at kernel level, and you can only transfer data among processes sitting in the same kernel."
(see source: https://stackoverflow.com/questions/783134/af-unix-domain-why-use-local-file-names-only )

Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share.

We need to stick with Ubuntu Linux as we have established management process only for this operating systems.

I hope I did not forget some information that was missing before. Thanks Hildegard



> Gesendet: Mittwoch, 22. September 2021 um 09:19 Uhr
> Von: "Damien Miller" <djm@mindrot.org>
> An: "Hildegard Meier" <daku8938@gmx.de>
> Cc: openssh-unix-dev@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> On Tue, 21 Sep 2021, Hildegard Meier wrote:
>
> > OpenSSH 5.9p1 + 7.6p1
> >
> > syslog-ng 3.3.4 + 3.13.2
> >
> > Hello, having an Ubuntu server with sftpd running where /var/data/chroot/ is an NFS mount from a remote central NFS server,
> > and each sftpd user's chroot home is /var/data/chroot/<username>/
> > and every user has a log device /var/data/chroot/<username>/dev/log which I read in successfully with syslog-ng:
> >
> > source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> > destination d_sftp_<username> { file("/var/log/sftp/<username>.log"); };
> > log { source(s_chroot_<username>); destination(d_sftp_<username>); };
> >
> > Now I have a second sftpd server in parallel, with the same user database and also mounts /var/data/chroot/ via NFS, and has the same syslog-ng config,
> > so every user can login on the one server or on the other. This is for high availability. This works so far.
> >
> > What is not working now is the sftpd logging: The sftp user's log is only available on one sftp server exclusively, and that is the one where syslog-ng was started least,
> > because as I understand it takes the exclusive unix socket file lock for each user's /dev/log.
> >
> > So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
> > But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
> >
> > If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.
> >
> > How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
>
> Right now there is no solution for this inside OpenSSH. There have been
> some proposals for post-auth logging to be proxied via the priviledged
> sshd monitor process but we haven't pursued them yet.
>
> Maybe someone with more Linux/NFS wit could suggest an OS-side solution
> for you?
>
> -d
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
Hallo Hildegard,

thanks for the summary.

Hildegard Meier wrote:
> ---------------------------------------------------------------
> AllowGroups sftp-cust
> Subsystem sftp internal-sftp -f LOCAL5 -l INFO
> Match Group sftp-cust
> ChrootDirectory %h
> ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
> AllowTcpForwarding no
> ---------------------------------------------------------------

This is very sensible configuration but it unfortunately makes the
LD_PRELOAD approach a bit less desirable. It will still work though.


> So a solution for the log problem would be, to not use the chroot
> logging, but to parse the global sftpd log
..
> But I think this is not trivial to make this reliable and robust,

Agreed. If going the log analysis path I would add -e to the
internal-sftp command line and run sshd with stderr connected to that
parser, to be able to keep track of events in a sensible way.
Log rotation would happen only after (or in) the parser.


> If we want to try to keep using the available session chroot
> logging, man (8) sftp-server says:
>
> "For logging to work, sftp-server must be able to access /dev/log. Use of sftp-server in a chroot configuration therefore requires that
> syslogd(8) establish a logging socket inside the chroot directory."
>
> As Peter has written here
> https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html
>
> as I understand it is hard coded in the system C library that the log
> devices name is /dev/log, so this cannot be changed (e.g. to log to
> chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2
> on hostname2).

Using LD_PRELOAD e.g. with the code I attached to that email allows
replacing the C library syslog functionality. My attachment does
indeed log to /dev/log_hostname1 on hostname1 instead of /dev/log.

However because of internal-sftp (which I still prefer over the alternative;
maintaining an sftp-server binary and possibly libraries in every homedir)
the entire sshd would have to be run with LD_PRELOAD.

Note that LD_PRELOAD is only neccessary on n-1 servers; the first
server can still use /dev/log and the full C library syslog() code.

If the second server where LD_PRELOAD is used does not need to
support anything besides SFTP, or if you can mandate that, then
it could be okay to run the entire sshd with LD_PRELOAD. Your IT
security colleagues may well object and they're not wrong; it's not
much code but still quite a mighty hammer.


> > Maybe someone with more Linux/NFS wit could suggest an OS-side solution
> > for you?

Before going down the LD_PRELOAD path I considered an overlayfs solution.

It would be nice, but the writable (upper) directory can't be NFS
with overlayfs so it can not work on the SFTP servers.

But using overlayfs can work on the NFS server. There would then be
one export for each SFTP server, all of them reaching the same
homedir contents except for different (empty) <homedir>/dev/ subdirs.

This requires maintaining a second homedir structure on the NFS server,
but that structure only needs to contain a homedir with one empty dev
subdir under and nothing else, so it may be doable. Homedirs need
correct permissions and only one of the two directory inodes for the
homedir itself will be changed by the respective SFTP server, but on
the plus side it requires only a single overlay mount, nothing per-user
and no LD_PRELOAD stuff on the SFTP servers.


On the NFS server, assuming that /home is where real homedirs exist
(this gets mounted directly by one SFTP server), that /var/home2 is
the second homedir structure with only homedirs and an empty dev subdir
in each, that /var/home2_work is an empty directory on the same filesystem
as /var/home2 and that /nfs/home2 is the second export mounted by the
second SFTP server, then set it up like so:

mount -t overlay overlay -olowerdir=/home,upperdir=/var/home2,workdir=/var/home2_work /nfs/home2


Then export /home and /nfs/home2 via NFS.

The SFTP server mouting /home will access /home/<user>/dev/log
(both syslog-ng and internal-sftp) and the SFTP server mouting
/nfs/home2 will instead access /var/home2/<user>/dev/log but
everything else within the homedir is to and from /home/<user>.

If this change can be done on the NFS server I think it's what I'd prefer.


Kind regards

//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
I like Peter's overlay concept. Ubuntu has aufs-tools to use builtin kernal code. These add mount.aufs

But here is another proposal.

You already have 800 NFs volumes and they are all mounted on each server.
(This is based on the syslog-ng configure which create a unix-stream in every volume when started.)

I assume that the only difference between these volumes is the user's home directory with
files they transfer to and from. And the user has only read access to whatever else is in the volume.
I assume they must have a /etc/passwd file with only one user.

Create 800 duplicate volumes on each server (not in NFS), but without the user's home directory.
for example /var/data/sftp/<username>/chroot/


modify the current 800 NFS volumes so they can be NFS mounted as the user's home in the above
new volumes.

Thus each sftp server uses the /var/data/sftp/<username>/chroot/dev/log which is on the same host
rather then a shared /dev/log. Its more complicated, but no local modifications.


On 9/28/2021 10:00 AM, Hildegard Meier wrote:
> Hallo all, thank You all very much for your answers and suggestions.
> Since the discussion has been shared between several mails now, I would like to try to summarize all up in one mail again here
> and fill in the informations that where missing before.
>
> We have 800 (eight hundred) sftp customers, each sftp customer has the same simple local Linux account on both sftp servers (simple entry in /etc/passwd and /etc/shadow etc.).
> (Before we had only one sftp server, but for higher availability we want now to run two or more sftp servers parallel, accessed via TCP (sftp) load balancer)
>
> Each customer account is in the group "sftp-cust", and only members of that group are allowed to login via sftp.
> Each customer has it's chrooted home dir in /var/data/chroot/<username>/.
>
> Here the (relevant part) of the sftpd config:
>
> ---------------------------------------------------------------
> AllowGroups sftp-cust
> Subsystem sftp internal-sftp -f LOCAL5 -l INFO
> Match Group sftp-cust
> ChrootDirectory %h
> ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
> AllowTcpForwarding no
> ---------------------------------------------------------------
>
> Each sftp customer's sftp activity needs to be available in a dedicated log file for each sftp customer, for our support to be able to look into it. Here is an example of this log file:
>
> Sep 28 15:54:25 myhostname internal-sftp[1618]: session opened for local user <username> from [1.2.3.4]
> Sep 28 15:55:52 myhostname internal-sftp[27918]: remove name "/in/file.dat"
> Sep 28 15:55:52 myhostname internal-sftp[27918]: sent status No such file
> Sep 28 15:55:52 myhostname internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE mode 0666
> Sep 28 15:55:52 myhostname internal-sftp[27918]: close "/in/file.dat" bytes read 0 written 2966
> Sep 28 15:55:52 myhostname internal-sftp[27918]: set "/in/file.dat" modtime 20210928-15:55:46
>
> If one does not use the /dev/log in the chroot environment (that is /var/data/chroot/<username>/dev/log absolute), you have a global sftpd log (I think in /var/log/messages on the server or something like that).
> So a solution for the log problem would be, to not use the chroot logging, but to parse the global sftpd log (which is available on both sftpd servers locally for all local logins separated).
> But I think this is not trivial to make this reliable and robust, since we need to parse the process id (sftpd session), check which username that session belongs to and write that log lines in the username specific log file.
> Of course, process IDs are reused for different sessions, we have overlapping sessions etc. But if there is no other solution, than we need to try that. Maybe someone has already written such a session tracking parser.
> Does somebody know of a log analyzer that can do this? A problem would be e.g. if the user never logs out, and the log file rotates daily, so after one day, the parser could not find any "session opened for local user xxx" log lines anymore.
>
> If we want to try to keep using the available session chroot logging, man (8) sftp-server says:
>
> "For logging to work, sftp-server must be able to access /dev/log. Use of sftp-server in a chroot configuration therefore requires that
> syslogd(8) establish a logging socket inside the chroot directory."
>
> As Peter has written here
> https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html
>
> as I understand it is hard coded in the system C library that the log devices name is /dev/log, so this cannot be changed (e.g. to log to chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2 on hostname2).
>
> With the syslog-ng we use, we only need to create the directory /var/data/chroot/<username>/dev for each user once we create a new account.
>
> We have the following syslog-ng config snippet file for each sftp user:
>
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> destination d_sftp_<username> { file("/var/log/app/sftp/<username>.log"); };
> log { source(s_chroot_<username>); destination(d_sftp_<username>); };
>
> When starting syslog-ng the unix stream file /var/data/chroot/<username>/dev/log is created automatically by syslog-ng. If you delete it it gets recreated upon syslog-ng restart.
>
> But the problem is that the last started syslog-ng aquires the lock for the NFS shared /var/data/chroot/<username>/dev/log so the other server cannot read it anymore and so there is no sftp log for the sessions on the other server.
>
> I guess this is because syslog-ng creates /dev/log in the user's chroot directories as Unix stream socket (see https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/28#TOPIC-1209171).
> This seems also to be called IPC socket (inter-process communication socket) or AF_UNIX socket.
> "It is used in POSIX operating systems for inter-process communication. The correct standard POSIX term is POSIX Local IPC Sockets. Unix domain connections appear as byte streams, much like network connections, but all data _remains within the local computer_."
>
> "It means that if you create a AF_UNIX socket on a NFS disk which is shared between two machines A and B, you cannot have a process on A writing data to the unix socket and a process on B reading data from that socket.
> The communication happens at kernel level, and you can only transfer data among processes sitting in the same kernel."
> (see source: https://stackoverflow.com/questions/783134/af-unix-domain-why-use-local-file-names-only )
>
> Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share.
>
> We need to stick with Ubuntu Linux as we have established management process only for this operating systems.
>
> I hope I did not forget some information that was missing before. Thanks Hildegard
>
>
>
>> Gesendet: Mittwoch, 22. September 2021 um 09:19 Uhr
>> Von: "Damien Miller" <djm@mindrot.org>
>> An: "Hildegard Meier" <daku8938@gmx.de>
>> Cc: openssh-unix-dev@mindrot.org
>> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>>
>> On Tue, 21 Sep 2021, Hildegard Meier wrote:
>>
>>> OpenSSH 5.9p1 + 7.6p1
>>>
>>> syslog-ng 3.3.4 + 3.13.2
>>>
>>> Hello, having an Ubuntu server with sftpd running where /var/data/chroot/ is an NFS mount from a remote central NFS server,
>>> and each sftpd user's chroot home is /var/data/chroot/<username>/
>>> and every user has a log device /var/data/chroot/<username>/dev/log which I read in successfully with syslog-ng:
>>>
>>> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
>>> destination d_sftp_<username> { file("/var/log/sftp/<username>.log"); };
>>> log { source(s_chroot_<username>); destination(d_sftp_<username>); };
>>>
>>> Now I have a second sftpd server in parallel, with the same user database and also mounts /var/data/chroot/ via NFS, and has the same syslog-ng config,
>>> so every user can login on the one server or on the other. This is for high availability. This works so far.
>>>
>>> What is not working now is the sftpd logging: The sftp user's log is only available on one sftp server exclusively, and that is the one where syslog-ng was started least,
>>> because as I understand it takes the exclusive unix socket file lock for each user's /dev/log.
>>>
>>> So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
>>> But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
>>>
>>> If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.
>>>
>>> How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
>>
>> Right now there is no solution for this inside OpenSSH. There have been
>> some proposals for post-auth logging to be proxied via the priviledged
>> sshd monitor process but we haven't pursued them yet.
>>
>> Maybe someone with more Linux/NFS wit could suggest an OS-side solution
>> for you?
>>
>> -d
>>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
On 28.09.21 17:00, Hildegard Meier wrote:
> Each sftp customer's sftp activity needs to be available in a
> dedicated log file for each sftp customer, for our support to
> be able to look into it. Here is an example of this log file:
>
> Sep 28 15:54:25 myhostname internal-sftp[1618]:
> session opened for local user <username> from [1.2.3.4]

(General warning from me: There are log messages from the user's session
*earlier* than that, from before the client successfully authenticated
for the login as <username>. Having those messages go into a per-account
log(file) is nontrivial because sshd *does not yet know* what username
it will be, and if your support staff cannot see those lines, they will
not be able to debug *failed* connection attempts.)

At this point, I'd say that the best approach would be to have the
post-auth sftp-server process log to a *file* within the chroot, with
the filename set to the server's hostname (in the ForceCommand statement
of the server's /etc/ssh/sshd_config). Unfortunately, while sftp-server
has command line options for logging to syslog and for logging to
stderr, it does *not* have a command line option to log to a *file* (or,
for that matter, to set/change the "ident" parameter of openlog(3) to
include the username) ...

> you have a global sftpd log (I think in /var/log/messages on the
> server or something like that)

(RHEL, Centos, Fedora etc. tend to have a /var/log/messages . Ubuntu, in
my experience, uses /var/log/syslog instead.)

> But I think this is not trivial to make this reliable and robust,
> since we need to parse the process id (sftpd session), check which
> username that session belongs to and write that log lines in the
> username specific log file.

That's essentially what tools like opentelemetry.io are talking about
when they make a difference between "logs" and "traces", but I suppose
that throwing one of those at your problem could easily prove to be
overkill ...

https://opentelemetry.io/docs/concepts/data-sources/

> A problem would be e.g. if the user never logs out, and the log file
> rotates daily, so after one day, the parser could not find any
> "session opened for local user xxx" log lines anymore.

That's why log-parsing utilities that are worth their salt (logwatch,
ConSol's check_logfiles, etc.) can be configured to find and include the
rotated-away previous logfile instances ...

> as I understand it is hard coded in the system C library that the
> log devices name is /dev/log, so this cannot be changed

Well, not on Linux, at least ...

[.still fondly remembers AIX "softlinks" where you could "ln -s"
something to "/foo/$BAR/baz" and the value of the env var BAR would be
taken into account at access/expansion time]

> But the problem is that the last started syslog-ng aquires the lock
> for the NFS shared /var/data/chroot/<username>/dev/log so the other
> server cannot read it anymore and so there is no sftp log for the
> sessions on the other server.

(I *still* suspect that if only you could configure the syslogd's to use
a file locking method that just does *not* work across NFS shares -
there's about half a dozen different methods available, see, e.g.,

https://dovecot.org/pipermail/dovecot/2011-July/060149.html

-, you could circumvent that effect from the get-go ...)

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
Re: Howto log multiple sftpd instances with their chroot shared via NFS [ In reply to ]
Hildegard Meier wrote:
> But the problem is that the last started syslog-ng aquires the lock
> for the NFS shared /var/data/chroot/<username>/dev/log so the other
> server cannot read it anymore

Is it known what kind of lock this is? Was it investigated? Maybe on
the NFS server?


Douglas E Engert wrote:
> You already have 800 NFs volumes and they are all mounted on each server.

AIUI there's only one NFS export with all homedirs mounted on each server,
and avoiding per-user runtime setup such as mounts is a requirement.


Jochen Bern wrote:
> I *still* suspect that if only you could configure the syslogd's to use
> a file locking method that just does *not* work across NFS shares -
> there's about half a dozen different methods available, see, e.g.,
>
> https://dovecot.org/pipermail/dovecot/2011-July/060149.html
>
> -, you could circumvent that effect from the get-go ...

Looking through the afsocket module in syslog-ng it does no file locking.


I'm curious what kind of locking it is. Maybe the contention is
all within the NFS layer and could be overcome by setting a nolock
or local_lock mount option on the SFTP servers, if either option is
acceptable for the use case.


Kind regards

//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

1 2 3  View All