Mailing List Archive

On Tue, 21 Sep 2021, Hildegard Meier wrote:

> OpenSSH 5.9p1 + 7.6p1
>
> syslog-ng 3.3.4 + 3.13.2
>
> Hello, having an Ubuntu server with sftpd running where /var/data/chroot/ is an NFS mount from a remote central NFS server,
> and each sftpd user's chroot home is /var/data/chroot/<username>/
> and every user has a log device /var/data/chroot/<username>/dev/log which I read in successfully with syslog-ng:
>
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> destination d_sftp_<username> { file("/var/log/sftp/<username>.log"); };
> log { source(s_chroot_<username>); destination(d_sftp_<username>); };
>
> Now I have a second sftpd server in parallel, with the same user database and also mounts /var/data/chroot/ via NFS, and has the same syslog-ng config,
> so every user can login on the one server or on the other. This is for high availability. This works so far.
>
> What is not working now is the sftpd logging: The sftp user's log is only available on one sftp server exclusively, and that is the one where syslog-ng was started least,
> because as I understand it takes the exclusive unix socket file lock for each user's /dev/log.
>
> So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
>
> If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.
>
> How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?

Right now there is no solution for this inside OpenSSH. There have been
some proposals for post-auth logging to be proxied via the priviledged
sshd monitor process but we haven't pursued them yet.

Maybe someone with more Linux/NFS wit could suggest an OS-side solution
for you?

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

On Tue, 21 Sep 2021, Hildegard Meier wrote:
> Now I have a second sftpd server in parallel, with the same user
> database and also mounts /var/data/chroot/ via NFS, and has the same
> syslog-ng config,
> so every user can login on the one server or on the other. This is for
> high availability. This works so far.
>
> What is not working now is the sftpd logging: The sftp user's log is
> only available on one sftp server exclusively, and that is the one
> where syslog-ng was started least, because as I understand it takes
> the exclusive unix socket file lock for each user's /dev/log.
>
> So, if a user logs in on the first server, where syslog-ng was started
> least, the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is
> not logged, neither on the second nor on the first server.

Forward the log entries on both machines to a log host.? E.g.

destination d_tcp {
network("log_host" port(1999));
};

Regards,

David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Tue, 21 Sept 2021 at 19:58, Hildegard Meier <daku8938@gmx.de> wrote:
[...]

> Now I have a second sftpd server in parallel, with the same user database
> and also mounts /var/data/chroot/ via NFS

[...]

> How can I get the sftp user's activity be logged on each sftp server, when
> a user logs in to that server, while the user's home is shared on both
> servers via NFS?
>

Mount a tiny local ramfs or tmpfs over /var/data/chroot/dev?

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 22.09.21 11:18, David Newall wrote:
> On Tue, 21 Sep 2021, Hildegard Meier wrote:
>> So, if a user logs in on the first server, where syslog-ng was started
>> least, the user's sftp activity is logged on the first server.
>> But if the user logs in on the second server, it's sftp activity is
>> not logged, neither on the second nor on the first server.
>
> Forward the log entries on both machines to a log host.

Considering that server B is not logging *at all* right now, I doubt
that it'll have anything to forward to a log host, either.

The problem *presumably* is that the syslogd on server A has put some
sort of file lock on the device that propagates through the NFS server
and interferes with syslogd on server B using it.

One solution might be to reconfigure the syslogd's to use a method of
locking that does *not* propagate through NFS. I'm afraid I don't know
syslog-ng well enough to advise on that.

Then there's the possibility of reconfiguring *NFS* to stop the
forwarding, but "breaking" file locking on NFS is, of course, a can of
worms of possible side effects ...

(Bind) mounting a local .../dev over the NFS-shared chroot dirtree ...
ought to work, but complicates unmounting/remounting, which was already
enough of a hair-puller in failure scenarios when I last worked with NFS.

What do the chrooted users have for a homedir *within* the chroot? Would
it be possible to have /var/data/chroot be a local FS and mount only
/var/data/chroot/home from the NFS server? (If there are files that you
need to keep identical on both servers, e.g., under
/var/data/chroot/etc, you can still symlink those to some special subdir
like /var/data/chroot/home/ETC to put the actual data onto the NFS share.)

Regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 2021/09/22 13:06, Jochen Bern wrote:
> What do the chrooted users have for a homedir *within* the chroot? Would it
> be possible to have /var/data/chroot be a local FS and mount only
> /var/data/chroot/home from the NFS server? (If there are files that you need
> to keep identical on both servers, e.g., under /var/data/chroot/etc, you can
> still symlink those to some special subdir like /var/data/chroot/home/ETC to
> put the actual data onto the NFS share.)

The description was for /var/data/chroot/<username>/dev/log i.e. each
user has their own separate chroot. So this type of approach would
require mounting a local fs of some sort over the top of each user's dir
which soon gets messy.

This is amongst the reasons why OpenBSD has the sendsyslog(2) syscall,
https://man.openbsd.org/sendsyslog.2 - the syslog daemon opens a
kernel socket to receive those messages, and processes which want to
write a log entry just call the standard syslog functions which use
sendsyslog(2) rather than /dev/log, so it works even through FD
exhaustion, in chroot, and with syscall filters that prohibit
filesystem access.

Not entirely pleasant but I suppose it could alternatively be done by
using a LD_PRELOAD wrapper to override syslog functions (I think just
syslog_r is probably enough for openssh) and have them send over a
network socket instead.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Fri, 24 Sep 2021, Stuart Henderson wrote:

> This is amongst the reasons why OpenBSD has the sendsyslog(2) syscall,
> https://man.openbsd.org/sendsyslog.2 - the syslog daemon opens a
> kernel socket to receive those messages, and processes which want to
> write a log entry just call the standard syslog functions which use

Oh, nice.

> The description was for /var/data/chroot/<username>/dev/log i.e. each
> user has their own separate chroot. So this type of approach would
> require mounting a local fs of some sort over the top of each user's dir

This made me curious, and I tried¹ this. It is possible to bind-mount
sockets on Linux iff the target exists as regular file.

sudo touch /var/data/chroot/<username>/dev/log # but beware of
# filesystem-based
# attacks here!
sudo mount --bind /dev/log /var/data/chroot/<username>/dev/log

? I went and began using this technology here:
https://github.com/mirabilos/shellsnippets/blob/master/posix/debchroot.sh

bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

****************************************************
/?\ The UTF-8 Ribbon
? ? Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
? HTML eMail! Also, https://www.tarent.de/newsletter
? ? header encryption!
****************************************************
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 9/21/2021 4:49 AM, Hildegard Meier wrote:

>
> How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?

based on:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/88

This might work for a few sftp servers.
On first server, hostname1, use /dev/<hostname1>/log for all its default logging and its normal syslog-ng.conf and for the special
syslog-ng.conf:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
(May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)

Do the above for second server, hostname2, use /dev/<hostname2>/log

Then add these to the chroot directories in NFS:
/var/data/chroot/<username>/dev/<hostname1>/log
/var/data/chroot/<username>/dev/<hostname2>/log

This should cause hostname1 to lock on /var/data/chroot/<username>/dev/<hostname1>/log
and hostname 2 to lock on /var/data/chroot/<username>/dev/<hostname2>/log
because the syslog-ng does not really look at the syslog-ng.conf in the chroot.


>
> Thanks
>
> Crossposting (because unanswered)
> https://unix.stackexchange.com/questions/666641/howto-log-multiple-sftpd-servers-activity-which-users-chrooted-home-is-on-shar
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Many thanks for your answers!
 
More ideas appreciated, since it would be relly important for us to have a solution for that.
 
 
>Date: Wed, 22 Sep 2021 13:06:43 +0200
>From: Jochen Bern <Jochen.Bern@binect.de>
>To: openssh-unix-dev@mindrot.org
>Subject: Re: Howto log multiple sftpd instances with their chroot
>shared via NFS
On 22.09.21 11:18, David Newall wrote:
> On Tue, 21 Sep 2021, Hildegard Meier wrote:
>> So, if a user logs in on the first server, where syslog-ng was started
>> least, the user's sftp activity is logged on the first server.
>> But if the user logs in on the second server, it's sftp activity is
>> not logged, neither on the second nor on the first server.
>
> Forward the log entries on both machines to a log host.
>Considering that server B is not logging *at all* right now, I doubt
>that it'll have anything to forward to a log host, either.
 
Yes the log destination is not the problem, the empty log source is.


>The problem *presumably* is that the syslogd on server A has put some
>sort of file lock on the device that propagates through the NFS server
>and interferes with syslogd on server B using it.
 
As I understand, syslog-ng creates /dev/log in the user's chroot directories as Unix stream socket (see https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/28#TOPIC-1209171).
 
This seems also to be called IPC socket (inter-process communication socket)   or AF_UNIX socket.
 
"It is used in POSIX operating systems for inter-process communication. The correct standard POSIX term is POSIX Local IPC Sockets. Unix domain connections appear as byte streams, much like network connections, but all data _remains within the local computer_."
 
"It means that if you create a AF_UNIX socket on a NFS disk which is shared between two machines A and B, you cannot have a process on A writing data to the unix socket and a process on B reading data from that socket.
The communication happens at kernel level, and you can only transfer data among processes sitting in the same kernel."
(see source: https://stackoverflow.com/questions/783134/af-unix-domain-why-use-local-file-names-only )


>One solution might be to reconfigure the syslogd's to use a method of
>locking that does *not* propagate through NFS. I'm afraid I don't know
>syslog-ng well enough to advise on that.
 
I think the problem is that the chrooted sftp users only have write access (for logging) under their chroot home dir (and there /devlog) , and since that is on NFS, how could one escape that. A possibility would be a symlink to a local unix stream device, but chroot does not allow to access out of it, that's the whole point of being chrooted (jailed).


>Then there's the possibility of reconfiguring *NFS* to stop the
>forwarding, but "breaking" file locking on NFS is, of course, a can of
>worms of possible side effects ...
Do not know what you exactly mean, but I think we cannot fiddle with NFS file locking since we have several hundrets of sftp customer accounts on the NFS chroot, which transfer many many files.


>(Bind) mounting a local .../dev over the NFS-shared chroot dirtree ...
>ought to work, but complicates unmounting/remounting, which was already
>enough of a hair-puller in failure scenarios when I last worked with NFS.
man bind says: "mount --bind olddir newdir"
/dev/log is not a directory, but created by syslog-ng as a unix stream socket (see above). I guess if it would a directory, the sftpd process could not log to it?
# mkdir -p /my/local/dir/
# mount -v --bind /var/data/chroot/test/dev/log /my/local/dir
mount: /my/local/dir: mount(2) system call failed: Not a directory.


>What do the chrooted users have for a homedir *within* the chroot? Would
>it be possible to have /var/data/chroot be a local FS and mount only
>/var/data/chroot/home from the NFS server? (If there are files that you
>need to keep identical on both servers, e.g., under
>/var/data/chroot/etc, you can still symlink those to some special subdir
>like /var/data/chroot/home/ETC to put the actual data onto the NFS share.)
 
Strict technically, we really _need_ to share every sftp user's chrooted upload directory (/var/data/chroot/<username>/in/) and download directory (/var/data/chroot/<username>/out/), since we have a single backend process that reads all files that the customers have uploaded into their chroot /in/ directory (be it that they have logged in to the one sftp server or the other, both sftp servers are TCP load balanced, but the time order of the uploads needs to be honoured), and writes files to the customers chroot /out/ dir (and the customer can download the files from it's chrooted /out/ dir whether having sftp logged in to the one sftp server or the other).
But since we have 800 sftp customers, so 800 /var/data/chroot/<username>/in/ and 800 /var/data/chroot/<username>/out/ it is not feasable to have 1600 single NFS mounts, to achieve that /var/data/chroot/<username>/dev/log is local. That is the problem.
 
 
>Mount a tiny local ramfs or tmpfs over /var/data/chroot/dev?
 
For the same reason, since we have so many /var/data/chroot/<username>/dev/log , this would not be feasable (apart of the question if this would technically be a solution).
 
We need also a single log source for each every sftp user, because we need a single log file for every single sftp user, to be able to view each sftp user's sftp activity.
 
Best regards
Hildegard
 
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Thanks, this sounds interesting and maybe in this direction could be the solution? But I do not understand you precisely.

> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
You mean there should be
/var/data/chroot/<username>/dev/hostname1/log
/var/data/chroot/<username>/dev/hostname2/log

?

> (May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
You mean there should be the Symlink
/var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname1/log

?

> Do the above for second server, hostname2, use /dev/<hostname2>/log

It would not be possible to have a Symlink
/var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname2/log

since /var/data/chroot/<username>/dev/log is already a Symlink to /var/data/chroot/<username>/dev/hostname1/log
(see first step), and it is the same NFS filesystem.



> Gesendet: Freitag, 24. September 2021 um 16:08 Uhr
> Von: "Douglas E Engert" <deengert@gmail.com>
> An: openssh-unix-dev@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> On 9/21/2021 4:49 AM, Hildegard Meier wrote:
>
> >
> > How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
>
> based on:
> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/88
>
> This might work for a few sftp servers.
> On first server, hostname1, use /dev/<hostname1>/log for all its default logging and its normal syslog-ng.conf and for the special
> syslog-ng.conf:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
> (May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
>
> Do the above for second server, hostname2, use /dev/<hostname2>/log
>
> Then add these to the chroot directories in NFS:
> /var/data/chroot/<username>/dev/<hostname1>/log
> /var/data/chroot/<username>/dev/<hostname2>/log
>
> This should cause hostname1 to lock on /var/data/chroot/<username>/dev/<hostname1>/log
> and hostname 2 to lock on /var/data/chroot/<username>/dev/<hostname2>/log
> because the syslog-ng does not really look at the syslog-ng.conf in the chroot.
>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Fri, 24 Sep 2021, Hildegard Meier wrote:

> It would not be possible to have a Symlink
> /var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname2/log
>
> since /var/data/chroot/<username>/dev/log is already a Symlink to
> /var/data/chroot/<username>/dev/hostname1/log (see first step), and it
> is the same NFS filesystem.

Not sure I understand the entire situation, but can’t you make
/var/data/chroot/<username>/dev/log a regular file (on the NFS
mount) then bind-mount (see my earlier mail) /dev/log from the
host over it?

bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

****************************************************
/?\ The UTF-8 Ribbon
? ? Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
? HTML eMail! Also, https://www.tarent.de/newsletter
? ? header encryption!
****************************************************
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Thanks, this sounds like the solution could be in this direction.
I think, the sftpd process should just not write to the /dev/log unix socket (because this leads to the problem here), but to the local kernel directly, something like what you describe here.

But how could I do this concrete with Ubuntu Linux? What you write is rather abstract and I am not so expert that I understand what you mean with LD_PRELOAD wrapper.

Unfortunately, I could not change our sftp server to OpenBSD operating system since we would not have the capacity to maintain this one special operating system. We maintain our 350 Ubuntu Linux servers with already established processes.



> This is amongst the reasons why OpenBSD has the sendsyslog(2) syscall,
> https://man.openbsd.org/sendsyslog.2 - the syslog daemon opens a
> kernel socket to receive those messages, and processes which want to
> write a log entry just call the standard syslog functions which use
> sendsyslog(2) rather than /dev/log, so it works even through FD
> exhaustion, in chroot, and with syscall filters that prohibit
> filesystem access.
>
> Not entirely pleasant but I suppose it could alternatively be done by
> using a LD_PRELOAD wrapper to override syslog functions (I think just
> syslog_r is probably enough for openssh) and have them send over a
> network socket instead.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 9/24/2021 10:03 AM, Hildegard Meier wrote:
> Thanks, this sounds interesting and maybe in this direction could be the solution? But I do not understand you precisely.
>
>> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
> You mean there should be
> /var/data/chroot/<username>/dev/hostname1/log
> /var/data/chroot/<username>/dev/hostname2/log
>

The point is each application's syslog library will write to /dev/hostnameX/log or if running in chroot
var/data/chroot/<username>/dev/hostnameX/log

Thus each sever has it own lock for each user.


So any lock done while application is running in chroot will lock will lock on different file.


>
>> (May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
> You mean there should be the Symlink
> /var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname1/log
>

Symlink may not be needed. I was refereing to a symlink when not using the chroot.

>
>> Do the above for second server, hostname2, use /dev/<hostname2>/log
>
> It would not be possible to have a Symlink
> /var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname2/log
>
> since /var/data/chroot/<username>/dev/log is already a Symlink to /var/data/chroot/<username>/dev/hostname1/log
> (see first step), and it is the same NFS filesystem.
>
>
>
>> Gesendet: Freitag, 24. September 2021 um 16:08 Uhr
>> Von: "Douglas E Engert" <deengert@gmail.com>
>> An: openssh-unix-dev@mindrot.org
>> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>>
>> On 9/21/2021 4:49 AM, Hildegard Meier wrote:
>>
>>>
>>> How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
>>
>> based on:
>> https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/88
>>
>> This might work for a few sftp servers.
>> On first server, hostname1, use /dev/<hostname1>/log for all its default logging and its normal syslog-ng.conf and for the special
>> syslog-ng.conf:
>> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/hostname1/log" optional(yes) ); };
>> (May need to add a symlink on each host /dev/log->/dev/<hostname1>/log)
>>
>> Do the above for second server, hostname2, use /dev/<hostname2>/log
>>
>> Then add these to the chroot directories in NFS:
>> /var/data/chroot/<username>/dev/<hostname1>/log
>> /var/data/chroot/<username>/dev/<hostname2>/log
>>
>> This should cause hostname1 to lock on /var/data/chroot/<username>/dev/<hostname1>/log
>> and hostname 2 to lock on /var/data/chroot/<username>/dev/<hostname2>/log
>> because the syslog-ng does not really look at the syslog-ng.conf in the chroot.
>>
>
> .
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 9/24/2021 10:11 AM, Thorsten Glaser wrote:
> On Fri, 24 Sep 2021, Hildegard Meier wrote:
>
>> It would not be possible to have a Symlink
>> /var/data/chroot/<username>/dev/log -> /var/data/chroot/<username>/dev/hostname2/log

Correct that is not needed.
It looks like the server has a line:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };

It is not clear if this is created ahead of time or on the fly when the NFS volume is mounted
or for each possible user to create a unix-stream socket in NFS.


What I was proposing is each of the two servers has it own /var/data/chroot/<username>/dev/hostnameX.log
What is not clear is when running in chroot, when an application does
an openlog, syslog or vsyslog are these system calls, or library calls.
i.e. do the read any configuration from the chroot directory?

I think it expects to find /dev/log in the chroot directory and this name
matches what is in the real root directory. So I was proposing
that each sftpd server has a different /dev/hostname.log so a user
could be on both of the servers at the same time.

Hildegard said these are Ubuntu servers, and Ubuntu uses rsyslogd that looks
similar to syslog-ng.

Google'ing for: rsyslog chroot /dev/log

These look interesting.

https://www.the-art-of-web.com/system/sftp-logging-chroot/
https://serverfault.com/questions/878627/logging-chrooted-sftp-user-activity



>>
>> since /var/data/chroot/<username>/dev/log is already a Symlink to
>> /var/data/chroot/<username>/dev/hostname1/log (see first step), and it
>> is the same NFS filesystem.
>
> Not sure I understand the entire situation, but can’t you make
> /var/data/chroot/<username>/dev/log a regular file (on the NFS
> mount) then bind-mount (see my earlier mail) /dev/log from the
> host over it?

Is this what you are trying to do?
https://serverfault.com/questions/878627/logging-chrooted-sftp-user-activity

Instead, use input(type="imuxsock" Socket="/opt/bindmount/dev/log") to create an additional
socket (which you can then bind-mount such that all chroots see it at /dev/log).

>
> bye,
> //mirabilos
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Fri, 24 Sep 2021, Douglas E Engert wrote:

> Instead, use input(type="imuxsock" Socket="/opt/bindmount/dev/log") to create
> an additional
> socket (which you can then bind-mount such that all chroots see it at
> /dev/log).

Wasn’t it said earlier in the thread that this doesn’t scale across
multiple chroots, especially when they may be added and removed dy?
namically? Restarting the syslog daemon all the time isn’t going to
fly… and this can avoid that.

bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

****************************************************
/?\ The UTF-8 Ribbon
? ? Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
? HTML eMail! Also, https://www.tarent.de/newsletter
? ? header encryption!
****************************************************
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

Hildegard Meier wrote:
> I think, the sftpd process should just not write to the /dev/log
> unix socket (because this leads to the problem here), but to the
> local kernel directly, something like what you describe here.

Exactly, but unfortunately the code that sets the socket path is in
the system C library, without any interface to control the setting.


> But how could I do this concrete with Ubuntu Linux? What you write
> is rather abstract and I am not so expert that I understand what
> you mean with LD_PRELOAD wrapper.

Setting the LD_PRELOAD environment variable to a library filename when
starting a program loads that library into memory before the system C
library and the program itself. Symbols (ie. functions) loaded from
the preloaded library are not replaced by later (system) libraries,
making it possible to override and/or wrap libc functionality in this
fairly brutish way, hence a bit unpleasant.

Attached is source code for a simple such wrapper overriding openlog(),
syslog() and closelog(). It might work only on the older system, or on
both. You only need this hack on one server, the other can use /dev/log.

The wrapper directs logging to /dev/log_<hostname> instead of /dev/log.
Since libc has no interface to manipulate the socket path these functions
override the normal functions with a very limited implementation.
Adjust the template on line 49 if you want to change the destinateion.

Although functionality is limited it could work well for sftp-server.

One simplification is that this openlog() ignores all option flags
except LOG_NDELAY and that syslog() always behaves as if openlog() was
called with the LOG_PID option. Since this is exactly what OpenSSH log.c
does all log messages should look the same as with the libc implementation.

Note that log messages are silently lost on any error, e.g. syslog-ng
not running. Also note that if log messages, the openlog() ident
parameter or the username (when ident == NULL) are very long then
messages are silently truncated to 2048 chars. OpenSSH sends max 500
chars.


You didn't mention if your two sshd are configured to use the in-process
sftp-server (Subsystem sftp internal-sftp in sshd_config) - if so then
this is probably not really the best solution, since LD_PRELOAD would
have to be set for the entire sshd process, which I don't recommend
because this code is so simplistic compared to libc.

Anyway, try it if you like.


The comment near the top of the file shows how to compile. sshd can't
be configured to set LD_PRELOAD so to use the compiled .so with
sftp-server you have to create a simple script:

--8<-- /usr/local/libexec/sftp-server-hostlog
#!/bin/sh
export LD_PRELOAD=/usr/local/libexec/hostlog.so
exec /usr/lib/sftp-server "$*"
-->8--

and then configure sshd to call that instead of sftp-server:

--8<-- /etc/ssh/sshd_config
Subsystem sftp /usr/local/libexec/sftp-server-hostlog
-->8--

Adjust paths as needed, also in the compile command.


For anyone else interested please go ahead but note that I haven't
put huge effort into portability; it might work outside Linux but
I haven't tested anywhere else so YMMV.


Kind regards

//Peter

On 24.09.21 16:37, Hildegard Meier wrote:
> More ideas appreciated, since it would be relly important for us
> to have a solution for that.

OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
have no idea how portable it is to other distribs, so just give it a try:

> # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
> Subsystem sftp internal-sftp
> Match group mandanten
> ForceCommand internal-sftp -l INFO -u 0077


> # ls -al ~lvinq4/dev
> insgesamt 0
> drwx--x---. 2 root mandanten 6 20. Mai 17:25 .
> drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..

As you can see, the chroots have an *empty* /dev subdir, but logging
*still works*, apparently because the chrooted process just keeps using
the system-central /dev/log it opened *before* chroot()ing.

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

Jochen Bern wrote:
> OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
> have no idea how portable it is to other distribs, so just give it a try:
>
> > # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
> > Subsystem sftp internal-sftp
> > Match group mandanten
> > ForceCommand internal-sftp -l INFO -u 0077
>
>
> > # ls -al ~lvinq4/dev
> > insgesamt 0
> > drwx--x---. 2 root mandanten 6 20. Mai 17:25 .
> > drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
>
> As you can see, the chroots have an *empty* /dev subdir, but logging
> *still works*, apparently because the chrooted process just keeps using
> the system-central /dev/log it opened *before* chroot()ing.

I think that works specifically because *no* new process is created
when using internal-sftp as opposed to executing the sftp-server binary.

If syslog-ng can be made to reliably route internal-sftp messages to
user-specific log files then this approach would work well in Hildegard's
setup and would be a lot more pleasant than messing with LD_PRELOAD.


//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 9/26/2021 3:20 PM, Peter Stuge wrote:
> Jochen Bern wrote:
>> OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
>> have no idea how portable it is to other distribs, so just give it a try:
>>
>>> # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
>>> Subsystem sftp internal-sftp
>>> Match group mandanten
>>> ForceCommand internal-sftp -l INFO -u 0077
>>
>>
>>> # ls -al ~lvinq4/dev
>>> insgesamt 0
>>> drwx--x---. 2 root mandanten 6 20. Mai 17:25 .
>>> drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
>>
>> As you can see, the chroots have an *empty* /dev subdir, but logging
>> *still works*, apparently because the chrooted process just keeps using
>> the system-central /dev/log it opened *before* chroot()ing.
>
> I think that works specifically because *no* new process is created
> when using internal-sftp as opposed to executing the sftp-server binary.
>

It is not clear from Hildegard's email what is in the user's home directory
and if the sftp server can run in interactive mode. i.e. if a shell is started,
and if it can run any commands that may require logging.

OpenSSH sshd_config man page discusses ChrootDirectory says: "For an interactive session this requires at least a shell"
and also says: "sessions which use logging may require /dev/log inside the chroot directory on some operating systems."

So I would say the /dev/log or logs are required.

Based on the problem in original note:

> So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
>
> If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.

Logging works on the server that started syslog-ng. It sounds like syslog-ng opens a "unix-stream" socket.
She also says: "for every user has a log device /var/data/chroot/<username>/dev/log"
(not sure what it changes, it may save the socket handle/index (or whatever it is) to be used in that server's kernel.)

Each server is overwriting the /dev/log in the chroot directories.

So if interactive is not required, then no chroot /dev/log would be needed and sftp-initeral could work.
But it sounds like there is a need to share these NFS volumes

So we are back to using bind-mounting as it is done in the kernel and NFS version of /dev/log is not touched.

OR

Each server has its own /dev/log_<hostname> and the syslog-ng files on each server do the same
and use LD_PRELOAD= as proposed by Peter Stuge. I have used the LD_PRELOAD before when running
valgrind and/or gdb to not unload modules so as the debug tables stay available,


> If syslog-ng can be made to reliably route internal-sftp messages to
> user-specific log files then this approach would work well in Hildegard's
> setup and would be a lot more pleasant than messing with LD_PRELOAD.

As noted above, depends if any other process is stared that does logging is started after the
change to chroot.

>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hallo all, thank You all very much for your answers and suggestions.
Since the discussion has been shared between several mails now, I would like to try to summarize all up in one mail again here
and fill in the informations that where missing before.

We have 800 (eight hundred) sftp customers, each sftp customer has the same simple local Linux account on both sftp servers (simple entry in /etc/passwd and /etc/shadow etc.).
(Before we had only one sftp server, but for higher availability we want now to run two or more sftp servers parallel, accessed via TCP (sftp) load balancer)

Each customer account is in the group "sftp-cust", and only members of that group are allowed to login via sftp.
Each customer has it's chrooted home dir in /var/data/chroot/<username>/.

Here the (relevant part) of the sftpd config:

---------------------------------------------------------------
AllowGroups sftp-cust
Subsystem sftp internal-sftp -f LOCAL5 -l INFO
Match Group sftp-cust
ChrootDirectory %h
ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
AllowTcpForwarding no
---------------------------------------------------------------

Each sftp customer's sftp activity needs to be available in a dedicated log file for each sftp customer, for our support to be able to look into it. Here is an example of this log file:

Sep 28 15:54:25 myhostname internal-sftp[1618]: session opened for local user <username> from [1.2.3.4]
Sep 28 15:55:52 myhostname internal-sftp[27918]: remove name "/in/file.dat"
Sep 28 15:55:52 myhostname internal-sftp[27918]: sent status No such file
Sep 28 15:55:52 myhostname internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE mode 0666
Sep 28 15:55:52 myhostname internal-sftp[27918]: close "/in/file.dat" bytes read 0 written 2966
Sep 28 15:55:52 myhostname internal-sftp[27918]: set "/in/file.dat" modtime 20210928-15:55:46

If one does not use the /dev/log in the chroot environment (that is /var/data/chroot/<username>/dev/log absolute), you have a global sftpd log (I think in /var/log/messages on the server or something like that).
So a solution for the log problem would be, to not use the chroot logging, but to parse the global sftpd log (which is available on both sftpd servers locally for all local logins separated).
But I think this is not trivial to make this reliable and robust, since we need to parse the process id (sftpd session), check which username that session belongs to and write that log lines in the username specific log file.
Of course, process IDs are reused for different sessions, we have overlapping sessions etc. But if there is no other solution, than we need to try that. Maybe someone has already written such a session tracking parser.
Does somebody know of a log analyzer that can do this? A problem would be e.g. if the user never logs out, and the log file rotates daily, so after one day, the parser could not find any "session opened for local user xxx" log lines anymore.

If we want to try to keep using the available session chroot logging, man (8) sftp-server says:

"For logging to work, sftp-server must be able to access /dev/log. Use of sftp-server in a chroot configuration therefore requires that
syslogd(8) establish a logging socket inside the chroot directory."

As Peter has written here
https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html

as I understand it is hard coded in the system C library that the log devices name is /dev/log, so this cannot be changed (e.g. to log to chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2 on hostname2).

With the syslog-ng we use, we only need to create the directory /var/data/chroot/<username>/dev for each user once we create a new account.

We have the following syslog-ng config snippet file for each sftp user:

source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
destination d_sftp_<username> { file("/var/log/app/sftp/<username>.log"); };
log { source(s_chroot_<username>); destination(d_sftp_<username>); };

When starting syslog-ng the unix stream file /var/data/chroot/<username>/dev/log is created automatically by syslog-ng. If you delete it it gets recreated upon syslog-ng restart.

But the problem is that the last started syslog-ng aquires the lock for the NFS shared /var/data/chroot/<username>/dev/log so the other server cannot read it anymore and so there is no sftp log for the sessions on the other server.

I guess this is because syslog-ng creates /dev/log in the user's chroot directories as Unix stream socket (see https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/28#TOPIC-1209171).
This seems also to be called IPC socket (inter-process communication socket) or AF_UNIX socket.
"It is used in POSIX operating systems for inter-process communication. The correct standard POSIX term is POSIX Local IPC Sockets. Unix domain connections appear as byte streams, much like network connections, but all data _remains within the local computer_."

"It means that if you create a AF_UNIX socket on a NFS disk which is shared between two machines A and B, you cannot have a process on A writing data to the unix socket and a process on B reading data from that socket.
The communication happens at kernel level, and you can only transfer data among processes sitting in the same kernel."
(see source: https://stackoverflow.com/questions/783134/af-unix-domain-why-use-local-file-names-only )

Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share.

We need to stick with Ubuntu Linux as we have established management process only for this operating systems.

I hope I did not forget some information that was missing before. Thanks Hildegard



> Gesendet: Mittwoch, 22. September 2021 um 09:19 Uhr
> Von: "Damien Miller" <djm@mindrot.org>
> An: "Hildegard Meier" <daku8938@gmx.de>
> Cc: openssh-unix-dev@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> On Tue, 21 Sep 2021, Hildegard Meier wrote:
>
> > OpenSSH 5.9p1 + 7.6p1
> >
> > syslog-ng 3.3.4 + 3.13.2
> >
> > Hello, having an Ubuntu server with sftpd running where /var/data/chroot/ is an NFS mount from a remote central NFS server,
> > and each sftpd user's chroot home is /var/data/chroot/<username>/
> > and every user has a log device /var/data/chroot/<username>/dev/log which I read in successfully with syslog-ng:
> >
> > source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> > destination d_sftp_<username> { file("/var/log/sftp/<username>.log"); };
> > log { source(s_chroot_<username>); destination(d_sftp_<username>); };
> >
> > Now I have a second sftpd server in parallel, with the same user database and also mounts /var/data/chroot/ via NFS, and has the same syslog-ng config,
> > so every user can login on the one server or on the other. This is for high availability. This works so far.
> >
> > What is not working now is the sftpd logging: The sftp user's log is only available on one sftp server exclusively, and that is the one where syslog-ng was started least,
> > because as I understand it takes the exclusive unix socket file lock for each user's /dev/log.
> >
> > So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
> > But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
> >
> > If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.
> >
> > How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
>
> Right now there is no solution for this inside OpenSSH. There have been
> some proposals for post-auth logging to be proxied via the priviledged
> sshd monitor process but we haven't pursued them yet.
>
> Maybe someone with more Linux/NFS wit could suggest an OS-side solution
> for you?
>
> -d
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hallo Hildegard,

thanks for the summary.

Hildegard Meier wrote:
> ---------------------------------------------------------------
> AllowGroups sftp-cust
> Subsystem sftp internal-sftp -f LOCAL5 -l INFO
> Match Group sftp-cust
> ChrootDirectory %h
> ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
> AllowTcpForwarding no
> ---------------------------------------------------------------

This is very sensible configuration but it unfortunately makes the
LD_PRELOAD approach a bit less desirable. It will still work though.


> So a solution for the log problem would be, to not use the chroot
> logging, but to parse the global sftpd log
..
> But I think this is not trivial to make this reliable and robust,

Agreed. If going the log analysis path I would add -e to the
internal-sftp command line and run sshd with stderr connected to that
parser, to be able to keep track of events in a sensible way.
Log rotation would happen only after (or in) the parser.


> If we want to try to keep using the available session chroot
> logging, man (8) sftp-server says:
>
> "For logging to work, sftp-server must be able to access /dev/log. Use of sftp-server in a chroot configuration therefore requires that
> syslogd(8) establish a logging socket inside the chroot directory."
>
> As Peter has written here
> https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html
>
> as I understand it is hard coded in the system C library that the log
> devices name is /dev/log, so this cannot be changed (e.g. to log to
> chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2
> on hostname2).

Using LD_PRELOAD e.g. with the code I attached to that email allows
replacing the C library syslog functionality. My attachment does
indeed log to /dev/log_hostname1 on hostname1 instead of /dev/log.

However because of internal-sftp (which I still prefer over the alternative;
maintaining an sftp-server binary and possibly libraries in every homedir)
the entire sshd would have to be run with LD_PRELOAD.

Note that LD_PRELOAD is only neccessary on n-1 servers; the first
server can still use /dev/log and the full C library syslog() code.

If the second server where LD_PRELOAD is used does not need to
support anything besides SFTP, or if you can mandate that, then
it could be okay to run the entire sshd with LD_PRELOAD. Your IT
security colleagues may well object and they're not wrong; it's not
much code but still quite a mighty hammer.


> > Maybe someone with more Linux/NFS wit could suggest an OS-side solution
> > for you?

Before going down the LD_PRELOAD path I considered an overlayfs solution.

It would be nice, but the writable (upper) directory can't be NFS
with overlayfs so it can not work on the SFTP servers.

But using overlayfs can work on the NFS server. There would then be
one export for each SFTP server, all of them reaching the same
homedir contents except for different (empty) <homedir>/dev/ subdirs.

This requires maintaining a second homedir structure on the NFS server,
but that structure only needs to contain a homedir with one empty dev
subdir under and nothing else, so it may be doable. Homedirs need
correct permissions and only one of the two directory inodes for the
homedir itself will be changed by the respective SFTP server, but on
the plus side it requires only a single overlay mount, nothing per-user
and no LD_PRELOAD stuff on the SFTP servers.


On the NFS server, assuming that /home is where real homedirs exist
(this gets mounted directly by one SFTP server), that /var/home2 is
the second homedir structure with only homedirs and an empty dev subdir
in each, that /var/home2_work is an empty directory on the same filesystem
as /var/home2 and that /nfs/home2 is the second export mounted by the
second SFTP server, then set it up like so:

mount -t overlay overlay -olowerdir=/home,upperdir=/var/home2,workdir=/var/home2_work /nfs/home2


Then export /home and /nfs/home2 via NFS.

The SFTP server mouting /home will access /home/<user>/dev/log
(both syslog-ng and internal-sftp) and the SFTP server mouting
/nfs/home2 will instead access /var/home2/<user>/dev/log but
everything else within the homedir is to and from /home/<user>.

If this change can be done on the NFS server I think it's what I'd prefer.


Kind regards

//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

I like Peter's overlay concept. Ubuntu has aufs-tools to use builtin kernal code. These add mount.aufs

But here is another proposal.

You already have 800 NFs volumes and they are all mounted on each server.
(This is based on the syslog-ng configure which create a unix-stream in every volume when started.)

I assume that the only difference between these volumes is the user's home directory with
files they transfer to and from. And the user has only read access to whatever else is in the volume.
I assume they must have a /etc/passwd file with only one user.

Create 800 duplicate volumes on each server (not in NFS), but without the user's home directory.
for example /var/data/sftp/<username>/chroot/


modify the current 800 NFS volumes so they can be NFS mounted as the user's home in the above
new volumes.

Thus each sftp server uses the /var/data/sftp/<username>/chroot/dev/log which is on the same host
rather then a shared /dev/log. Its more complicated, but no local modifications.


On 9/28/2021 10:00 AM, Hildegard Meier wrote:
> Hallo all, thank You all very much for your answers and suggestions.
> Since the discussion has been shared between several mails now, I would like to try to summarize all up in one mail again here
> and fill in the informations that where missing before.
>
> We have 800 (eight hundred) sftp customers, each sftp customer has the same simple local Linux account on both sftp servers (simple entry in /etc/passwd and /etc/shadow etc.).
> (Before we had only one sftp server, but for higher availability we want now to run two or more sftp servers parallel, accessed via TCP (sftp) load balancer)
>
> Each customer account is in the group "sftp-cust", and only members of that group are allowed to login via sftp.
> Each customer has it's chrooted home dir in /var/data/chroot/<username>/.
>
> Here the (relevant part) of the sftpd config:
>
> ---------------------------------------------------------------
> AllowGroups sftp-cust
> Subsystem sftp internal-sftp -f LOCAL5 -l INFO
> Match Group sftp-cust
> ChrootDirectory %h
> ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
> AllowTcpForwarding no
> ---------------------------------------------------------------
>
> Each sftp customer's sftp activity needs to be available in a dedicated log file for each sftp customer, for our support to be able to look into it. Here is an example of this log file:
>
> Sep 28 15:54:25 myhostname internal-sftp[1618]: session opened for local user <username> from [1.2.3.4]
> Sep 28 15:55:52 myhostname internal-sftp[27918]: remove name "/in/file.dat"
> Sep 28 15:55:52 myhostname internal-sftp[27918]: sent status No such file
> Sep 28 15:55:52 myhostname internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE mode 0666
> Sep 28 15:55:52 myhostname internal-sftp[27918]: close "/in/file.dat" bytes read 0 written 2966
> Sep 28 15:55:52 myhostname internal-sftp[27918]: set "/in/file.dat" modtime 20210928-15:55:46
>
> If one does not use the /dev/log in the chroot environment (that is /var/data/chroot/<username>/dev/log absolute), you have a global sftpd log (I think in /var/log/messages on the server or something like that).
> So a solution for the log problem would be, to not use the chroot logging, but to parse the global sftpd log (which is available on both sftpd servers locally for all local logins separated).
> But I think this is not trivial to make this reliable and robust, since we need to parse the process id (sftpd session), check which username that session belongs to and write that log lines in the username specific log file.
> Of course, process IDs are reused for different sessions, we have overlapping sessions etc. But if there is no other solution, than we need to try that. Maybe someone has already written such a session tracking parser.
> Does somebody know of a log analyzer that can do this? A problem would be e.g. if the user never logs out, and the log file rotates daily, so after one day, the parser could not find any "session opened for local user xxx" log lines anymore.
>
> If we want to try to keep using the available session chroot logging, man (8) sftp-server says:
>
> "For logging to work, sftp-server must be able to access /dev/log. Use of sftp-server in a chroot configuration therefore requires that
> syslogd(8) establish a logging socket inside the chroot directory."
>
> As Peter has written here
> https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html
>
> as I understand it is hard coded in the system C library that the log devices name is /dev/log, so this cannot be changed (e.g. to log to chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2 on hostname2).
>
> With the syslog-ng we use, we only need to create the directory /var/data/chroot/<username>/dev for each user once we create a new account.
>
> We have the following syslog-ng config snippet file for each sftp user:
>
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> destination d_sftp_<username> { file("/var/log/app/sftp/<username>.log"); };
> log { source(s_chroot_<username>); destination(d_sftp_<username>); };
>
> When starting syslog-ng the unix stream file /var/data/chroot/<username>/dev/log is created automatically by syslog-ng. If you delete it it gets recreated upon syslog-ng restart.
>
> But the problem is that the last started syslog-ng aquires the lock for the NFS shared /var/data/chroot/<username>/dev/log so the other server cannot read it anymore and so there is no sftp log for the sessions on the other server.
>
> I guess this is because syslog-ng creates /dev/log in the user's chroot directories as Unix stream socket (see https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/28#TOPIC-1209171).
> This seems also to be called IPC socket (inter-process communication socket) or AF_UNIX socket.
> "It is used in POSIX operating systems for inter-process communication. The correct standard POSIX term is POSIX Local IPC Sockets. Unix domain connections appear as byte streams, much like network connections, but all data _remains within the local computer_."
>
> "It means that if you create a AF_UNIX socket on a NFS disk which is shared between two machines A and B, you cannot have a process on A writing data to the unix socket and a process on B reading data from that socket.
> The communication happens at kernel level, and you can only transfer data among processes sitting in the same kernel."
> (see source: https://stackoverflow.com/questions/783134/af-unix-domain-why-use-local-file-names-only )
>
> Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share.
>
> We need to stick with Ubuntu Linux as we have established management process only for this operating systems.
>
> I hope I did not forget some information that was missing before. Thanks Hildegard
>
>
>
>> Gesendet: Mittwoch, 22. September 2021 um 09:19 Uhr
>> Von: "Damien Miller" <djm@mindrot.org>
>> An: "Hildegard Meier" <daku8938@gmx.de>
>> Cc: openssh-unix-dev@mindrot.org
>> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>>
>> On Tue, 21 Sep 2021, Hildegard Meier wrote:
>>
>>> OpenSSH 5.9p1 + 7.6p1
>>>
>>> syslog-ng 3.3.4 + 3.13.2
>>>
>>> Hello, having an Ubuntu server with sftpd running where /var/data/chroot/ is an NFS mount from a remote central NFS server,
>>> and each sftpd user's chroot home is /var/data/chroot/<username>/
>>> and every user has a log device /var/data/chroot/<username>/dev/log which I read in successfully with syslog-ng:
>>>
>>> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
>>> destination d_sftp_<username> { file("/var/log/sftp/<username>.log"); };
>>> log { source(s_chroot_<username>); destination(d_sftp_<username>); };
>>>
>>> Now I have a second sftpd server in parallel, with the same user database and also mounts /var/data/chroot/ via NFS, and has the same syslog-ng config,
>>> so every user can login on the one server or on the other. This is for high availability. This works so far.
>>>
>>> What is not working now is the sftpd logging: The sftp user's log is only available on one sftp server exclusively, and that is the one where syslog-ng was started least,
>>> because as I understand it takes the exclusive unix socket file lock for each user's /dev/log.
>>>
>>> So, if a user logs in on the first server, where syslog-ng was started least, the user's sftp activity is logged on the first server.
>>> But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
>>>
>>> If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.
>>>
>>> How can I get the sftp user's activity be logged on each sftp server, when a user logs in to that server, while the user's home is shared on both servers via NFS?
>>
>> Right now there is no solution for this inside OpenSSH. There have been
>> some proposals for post-auth logging to be proxied via the priviledged
>> sshd monitor process but we haven't pursued them yet.
>>
>> Maybe someone with more Linux/NFS wit could suggest an OS-side solution
>> for you?
>>
>> -d
>>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 28.09.21 17:00, Hildegard Meier wrote:
> Each sftp customer's sftp activity needs to be available in a
> dedicated log file for each sftp customer, for our support to
> be able to look into it. Here is an example of this log file:
>
> Sep 28 15:54:25 myhostname internal-sftp[1618]:
> session opened for local user <username> from [1.2.3.4]

(General warning from me: There are log messages from the user's session
*earlier* than that, from before the client successfully authenticated
for the login as <username>. Having those messages go into a per-account
log(file) is nontrivial because sshd *does not yet know* what username
it will be, and if your support staff cannot see those lines, they will
not be able to debug *failed* connection attempts.)

At this point, I'd say that the best approach would be to have the
post-auth sftp-server process log to a *file* within the chroot, with
the filename set to the server's hostname (in the ForceCommand statement
of the server's /etc/ssh/sshd_config). Unfortunately, while sftp-server
has command line options for logging to syslog and for logging to
stderr, it does *not* have a command line option to log to a *file* (or,
for that matter, to set/change the "ident" parameter of openlog(3) to
include the username) ...

> you have a global sftpd log (I think in /var/log/messages on the
> server or something like that)

(RHEL, Centos, Fedora etc. tend to have a /var/log/messages . Ubuntu, in
my experience, uses /var/log/syslog instead.)

> But I think this is not trivial to make this reliable and robust,
> since we need to parse the process id (sftpd session), check which
> username that session belongs to and write that log lines in the
> username specific log file.

That's essentially what tools like opentelemetry.io are talking about
when they make a difference between "logs" and "traces", but I suppose
that throwing one of those at your problem could easily prove to be
overkill ...

https://opentelemetry.io/docs/concepts/data-sources/

> A problem would be e.g. if the user never logs out, and the log file
> rotates daily, so after one day, the parser could not find any
> "session opened for local user xxx" log lines anymore.

That's why log-parsing utilities that are worth their salt (logwatch,
ConSol's check_logfiles, etc.) can be configured to find and include the
rotated-away previous logfile instances ...

> as I understand it is hard coded in the system C library that the
> log devices name is /dev/log, so this cannot be changed

Well, not on Linux, at least ...

[.still fondly remembers AIX "softlinks" where you could "ln -s"
something to "/foo/$BAR/baz" and the value of the env var BAR would be
taken into account at access/expansion time]

> But the problem is that the last started syslog-ng aquires the lock
> for the NFS shared /var/data/chroot/<username>/dev/log so the other
> server cannot read it anymore and so there is no sftp log for the
> sessions on the other server.

(I *still* suspect that if only you could configure the syslogd's to use
a file locking method that just does *not* work across NFS shares -
there's about half a dozen different methods available, see, e.g.,

https://dovecot.org/pipermail/dovecot/2011-July/060149.html

-, you could circumvent that effect from the get-go ...)

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

Hildegard Meier wrote:
> But the problem is that the last started syslog-ng aquires the lock
> for the NFS shared /var/data/chroot/<username>/dev/log so the other
> server cannot read it anymore

Is it known what kind of lock this is? Was it investigated? Maybe on
the NFS server?


Douglas E Engert wrote:
> You already have 800 NFs volumes and they are all mounted on each server.

AIUI there's only one NFS export with all homedirs mounted on each server,
and avoiding per-user runtime setup such as mounts is a requirement.


Jochen Bern wrote:
> I *still* suspect that if only you could configure the syslogd's to use
> a file locking method that just does *not* work across NFS shares -
> there's about half a dozen different methods available, see, e.g.,
>
> https://dovecot.org/pipermail/dovecot/2011-July/060149.html
>
> -, you could circumvent that effect from the get-go ...

Looking through the afsocket module in syslog-ng it does no file locking.


I'm curious what kind of locking it is. Maybe the contention is
all within the NFS layer and could be overcome by setting a nolock
or local_lock mount option on the SFTP servers, if either option is
acceptable for the use case.


Kind regards

//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 9/28/2021 6:29 PM, Peter Stuge wrote:
> Hildegard Meier wrote:
>> But the problem is that the last started syslog-ng aquires the lock
>> for the NFS shared /var/data/chroot/<username>/dev/log so the other
>> server cannot read it anymore
>
> Is it known what kind of lock this is? Was it investigated? Maybe on
> the NFS server?

I suspect it is not a lock, but the "struct sockaddr_un" used with bind and connect or
some index into the server's kernal unix-stream like process and fd used in the bind is stored in the chrooted /dev/log when
the syslog-ng creates it. The would match the statement:
> So, if a user logs in on the first server, where syslog-ng was started least(last?), the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
>
> If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.

http://manpages.ubuntu.com/manpages/focal/man7/unix.7.html

Says: "The AF_UNIX (also known as AF_LOCAL) socket family is used to communicate between
processes on the same machine efficiently.

The "same machine" is the key.

netstat should show what sockets are in use.

>
>
> Douglas E Engert wrote:
>> You already have 800 NFs volumes and they are all mounted on each server.
>
> AIUI there's only one NFS export with all homedirs mounted on each server,
> and avoiding per-user runtime setup such as mounts is a requirement.
>
>
> Jochen Bern wrote:
>> I *still* suspect that if only you could configure the syslogd's to use
>> a file locking method that just does *not* work across NFS shares -
>> there's about half a dozen different methods available, see, e.g.,
>>
>> https://dovecot.org/pipermail/dovecot/2011-July/060149.html
>>
>> -, you could circumvent that effect from the get-go ...
>
> Looking through the afsocket module in syslog-ng it does no file locking.
>
>
> I'm curious what kind of locking it is. Maybe the contention is
> all within the NFS layer and could be overcome by setting a nolock
> or local_lock mount option on the SFTP servers, if either option is
> acceptable for the use case.
>
>
> Kind regards
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 29.09.21 02:04, Douglas E Engert wrote:
> On 9/28/2021 6:29 PM, Peter Stuge wrote:
>> Hildegard Meier wrote:
>>> But the problem is that the last started syslog-ng aquires the lock
>>> for the NFS shared /var/data/chroot/<username>/dev/log so the other
>>> server cannot read it anymore
>>
>> Is it known what kind of lock this is? Was it investigated? Maybe on
>> the NFS server?
>
> I suspect it is not a lock, but the "struct sockaddr_un" used with bind
> and connect or
> some index into the server's kernal unix-stream  like process and fd
> used in the bind is stored in the chrooted /dev/log when
> the syslog-ng creates it. The would match the statement:
>       > So, if a user logs in on the first server, where syslog-ng was
> started least(last?), the user's sftp activity is logged on the first
> server.
>     > But if the user logs in on the second server, it's sftp activity
> is not logged, neither on the second nor on the first server.
>     >
>     > If the syslog-ng is then restarted on the second server, the sftp
> user's activity is exclusively logged only on the second server and only
> for logins on the second server.

Somewhat different thought: If a newly-started syslogd on server A does
indeed REMOVE AND RECREATE the /dev/log sockets, then any user logging
into server B afterwards would find and open a socket that is not the
same (inode number) as the one server B's syslogd (created and) opened
when *it* started. Presto, implicit disconnect.

(The difference being that the socket itself would not have to store any
additional state about the open FDs pointing to it, which I doubt it has
any capacity for.)

If that theory holds, then all we'd need to do is to force both
syslogd's to use *pre-existing* sockets instead ... (And add a little
housekeeping background job that detects new users / chroots, creates
the matching socket, and HUP's both syslogd's so as to reopen them all.)

Regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

Hi, I tried now the following:

Add the following line to /etc/fstab:

/var/data/chroot /usr/local/sftp_chroot_bind_mount none bind 0 2

Then:
mkdir /usr/local/sftp_chroot_bind_mount
mount /usr/local/sftp_chroot_bind_mount

to bind-mount the users's chroot home to /usr/local/sftp_chroot_bind_mount

Then replace in the syslog-ng config:

source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
with
source s_chroot_<username> { unix-stream("/usr/local/sftp_chroot_bind_mount/<username>/dev/log" optional(yes) ); };

Restart syslog-ng

That would have been such a simple workaround.
But unfortunately, the problem is not solved with this.
The problem is the same, that there is sftp logging only on the sftp server where syslog-ng was restarted least.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Jochen, are you sure that you see the real sftp user session detailed activity log, e.g.

internal-sftp[27918]: session opened for local user <username> from [1.2.3.4]
internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE mode 0666

etc. and not just the sshd auth log, e.g.

sftpd[4772]: Accepted publickey for <username> from 1.2.3.4 port 45504 ssh2

? With Ubuntu 18.04.1 LTS and it's shipped OpenSSH 7.6p1-4ubuntu0.5 I can not get any sftp session logging when having only an empty /dev directory under the user's chroot directory, without it's "log" file.

I wonder if it would be a bug or a feature if you can manage to get sftp session logging without /dev/log in the sftp user's chroot dir?
What CentOS and OpenSSH version do you have exactly?
What could be the difference of the CentOS OpenSSH? Do you have special starting options?

Peter,
> I think that works specifically because *no* new process is created
> when using internal-sftp as opposed to executing the sftp-server binary.

For every sftp subsystem login (here with user "sftp_nagios"), I see a new sftpd process created:

First login:

ps auxww |grep sftp
root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp

Then logout and login again, second login:

ps auxww |grep sftp
root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
root 4630 0.5 0.1 74736 6596 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
sftp_na+ 4632 0.0 0.0 74736 3552 ? S 11:05 0:00 sftpd: sftp_nagios@notty
sftp_na+ 4633 0.0 0.0 74736 3164 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp


> Gesendet: Sonntag, 26. September 2021 um 22:20 Uhr
> Von: "Peter Stuge" <peter@stuge.se>
> An: openssh-unix-dev@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> Jochen Bern wrote:
> > OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
> > have no idea how portable it is to other distribs, so just give it a try:
> >
> > > # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
> > > Subsystem sftp internal-sftp
> > > Match group mandanten
> > > ForceCommand internal-sftp -l INFO -u 0077
> >
> >
> > > # ls -al ~lvinq4/dev
> > > insgesamt 0
> > > drwx--x---. 2 root mandanten 6 20. Mai 17:25 .
> > > drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
> >
> > As you can see, the chroots have an *empty* /dev subdir, but logging
> > *still works*, apparently because the chrooted process just keeps using
> > the system-central /dev/log it opened *before* chroot()ing.
>
> I think that works specifically because *no* new process is created
> when using internal-sftp as opposed to executing the sftp-server binary.
>
> If syslog-ng can be made to reliably route internal-sftp messages to
> user-specific log files then this approach would work well in Hildegard's
> setup and would be a lot more pleasant than messing with LD_PRELOAD.
>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

> Von: "Hildegard Meier" <daku8938@gmx.de>
> If one does not use the /dev/log in the chroot environment (that is /var/data/chroot/<username>/dev/log absolute), you have a global sftpd log (I think in /var/log/messages on the server or something like that).

Sorry I think this is not true, I think I confused this with the first situation when I did not fetch each sftp chrooted user's /dev/log with a specific syslog-ng source config,
but just the whole sftp session logs from all users wnet to facility LOCAL5 and I fetched facility LOCAL5 into one log file. But /dev/log in each sftp user's chroot dir was required for that, though.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Thanks Peter,

I think both the LD_PRELOAD and overlay fs approach are too complicated and hacky for me. Additionally, the user's chroot NFS share is exported by a proprietary NFS storage appliance (Tegile), and I would like to keep that as a simple nfs server, as we experienced already some stramnge behaviour with that appliance in the past, and I am happy it is working now stable finally.

> Gesendet: Dienstag, 28. September 2021 um 20:27 Uhr
> Von: "Peter Stuge" <peter@stuge.se>
> An: openssh-unix-dev@mindrot.org
> Cc: "Hildegard Meier" <daku8938@gmx.de>
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> Hallo Hildegard,
>
> thanks for the summary.
>
> Hildegard Meier wrote:
> > ---------------------------------------------------------------
> > AllowGroups sftp-cust
> > Subsystem sftp internal-sftp -f LOCAL5 -l INFO
> > Match Group sftp-cust
> > ChrootDirectory %h
> > ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
> > AllowTcpForwarding no
> > ---------------------------------------------------------------
>
> This is very sensible configuration but it unfortunately makes the
> LD_PRELOAD approach a bit less desirable. It will still work though.
>
>
> > So a solution for the log problem would be, to not use the chroot
> > logging, but to parse the global sftpd log
> ..
> > But I think this is not trivial to make this reliable and robust,
>
> Agreed. If going the log analysis path I would add -e to the
> internal-sftp command line and run sshd with stderr connected to that
> parser, to be able to keep track of events in a sensible way.
> Log rotation would happen only after (or in) the parser.
>
>
> > If we want to try to keep using the available session chroot
> > logging, man (8) sftp-server says:
> >
> > "For logging to work, sftp-server must be able to access /dev/log. Use of sftp-server in a chroot configuration therefore requires that
> > syslogd(8) establish a logging socket inside the chroot directory."
> >
> > As Peter has written here
> > https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html
> >
> > as I understand it is hard coded in the system C library that the log
> > devices name is /dev/log, so this cannot be changed (e.g. to log to
> > chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2
> > on hostname2).
>
> Using LD_PRELOAD e.g. with the code I attached to that email allows
> replacing the C library syslog functionality. My attachment does
> indeed log to /dev/log_hostname1 on hostname1 instead of /dev/log.
>
> However because of internal-sftp (which I still prefer over the alternative;
> maintaining an sftp-server binary and possibly libraries in every homedir)
> the entire sshd would have to be run with LD_PRELOAD.
>
> Note that LD_PRELOAD is only neccessary on n-1 servers; the first
> server can still use /dev/log and the full C library syslog() code.
>
> If the second server where LD_PRELOAD is used does not need to
> support anything besides SFTP, or if you can mandate that, then
> it could be okay to run the entire sshd with LD_PRELOAD. Your IT
> security colleagues may well object and they're not wrong; it's not
> much code but still quite a mighty hammer.
>
>
> > > Maybe someone with more Linux/NFS wit could suggest an OS-side solution
> > > for you?
>
> Before going down the LD_PRELOAD path I considered an overlayfs solution.
>
> It would be nice, but the writable (upper) directory can't be NFS
> with overlayfs so it can not work on the SFTP servers.
>
> But using overlayfs can work on the NFS server. There would then be
> one export for each SFTP server, all of them reaching the same
> homedir contents except for different (empty) <homedir>/dev/ subdirs.
>
> This requires maintaining a second homedir structure on the NFS server,
> but that structure only needs to contain a homedir with one empty dev
> subdir under and nothing else, so it may be doable. Homedirs need
> correct permissions and only one of the two directory inodes for the
> homedir itself will be changed by the respective SFTP server, but on
> the plus side it requires only a single overlay mount, nothing per-user
> and no LD_PRELOAD stuff on the SFTP servers.
>
>
> On the NFS server, assuming that /home is where real homedirs exist
> (this gets mounted directly by one SFTP server), that /var/home2 is
> the second homedir structure with only homedirs and an empty dev subdir
> in each, that /var/home2_work is an empty directory on the same filesystem
> as /var/home2 and that /nfs/home2 is the second export mounted by the
> second SFTP server, then set it up like so:
>
> mount -t overlay overlay -olowerdir=/home,upperdir=/var/home2,workdir=/var/home2_work /nfs/home2
>
>
> Then export /home and /nfs/home2 via NFS.
>
> The SFTP server mouting /home will access /home/<user>/dev/log
> (both syslog-ng and internal-sftp) and the SFTP server mouting
> /nfs/home2 will instead access /var/home2/<user>/dev/log but
> everything else within the homedir is to and from /home/<user>.
>
> If this change can be done on the NFS server I think it's what I'd prefer.
>
>
> Kind regards
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

On 29/9/21 6:54 pm, Hildegard Meier wrote:
>> Von: "Hildegard Meier"<daku8938@gmx.de>
>> If one does not use the/dev/log in the chroot environment (that is /var/data/chroot/<username>/dev/log absolute), you have a global sftpd log (I think in /var/log/messages on the server or something like that).
> Sorry I think this is not true, I think I confused this with the first situation when I did not fetch each sftp chrooted user's /dev/log with a specific syslog-ng source config,
> but just the whole sftp session logs from all users wnet to facility LOCAL5 and I fetched facility LOCAL5 into one log file. But /dev/log in each sftp user's chroot dir was required for that, though.

I don't think it should be this hard to fix.  I'd like to help.

I'm sorry if the questions have already been asked; if so, I missed it.

What is /dev/log on each server (not the one in chroot).  I.e. output of
ls -l /dev/log

As Peter Stuge said, if you're using internal-sftp, the log device would
be opened at start of sshd.  For external sftp, what is /dev/log in each
NFS chroot?  I.e. output of ls -l /var/data/chroot/dev/log

What OS are used on your sftp server (i.e. not the NFS server, and not
the sftp client machine.)

Regards,

David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

> Von: "Douglas E Engert" <deengert@gmail.com>
> You already have 800 NFs volumes and they are all mounted on each server.
> (This is based on the syslog-ng configure which create a unix-stream in every volume when started.)

I guess you have misunderstood this. /var/data/chroot/ is only exaclty _one_ NFS volume, which is simply mounted once.
All the sftp user's chrrot directories are under that NFS mount.
There are only 800 sftp config objects, each reads the /dev/log of each sftp user (/var/data/chroot/<username>/dev/log)

The directory structure is like this (for the user sftp_nagios):

drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 .
drwxr-xr-x+ 1143 root root 1145 Sep 23 15:07 ..
drwxr-x---+ 2 root sftp_nagios 3 Sep 29 11:26 dev
drwxr-xr-x+ 2 root root 3 Oct 31 2014 etc
dr-xr-x---+ 3 root sftp_nagios 3 Sep 10 09:19 in
dr-x------+ 2 sftp_nagios sftp_nagios 3 Sep 10 09:59 .ssh

ls -al /var/data/chroot/sftp_nagios/dev
total 4
drwxr-x---+ 2 root sftp_nagios 3 Sep 29 11:26 .
drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
srw-rw-rw-+ 1 root root 0 Sep 29 11:26 log

ls -al /var/data/chroot/sftp_nagios/etc/
total 6
drwxr-xr-x+ 2 root root 3 Oct 31 2014 .
drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
-rw-r--r--+ 1 root root 2309 Oct 31 2014 localtime

ls -al /var/data/chroot/sftp_nagios/in/test/
total 5
dr-xr-x---+ 3 root sftp_nagios 3 Sep 10 09:21 .
dr-xr-x---+ 3 root sftp_nagios 3 Sep 10 09:19 ..
drwxrwx--- 2 sftp_nagios sftp_nagios 5 Sep 21 16:47 nagios

(nagios does test uploads to it's chrroted dir /in/test/nagios/)

ls -al /var/data/chroot/sftp_nagios/.ssh/
total 4
dr-x------+ 2 sftp_nagios sftp_nagios 3 Sep 10 09:59 .
drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
-r--r-----+ 1 root sftp_nagios 401 Sep 10 09:30 authorized_keys

(this is for public key auth, in the future this shall be moved out of the user's chroot dir structure as it is unwanted that the users can change/view that file)
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

> Von: "Jochen Bern" <Jochen.Bern@binect.de>
> If a newly-started syslogd on server A does
> indeed REMOVE AND RECREATE the /dev/log sockets,

If /dev dir under sftp user's chroot dir exists but there is no "log" file in it, it gets created by syslog-ng.
It is never removed afterwards.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

>I don't think it should be this hard to fix.  I'd like to help.

Thanks David, your help is much appreciated.

>What is /dev/log on each server (not the one in chroot).  I.e. output of ls -l /dev/log

On the next generation sftp server (Ubuntu 18.04.1 LTS with it's shipped OpenSSH 7.6p1-4ubuntu0.5):
ls -l /dev/log
lrwxrwxrwx 1 root root 28 Aug 27 11:54 /dev/log -> /run/systemd/journal/dev-log

On the old (to be replaced with the new) sftp server (Ubuntu 12.04.5 LTS with it's shipped OpenSSH 5.9p1-5ubuntu1.10):
ls -l /dev/log
srw-rw-rw- 1 root root 0 Sep 23 15:07 /dev/log

>For external sftp, what is /dev/log in each NFS chroot?  I.e. output of ls -l /var/data/chroot/dev/log

Both above sftp servers (old and new generation) mount /var/data/chroot/ from a third NFS server appliance.
Every sftp user has it's (chroot) home dir /var/data/chroot/<username>/ under that single NFS mount.

E.g. for the user "sftp_nagios" this is

ls -l /var/data/chroot/sftp_nagios/dev/log
srw-rw-rw-+ 1 root root 0 Sep 29 11:26 /var/data/chroot/sftp_nagios/dev/log

Best regards
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 29.09.21 11:18, Hildegard Meier wrote:
> Jochen, are you sure that you see the real sftp user session detailed activity log, e.g.
>
> internal-sftp[27918]: session opened for local user <username> from [1.2.3.4]
> internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE mode 0666
>
> etc. and not just the sshd auth log, e.g.
>
> sftpd[4772]: Accepted publickey for <username> from 1.2.3.4 port 45504 ssh2

Considering that I'm the one who gets to debug both customers'
connectivity *and* concurrent-file-operations problems, I'm *quite* sure
of that. :-)

> sshd[27049]: Accepted publickey for [REDACTED] from [REDACTED] port 54343 ssh2: RSA SHA256:[REDACTED]
> sshd[27049]: pam_unix(sshd:session): session opened for user [REDACTED] by (uid=0)
> sshd[27049]: session opened for local user [REDACTED] from [REDACTED] [postauth]
> sshd[27049]: sent status No such file [postauth]
> sshd[27049]: sent status No such file [postauth]
> sshd[27049]: open "[REDACTED]" flags WRITE,CREATE,TRUNCATE mode 0666 [postauth]
> sshd[27049]: close "[REDACTED]" bytes read 0 written 5870358 [postauth]
> sshd[27049]: session closed for local user [REDACTED] from [REDACTED] [postauth]
> sshd[27049]: pam_unix(sshd:session): session closed for user [REDACTED]

- all from today's /var/log/messages .

> I wonder if it would be a bug or a feature if you can manage to get sftp
> session logging without /dev/log in the sftp user's chroot dir?

I'm in the dark whether that behavior is *intended* (and if so, by whom
exactly), hence my reluctance to openly recommend my setup to others ...

> What CentOS and OpenSSH version do you have exactly?

Current CentOS 7 with its genuine OpenSSH package
(openssh-7.4p1-21.el7.x86_64).

> Do you have special starting options?

Various hardened settings, but the only ones I'd *expect* to affect
*logging* in *any* way would be:

> SyslogFacility AUTHPRIV
> UsePAM yes # That's why there's messages from PAM in the log above
> UsePrivilegeSeparation sandbox
> Subsystem sftp internal-sftp
> Match group mandanten
> PermitTTY no
> ForceCommand internal-sftp -l INFO -u 0077


>> If a newly-started syslogd on server A does
>> indeed REMOVE AND RECREATE the /dev/log sockets,
>
> If /dev dir under sftp user's chroot dir exists but there is no "log" file in it, it gets created by syslog-ng.
> It is never removed afterwards.

If a .../dev/log is created within the .../dev/ directory *on the NFS
share*, and never removed, that means that all the .../dev/log's there
are were created *ONCE* by whichever syslogd got restarted *first* after
the user was created, correct? But still only the syslogd restarted
*last*, no matter whether on the same server or the other, gets that
user's log messages? I'm getting a murder mystery vibe here ...

Regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

On 29.09.21 11:54, Hildegard Meier wrote:
> ls -al /var/data/chroot/sftp_nagios/etc/
> total 6
> drwxr-xr-x+ 2 root root 3 Oct 31 2014 .
> drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
> -rw-r--r--+ 1 root root 2309 Oct 31 2014 localtime

(Semi-off-topic suggestion:

> # ls -al ~binect/etc
> insgesamt 8
> drwx--x---. 2 root mandanten 31 26. Jan 2018 .
> drwxr-x---. 5 root mandanten 62 4. Nov 2019 ..
> -rw-r-----. 1 root mandanten 24 26. Jan 2018 group
> -rw-r-----. 1 root mandanten 90 26. Jan 2018 passwd

> # grep . ~binect/etc/*
> /home/chroot/binect/etc/group:root:x:0:
> /home/chroot/binect/etc/group:users:x:[GID of group "mandanten"]:
> /home/chroot/binect/etc/passwd:root:x:0:0:root:/:/usr/sbin/nologin
> /home/chroot/binect/etc/passwd:binect:x:[UID of "binect"]:[GID of "mandanten"]:Mandant binect:/:/usr/sbin/nologin

- just so that the user's "ls -l" output is more readable than listing
raw UIDs and GIDs.)

> ls -al /var/data/chroot/sftp_nagios/.ssh/
> total 4
> dr-x------+ 2 sftp_nagios sftp_nagios 3 Sep 10 09:59 .
> drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
> -r--r-----+ 1 root sftp_nagios 401 Sep 10 09:30 authorized_keys
>
> (this is for public key auth, in the future this shall be moved out of the user's chroot dir structure as it is unwanted that the users can change/view that file)

Another suggestion:

> Match group mandanten
> AuthorizedKeysCommand /usr/local/sbin/MKLookup
> AuthorizedKeysCommandUser akcu


> # cat /usr/local/sbin/MKLookup
> #!/bin/sh
>
> MAIN_FILE="/etc/mand/pubkeys"
> MASTER_FILE="/etc/mand/masterkeys"
>
> MANDANT="$1"
> if [ "`echo $MANDANT | tr 'A-Za-z0-9-' _ | sed -e 's/^_*$/_/'`" != "_" ]; then
> # Unsupported characters in username. Refuse to work.
> exit 0
> fi
> if [ -r "$MAIN_FILE" ]; then
> grep '^ *#'"$MANDANT"'# *ssh-' "$MAIN_FILE" | sed -e 's/^ *#'"$MANDANT"'# *//'
> fi
> if [ -r "$MASTER_FILE" ]; then
> cat "$MASTER_FILE"
> fi
> exit 0


> # grep '^#binect#ssh-r.*Bern' /etc/mand/pubkeys | sed -e 's/ .* / ... /'
> #binect#ssh-rsa ... Jochen.Bern@Binect.de

(Making the entries' format so that they'd be *nonfunctional comments*
if they'd ever be read as normal authorized_keys lines is an extra
security precaution by paranoid /me ;-)

Regards,
--
Jochen Bern
Systemingenieur

T +49 6151 9067-231
F +49 6151 9067-290
E jochen.bern@binect.de
W www.binect.de


Binect GmbH
Robert-Koch-Str. 9
64331 Weiterstadt

Geschäftspost.Einfach.Digital.
Wir sind nach ISO/IEC 27001:2013 und 9001:2015 zertifiziert.
BMWi fördert digitale Lösungen für KMU.

Geschäftsführung: Dr. Frank Wermeyer, Michael Imiolczyk
Unternehmenssitz: Weiterstadt
Register: Amtsgericht Darmstadt, HRB 94685
Umsatzsteuer-ID: DE 221 302 264

Thanks Jochen,

> > sshd[27049]: Accepted publickey for [REDACTED] from [REDACTED] port 54343 ssh2: RSA SHA256:[REDACTED]
> > sshd[27049]: pam_unix(sshd:session): session opened for user [REDACTED] by (uid=0)
> > sshd[27049]: session opened for local user [REDACTED] from [REDACTED] [postauth]
> > sshd[27049]: sent status No such file [postauth]
> > sshd[27049]: sent status No such file [postauth]
> > sshd[27049]: open "[REDACTED]" flags WRITE,CREATE,TRUNCATE mode 0666 [postauth]
> > sshd[27049]: close "[REDACTED]" bytes read 0 written 5870358 [postauth]
> > sshd[27049]: session closed for local user [REDACTED] from [REDACTED] [postauth]
> > sshd[27049]: pam_unix(sshd:session): session closed for user [REDACTED]

Have all sftp log messages from today the prefix sshd[27049]? No other PIDs logged? I guess PID 27049 is the PID of the main sftpd(sshs) process?

> > SyslogFacility AUTHPRIV
> > UsePAM yes # That's why there's messages from PAM in the log above
> > UsePrivilegeSeparation sandbox
> > Subsystem sftp internal-sftp
> > Match group mandanten
> > PermitTTY no
> > ForceCommand internal-sftp -l INFO -u 0077
>

Sorry for this question, but just to be sure because you neither posted this here nor in your other post
https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039673.html
where you post the output of "egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config"

You have "ChrootDirectory" set in sshd_config, right?

E.g. I have set
ChrootDirectory %h


> If a .../dev/log is created within the .../dev/ directory *on the NFS
> share*, and never removed, that means that all the .../dev/log's there
> are were created *ONCE* by whichever syslogd got restarted *first* after
> the user was created, correct? But still only the syslogd restarted
> *last*, no matter whether on the same server or the other, gets that
> user's log messages?

That's correct

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 29.09.21 13:27, Hildegard Meier wrote:
>>> sshd[27049]: Accepted publickey for [REDACTED] from [REDACTED] port 54343 ssh2: RSA SHA256:[REDACTED]
>>> sshd[27049]: pam_unix(sshd:session): session opened for user [REDACTED] by (uid=0)
>>> sshd[27049]: session opened for local user [REDACTED] from [REDACTED] [postauth]
>>> sshd[27049]: sent status No such file [postauth]
>>> sshd[27049]: sent status No such file [postauth]
>>> sshd[27049]: open "[REDACTED]" flags WRITE,CREATE,TRUNCATE mode 0666 [postauth]
>>> sshd[27049]: close "[REDACTED]" bytes read 0 written 5870358 [postauth]
>>> sshd[27049]: session closed for local user [REDACTED] from [REDACTED] [postauth]
>>> sshd[27049]: pam_unix(sshd:session): session closed for user [REDACTED]
>
> Have all sftp log messages from today the prefix sshd[27049]?

No, the PID changes from login to login (and the master "/usr/sbin/sshd
-D"'s PID does not show up at all).

(Did I say in previous posts that I took this from /var/log/messages ?
Of course *not*, thanks to:

>>> SyslogFacility AUTHPRIV

and CentOS' default syslogd configs, it's all in /var/log/secure .)

> Sorry for this question, but [...]
> You have "ChrootDirectory" set in sshd_config, right?
>
> E.g. I have set
> ChrootDirectory %h

There's a "ChrootDirectory %h" within the "Match group mandanten" block,
yes. I also see the cwd as "/" and the group ownerships as "users" in a
test login (while the GID is named "mandanten" in the *real* /etc/group
), so the chroot() definitely works. I'd be in HUGE trouble if it
didn't. :-3

Regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

Thanks for the info on the single NFS volume mount.

Another take on the direct bind mount:


If server1 did:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };

Then for each user server1 would also do (or have in fstab):
mount -bind /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server1

and server2 did:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
and also do this:
mount -move /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server2

This would allow each server to have its own /dev/log_host so they would not overwrite each others,
but when the libc went to write to the chrooted /dev/log, the kernel would write to /dev/log_serverX because
the direct bind is in the host's kernel.



I have not tried this.



On 9/29/2021 3:42 AM, Hildegard Meier wrote:
> Hi, I tried now the following:
>
> Add the following line to /etc/fstab:
>
> /var/data/chroot /usr/local/sftp_chroot_bind_mount none bind 0 2
>
> Then:
> mkdir /usr/local/sftp_chroot_bind_mount
> mount /usr/local/sftp_chroot_bind_mount
>
> to bind-mount the users's chroot home to /usr/local/sftp_chroot_bind_mount
>
> Then replace in the syslog-ng config:
>
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> with
> source s_chroot_<username> { unix-stream("/usr/local/sftp_chroot_bind_mount/<username>/dev/log" optional(yes) ); };
>
> Restart syslog-ng
>
> That would have been such a simple workaround.
> But unfortunately, the problem is not solved with this.
> The problem is the same, that there is sftp logging only on the sftp server where syslog-ng was restarted least.
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
> .
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Correction:


On 9/29/2021 8:34 AM, Douglas E Engert wrote:
> Thanks for the info on the single NFS volume mount.
>
> Another take on the direct bind mount:
>
>
> If server1 did:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
>
> Then for each user server1 would also do (or have in fstab):
> mount -bind /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server1
>
> and server2 did:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };

Should read:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server2" optional(yes) ); };



> and also do this:
> mount -move /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server2
>
> This would allow each server to have its own /dev/log_host so they would not overwrite each others,
> but when the libc went to write to the chrooted  /dev/log, the kernel would write to /dev/log_serverX because
> the direct bind is in the host's kernel.
>
>
>
> I have not tried this.
>
>
>
> On 9/29/2021 3:42 AM, Hildegard Meier wrote:
>> Hi, I tried now the following:
>>
>> Add the following line to /etc/fstab:
>>
>> /var/data/chroot                            /usr/local/sftp_chroot_bind_mount   none    bind                0       2
>>
>> Then:
>> mkdir /usr/local/sftp_chroot_bind_mount
>> mount /usr/local/sftp_chroot_bind_mount
>>
>> to bind-mount the users's chroot home to /usr/local/sftp_chroot_bind_mount
>>
>> Then replace in the syslog-ng config:
>>
>> source s_chroot_<username>    { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
>> with
>> source s_chroot_<username>    { unix-stream("/usr/local/sftp_chroot_bind_mount/<username>/dev/log" optional(yes) ); };
>>
>> Restart syslog-ng
>>
>> That would have been such a simple workaround.
>> But unfortunately, the problem is not solved with this.
>> The problem is the same, that there is sftp logging only on the sftp server where syslog-ng was restarted least.
>> _______________________________________________
>> openssh-unix-dev mailing list
>> openssh-unix-dev@mindrot.org
>> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>> .
>>
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

This, I believe, explains your problem and how to achieve your goal:

On 29/9/21 7:55 pm, Hildegard Meier wrote:
>> What is /dev/log on each server (not the one in chroot).  I.e. output of ls -l /dev/log
> On the next generation sftp server (Ubuntu 18.04.1 LTS with it's shipped OpenSSH 7.6p1-4ubuntu0.5):
> ls -l /dev/log
> lrwxrwxrwx 1 root root 28 Aug 27 11:54 /dev/log -> /run/systemd/journal/dev-log
>
> On the old (to be replaced with the new) sftp server (Ubuntu 12.04.5 LTS with it's shipped OpenSSH 5.9p1-5ubuntu1.10):
> ls -l /dev/log
> srw-rw-rw- 1 root root 0 Sep 23 15:07 /dev/log

On Linux, a process opens /dev/log when it first uses syslog(3). It
writes log messages there.  On your new server, /dev/log is a link to
/run/systemd/journal/dev-log, which is a UNIX domain socket.  On the old
server, /dev/log is /dev/log, which is also a UNIX domain socket.

The syslog daemon conventionally also opens /dev/log to read incoming
log messages.  That's how log messages get from arbitrary process to
syslog daemon: they communicate via the same UNIX domain socket.

>> For external sftp, what is /dev/log in each NFS chroot?  I.e. output of ls -l /var/data/chroot/dev/log
> Both above sftp servers (old and new generation) mount/var/data/chroot/ from a third NFS server appliance.
> Every sftp user has it's (chroot) home dir/var/data/chroot/<username>/ under that single NFS mount.
>
> E.g. for the user "sftp_nagios" this is
>
> ls -l /var/data/chroot/sftp_nagios/dev/log
> srw-rw-rw-+ 1 root root 0 Sep 29 11:26 /var/data/chroot/sftp_nagios/dev/log

When the syslog daemon starts, it needs to be able to open (in fact,
create) the same UNIX domain socket that the external sftp server will
write to.  When it starts!  That means, the socket that the external
sftp server will write to must be visible in the chroot environment and
also must be visible when syslog starts.

I'm imagining you have an NFS share to mount over /var/data/chroot on
both of your SFTP servers so that you can pre-populate an empty
directory for each sftp user.  I'm imagining that you NFS mount each
user's home directory inside that when the user logs in (i.e. connects
using sftp, in your case).

Finally, I infer that when a user connects with sftp, they get an
external sftp process chrooted to /var/data/chroot/<username> with a
/dev/log inside that. This can't work.  The syslog daemon on neither old
nor new server can open /var/data/chroot/<username>/dev/log at startup
because there is no such file at startup.

What you should do is create /var/data/chroot/{dev,home} on both
servers.  These are local directories, and not NFS mounts.  This allows
you to configure syslog daemon to read log messages from
/var/data/chroot/dev/log (wich syslog will create when it starts.)  You
can't NFS mount over /var/data/chroot because you've got two different
sftp servers, each running syslog daemon.  The server that starts second
will remove the /var/data/chroot/dev/log socket that the first one
created, and create it's own socket in that place.  The
/var/data/chroot/dev directory must be truly local on each machine.

You'll also have /var/data/chroot/lib/sftp-server (or
/var/data/chroot/lib/openssh/sftp-server) on each machine, so that sshd
can exec that inside the chroot when a user connects.  The user will be
in chroot at /var/data/chroot.

You mount the NFS share containing the empty directories for each user. 
You mount it at /var/data/chroot/home.

You dynamically mount the user's home directory at
/var/data/chroot/home/<username>.

When a user logs in using sftp, they're in a chroot at /var/data/chroot
with their home directory at /home/<username> (as they see it).

What you might not like about this is that while two different users are
logged on to the same server at the same time, their home directories
are also mounted and accessible at the same time.  Mitigate this by
chmod a=x on the NFS share that you'll mount over
/var/data/chroot/home.  This is security through obscurity: Users can't
see what's in home.  Also, something like chmod go= on each users
NFS-shared home directory.  This prevents anybody from writing in any
home directory other than their own.

Regards,

David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Thanks Douglas,

did not know abount "mount move" before, interesting.

But unfortunateley, it would be an impractical complicated mess to have this hundrets of extra mounts, which additionally are server specific, so at least for me this would not be a possible approach.

Additionally, reading the mount man page I doubt this would be possible, anyway. I guess /dev/log would not be available in the chroot anymore.
And it says "moving a mount residing under a shared mount is invalid and unsupported".
But as said, this is approach is impractical for me anyways.

"The move operation
Move a mounted tree to another place (atomically). The call is:

mount --move olddir newdir

This will cause the contents which previously appeared under olddir to now be accessible under newdir. The physical location of the files
is not changed. Note that olddir has to be a mountpoint.

Note also that moving a mount residing under a shared mount is invalid and unsupported. Use findmnt -o TARGET,PROPAGATION to see the cur?
rent propagation flags."


> Gesendet: Mittwoch, 29. September 2021 um 15:34 Uhr
> Von: "Douglas E Engert" <deengert@gmail.com>
> An: openssh-unix-dev@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> Thanks for the info on the single NFS volume mount.
>
> Another take on the direct bind mount:
>
>
> If server1 did:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
>
> Then for each user server1 would also do (or have in fstab):
> mount -bind /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server1
>
> and server2 did:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
> and also do this:
> mount -move /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server2
>
> This would allow each server to have its own /dev/log_host so they would not overwrite each others,
> but when the libc went to write to the chrooted /dev/log, the kernel would write to /dev/log_serverX because
> the direct bind is in the host's kernel.
>
>
>
> I have not tried this.
>
>
>
> On 9/29/2021 3:42 AM, Hildegard Meier wrote:
> > Hi, I tried now the following:
> >
> > Add the following line to /etc/fstab:
> >
> > /var/data/chroot /usr/local/sftp_chroot_bind_mount none bind 0 2
> >
> > Then:
> > mkdir /usr/local/sftp_chroot_bind_mount
> > mount /usr/local/sftp_chroot_bind_mount
> >
> > to bind-mount the users's chroot home to /usr/local/sftp_chroot_bind_mount
> >
> > Then replace in the syslog-ng config:
> >
> > source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> > with
> > source s_chroot_<username> { unix-stream("/usr/local/sftp_chroot_bind_mount/<username>/dev/log" optional(yes) ); };
> >
> > Restart syslog-ng
> >
> > That would have been such a simple workaround.
> > But unfortunately, the problem is not solved with this.
> > The problem is the same, that there is sftp logging only on the sftp server where syslog-ng was restarted least.
> > _______________________________________________
> > openssh-unix-dev mailing list
> > openssh-unix-dev@mindrot.org
> > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
> > .
> >
>
> --
>
> Douglas E. Engert <DEEngert@gmail.com>
>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Thanks David,

I think you describe the problem correctly, and your approach could work, but unfortunately it would be a way too complicated mess.

As I had written in my summary
https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039677.html

"Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share."
 
 Best regards
 

Gesendet: Mittwoch, 29. September 2021 um 17:00 Uhr
Von: "David Newall" <openssh@davidnewall.com>
An: "Hildegard Meier" <daku8938@gmx.de>
Cc: "Damien Miller" <djm@mindrot.org>, "OpenSSH List" <openssh-unix-dev@mindrot.org>
Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
Hi Hildegard,
This, I believe, explains your problem and how to achieve your goal:

On 29/9/21 7:55 pm, Hildegard Meier wrote:
What is /dev/log on each server (not the one in chroot).  I.e. output of ls -l /dev/log
On the next generation sftp server (Ubuntu 18.04.1 LTS with it's shipped OpenSSH 7.6p1-4ubuntu0.5):
ls -l /dev/log
lrwxrwxrwx 1 root root 28 Aug 27 11:54 /dev/log -> /run/systemd/journal/dev-log

On the old (to be replaced with the new) sftp server (Ubuntu 12.04.5 LTS with it's shipped OpenSSH 5.9p1-5ubuntu1.10):
ls -l /dev/log
srw-rw-rw- 1 root root 0 Sep 23 15:07 /dev/log
On Linux, a process opens /dev/log when it first uses syslog(3).  It writes log messages there.  On your new server, /dev/log is a link to /run/systemd/journal/dev-log, which is a UNIX domain socket.  On the old server, /dev/log is /dev/log, which is also a UNIX domain socket.
The syslog daemon conventionally also opens /dev/log to read incoming log messages.  That's how log messages get from arbitrary process to syslog daemon: they communicate via the same UNIX domain socket.
For external sftp, what is /dev/log in each NFS chroot?  I.e. output of ls -l /var/data/chroot/dev/log
Both above sftp servers (old and new generation) mount /var/data/chroot/ from a third NFS server appliance.
Every sftp user has it's (chroot) home dir /var/data/chroot/<username>/ under that single NFS mount.

E.g. for the user "sftp_nagios" this is

ls -l /var/data/chroot/sftp_nagios/dev/log
srw-rw-rw-+ 1 root root 0 Sep 29 11:26 /var/data/chroot/sftp_nagios/dev/log
When the syslog daemon starts, it needs to be able to open (in fact, create) the same UNIX domain socket that the external sftp server will write to.  When it starts!  That means, the socket that the external sftp server will write to must be visible in the chroot environment and also must be visible when syslog starts.
I'm imagining you have an NFS share to mount over /var/data/chroot on both of your SFTP servers so that you can pre-populate an empty directory for each sftp user.  I'm imagining that you NFS mount each user's home directory inside that when the user logs in (i.e. connects using sftp, in your case).
Finally, I infer that when a user connects with sftp, they get an external sftp process chrooted to /var/data/chroot/<username> with a /dev/log inside that.  This can't work.  The syslog daemon on neither old nor new server can open /var/data/chroot/<username>/dev/log at startup because there is no such file at startup.
What you should do is create /var/data/chroot/{dev,home} on both servers.  These are local directories, and not NFS mounts.  This allows you to configure syslog daemon to read log messages from /var/data/chroot/dev/log (wich syslog will create when it starts.)  You can't NFS mount over /var/data/chroot because you've got two different sftp servers, each running syslog daemon.  The server that starts second will remove the /var/data/chroot/dev/log socket that the first one created, and create it's own socket in that place.  The /var/data/chroot/dev directory must be truly local on each machine.
You'll also have /var/data/chroot/lib/sftp-server (or /var/data/chroot/lib/openssh/sftp-server) on each machine, so that sshd can exec that inside the chroot when a user connects.  The user will be in chroot at /var/data/chroot.
You mount the NFS share containing the empty directories for each user.  You mount it at /var/data/chroot/home.
You dynamically mount the user's home directory at /var/data/chroot/home/<username>.
When a user logs in using sftp, they're in a chroot at /var/data/chroot with their home directory at /home/<username> (as they see it).
What you might not like about this is that while two different users are logged on to the same server at the same time, their home directories are also mounted and accessible at the same time.  Mitigate this by chmod a=x on the NFS share that you'll mount over /var/data/chroot/home.  This is security through obscurity: Users can't see what's in home.  Also, something like chmod go= on each users NFS-shared home directory.  This prevents anybody from writing in any home directory other than their own.
Regards,
David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

On 30/9/21 1:01 am, Hildegard Meier wrote:
> Thanks David,
>
> I think you describe the problem correctly, and your approach could work, but unfortunately it would be a way too complicated mess.
>
> As I had written in my summary
> https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039677.html
>
> "Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share."

I may have explained myself poorly.  No bind mount is required. You need
one NFS mount over /var/data/chroot/home (yielding /home/<user> for all
possible users), and one dynamic mount of user's home directory when
they log in.  (Dynamically mounting users home directory is quite common
and is handled by automount.)

I might also have misunderstood your desire to automount home
directories.  Perhaps you don't want to do that, and have all user home
directories on the one NFS share, mounted over /var/data/chroot/home. 
If so, great, that's all you need: create /var/data/chroot (a local
directory containg dev, home and lib directories, as well as ssh-server
executable) and NFS mount the share containing all of your user
directories over /var/data/chroot/home.

If I've still not got it right, you might explain to me what is
complicated so that I can better understand.

Regards,

David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

Hildegard Meier wrote:
> > I think that works specifically because *no* new process is created
> > when using internal-sftp as opposed to executing the sftp-server binary.
>
> For every sftp subsystem login (here with user "sftp_nagios"), I
> see a new sftpd process created:
..
> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp

Thanks, yes, you're correct that there are new processes. I should
have written different, refering to sshd vs. sftp-server:

sshd creates one child process for each incoming TCP connection (4590)
and after authentication it creates another child process (4592) which
runs as the authenticated user to handle the session, and then the
sftp subsystem channel is handled by yet another child process (4593).

These processes all run (different parts of) sshd, they do not
execute any other program, when using internal-sftp not even 4593.

This is relevant because as mentioned there is then no point where
sshd executes the sftp-server binary, so also no opportunity to use
LD_PRELOAD only for SFTP. All of sftp-server is built into sshd,
specifically to enable internal-sftp, with the great benefit that no
sftp-server binary is needed inside the chroot.

My thought was still wrong though. It could have been relevant for
the syslog() socket because that is closed automatically on exec()
and not on fork(), but sshd re-opens the syslog() socket right after
fork() so the internal-sftp vs. sftp-server makes no difference for
logging, only for LD_PRELOAD convenience.


Hildegard Meier wrote:
> Jochen Bern wrote:
> > If a newly-started syslogd on server A does
> > indeed REMOVE AND RECREATE the /dev/log sockets,
>
> If /dev dir under sftp user's chroot dir exists but there is no "log"
> file in it, it gets created by syslog-ng. It is never removed afterwards.

I had forgotten: AF_UNIX bind() to a filesystem address only succeeds
if the socket inode does not already exist in the filesystem.

So syslog-ng does what all AF_UNIX socket servers must: it unlinks the
socket from the filesystem first, so that a new socket inode can be
created by bind().

Jochen identified the problem.


Jochen Bern wrote:
> (The difference being that the socket itself would not have to store any
> additional state about the open FDs pointing to it, which I doubt it has
> any capacity for.)

The inode itself doesn't store anything, the kernel just maps the
filesystem inode to its socket object. Since the inode created by
daemon B means nothing within the kernel on system A, connecting to
that inode now also visible on system A doesn't reach daemon A.


Jochen Bern wrote:
> If that theory holds, then all we'd need to do is to force both
> syslogd's to use *pre-existing* sockets instead ...

That's unfortunately not possible.


Hildegard Meier wrote:
> I think both the LD_PRELOAD and overlay fs approach are too
> complicated and hacky for me.

Fair enough, and overlayfs is only possible with a Linux NFS server.

There aren't very many other solutions:


* A patch to change SFTP logging

The attached patch adds an -E option to sftp-server (as exists for sshd)
to log to a specific file. With -E /sftp.log on the internal-sftp command
line in sshd_config on both servers they would then both log to a file in
the user's home directory. Beware of write contention on that log file
when the same user is connected to both servers.

Ensure that the file exists before using this configuration. Make file
ownership root:user and mode 620 if the user shouldn't be able to read it.

The patch is small, but until an equivalent is accepted in OpenSSH,
if ever, it requires maintaining a local package.

Note that as proposed this patch only works with a regular file, not
a syslog socket. Supporting a socket is of course possible but requires
a larger patch re-implementing much of syslog(). I can write that if
it would be accepted into OpenSSH.


* Add -e to the internal-sftp command line and create a custom parser

Because internal-sftp is still the sshd process this log message
parser must be connected already to the initial sshd stderr and must
handle anything that can be output on stderr by any sshd child process,
not just SFTP messages, making this alternative less attractive.


Kind regards

//Peter

Small clarification:

Hildegard Meier wrote:
> ps auxww |grep sftp
> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp

4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
I guess that this is a TLS FTP server.
Try ps fax to see process relationships.

The "sftpd:" prefix for the sshd processes may be related to SELinux;
nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.


//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

If 'internal-sftp' means that the SFTP server is just a forked child of sshd, would the 'sshd -e' flag work to send log messages from internal-sftp to stderr?

If so, it may be worth just sending all logging from sshd to stderr (sowie [daemontools]).

Someone with time and ingenuity on their hands may even be able to send the stderr output on to syslog-ng and dispense with any need to filter it or send it to a file....

If internal-sftp doesn't work that way for ${reasons}, it may be worth adding that functionality for this sort of use case.

____________________
[daemontools]: https://cr.yp.to/daemontools.html


--
jmk

> On Sep 29, 2021, at 19:44, Peter Stuge <peter@stuge.se> wrote:
>
> ?Small clarification:
>
> Hildegard Meier wrote:
>> ps auxww |grep sftp
>> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
>> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
>> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
>> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp
>
> 4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
> I guess that this is a TLS FTP server.
> Try ps fax to see process relationships.
>
> The "sftpd:" prefix for the sshd processes may be related to SELinux;
> nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.
>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

If 'internal-sftp' means that the SFTP server is just a forked child of sshd, would the 'sshd -e' flag work to send log messages from internal-sftp to stderr?

If so, it may be worth just sending all logging from sshd to stderr (sowie [daemontools]).

Someone with time and ingenuity on their hands may even be able to send the stderr output on to syslog-ng and dispense with any need to filter it or send it to a file....

If internal-sftp doesn't work that way for ${reasons}, it may be worth adding that functionality for this sort of use case.

____________________
[daemontools]: https://cr.yp.to/daemontools.html


--
jmk

> On Sep 29, 2021, at 19:44, Peter Stuge <peter@stuge.se> wrote:
>
> ?Small clarification:
>
> Hildegard Meier wrote:
>> ps auxww |grep sftp
>> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
>> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
>> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
>> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp
>
> 4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
> I guess that this is a TLS FTP server.
> Try ps fax to see process relationships.
>
> The "sftpd:" prefix for the sshd processes may be related to SELinux;
> nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.
>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

If 'internal-sftp' means that the SFTP server is just a forked child of sshd, would the 'sshd -e' flag work to send log messages from internal-sftp to stderr?

If so, it may be worth just sending all logging from sshd to stderr (sowie [daemontools]).

Someone with time and ingenuity on their hands may even be able to send the stderr output on to syslog-ng and dispense with any need to filter it or send it to a file....

If internal-sftp doesn't work that way for ${reasons}, it may be worth adding that functionality for this sort of use case.

____________________
[daemontools]: https://cr.yp.to/daemontools.html


--
jmk

> On Sep 29, 2021, at 19:44, Peter Stuge <peter@stuge.se> wrote:
>
> ?Small clarification:
>
> Hildegard Meier wrote:
>> ps auxww |grep sftp
>> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
>> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
>> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
>> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp
>
> 4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
> I guess that this is a TLS FTP server.
> Try ps fax to see process relationships.
>
> The "sftpd:" prefix for the sshd processes may be related to SELinux;
> nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.
>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

> 4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
> I guess that this is a TLS FTP server.
> Try ps fax to see process relationships.
>
> The "sftpd:" prefix for the sshd processes may be related to SELinux;
> nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.

Sorry Peter I should have explained this at once.

To have a clean seperation of the standard sshd service on port 22 and our sftp server that listens on another port,
and both have and shall have really nothing to do which each other,
I created a new sshd instance named "sftpd":

/usr/sbin/sftpd is a symlink to /usr/sbin/sshd

Copied /etc/ssh/ and /etc/default/ssh to /etc/sftpd/ and /etc/default/sftpd

Copied /lib/systemd/system/ssh.service to /etc/systemd/system/sftpd.service

end replaced in the copy all occurences of regex "sshd?" with sftpd

E.g.

[Service]
EnvironmentFile=-/etc/default/sftpd
ExecStartPre=/usr/sbin/sftpd -t
ExecStart=/usr/sbin/sftpd -D $SFTPD_OPTS -f /etc/sftpd/sftpd_config

Important for correct autostart of sftpd on boot is:
[Unit]
After=network.target auditd.service sshd.service

Because sftpd needs to be started after sshd because only sshd creates /run/sshd/
and that is needed by sftpd (hard compiled in sshd).

Pitfall:
When stopping sshd /run/sshd/ gets deleted and you get the following sftpd error:
fatal: Missing privilege separation directory: /run/sshd
and sftpd does not accept new connections anymore etc.


Here ps fax output :)

12981 ? Ss 0:02 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
18846 ? Ss 0:00 \_ sftpd: sftp_nagios [priv]
18848 ? S 0:00 \_ sftpd: sftp_nagios@notty
18849 ? Ss 0:00 \_ sftpd: sftp_nagios@internal-sftp

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildgard,

Hildegard Meier wrote:
> > 4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
>
> To have a clean seperation of the standard sshd service on port 22
> and our sftp server that listens on another port, and both have and
> shall have really nothing to do which each other, I created a new
> sshd instance named "sftpd":

I understand! This could actually make it much easier to use a local
build for the standalone sftp server. Does the patch idea seem viable?


> [Service]
> EnvironmentFile=-/etc/default/sftpd
> ExecStartPre=/usr/sbin/sftpd -t
> ExecStart=/usr/sbin/sftpd -D $SFTPD_OPTS -f /etc/sftpd/sftpd_config
>
> Important for correct autostart of sftpd on boot is:
> [Unit]
> After=network.target auditd.service sshd.service
>
> Because sftpd needs to be started after sshd because only sshd
> creates /run/sshd/ and that is needed by sftpd (hard compiled in sshd).

systemd creates /run/sshd for the sshd.service, it's not great that
the sftpd service needs to use the same path.

A local sshd build allows cleanly solving that as well.


These commands build an OpenSSH-8.8p1 sshd with my sftp-server patch
and the alternate privsep directory:

git clone https://anongit.mindrot.org/openssh.git &&
cd openssh &&
git checkout -b v88_sftplog V_8_8_P1 &&
wget -O 0001-sftplog.patch \
https://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20210930/9b40e2d4/attachment.bin &&
git am 0001-sftplog.patch &&
autoreconf -fi -I m4 &&
./configure --prefix=/usr/local/sftpd --with-privsep-path=/run/sftpd &&
make sshd

Thanks to --prefix this sshd will not interfere with anything installed
via package management. Place configuration and keys (or symlinks) in
/usr/local/sftpd/etc/ and remember PidFile /run/sftpd.pid in sshd_config.


In any case you can express the current dependency to systemd to avoid
errors:

> When stopping sshd /run/sshd/ gets deleted and you get the
> following sftpd error:
> fatal: Missing privilege separation directory: /run/sshd
> and sftpd does not accept new connections anymore etc.

Set BindsTo=sshd.service in the sftpd.service [Unit] section to
tell systemd that sftpd requires sshd, and should be stopped first
if sshd is being stopped.

Also create a Wants:

mkdir /etc/systemd/system/sshd.service.wants
ln -s ../sftpd.service /etc/systemd/system/sshd.service.wants/

..to tell systemd that it should try to start sftpd when sshd starts.

Keep After=sshd.service in sftpd.service.

With those changes you can stop and start sftpd while sshd continues
running like before, if you stop sshd then sftpd is now stopped first,
and if you start either sshd or sftpd when neither is running then
both are started in the right order.


> Here ps fax output :)

That makes sense now. :)


Kind regards

//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

I'm staring at this thread, and all of this complexity, and trying to
figure out "why are you hurting yourself doing this"? In the amount of
time burned trying to integrate hundreds of NFS mounts and chroot cage
configurations, you could have "vsftpd" to support FTPS rather than
SFTP, gotten clean isolated upload and download cages without having
to embed /dev/log in the chroot cages, enabled much simpler and stable
automounting, and spent the rest of the time on the lobbying effort
necessary to convince your user community to switch to FTPS enabled
clients.

Unless you have some compelling need for SSH key based access, why are
you spending these cycles on this work?
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Following up on my own thread.

I was able to get logging from sshd to stderr using 'sshd -e', but it appears as if internal-sftp does not send messages to stderr (or, at least, to the same file handle). With 'LogLevel VERBOSE', sshd emits expected messages about connection and accepting keys, then:

User child is on pid 1234
Starting session: subsystem'sftp' for user1 from 1.2.3.4 port 5678 id 0

It's not until closing the session that we get more:

Close session: user user1 from 1.2.3.4 port 5678 id 0
Received disconnect from 1.2.3.4 port 5678:12: disconnected by user
[...]

As expected, command-line flags in sshd_config for the internal-sftp command have no effect ('-e -l DEBUG', for example).

I haven't looked at the code to see whether there's an easy path here or not. Too bad, it seemed like a reasonable approach -- and could still be, if someone were to implement a reasonable "log-to-stderr" for internal sftp.

The only other approach that's seems at all reasonable to me is to add code and a config twiddle to sshd to send log messages directly to, say, an RFC5425 syslog-over-TLS server at a configurable address and port. That seems like a lot of code for not so much benefit.

--
jmk

> On Sep 29, 2021, at 20:44, Jim Knoble <jmknoble@pobox.com> wrote:
>
> ?
> If 'internal-sftp' means that the SFTP server is just a forked child of sshd, would the 'sshd -e' flag work to send log messages from internal-sftp to stderr?
>
> If so, it may be worth just sending all logging from sshd to stderr (sowie [daemontools]).
>
> Someone with time and ingenuity on their hands may even be able to send the stderr output on to syslog-ng and dispense with any need to filter it or send it to a file....
>
> If internal-sftp doesn't work that way for ${reasons}, it may be worth adding that functionality for this sort of use case.
>
> ____________________
> [daemontools]: https://cr.yp.to/daemontools.html
>
>
> --
> jmk
>
>>> On Sep 29, 2021, at 19:44, Peter Stuge <peter@stuge.se> wrote:
>>>
>> ?Small clarification:
>>
>> Hildegard Meier wrote:
>>> ps auxww |grep sftp
>>> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
>>> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
>>> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
>>> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp
>>
>> 4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
>> I guess that this is a TLS FTP server.
>> Try ps fax to see process relationships.
>>
>> The "sftpd:" prefix for the sshd processes may be related to SELinux;
>> nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.
>>
>>
>> //Peter
>> _______________________________________________
>> openssh-unix-dev mailing list
>> openssh-unix-dev@mindrot.org
>> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hello all, first let me thank you all for your input and suggestions.

I think I have found now a workaround for the problem.

> Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share.

It seems, that I cannot uphold this requirement of not having user-specific bind-over-mounts. (while the requirement that there is only exactly one nfs mount is still true).

My workaround is the following:

sudo mkdir /var/data/dev # Create a directroy under which user subdirectories are created

For very username <username> and the user's primary group <groupname> do the following:

sudo mkdir /var/data/dev/<username>
sudo chmod 550 /var/data/dev/<username> # This restrictive permission is a requirement I think
sudo chgrp <groupname> /var/data/dev/<username> # so the user can read the directory

So the new directory is exactly the same as the existing /var/data/chroot/<username>/dev directory (which is on the nfs mount /var/data/chroot/).

Then do
mount --bind /var/data/dev/<username> /var/data/chroot/<username>/dev

so /var/data/chroot/<username>/dev is now effectively local on the sftp server, not anymore on nfs mount.

Then change the syslog-ng config

from
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };

to
source s_chroot_<username> { unix-stream("/var/data/dev/<username>/log" optional(yes) ); };

(this is not strictly needed, but I think it's nice having syslog-ng definitely now only reading from local file, guaranteed not from nfs mount anymore)

I have tested this successfully with one user so far - whether the user logs in on the one sftp server or the other, syslog-ng can now log the sftp session on the affected sftp server.

While it is still a mess to have 800 (or in the future maybe 2000 or 3000) bind mounts which I very much would like to avoid, I think this workaround is acceptable because the impact is limited to the logging functionality and not the sftp service itself. Speaking, if there are problems with the bind mounts, the impact is only that there is no sftp session logging.
Also, this is very simple, uniform and clear.
And one has 100% reliable sftp session logging.
No need to change anything else of the well established production system.

I think I will create systemd unit files for every bind mount (I experienced systemd dependency cycles with bind mounts in /etc/fstab on Ubuntu18 boot before anyway, so systemd units are needed anyway).
Maybe one can define a systemd dependency so that when you unmount the nfs mount /var/data/chroot/, all the bind mounts under it are automatically unmounted before (sftpd needs to be stopped first of course, otherwise mount would be busy), and also the bind-mounts automatically be mounted after /var/dat/chroot is mounted again.

So there should not be much hassle with all those simple uniform bind mounts.

I think I will implement a nagios monitoring check that checks if for every username /var/data/chroot/<username>/dev is a mountpoint, so logging is assured.

According to https://serverfault.com/questions/102588/maximum-numbers-for-file-system-mounts-in-linux/927860#927860 the Linux Kernel mount maximum is 100 000 so hopefully 2000 or 3000 will not be a problem.

--
Possible alternative
I guess (not tested) an alternative could be to hardlink (so needs to be on the same (nfs) file system) every /var/data/chroot/<username>/dev/log to e.g. /var/data/chroot/general_dev_log (which would need access permission for all users of course)
and do
mount --bind /var/data/general_dev_log /var/data/chroot/general_dev_log

so every sftp session would log to the one and only log device (unix socket), again using the local bind mount.
So you could write all the sftp session logs of all users into one log file with syslog-ng.
Advantage:
- only one bind mount needed
- only one syslog-ng unix-stream configuration needed

Disadvantage:
You need to parse and filter the big one session log to create user-specific log files which only conatin the sessions for that user.
I have no experience with such tools and I think this filtering can not be 100% reliable, there will always be some cases where the filter will fail and log entries will be missing.

Best regards
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

> Does the patch idea seem viable?
> A local sshd build allows cleanly solving that as well.

Thanks Peter, but one reason for the new sftp-server cluster HA architecture (and therefore the central NFS mount for the users, leading to this problem)
was to be able to have maintenance of the single sftp servers without service outage, to be able to apply operating system security patches delivered by the distribution (Ubuntu in this case).
I have no capacity to follow the OpenSSH security issues myself an then if needed re-compile newer patched versions (and not even then apply your patch additionally every time to it :)
We do here industry production service and need to stick with vanilla distribution OpenSSH and leave the delivery security patches to Ubuntu.


> Set BindsTo=sshd.service in the sftpd.service [Unit] section to
> tell systemd that sftpd requires sshd, and should be stopped first
> if sshd is being stopped.
>
> Also create a Wants:
>
> mkdir /etc/systemd/system/sshd.service.wants
> ln -s ../sftpd.service /etc/systemd/system/sshd.service.wants/
>
> ..to tell systemd that it should try to start sftpd when sshd starts.
>
> Keep After=sshd.service in sftpd.service.
>

Thanks for the hint, I will look into it, I am not so experienced with systemd yet.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Sorry for answering this off topic post.

> you could have "vsftpd" to support FTPS rather than SFTP

In the past we offered FTPS to our industry clients, but my collegues which admister the old FTPS server told that FTPS would often be problematic with customers because of firewall/port/certificate issues or something like that (I do not know FTPS), so we migrate customers from FTPS to SFTP, which has simply one TCP port (and luckily, PKI is not established in practice with OpenSSH!).
(if customers insist to use FTPS they can use it on the old server, though)

> convince your user community to switch to FTPS enabled clients

Well we do not have a "user community" but industry clients, some of them big companies, so to change this is practically impossible.
The decision for SFTP was made ten years ago, and we need to support SFTP for decades now on :)

It took years to achieve the sftp service to become reasonably robust stable, there popped up many problems through the years which was then fixed, and with so many customers you experience ever new unexpected fancy client behaviour.
And then we have the forwarding from the sftp service to the backend processing and back, which is also not trivial.

Just to mention, 18 million files transferred via SFTP last month:
inbound 8 252 591
outbound 10 074 881

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

On 1/10/21 2:44 pm, Hildegard Meier wrote:
> mount --bind/var/data/dev/<username>/var/data/chroot/<username>/dev
>
> so/var/data/chroot/<username>/dev is now effectively local on the sftp server, not anymore on nfs mount.

That's not right.  This doesn't magically turn an NFS mount into a local
mount.  If /var/data/chroot/<username>/dev is NFS mounted,
/var/data/dev/<username> is also NFS mounted.

I believe I explained what you need to do but for some reason you didn't
like it, and when I asked why, you never replied.

To repeat myself:

1. Create /var/data/chroot/dev, /var/data/chroot/home and
/var/data/chroot/lib on both machines.

2. Configure your syslog daemon to read from /var/data/chroot/dev/log.

3. Configure SSH to chroot for the SFTP users to /var/data/chroot.

4. Mount your current NFS share which contains all 800 user directories
over /var/data/chroot/home.

For extra points, instead of step 4, although I think is the bit you
didn't like, while at the same time I think is the bit that you said you
really want, so there's cognitive dissonance for you:

4. Mount a NFS share which contains empty directories for all 800 users
over /var/data/chroot/home.

5. When a user logs in use automount to NSF mount their home directory
over /var/data/chroot/home/<username>.

This is not hard.

Regards,

David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

Hildegard Meier wrote:
> > Does the patch idea seem viable?
..
> I have no capacity to follow the OpenSSH security issues myself an
> then if needed re-compile newer patched versions (and not even then
> apply your patch additionally every time to it :)

Fair enough, although it's not too much effort..

OpenSSH has a few releases per year (including security updates), there's
a release announcement mailing list with no other traffic, and the build
commands I sent need only minimal adjustment for new version numbers in
the git checkout command - or remove the git checkout command to always
build the very latest code, then the commands stay identical, including
applying the patch. The patch is small so should apply cleanly for some
time.


> We do here industry production service and need to stick with
> vanilla distribution OpenSSH and leave the delivery security
> patches to Ubuntu.

Ack. I consider upstream releases to have higher quality, because
distributions usually ship OpenSSH with their own patches, which
every now and then introduce some problems.

And Ubuntu like all other distributions usually take maybe a day or two
even for critical updates, full new releases can take months or years,
so a local build on release day is much faster, but you are right:

A local build does require tracking releases and running commands other
than the standard update process, and while few packages are as easy to
build as OpenSSH I understand that it can still be out of scope.



> > BindsTo=sshd.service
> > mkdir /etc/systemd/system/sshd.service.wants
> > ln -s ../sftpd.service /etc/systemd/system/sshd.service.wants/
> > Keep After=sshd.service in sftpd.service.
>
> Thanks for the hint, I will look into it, I am not so experienced
> with systemd yet.

The systemd dependency data model is simple but very capable and allows
for neat yet precise configuration. I like it a lot. Can recommend.


Hildegard Meier wrote:
> My workaround is the following:
..
> mount --bind /var/data/dev/<username> /var/data/chroot/<username>/dev
>
> so /var/data/chroot/<username>/dev is now effectively local on the
> sftp server, not anymore on nfs mount.

This solution is perfect if per-user mounts are acceptable.


> Then change the syslog-ng config
>
> from
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
>
> to
> source s_chroot_<username> { unix-stream("/var/data/dev/<username>/log" optional(yes) ); };
>
> (this is not strictly needed, but I think it's nice having syslog-ng
> definitely now only reading from local file, guaranteed not from nfs
> mount anymore)

I agree, I think this is an important step for reliability.


> I think I will create systemd unit files for every bind mount
> (I experienced systemd dependency cycles with bind mounts in /etc/fstab
> on Ubuntu18 boot before anyway, so systemd units are needed anyway).

If users have local accounts then a generator[1] can create units
dynamically (on boot and on systemctl daemon-reload), otherwise units
must be created and deleted explicitly by user account management.


I've attached a suggested generator script that reads /etc/passwd and
creates units for all members of the group set at the top of the script.
The sshd_config you sent matches on group sftp-cust so I used that.

The generator can be tested in a temporary directory:

dir=$(mktemp -d); ./sftplog-generator.sh "$dir" && ls -lR "$dir"


When all is good copy the script to /etc/systemd/system-generators/
and after running systemctl daemon-reload the generated units will
be in /run/systemd/system-generators/ (never stored on disk) and have
all been loaded.

systemctl restart sftpd.service then mounts everything before
starting the new sftpd process.


> Maybe one can define a systemd dependency so that when you unmount
> the nfs mount /var/data/chroot/, all the bind mounts under it are
> automatically unmounted before (sftpd needs to be stopped first of
> course, otherwise mount would be busy),

Yes, but it requires using systemctl stop var-data-chroot.mount
to unmount. systemd doesn't know about unsuccessful attempts to
directly umount a busy /var/data/chroot, but when stopping the
mount unit it knows to first stop the dependency tree. sftpd can
be included there.


> and also the bind-mounts automatically be mounted after /var/dat/chroot
> is mounted again.

Yes, that's a great idea. So looking at the dependencies:

systemd implicitly[2] adds a Requires= dependency with After= ordering
for directories above the target of mount units, so the generated units
get this for /var/data/chroot automatically, ensuring that /var/data/chroot
is mounted first.

The generator creates units with Before=sftpd.service ordering, it creates
Wants= dependencies for var-data-chroot.mount on each bind mount using a
symlink in var-data-chroot.mount.wants/, and finally it creates
BindsTo= dependencies for sftpd.service on each bind mount using so-called
drop-in[3] files in sftpd.service.d/.

The Wants= symlinks automatically mount the bind mounts when /var/data/chroot
is mounted, and the BindsTo= drop-in files ensure that when sftpd.service
is started all bind mounts are mounted first, and also that if any bind
mount is unmounted then sftpd.service is stopped.

A good example of the dependency model capabilities!


> I think I will implement a nagios monitoring check that checks if
> for every username /var/data/chroot/<username>/dev is a mountpoint,
> so logging is assured.

Can do, but with the above dependencies systemd will never start sftpd
without all bind mounts and will stop sftpd if any bind mount is unmounted.

If stopping sftpd.service is too violent then change BindsTo=${unit}
to Requires=${unit} near the bottom of the generator, but add
BindsTo=var-data-chroot.mount to sftpd.service, otherwise sftpd will not
be stopped when stopping var-data-chroot.mount, causing that to fail.

I like how BindsTo= ensures that sftpd either works fully or not at all,
but that may not suit everyone.



> Possible alternative
> --
> I guess (not tested) an alternative could be to hardlink (so needs
> to be on the same (nfs) file system) every
> /var/data/chroot/<username>/dev/log

Unfortunately it doesn't work; the hardlinks must be (re)created after
syslog-ng creates the general socket, but syslog-ng A and syslog-ng B
will create different general socket inodes, the later will overwrite
the earlier, so it's back to square one.


> Disadvantage:
> You need to parse and filter the big one session log to create
> user-specific log files which only conatin the sessions for that user.
> I have no experience with such tools and I think this filtering can
> not be 100% reliable

Exactly right; sftp-server doesn't include the username in most
messages so it's difficult to reliably associate messages with a user.


Kind regards

//Peter

[1] https://www.freedesktop.org/software/systemd/man/systemd.generator.html
[2] https://www.freedesktop.org/software/systemd/man/systemd.mount.html
[3] https://www.freedesktop.org/software/systemd/man/systemd.unit.html

David,

David Newall wrote:
> I believe I explained what you need to do but for some reason you didn't
> like it, and when I asked why, you never replied.
..
> This is not hard.

I find your tone condescending, even if unintended, and I probably
wouldn't reply to it either. If you'd like to discuss that further
please email me off-list.


Have a nice weekend everyone

//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

All,

On 2/10/21 1:52 am, Peter Stuge wrote:
> David,
>
> David Newall wrote:
>> I believe I explained what you need to do but for some reason you didn't
>> like it, and when I asked why, you never replied.
> ..
>> This is not hard.
> I find your tone condescending, even if unintended

Apologies.  I didn't intend to be condescending.  I was trying to help
with what seems to me to be an already solved problem.

David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Can confirm this is working in production successfully now.

> Gesendet: Freitag, 01. Oktober 2021 um 06:14 Uhr
> Von: "Hildegard Meier" <daku8938@gmx.de>
> Cc: openssh-unix-dev@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> My workaround is the following:
>
> sudo mkdir /var/data/dev # Create a directroy under which user subdirectories are created
>
> For very username <username> and the user's primary group <groupname> do the following:
>
> sudo mkdir /var/data/dev/<username>
> sudo chmod 550 /var/data/dev/<username> # This restrictive permission is a requirement I think
> sudo chgrp <groupname> /var/data/dev/<username> # so the user can read the directory
>
> So the new directory is exactly the same as the existing /var/data/chroot/<username>/dev directory (which is on the nfs mount /var/data/chroot/).
>
> Then do
> mount --bind /var/data/dev/<username> /var/data/chroot/<username>/dev
>
> so /var/data/chroot/<username>/dev is now effectively local on the sftp server, not anymore on nfs mount.
>
> Then change the syslog-ng config
>
> from
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
>
> to
> source s_chroot_<username> { unix-stream("/var/data/dev/<username>/log" optional(yes) ); };
>
> (this is not strictly needed, but I think it's nice having syslog-ng definitely now only reading from local file, guaranteed not from nfs mount anymore)
>
> I have tested this successfully with one user so far - whether the user logs in on the one sftp server or the other, syslog-ng can now log the sftp session on the affected sftp server.
>
> While it is still a mess to have 800 (or in the future maybe 2000 or 3000) bind mounts which I very much would like to avoid, I think this workaround is acceptable because the impact is limited to the logging functionality and not the sftp service itself. Speaking, if there are problems with the bind mounts, the impact is only that there is no sftp session logging.
> Also, this is very simple, uniform and clear.
> And one has 100% reliable sftp session logging.
> No need to change anything else of the well established production system.
>
> I think I will create systemd unit files for every bind mount (I experienced systemd dependency cycles with bind mounts in /etc/fstab on Ubuntu18 boot before anyway, so systemd units are needed anyway).
> Maybe one can define a systemd dependency so that when you unmount the nfs mount /var/data/chroot/, all the bind mounts under it are automatically unmounted before (sftpd needs to be stopped first of course, otherwise mount would be busy), and also the bind-mounts automatically be mounted after /var/dat/chroot is mounted again.
>
> So there should not be much hassle with all those simple uniform bind mounts.
>
> I think I will implement a nagios monitoring check that checks if for every username /var/data/chroot/<username>/dev is a mountpoint, so logging is assured.
>
> According to https://serverfault.com/questions/102588/maximum-numbers-for-file-system-mounts-in-linux/927860#927860 the Linux Kernel mount maximum is 100 000 so hopefully 2000 or 3000 will not be a problem.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev