Mailing List Archive

On 9/28/2021 6:29 PM, Peter Stuge wrote:
> Hildegard Meier wrote:
>> But the problem is that the last started syslog-ng aquires the lock
>> for the NFS shared /var/data/chroot/<username>/dev/log so the other
>> server cannot read it anymore
>
> Is it known what kind of lock this is? Was it investigated? Maybe on
> the NFS server?

I suspect it is not a lock, but the "struct sockaddr_un" used with bind and connect or
some index into the server's kernal unix-stream like process and fd used in the bind is stored in the chrooted /dev/log when
the syslog-ng creates it. The would match the statement:
> So, if a user logs in on the first server, where syslog-ng was started least(last?), the user's sftp activity is logged on the first server.
> But if the user logs in on the second server, it's sftp activity is not logged, neither on the second nor on the first server.
>
> If the syslog-ng is then restarted on the second server, the sftp user's activity is exclusively logged only on the second server and only for logins on the second server.

http://manpages.ubuntu.com/manpages/focal/man7/unix.7.html

Says: "The AF_UNIX (also known as AF_LOCAL) socket family is used to communicate between
processes on the same machine efficiently.

The "same machine" is the key.

netstat should show what sockets are in use.

>
>
> Douglas E Engert wrote:
>> You already have 800 NFs volumes and they are all mounted on each server.
>
> AIUI there's only one NFS export with all homedirs mounted on each server,
> and avoiding per-user runtime setup such as mounts is a requirement.
>
>
> Jochen Bern wrote:
>> I *still* suspect that if only you could configure the syslogd's to use
>> a file locking method that just does *not* work across NFS shares -
>> there's about half a dozen different methods available, see, e.g.,
>>
>> https://dovecot.org/pipermail/dovecot/2011-July/060149.html
>>
>> -, you could circumvent that effect from the get-go ...
>
> Looking through the afsocket module in syslog-ng it does no file locking.
>
>
> I'm curious what kind of locking it is. Maybe the contention is
> all within the NFS layer and could be overcome by setting a nolock
> or local_lock mount option on the SFTP servers, if either option is
> acceptable for the use case.
>
>
> Kind regards
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 29.09.21 02:04, Douglas E Engert wrote:
> On 9/28/2021 6:29 PM, Peter Stuge wrote:
>> Hildegard Meier wrote:
>>> But the problem is that the last started syslog-ng aquires the lock
>>> for the NFS shared /var/data/chroot/<username>/dev/log so the other
>>> server cannot read it anymore
>>
>> Is it known what kind of lock this is? Was it investigated? Maybe on
>> the NFS server?
>
> I suspect it is not a lock, but the "struct sockaddr_un" used with bind
> and connect or
> some index into the server's kernal unix-stream  like process and fd
> used in the bind is stored in the chrooted /dev/log when
> the syslog-ng creates it. The would match the statement:
>       > So, if a user logs in on the first server, where syslog-ng was
> started least(last?), the user's sftp activity is logged on the first
> server.
>     > But if the user logs in on the second server, it's sftp activity
> is not logged, neither on the second nor on the first server.
>     >
>     > If the syslog-ng is then restarted on the second server, the sftp
> user's activity is exclusively logged only on the second server and only
> for logins on the second server.

Somewhat different thought: If a newly-started syslogd on server A does
indeed REMOVE AND RECREATE the /dev/log sockets, then any user logging
into server B afterwards would find and open a socket that is not the
same (inode number) as the one server B's syslogd (created and) opened
when *it* started. Presto, implicit disconnect.

(The difference being that the socket itself would not have to store any
additional state about the open FDs pointing to it, which I doubt it has
any capacity for.)

If that theory holds, then all we'd need to do is to force both
syslogd's to use *pre-existing* sockets instead ... (And add a little
housekeeping background job that detects new users / chroots, creates
the matching socket, and HUP's both syslogd's so as to reopen them all.)

Regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

Hi, I tried now the following:

Add the following line to /etc/fstab:

/var/data/chroot /usr/local/sftp_chroot_bind_mount none bind 0 2

Then:
mkdir /usr/local/sftp_chroot_bind_mount
mount /usr/local/sftp_chroot_bind_mount

to bind-mount the users's chroot home to /usr/local/sftp_chroot_bind_mount

Then replace in the syslog-ng config:

source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
with
source s_chroot_<username> { unix-stream("/usr/local/sftp_chroot_bind_mount/<username>/dev/log" optional(yes) ); };

Restart syslog-ng

That would have been such a simple workaround.
But unfortunately, the problem is not solved with this.
The problem is the same, that there is sftp logging only on the sftp server where syslog-ng was restarted least.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Jochen, are you sure that you see the real sftp user session detailed activity log, e.g.

internal-sftp[27918]: session opened for local user <username> from [1.2.3.4]
internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE mode 0666

etc. and not just the sshd auth log, e.g.

sftpd[4772]: Accepted publickey for <username> from 1.2.3.4 port 45504 ssh2

? With Ubuntu 18.04.1 LTS and it's shipped OpenSSH 7.6p1-4ubuntu0.5 I can not get any sftp session logging when having only an empty /dev directory under the user's chroot directory, without it's "log" file.

I wonder if it would be a bug or a feature if you can manage to get sftp session logging without /dev/log in the sftp user's chroot dir?
What CentOS and OpenSSH version do you have exactly?
What could be the difference of the CentOS OpenSSH? Do you have special starting options?

Peter,
> I think that works specifically because *no* new process is created
> when using internal-sftp as opposed to executing the sftp-server binary.

For every sftp subsystem login (here with user "sftp_nagios"), I see a new sftpd process created:

First login:

ps auxww |grep sftp
root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp

Then logout and login again, second login:

ps auxww |grep sftp
root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
root 4630 0.5 0.1 74736 6596 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
sftp_na+ 4632 0.0 0.0 74736 3552 ? S 11:05 0:00 sftpd: sftp_nagios@notty
sftp_na+ 4633 0.0 0.0 74736 3164 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp


> Gesendet: Sonntag, 26. September 2021 um 22:20 Uhr
> Von: "Peter Stuge" <peter@stuge.se>
> An: openssh-unix-dev@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> Jochen Bern wrote:
> > OK. This is a behavior I see (and use) on a CentOS SFTP server, but I
> > have no idea how portable it is to other distribs, so just give it a try:
> >
> > > # egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config
> > > Subsystem sftp internal-sftp
> > > Match group mandanten
> > > ForceCommand internal-sftp -l INFO -u 0077
> >
> >
> > > # ls -al ~lvinq4/dev
> > > insgesamt 0
> > > drwx--x---. 2 root mandanten 6 20. Mai 17:25 .
> > > drwxr-x---. 5 root mandanten 54 24. Aug 15:38 ..
> >
> > As you can see, the chroots have an *empty* /dev subdir, but logging
> > *still works*, apparently because the chrooted process just keeps using
> > the system-central /dev/log it opened *before* chroot()ing.
>
> I think that works specifically because *no* new process is created
> when using internal-sftp as opposed to executing the sftp-server binary.
>
> If syslog-ng can be made to reliably route internal-sftp messages to
> user-specific log files then this approach would work well in Hildegard's
> setup and would be a lot more pleasant than messing with LD_PRELOAD.
>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

> Von: "Hildegard Meier" <daku8938@gmx.de>
> If one does not use the /dev/log in the chroot environment (that is /var/data/chroot/<username>/dev/log absolute), you have a global sftpd log (I think in /var/log/messages on the server or something like that).

Sorry I think this is not true, I think I confused this with the first situation when I did not fetch each sftp chrooted user's /dev/log with a specific syslog-ng source config,
but just the whole sftp session logs from all users wnet to facility LOCAL5 and I fetched facility LOCAL5 into one log file. But /dev/log in each sftp user's chroot dir was required for that, though.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Thanks Peter,

I think both the LD_PRELOAD and overlay fs approach are too complicated and hacky for me. Additionally, the user's chroot NFS share is exported by a proprietary NFS storage appliance (Tegile), and I would like to keep that as a simple nfs server, as we experienced already some stramnge behaviour with that appliance in the past, and I am happy it is working now stable finally.

> Gesendet: Dienstag, 28. September 2021 um 20:27 Uhr
> Von: "Peter Stuge" <peter@stuge.se>
> An: openssh-unix-dev@mindrot.org
> Cc: "Hildegard Meier" <daku8938@gmx.de>
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> Hallo Hildegard,
>
> thanks for the summary.
>
> Hildegard Meier wrote:
> > ---------------------------------------------------------------
> > AllowGroups sftp-cust
> > Subsystem sftp internal-sftp -f LOCAL5 -l INFO
> > Match Group sftp-cust
> > ChrootDirectory %h
> > ForceCommand internal-sftp -u 0002 -f LOCAL5 -l INFO
> > AllowTcpForwarding no
> > ---------------------------------------------------------------
>
> This is very sensible configuration but it unfortunately makes the
> LD_PRELOAD approach a bit less desirable. It will still work though.
>
>
> > So a solution for the log problem would be, to not use the chroot
> > logging, but to parse the global sftpd log
> ..
> > But I think this is not trivial to make this reliable and robust,
>
> Agreed. If going the log analysis path I would add -e to the
> internal-sftp command line and run sshd with stderr connected to that
> parser, to be able to keep track of events in a sensible way.
> Log rotation would happen only after (or in) the parser.
>
>
> > If we want to try to keep using the available session chroot
> > logging, man (8) sftp-server says:
> >
> > "For logging to work, sftp-server must be able to access /dev/log. Use of sftp-server in a chroot configuration therefore requires that
> > syslogd(8) establish a logging socket inside the chroot directory."
> >
> > As Peter has written here
> > https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039669.html
> >
> > as I understand it is hard coded in the system C library that the log
> > devices name is /dev/log, so this cannot be changed (e.g. to log to
> > chrooted /dev/log_hostname1 on hostname1 and chrooted /dev/log_hostname2
> > on hostname2).
>
> Using LD_PRELOAD e.g. with the code I attached to that email allows
> replacing the C library syslog functionality. My attachment does
> indeed log to /dev/log_hostname1 on hostname1 instead of /dev/log.
>
> However because of internal-sftp (which I still prefer over the alternative;
> maintaining an sftp-server binary and possibly libraries in every homedir)
> the entire sshd would have to be run with LD_PRELOAD.
>
> Note that LD_PRELOAD is only neccessary on n-1 servers; the first
> server can still use /dev/log and the full C library syslog() code.
>
> If the second server where LD_PRELOAD is used does not need to
> support anything besides SFTP, or if you can mandate that, then
> it could be okay to run the entire sshd with LD_PRELOAD. Your IT
> security colleagues may well object and they're not wrong; it's not
> much code but still quite a mighty hammer.
>
>
> > > Maybe someone with more Linux/NFS wit could suggest an OS-side solution
> > > for you?
>
> Before going down the LD_PRELOAD path I considered an overlayfs solution.
>
> It would be nice, but the writable (upper) directory can't be NFS
> with overlayfs so it can not work on the SFTP servers.
>
> But using overlayfs can work on the NFS server. There would then be
> one export for each SFTP server, all of them reaching the same
> homedir contents except for different (empty) <homedir>/dev/ subdirs.
>
> This requires maintaining a second homedir structure on the NFS server,
> but that structure only needs to contain a homedir with one empty dev
> subdir under and nothing else, so it may be doable. Homedirs need
> correct permissions and only one of the two directory inodes for the
> homedir itself will be changed by the respective SFTP server, but on
> the plus side it requires only a single overlay mount, nothing per-user
> and no LD_PRELOAD stuff on the SFTP servers.
>
>
> On the NFS server, assuming that /home is where real homedirs exist
> (this gets mounted directly by one SFTP server), that /var/home2 is
> the second homedir structure with only homedirs and an empty dev subdir
> in each, that /var/home2_work is an empty directory on the same filesystem
> as /var/home2 and that /nfs/home2 is the second export mounted by the
> second SFTP server, then set it up like so:
>
> mount -t overlay overlay -olowerdir=/home,upperdir=/var/home2,workdir=/var/home2_work /nfs/home2
>
>
> Then export /home and /nfs/home2 via NFS.
>
> The SFTP server mouting /home will access /home/<user>/dev/log
> (both syslog-ng and internal-sftp) and the SFTP server mouting
> /nfs/home2 will instead access /var/home2/<user>/dev/log but
> everything else within the homedir is to and from /home/<user>.
>
> If this change can be done on the NFS server I think it's what I'd prefer.
>
>
> Kind regards
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

On 29/9/21 6:54 pm, Hildegard Meier wrote:
>> Von: "Hildegard Meier"<daku8938@gmx.de>
>> If one does not use the/dev/log in the chroot environment (that is /var/data/chroot/<username>/dev/log absolute), you have a global sftpd log (I think in /var/log/messages on the server or something like that).
> Sorry I think this is not true, I think I confused this with the first situation when I did not fetch each sftp chrooted user's /dev/log with a specific syslog-ng source config,
> but just the whole sftp session logs from all users wnet to facility LOCAL5 and I fetched facility LOCAL5 into one log file. But /dev/log in each sftp user's chroot dir was required for that, though.

I don't think it should be this hard to fix.  I'd like to help.

I'm sorry if the questions have already been asked; if so, I missed it.

What is /dev/log on each server (not the one in chroot).  I.e. output of
ls -l /dev/log

As Peter Stuge said, if you're using internal-sftp, the log device would
be opened at start of sshd.  For external sftp, what is /dev/log in each
NFS chroot?  I.e. output of ls -l /var/data/chroot/dev/log

What OS are used on your sftp server (i.e. not the NFS server, and not
the sftp client machine.)

Regards,

David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

> Von: "Douglas E Engert" <deengert@gmail.com>
> You already have 800 NFs volumes and they are all mounted on each server.
> (This is based on the syslog-ng configure which create a unix-stream in every volume when started.)

I guess you have misunderstood this. /var/data/chroot/ is only exaclty _one_ NFS volume, which is simply mounted once.
All the sftp user's chrrot directories are under that NFS mount.
There are only 800 sftp config objects, each reads the /dev/log of each sftp user (/var/data/chroot/<username>/dev/log)

The directory structure is like this (for the user sftp_nagios):

drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 .
drwxr-xr-x+ 1143 root root 1145 Sep 23 15:07 ..
drwxr-x---+ 2 root sftp_nagios 3 Sep 29 11:26 dev
drwxr-xr-x+ 2 root root 3 Oct 31 2014 etc
dr-xr-x---+ 3 root sftp_nagios 3 Sep 10 09:19 in
dr-x------+ 2 sftp_nagios sftp_nagios 3 Sep 10 09:59 .ssh

ls -al /var/data/chroot/sftp_nagios/dev
total 4
drwxr-x---+ 2 root sftp_nagios 3 Sep 29 11:26 .
drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
srw-rw-rw-+ 1 root root 0 Sep 29 11:26 log

ls -al /var/data/chroot/sftp_nagios/etc/
total 6
drwxr-xr-x+ 2 root root 3 Oct 31 2014 .
drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
-rw-r--r--+ 1 root root 2309 Oct 31 2014 localtime

ls -al /var/data/chroot/sftp_nagios/in/test/
total 5
dr-xr-x---+ 3 root sftp_nagios 3 Sep 10 09:21 .
dr-xr-x---+ 3 root sftp_nagios 3 Sep 10 09:19 ..
drwxrwx--- 2 sftp_nagios sftp_nagios 5 Sep 21 16:47 nagios

(nagios does test uploads to it's chrroted dir /in/test/nagios/)

ls -al /var/data/chroot/sftp_nagios/.ssh/
total 4
dr-x------+ 2 sftp_nagios sftp_nagios 3 Sep 10 09:59 .
drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
-r--r-----+ 1 root sftp_nagios 401 Sep 10 09:30 authorized_keys

(this is for public key auth, in the future this shall be moved out of the user's chroot dir structure as it is unwanted that the users can change/view that file)
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

> Von: "Jochen Bern" <Jochen.Bern@binect.de>
> If a newly-started syslogd on server A does
> indeed REMOVE AND RECREATE the /dev/log sockets,

If /dev dir under sftp user's chroot dir exists but there is no "log" file in it, it gets created by syslog-ng.
It is never removed afterwards.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

>I don't think it should be this hard to fix.  I'd like to help.

Thanks David, your help is much appreciated.

>What is /dev/log on each server (not the one in chroot).  I.e. output of ls -l /dev/log

On the next generation sftp server (Ubuntu 18.04.1 LTS with it's shipped OpenSSH 7.6p1-4ubuntu0.5):
ls -l /dev/log
lrwxrwxrwx 1 root root 28 Aug 27 11:54 /dev/log -> /run/systemd/journal/dev-log

On the old (to be replaced with the new) sftp server (Ubuntu 12.04.5 LTS with it's shipped OpenSSH 5.9p1-5ubuntu1.10):
ls -l /dev/log
srw-rw-rw- 1 root root 0 Sep 23 15:07 /dev/log

>For external sftp, what is /dev/log in each NFS chroot?  I.e. output of ls -l /var/data/chroot/dev/log

Both above sftp servers (old and new generation) mount /var/data/chroot/ from a third NFS server appliance.
Every sftp user has it's (chroot) home dir /var/data/chroot/<username>/ under that single NFS mount.

E.g. for the user "sftp_nagios" this is

ls -l /var/data/chroot/sftp_nagios/dev/log
srw-rw-rw-+ 1 root root 0 Sep 29 11:26 /var/data/chroot/sftp_nagios/dev/log

Best regards
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 29.09.21 11:18, Hildegard Meier wrote:
> Jochen, are you sure that you see the real sftp user session detailed activity log, e.g.
>
> internal-sftp[27918]: session opened for local user <username> from [1.2.3.4]
> internal-sftp[27918]: open "/in/file.dat" flags WRITE,CREATE,TRUNCATE mode 0666
>
> etc. and not just the sshd auth log, e.g.
>
> sftpd[4772]: Accepted publickey for <username> from 1.2.3.4 port 45504 ssh2

Considering that I'm the one who gets to debug both customers'
connectivity *and* concurrent-file-operations problems, I'm *quite* sure
of that. :-)

> sshd[27049]: Accepted publickey for [REDACTED] from [REDACTED] port 54343 ssh2: RSA SHA256:[REDACTED]
> sshd[27049]: pam_unix(sshd:session): session opened for user [REDACTED] by (uid=0)
> sshd[27049]: session opened for local user [REDACTED] from [REDACTED] [postauth]
> sshd[27049]: sent status No such file [postauth]
> sshd[27049]: sent status No such file [postauth]
> sshd[27049]: open "[REDACTED]" flags WRITE,CREATE,TRUNCATE mode 0666 [postauth]
> sshd[27049]: close "[REDACTED]" bytes read 0 written 5870358 [postauth]
> sshd[27049]: session closed for local user [REDACTED] from [REDACTED] [postauth]
> sshd[27049]: pam_unix(sshd:session): session closed for user [REDACTED]

- all from today's /var/log/messages .

> I wonder if it would be a bug or a feature if you can manage to get sftp
> session logging without /dev/log in the sftp user's chroot dir?

I'm in the dark whether that behavior is *intended* (and if so, by whom
exactly), hence my reluctance to openly recommend my setup to others ...

> What CentOS and OpenSSH version do you have exactly?

Current CentOS 7 with its genuine OpenSSH package
(openssh-7.4p1-21.el7.x86_64).

> Do you have special starting options?

Various hardened settings, but the only ones I'd *expect* to affect
*logging* in *any* way would be:

> SyslogFacility AUTHPRIV
> UsePAM yes # That's why there's messages from PAM in the log above
> UsePrivilegeSeparation sandbox
> Subsystem sftp internal-sftp
> Match group mandanten
> PermitTTY no
> ForceCommand internal-sftp -l INFO -u 0077


>> If a newly-started syslogd on server A does
>> indeed REMOVE AND RECREATE the /dev/log sockets,
>
> If /dev dir under sftp user's chroot dir exists but there is no "log" file in it, it gets created by syslog-ng.
> It is never removed afterwards.

If a .../dev/log is created within the .../dev/ directory *on the NFS
share*, and never removed, that means that all the .../dev/log's there
are were created *ONCE* by whichever syslogd got restarted *first* after
the user was created, correct? But still only the syslogd restarted
*last*, no matter whether on the same server or the other, gets that
user's log messages? I'm getting a murder mystery vibe here ...

Regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

On 29.09.21 11:54, Hildegard Meier wrote:
> ls -al /var/data/chroot/sftp_nagios/etc/
> total 6
> drwxr-xr-x+ 2 root root 3 Oct 31 2014 .
> drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
> -rw-r--r--+ 1 root root 2309 Oct 31 2014 localtime

(Semi-off-topic suggestion:

> # ls -al ~binect/etc
> insgesamt 8
> drwx--x---. 2 root mandanten 31 26. Jan 2018 .
> drwxr-x---. 5 root mandanten 62 4. Nov 2019 ..
> -rw-r-----. 1 root mandanten 24 26. Jan 2018 group
> -rw-r-----. 1 root mandanten 90 26. Jan 2018 passwd

> # grep . ~binect/etc/*
> /home/chroot/binect/etc/group:root:x:0:
> /home/chroot/binect/etc/group:users:x:[GID of group "mandanten"]:
> /home/chroot/binect/etc/passwd:root:x:0:0:root:/:/usr/sbin/nologin
> /home/chroot/binect/etc/passwd:binect:x:[UID of "binect"]:[GID of "mandanten"]:Mandant binect:/:/usr/sbin/nologin

- just so that the user's "ls -l" output is more readable than listing
raw UIDs and GIDs.)

> ls -al /var/data/chroot/sftp_nagios/.ssh/
> total 4
> dr-x------+ 2 sftp_nagios sftp_nagios 3 Sep 10 09:59 .
> drwxr-x---+ 6 root sftp_nagios 6 Sep 28 17:09 ..
> -r--r-----+ 1 root sftp_nagios 401 Sep 10 09:30 authorized_keys
>
> (this is for public key auth, in the future this shall be moved out of the user's chroot dir structure as it is unwanted that the users can change/view that file)

Another suggestion:

> Match group mandanten
> AuthorizedKeysCommand /usr/local/sbin/MKLookup
> AuthorizedKeysCommandUser akcu


> # cat /usr/local/sbin/MKLookup
> #!/bin/sh
>
> MAIN_FILE="/etc/mand/pubkeys"
> MASTER_FILE="/etc/mand/masterkeys"
>
> MANDANT="$1"
> if [ "`echo $MANDANT | tr 'A-Za-z0-9-' _ | sed -e 's/^_*$/_/'`" != "_" ]; then
> # Unsupported characters in username. Refuse to work.
> exit 0
> fi
> if [ -r "$MAIN_FILE" ]; then
> grep '^ *#'"$MANDANT"'# *ssh-' "$MAIN_FILE" | sed -e 's/^ *#'"$MANDANT"'# *//'
> fi
> if [ -r "$MASTER_FILE" ]; then
> cat "$MASTER_FILE"
> fi
> exit 0


> # grep '^#binect#ssh-r.*Bern' /etc/mand/pubkeys | sed -e 's/ .* / ... /'
> #binect#ssh-rsa ... Jochen.Bern@Binect.de

(Making the entries' format so that they'd be *nonfunctional comments*
if they'd ever be read as normal authorized_keys lines is an extra
security precaution by paranoid /me ;-)

Regards,
--
Jochen Bern
Systemingenieur

T +49 6151 9067-231
F +49 6151 9067-290
E jochen.bern@binect.de
W www.binect.de


Binect GmbH
Robert-Koch-Str. 9
64331 Weiterstadt

Geschäftspost.Einfach.Digital.
Wir sind nach ISO/IEC 27001:2013 und 9001:2015 zertifiziert.
BMWi fördert digitale Lösungen für KMU.

Geschäftsführung: Dr. Frank Wermeyer, Michael Imiolczyk
Unternehmenssitz: Weiterstadt
Register: Amtsgericht Darmstadt, HRB 94685
Umsatzsteuer-ID: DE 221 302 264

Thanks Jochen,

> > sshd[27049]: Accepted publickey for [REDACTED] from [REDACTED] port 54343 ssh2: RSA SHA256:[REDACTED]
> > sshd[27049]: pam_unix(sshd:session): session opened for user [REDACTED] by (uid=0)
> > sshd[27049]: session opened for local user [REDACTED] from [REDACTED] [postauth]
> > sshd[27049]: sent status No such file [postauth]
> > sshd[27049]: sent status No such file [postauth]
> > sshd[27049]: open "[REDACTED]" flags WRITE,CREATE,TRUNCATE mode 0666 [postauth]
> > sshd[27049]: close "[REDACTED]" bytes read 0 written 5870358 [postauth]
> > sshd[27049]: session closed for local user [REDACTED] from [REDACTED] [postauth]
> > sshd[27049]: pam_unix(sshd:session): session closed for user [REDACTED]

Have all sftp log messages from today the prefix sshd[27049]? No other PIDs logged? I guess PID 27049 is the PID of the main sftpd(sshs) process?

> > SyslogFacility AUTHPRIV
> > UsePAM yes # That's why there's messages from PAM in the log above
> > UsePrivilegeSeparation sandbox
> > Subsystem sftp internal-sftp
> > Match group mandanten
> > PermitTTY no
> > ForceCommand internal-sftp -l INFO -u 0077
>

Sorry for this question, but just to be sure because you neither posted this here nor in your other post
https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039673.html
where you post the output of "egrep '^[^#]*( mand|sftp)' /etc/ssh/sshd_config"

You have "ChrootDirectory" set in sshd_config, right?

E.g. I have set
ChrootDirectory %h


> If a .../dev/log is created within the .../dev/ directory *on the NFS
> share*, and never removed, that means that all the .../dev/log's there
> are were created *ONCE* by whichever syslogd got restarted *first* after
> the user was created, correct? But still only the syslogd restarted
> *last*, no matter whether on the same server or the other, gets that
> user's log messages?

That's correct

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 29.09.21 13:27, Hildegard Meier wrote:
>>> sshd[27049]: Accepted publickey for [REDACTED] from [REDACTED] port 54343 ssh2: RSA SHA256:[REDACTED]
>>> sshd[27049]: pam_unix(sshd:session): session opened for user [REDACTED] by (uid=0)
>>> sshd[27049]: session opened for local user [REDACTED] from [REDACTED] [postauth]
>>> sshd[27049]: sent status No such file [postauth]
>>> sshd[27049]: sent status No such file [postauth]
>>> sshd[27049]: open "[REDACTED]" flags WRITE,CREATE,TRUNCATE mode 0666 [postauth]
>>> sshd[27049]: close "[REDACTED]" bytes read 0 written 5870358 [postauth]
>>> sshd[27049]: session closed for local user [REDACTED] from [REDACTED] [postauth]
>>> sshd[27049]: pam_unix(sshd:session): session closed for user [REDACTED]
>
> Have all sftp log messages from today the prefix sshd[27049]?

No, the PID changes from login to login (and the master "/usr/sbin/sshd
-D"'s PID does not show up at all).

(Did I say in previous posts that I took this from /var/log/messages ?
Of course *not*, thanks to:

>>> SyslogFacility AUTHPRIV

and CentOS' default syslogd configs, it's all in /var/log/secure .)

> Sorry for this question, but [...]
> You have "ChrootDirectory" set in sshd_config, right?
>
> E.g. I have set
> ChrootDirectory %h

There's a "ChrootDirectory %h" within the "Match group mandanten" block,
yes. I also see the cwd as "/" and the group ownerships as "users" in a
test login (while the GID is named "mandanten" in the *real* /etc/group
), so the chroot() definitely works. I'd be in HUGE trouble if it
didn't. :-3

Regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

Thanks for the info on the single NFS volume mount.

Another take on the direct bind mount:


If server1 did:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };

Then for each user server1 would also do (or have in fstab):
mount -bind /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server1

and server2 did:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
and also do this:
mount -move /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server2

This would allow each server to have its own /dev/log_host so they would not overwrite each others,
but when the libc went to write to the chrooted /dev/log, the kernel would write to /dev/log_serverX because
the direct bind is in the host's kernel.



I have not tried this.



On 9/29/2021 3:42 AM, Hildegard Meier wrote:
> Hi, I tried now the following:
>
> Add the following line to /etc/fstab:
>
> /var/data/chroot /usr/local/sftp_chroot_bind_mount none bind 0 2
>
> Then:
> mkdir /usr/local/sftp_chroot_bind_mount
> mount /usr/local/sftp_chroot_bind_mount
>
> to bind-mount the users's chroot home to /usr/local/sftp_chroot_bind_mount
>
> Then replace in the syslog-ng config:
>
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> with
> source s_chroot_<username> { unix-stream("/usr/local/sftp_chroot_bind_mount/<username>/dev/log" optional(yes) ); };
>
> Restart syslog-ng
>
> That would have been such a simple workaround.
> But unfortunately, the problem is not solved with this.
> The problem is the same, that there is sftp logging only on the sftp server where syslog-ng was restarted least.
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
> .
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Correction:


On 9/29/2021 8:34 AM, Douglas E Engert wrote:
> Thanks for the info on the single NFS volume mount.
>
> Another take on the direct bind mount:
>
>
> If server1 did:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
>
> Then for each user server1 would also do (or have in fstab):
> mount -bind /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server1
>
> and server2 did:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };

Should read:
source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server2" optional(yes) ); };



> and also do this:
> mount -move /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server2
>
> This would allow each server to have its own /dev/log_host so they would not overwrite each others,
> but when the libc went to write to the chrooted  /dev/log, the kernel would write to /dev/log_serverX because
> the direct bind is in the host's kernel.
>
>
>
> I have not tried this.
>
>
>
> On 9/29/2021 3:42 AM, Hildegard Meier wrote:
>> Hi, I tried now the following:
>>
>> Add the following line to /etc/fstab:
>>
>> /var/data/chroot                            /usr/local/sftp_chroot_bind_mount   none    bind                0       2
>>
>> Then:
>> mkdir /usr/local/sftp_chroot_bind_mount
>> mount /usr/local/sftp_chroot_bind_mount
>>
>> to bind-mount the users's chroot home to /usr/local/sftp_chroot_bind_mount
>>
>> Then replace in the syslog-ng config:
>>
>> source s_chroot_<username>    { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
>> with
>> source s_chroot_<username>    { unix-stream("/usr/local/sftp_chroot_bind_mount/<username>/dev/log" optional(yes) ); };
>>
>> Restart syslog-ng
>>
>> That would have been such a simple workaround.
>> But unfortunately, the problem is not solved with this.
>> The problem is the same, that there is sftp logging only on the sftp server where syslog-ng was restarted least.
>> _______________________________________________
>> openssh-unix-dev mailing list
>> openssh-unix-dev@mindrot.org
>> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>> .
>>
>

--

Douglas E. Engert <DEEngert@gmail.com>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

This, I believe, explains your problem and how to achieve your goal:

On 29/9/21 7:55 pm, Hildegard Meier wrote:
>> What is /dev/log on each server (not the one in chroot).  I.e. output of ls -l /dev/log
> On the next generation sftp server (Ubuntu 18.04.1 LTS with it's shipped OpenSSH 7.6p1-4ubuntu0.5):
> ls -l /dev/log
> lrwxrwxrwx 1 root root 28 Aug 27 11:54 /dev/log -> /run/systemd/journal/dev-log
>
> On the old (to be replaced with the new) sftp server (Ubuntu 12.04.5 LTS with it's shipped OpenSSH 5.9p1-5ubuntu1.10):
> ls -l /dev/log
> srw-rw-rw- 1 root root 0 Sep 23 15:07 /dev/log

On Linux, a process opens /dev/log when it first uses syslog(3). It
writes log messages there.  On your new server, /dev/log is a link to
/run/systemd/journal/dev-log, which is a UNIX domain socket.  On the old
server, /dev/log is /dev/log, which is also a UNIX domain socket.

The syslog daemon conventionally also opens /dev/log to read incoming
log messages.  That's how log messages get from arbitrary process to
syslog daemon: they communicate via the same UNIX domain socket.

>> For external sftp, what is /dev/log in each NFS chroot?  I.e. output of ls -l /var/data/chroot/dev/log
> Both above sftp servers (old and new generation) mount/var/data/chroot/ from a third NFS server appliance.
> Every sftp user has it's (chroot) home dir/var/data/chroot/<username>/ under that single NFS mount.
>
> E.g. for the user "sftp_nagios" this is
>
> ls -l /var/data/chroot/sftp_nagios/dev/log
> srw-rw-rw-+ 1 root root 0 Sep 29 11:26 /var/data/chroot/sftp_nagios/dev/log

When the syslog daemon starts, it needs to be able to open (in fact,
create) the same UNIX domain socket that the external sftp server will
write to.  When it starts!  That means, the socket that the external
sftp server will write to must be visible in the chroot environment and
also must be visible when syslog starts.

I'm imagining you have an NFS share to mount over /var/data/chroot on
both of your SFTP servers so that you can pre-populate an empty
directory for each sftp user.  I'm imagining that you NFS mount each
user's home directory inside that when the user logs in (i.e. connects
using sftp, in your case).

Finally, I infer that when a user connects with sftp, they get an
external sftp process chrooted to /var/data/chroot/<username> with a
/dev/log inside that. This can't work.  The syslog daemon on neither old
nor new server can open /var/data/chroot/<username>/dev/log at startup
because there is no such file at startup.

What you should do is create /var/data/chroot/{dev,home} on both
servers.  These are local directories, and not NFS mounts.  This allows
you to configure syslog daemon to read log messages from
/var/data/chroot/dev/log (wich syslog will create when it starts.)  You
can't NFS mount over /var/data/chroot because you've got two different
sftp servers, each running syslog daemon.  The server that starts second
will remove the /var/data/chroot/dev/log socket that the first one
created, and create it's own socket in that place.  The
/var/data/chroot/dev directory must be truly local on each machine.

You'll also have /var/data/chroot/lib/sftp-server (or
/var/data/chroot/lib/openssh/sftp-server) on each machine, so that sshd
can exec that inside the chroot when a user connects.  The user will be
in chroot at /var/data/chroot.

You mount the NFS share containing the empty directories for each user. 
You mount it at /var/data/chroot/home.

You dynamically mount the user's home directory at
/var/data/chroot/home/<username>.

When a user logs in using sftp, they're in a chroot at /var/data/chroot
with their home directory at /home/<username> (as they see it).

What you might not like about this is that while two different users are
logged on to the same server at the same time, their home directories
are also mounted and accessible at the same time.  Mitigate this by
chmod a=x on the NFS share that you'll mount over
/var/data/chroot/home.  This is security through obscurity: Users can't
see what's in home.  Also, something like chmod go= on each users
NFS-shared home directory.  This prevents anybody from writing in any
home directory other than their own.

Regards,

David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Thanks Douglas,

did not know abount "mount move" before, interesting.

But unfortunateley, it would be an impractical complicated mess to have this hundrets of extra mounts, which additionally are server specific, so at least for me this would not be a possible approach.

Additionally, reading the mount man page I doubt this would be possible, anyway. I guess /dev/log would not be available in the chroot anymore.
And it says "moving a mount residing under a shared mount is invalid and unsupported".
But as said, this is approach is impractical for me anyways.

"The move operation
Move a mounted tree to another place (atomically). The call is:

mount --move olddir newdir

This will cause the contents which previously appeared under olddir to now be accessible under newdir. The physical location of the files
is not changed. Note that olddir has to be a mountpoint.

Note also that moving a mount residing under a shared mount is invalid and unsupported. Use findmnt -o TARGET,PROPAGATION to see the cur?
rent propagation flags."


> Gesendet: Mittwoch, 29. September 2021 um 15:34 Uhr
> Von: "Douglas E Engert" <deengert@gmail.com>
> An: openssh-unix-dev@mindrot.org
> Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
>
> Thanks for the info on the single NFS volume mount.
>
> Another take on the direct bind mount:
>
>
> If server1 did:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
>
> Then for each user server1 would also do (or have in fstab):
> mount -bind /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server1
>
> and server2 did:
> source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log_server1" optional(yes) ); };
> and also do this:
> mount -move /var/data/chroot/<username>/dev/log /var/data/chroot/<username>/dev/log_server2
>
> This would allow each server to have its own /dev/log_host so they would not overwrite each others,
> but when the libc went to write to the chrooted /dev/log, the kernel would write to /dev/log_serverX because
> the direct bind is in the host's kernel.
>
>
>
> I have not tried this.
>
>
>
> On 9/29/2021 3:42 AM, Hildegard Meier wrote:
> > Hi, I tried now the following:
> >
> > Add the following line to /etc/fstab:
> >
> > /var/data/chroot /usr/local/sftp_chroot_bind_mount none bind 0 2
> >
> > Then:
> > mkdir /usr/local/sftp_chroot_bind_mount
> > mount /usr/local/sftp_chroot_bind_mount
> >
> > to bind-mount the users's chroot home to /usr/local/sftp_chroot_bind_mount
> >
> > Then replace in the syslog-ng config:
> >
> > source s_chroot_<username> { unix-stream("/var/data/chroot/<username>/dev/log" optional(yes) ); };
> > with
> > source s_chroot_<username> { unix-stream("/usr/local/sftp_chroot_bind_mount/<username>/dev/log" optional(yes) ); };
> >
> > Restart syslog-ng
> >
> > That would have been such a simple workaround.
> > But unfortunately, the problem is not solved with this.
> > The problem is the same, that there is sftp logging only on the sftp server where syslog-ng was restarted least.
> > _______________________________________________
> > openssh-unix-dev mailing list
> > openssh-unix-dev@mindrot.org
> > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
> > .
> >
>
> --
>
> Douglas E. Engert <DEEngert@gmail.com>
>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Thanks David,

I think you describe the problem correctly, and your approach could work, but unfortunately it would be a way too complicated mess.

As I had written in my summary
https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039677.html

"Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share."
 
 Best regards
 

Gesendet: Mittwoch, 29. September 2021 um 17:00 Uhr
Von: "David Newall" <openssh@davidnewall.com>
An: "Hildegard Meier" <daku8938@gmx.de>
Cc: "Damien Miller" <djm@mindrot.org>, "OpenSSH List" <openssh-unix-dev@mindrot.org>
Betreff: Re: Howto log multiple sftpd instances with their chroot shared via NFS
Hi Hildegard,
This, I believe, explains your problem and how to achieve your goal:

On 29/9/21 7:55 pm, Hildegard Meier wrote:
What is /dev/log on each server (not the one in chroot).  I.e. output of ls -l /dev/log
On the next generation sftp server (Ubuntu 18.04.1 LTS with it's shipped OpenSSH 7.6p1-4ubuntu0.5):
ls -l /dev/log
lrwxrwxrwx 1 root root 28 Aug 27 11:54 /dev/log -> /run/systemd/journal/dev-log

On the old (to be replaced with the new) sftp server (Ubuntu 12.04.5 LTS with it's shipped OpenSSH 5.9p1-5ubuntu1.10):
ls -l /dev/log
srw-rw-rw- 1 root root 0 Sep 23 15:07 /dev/log
On Linux, a process opens /dev/log when it first uses syslog(3).  It writes log messages there.  On your new server, /dev/log is a link to /run/systemd/journal/dev-log, which is a UNIX domain socket.  On the old server, /dev/log is /dev/log, which is also a UNIX domain socket.
The syslog daemon conventionally also opens /dev/log to read incoming log messages.  That's how log messages get from arbitrary process to syslog daemon: they communicate via the same UNIX domain socket.
For external sftp, what is /dev/log in each NFS chroot?  I.e. output of ls -l /var/data/chroot/dev/log
Both above sftp servers (old and new generation) mount /var/data/chroot/ from a third NFS server appliance.
Every sftp user has it's (chroot) home dir /var/data/chroot/<username>/ under that single NFS mount.

E.g. for the user "sftp_nagios" this is

ls -l /var/data/chroot/sftp_nagios/dev/log
srw-rw-rw-+ 1 root root 0 Sep 29 11:26 /var/data/chroot/sftp_nagios/dev/log
When the syslog daemon starts, it needs to be able to open (in fact, create) the same UNIX domain socket that the external sftp server will write to.  When it starts!  That means, the socket that the external sftp server will write to must be visible in the chroot environment and also must be visible when syslog starts.
I'm imagining you have an NFS share to mount over /var/data/chroot on both of your SFTP servers so that you can pre-populate an empty directory for each sftp user.  I'm imagining that you NFS mount each user's home directory inside that when the user logs in (i.e. connects using sftp, in your case).
Finally, I infer that when a user connects with sftp, they get an external sftp process chrooted to /var/data/chroot/<username> with a /dev/log inside that.  This can't work.  The syslog daemon on neither old nor new server can open /var/data/chroot/<username>/dev/log at startup because there is no such file at startup.
What you should do is create /var/data/chroot/{dev,home} on both servers.  These are local directories, and not NFS mounts.  This allows you to configure syslog daemon to read log messages from /var/data/chroot/dev/log (wich syslog will create when it starts.)  You can't NFS mount over /var/data/chroot because you've got two different sftp servers, each running syslog daemon.  The server that starts second will remove the /var/data/chroot/dev/log socket that the first one created, and create it's own socket in that place.  The /var/data/chroot/dev directory must be truly local on each machine.
You'll also have /var/data/chroot/lib/sftp-server (or /var/data/chroot/lib/openssh/sftp-server) on each machine, so that sshd can exec that inside the chroot when a user connects.  The user will be in chroot at /var/data/chroot.
You mount the NFS share containing the empty directories for each user.  You mount it at /var/data/chroot/home.
You dynamically mount the user's home directory at /var/data/chroot/home/<username>.
When a user logs in using sftp, they're in a chroot at /var/data/chroot with their home directory at /home/<username> (as they see it).
What you might not like about this is that while two different users are logged on to the same server at the same time, their home directories are also mounted and accessible at the same time.  Mitigate this by chmod a=x on the NFS share that you'll mount over /var/data/chroot/home.  This is security through obscurity: Users can't see what's in home.  Also, something like chmod go= on each users NFS-shared home directory.  This prevents anybody from writing in any home directory other than their own.
Regards,
David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

On 30/9/21 1:01 am, Hildegard Meier wrote:
> Thanks David,
>
> I think you describe the problem correctly, and your approach could work, but unfortunately it would be a way too complicated mess.
>
> As I had written in my summary
> https://lists.mindrot.org/pipermail/openssh-unix-dev/2021-September/039677.html
>
> "Since we have 800 users, it would be impractical unrobust to use user-specifc e.g. bind) mounts (e.g. 800 bind-over-mounts). To keep it simple, clear and coherent, all user's homes must be on the same one singular NFS-Share."

I may have explained myself poorly.  No bind mount is required. You need
one NFS mount over /var/data/chroot/home (yielding /home/<user> for all
possible users), and one dynamic mount of user's home directory when
they log in.  (Dynamically mounting users home directory is quite common
and is handled by automount.)

I might also have misunderstood your desire to automount home
directories.  Perhaps you don't want to do that, and have all user home
directories on the one NFS share, mounted over /var/data/chroot/home. 
If so, great, that's all you need: create /var/data/chroot (a local
directory containg dev, home and lib directories, as well as ssh-server
executable) and NFS mount the share containing all of your user
directories over /var/data/chroot/home.

If I've still not got it right, you might explain to me what is
complicated so that I can better understand.

Regards,

David

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Hi Hildegard,

Hildegard Meier wrote:
> > I think that works specifically because *no* new process is created
> > when using internal-sftp as opposed to executing the sftp-server binary.
>
> For every sftp subsystem login (here with user "sftp_nagios"), I
> see a new sftpd process created:
..
> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp

Thanks, yes, you're correct that there are new processes. I should
have written different, refering to sshd vs. sftp-server:

sshd creates one child process for each incoming TCP connection (4590)
and after authentication it creates another child process (4592) which
runs as the authenticated user to handle the session, and then the
sftp subsystem channel is handled by yet another child process (4593).

These processes all run (different parts of) sshd, they do not
execute any other program, when using internal-sftp not even 4593.

This is relevant because as mentioned there is then no point where
sshd executes the sftp-server binary, so also no opportunity to use
LD_PRELOAD only for SFTP. All of sftp-server is built into sshd,
specifically to enable internal-sftp, with the great benefit that no
sftp-server binary is needed inside the chroot.

My thought was still wrong though. It could have been relevant for
the syslog() socket because that is closed automatically on exec()
and not on fork(), but sshd re-opens the syslog() socket right after
fork() so the internal-sftp vs. sftp-server makes no difference for
logging, only for LD_PRELOAD convenience.


Hildegard Meier wrote:
> Jochen Bern wrote:
> > If a newly-started syslogd on server A does
> > indeed REMOVE AND RECREATE the /dev/log sockets,
>
> If /dev dir under sftp user's chroot dir exists but there is no "log"
> file in it, it gets created by syslog-ng. It is never removed afterwards.

I had forgotten: AF_UNIX bind() to a filesystem address only succeeds
if the socket inode does not already exist in the filesystem.

So syslog-ng does what all AF_UNIX socket servers must: it unlinks the
socket from the filesystem first, so that a new socket inode can be
created by bind().

Jochen identified the problem.


Jochen Bern wrote:
> (The difference being that the socket itself would not have to store any
> additional state about the open FDs pointing to it, which I doubt it has
> any capacity for.)

The inode itself doesn't store anything, the kernel just maps the
filesystem inode to its socket object. Since the inode created by
daemon B means nothing within the kernel on system A, connecting to
that inode now also visible on system A doesn't reach daemon A.


Jochen Bern wrote:
> If that theory holds, then all we'd need to do is to force both
> syslogd's to use *pre-existing* sockets instead ...

That's unfortunately not possible.


Hildegard Meier wrote:
> I think both the LD_PRELOAD and overlay fs approach are too
> complicated and hacky for me.

Fair enough, and overlayfs is only possible with a Linux NFS server.

There aren't very many other solutions:


* A patch to change SFTP logging

The attached patch adds an -E option to sftp-server (as exists for sshd)
to log to a specific file. With -E /sftp.log on the internal-sftp command
line in sshd_config on both servers they would then both log to a file in
the user's home directory. Beware of write contention on that log file
when the same user is connected to both servers.

Ensure that the file exists before using this configuration. Make file
ownership root:user and mode 620 if the user shouldn't be able to read it.

The patch is small, but until an equivalent is accepted in OpenSSH,
if ever, it requires maintaining a local package.

Note that as proposed this patch only works with a regular file, not
a syslog socket. Supporting a socket is of course possible but requires
a larger patch re-implementing much of syslog(). I can write that if
it would be accepted into OpenSSH.


* Add -e to the internal-sftp command line and create a custom parser

Because internal-sftp is still the sshd process this log message
parser must be connected already to the initial sshd stderr and must
handle anything that can be output on stderr by any sshd child process,
not just SFTP messages, making this alternative less attractive.


Kind regards

//Peter

Small clarification:

Hildegard Meier wrote:
> ps auxww |grep sftp
> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp

4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
I guess that this is a TLS FTP server.
Try ps fax to see process relationships.

The "sftpd:" prefix for the sshd processes may be related to SELinux;
nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.


//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

If 'internal-sftp' means that the SFTP server is just a forked child of sshd, would the 'sshd -e' flag work to send log messages from internal-sftp to stderr?

If so, it may be worth just sending all logging from sshd to stderr (sowie [daemontools]).

Someone with time and ingenuity on their hands may even be able to send the stderr output on to syslog-ng and dispense with any need to filter it or send it to a file....

If internal-sftp doesn't work that way for ${reasons}, it may be worth adding that functionality for this sort of use case.

____________________
[daemontools]: https://cr.yp.to/daemontools.html


--
jmk

> On Sep 29, 2021, at 19:44, Peter Stuge <peter@stuge.se> wrote:
>
> ?Small clarification:
>
> Hildegard Meier wrote:
>> ps auxww |grep sftp
>> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
>> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
>> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
>> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp
>
> 4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
> I guess that this is a TLS FTP server.
> Try ps fax to see process relationships.
>
> The "sftpd:" prefix for the sshd processes may be related to SELinux;
> nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.
>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

If 'internal-sftp' means that the SFTP server is just a forked child of sshd, would the 'sshd -e' flag work to send log messages from internal-sftp to stderr?

If so, it may be worth just sending all logging from sshd to stderr (sowie [daemontools]).

Someone with time and ingenuity on their hands may even be able to send the stderr output on to syslog-ng and dispense with any need to filter it or send it to a file....

If internal-sftp doesn't work that way for ${reasons}, it may be worth adding that functionality for this sort of use case.

____________________
[daemontools]: https://cr.yp.to/daemontools.html


--
jmk

> On Sep 29, 2021, at 19:44, Peter Stuge <peter@stuge.se> wrote:
>
> ?Small clarification:
>
> Hildegard Meier wrote:
>> ps auxww |grep sftp
>> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
>> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
>> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
>> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp
>
> 4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
> I guess that this is a TLS FTP server.
> Try ps fax to see process relationships.
>
> The "sftpd:" prefix for the sshd processes may be related to SELinux;
> nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.
>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

If 'internal-sftp' means that the SFTP server is just a forked child of sshd, would the 'sshd -e' flag work to send log messages from internal-sftp to stderr?

If so, it may be worth just sending all logging from sshd to stderr (sowie [daemontools]).

Someone with time and ingenuity on their hands may even be able to send the stderr output on to syslog-ng and dispense with any need to filter it or send it to a file....

If internal-sftp doesn't work that way for ${reasons}, it may be worth adding that functionality for this sort of use case.

____________________
[daemontools]: https://cr.yp.to/daemontools.html


--
jmk

> On Sep 29, 2021, at 19:44, Peter Stuge <peter@stuge.se> wrote:
>
> ?Small clarification:
>
> Hildegard Meier wrote:
>> ps auxww |grep sftp
>> root 4192 0.0 0.1 72304 6512 ? Ss 11:01 0:00 /usr/sbin/sftpd -D -f /etc/sftpd/sftpd_config
>> root 4590 0.2 0.1 74736 6632 ? Ss 11:05 0:00 sftpd: sftp_nagios [priv]
>> sftp_na+ 4592 0.0 0.0 74736 3432 ? S 11:05 0:00 sftpd: sftp_nagios@notty
>> sftp_na+ 4593 0.0 0.0 74736 3108 ? Ss 11:05 0:00 sftpd: sftp_nagios@internal-sftp
>
> 4192 /usr/sbin/sftpd is likely unrelated to OpenSSH.
> I guess that this is a TLS FTP server.
> Try ps fax to see process relationships.
>
> The "sftpd:" prefix for the sshd processes may be related to SELinux;
> nothing in OpenSSH is called sftpd, except the "sftpd_t" SELinux context.
>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev