Mailing List Archive

On Thu, 20 Jul 2023 at 10:54, Johnnie W Adams <jxadams@ualr.edu> wrote:
> We're experiencing an odd ten-second delay intermittently when logging
> into any of our Linux boxes [...] I'm
> having a heck of a time figuring out what or why. Any guidance would be
> greatly appreciated.

You can strace sshd to see what it's doing when it stalls. Assuming
you can connect to another port, you can do something like this to run
sshd on 2222, tracing both parent and child sshds:

$ sudo strace -f /usr/sbin/sshd -De -ologlevel=fatal -p 2222

(note that the trace may contain sensitive information so treat it with caution)

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

> On 20 Jul 2023, at 19:45, Johnnie W Adams <jxadams@ualr.edu> wrote:
>
> Hi, folks,
>
> We're experiencing an odd ten-second delay intermittently when logging
> into any of our Linux boxes which authenticate against LDAP. Here's where
> it happens:
>
> Jul 13 11:54:23 console2 sshd[1853]: debug1: temporarily_use_uid: <my
> uid\gid> (e=0/0)
>
> Jul 13 11:54:35 console2 sshd[1853]: debug1: trying public key file <my key
> file>
>
> My assumption is there's something in sssd slowing it down, but I'm
> having a heck of a time figuring out what or why. Any guidance would be
> greatly appreciated.

The current (by default for me, after DNS is fixed) guilty party since Debian 11, had been dbus on especially LXC containers.
Disabling that had been one of the easy fixes for me when I get hit by that "hang" .. its actually, so frequent for me, it's part of my script to copy-paste when prepping a LXC DEbian 11&12 instance

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Thu, Jul 20, 2023 at 1:49?PM Johnnie W Adams <jxadams@ualr.edu> wrote:
>
> Hi, folks,
>
> We're experiencing an odd ten-second delay intermittently when logging
> into any of our Linux boxes which authenticate against LDAP. Here's where
> it happens:
>
> Jul 13 11:54:23 console2 sshd[1853]: debug1: temporarily_use_uid: <my
> uid\gid> (e=0/0)
>
> Jul 13 11:54:35 console2 sshd[1853]: debug1: trying public key file <my key
> file>
>
> My assumption is there's something in sssd slowing it down, but I'm
> having a heck of a time figuring out what or why. Any guidance would be
> greatly appreciated.
>
> Thanks,
>
> John A

sssd is a pretty aggressively "optimized" tool. It's designed, not to
issue LDAP queries, but to pull from a locally stashed copy of the
*entire* upstream LDAP directory, or at least enough of the LDAP
directory to contain every dolder it may reference. The result can be
really nasty when the VPN connection between an internal AD and a
cloud environment, especially when it thinks it has to refresh that
cache. All of it. Without notice. And crash, if it doesn't succeed
within the hard-coded and un-tunable timeout periods.

I'm not happy with some of sssd's behavior, especially the head games
it plays with systemd about "I'm started, I'm running, I'm allowing
logins via SSH, la-la-la-la-la, I failed to cache the full LDAP and
now I will crash hard with systemd not noticing and recovering the
service". It's an unpleasant problem.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Nico Kadel-Garcia wrote:
> On Thu, Jul 20, 2023 at 1:49?PM Johnnie W Adams <jxadams@ualr.edu> wrote:
>>
>> Hi, folks,
>>
>> We're experiencing an odd ten-second delay intermittently when logging
>> into any of our Linux boxes which authenticate against LDAP. Here's where
>> it happens:
>>
>> Jul 13 11:54:23 console2 sshd[1853]: debug1: temporarily_use_uid: <my
>> uid\gid> (e=0/0)
>>
>> Jul 13 11:54:35 console2 sshd[1853]: debug1: trying public key file <my key
>> file>
>>
>> My assumption is there's something in sssd slowing it down, but I'm
>> having a heck of a time figuring out what or why. Any guidance would be
>> greatly appreciated.
>>
>> Thanks,
>>
>> John A
>
> sssd is a pretty aggressively "optimized" tool. It's designed, not to
> issue LDAP queries, but to pull from a locally stashed copy of the
> *entire* upstream LDAP directory, or at least enough of the LDAP
> directory to contain every dolder it may reference. The result can be
> really nasty when the VPN connection between an internal AD and a
> cloud environment, especially when it thinks it has to refresh that
> cache. All of it. Without notice. And crash, if it doesn't succeed
> within the hard-coded and un-tunable timeout periods.
>
> I'm not happy with some of sssd's behavior, especially the head games
> it plays with systemd about "I'm started, I'm running, I'm allowing
> logins via SSH, la-la-la-la-la, I failed to cache the full LDAP and
> now I will crash hard with systemd not noticing and recovering the
> service". It's an unpleasant problem.

Sounds like you'd be better off using nslcd. And if you want caching
of the LDAP info, use a local OpenLDAP slapd with slapo-pcache instead, which
has all cache refresh/expiration/etc intervals fully configurable.

--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Sat, Jul 22, 2023 at 3:20?PM Howard Chu <hyc@symas.com> wrote:
>
> Nico Kadel-Garcia wrote:
> > On Thu, Jul 20, 2023 at 1:49?PM Johnnie W Adams <jxadams@ualr.edu> wrote:
> >>
> >> Hi, folks,
> >>
> >> We're experiencing an odd ten-second delay intermittently when logging
> >> into any of our Linux boxes which authenticate against LDAP. Here's where
> >> it happens:
> >>
> >> Jul 13 11:54:23 console2 sshd[1853]: debug1: temporarily_use_uid: <my
> >> uid\gid> (e=0/0)
> >>
> >> Jul 13 11:54:35 console2 sshd[1853]: debug1: trying public key file <my key
> >> file>
> >>
> >> My assumption is there's something in sssd slowing it down, but I'm
> >> having a heck of a time figuring out what or why. Any guidance would be
> >> greatly appreciated.
> >>
> >> Thanks,
> >>
> >> John A
> >
> > sssd is a pretty aggressively "optimized" tool. It's designed, not to
> > issue LDAP queries, but to pull from a locally stashed copy of the
> > *entire* upstream LDAP directory, or at least enough of the LDAP
> > directory to contain every dolder it may reference. The result can be
> > really nasty when the VPN connection between an internal AD and a
> > cloud environment, especially when it thinks it has to refresh that
> > cache. All of it. Without notice. And crash, if it doesn't succeed
> > within the hard-coded and un-tunable timeout periods.
> >
> > I'm not happy with some of sssd's behavior, especially the head games
> > it plays with systemd about "I'm started, I'm running, I'm allowing
> > logins via SSH, la-la-la-la-la, I failed to cache the full LDAP and
> > now I will crash hard with systemd not noticing and recovering the
> > service". It's an unpleasant problem.
>
> Sounds like you'd be better off using nslcd. And if you want caching
> of the LDAP info, use a local OpenLDAP slapd with slapo-pcache instead, which
> has all cache refresh/expiration/etc intervals fully configurable.

At that time, I probably would have been better off doing that for the
environment I was organizing. One of the problems of consulting work
is the need to leave something well documented and supportable by the
permanent staff. Another can be getting the people behind the
bureaucrati cwalls to admit what they're doing, and its consequences,
when you weren't hired as part of their department. I'm afraid that
they were fairly miffed at me for usin "ldapbrowser" to look around at
their AD structure. I was *not* supposed to have permission to look,
and the supervisor of security demanded that even his own personnel
not share insights with each other. It got weirder when I yanked out
the secret root user public SSH keys in the OS images I was supposed
to secure for deployment.

It took time to negotiate with the AD admins, and them to realize I
understood what I was asking for, to organize the relevant OU's in one
LDAP directory so sssd only had to stash that LDAP directory. It's a
good technique for improving sssd performance, especially in remotely
deployed environments without a high availability AD domain controller
at the end of a VPN for big cloud environments.

For our original poster here, the lesson is "take a look at your sssd
setup and see if you can reduce the relevant LDAP directory space".
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev