Mailing List Archive

Minimize sshd log clutter/spam from unauthenticated connections
Dear OpenSSH developers,

a publicly accessible sshd on port 22 generates a lot of log clutter
from unauthenticated connections. For an exemplary host on a university
network, sshd accumulates 5~20k log lines on a single day (more than 90%
of the total amount of syslog lines). That is despite the host having a
restricted configuration (no SSH password authentication, firewall rate
limit for new SSH connections on /24 subnets permitting a few
connections per hour, however with a shorter timeout). I'd expect even
more log messages for a default configuration (password auth enabled and
no firewall rate limit).

Would you be open to introducing a new config option to suppress any log
messages from yet unauthenticated connections? If such a suggestion has
been discussed before, please direct me to it. I haven't found anything
in the archives.

Any log messages including successful authentication and afterwards are
still desired, so changing the log level to above INFO will not help.
Additionally, even unauthenticated connections cause messages with
levels ERROR ("kex_exchange_identification: Connection closed by remote
host") or even CRITICAL ("Timeout before authentication"). As I
periodically scan my hosts' syslogs for messages with level WARNING or
above, I currently have to filter these messages to keep my inbox from
overflowing.

Thanks and best regards,
Carsten

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
I guess you might find fail2ban useful.

It scans logfiles (like /var/log/sshd.log), and when it sees too many authentication failures from an IP address (or network range) it can issue commands to drop any further attempts via a firewall.

By having it read its own logfile it's possible to have repeated offenders be cut out for longer and longer time spans.

https://www.fail2ban.org/wiki/index.php/Main_Page
https://supine.com/posts/2012/08/fail2ban-monitoring-itself-recursively/
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
modern syslog daemons (including rsyslog, which is default on just about every
linux system) allow you to filter efficiently on the message contents, not just
the severity, so you can opt to throw out the messages you don't want.

I advocate for a slightly different way of dealing with it, filter these
messages from your main logstream, but put them into either a script directly,
or a separate file and have a script run against it. Have the script report the
number of these messgaes that you get in a time period (minute, hour, whatever
you want) and log that count back into your log stream

as Marcus Ranum said in his Artificial Ignorance writeup, the number of times
that an uninteresting thing happens can be interesting.

If you see a big spike (or drop) is these attempts, it can indicate cause for
concern.

David Lang

On Sat,
18 Mar 2023, Carsten Andrich wrote:

> Date: Sat, 18 Mar 2023 13:15:29 +0100
> From: Carsten Andrich <carsten.andrich@tu-ilmenau.de>
> To: openssh-unix-dev@mindrot.org
> Subject: Minimize sshd log clutter/spam from unauthenticated connections
>
> Dear OpenSSH developers,
>
> a publicly accessible sshd on port 22 generates a lot of log clutter from
> unauthenticated connections. For an exemplary host on a university network,
> sshd accumulates 5~20k log lines on a single day (more than 90% of the total
> amount of syslog lines). That is despite the host having a restricted
> configuration (no SSH password authentication, firewall rate limit for new
> SSH connections on /24 subnets permitting a few connections per hour, however
> with a shorter timeout). I'd expect even more log messages for a default
> configuration (password auth enabled and no firewall rate limit).
>
> Would you be open to introducing a new config option to suppress any log
> messages from yet unauthenticated connections? If such a suggestion has been
> discussed before, please direct me to it. I haven't found anything in the
> archives.
>
> Any log messages including successful authentication and afterwards are still
> desired, so changing the log level to above INFO will not help. Additionally,
> even unauthenticated connections cause messages with levels ERROR
> ("kex_exchange_identification: Connection closed by remote host") or even
> CRITICAL ("Timeout before authentication"). As I periodically scan my hosts'
> syslogs for messages with level WARNING or above, I currently have to filter
> these messages to keep my inbox from overflowing.
>
> Thanks and best regards,
> Carsten
>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
On 18.03.23 14:19, Philipp Marek wrote:
> I guess you might find fail2ban useful.
>
> It scans logfiles (like /var/log/sshd.log), and when it sees too many
> authentication failures from an IP address (or network range) it can
> issue commands to drop any further attempts via a firewall.
>
> By having it read its own logfile it's possible to have repeated
> offenders be cut out for longer and longer time spans.
>
> https://www.fail2ban.org/wiki/index.php/Main_Page
> https://supine.com/posts/2012/08/fail2ban-monitoring-itself-recursively/

Thanks for the suggestion. I've looked into solutions like fail2ban in
the past, but have decided for a simpler approach. On some Linux hosts I
use the following nftables rules (commented and stripped for clarity):

table inet filter {
# set of IP addresses that have successfully authenticated
# filled via, e.g., the following /root/.ssh/rc (simple example without error handling):
# `nft add element inet filter sshauth { ${SSH_CONNECTION%% *} timeout 4h }`
set sshauth {
type ipv4_addr
flags timeout, dynamic
}

# set of IP addresses (or rather /24 subnets, see below) that have
# established new TCP connections to SSHD
set sshlimit {
type ipv4_addr
flags timeout, dynamic
}

chain input {
type filter hook input priority 0; policy drop;

# accept new connections from IP addresses that have authenticated before
ip saddr @sshauth tcp dport 22 ct state new counter accept
# accept new connections from all other addresses with significant rate
# limit on /24 subnet
ip protocol tcp tcp dport 22 ct state new add @sshlimit { ip saddr & 255.255.255.0 timeout 1h limit rate 2/hour } counter accept

# accept established connections and reject the rest (whatever exceeds
# above rate limit)
ct state { established, related } accept
meta pkttype unicast ip protocol tcp counter reject with tcp reset
}
}

The result is similar to fail2ban in that it aggressively limits any
repeat connections that do not authenticate successfully. Albeit with a
significantly smaller attack surface and configuration effort. The trick
to make it usable despite the 2/hour connection limit is to manually
fill the set sshauth either via an .ssh/rc file (will only work for
root) or by parsing the ssh log and adding IP addresses that
authenticate successfully.

Best regards,
Carsten

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
On 18.03.23 14:34, David Lang wrote:
> modern syslog daemons (including rsyslog, which is default on just
> about every linux system) allow you to filter efficiently on the
> message contents, not just the severity, so you can opt to throw out
> the messages you don't want.
>
> I advocate for a slightly different way of dealing with it, filter
> these messages from your main logstream, but put them into either a
> script directly, or a separate file and have a script run against it.
> Have the script report the number of these messgaes that you get in a
> time period (minute, hour, whatever you want) and log that count back
> into your log stream
>
> as Marcus Ranum said in his Artificial Ignorance writeup, the number
> of times that an uninteresting thing happens can be interesting.
>
> If you see a big spike (or drop) is these attempts, it can indicate
> cause for concern.

I run Debian with systemd-journald instead of rsyslog. AFAIK journald
does not support filtering of its ingress log messages. Only the output
can be filtered with journalctl, but by then it's already too late in
terms of log spam on disk.

Regardless of which syslog solution is being used, I find it
inconvenient to manually assemble and maintain a filter list. I believe
an sshd configuration option, which of course defaults to false, would
be the best solution. Currently, the sheer amount of messages from
unauthenticated connections drowns out anything else. I took this
opportunity to sift through sshd's messages of almost 2 years via:

journalctl -t sshd -o cat \
| grep -v '^Accepted ' \
| sed -E 's/[Uu]ser \S+/user .../' \
| sed -E 's/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/0.0.0.0/' \
| sed -E 's/port\s+[0-9]+/port 0/' \
| sed -E 's/".*"/"..."/' \
| sed -E 's/Change of username or service not allowed: .+/Change of username or service not allowed: .../' \
| sed -E 's/Their offer: .+/Their offer: .../' \
| sort -u

I found a select few attempts to mess with, identify, or exploit log
parsing IDS/IPS software like fail2ban (and feel confirmed in my choice
of an alternative solution with far less attack surface, see my other mail):

Invalid user $(ping -c 1 16e939dc.ad.xspzo.com) from ...
Invalid user ' $(ping -c 1 16e939dc.ad.xspzo.com) from ...
Invalid user ' or '1'='1' - from 176.100.42.41

Only two concerning messages came up:

error: beginning MaxStartups throttling
fatal: ssh_sandbox_violation: unexpected system call (arch:0xc000003e,syscall:20 @ 0x7f1d8469e1f5) [preauth]

I definitely wouldn't want to miss the above among an almost endless
list of (invalid) usernames, (dis-)connecting IP addresses, invalid
version strings, etc. that various script kiddies are playing with.
While I do get your point, I doubt analyzing all of these individual
messages has a reasonable cost-benefit ratio. I occasionally look at the
amount of rejected connections (my nftables rules keep track of that) to
see any unusual uptick of interest, upon which I investigate.

Best regards,
Carsten

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
On Sat, 18 Mar 2023, Carsten Andrich wrote:

> Date: Sat, 18 Mar 2023 18:16:44 +0100
> From: Carsten Andrich <carsten.andrich@tu-ilmenau.de>
> To: David Lang <david@lang.hm>
> Cc: openssh-unix-dev@mindrot.org
> Subject: Re: Minimize sshd log clutter/spam from unauthenticated connections
>
> On 18.03.23 14:34, David Lang wrote:
>> modern syslog daemons (including rsyslog, which is default on just about
>> every linux system) allow you to filter efficiently on the message
>> contents, not just the severity, so you can opt to throw out the messages
>> you don't want.
>>
>> I advocate for a slightly different way of dealing with it, filter these
>> messages from your main logstream, but put them into either a script
>> directly, or a separate file and have a script run against it. Have the
>> script report the number of these messgaes that you get in a time period
>> (minute, hour, whatever you want) and log that count back into your log
>> stream
>>
>> as Marcus Ranum said in his Artificial Ignorance writeup, the number of
>> times that an uninteresting thing happens can be interesting.
>>
>> If you see a big spike (or drop) is these attempts, it can indicate cause
>> for concern.
>
> I run Debian with systemd-journald instead of rsyslog. AFAIK journald does
> not support filtering of its ingress log messages. Only the output can be
> filtered with journalctl, but by then it's already too late in terms of log
> spam on disk.

rsyslog is still available, and you don't have to keep everything in the journal
files (journald is not a modern logging system, in spite of it's date of
implementation :-) )

David Lang
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
To radically cut down on SSH log spam you can also hide it completely behind a firewall, and allow access only by some port knocking sequence.


I quite like having a process listen on port 53 and wait for a dns query containing a totp string to grant (temporary) access; that's a 2fa, and doing a "host 123456. my-ip" is easily automated in a shell script as well...
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
[...]

> journalctl -t sshd -o cat \
>     | grep -v '^Accepted ' \
>     | sed -E 's/[Uu]ser \S+/user .../' \
>     | sed -E 's/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/0.0.0.0/' \
>     | sed -E 's/port\s+[0-9]+/port 0/' \
>     | sed -E 's/".*"/"..."/' \
>     | sed -E 's/Change of username or service not allowed: .+/Change of username or service not allowed: .../' \
>     | sed -E 's/Their offer: .+/Their offer: .../' \
>     | sort -u
>
> I found a select few attempts to mess with, identify, or exploit log parsing IDS/IPS software like fail2ban (and feel confirmed in my choice of an alternative solution with far less attack surface, see my other mail):
>
> Invalid user $(ping -c 1 16e939dc.ad.xspzo.com) from ...
> Invalid user ' $(ping -c 1 16e939dc.ad.xspzo.com) from ...
> Invalid user ' or '1'='1' - from 176.100.42.41
[...]

May I suggest, you take a look at logcheck(8). It seems, that this what you are looking for.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
Dropping logs/audits and possible indications-of-attack is NOT cyber security best practices.



> On Mar 18, 2023, at 6:15 AM, Carsten Andrich <carsten.andrich@tu-ilmenau.de> wrote:
>
> Dear OpenSSH developers,
>
> a publicly accessible sshd on port 22 generates a lot of log clutter from unauthenticated connections. For an exemplary host on a university network, sshd accumulates 5~20k log lines on a single day (more than 90% of the total amount of syslog lines). That is despite the host having a restricted configuration (no SSH password authentication, firewall rate limit for new SSH connections on /24 subnets permitting a few connections per hour, however with a shorter timeout). I'd expect even more log messages for a default configuration (password auth enabled and no firewall rate limit).
>
> Would you be open to introducing a new config option to suppress any log messages from yet unauthenticated connections? If such a suggestion has been discussed before, please direct me to it. I haven't found anything in the archives.
>
> Any log messages including successful authentication and afterwards are still desired, so changing the log level to above INFO will not help. Additionally, even unauthenticated connections cause messages with levels ERROR ("kex_exchange_identification: Connection closed by remote host") or even CRITICAL ("Timeout before authentication"). As I periodically scan my hosts' syslogs for messages with level WARNING or above, I currently have to filter these messages to keep my inbox from overflowing.
>
> Thanks and best regards,
> Carsten

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
Good suggestion. You could also use rate-limiting on your firewall to slow down inbound connections from the same attacker.



> On Mar 18, 2023, at 7:19 AM, Philipp Marek <philipp@marek.priv.at> wrote:
>
> I guess you might find fail2ban useful.
>
> It scans logfiles (like /var/log/sshd.log), and when it sees too many authentication failures from an IP address (or network range) it can issue commands to drop any further attempts via a firewall.
>
> By having it read its own logfile it's possible to have repeated offenders be cut out for longer and longer time spans.
>
> https://www.fail2ban.org/wiki/index.php/Main_Page
> https://supine.com/posts/2012/08/fail2ban-monitoring-itself-recursively/

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
1On Sat, Mar 18, 2023 at 8:23?AM Carsten Andrich
<carsten.andrich@tu-ilmenau.de> wrote:

> a publicly accessible sshd on port 22 generates a lot of log clutter
> from unauthenticated connections.

This is not unique to OpenSSH. Any well-known service on any
publicly-accessible host will be probed incessantly.

In addition to source-blocking solutions (portknocking, fail2ban, et.
al.) and log filtering solutions, something that can be surprisingly
effective is to simply configure sshd to listen on a TCP port other
than 22. Virtually no attackers perform full port scans looking for
hidden ssh daemons, because there’s so much low-hanging fruit on port
22.

On my home network, I have a publicly-accessible sshd instance that is
listening on a TCP port <1024 that is not port 22. For more than 10
years (until a botnet finally found it), I was the only person who
ever attempted to connect to it. (In response to its discovery, I
could have simply moved the sshd instance to a different port, but
instead I used that as an excuse to experiment with a portknocker
configuration.)

In InfoSec, obfuscation has a bad rap, because it is frequently
deployed in an attempt to hide a security vulnerability instead of
applying remediation. But there are a few instances where obfuscation
can be a valuable and appropriate tool in one’s toolbox. And reducing
voluminous log noise by evading 99.999% of ankle-biting script kiddies
is one of those instances.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
On 19.03.23 07:03, Philipp Marek wrote:
> I quite like having a process listen on port 53 and wait for a dns
> query containing a totp string to grant (temporary) access; that's a
> 2fa, and doing a "host 123456. my-ip" is easily automated in a shell
> script as well...

I have to admit, I *really* like the TOTP idea.

For the time being, I've deployed a quasi-knocking KISS solution that
sends an unencrypted secret via a single UDP packet. Server side is
realized entirely with nftables:

table inet filter {
# set of IP addresses that have authenticated via knocking
set sshauth {
type ipv4_addr
size 255
flags timeout, dynamic
}

chain input {
type filter hook input priority filter; policy drop;

# SSH knocking with single UDP packet containing unencrypted secret
udp dport 12345 @ih,0,32 0xdeadbeef add @sshauth { ip saddr timeout 4h } counter
# alternatively TCP fast open can be used to knock with firewalls that only permit TCP port 22
tcp dport 22 tcp flags syn @ih,0,32 0xdeadbeef add @sshauth { ip saddr timeout 4h } counter reject with tcp reset

# accept new SSH connections from IP addresses that have knocked
ip saddr @sshauth tcp dport 22 ct state new counter accept

# accept established connections and reject the rest
ct state { established, related } accept
meta pkttype unicast ip protocol tcp counter reject with tcp reset
}
}

For SSH hosts only accessed by a select few of technically experienced
users that approach has worked like a charm.

Best regards,
Carsten

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Minimize sshd log clutter/spam from unauthenticated connections [ In reply to ]
On 10.06.23 11:19, Carsten Andrich wrote:
> For the time being, I've deployed a quasi-knocking KISS solution that
> sends an unencrypted secret via a single UDP packet. Server side is
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> realized entirely with nftables

... frankly, for that reason, I like fwknop (in my case, straight from
OS repos) better ... I'd still have to see fwknopd exit unexpectedly,
which is where a host-firewall-only mechanism on the server side would
have an advantage ...

http://www.cipherdyne.org/fwknop/

> ~# cd /etc/fwknop

> fwknop# diff access.conf.orig access.conf | sed -e '/> .*KEY/s/\t.*/\t.../'
> 204,206c204,211
> < SOURCE ANY
> < KEY_BASE64 __CHANGEME__
> < HMAC_KEY_BASE64 __CHANGEME__
> ---
>> SOURCE ANY
>> KEY_BASE64 ...
>> HMAC_KEY_BASE64 ...
>> REQUIRE_SOURCE_ADDRESS Y
>> # fwknopd fiddles with iptables, we need to have nftables modified.
>> CMD_CYCLE_OPEN /usr/local/sbin/fwknop2nftables $IP $PORT
>> CMD_CYCLE_CLOSE NONE
>> CMD_CYCLE_TIMER 30s

> fwknop# diff fwknopd.conf.orig fwknopd.conf
> 40a41
>> PCAP_INTF enp0s25

> fwknop# cat /usr/local/sbin/fwknop2nftables
> #!/bin/sh
>
> # Syntax: $0 SRC_IP PORT
>
> NFT="/usr/sbin/nft"
> SET="fwkn"
> # Note that we are ignoring everything from the accepted fwknop
> # requests except the src IP and tgt port to be allowed ...
>
> PREP=`$NFT list chain inet firewalld filter_IN_public_allow | grep -c "@${SET}_$2"`
>
> if [ $PREP -eq 0 ]; then
> $NFT add set inet firewalld "${SET}_$2" '{ type ipv4_addr ; timeout 30s ; size 32 ; }'
> $NFT add rule inet firewalld filter_IN_public_allow ip saddr "@${SET}_$2" tcp dport "$2" accept
> fi
>
> $NFT add element inet firewalld "${SET}_$2" { $1 }

> ~$ tail -8 .fwknoprc | sed -e '/^[SKH]/s/\t.*/\t.../' -e '/^\[/s/[a-z][a-z]*/.../g'
> [...]
> ACCESS tcp/22
> SPA_SERVER ...
> #ALLOW_IP TBD
> KEY_BASE64 ...
> HMAC_KEY_BASE64 ...
> USE_HMAC Y
> RESOLVE_IP_HTTPS N
Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH