Mailing List Archive

Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix
Hi,

I'm new to rsyslog and I'm trying to move forward on rsyslog due to a security advisory.
I'm trying to achieve that rsyslog will no long accept ssl2, ssl3 or tls1 and only accepts tls 1.2

I figured this would work:
module(
load="imtcp"
# see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
StreamDriver.Name="gtls"
StreamDriver.Mode="1"
StreamDriver.Authmode="x509/name"
# See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
# PermittedPeer=["*.enexis.nl"]
PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1
MinProtocol=TLSv1.2"
)
input(
type="imtcp"
port="6518"
)

It did not, and I started seeing these errors in journal:
peer did not provide a certificate, not permitted to talk to it [v8.24.0-34.el7 try http://www.rsyslog.com/e/2085 ]
netstream session 0x7f10e40e8d80 from 10.xx.xx.xx will be closed due to error [v8.24.0-34.el7 try http://www.rsyslog.com/e/2089 ]



I figured, well there is no change on the other end, so a rollback will fix it. That would give me re-assurance that I can rollback any time. Since we haven't tested that before.
It did not, and now I have a broken rsyslog server ????

Old config:
module(
load="imtcp"
# see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
StreamDriver.Name="gtls"
StreamDriver.Mode="1"
StreamDriver.Authmode="x509/name"
# See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
# PermittedPeer=["*.enexis.nl"]
PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
)
input(
type="imtcp"
port="6518"
)


I don't have many things I can check, as there is only 1 service relying on this. But that service is actually the only thing that's providing us insight into network logs. Incidentally our SIEM uses that as input.
So I did do a small check on the certificates:
openssl s_client -connect syslog.xx.nl:6518 -CAfile tls-ca-bundle.pem

SSL handshake has read 17073 bytes and written 338 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 966666C3F3245570181E11CB525457BAD70D3F0E208D0CED75361980B7935465
Session-ID-ctx:
Master-Key: 2570E3FEC0FEBCB661A213FC38EF685FBACEA796E31CDSADH78HDFG6ADDFD0FDE9788CFD4C43B101A307345EC66C5C5BC0
Start Time: 1562755867
Timeout : 7200 (sec)
Verify return code: 0 (ok)


Trying the endpoint of this server always fails on the first certificate in the chain, hence why I'm adding it into that command.

And in the config it does specify certs that are needed:
global(
DefaultNetstreamDriver="gtls"?
DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls-ca-bundle.pem"?
DefaultNetstreamDriverCertFile="/etc/rsyslog.d/server.crt"?
DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/server.key"?
)?

I believe the use of gnutlsprioritystring setting requires version 8.29
We are running 8.24, however I do not see version 8.29 in the public yum repos for centos7? I might be looking in the wrong place though.

Since it is broken I would rather move forward in limiting connections to just tls 1.2 instead of investing time into getting it rolled back to a working state. But I guess that all depends on how easy moving forward goes.
Is there anyone who has tips on moving forward?

Best regards,

Marco Moll
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix [ In reply to ]
sry, just a terse reply, hopefully still useful:

8.24 is very old. I think it does not have the ability to do the detail
configuration.

The rsyslog homepage (top right hand) lists project repos with current
versions.

No matter what you do to a config, rsyslog does not store anything of that
persistently. So if you make a change and undo it, this will always work.
As such it looks like you did not completely undo it or made some other
changes which affect rsyslog outside it's config context.

HTH
Rainer

El jue., 20 feb. 2020 a las 14:03, Moll, Marco via rsyslog (<
rsyslog@lists.adiscon.com>) escribió:

> Hi,
>
> I'm new to rsyslog and I'm trying to move forward on rsyslog due to a
> security advisory.
> I'm trying to achieve that rsyslog will no long accept ssl2, ssl3 or tls1
> and only accepts tls 1.2
>
> I figured this would work:
> module(
> load="imtcp"
> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
> StreamDriver.Name="gtls"
> StreamDriver.Mode="1"
> StreamDriver.Authmode="x509/name"
> # See
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
> # PermittedPeer=["*.enexis.nl"]
> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
> gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1
> MinProtocol=TLSv1.2"
> )
> input(
> type="imtcp"
> port="6518"
> )
>
> It did not, and I started seeing these errors in journal:
> peer did not provide a certificate, not permitted to talk to it
> [v8.24.0-34.el7 try http://www.rsyslog.com/e/2085 ]
> netstream session 0x7f10e40e8d80 from 10.xx.xx.xx will be closed due to
> error [v8.24.0-34.el7 try http://www.rsyslog.com/e/2089 ]
>
>
>
> I figured, well there is no change on the other end, so a rollback will
> fix it. That would give me re-assurance that I can rollback any time. Since
> we haven't tested that before.
> It did not, and now I have a broken rsyslog server ????
>
> Old config:
> module(
> load="imtcp"
> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
> StreamDriver.Name="gtls"
> StreamDriver.Mode="1"
> StreamDriver.Authmode="x509/name"
> # See
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
> # PermittedPeer=["*.enexis.nl"]
> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
> )
> input(
> type="imtcp"
> port="6518"
> )
>
>
> I don't have many things I can check, as there is only 1 service relying
> on this. But that service is actually the only thing that's providing us
> insight into network logs. Incidentally our SIEM uses that as input.
> So I did do a small check on the certificates:
> openssl s_client -connect syslog.xx.nl:6518 -CAfile tls-ca-bundle.pem
>
> SSL handshake has read 17073 bytes and written 338 bytes
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-GCM-SHA384
> Session-ID:
> 966666C3F3245570181E11CB525457BAD70D3F0E208D0CED75361980B7935465
> Session-ID-ctx:
> Master-Key:
> 2570E3FEC0FEBCB661A213FC38EF685FBACEA796E31CDSADH78HDFG6ADDFD0FDE9788CFD4C43B101A307345EC66C5C5BC0
> Start Time: 1562755867
> Timeout : 7200 (sec)
> Verify return code: 0 (ok)
>
>
> Trying the endpoint of this server always fails on the first certificate
> in the chain, hence why I'm adding it into that command.
>
> And in the config it does specify certs that are needed:
> global(
> DefaultNetstreamDriver="gtls"?
> DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls-ca-bundle.pem"?
> DefaultNetstreamDriverCertFile="/etc/rsyslog.d/server.crt"?
> DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/server.key"?
> )?
>
> I believe the use of gnutlsprioritystring setting requires version 8.29
> We are running 8.24, however I do not see version 8.29 in the public yum
> repos for centos7? I might be looking in the wrong place though.
>
> Since it is broken I would rather move forward in limiting connections to
> just tls 1.2 instead of investing time into getting it rolled back to a
> working state. But I guess that all depends on how easy moving forward goes.
> Is there anyone who has tips on moving forward?
>
> Best regards,
>
> Marco Moll
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix [ In reply to ]
8.24 is over 3 years old now, after 8.39 (over a year old) we switched to a date
based version, and the current version is 8.2001 (8.YYMM)

GNUTLS has some problems that will cause it to segfault under some conditions.
You should switch to openssl (requires a newer version of rsyslog)

The PriorityString requires a newer version of gnutls and openssl libraries, I'm
not sure if centos 7 includes it, but centos6 does not.

David Lang

On Thu, 20 Feb 2020, Moll, Marco via rsyslog wrote:

> Date: Thu, 20 Feb 2020 13:03:52 +0000
> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: "Moll, Marco" <marco.moll@sogeti.com>
> Subject: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections
> after config change, rollback doesn't fix
>
> Hi,
>
> I'm new to rsyslog and I'm trying to move forward on rsyslog due to a security advisory.
> I'm trying to achieve that rsyslog will no long accept ssl2, ssl3 or tls1 and only accepts tls 1.2
>
> I figured this would work:
> module(
> load="imtcp"
> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
> StreamDriver.Name="gtls"
> StreamDriver.Mode="1"
> StreamDriver.Authmode="x509/name"
> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
> # PermittedPeer=["*.enexis.nl"]
> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
> gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1
> MinProtocol=TLSv1.2"
> )
> input(
> type="imtcp"
> port="6518"
> )
>
> It did not, and I started seeing these errors in journal:
> peer did not provide a certificate, not permitted to talk to it [v8.24.0-34.el7 try http://www.rsyslog.com/e/2085 ]
> netstream session 0x7f10e40e8d80 from 10.xx.xx.xx will be closed due to error [v8.24.0-34.el7 try http://www.rsyslog.com/e/2089 ]
>
>
>
> I figured, well there is no change on the other end, so a rollback will fix it. That would give me re-assurance that I can rollback any time. Since we haven't tested that before.
> It did not, and now I have a broken rsyslog server ????
>
> Old config:
> module(
> load="imtcp"
> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
> StreamDriver.Name="gtls"
> StreamDriver.Mode="1"
> StreamDriver.Authmode="x509/name"
> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
> # PermittedPeer=["*.enexis.nl"]
> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
> )
> input(
> type="imtcp"
> port="6518"
> )
>
>
> I don't have many things I can check, as there is only 1 service relying on this. But that service is actually the only thing that's providing us insight into network logs. Incidentally our SIEM uses that as input.
> So I did do a small check on the certificates:
> openssl s_client -connect syslog.xx.nl:6518 -CAfile tls-ca-bundle.pem
>
> SSL handshake has read 17073 bytes and written 338 bytes
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-GCM-SHA384
> Session-ID: 966666C3F3245570181E11CB525457BAD70D3F0E208D0CED75361980B7935465
> Session-ID-ctx:
> Master-Key: 2570E3FEC0FEBCB661A213FC38EF685FBACEA796E31CDSADH78HDFG6ADDFD0FDE9788CFD4C43B101A307345EC66C5C5BC0
> Start Time: 1562755867
> Timeout : 7200 (sec)
> Verify return code: 0 (ok)
>
>
> Trying the endpoint of this server always fails on the first certificate in the chain, hence why I'm adding it into that command.
>
> And in the config it does specify certs that are needed:
> global(
> DefaultNetstreamDriver="gtls"?
> DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls-ca-bundle.pem"?
> DefaultNetstreamDriverCertFile="/etc/rsyslog.d/server.crt"?
> DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/server.key"?
> )?
>
> I believe the use of gnutlsprioritystring setting requires version 8.29
> We are running 8.24, however I do not see version 8.29 in the public yum repos for centos7? I might be looking in the wrong place though.
>
> Since it is broken I would rather move forward in limiting connections to just tls 1.2 instead of investing time into getting it rolled back to a working state. But I guess that all depends on how easy moving forward goes.
> Is there anyone who has tips on moving forward?
>
> Best regards,
>
> Marco Moll
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix [ In reply to ]
I did try sudo yum update rsyslog, but all I get is that the latest packages are already installed. After checking, it seems we are using private repositories from one of our partners. But they say their repo is synced with the public repositories.
I'm going to take a guess it's this one: http://mirror.centos.org/centos/7/os/x86_64/Packages/

And as far as I can see that one doesn't go beyond 8.24 ..

And the entire system is ansible-managed, so whatever change I make: any rollback is basically just a revert on code. It's impossible for me to forget any part that got changed.

So in order to upgrade I guess that means I need to add the repo via ansible, install the latest version and test.
Still leaves me hanging with a non-working rollbacked scenario where rsyslog should not be persistent.
What's the best way for me to figure out what's not working correct, and to fix it?
turn on rsyslog with debugging on 2? (Guess i'll go do that anyways, see how far that gets me)
Pointers for me to look at specifically maybe?
________________________________
From: David Lang <david@lang.hm>
Sent: Thursday, February 20, 2020 8:09 PM
To: Moll, Marco via rsyslog <rsyslog@lists.adiscon.com>
Cc: Moll, Marco <marco.moll@sogeti.com>
Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix

8.24 is over 3 years old now, after 8.39 (over a year old) we switched to a date
based version, and the current version is 8.2001 (8.YYMM)

GNUTLS has some problems that will cause it to segfault under some conditions.
You should switch to openssl (requires a newer version of rsyslog)

The PriorityString requires a newer version of gnutls and openssl libraries, I'm
not sure if centos 7 includes it, but centos6 does not.

David Lang

On Thu, 20 Feb 2020, Moll, Marco via rsyslog wrote:

> Date: Thu, 20 Feb 2020 13:03:52 +0000
> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: "Moll, Marco" <marco.moll@sogeti.com>
> Subject: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections
> after config change, rollback doesn't fix
>
> Hi,
>
> I'm new to rsyslog and I'm trying to move forward on rsyslog due to a security advisory.
> I'm trying to achieve that rsyslog will no long accept ssl2, ssl3 or tls1 and only accepts tls 1.2
>
> I figured this would work:
> module(
> load="imtcp"
> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
> StreamDriver.Name="gtls"
> StreamDriver.Mode="1"
> StreamDriver.Authmode="x509/name"
> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
> # PermittedPeer=["*.enexis.nl"]
> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
> gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1
> MinProtocol=TLSv1.2"
> )
> input(
> type="imtcp"
> port="6518"
> )
>
> It did not, and I started seeing these errors in journal:
> peer did not provide a certificate, not permitted to talk to it [v8.24.0-34.el7 try http://www.rsyslog.com/e/2085 ]
> netstream session 0x7f10e40e8d80 from 10.xx.xx.xx will be closed due to error [v8.24.0-34.el7 try http://www.rsyslog.com/e/2089 ]
>
>
>
> I figured, well there is no change on the other end, so a rollback will fix it. That would give me re-assurance that I can rollback any time. Since we haven't tested that before.
> It did not, and now I have a broken rsyslog server ????
>
> Old config:
> module(
> load="imtcp"
> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
> StreamDriver.Name="gtls"
> StreamDriver.Mode="1"
> StreamDriver.Authmode="x509/name"
> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
> # PermittedPeer=["*.enexis.nl"]
> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
> )
> input(
> type="imtcp"
> port="6518"
> )
>
>
> I don't have many things I can check, as there is only 1 service relying on this. But that service is actually the only thing that's providing us insight into network logs. Incidentally our SIEM uses that as input.
> So I did do a small check on the certificates:
> openssl s_client -connect syslog.xx.nl:6518 -CAfile tls-ca-bundle.pem
>
> SSL handshake has read 17073 bytes and written 338 bytes
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-GCM-SHA384
> Session-ID: 966666C3F3245570181E11CB525457BAD70D3F0E208D0CED75361980B7935465
> Session-ID-ctx:
> Master-Key: 2570E3FEC0FEBCB661A213FC38EF685FBACEA796E31CDSADH78HDFG6ADDFD0FDE9788CFD4C43B101A307345EC66C5C5BC0
> Start Time: 1562755867
> Timeout : 7200 (sec)
> Verify return code: 0 (ok)
>
>
> Trying the endpoint of this server always fails on the first certificate in the chain, hence why I'm adding it into that command.
>
> And in the config it does specify certs that are needed:
> global(
> DefaultNetstreamDriver="gtls"?
> DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls-ca-bundle.pem"?
> DefaultNetstreamDriverCertFile="/etc/rsyslog.d/server.crt"?
> DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/server.key"?
> )?
>
> I believe the use of gnutlsprioritystring setting requires version 8.29
> We are running 8.24, however I do not see version 8.29 in the public yum repos for centos7? I might be looking in the wrong place though.
>
> Since it is broken I would rather move forward in limiting connections to just tls 1.2 instead of investing time into getting it rolled back to a working state. But I guess that all depends on how easy moving forward goes.
> Is there anyone who has tips on moving forward?
>
> Best regards,
>
> Marco Moll
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix [ In reply to ]
the only thing that rsyslog persists between runs is state files for queues, do
you have any of those configured in your system?, if so you can try to remove
them

do you get a syntax error if you do 'rsyslog -N 1'

what error do you get at startup?


On Fri, 21 Feb 2020, Moll, Marco via rsyslog wrote:

> Date: Fri, 21 Feb 2020 08:27:24 +0000
> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
> To: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
> Cc: "Moll, Marco" <marco.moll@sogeti.com>
> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls
> connections after config change, rollback doesn't fix
>
> I did try sudo yum update rsyslog, but all I get is that the latest packages are already installed. After checking, it seems we are using private repositories from one of our partners. But they say their repo is synced with the public repositories.
> I'm going to take a guess it's this one: http://mirror.centos.org/centos/7/os/x86_64/Packages/
>
> And as far as I can see that one doesn't go beyond 8.24 ..
>
> And the entire system is ansible-managed, so whatever change I make: any rollback is basically just a revert on code. It's impossible for me to forget any part that got changed.
>
> So in order to upgrade I guess that means I need to add the repo via ansible, install the latest version and test.
> Still leaves me hanging with a non-working rollbacked scenario where rsyslog should not be persistent.
> What's the best way for me to figure out what's not working correct, and to fix it?
> turn on rsyslog with debugging on 2? (Guess i'll go do that anyways, see how far that gets me)
> Pointers for me to look at specifically maybe?
> ________________________________
> From: David Lang <david@lang.hm>
> Sent: Thursday, February 20, 2020 8:09 PM
> To: Moll, Marco via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Moll, Marco <marco.moll@sogeti.com>
> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix
>
> 8.24 is over 3 years old now, after 8.39 (over a year old) we switched to a date
> based version, and the current version is 8.2001 (8.YYMM)
>
> GNUTLS has some problems that will cause it to segfault under some conditions.
> You should switch to openssl (requires a newer version of rsyslog)
>
> The PriorityString requires a newer version of gnutls and openssl libraries, I'm
> not sure if centos 7 includes it, but centos6 does not.
>
> David Lang
>
> On Thu, 20 Feb 2020, Moll, Marco via rsyslog wrote:
>
>> Date: Thu, 20 Feb 2020 13:03:52 +0000
>> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
>> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
>> Cc: "Moll, Marco" <marco.moll@sogeti.com>
>> Subject: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections
>> after config change, rollback doesn't fix
>>
>> Hi,
>>
>> I'm new to rsyslog and I'm trying to move forward on rsyslog due to a security advisory.
>> I'm trying to achieve that rsyslog will no long accept ssl2, ssl3 or tls1 and only accepts tls 1.2
>>
>> I figured this would work:
>> module(
>> load="imtcp"
>> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
>> StreamDriver.Name="gtls"
>> StreamDriver.Mode="1"
>> StreamDriver.Authmode="x509/name"
>> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
>> # PermittedPeer=["*.enexis.nl"]
>> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
>> gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1
>> MinProtocol=TLSv1.2"
>> )
>> input(
>> type="imtcp"
>> port="6518"
>> )
>>
>> It did not, and I started seeing these errors in journal:
>> peer did not provide a certificate, not permitted to talk to it [v8.24.0-34.el7 try http://www.rsyslog.com/e/2085 ]
>> netstream session 0x7f10e40e8d80 from 10.xx.xx.xx will be closed due to error [v8.24.0-34.el7 try http://www.rsyslog.com/e/2089 ]
>>
>>
>>
>> I figured, well there is no change on the other end, so a rollback will fix it. That would give me re-assurance that I can rollback any time. Since we haven't tested that before.
>> It did not, and now I have a broken rsyslog server ????
>>
>> Old config:
>> module(
>> load="imtcp"
>> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
>> StreamDriver.Name="gtls"
>> StreamDriver.Mode="1"
>> StreamDriver.Authmode="x509/name"
>> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
>> # PermittedPeer=["*.enexis.nl"]
>> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
>> )
>> input(
>> type="imtcp"
>> port="6518"
>> )
>>
>>
>> I don't have many things I can check, as there is only 1 service relying on this. But that service is actually the only thing that's providing us insight into network logs. Incidentally our SIEM uses that as input.
>> So I did do a small check on the certificates:
>> openssl s_client -connect syslog.xx.nl:6518 -CAfile tls-ca-bundle.pem
>>
>> SSL handshake has read 17073 bytes and written 338 bytes
>> ---
>> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
>> Server public key is 2048 bit
>> Secure Renegotiation IS supported
>> Compression: NONE
>> Expansion: NONE
>> No ALPN negotiated
>> SSL-Session:
>> Protocol : TLSv1.2
>> Cipher : ECDHE-RSA-AES256-GCM-SHA384
>> Session-ID: 966666C3F3245570181E11CB525457BAD70D3F0E208D0CED75361980B7935465
>> Session-ID-ctx:
>> Master-Key: 2570E3FEC0FEBCB661A213FC38EF685FBACEA796E31CDSADH78HDFG6ADDFD0FDE9788CFD4C43B101A307345EC66C5C5BC0
>> Start Time: 1562755867
>> Timeout : 7200 (sec)
>> Verify return code: 0 (ok)
>>
>>
>> Trying the endpoint of this server always fails on the first certificate in the chain, hence why I'm adding it into that command.
>>
>> And in the config it does specify certs that are needed:
>> global(
>> DefaultNetstreamDriver="gtls"?
>> DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls-ca-bundle.pem"?
>> DefaultNetstreamDriverCertFile="/etc/rsyslog.d/server.crt"?
>> DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/server.key"?
>> )?
>>
>> I believe the use of gnutlsprioritystring setting requires version 8.29
>> We are running 8.24, however I do not see version 8.29 in the public yum repos for centos7? I might be looking in the wrong place though.
>>
>> Since it is broken I would rather move forward in limiting connections to just tls 1.2 instead of investing time into getting it rolled back to a working state. But I guess that all depends on how easy moving forward goes.
>> Is there anyone who has tips on moving forward?
>>
>> Best regards,
>>
>> Marco Moll
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix [ In reply to ]
These are the steps I take any time I do a restart or stop/start for rsyslog, and this is the current output:
sudo systemctl disable rsyslog
Removed symlink /etc/systemd/system/multi-user.target.wants/rsyslog.service.?sudo systemctl stop rsyslog?sudo systemctl enable rsyslog?
Created symlink from /etc/systemd/system/multi-user.target.wants/rsyslog.service to /usr/lib/systemd/system/rsyslog.service.?
sudo systemctl start rsyslog?sudo systemctl status rsyslog?
? rsyslog.service - System Logging Service?
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor pres et: enabled)?
Active: active (running) since Fri 2020-02-21 09:36:36 CET; 5s ago?
Docs: man:rsyslogd(8)?
http://www.rsyslog.com/doc/?
Main PID: 5372 (rsyslogd)?
CGroup: /system.slice/rsyslog.service?
??5372 /usr/sbin/rsyslogd -n?
?
Feb 21 09:36:36 systemd[1]: Started System Loggi...?
Feb 21 09:36:37 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
Feb 21 09:36:37 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
Feb 21 09:36:38 rsyslogd[5372]: peer did not provide a certificate, not permitted to talk to it ...?
Feb 21 09:36:38 rsyslogd[5372]: netstream session 0x7fa29004a7f0 from 10.xx.xx.xx will be closed due...?
Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
Feb 21 09:36:40 rsyslogd[5372]: unexpected GnuTLS error -110 in nsdsel_gtls.c:178: The TLS connection ...?
Feb 21 09:36:40 rsyslogd[5372]: netstream session 0x7fa29004a7f0 from 10.xx.xx.xx will be closed due...?
Hint: Some lines were ellipsized, use -l to show in full.?

Im doing the disable/enable because I read in some other topic that a reboot was needed, but rebooting means alerts get send out to our pagerduty. So im circumventing that.

When I try your command:
sudo rsyslogd -N 1
rsyslogd: version 8.24.0-34.el7, config validation run (level 1), master config /etc/rsyslog.conf?
rsyslogd: End of config validation run. Bye.?

________________________________
From: David Lang <david@lang.hm>
Sent: Friday, February 21, 2020 9:31 AM
To: Moll, Marco via rsyslog <rsyslog@lists.adiscon.com>
Cc: Moll, Marco <marco.moll@sogeti.com>
Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix

the only thing that rsyslog persists between runs is state files for queues, do
you have any of those configured in your system?, if so you can try to remove
them

do you get a syntax error if you do 'rsyslog -N 1'

what error do you get at startup?


On Fri, 21 Feb 2020, Moll, Marco via rsyslog wrote:

> Date: Fri, 21 Feb 2020 08:27:24 +0000
> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
> To: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
> Cc: "Moll, Marco" <marco.moll@sogeti.com>
> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls
> connections after config change, rollback doesn't fix
>
> I did try sudo yum update rsyslog, but all I get is that the latest packages are already installed. After checking, it seems we are using private repositories from one of our partners. But they say their repo is synced with the public repositories.
> I'm going to take a guess it's this one: http://mirror.centos.org/centos/7/os/x86_64/Packages/
>
> And as far as I can see that one doesn't go beyond 8.24 ..
>
> And the entire system is ansible-managed, so whatever change I make: any rollback is basically just a revert on code. It's impossible for me to forget any part that got changed.
>
> So in order to upgrade I guess that means I need to add the repo via ansible, install the latest version and test.
> Still leaves me hanging with a non-working rollbacked scenario where rsyslog should not be persistent.
> What's the best way for me to figure out what's not working correct, and to fix it?
> turn on rsyslog with debugging on 2? (Guess i'll go do that anyways, see how far that gets me)
> Pointers for me to look at specifically maybe?
> ________________________________
> From: David Lang <david@lang.hm>
> Sent: Thursday, February 20, 2020 8:09 PM
> To: Moll, Marco via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Moll, Marco <marco.moll@sogeti.com>
> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix
>
> 8.24 is over 3 years old now, after 8.39 (over a year old) we switched to a date
> based version, and the current version is 8.2001 (8.YYMM)
>
> GNUTLS has some problems that will cause it to segfault under some conditions.
> You should switch to openssl (requires a newer version of rsyslog)
>
> The PriorityString requires a newer version of gnutls and openssl libraries, I'm
> not sure if centos 7 includes it, but centos6 does not.
>
> David Lang
>
> On Thu, 20 Feb 2020, Moll, Marco via rsyslog wrote:
>
>> Date: Thu, 20 Feb 2020 13:03:52 +0000
>> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
>> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
>> Cc: "Moll, Marco" <marco.moll@sogeti.com>
>> Subject: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections
>> after config change, rollback doesn't fix
>>
>> Hi,
>>
>> I'm new to rsyslog and I'm trying to move forward on rsyslog due to a security advisory.
>> I'm trying to achieve that rsyslog will no long accept ssl2, ssl3 or tls1 and only accepts tls 1.2
>>
>> I figured this would work:
>> module(
>> load="imtcp"
>> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
>> StreamDriver.Name="gtls"
>> StreamDriver.Mode="1"
>> StreamDriver.Authmode="x509/name"
>> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
>> # PermittedPeer=["*.enexis.nl"]
>> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
>> gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1
>> MinProtocol=TLSv1.2"
>> )
>> input(
>> type="imtcp"
>> port="6518"
>> )
>>
>> It did not, and I started seeing these errors in journal:
>> peer did not provide a certificate, not permitted to talk to it [v8.24.0-34.el7 try http://www.rsyslog.com/e/2085 ]
>> netstream session 0x7f10e40e8d80 from 10.xx.xx.xx will be closed due to error [v8.24.0-34.el7 try http://www.rsyslog.com/e/2089 ]
>>
>>
>>
>> I figured, well there is no change on the other end, so a rollback will fix it. That would give me re-assurance that I can rollback any time. Since we haven't tested that before.
>> It did not, and now I have a broken rsyslog server ????
>>
>> Old config:
>> module(
>> load="imtcp"
>> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
>> StreamDriver.Name="gtls"
>> StreamDriver.Mode="1"
>> StreamDriver.Authmode="x509/name"
>> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
>> # PermittedPeer=["*.enexis.nl"]
>> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
>> )
>> input(
>> type="imtcp"
>> port="6518"
>> )
>>
>>
>> I don't have many things I can check, as there is only 1 service relying on this. But that service is actually the only thing that's providing us insight into network logs. Incidentally our SIEM uses that as input.
>> So I did do a small check on the certificates:
>> openssl s_client -connect syslog.xx.nl:6518 -CAfile tls-ca-bundle.pem
>>
>> SSL handshake has read 17073 bytes and written 338 bytes
>> ---
>> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
>> Server public key is 2048 bit
>> Secure Renegotiation IS supported
>> Compression: NONE
>> Expansion: NONE
>> No ALPN negotiated
>> SSL-Session:
>> Protocol : TLSv1.2
>> Cipher : ECDHE-RSA-AES256-GCM-SHA384
>> Session-ID: 966666C3F3245570181E11CB525457BAD70D3F0E208D0CED75361980B7935465
>> Session-ID-ctx:
>> Master-Key: 2570E3FEC0FEBCB661A213FC38EF685FBACEA796E31CDSADH78HDFG6ADDFD0FDE9788CFD4C43B101A307345EC66C5C5BC0
>> Start Time: 1562755867
>> Timeout : 7200 (sec)
>> Verify return code: 0 (ok)
>>
>>
>> Trying the endpoint of this server always fails on the first certificate in the chain, hence why I'm adding it into that command.
>>
>> And in the config it does specify certs that are needed:
>> global(
>> DefaultNetstreamDriver="gtls"?
>> DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls-ca-bundle.pem"?
>> DefaultNetstreamDriverCertFile="/etc/rsyslog.d/server.crt"?
>> DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/server.key"?
>> )?
>>
>> I believe the use of gnutlsprioritystring setting requires version 8.29
>> We are running 8.24, however I do not see version 8.29 in the public yum repos for centos7? I might be looking in the wrong place though.
>>
>> Since it is broken I would rather move forward in limiting connections to just tls 1.2 instead of investing time into getting it rolled back to a working state. But I guess that all depends on how easy moving forward goes.
>> Is there anyone who has tips on moving forward?
>>
>> Best regards,
>>
>> Marco Moll
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix [ In reply to ]
rsyslog restarting never requires a reboot.

it looks like rsyslog is restarting, what is the problem you think is sticking
around?

the gnutls errors are indications that things are connecting to your encrypted
port and not talking TLS properly, not something that rsyslog can fix.

David Lang

On Fri, 21 Feb 2020, Moll, Marco via rsyslog wrote:

> Date: Fri, 21 Feb 2020 08:45:26 +0000
> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
> To: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
> Cc: "Moll, Marco" <marco.moll@sogeti.com>
> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls
> connections after config change, rollback doesn't fix
>
> These are the steps I take any time I do a restart or stop/start for rsyslog, and this is the current output:
> sudo systemctl disable rsyslog
> Removed symlink /etc/systemd/system/multi-user.target.wants/rsyslog.service.?sudo systemctl stop rsyslog?sudo systemctl enable rsyslog?
> Created symlink from /etc/systemd/system/multi-user.target.wants/rsyslog.service to /usr/lib/systemd/system/rsyslog.service.?
> sudo systemctl start rsyslog?sudo systemctl status rsyslog?
> ? rsyslog.service - System Logging Service?
> Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor pres et: enabled)?
> Active: active (running) since Fri 2020-02-21 09:36:36 CET; 5s ago?
> Docs: man:rsyslogd(8)?
> http://www.rsyslog.com/doc/?
> Main PID: 5372 (rsyslogd)?
> CGroup: /system.slice/rsyslog.service?
> ??5372 /usr/sbin/rsyslogd -n?
> ?
> Feb 21 09:36:36 systemd[1]: Started System Loggi...?
> Feb 21 09:36:37 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
> Feb 21 09:36:37 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
> Feb 21 09:36:38 rsyslogd[5372]: peer did not provide a certificate, not permitted to talk to it ...?
> Feb 21 09:36:38 rsyslogd[5372]: netstream session 0x7fa29004a7f0 from 10.xx.xx.xx will be closed due...?
> Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
> Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
> Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
> Feb 21 09:36:40 rsyslogd[5372]: unexpected GnuTLS error -110 in nsdsel_gtls.c:178: The TLS connection ...?
> Feb 21 09:36:40 rsyslogd[5372]: netstream session 0x7fa29004a7f0 from 10.xx.xx.xx will be closed due...?
> Hint: Some lines were ellipsized, use -l to show in full.?
>
> Im doing the disable/enable because I read in some other topic that a reboot was needed, but rebooting means alerts get send out to our pagerduty. So im circumventing that.
>
> When I try your command:
> sudo rsyslogd -N 1
> rsyslogd: version 8.24.0-34.el7, config validation run (level 1), master config /etc/rsyslog.conf?
> rsyslogd: End of config validation run. Bye.?
>
> ________________________________
> From: David Lang <david@lang.hm>
> Sent: Friday, February 21, 2020 9:31 AM
> To: Moll, Marco via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Moll, Marco <marco.moll@sogeti.com>
> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix
>
> the only thing that rsyslog persists between runs is state files for queues, do
> you have any of those configured in your system?, if so you can try to remove
> them
>
> do you get a syntax error if you do 'rsyslog -N 1'
>
> what error do you get at startup?
>
>
> On Fri, 21 Feb 2020, Moll, Marco via rsyslog wrote:
>
>> Date: Fri, 21 Feb 2020 08:27:24 +0000
>> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
>> To: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
>> Cc: "Moll, Marco" <marco.moll@sogeti.com>
>> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls
>> connections after config change, rollback doesn't fix
>>
>> I did try sudo yum update rsyslog, but all I get is that the latest packages are already installed. After checking, it seems we are using private repositories from one of our partners. But they say their repo is synced with the public repositories.
>> I'm going to take a guess it's this one: http://mirror.centos.org/centos/7/os/x86_64/Packages/
>>
>> And as far as I can see that one doesn't go beyond 8.24 ..
>>
>> And the entire system is ansible-managed, so whatever change I make: any rollback is basically just a revert on code. It's impossible for me to forget any part that got changed.
>>
>> So in order to upgrade I guess that means I need to add the repo via ansible, install the latest version and test.
>> Still leaves me hanging with a non-working rollbacked scenario where rsyslog should not be persistent.
>> What's the best way for me to figure out what's not working correct, and to fix it?
>> turn on rsyslog with debugging on 2? (Guess i'll go do that anyways, see how far that gets me)
>> Pointers for me to look at specifically maybe?
>> ________________________________
>> From: David Lang <david@lang.hm>
>> Sent: Thursday, February 20, 2020 8:09 PM
>> To: Moll, Marco via rsyslog <rsyslog@lists.adiscon.com>
>> Cc: Moll, Marco <marco.moll@sogeti.com>
>> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix
>>
>> 8.24 is over 3 years old now, after 8.39 (over a year old) we switched to a date
>> based version, and the current version is 8.2001 (8.YYMM)
>>
>> GNUTLS has some problems that will cause it to segfault under some conditions.
>> You should switch to openssl (requires a newer version of rsyslog)
>>
>> The PriorityString requires a newer version of gnutls and openssl libraries, I'm
>> not sure if centos 7 includes it, but centos6 does not.
>>
>> David Lang
>>
>> On Thu, 20 Feb 2020, Moll, Marco via rsyslog wrote:
>>
>>> Date: Thu, 20 Feb 2020 13:03:52 +0000
>>> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
>>> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
>>> Cc: "Moll, Marco" <marco.moll@sogeti.com>
>>> Subject: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections
>>> after config change, rollback doesn't fix
>>>
>>> Hi,
>>>
>>> I'm new to rsyslog and I'm trying to move forward on rsyslog due to a security advisory.
>>> I'm trying to achieve that rsyslog will no long accept ssl2, ssl3 or tls1 and only accepts tls 1.2
>>>
>>> I figured this would work:
>>> module(
>>> load="imtcp"
>>> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
>>> StreamDriver.Name="gtls"
>>> StreamDriver.Mode="1"
>>> StreamDriver.Authmode="x509/name"
>>> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
>>> # PermittedPeer=["*.enexis.nl"]
>>> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
>>> gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1
>>> MinProtocol=TLSv1.2"
>>> )
>>> input(
>>> type="imtcp"
>>> port="6518"
>>> )
>>>
>>> It did not, and I started seeing these errors in journal:
>>> peer did not provide a certificate, not permitted to talk to it [v8.24.0-34.el7 try http://www.rsyslog.com/e/2085 ]
>>> netstream session 0x7f10e40e8d80 from 10.xx.xx.xx will be closed due to error [v8.24.0-34.el7 try http://www.rsyslog.com/e/2089 ]
>>>
>>>
>>>
>>> I figured, well there is no change on the other end, so a rollback will fix it. That would give me re-assurance that I can rollback any time. Since we haven't tested that before.
>>> It did not, and now I have a broken rsyslog server ????
>>>
>>> Old config:
>>> module(
>>> load="imtcp"
>>> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
>>> StreamDriver.Name="gtls"
>>> StreamDriver.Mode="1"
>>> StreamDriver.Authmode="x509/name"
>>> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
>>> # PermittedPeer=["*.enexis.nl"]
>>> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
>>> )
>>> input(
>>> type="imtcp"
>>> port="6518"
>>> )
>>>
>>>
>>> I don't have many things I can check, as there is only 1 service relying on this. But that service is actually the only thing that's providing us insight into network logs. Incidentally our SIEM uses that as input.
>>> So I did do a small check on the certificates:
>>> openssl s_client -connect syslog.xx.nl:6518 -CAfile tls-ca-bundle.pem
>>>
>>> SSL handshake has read 17073 bytes and written 338 bytes
>>> ---
>>> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
>>> Server public key is 2048 bit
>>> Secure Renegotiation IS supported
>>> Compression: NONE
>>> Expansion: NONE
>>> No ALPN negotiated
>>> SSL-Session:
>>> Protocol : TLSv1.2
>>> Cipher : ECDHE-RSA-AES256-GCM-SHA384
>>> Session-ID: 966666C3F3245570181E11CB525457BAD70D3F0E208D0CED75361980B7935465
>>> Session-ID-ctx:
>>> Master-Key: 2570E3FEC0FEBCB661A213FC38EF685FBACEA796E31CDSADH78HDFG6ADDFD0FDE9788CFD4C43B101A307345EC66C5C5BC0
>>> Start Time: 1562755867
>>> Timeout : 7200 (sec)
>>> Verify return code: 0 (ok)
>>>
>>>
>>> Trying the endpoint of this server always fails on the first certificate in the chain, hence why I'm adding it into that command.
>>>
>>> And in the config it does specify certs that are needed:
>>> global(
>>> DefaultNetstreamDriver="gtls"?
>>> DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls-ca-bundle.pem"?
>>> DefaultNetstreamDriverCertFile="/etc/rsyslog.d/server.crt"?
>>> DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/server.key"?
>>> )?
>>>
>>> I believe the use of gnutlsprioritystring setting requires version 8.29
>>> We are running 8.24, however I do not see version 8.29 in the public yum repos for centos7? I might be looking in the wrong place though.
>>>
>>> Since it is broken I would rather move forward in limiting connections to just tls 1.2 instead of investing time into getting it rolled back to a working state. But I guess that all depends on how easy moving forward goes.
>>> Is there anyone who has tips on moving forward?
>>>
>>> Best regards,
>>>
>>> Marco Moll
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix [ In reply to ]
I was initially thinking it might be the certificate checks.
But other then the openssl command I already used I don't see a way to verify if rsyslog is having issues with that or not.

Because doing this: openssl s_client -connect syslog.xx.nl:6518
Tells me the first certificate in the chain failed. I would expect that to work without the -CAfile tls-ca-bundle.pem parameter if the ca-cert is available on the system.

Could it be possible that it's not working because I am providing a .pem instead of a .crt in the config?

But I haven't had time to enable debugging and delve deeper into it. So I hope to get on that today.
Also I found that CentOS7 doesn't include your latest version, so I also need to go check their policies or what else why they don't do that..
________________________________
From: David Lang <david@lang.hm>
Sent: Friday, February 21, 2020 6:54 PM
To: Moll, Marco via rsyslog <rsyslog@lists.adiscon.com>
Cc: Moll, Marco <marco.moll@sogeti.com>
Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix

rsyslog restarting never requires a reboot.

it looks like rsyslog is restarting, what is the problem you think is sticking
around?

the gnutls errors are indications that things are connecting to your encrypted
port and not talking TLS properly, not something that rsyslog can fix.

David Lang

On Fri, 21 Feb 2020, Moll, Marco via rsyslog wrote:

> Date: Fri, 21 Feb 2020 08:45:26 +0000
> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
> To: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
> Cc: "Moll, Marco" <marco.moll@sogeti.com>
> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls
> connections after config change, rollback doesn't fix
>
> These are the steps I take any time I do a restart or stop/start for rsyslog, and this is the current output:
> sudo systemctl disable rsyslog
> Removed symlink /etc/systemd/system/multi-user.target.wants/rsyslog.service.?sudo systemctl stop rsyslog?sudo systemctl enable rsyslog?
> Created symlink from /etc/systemd/system/multi-user.target.wants/rsyslog.service to /usr/lib/systemd/system/rsyslog.service.?
> sudo systemctl start rsyslog?sudo systemctl status rsyslog?
> ? rsyslog.service - System Logging Service?
> Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor pres et: enabled)?
> Active: active (running) since Fri 2020-02-21 09:36:36 CET; 5s ago?
> Docs: man:rsyslogd(8)?
> http://www.rsyslog.com/doc/?
> Main PID: 5372 (rsyslogd)?
> CGroup: /system.slice/rsyslog.service?
> ??5372 /usr/sbin/rsyslogd -n?
> ?
> Feb 21 09:36:36 systemd[1]: Started System Loggi...?
> Feb 21 09:36:37 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
> Feb 21 09:36:37 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
> Feb 21 09:36:38 rsyslogd[5372]: peer did not provide a certificate, not permitted to talk to it ...?
> Feb 21 09:36:38 rsyslogd[5372]: netstream session 0x7fa29004a7f0 from 10.xx.xx.xx will be closed due...?
> Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
> Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
> Feb 21 09:36:38 rsyslogd[5372]: gnutls returned error on handshake: The TLS connection was non-properl ...?
> Feb 21 09:36:40 rsyslogd[5372]: unexpected GnuTLS error -110 in nsdsel_gtls.c:178: The TLS connection ...?
> Feb 21 09:36:40 rsyslogd[5372]: netstream session 0x7fa29004a7f0 from 10.xx.xx.xx will be closed due...?
> Hint: Some lines were ellipsized, use -l to show in full.?
>
> Im doing the disable/enable because I read in some other topic that a reboot was needed, but rebooting means alerts get send out to our pagerduty. So im circumventing that.
>
> When I try your command:
> sudo rsyslogd -N 1
> rsyslogd: version 8.24.0-34.el7, config validation run (level 1), master config /etc/rsyslog.conf?
> rsyslogd: End of config validation run. Bye.?
>
> ________________________________
> From: David Lang <david@lang.hm>
> Sent: Friday, February 21, 2020 9:31 AM
> To: Moll, Marco via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Moll, Marco <marco.moll@sogeti.com>
> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix
>
> the only thing that rsyslog persists between runs is state files for queues, do
> you have any of those configured in your system?, if so you can try to remove
> them
>
> do you get a syntax error if you do 'rsyslog -N 1'
>
> what error do you get at startup?
>
>
> On Fri, 21 Feb 2020, Moll, Marco via rsyslog wrote:
>
>> Date: Fri, 21 Feb 2020 08:27:24 +0000
>> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
>> To: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
>> Cc: "Moll, Marco" <marco.moll@sogeti.com>
>> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls
>> connections after config change, rollback doesn't fix
>>
>> I did try sudo yum update rsyslog, but all I get is that the latest packages are already installed. After checking, it seems we are using private repositories from one of our partners. But they say their repo is synced with the public repositories.
>> I'm going to take a guess it's this one: http://mirror.centos.org/centos/7/os/x86_64/Packages/
>>
>> And as far as I can see that one doesn't go beyond 8.24 ..
>>
>> And the entire system is ansible-managed, so whatever change I make: any rollback is basically just a revert on code. It's impossible for me to forget any part that got changed.
>>
>> So in order to upgrade I guess that means I need to add the repo via ansible, install the latest version and test.
>> Still leaves me hanging with a non-working rollbacked scenario where rsyslog should not be persistent.
>> What's the best way for me to figure out what's not working correct, and to fix it?
>> turn on rsyslog with debugging on 2? (Guess i'll go do that anyways, see how far that gets me)
>> Pointers for me to look at specifically maybe?
>> ________________________________
>> From: David Lang <david@lang.hm>
>> Sent: Thursday, February 20, 2020 8:09 PM
>> To: Moll, Marco via rsyslog <rsyslog@lists.adiscon.com>
>> Cc: Moll, Marco <marco.moll@sogeti.com>
>> Subject: Re: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections after config change, rollback doesn't fix
>>
>> 8.24 is over 3 years old now, after 8.39 (over a year old) we switched to a date
>> based version, and the current version is 8.2001 (8.YYMM)
>>
>> GNUTLS has some problems that will cause it to segfault under some conditions.
>> You should switch to openssl (requires a newer version of rsyslog)
>>
>> The PriorityString requires a newer version of gnutls and openssl libraries, I'm
>> not sure if centos 7 includes it, but centos6 does not.
>>
>> David Lang
>>
>> On Thu, 20 Feb 2020, Moll, Marco via rsyslog wrote:
>>
>>> Date: Thu, 20 Feb 2020 13:03:52 +0000
>>> From: "Moll, Marco via rsyslog" <rsyslog@lists.adiscon.com>
>>> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
>>> Cc: "Moll, Marco" <marco.moll@sogeti.com>
>>> Subject: [rsyslog] Limitting rsyslog to just TLS v1.2 - broke tls connections
>>> after config change, rollback doesn't fix
>>>
>>> Hi,
>>>
>>> I'm new to rsyslog and I'm trying to move forward on rsyslog due to a security advisory.
>>> I'm trying to achieve that rsyslog will no long accept ssl2, ssl3 or tls1 and only accepts tls 1.2
>>>
>>> I figured this would work:
>>> module(
>>> load="imtcp"
>>> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
>>> StreamDriver.Name="gtls"
>>> StreamDriver.Mode="1"
>>> StreamDriver.Authmode="x509/name"
>>> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
>>> # PermittedPeer=["*.enexis.nl"]
>>> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
>>> gnutlsPriorityString="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1
>>> MinProtocol=TLSv1.2"
>>> )
>>> input(
>>> type="imtcp"
>>> port="6518"
>>> )
>>>
>>> It did not, and I started seeing these errors in journal:
>>> peer did not provide a certificate, not permitted to talk to it [v8.24.0-34.el7 try http://www.rsyslog.com/e/2085 ]
>>> netstream session 0x7f10e40e8d80 from 10.xx.xx.xx will be closed due to error [v8.24.0-34.el7 try http://www.rsyslog.com/e/2089 ]
>>>
>>>
>>>
>>> I figured, well there is no change on the other end, so a rollback will fix it. That would give me re-assurance that I can rollback any time. Since we haven't tested that before.
>>> It did not, and now I have a broken rsyslog server ????
>>>
>>> Old config:
>>> module(
>>> load="imtcp"
>>> # see https://www.rsyslog.com/doc/v8-stable/concepts/ns_gtls.html
>>> StreamDriver.Name="gtls"
>>> StreamDriver.Mode="1"
>>> StreamDriver.Authmode="x509/name"
>>> # See https://www.rsyslog.com/doc/v8-stable/configuration/modules/imtcp.html#permittedpeer
>>> # PermittedPeer=["*.enexis.nl"]
>>> PermittedPeer=["*.xx.nl", "*.yy.com", "*.yy.com.crt"]
>>> )
>>> input(
>>> type="imtcp"
>>> port="6518"
>>> )
>>>
>>>
>>> I don't have many things I can check, as there is only 1 service relying on this. But that service is actually the only thing that's providing us insight into network logs. Incidentally our SIEM uses that as input.
>>> So I did do a small check on the certificates:
>>> openssl s_client -connect syslog.xx.nl:6518 -CAfile tls-ca-bundle.pem
>>>
>>> SSL handshake has read 17073 bytes and written 338 bytes
>>> ---
>>> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
>>> Server public key is 2048 bit
>>> Secure Renegotiation IS supported
>>> Compression: NONE
>>> Expansion: NONE
>>> No ALPN negotiated
>>> SSL-Session:
>>> Protocol : TLSv1.2
>>> Cipher : ECDHE-RSA-AES256-GCM-SHA384
>>> Session-ID: 966666C3F3245570181E11CB525457BAD70D3F0E208D0CED75361980B7935465
>>> Session-ID-ctx:
>>> Master-Key: 2570E3FEC0FEBCB661A213FC38EF685FBACEA796E31CDSADH78HDFG6ADDFD0FDE9788CFD4C43B101A307345EC66C5C5BC0
>>> Start Time: 1562755867
>>> Timeout : 7200 (sec)
>>> Verify return code: 0 (ok)
>>>
>>>
>>> Trying the endpoint of this server always fails on the first certificate in the chain, hence why I'm adding it into that command.
>>>
>>> And in the config it does specify certs that are needed:
>>> global(
>>> DefaultNetstreamDriver="gtls"?
>>> DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls-ca-bundle.pem"?
>>> DefaultNetstreamDriverCertFile="/etc/rsyslog.d/server.crt"?
>>> DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/server.key"?
>>> )?
>>>
>>> I believe the use of gnutlsprioritystring setting requires version 8.29
>>> We are running 8.24, however I do not see version 8.29 in the public yum repos for centos7? I might be looking in the wrong place though.
>>>
>>> Since it is broken I would rather move forward in limiting connections to just tls 1.2 instead of investing time into getting it rolled back to a working state. But I guess that all depends on how easy moving forward goes.
>>> Is there anyone who has tips on moving forward?
>>>
>>> Best regards,
>>>
>>> Marco Moll
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.