Mailing List Archive

FWIW, we use FQDN to forward messages, but whenever our campus DNS servers experience issues our clients will backup and Nagios will start screaming about stuck items in the forward queue.

IP Address appear to make a more resilient forwarding target.

In our case we had good success with migrating a central receiver between subnets (i.e., IP change) and the clients picked up the change. I don't know whether this is because the receiver was down for a sufficient amount of time to force disconnect/reconnect behavior on the clients or if it's because we used the newer configuration format where you configure forwarding as an "action". To further stir mud in the water we are also using RELP, so that could have a bearing.

I recall seeing on the list somewhere some discussion about load-balancers and how forced disconnections can be used to switch targets. I might be thinking of forwarding into elasticsearch, so take that for what it's worth.

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Marki via rsyslog
Sent: Tuesday, October 15, 2019 11:31 AM
To: rsyslog@lists.adiscon.com
Cc: Marki <rsyslog@lists.roth.lu>
Subject: [rsyslog] Hostname resolution updates (remote logging) not picked up


Hey,

When using remote logging (*.* @syslog.example.com) "syslog" is an alias
(CNAME with low TTL) in our DNS, like all service names.

Now it seems when we change this alias' destination in DNS, the change
is never picked up. Not even on reload, only on restart. On reload would
at least make it use the new IP address after logrotation for example.

I don't even think it's about rsyslog. Seems to be how all syslog
implementations usually behave. But it is still a topic of discussion:

Are people just not using hostnames? I understand that for example on
network equipment you would rather hardcode IPs than use hostnames. But
what do you do on the servers?

Is there a best practice with valid reasons why it should be done that
way? What do you think?

Cheers.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

so there are a few things going on here

with rsyslog, a HUP signal does not reload the config, it closes outputs to
support log rotation

I'm not sure if it closes network connections or not, it not there is not going
to be a name lookup, I'm not sure if a name lookup would happen anyway as I
think rsyslog is farily aggressive in caching the results of a lookup.

We have the config option to close and re-open the connections after every X
messages so that any load balancing can take place.

David Lang


On Tue, 15 Oct 2019, Adam Chalkley via rsyslog wrote:

> Date: Tue, 15 Oct 2019 17:35:02 +0000
> From: Adam Chalkley via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Adam Chalkley <atc0005@auburn.edu>
> Subject: Re: [rsyslog] Hostname resolution updates (remote logging) not picked
> up
>
> FWIW, we use FQDN to forward messages, but whenever our campus DNS servers experience issues our clients will backup and Nagios will start screaming about stuck items in the forward queue.
>
> IP Address appear to make a more resilient forwarding target.
>
> In our case we had good success with migrating a central receiver between subnets (i.e., IP change) and the clients picked up the change. I don't know whether this is because the receiver was down for a sufficient amount of time to force disconnect/reconnect behavior on the clients or if it's because we used the newer configuration format where you configure forwarding as an "action". To further stir mud in the water we are also using RELP, so that could have a bearing.
>
> I recall seeing on the list somewhere some discussion about load-balancers and how forced disconnections can be used to switch targets. I might be thinking of forwarding into elasticsearch, so take that for what it's worth.
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Marki via rsyslog
> Sent: Tuesday, October 15, 2019 11:31 AM
> To: rsyslog@lists.adiscon.com
> Cc: Marki <rsyslog@lists.roth.lu>
> Subject: [rsyslog] Hostname resolution updates (remote logging) not picked up
>
>
> Hey,
>
> When using remote logging (*.* @syslog.example.com) "syslog" is an alias
> (CNAME with low TTL) in our DNS, like all service names.
>
> Now it seems when we change this alias' destination in DNS, the change
> is never picked up. Not even on reload, only on restart. On reload would
> at least make it use the new IP address after logrotation for example.
>
> I don't even think it's about rsyslog. Seems to be how all syslog
> implementations usually behave. But it is still a topic of discussion:
>
> Are people just not using hostnames? I understand that for example on
> network equipment you would rather hardcode IPs than use hostnames. But
> what do you do on the servers?
>
> Is there a best practice with valid reasons why it should be done that
> way? What do you think?
>
> Cheers.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

We had a little discussion about TCP reopening (which might include name
resolution) in following bug report (Reopen TCP sockets on HUP signal).
https://github.com/rsyslog/rsyslog/issues/3683

The outcome is to use rebindinterval omfwd config option which makes the
same, but cannot be enforced by user.

The name resolution takes effect within establishing the TCP connection.
Once established, there is no easy way to propagate the change to the
application. The POSIX name resolver just do not count with DNS TTL.
https://curl.haxx.se/mail/lib-2017-06/0022.html

--
Peter

On Tue, Oct 15, 2019 at 8:57 PM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> so there are a few things going on here
>
> with rsyslog, a HUP signal does not reload the config, it closes outputs
> to
> support log rotation
>
> I'm not sure if it closes network connections or not, it not there is not
> going
> to be a name lookup, I'm not sure if a name lookup would happen anyway as
> I
> think rsyslog is farily aggressive in caching the results of a lookup.
>
> We have the config option to close and re-open the connections after every
> X
> messages so that any load balancing can take place.
>
> David Lang
>
>
> On Tue, 15 Oct 2019, Adam Chalkley via rsyslog wrote:
>
> > Date: Tue, 15 Oct 2019 17:35:02 +0000
> > From: Adam Chalkley via rsyslog <rsyslog@lists.adiscon.com>
> > To: rsyslog-users <rsyslog@lists.adiscon.com>
> > Cc: Adam Chalkley <atc0005@auburn.edu>
> > Subject: Re: [rsyslog] Hostname resolution updates (remote logging) not
> picked
> > up
> >
> > FWIW, we use FQDN to forward messages, but whenever our campus DNS
> servers experience issues our clients will backup and Nagios will start
> screaming about stuck items in the forward queue.
> >
> > IP Address appear to make a more resilient forwarding target.
> >
> > In our case we had good success with migrating a central receiver
> between subnets (i.e., IP change) and the clients picked up the change. I
> don't know whether this is because the receiver was down for a sufficient
> amount of time to force disconnect/reconnect behavior on the clients or if
> it's because we used the newer configuration format where you configure
> forwarding as an "action". To further stir mud in the water we are also
> using RELP, so that could have a bearing.
> >
> > I recall seeing on the list somewhere some discussion about
> load-balancers and how forced disconnections can be used to switch targets.
> I might be thinking of forwarding into elasticsearch, so take that for what
> it's worth.
> >
> > -----Original Message-----
> > From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Marki
> via rsyslog
> > Sent: Tuesday, October 15, 2019 11:31 AM
> > To: rsyslog@lists.adiscon.com
> > Cc: Marki <rsyslog@lists.roth.lu>
> > Subject: [rsyslog] Hostname resolution updates (remote logging) not
> picked up
> >
> >
> > Hey,
> >
> > When using remote logging (*.* @syslog.example.com) "syslog" is an alias
> > (CNAME with low TTL) in our DNS, like all service names.
> >
> > Now it seems when we change this alias' destination in DNS, the change
> > is never picked up. Not even on reload, only on restart. On reload would
> > at least make it use the new IP address after logrotation for example.
> >
> > I don't even think it's about rsyslog. Seems to be how all syslog
> > implementations usually behave. But it is still a topic of discussion:
> >
> > Are people just not using hostnames? I understand that for example on
> > network equipment you would rather hardcode IPs than use hostnames. But
> > what do you do on the servers?
> >
> > Is there a best practice with valid reasons why it should be done that
> > way? What do you think?
> >
> > Cheers.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.