Mailing List Archive

The main question would be _why_ do you use LB in the first place. The
reasonable solution depends on the answer to this question.

If you had a network-level load-balancer, it would work without any
further configuration, as you have a proxy-type load-balancer, it's only
natural to have the LB IP as your $fromhost-ip and there is not much you
can do about.it.

You either need to configure your sources to provide individually
distinguishable source identifiers in the events (so that if you have
multiple logging servers they don't all report hostname of "server1") or
think about reorganizing your infrastructure.

In syslog there is no concept of "headers" so you can't add proxy
headers on your nginx. If you want to modify your event on the
loadbalancer, you'd need to use a syslog daemon (rsyslog?) for
load-balancing. But that seems a bit overengineered since probably just
a single rsyslog receiving events would suffice.

MK


On 22.08.2023 10:58, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
> Hello All rsyslog users and developers ????
>
> The main problem.
> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP added to the variable $fromhost-ip no matter what we do. This is due to the fact that the LB is doing TCP routing and becomes the sending IP on the network layer. The syslog message does not contain the IP of the sending node.
>
> The Setup:
> The setup contains only 2 servers which have 2 subnets (Frontend and a Backend(For TCP messages))
> The servers are using the following software
>
> * Nginx (For LB)
> * Keepalived (For VIP handling)
> * Rsyslog (yes for syslog messages)
>
>
> Question 1:
> I was thinking about adding some Proxy Headers to the log message on the LB (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is this possible in anyway to have rsyslog to use Proxy Headers like:
>
> * X-Forwarded-For
> * X-Real-IP
>
> If possible howto do it ?
> If not, any other great suggestions to preserve IP address of sending source in a LB TCP syslog setup is appreciated
>
> Please be aware, adding more HW to the setup is not an option.
>
> Best regards
> Jan P. Madsen
>
> This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

the syslog message is supposed to have the hostname of the sending system in it,
you can use that directly if you want.

fromhost-ip is specifically bypassing the metadata in the log itself to tell you
the IP of the last hop that sent it to you, and as you see, when you run it
through a non-transparent load balancer, you get the load balancer instead of
the initial sender.

where would you set your additional headers? you can set the hostname there at
least as easily as you could add something extra (I do like to add extra stuff,
I make the body of the message JSON and add a subtree with a company-specific
name that's unlikely to match anything in the message, and then make the
original message $!msg inside this json

one problem with using a load balancer like nginx is that it doesn't understand
the syslog protocol, so it can't break the connection to rebalance at a sane
point

If you are doing the load balancing on the two systems that are receiving the
messages, instead of using nginx/keepalived, look at using corosync/pacemaker
and use the CLUSTERIP option to share a single VIP across both machines. If you
include the senders port number in the balancing, it works well to spread the
incoming load pretty evenly

Rsyslog also has the rebindinterval option that you can set on the senders to
have them disconnect and reconnect periodically (which give a load balancer the
chance to rebalance when it reconnects

I like to put a pair of load balanced syslog relays on each subnet (sometimes
multi-homed as you are doing) to gather all the logs from that subnet, clean
them up, parse them, add metadata, and then deliver to the central syslog server
(if that central server is in a different datacenter, the RELP protocol for that
hop helps avoid log loss that can happen with TCP)


David Lang

On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:

> Date: Tue, 22 Aug 2023 08:58:02 +0000
> From: "JPIM (Jan Primdahl Madsen) via rsyslog" <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: "JPIM (Jan Primdahl Madsen)" <JPIM@aevengroup.com>
> Subject: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages
>
>
> Hello All rsyslog users and developers ????
>
> The main problem.
> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP added to the variable $fromhost-ip no matter what we do. This is due to the fact that the LB is doing TCP routing and becomes the sending IP on the network layer. The syslog message does not contain the IP of the sending node.
>
> The Setup:
> The setup contains only 2 servers which have 2 subnets (Frontend and a Backend(For TCP messages))
> The servers are using the following software
>
> * Nginx (For LB)
> * Keepalived (For VIP handling)
> * Rsyslog (yes for syslog messages)
>
>
> Question 1:
> I was thinking about adding some Proxy Headers to the log message on the LB (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is this possible in anyway to have rsyslog to use Proxy Headers like:
>
> * X-Forwarded-For
> * X-Real-IP
>
> If possible howto do it ?
> If not, any other great suggestions to preserve IP address of sending source in a LB TCP syslog setup is appreciated
>
> Please be aware, adding more HW to the setup is not an option.
>
> Best regards
> Jan P. Madsen
>
> This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Having a syslog cluster with LB in front is for scalability MK.
I'm going to receive a huge amount of syslogs from servers, network equipment as Firewalls connections.
So yes I want to be able to scale enough by LB between multiple log servers.

But got your point of not able to use header stuff.
I will try to look on some of the point David Lang mention by looking into the use of corosync/pacemaker if this is something that can help solving our challenge.

Thanks a lot for the answer.
Best regards
Jan


-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
Sent: 22. august 2023 11:09
To: rsyslog@lists.adiscon.com
Cc: Mariusz Kruk <kruk@epsilon.eu.org>
Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages

Caution: This email originates from outside the company. You should carefully examine the sender and any links or actions before clicking. Use the "Report Phishing" button on the ribbon to have Security analyze the email if in doubt.


The main question would be _why_ do you use LB in the first place. The reasonable solution depends on the answer to this question.

If you had a network-level load-balancer, it would work without any further configuration, as you have a proxy-type load-balancer, it's only natural to have the LB IP as your $fromhost-ip and there is not much you can do about.it.

You either need to configure your sources to provide individually distinguishable source identifiers in the events (so that if you have multiple logging servers they don't all report hostname of "server1") or think about reorganizing your infrastructure.

In syslog there is no concept of "headers" so you can't add proxy headers on your nginx. If you want to modify your event on the loadbalancer, you'd need to use a syslog daemon (rsyslog?) for load-balancing. But that seems a bit overengineered since probably just a single rsyslog receiving events would suffice.

MK


On 22.08.2023 10:58, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
> Hello All rsyslog users and developers ????
>
> The main problem.
> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP added to the variable $fromhost-ip no matter what we do. This is due to the fact that the LB is doing TCP routing and becomes the sending IP on the network layer. The syslog message does not contain the IP of the sending node.
>
> The Setup:
> The setup contains only 2 servers which have 2 subnets (Frontend and a
> Backend(For TCP messages)) The servers are using the following
> software
>
> * Nginx (For LB)
> * Keepalived (For VIP handling)
> * Rsyslog (yes for syslog messages)
>
>
> Question 1:
> I was thinking about adding some Proxy Headers to the log message on the LB (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is this possible in anyway to have rsyslog to use Proxy Headers like:
>
> * X-Forwarded-For
> * X-Real-IP
>
> If possible howto do it ?
> If not, any other great suggestions to preserve IP address of sending
> source in a LB TCP syslog setup is appreciated
>
> Please be aware, adding more HW to the setup is not an option.
>
> Best regards
> Jan P. Madsen
>
> This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

what are you considering 'a huge amount of syslogs'? Rsyslog is fast enough that
you seldom need multiple systems for capacity, just for redundency.

David Lang

On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:

> Having a syslog cluster with LB in front is for scalability MK.
> I'm going to receive a huge amount of syslogs from servers, network equipment as Firewalls connections.
> So yes I want to be able to scale enough by LB between multiple log servers.
>
> But got your point of not able to use header stuff.
> I will try to look on some of the point David Lang mention by looking into the use of corosync/pacemaker if this is something that can help solving our challenge.
>
> Thanks a lot for the answer.
> Best regards
> Jan
>
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
> Sent: 22. august 2023 11:09
> To: rsyslog@lists.adiscon.com
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages
>
> Caution: This email originates from outside the company. You should carefully examine the sender and any links or actions before clicking. Use the "Report Phishing" button on the ribbon to have Security analyze the email if in doubt.
>
>
> The main question would be _why_ do you use LB in the first place. The reasonable solution depends on the answer to this question.
>
> If you had a network-level load-balancer, it would work without any further configuration, as you have a proxy-type load-balancer, it's only natural to have the LB IP as your $fromhost-ip and there is not much you can do about.it.
>
> You either need to configure your sources to provide individually distinguishable source identifiers in the events (so that if you have multiple logging servers they don't all report hostname of "server1") or think about reorganizing your infrastructure.
>
> In syslog there is no concept of "headers" so you can't add proxy headers on your nginx. If you want to modify your event on the loadbalancer, you'd need to use a syslog daemon (rsyslog?) for load-balancing. But that seems a bit overengineered since probably just a single rsyslog receiving events would suffice.
>
> MK
>
>
> On 22.08.2023 10:58, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>> Hello All rsyslog users and developers ????
>>
>> The main problem.
>> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP added to the variable $fromhost-ip no matter what we do. This is due to the fact that the LB is doing TCP routing and becomes the sending IP on the network layer. The syslog message does not contain the IP of the sending node.
>>
>> The Setup:
>> The setup contains only 2 servers which have 2 subnets (Frontend and a
>> Backend(For TCP messages)) The servers are using the following
>> software
>>
>> * Nginx (For LB)
>> * Keepalived (For VIP handling)
>> * Rsyslog (yes for syslog messages)
>>
>>
>> Question 1:
>> I was thinking about adding some Proxy Headers to the log message on the LB (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is this possible in anyway to have rsyslog to use Proxy Headers like:
>>
>> * X-Forwarded-For
>> * X-Real-IP
>>
>> If possible howto do it ?
>> If not, any other great suggestions to preserve IP address of sending
>> source in a LB TCP syslog setup is appreciated
>>
>> Please be aware, adding more HW to the setup is not an option.
>>
>> Best regards
>> Jan P. Madsen
>>
>> This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On Tue, 22 Aug 2023, Mariusz Kruk via rsyslog wrote:

> In syslog there is no concept of "headers" so you can't add proxy
> headers on your nginx. If you want to modify your event on the
> loadbalancer, you'd need to use a syslog daemon (rsyslog?) for
> load-balancing. But that seems a bit overengineered since probably just
> a single rsyslog receiving events would suffice.

while rsyslog doesn't have 'headers' you can simulate it quite well by changing
the format of the message body from plain text to json (with a msg field that
contains the original plain text)

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi David

Point taken on the fromhost-ip is the last hop just before "me"(the syslog hosts)

This is what I wanted to do:
template(name="forwardlog" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg% <%fromhost-ip%|%timereported:::date-unixtimestamp%|%$myhostname%|hlu>\n")

I was thinking of adding the headers on the LB side, and then use these headers on the rsyslog side to add the source IP to my log message using a template.
So what I'm actually is asking. Would rsyslog be able to fetch a X-Forward Header or X-Real-IP header and use that in a template and add it to the message ?

So I "just" wanted to pickup the X-Forward header and use that instead of fromhost-ip

I will try to look into the corosync/pacemaker and see if this can help me with the challenge.

Best regards
Jan

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: 22. august 2023 11:11
To: JPIM (Jan Primdahl Madsen) via rsyslog <rsyslog@lists.adiscon.com>
Cc: JPIM (Jan Primdahl Madsen) <JPIM@aevengroup.com>
Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages

Caution: This email originates from outside the company. You should carefully examine the sender and any links or actions before clicking. Use the "Report Phishing" button on the ribbon to have Security analyze the email if in doubt.


the syslog message is supposed to have the hostname of the sending system in it, you can use that directly if you want.

fromhost-ip is specifically bypassing the metadata in the log itself to tell you the IP of the last hop that sent it to you, and as you see, when you run it through a non-transparent load balancer, you get the load balancer instead of the initial sender.

where would you set your additional headers? you can set the hostname there at least as easily as you could add something extra (I do like to add extra stuff, I make the body of the message JSON and add a subtree with a company-specific name that's unlikely to match anything in the message, and then make the original message $!msg inside this json

one problem with using a load balancer like nginx is that it doesn't understand the syslog protocol, so it can't break the connection to rebalance at a sane point

If you are doing the load balancing on the two systems that are receiving the messages, instead of using nginx/keepalived, look at using corosync/pacemaker and use the CLUSTERIP option to share a single VIP across both machines. If you include the senders port number in the balancing, it works well to spread the incoming load pretty evenly

Rsyslog also has the rebindinterval option that you can set on the senders to have them disconnect and reconnect periodically (which give a load balancer the chance to rebalance when it reconnects

I like to put a pair of load balanced syslog relays on each subnet (sometimes multi-homed as you are doing) to gather all the logs from that subnet, clean them up, parse them, add metadata, and then deliver to the central syslog server (if that central server is in a different datacenter, the RELP protocol for that hop helps avoid log loss that can happen with TCP)


David Lang

On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:

> Date: Tue, 22 Aug 2023 08:58:02 +0000
> From: "JPIM (Jan Primdahl Madsen) via rsyslog"
> <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: "JPIM (Jan Primdahl Madsen)" <JPIM@aevengroup.com>
> Subject: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
> messages
>
>
> Hello All rsyslog users and developers ????
>
> The main problem.
> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP added to the variable $fromhost-ip no matter what we do. This is due to the fact that the LB is doing TCP routing and becomes the sending IP on the network layer. The syslog message does not contain the IP of the sending node.
>
> The Setup:
> The setup contains only 2 servers which have 2 subnets (Frontend and a
> Backend(For TCP messages)) The servers are using the following
> software
>
> * Nginx (For LB)
> * Keepalived (For VIP handling)
> * Rsyslog (yes for syslog messages)
>
>
> Question 1:
> I was thinking about adding some Proxy Headers to the log message on the LB (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is this possible in anyway to have rsyslog to use Proxy Headers like:
>
> * X-Forwarded-For
> * X-Real-IP
>
> If possible howto do it ?
> If not, any other great suggestions to preserve IP address of sending
> source in a LB TCP syslog setup is appreciated
>
> Please be aware, adding more HW to the setup is not an option.
>
> Best regards
> Jan P. Madsen
>
> This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Huge amount.
2000+ servers
2200 network nodes where 200 is firewalls logging GB of date each every day.

This is where I wanted to be able to scale out when 1 node is not enough, and 2 nodes is currently added for redundancy. (Yes could run Active/Passive setup sharing a VIP with no LB and this would probably fix my challenges for now, but will break the scalability option)
I also do understand that NIC's could be a bottleneck on the LB side in the huge amount of data which is another headache I need to crack if possible.
But for making sure the the Log Servers can keep up and not hit IO Waite state on disk side I'm scaling out.

Best regards
Jan

-----Original Message-----
From: David Lang <david@lang.hm>
Sent: 22. august 2023 11:32
To: JPIM (Jan Primdahl Madsen) via rsyslog <rsyslog@lists.adiscon.com>
Cc: JPIM (Jan Primdahl Madsen) <JPIM@aevengroup.com>
Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages

what are you considering 'a huge amount of syslogs'? Rsyslog is fast enough that you seldom need multiple systems for capacity, just for redundency.

David Lang

On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:

> Having a syslog cluster with LB in front is for scalability MK.
> I'm going to receive a huge amount of syslogs from servers, network equipment as Firewalls connections.
> So yes I want to be able to scale enough by LB between multiple log servers.
>
> But got your point of not able to use header stuff.
> I will try to look on some of the point David Lang mention by looking into the use of corosync/pacemaker if this is something that can help solving our challenge.
>
> Thanks a lot for the answer.
> Best regards
> Jan
>
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz
> Kruk via rsyslog
> Sent: 22. august 2023 11:09
> To: rsyslog@lists.adiscon.com
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
> messages
>
> Caution: This email originates from outside the company. You should carefully examine the sender and any links or actions before clicking. Use the "Report Phishing" button on the ribbon to have Security analyze the email if in doubt.
>
>
> The main question would be _why_ do you use LB in the first place. The reasonable solution depends on the answer to this question.
>
> If you had a network-level load-balancer, it would work without any further configuration, as you have a proxy-type load-balancer, it's only natural to have the LB IP as your $fromhost-ip and there is not much you can do about.it.
>
> You either need to configure your sources to provide individually distinguishable source identifiers in the events (so that if you have multiple logging servers they don't all report hostname of "server1") or think about reorganizing your infrastructure.
>
> In syslog there is no concept of "headers" so you can't add proxy headers on your nginx. If you want to modify your event on the loadbalancer, you'd need to use a syslog daemon (rsyslog?) for load-balancing. But that seems a bit overengineered since probably just a single rsyslog receiving events would suffice.
>
> MK
>
>
> On 22.08.2023 10:58, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>> Hello All rsyslog users and developers ????
>>
>> The main problem.
>> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP added to the variable $fromhost-ip no matter what we do. This is due to the fact that the LB is doing TCP routing and becomes the sending IP on the network layer. The syslog message does not contain the IP of the sending node.
>>
>> The Setup:
>> The setup contains only 2 servers which have 2 subnets (Frontend and
>> a Backend(For TCP messages)) The servers are using the following
>> software
>>
>> * Nginx (For LB)
>> * Keepalived (For VIP handling)
>> * Rsyslog (yes for syslog messages)
>>
>>
>> Question 1:
>> I was thinking about adding some Proxy Headers to the log message on the LB (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is this possible in anyway to have rsyslog to use Proxy Headers like:
>>
>> * X-Forwarded-For
>> * X-Real-IP
>>
>> If possible howto do it ?
>> If not, any other great suggestions to preserve IP address of sending
>> source in a LB TCP syslog setup is appreciated
>>
>> Please be aware, adding more HW to the setup is not an option.
>>
>> Best regards
>> Jan P. Madsen
>>
>> This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

It's not about the number of nodes :-)

I have multi-layered rsyslog-based solution where the central node
processes about 35k EPS and load-balances them over several outgoing
RELP connections and it doesn't break a sweat (true, it sends the data
out to "workers" for the heavier parts of event processing).

@David Yes, I know you can embed your original message in json. You can
do a similar thing by cleverly formatting the forwarder event and
appending the original IP to the message in a form that can be
"unpacked". But it of course does need a syslog daemon as the load
balancer, not a simple connection relay.

MK

On 22.08.2023 11:38, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
> Huge amount.
> 2000+ servers
> 2200 network nodes where 200 is firewalls logging GB of date each every day.
>
> This is where I wanted to be able to scale out when 1 node is not enough, and 2 nodes is currently added for redundancy. (Yes could run Active/Passive setup sharing a VIP with no LB and this would probably fix my challenges for now, but will break the scalability option)
> I also do understand that NIC's could be a bottleneck on the LB side in the huge amount of data which is another headache I need to crack if possible.
> But for making sure the the Log Servers can keep up and not hit IO Waite state on disk side I'm scaling out.
>
> Best regards
> Jan
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: 22. august 2023 11:32
> To: JPIM (Jan Primdahl Madsen) via rsyslog <rsyslog@lists.adiscon.com>
> Cc: JPIM (Jan Primdahl Madsen) <JPIM@aevengroup.com>
> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages
>
> what are you considering 'a huge amount of syslogs'? Rsyslog is fast enough that you seldom need multiple systems for capacity, just for redundency.
>
> David Lang
>
> On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>
>> Having a syslog cluster with LB in front is for scalability MK.
>> I'm going to receive a huge amount of syslogs from servers, network equipment as Firewalls connections.
>> So yes I want to be able to scale enough by LB between multiple log servers.
>>
>> But got your point of not able to use header stuff.
>> I will try to look on some of the point David Lang mention by looking into the use of corosync/pacemaker if this is something that can help solving our challenge.
>>
>> Thanks a lot for the answer.
>> Best regards
>> Jan
>>
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz
>> Kruk via rsyslog
>> Sent: 22. august 2023 11:09
>> To: rsyslog@lists.adiscon.com
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
>> messages
>>
>> Caution: This email originates from outside the company. You should carefully examine the sender and any links or actions before clicking. Use the "Report Phishing" button on the ribbon to have Security analyze the email if in doubt.
>>
>>
>> The main question would be _why_ do you use LB in the first place. The reasonable solution depends on the answer to this question.
>>
>> If you had a network-level load-balancer, it would work without any further configuration, as you have a proxy-type load-balancer, it's only natural to have the LB IP as your $fromhost-ip and there is not much you can do about.it.
>>
>> You either need to configure your sources to provide individually distinguishable source identifiers in the events (so that if you have multiple logging servers they don't all report hostname of "server1") or think about reorganizing your infrastructure.
>>
>> In syslog there is no concept of "headers" so you can't add proxy headers on your nginx. If you want to modify your event on the loadbalancer, you'd need to use a syslog daemon (rsyslog?) for load-balancing. But that seems a bit overengineered since probably just a single rsyslog receiving events would suffice.
>>
>> MK
>>
>>
>> On 22.08.2023 10:58, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>>> Hello All rsyslog users and developers ????
>>>
>>> The main problem.
>>> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP added to the variable $fromhost-ip no matter what we do. This is due to the fact that the LB is doing TCP routing and becomes the sending IP on the network layer. The syslog message does not contain the IP of the sending node.
>>>
>>> The Setup:
>>> The setup contains only 2 servers which have 2 subnets (Frontend and
>>> a Backend(For TCP messages)) The servers are using the following
>>> software
>>>
>>> * Nginx (For LB)
>>> * Keepalived (For VIP handling)
>>> * Rsyslog (yes for syslog messages)
>>>
>>>
>>> Question 1:
>>> I was thinking about adding some Proxy Headers to the log message on the LB (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is this possible in anyway to have rsyslog to use Proxy Headers like:
>>>
>>> * X-Forwarded-For
>>> * X-Real-IP
>>>
>>> If possible howto do it ?
>>> If not, any other great suggestions to preserve IP address of sending
>>> source in a LB TCP syslog setup is appreciated
>>>
>>> Please be aware, adding more HW to the setup is not an option.
>>>
>>> Best regards
>>> Jan P. Madsen
>>>
>>> This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hey guys.
So what you are mentioning here is that I could make something like:

Client --> TCP 10517 --> rsyslog(Frontend) --> process data, change message based on Template --> LB Data to other rsyslog servers --> Store Data

What would be used to LB the data. Yet another "frontend" to do the connection relay ?

Best regards
Jan

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
Sent: 22. august 2023 11:44
To: rsyslog@lists.adiscon.com
Cc: Mariusz Kruk <kruk@epsilon.eu.org>
Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages

It's not about the number of nodes :-)

I have multi-layered rsyslog-based solution where the central node processes about 35k EPS and load-balances them over several outgoing RELP connections and it doesn't break a sweat (true, it sends the data out to "workers" for the heavier parts of event processing).

@David Yes, I know you can embed your original message in json. You can do a similar thing by cleverly formatting the forwarder event and appending the original IP to the message in a form that can be "unpacked". But it of course does need a syslog daemon as the load balancer, not a simple connection relay.

MK

On 22.08.2023 11:38, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
> Huge amount.
> 2000+ servers
> 2200 network nodes where 200 is firewalls logging GB of date each every day.
>
> This is where I wanted to be able to scale out when 1 node is not
> enough, and 2 nodes is currently added for redundancy. (Yes could run Active/Passive setup sharing a VIP with no LB and this would probably fix my challenges for now, but will break the scalability option) I also do understand that NIC's could be a bottleneck on the LB side in the huge amount of data which is another headache I need to crack if possible.
> But for making sure the the Log Servers can keep up and not hit IO Waite state on disk side I'm scaling out.
>
> Best regards
> Jan
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: 22. august 2023 11:32
> To: JPIM (Jan Primdahl Madsen) via rsyslog <rsyslog@lists.adiscon.com>
> Cc: JPIM (Jan Primdahl Madsen) <JPIM@aevengroup.com>
> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
> messages
>
> what are you considering 'a huge amount of syslogs'? Rsyslog is fast enough that you seldom need multiple systems for capacity, just for redundency.
>
> David Lang
>
> On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>
>> Having a syslog cluster with LB in front is for scalability MK.
>> I'm going to receive a huge amount of syslogs from servers, network equipment as Firewalls connections.
>> So yes I want to be able to scale enough by LB between multiple log servers.
>>
>> But got your point of not able to use header stuff.
>> I will try to look on some of the point David Lang mention by looking into the use of corosync/pacemaker if this is something that can help solving our challenge.
>>
>> Thanks a lot for the answer.
>> Best regards
>> Jan
>>
>>
>> -----Original Message-----
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>> Mariusz Kruk via rsyslog
>> Sent: 22. august 2023 11:09
>> To: rsyslog@lists.adiscon.com
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
>> messages
>>
>> Caution: This email originates from outside the company. You should carefully examine the sender and any links or actions before clicking. Use the "Report Phishing" button on the ribbon to have Security analyze the email if in doubt.
>>
>>
>> The main question would be _why_ do you use LB in the first place. The reasonable solution depends on the answer to this question.
>>
>> If you had a network-level load-balancer, it would work without any further configuration, as you have a proxy-type load-balancer, it's only natural to have the LB IP as your $fromhost-ip and there is not much you can do about.it.
>>
>> You either need to configure your sources to provide individually distinguishable source identifiers in the events (so that if you have multiple logging servers they don't all report hostname of "server1") or think about reorganizing your infrastructure.
>>
>> In syslog there is no concept of "headers" so you can't add proxy headers on your nginx. If you want to modify your event on the loadbalancer, you'd need to use a syslog daemon (rsyslog?) for load-balancing. But that seems a bit overengineered since probably just a single rsyslog receiving events would suffice.
>>
>> MK
>>
>>
>> On 22.08.2023 10:58, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>>> Hello All rsyslog users and developers ????
>>>
>>> The main problem.
>>> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP added to the variable $fromhost-ip no matter what we do. This is due to the fact that the LB is doing TCP routing and becomes the sending IP on the network layer. The syslog message does not contain the IP of the sending node.
>>>
>>> The Setup:
>>> The setup contains only 2 servers which have 2 subnets (Frontend and
>>> a Backend(For TCP messages)) The servers are using the following
>>> software
>>>
>>> * Nginx (For LB)
>>> * Keepalived (For VIP handling)
>>> * Rsyslog (yes for syslog messages)
>>>
>>>
>>> Question 1:
>>> I was thinking about adding some Proxy Headers to the log message on the LB (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is this possible in anyway to have rsyslog to use Proxy Headers like:
>>>
>>> * X-Forwarded-For
>>> * X-Real-IP
>>>
>>> If possible howto do it ?
>>> If not, any other great suggestions to preserve IP address of
>>> sending source in a LB TCP syslog setup is appreciated
>>>
>>> Please be aware, adding more HW to the setup is not an option.
>>>
>>> Best regards
>>> Jan P. Madsen
>>>
>>> This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

the problem is that nginx doesn't understand syslog, so it doesn't know what a
single message is and can't modify per message.

the x-forwarder type header works in http because nginx understands the
protocol and can modify it. I haven't heard that nginx has grown that
capability.

CLUSTERIP is a way to have multiple machines on a subnet share the same IP (at
the network layer, the switch send the logs to all of them and the IP stack on
the syslog servers ignores all connections except it's 'share' of them), so you
still have to have a network that's fast enough, but a gigabit network can take
you over 300k logs/sec, which would be ~10TB/day of logs if kept fully saturated
for 24 hours.

I have run systems at those rates for testing, my real world log data from ~5k
servers ended up being <100k logs/sec at peak, but I tested to saturation.

David Lang

On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) wrote:

> Date: Tue, 22 Aug 2023 09:34:38 +0000
> From: "JPIM (Jan Primdahl Madsen)" <JPIM@aevengroup.com>
> To: David Lang <david@lang.hm>,
> "JPIM (Jan Primdahl Madsen) via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: RE: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages
>
> Hi David
>
> Point taken on the fromhost-ip is the last hop just before "me"(the syslog hosts)
>
> This is what I wanted to do:
> template(name="forwardlog" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg% <%fromhost-ip%|%timereported:::date-unixtimestamp%|%$myhostname%|hlu>\n")
>
> I was thinking of adding the headers on the LB side, and then use these headers on the rsyslog side to add the source IP to my log message using a template.
> So what I'm actually is asking. Would rsyslog be able to fetch a X-Forward Header or X-Real-IP header and use that in a template and add it to the message ?
>
> So I "just" wanted to pickup the X-Forward header and use that instead of fromhost-ip
>
> I will try to look into the corosync/pacemaker and see if this can help me with the challenge.
>
> Best regards
> Jan
>
> -----Original Message-----
> From: David Lang <david@lang.hm>
> Sent: 22. august 2023 11:11
> To: JPIM (Jan Primdahl Madsen) via rsyslog <rsyslog@lists.adiscon.com>
> Cc: JPIM (Jan Primdahl Madsen) <JPIM@aevengroup.com>
> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages
>
> Caution: This email originates from outside the company. You should carefully examine the sender and any links or actions before clicking. Use the "Report Phishing" button on the ribbon to have Security analyze the email if in doubt.
>
>
> the syslog message is supposed to have the hostname of the sending system in it, you can use that directly if you want.
>
> fromhost-ip is specifically bypassing the metadata in the log itself to tell you the IP of the last hop that sent it to you, and as you see, when you run it through a non-transparent load balancer, you get the load balancer instead of the initial sender.
>
> where would you set your additional headers? you can set the hostname there at least as easily as you could add something extra (I do like to add extra stuff, I make the body of the message JSON and add a subtree with a company-specific name that's unlikely to match anything in the message, and then make the original message $!msg inside this json
>
> one problem with using a load balancer like nginx is that it doesn't understand the syslog protocol, so it can't break the connection to rebalance at a sane point
>
> If you are doing the load balancing on the two systems that are receiving the messages, instead of using nginx/keepalived, look at using corosync/pacemaker and use the CLUSTERIP option to share a single VIP across both machines. If you include the senders port number in the balancing, it works well to spread the incoming load pretty evenly
>
> Rsyslog also has the rebindinterval option that you can set on the senders to have them disconnect and reconnect periodically (which give a load balancer the chance to rebalance when it reconnects
>
> I like to put a pair of load balanced syslog relays on each subnet (sometimes multi-homed as you are doing) to gather all the logs from that subnet, clean them up, parse them, add metadata, and then deliver to the central syslog server (if that central server is in a different datacenter, the RELP protocol for that hop helps avoid log loss that can happen with TCP)
>
>
> David Lang
>
> On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>
>> Date: Tue, 22 Aug 2023 08:58:02 +0000
>> From: "JPIM (Jan Primdahl Madsen) via rsyslog"
>> <rsyslog@lists.adiscon.com>
>> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
>> Cc: "JPIM (Jan Primdahl Madsen)" <JPIM@aevengroup.com>
>> Subject: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
>> messages
>>
>>
>> Hello All rsyslog users and developers ????
>>
>> The main problem.
>> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP added to the variable $fromhost-ip no matter what we do. This is due to the fact that the LB is doing TCP routing and becomes the sending IP on the network layer. The syslog message does not contain the IP of the sending node.
>>
>> The Setup:
>> The setup contains only 2 servers which have 2 subnets (Frontend and a
>> Backend(For TCP messages)) The servers are using the following
>> software
>>
>> * Nginx (For LB)
>> * Keepalived (For VIP handling)
>> * Rsyslog (yes for syslog messages)
>>
>>
>> Question 1:
>> I was thinking about adding some Proxy Headers to the log message on the LB (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is this possible in anyway to have rsyslog to use Proxy Headers like:
>>
>> * X-Forwarded-For
>> * X-Real-IP
>>
>> If possible howto do it ?
>> If not, any other great suggestions to preserve IP address of sending
>> source in a LB TCP syslog setup is appreciated
>>
>> Please be aware, adding more HW to the setup is not an option.
>>
>> Best regards
>> Jan P. Madsen
>>
>> This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

With sufficiently big and fragmented infrastructure you could even go
further and separate your environment into separate layers -

source -> local rsyslog receiver(possibly duplicated in HA pair with
VIP) --- load-balanced RELP connections -> final rsyslog receivers ->
log files

On a bit smaller scale you could just do

source -> rsyslog-based load-balancer (in HA pair with VIP) --
load-balanced RELP connections -> final rsyslog receivers -> log files

There are many possible architectures.

It does involve a bit of configuration magic to relay that source IP
from the source to the destination instance but it is definitely possible.

MK

On 22.08.2023 11:51, JPIM (Jan Primdahl Madsen) wrote:
> Hey guys.
> So what you are mentioning here is that I could make something like:
>
> Client --> TCP 10517 --> rsyslog(Frontend) --> process data, change message based on Template --> LB Data to other rsyslog servers --> Store Data
>
> What would be used to LB the data. Yet another "frontend" to do the connection relay ?
>
> Best regards
> Jan
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
> Sent: 22. august 2023 11:44
> To: rsyslog@lists.adiscon.com
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages
>
> It's not about the number of nodes :-)
>
> I have multi-layered rsyslog-based solution where the central node processes about 35k EPS and load-balances them over several outgoing RELP connections and it doesn't break a sweat (true, it sends the data out to "workers" for the heavier parts of event processing).
>
> @David Yes, I know you can embed your original message in json. You can do a similar thing by cleverly formatting the forwarder event and appending the original IP to the message in a form that can be "unpacked". But it of course does need a syslog daemon as the load balancer, not a simple connection relay.
>
> MK
>
> On 22.08.2023 11:38, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>> Huge amount.
>> 2000+ servers
>> 2200 network nodes where 200 is firewalls logging GB of date each every day.
>>
>> This is where I wanted to be able to scale out when 1 node is not
>> enough, and 2 nodes is currently added for redundancy. (Yes could run Active/Passive setup sharing a VIP with no LB and this would probably fix my challenges for now, but will break the scalability option) I also do understand that NIC's could be a bottleneck on the LB side in the huge amount of data which is another headache I need to crack if possible.
>> But for making sure the the Log Servers can keep up and not hit IO Waite state on disk side I'm scaling out.
>>
>> Best regards
>> Jan
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: 22. august 2023 11:32
>> To: JPIM (Jan Primdahl Madsen) via rsyslog <rsyslog@lists.adiscon.com>
>> Cc: JPIM (Jan Primdahl Madsen) <JPIM@aevengroup.com>
>> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
>> messages
>>
>> what are you considering 'a huge amount of syslogs'? Rsyslog is fast enough that you seldom need multiple systems for capacity, just for redundency.
>>
>> David Lang
>>
>> On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>>
>>> Having a syslog cluster with LB in front is for scalability MK.
>>> I'm going to receive a huge amount of syslogs from servers, network equipment as Firewalls connections.
>>> So yes I want to be able to scale enough by LB between multiple log servers.
>>>
>>> But got your point of not able to use header stuff.
>>> I will try to look on some of the point David Lang mention by looking into the use of corosync/pacemaker if this is something that can help solving our challenge.
>>>
>>> Thanks a lot for the answer.
>>> Best regards
>>> Jan
>>>
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Mariusz Kruk via rsyslog
>>> Sent: 22. august 2023 11:09
>>> To: rsyslog@lists.adiscon.com
>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
>>> messages
>>>
>>> Caution: This email originates from outside the company. You should carefully examine the sender and any links or actions before clicking. Use the "Report Phishing" button on the ribbon to have Security analyze the email if in doubt.
>>>
>>>
>>> The main question would be _why_ do you use LB in the first place. The reasonable solution depends on the answer to this question.
>>>
>>> If you had a network-level load-balancer, it would work without any further configuration, as you have a proxy-type load-balancer, it's only natural to have the LB IP as your $fromhost-ip and there is not much you can do about.it.
>>>
>>> You either need to configure your sources to provide individually distinguishable source identifiers in the events (so that if you have multiple logging servers they don't all report hostname of "server1") or think about reorganizing your infrastructure.
>>>
>>> In syslog there is no concept of "headers" so you can't add proxy headers on your nginx. If you want to modify your event on the loadbalancer, you'd need to use a syslog daemon (rsyslog?) for load-balancing. But that seems a bit overengineered since probably just a single rsyslog receiving events would suffice.
>>>
>>> MK
>>>
>>>
>>> On 22.08.2023 10:58, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>>>> Hello All rsyslog users and developers ????
>>>>
>>>> The main problem.
>>>> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP added to the variable $fromhost-ip no matter what we do. This is due to the fact that the LB is doing TCP routing and becomes the sending IP on the network layer. The syslog message does not contain the IP of the sending node.
>>>>
>>>> The Setup:
>>>> The setup contains only 2 servers which have 2 subnets (Frontend and
>>>> a Backend(For TCP messages)) The servers are using the following
>>>> software
>>>>
>>>> * Nginx (For LB)
>>>> * Keepalived (For VIP handling)
>>>> * Rsyslog (yes for syslog messages)
>>>>
>>>>
>>>> Question 1:
>>>> I was thinking about adding some Proxy Headers to the log message on the LB (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is this possible in anyway to have rsyslog to use Proxy Headers like:
>>>>
>>>> * X-Forwarded-For
>>>> * X-Real-IP
>>>>
>>>> If possible howto do it ?
>>>> If not, any other great suggestions to preserve IP address of
>>>> sending source in a LB TCP syslog setup is appreciated
>>>>
>>>> Please be aware, adding more HW to the setup is not an option.
>>>>
>>>> Best regards
>>>> Jan P. Madsen
>>>>
>>>> This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:

> Hey guys.
> So what you are mentioning here is that I could make something like:
>
> Client --> TCP 10517 --> rsyslog(Frontend) --> process data, change message based on Template --> LB Data to other rsyslog servers --> Store Data
>
> What would be used to LB the data. Yet another "frontend" to do the connection relay ?

once you change the logs, you can then use a proxy-type TCP load balancer since
you aren't depending on fromhost-ip, or you could use CLUSTERIP to balance on
the receiving machines

see my presentation/slides on the topic
https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david

David Lang

> Best regards
> Jan
>
> -----Original Message-----
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Mariusz Kruk via rsyslog
> Sent: 22. august 2023 11:44
> To: rsyslog@lists.adiscon.com
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log messages
>
> It's not about the number of nodes :-)
>
> I have multi-layered rsyslog-based solution where the central node processes about 35k EPS and load-balances them over several outgoing RELP connections and it doesn't break a sweat (true, it sends the data out to "workers" for the heavier parts of event processing).
>
> @David Yes, I know you can embed your original message in json. You can do a similar thing by cleverly formatting the forwarder event and appending the original IP to the message in a form that can be "unpacked". But it of course does need a syslog daemon as the load balancer, not a simple connection relay.
>
> MK
>
> On 22.08.2023 11:38, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>> Huge amount.
>> 2000+ servers
>> 2200 network nodes where 200 is firewalls logging GB of date each every day.
>>
>> This is where I wanted to be able to scale out when 1 node is not
>> enough, and 2 nodes is currently added for redundancy. (Yes could run Active/Passive setup sharing a VIP with no LB and this would probably fix my challenges for now, but will break the scalability option) I also do understand that NIC's could be a bottleneck on the LB side in the huge amount of data which is another headache I need to crack if possible.
>> But for making sure the the Log Servers can keep up and not hit IO Waite state on disk side I'm scaling out.
>>
>> Best regards
>> Jan
>>
>> -----Original Message-----
>> From: David Lang <david@lang.hm>
>> Sent: 22. august 2023 11:32
>> To: JPIM (Jan Primdahl Madsen) via rsyslog <rsyslog@lists.adiscon.com>
>> Cc: JPIM (Jan Primdahl Madsen) <JPIM@aevengroup.com>
>> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
>> messages
>>
>> what are you considering 'a huge amount of syslogs'? Rsyslog is fast enough that you seldom need multiple systems for capacity, just for redundency.
>>
>> David Lang
>>
>> On Tue, 22 Aug 2023, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>>
>>> Having a syslog cluster with LB in front is for scalability MK.
>>> I'm going to receive a huge amount of syslogs from servers, network equipment as Firewalls connections.
>>> So yes I want to be able to scale enough by LB between multiple log servers.
>>>
>>> But got your point of not able to use header stuff.
>>> I will try to look on some of the point David Lang mention by looking into the use of corosync/pacemaker if this is something that can help solving our challenge.
>>>
>>> Thanks a lot for the answer.
>>> Best regards
>>> Jan
>>>
>>>
>>> -----Original Message-----
>>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of
>>> Mariusz Kruk via rsyslog
>>> Sent: 22. august 2023 11:09
>>> To: rsyslog@lists.adiscon.com
>>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>>> Subject: Re: [rsyslog] Handling Fromhost-IP on Loadbalanced TCP log
>>> messages
>>>
>>> Caution: This email originates from outside the company. You should carefully examine the sender and any links or actions before clicking. Use the "Report Phishing" button on the ribbon to have Security analyze the email if in doubt.
>>>
>>>
>>> The main question would be _why_ do you use LB in the first place. The reasonable solution depends on the answer to this question.
>>>
>>> If you had a network-level load-balancer, it would work without any further configuration, as you have a proxy-type load-balancer, it's only natural to have the LB IP as your $fromhost-ip and there is not much you can do about.it.
>>>
>>> You either need to configure your sources to provide individually distinguishable source identifiers in the events (so that if you have multiple logging servers they don't all report hostname of "server1") or think about reorganizing your infrastructure.
>>>
>>> In syslog there is no concept of "headers" so you can't add proxy headers on your nginx. If you want to modify your event on the loadbalancer, you'd need to use a syslog daemon (rsyslog?) for load-balancing. But that seems a bit overengineered since probably just a single rsyslog receiving events would suffice.
>>>
>>> MK
>>>
>>>
>>> On 22.08.2023 10:58, JPIM (Jan Primdahl Madsen) via rsyslog wrote:
>>>> Hello All rsyslog users and developers ????
>>>>
>>>> The main problem.
>>>> When receiving loadbalanced TCP syslog messages our Loadbalancer IP is the IP added to the variable $fromhost-ip no matter what we do. This is due to the fact that the LB is doing TCP routing and becomes the sending IP on the network layer. The syslog message does not contain the IP of the sending node.
>>>>
>>>> The Setup:
>>>> The setup contains only 2 servers which have 2 subnets (Frontend and
>>>> a Backend(For TCP messages)) The servers are using the following
>>>> software
>>>>
>>>> * Nginx (For LB)
>>>> * Keepalived (For VIP handling)
>>>> * Rsyslog (yes for syslog messages)
>>>>
>>>>
>>>> Question 1:
>>>> I was thinking about adding some Proxy Headers to the log message on the LB (nginx) side and then using that in rsyslog to overwrite the $fronhost-ip. Is this possible in anyway to have rsyslog to use Proxy Headers like:
>>>>
>>>> * X-Forwarded-For
>>>> * X-Real-IP
>>>>
>>>> If possible howto do it ?
>>>> If not, any other great suggestions to preserve IP address of
>>>> sending source in a LB TCP syslog setup is appreciated
>>>>
>>>> Please be aware, adding more HW to the setup is not an option.
>>>>
>>>> Best regards
>>>> Jan P. Madsen
>>>>
>>>> This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.