Mailing List Archive

rsyslog cluster with NFS backend
Hi.

I have a single rsyslog server (on cloud) receiving logs from a number of systems (also in cloud). I am planning to add a second server and use NFS cloud storage as the backend. This storage will be shared across the two syslog systems.

At the moment, the single rsyslog system receives around 380.000 messages per minute and stores them on local disk.

The reason for opting NFS shared storage is that logs from systems will be on one location instead of two. For example, without shared storage and due to load balancing, Monday logs from serverA can be found in rsyslog1 server and Tuesday logs for same server can be on rsyslog2 server.

However, I am thinking about the performance of NFS and the (negative) impact that it could cause on rsyslog. I am concerned about:


* NFS deadlocks
* behaviour when two rsyslog processes write to same file
* encryption and network delay of NFS backend
* rotation of files


Has anyone implemented such rsyslog setup and found it problematic?

Any thoughts/comments for such setup?

Thanks


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog cluster with NFS backend [ In reply to ]
having two systems writing to the same NFS file is very problematic. When I've
done this sort of thing I've had the two copies of rsyslog write to different
directories.

David Lang

On Thu, 30 Jun 2022, Dimi Onobodies via rsyslog wrote:

> Date: Thu, 30 Jun 2022 10:36:42 +0000
> From: Dimi Onobodies via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Dimi Onobodies <dimi_kdj@hotmail.com>
> Subject: [rsyslog] rsyslog cluster with NFS backend
>
> Hi.
>
> I have a single rsyslog server (on cloud) receiving logs from a number of systems (also in cloud). I am planning to add a second server and use NFS cloud storage as the backend. This storage will be shared across the two syslog systems.
>
> At the moment, the single rsyslog system receives around 380.000 messages per minute and stores them on local disk.
>
> The reason for opting NFS shared storage is that logs from systems will be on one location instead of two. For example, without shared storage and due to load balancing, Monday logs from serverA can be found in rsyslog1 server and Tuesday logs for same server can be on rsyslog2 server.
>
> However, I am thinking about the performance of NFS and the (negative) impact that it could cause on rsyslog. I am concerned about:
>
>
> * NFS deadlocks
> * behaviour when two rsyslog processes write to same file
> * encryption and network delay of NFS backend
> * rotation of files
>
>
> Has anyone implemented such rsyslog setup and found it problematic?
>
> Any thoughts/comments for such setup?
>
> Thanks
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog cluster with NFS backend [ In reply to ]
I'd simply forward from two remote rsyslogs to a central one using RELP
and not bother with NFS and such. The upside is that you can scale this
easily to other non-local installation (it's way easier to add another
RELP source than to export your NFS to some external site).

On 30.06.2022 12:36, Dimi Onobodies via rsyslog wrote:
> Hi.
>
> I have a single rsyslog server (on cloud) receiving logs from a number of systems (also in cloud). I am planning to add a second server and use NFS cloud storage as the backend. This storage will be shared across the two syslog systems.
>
> At the moment, the single rsyslog system receives around 380.000 messages per minute and stores them on local disk.
>
> The reason for opting NFS shared storage is that logs from systems will be on one location instead of two. For example, without shared storage and due to load balancing, Monday logs from serverA can be found in rsyslog1 server and Tuesday logs for same server can be on rsyslog2 server.
>
> However, I am thinking about the performance of NFS and the (negative) impact that it could cause on rsyslog. I am concerned about:
>
>
> * NFS deadlocks
> * behaviour when two rsyslog processes write to same file
> * encryption and network delay of NFS backend
> * rotation of files
>
>
> Has anyone implemented such rsyslog setup and found it problematic?
>
> Any thoughts/comments for such setup?
>
> Thanks
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog cluster with NFS backend [ In reply to ]
That could also be a solution. Iv seen numerous implementations using RELP.

Thanks everyone for sharing your thoughts
________________________________
From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
Sent: Thursday, June 30, 2022 1:42 PM
To: rsyslog@lists.adiscon.com <rsyslog@lists.adiscon.com>
Cc: Mariusz Kruk <kruk@epsilon.eu.org>
Subject: Re: [rsyslog] rsyslog cluster with NFS backend

I'd simply forward from two remote rsyslogs to a central one using RELP
and not bother with NFS and such. The upside is that you can scale this
easily to other non-local installation (it's way easier to add another
RELP source than to export your NFS to some external site).

On 30.06.2022 12:36, Dimi Onobodies via rsyslog wrote:
> Hi.
>
> I have a single rsyslog server (on cloud) receiving logs from a number of systems (also in cloud). I am planning to add a second server and use NFS cloud storage as the backend. This storage will be shared across the two syslog systems.
>
> At the moment, the single rsyslog system receives around 380.000 messages per minute and stores them on local disk.
>
> The reason for opting NFS shared storage is that logs from systems will be on one location instead of two. For example, without shared storage and due to load balancing, Monday logs from serverA can be found in rsyslog1 server and Tuesday logs for same server can be on rsyslog2 server.
>
> However, I am thinking about the performance of NFS and the (negative) impact that it could cause on rsyslog. I am concerned about:
>
>
> * NFS deadlocks
> * behaviour when two rsyslog processes write to same file
> * encryption and network delay of NFS backend
> * rotation of files
>
>
> Has anyone implemented such rsyslog setup and found it problematic?
>
> Any thoughts/comments for such setup?
>
> Thanks
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog cluster with NFS backend [ In reply to ]
I have both RELP (TCP/20514) and traditional syslog (UDP/514) setup on
two hosts.  the RELP instances are load balanced behind a single IP on
HAProxy for high availability and fault tolerance. the syslog instances
are load sharing behind an Anycast IP, so they both listen on the same
IP, but are not in conflict because of the Anycast.  in all cases, the
log receivers insert the records into a multi-master MariaDB database on
3 nodes that is behind HAProxy as well.

i dont see anywhere near 380k messages per minute, but do have about
440k messages per day.  You can probably tune and tweak to support that
message rate with more instances and better gear than i have.  Without a
database backend, you can have all your log in one place, without having
to work about file system locking and contention.

i think i have seen RELP plus TLS, and maybe RELP plus GSSAPI, which may
offer encryption in transit.  you can secure the database access with
TLS, too, and do encyption within the database for encryption at rest. 
I am not sure if that is easier or better than file system encryption
via LUKS, etc.  I dont have any of that going on.

On 7/1/22 6:59 AM, Dimi Onobodies via rsyslog wrote:
> That could also be a solution. Iv seen numerous implementations using RELP.
>
> Thanks everyone for sharing your thoughts
> ________________________________
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
> Sent: Thursday, June 30, 2022 1:42 PM
> To: rsyslog@lists.adiscon.com <rsyslog@lists.adiscon.com>
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] rsyslog cluster with NFS backend
>
> I'd simply forward from two remote rsyslogs to a central one using RELP
> and not bother with NFS and such. The upside is that you can scale this
> easily to other non-local installation (it's way easier to add another
> RELP source than to export your NFS to some external site).
>
> On 30.06.2022 12:36, Dimi Onobodies via rsyslog wrote:
>> Hi.
>>
>> I have a single rsyslog server (on cloud) receiving logs from a number of systems (also in cloud). I am planning to add a second server and use NFS cloud storage as the backend. This storage will be shared across the two syslog systems.
>>
>> At the moment, the single rsyslog system receives around 380.000 messages per minute and stores them on local disk.
>>
>> The reason for opting NFS shared storage is that logs from systems will be on one location instead of two. For example, without shared storage and due to load balancing, Monday logs from serverA can be found in rsyslog1 server and Tuesday logs for same server can be on rsyslog2 server.
>>
>> However, I am thinking about the performance of NFS and the (negative) impact that it could cause on rsyslog. I am concerned about:
>>
>>
>> * NFS deadlocks
>> * behaviour when two rsyslog processes write to same file
>> * encryption and network delay of NFS backend
>> * rotation of files
>>
>>
>> Has anyone implemented such rsyslog setup and found it problematic?
>>
>> Any thoughts/comments for such setup?
>>
>> Thanks
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog cluster with NFS backend [ In reply to ]
I have a multi-layered setup with a central rsyslog-based
"load-balancer". Said load balancer reaches up to some 8-10 kEPS without
breaking a sweat (other components which do some heavier filtering and
http output tend to be significantly more stressed). I user RELP with
TLS (mutual authentication). It has 16CPUs but the load rarely goes
above 5 or so.

On 01.07.2022 13:40, Brendan Kearney via rsyslog wrote:
> I have both RELP (TCP/20514) and traditional syslog (UDP/514) setup on
> two hosts.  the RELP instances are load balanced behind a single IP on
> HAProxy for high availability and fault tolerance. the syslog
> instances are load sharing behind an Anycast IP, so they both listen
> on the same IP, but are not in conflict because of the Anycast.  in
> all cases, the log receivers insert the records into a multi-master
> MariaDB database on 3 nodes that is behind HAProxy as well.
>
> i dont see anywhere near 380k messages per minute, but do have about
> 440k messages per day.  You can probably tune and tweak to support
> that message rate with more instances and better gear than i have. 
> Without a database backend, you can have all your log in one place,
> without having to work about file system locking and contention.
>
> i think i have seen RELP plus TLS, and maybe RELP plus GSSAPI, which
> may offer encryption in transit.  you can secure the database access
> with TLS, too, and do encyption within the database for encryption at
> rest.  I am not sure if that is easier or better than file system
> encryption via LUKS, etc.  I dont have any of that going on.
>
> On 7/1/22 6:59 AM, Dimi Onobodies via rsyslog wrote:
>> That could also be a solution. Iv seen numerous implementations using
>> RELP.
>>
>> Thanks everyone for sharing your thoughts
>> ________________________________
>> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of
>> Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
>> Sent: Thursday, June 30, 2022 1:42 PM
>> To: rsyslog@lists.adiscon.com <rsyslog@lists.adiscon.com>
>> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
>> Subject: Re: [rsyslog] rsyslog cluster with NFS backend
>>
>> I'd simply forward from two remote rsyslogs to a central one using RELP
>> and not bother with NFS and such. The upside is that you can scale this
>> easily to other non-local installation (it's way easier to add another
>> RELP source than to export your NFS to some external site).
>>
>> On 30.06.2022 12:36, Dimi Onobodies via rsyslog wrote:
>>> Hi.
>>>
>>> I have a single rsyslog server (on cloud) receiving logs from a
>>> number of systems (also in cloud). I am planning to add a second
>>> server and use NFS cloud storage as the backend. This storage will
>>> be shared across the two syslog systems.
>>>
>>> At the moment, the single rsyslog system receives around 380.000
>>> messages per minute and stores them on local disk.
>>>
>>> The reason for opting NFS shared storage is that logs from systems
>>> will be on one location instead of two. For example, without shared
>>> storage and due to load balancing, Monday logs from serverA can be
>>> found in rsyslog1 server and Tuesday logs for same server can be on
>>> rsyslog2 server.
>>>
>>> However, I am thinking about the performance of NFS and the
>>> (negative) impact that it could cause on rsyslog. I am concerned about:
>>>
>>>
>>>     *   NFS deadlocks
>>>     *   behaviour when two rsyslog processes write to same file
>>>     *   encryption and network delay of NFS backend
>>>     *   rotation of files
>>>
>>>
>>> Has anyone implemented such rsyslog setup and found it problematic?
>>>
>>> Any thoughts/comments for such setup?
>>>
>>> Thanks
>>>
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.