Mailing List Archive

To be honest,
the main reason Debian chosen rsyslog as primary syslog daemon was
that it does work with "standard syslog" configuration (more
information can be read on https://wiki.debian.org/Rsyslog ).
Nevertheless in newest versions of rsyslog you are always recommended
to move to "rainer-script" configuration.

Latest open-sourced versions of syslog-ng provide the TLS encryption
for message forwarding.

Have a look on comparison of syslog-ng releases to have some quick reference.
https://www.balasys.hu/en/network-security/syslog-ng/opensource-logging-system/features/comparison

Both provide "reliable" syslog forwarding. Rsyslog open-sourced,
syslog-ng closed-sourced within enterprise support.

If your budget is large enough, you can pay for enterprise support.
https://www.rsyslog.com/professional-services/enterprise-support/
https://support.oneidentity.com/syslog-ng-premium-edition

From my feeling on rsyslog it seems that this project has serious
issues with project management. More regression occur last year, even
last stable versions were released with serious bugs. But that's fruit
of our today's "agile development" mania. There are long standing
issues with TLS encryption still waiting for fix. Even when not having
experience with syslog-ng in large environments, it seems to me like
more mature project. Last year the Balabit company (originated in
Hungary), responsible for syslog-ng development, was bought by One
Identity.
https://www.oneidentity.com/balabit-acquisition/

To have a better feeling, you can check the list of issues for both projects
https://github.com/balabit/syslog-ng/issues
https://github.com/rsyslog/rsyslog/issues

After that you might be able to do serious decision.

Peter

On Mon, Feb 4, 2019 at 7:46 AM vishal via rsyslog
<rsyslog@lists.adiscon.com> wrote:
>
> Hi,
> I am evaluating rsyslog and syslogng for our project.
> Though aware of some of the differences and pros and cons, but still
> would like to know the differences which users have faced and evaluated
> in terms of ease of use, robustness, handling huge volumes of logs and
> deployment scenarios (single host to multi host cluster) , and if there
> are any other important areas to be considered.
>
> The general deployment would be,
>
> Log sources -> rsyslog/syslogng -> elasticsearch
>
>
> Thanks.
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On 2/5/19 2:40 AM, Peter Viskup via rsyslog wrote:
> To be honest,
> the main reason Debian chosen rsyslog as primary syslog daemon was
> that it does work with "standard syslog" configuration (more
> information can be read on https://wiki.debian.org/Rsyslog ).
> Nevertheless in newest versions of rsyslog you are always recommended
> to move to "rainer-script" configuration.
>
> Latest open-sourced versions of syslog-ng provide the TLS encryption
> for message forwarding.
>
> Have a look on comparison of syslog-ng releases to have some quick reference.
> https://www.balasys.hu/en/network-security/syslog-ng/opensource-logging-system/features/comparison
>
> Both provide "reliable" syslog forwarding. Rsyslog open-sourced,
> syslog-ng closed-sourced within enterprise support.
>
> If your budget is large enough, you can pay for enterprise support.
> https://www.rsyslog.com/professional-services/enterprise-support/
> https://support.oneidentity.com/syslog-ng-premium-edition
>
> From my feeling on rsyslog it seems that this project has serious
> issues with project management. More regression occur last year, even
> last stable versions were released with serious bugs. But that's fruit
> of our today's "agile development" mania.


I can appreciate from your statements that you appear to be frustrated with the direction that rsyslog has taken recently, but these are pretty serious accusations to level against the rsyslog
team based on pure conjecture or "feelings", as well as blatant advertising for a competitive commercial product, and not even giving Adsicon an equal opportunity for pointing out that they
also provide commercial support for rsyslog.


> There are long standing
> issues with TLS encryption still waiting for fix. Even when not having
> experience with syslog-ng in large environments, it seems to me like
> more mature project. Last year the Balabit company (originated in
> Hungary), responsible for syslog-ng development, was bought by One
> Identity.
> https://www.oneidentity.com/balabit-acquisition/
>
> To have a better feeling, you can check the list of issues for both projects
> https://github.com/balabit/syslog-ng/issues
> https://github.com/rsyslog/rsyslog/issues


Also keep in mind that this is not necessarily an "apples to apples" comparison - it is but one data point among many, many others to use to compare the products.  And even this one source of
data must be carefully filtered in terms of user base, usage, feature set, etc.  For example, rsyslog is used by _every single_ RHEL7/CentOS7 system _by default_, out of the box, and has been
part of RHEL/CentOS/Fedora for many years, so it has the full support of Red Hat and a dedicated team of rsyslog maintainers who file and fix bugs upstream.  Another way of thinking about this
is that if you are a RHEL user, you are already paying for rsyslog with your subscription.


>
> After that you might be able to do serious decision.
>
> Peter
>
> On Mon, Feb 4, 2019 at 7:46 AM vishal via rsyslog
> <rsyslog@lists.adiscon.com> wrote:
>> Hi,
>> I am evaluating rsyslog and syslogng for our project.
>> Though aware of some of the differences and pros and cons, but still
>> would like to know the differences which users have faced and evaluated
>> in terms of ease of use, robustness, handling huge volumes of logs and
>> deployment scenarios (single host to multi host cluster) , and if there
>> are any other important areas to be considered.
>>
>> The general deployment would be,
>>
>> Log sources -> rsyslog/syslogng -> elasticsearch
>>
>>
>> Thanks.
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.


_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi Vishal,

It's been many years since we switched from (open source) syslog-ng to rsyslog. We did it because we were struggling with configuration complexity and performance issues, and also because Balabit supported certain features (such as local disk buffering) only in the commercial-only version, but their pricing model was not justifiable for us at the time. I'm sure that syslog-ng has changed quite a bit in that time so a comparison today may not turn out the same way, but we've been very happy with rsyslog. If syslog collection is important to you or your organization, my opinion is that rsyslog is the best choice. The prompt adoption of new technologies over the years such as json, liblognorm, stream-compression, elasticsearch, kafka, and many others demonstrate that this project continues to have an active user and development community.

As for performance: we are certainly not the largest-volume users of rsyslog, but as one anecdotal example I happen to have handy, we've used rsyslog to collect over 24 million logs (around 12.5 TB) per day. You will need to learn about action queues, buffer sizing, pstats, and tuning, but rsyslog can handle it.

Additionally, we've been happy to help financially support the rsyslog project by maintaining an Adiscon support contract, as several others on this mailing-list do as well (just to clarify that unwillingness to pay was not the reason we decided not to go with syslog-ng pro).

Best regards,

--
Dave Caplinger | Chief Architect, Global Platform Engineering | NTT Security Corporation

> On Feb 4, 2019, at 12:45 AM, vishal via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> Hi,
> I am evaluating rsyslog and syslogng for our project.
> Though aware of some of the differences and pros and cons, but still would like to know the differences which users have faced and evaluated in terms of ease of use, robustness, handling huge volumes of logs and deployment scenarios (single host to multi host cluster) , and if there are any other important areas to be considered.
>
> The general deployment would be,
>
> Log sources -> rsyslog/syslogng -> elasticsearch
>
>
> Thanks.
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.


Confidentiality Notice: The content of this communication, along with any attachments, is covered by federal and state law governing electronic communications and may contain confidential and legally privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, use or copying of the information contained herein is strictly prohibited. If you have received this communication in error, please immediately contact us by telephone at 402.361.3000 or e-mail security-americas@nttsecurity.com.

Copyright 2000-2018 NTT Security (US) Inc., a wholly-owned subsidiary of NTT Group. All rights reserved. NTT Security is a trademark of NTT Security GMBH.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

I have used Rsyslog in the past to run a 66 node array (6 core VMs) to
process 6M log-events / second (avg size 1.5 KB), which boils down to
69 Gbps of log traffic. This was being received over ptcp and
delivered to the backend over omkafka.

Rsyslog and HDFS (of Hadoop) were by-far the most trouble-free
components in this architecture. In addition to data-plane, Rsyslog
did accounting based on custom fields (such as cluster-id or
application-id), reported those stats, performed L7 service defence
(blocking abuse) and helped get data out for escallation to L3 defense
and much more.

We saw everything break at this scale, including Rsyslog, but had to
make very few and relatively tiny changes to Rsyslog (all merged
upstream, btw) to get things going whereas most other components had
to be replaced or large parts of them had to be re-written.

So when it comes to stability and performance, I totally swear by Rsyslog.

When building such large systems one doesn't want to take over
maintainership of components used (which is sometimes the sorry state
things get into when upstream doesn't acknowledge the problems or
doesn't accept fixes for it). We were at 0 locally patched changes on
Rsyslog build, thanks to Rsyslog community which is super responsive
and helpful.

If I ever build such a system again, I would choose Rsyslog for
log-mux with my eyes closed.

On Wed, Feb 6, 2019 at 5:30 AM Dave Caplinger via rsyslog
<rsyslog@lists.adiscon.com> wrote:
>
> Hi Vishal,
>
> It's been many years since we switched from (open source) syslog-ng to rsyslog. We did it because we were struggling with configuration complexity and performance issues, and also because Balabit supported certain features (such as local disk buffering) only in the commercial-only version, but their pricing model was not justifiable for us at the time. I'm sure that syslog-ng has changed quite a bit in that time so a comparison today may not turn out the same way, but we've been very happy with rsyslog. If syslog collection is important to you or your organization, my opinion is that rsyslog is the best choice. The prompt adoption of new technologies over the years such as json, liblognorm, stream-compression, elasticsearch, kafka, and many others demonstrate that this project continues to have an active user and development community.
>
> As for performance: we are certainly not the largest-volume users of rsyslog, but as one anecdotal example I happen to have handy, we've used rsyslog to collect over 24 million logs (around 12.5 TB) per day. You will need to learn about action queues, buffer sizing, pstats, and tuning, but rsyslog can handle it.
>
> Additionally, we've been happy to help financially support the rsyslog project by maintaining an Adiscon support contract, as several others on this mailing-list do as well (just to clarify that unwillingness to pay was not the reason we decided not to go with syslog-ng pro).
>
> Best regards,
>
> --
> Dave Caplinger | Chief Architect, Global Platform Engineering | NTT Security Corporation
>
> > On Feb 4, 2019, at 12:45 AM, vishal via rsyslog <rsyslog@lists.adiscon.com> wrote:
> >
> > Hi,
> > I am evaluating rsyslog and syslogng for our project.
> > Though aware of some of the differences and pros and cons, but still would like to know the differences which users have faced and evaluated in terms of ease of use, robustness, handling huge volumes of logs and deployment scenarios (single host to multi host cluster) , and if there are any other important areas to be considered.
> >
> > The general deployment would be,
> >
> > Log sources -> rsyslog/syslogng -> elasticsearch
> >
> >
> > Thanks.
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
>
> Confidentiality Notice: The content of this communication, along with any attachments, is covered by federal and state law governing electronic communications and may contain confidential and legally privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, use or copying of the information contained herein is strictly prohibited. If you have received this communication in error, please immediately contact us by telephone at 402.361.3000 or e-mail security-americas@nttsecurity.com.
>
> Copyright 2000-2018 NTT Security (US) Inc., a wholly-owned subsidiary of NTT Group. All rights reserved. NTT Security is a trademark of NTT Security GMBH.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.



--
Regards,
Janmejay
http://codehunk.wordpress.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

El mar., 5 feb. 2019 a las 10:40, Peter Viskup via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> To be honest,
> the main reason Debian chosen rsyslog as primary syslog daemon was
> that it does work with "standard syslog" configuration (more
> information can be read on https://wiki.debian.org/Rsyslog ).
> Nevertheless in newest versions of rsyslog you are always recommended
> to move to "rainer-script" configuration.

Just to get the facts straight: we recommend this for more complex
things, only. Simple things are still very well done in sysklogd
format. See

https://www.rsyslog.com/doc/v8-stable/configuration/conf_formats.html

> Latest open-sourced versions of syslog-ng provide the TLS encryption
> for message forwarding.
>
> Have a look on comparison of syslog-ng releases to have some quick reference.
> https://www.balasys.hu/en/network-security/syslog-ng/opensource-logging-system/features/comparison
>
> Both provide "reliable" syslog forwarding. Rsyslog open-sourced,
> syslog-ng closed-sourced within enterprise support.
>
> If your budget is large enough, you can pay for enterprise support.
> https://www.rsyslog.com/professional-services/enterprise-support/
> https://support.oneidentity.com/syslog-ng-premium-edition

Yes, but you overlook that for syslog-ng you actually NEED to purchase
the "perimium edition" while for rsyslog you get all the software, all
features as part of the open source offering. The support package is
purely optional. For enterprise environments it usually pays quickly,
especially as it is so little money for a real enterprise (syslog-ng
is quite different according to the number I have).

BUT: I think a main difference is that rsyslog still is, and always
has been, a pure, traditional open source project. We depend on the
community, we give the software for free, the community even provides
great help for free. We do not request contributor license agreements
or anything else that comes with the now ever-popular freemium/open
core models that seem to infiltrate the open source movement. I am not
bashing the syslog-ng guys here. They are cool and they do real good
things. It's more a general political thingy: do we really depend
solely on software that is controlled by very big multi-nationals
(syslog-ng nowadays belongs to Quest Software) or do we want to use
community projects? If you are solely for "free beer", that may
probably not matter. If you still believe in "free speech", than you
will go along with some rough edges on community projects - and
hopefully contribute to iron them out. If you don't know about what I
am talking ... oh man, so far we have already come ;-)
>
> From my feeling on rsyslog it seems that this project has serious
> issues with project management. More regression occur last year, even
> last stable versions were released with serious bugs.

That really makes me disappointed. We have invested a tremendous
amount of time within the last 18..24 month to improve CI, testing and
all that. I was initially rather skeptic that this effort is actually
useful, but really have been convinced it is. Is it just so that you
notice some bad things the past year? Bad things that did not happen
before? I would really be interested in a more precise view.

I would also be really interested in which bugs you mean. I can
envision one or two but this sounds like a lot more.

As a side-note, with the decline of real open source software I also
see a decline in community participation. Around 5 to 7 years ago,
folks actually helped testing and trying out the software before we
crafted a final release. That was very useful. This has severely
changed since then. Nobody uses the devel versions, which was the main
reason why we gave up on them.

> But that's fruit
> of our today's "agile development" mania. There are long standing
> issues with TLS encryption still waiting for fix.

Which ones? I would be very interested to know, because we just wrote
openssl drivers to get around the restrictions, especially in error
reporting, that GnuTLS imposed on us. I am not aware of anything
serious right now (overlooked?). A lot of issue have been opened for
things that are actually done like specified in the relevant standards
(RFC5425), like certless authentication. We love standards. But we
finally gave up on this one and have a a PR active and almost ready
for merge that enables this useless and highly insecure mode.

Rainer

> Even when not having
> experience with syslog-ng in large environments, it seems to me like
> more mature project. Last year the Balabit company (originated in
> Hungary), responsible for syslog-ng development, was bought by One
> Identity.
> https://www.oneidentity.com/balabit-acquisition/
>
> To have a better feeling, you can check the list of issues for both projects
> https://github.com/balabit/syslog-ng/issues
> https://github.com/rsyslog/rsyslog/issues
>
> After that you might be able to do serious decision.
>
> Peter
>
> On Mon, Feb 4, 2019 at 7:46 AM vishal via rsyslog
> <rsyslog@lists.adiscon.com> wrote:
> >
> > Hi,
> > I am evaluating rsyslog and syslogng for our project.
> > Though aware of some of the differences and pros and cons, but still
> > would like to know the differences which users have faced and evaluated
> > in terms of ease of use, robustness, handling huge volumes of logs and
> > deployment scenarios (single host to multi host cluster) , and if there
> > are any other important areas to be considered.
> >
> > The general deployment would be,
> >
> > Log sources -> rsyslog/syslogng -> elasticsearch
> >
> >
> > Thanks.
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi Janmejay,

Thanks for sharing your scenario as well. Based on your log size data I realized that I meant 24B logs/day rather than 24M!

Here are some more details in case anyone else here is interested. The data I referred to was for a single day a few years ago, and the specifics based on rsyslog-pstats data were:

| 24,488,120,824 log lines
| 12,649,411,012,274 bytes total
| 15.1:1 compression ratio (6.6% of original)
| 283,427 logs/sec avg
| 516 bytes/log avg

This data was received via ptcp by a pool of 12 central aggregators and came from about 1000 remote collectors. Since we control both the collectors and the aggregators and they both run rsyslog, we could take advantage of the stream compression support (which resulted in a significant reduction in network bandwidth).

There is a definite "long tail" to our log size distribution. I don't have the exact min/max numbers available any more for that day, but they can range roughly between 50 and 8K bytes. A size distribution similar to this sample is pretty typical:

| # NumSamples = 53000; Min = 92.00; Max = 8206.00
| # Mean = 653.060623; Variance = 112272.110834; SD = 335.070307; Median 521.500000
| # each ? represents a count of 485
| 92.0000 - 903.4000 [ 36375]: ???????????????????????????????????????????????????????????????????????????
| 903.4000 - 1714.8000 [ 16362]: ?????????????????????????????????
| 1714.8000 - 2526.2000 [ 219]:
| 2526.2000 - 3337.6000 [ 19]:
| 3337.6000 - 4149.0000 [ 14]:
| 4149.0000 - 4960.4000 [ 4]:
| 4960.4000 - 5771.8000 [ 1]:
| 5771.8000 - 6583.2000 [ 0]:
| 6583.2000 - 7394.6000 [ 2]:
| 7394.6000 - 8206.0000 [ 4]:

I hope this discussion prompts some of the other high-volume rsyslog users on this list to share some details also...

Best regards,

--
Dave Caplinger | Chief Architect, Global Platform Engineering | NTT Security Corporation


> On Feb 6, 2019, at 3:03 AM, singh.janmejay <singh.janmejay@gmail.com> wrote:
>
> I have used Rsyslog in the past to run a 66 node array (6 core VMs) to
> process 6M log-events / second (avg size 1.5 KB), which boils down to
> 69 Gbps of log traffic. This was being received over ptcp and
> delivered to the backend over omkafka.
>
> Rsyslog and HDFS (of Hadoop) were by-far the most trouble-free
> components in this architecture. In addition to data-plane, Rsyslog
> did accounting based on custom fields (such as cluster-id or
> application-id), reported those stats, performed L7 service defence
> (blocking abuse) and helped get data out for escallation to L3 defense
> and much more.
>
> We saw everything break at this scale, including Rsyslog, but had to
> make very few and relatively tiny changes to Rsyslog (all merged
> upstream, btw) to get things going whereas most other components had
> to be replaced or large parts of them had to be re-written.
>
> So when it comes to stability and performance, I totally swear by Rsyslog.
>
> When building such large systems one doesn't want to take over
> maintainership of components used (which is sometimes the sorry state
> things get into when upstream doesn't acknowledge the problems or
> doesn't accept fixes for it). We were at 0 locally patched changes on
> Rsyslog build, thanks to Rsyslog community which is super responsive
> and helpful.
>
> If I ever build such a system again, I would choose Rsyslog for
> log-mux with my eyes closed.
>
> On Wed, Feb 6, 2019 at 5:30 AM Dave Caplinger via rsyslog
> <rsyslog@lists.adiscon.com> wrote:
>>
>> Hi Vishal,
>>
>> It's been many years since we switched from (open source) syslog-ng to rsyslog. We did it because we were struggling with configuration complexity and performance issues, and also because Balabit supported certain features (such as local disk buffering) only in the commercial-only version, but their pricing model was not justifiable for us at the time. I'm sure that syslog-ng has changed quite a bit in that time so a comparison today may not turn out the same way, but we've been very happy with rsyslog. If syslog collection is important to you or your organization, my opinion is that rsyslog is the best choice. The prompt adoption of new technologies over the years such as json, liblognorm, stream-compression, elasticsearch, kafka, and many others demonstrate that this project continues to have an active user and development community.
>>
>> As for performance: we are certainly not the largest-volume users of rsyslog, but as one anecdotal example I happen to have handy, we've used rsyslog to collect over 24 million logs (around 12.5 TB) per day. You will need to learn about action queues, buffer sizing, pstats, and tuning, but rsyslog can handle it.
>>
>> Additionally, we've been happy to help financially support the rsyslog project by maintaining an Adiscon support contract, as several others on this mailing-list do as well (just to clarify that unwillingness to pay was not the reason we decided not to go with syslog-ng pro).
>>
>> Best regards,
>>
>> --
>> Dave Caplinger | Chief Architect, Global Platform Engineering | NTT Security Corporation
>>
>>> On Feb 4, 2019, at 12:45 AM, vishal via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>>
>>> Hi,
>>> I am evaluating rsyslog and syslogng for our project.
>>> Though aware of some of the differences and pros and cons, but still would like to know the differences which users have faced and evaluated in terms of ease of use, robustness, handling huge volumes of logs and deployment scenarios (single host to multi host cluster) , and if there are any other important areas to be considered.
>>>
>>> The general deployment would be,
>>>
>>> Log sources -> rsyslog/syslogng -> elasticsearch
>>>
>>>
>>> Thanks.
>>>
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>>
>> Confidentiality Notice: The content of this communication, along with any attachments, is covered by federal and state law governing electronic communications and may contain confidential and legally privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, use or copying of the information contained herein is strictly prohibited. If you have received this communication in error, please immediately contact us by telephone at 402.361.3000 or e-mail security-americas@nttsecurity.com.
>>
>> Copyright 2000-2018 NTT Security (US) Inc., a wholly-owned subsidiary of NTT Group. All rights reserved. NTT Security is a trademark of NTT Security GMBH.
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com


Confidentiality Notice: The content of this communication, along with any attachments, is covered by federal and state law governing electronic communications and may contain confidential and legally privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, use or copying of the information contained herein is strictly prohibited. If you have received this communication in error, please immediately contact us by telephone at 402.361.3000 or e-mail security-americas@nttsecurity.com.

Copyright 2000-2018 NTT Security (US) Inc., a wholly-owned subsidiary of NTT Group. All rights reserved. NTT Security is a trademark of NTT Security GMBH.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Relays (senders) in my case were Rsyslog too, ptcp stream-compressed
octate-counted pipes in and batched pipes out using omkafka (which was
using librdkafka with https://github.com/edenhill/librdkafka/pull/679
local patch, like I said earlier, upstream isn't always as nice as
here).

~ 600 relays per ingester-node (that is 600 connections) shooting into
each of the 66 nodes. We had alerts setup for rx buildup (on socket on
the receiver array), iirc at 3 MB, which never fired! Assuming equal
core utilization on rx and tx (remember tx was relatively more
expensive), we were doing all of this with 2 - 3 threads on rx side
(of 6 cores).

This is what Rsyslog is capable of!

On Thu, Feb 7, 2019 at 12:52 AM Dave Caplinger
<Dave.Caplinger@nttsecurity.com> wrote:
>
> Hi Janmejay,
>
> Thanks for sharing your scenario as well. Based on your log size data I realized that I meant 24B logs/day rather than 24M!
>
> Here are some more details in case anyone else here is interested. The data I referred to was for a single day a few years ago, and the specifics based on rsyslog-pstats data were:
>
> | 24,488,120,824 log lines
> | 12,649,411,012,274 bytes total
> | 15.1:1 compression ratio (6.6% of original)
> | 283,427 logs/sec avg
> | 516 bytes/log avg
>
> This data was received via ptcp by a pool of 12 central aggregators and came from about 1000 remote collectors. Since we control both the collectors and the aggregators and they both run rsyslog, we could take advantage of the stream compression support (which resulted in a significant reduction in network bandwidth).
>
> There is a definite "long tail" to our log size distribution. I don't have the exact min/max numbers available any more for that day, but they can range roughly between 50 and 8K bytes. A size distribution similar to this sample is pretty typical:
>
> | # NumSamples = 53000; Min = 92.00; Max = 8206.00
> | # Mean = 653.060623; Variance = 112272.110834; SD = 335.070307; Median 521.500000
> | # each ? represents a count of 485
> | 92.0000 - 903.4000 [ 36375]: ???????????????????????????????????????????????????????????????????????????
> | 903.4000 - 1714.8000 [ 16362]: ?????????????????????????????????
> | 1714.8000 - 2526.2000 [ 219]:
> | 2526.2000 - 3337.6000 [ 19]:
> | 3337.6000 - 4149.0000 [ 14]:
> | 4149.0000 - 4960.4000 [ 4]:
> | 4960.4000 - 5771.8000 [ 1]:
> | 5771.8000 - 6583.2000 [ 0]:
> | 6583.2000 - 7394.6000 [ 2]:
> | 7394.6000 - 8206.0000 [ 4]:
>
> I hope this discussion prompts some of the other high-volume rsyslog users on this list to share some details also...
>
> Best regards,
>
> --
> Dave Caplinger | Chief Architect, Global Platform Engineering | NTT Security Corporation
>
>
> > On Feb 6, 2019, at 3:03 AM, singh.janmejay <singh.janmejay@gmail.com> wrote:
> >
> > I have used Rsyslog in the past to run a 66 node array (6 core VMs) to
> > process 6M log-events / second (avg size 1.5 KB), which boils down to
> > 69 Gbps of log traffic. This was being received over ptcp and
> > delivered to the backend over omkafka.
> >
> > Rsyslog and HDFS (of Hadoop) were by-far the most trouble-free
> > components in this architecture. In addition to data-plane, Rsyslog
> > did accounting based on custom fields (such as cluster-id or
> > application-id), reported those stats, performed L7 service defence
> > (blocking abuse) and helped get data out for escallation to L3 defense
> > and much more.
> >
> > We saw everything break at this scale, including Rsyslog, but had to
> > make very few and relatively tiny changes to Rsyslog (all merged
> > upstream, btw) to get things going whereas most other components had
> > to be replaced or large parts of them had to be re-written.
> >
> > So when it comes to stability and performance, I totally swear by Rsyslog.
> >
> > When building such large systems one doesn't want to take over
> > maintainership of components used (which is sometimes the sorry state
> > things get into when upstream doesn't acknowledge the problems or
> > doesn't accept fixes for it). We were at 0 locally patched changes on
> > Rsyslog build, thanks to Rsyslog community which is super responsive
> > and helpful.
> >
> > If I ever build such a system again, I would choose Rsyslog for
> > log-mux with my eyes closed.
> >
> > On Wed, Feb 6, 2019 at 5:30 AM Dave Caplinger via rsyslog
> > <rsyslog@lists.adiscon.com> wrote:
> >>
> >> Hi Vishal,
> >>
> >> It's been many years since we switched from (open source) syslog-ng to rsyslog. We did it because we were struggling with configuration complexity and performance issues, and also because Balabit supported certain features (such as local disk buffering) only in the commercial-only version, but their pricing model was not justifiable for us at the time. I'm sure that syslog-ng has changed quite a bit in that time so a comparison today may not turn out the same way, but we've been very happy with rsyslog. If syslog collection is important to you or your organization, my opinion is that rsyslog is the best choice. The prompt adoption of new technologies over the years such as json, liblognorm, stream-compression, elasticsearch, kafka, and many others demonstrate that this project continues to have an active user and development community.
> >>
> >> As for performance: we are certainly not the largest-volume users of rsyslog, but as one anecdotal example I happen to have handy, we've used rsyslog to collect over 24 million logs (around 12.5 TB) per day. You will need to learn about action queues, buffer sizing, pstats, and tuning, but rsyslog can handle it.
> >>
> >> Additionally, we've been happy to help financially support the rsyslog project by maintaining an Adiscon support contract, as several others on this mailing-list do as well (just to clarify that unwillingness to pay was not the reason we decided not to go with syslog-ng pro).
> >>
> >> Best regards,
> >>
> >> --
> >> Dave Caplinger | Chief Architect, Global Platform Engineering | NTT Security Corporation
> >>
> >>> On Feb 4, 2019, at 12:45 AM, vishal via rsyslog <rsyslog@lists.adiscon.com> wrote:
> >>>
> >>> Hi,
> >>> I am evaluating rsyslog and syslogng for our project.
> >>> Though aware of some of the differences and pros and cons, but still would like to know the differences which users have faced and evaluated in terms of ease of use, robustness, handling huge volumes of logs and deployment scenarios (single host to multi host cluster) , and if there are any other important areas to be considered.
> >>>
> >>> The general deployment would be,
> >>>
> >>> Log sources -> rsyslog/syslogng -> elasticsearch
> >>>
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >>
> >>
> >> Confidentiality Notice: The content of this communication, along with any attachments, is covered by federal and state law governing electronic communications and may contain confidential and legally privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, use or copying of the information contained herein is strictly prohibited. If you have received this communication in error, please immediately contact us by telephone at 402.361.3000 or e-mail security-americas@nttsecurity.com.
> >>
> >> Copyright 2000-2018 NTT Security (US) Inc., a wholly-owned subsidiary of NTT Group. All rights reserved. NTT Security is a trademark of NTT Security GMBH.
> >>
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >
> >
> >
> > --
> > Regards,
> > Janmejay
> > http://codehunk.wordpress.com
>
>
> Confidentiality Notice: The content of this communication, along with any attachments, is covered by federal and state law governing electronic communications and may contain confidential and legally privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, use or copying of the information contained herein is strictly prohibited. If you have received this communication in error, please immediately contact us by telephone at 402.361.3000 or e-mail security-americas@nttsecurity.com.
>
> Copyright 2000-2018 NTT Security (US) Inc., a wholly-owned subsidiary of NTT Group. All rights reserved. NTT Security is a trademark of NTT Security GMBH.
>


--
Regards,
Janmejay
http://codehunk.wordpress.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.