Hi all,
it is a great pleasure for me to do a fully buzzword-compliant
announcement for rsyslog 3.11.0, which is available immediately. This
release is a very important milestone: it is the first time ever that
the full, completely new queuing engine is in place.
Now rsyslog is the world's first-ever purely open-sourced syslogd
supporting disk based buffering individually for each log destination[*]
(called "action" in rsyslog speak). Besides, each action can now be
executed on its own thread. Together with the multi-threaded main worker
pool, rsyslog is more than ready for tomorrow's massively multi-core
machines. I think another "world's first" is the ability to
automatically discard low priority syslog messages under congestion
conditions.
The new engine provides, among others:
* reliable storage of messages while a destination is down
* persistence of message even when rsyslogd is shut down
* automatic retry of failed actions until they succeed,
of course without any message loss
* unparalleled ability to handle message burst
* fully automatic worker thread pools
* slow actions (e.g. database writes) are de-coupled from
quick local actions (e.g. file writes)
* rate-limited action processing
* flow control slows senders down when queues get full
* ability to discard less important messages in favor
of higher important ones when the queue runs out of
space
All of this while using minimal system resources.
The system can be set up with few configuration commands, but provides
very in-depth commands to manipulate aspects like worker thread startup
and shutdown parameters, queue sizes, disk quotas, watermark algorithm
settings and many more (sometimes exotic) tweaking knobs.
Full details about the new system and its parameters can be found at:
http://www.rsyslog.com/doc-queues.html
An actual use case, doing massive database inserts, is documented here:
http://www.rsyslog.com/doc-rsyslog_high_database_rate.html
Rsyslog 3.11.0 is fresh off the "development press". While it is well
tested in lab, I expect that there are bugs in it, especially given the
magnitude of changes and the massively multithreaded processing flow. As
such, I do not yet advise to use it on important production machines.
I would, however, be very interested in feedback from the field,
including bug reports. My focus in the next weeks will be on making the
new engine rock-solid while at the same time introducing many new
features which are quite easy to build on top of the new engine (like
more advanced rate limiting algorithms, action execution-time windows
and more advanced ways to flow-control inputs). My hope is that within
two month or so we will have a very stable version suitable for
production use in the most demanding environments.
I hope that this announcement will motivate some people to try rsyslog
out and provide feedback to the project. Just be warned that I will not
be at my desk next week: I am taking my now hopefully deserved break
after a long time of hard coding. I thought to delay the announcement,
but have decided it is probably better to get some head start. So please
do not be disappointed if I do not reply immediately on comments posted.
For comments and questions, I recommend using either the mailing list or
the web forum at
http://www.rsyslog.com
I hope that rsyslog will be useful. Any feedback is appreciated.
Best regards,
Rainer Gerhards
Adiscon
[*] I know that syslog-ng does disk buffering, but it does not support
it in the open source edition.
it is a great pleasure for me to do a fully buzzword-compliant
announcement for rsyslog 3.11.0, which is available immediately. This
release is a very important milestone: it is the first time ever that
the full, completely new queuing engine is in place.
Now rsyslog is the world's first-ever purely open-sourced syslogd
supporting disk based buffering individually for each log destination[*]
(called "action" in rsyslog speak). Besides, each action can now be
executed on its own thread. Together with the multi-threaded main worker
pool, rsyslog is more than ready for tomorrow's massively multi-core
machines. I think another "world's first" is the ability to
automatically discard low priority syslog messages under congestion
conditions.
The new engine provides, among others:
* reliable storage of messages while a destination is down
* persistence of message even when rsyslogd is shut down
* automatic retry of failed actions until they succeed,
of course without any message loss
* unparalleled ability to handle message burst
* fully automatic worker thread pools
* slow actions (e.g. database writes) are de-coupled from
quick local actions (e.g. file writes)
* rate-limited action processing
* flow control slows senders down when queues get full
* ability to discard less important messages in favor
of higher important ones when the queue runs out of
space
All of this while using minimal system resources.
The system can be set up with few configuration commands, but provides
very in-depth commands to manipulate aspects like worker thread startup
and shutdown parameters, queue sizes, disk quotas, watermark algorithm
settings and many more (sometimes exotic) tweaking knobs.
Full details about the new system and its parameters can be found at:
http://www.rsyslog.com/doc-queues.html
An actual use case, doing massive database inserts, is documented here:
http://www.rsyslog.com/doc-rsyslog_high_database_rate.html
Rsyslog 3.11.0 is fresh off the "development press". While it is well
tested in lab, I expect that there are bugs in it, especially given the
magnitude of changes and the massively multithreaded processing flow. As
such, I do not yet advise to use it on important production machines.
I would, however, be very interested in feedback from the field,
including bug reports. My focus in the next weeks will be on making the
new engine rock-solid while at the same time introducing many new
features which are quite easy to build on top of the new engine (like
more advanced rate limiting algorithms, action execution-time windows
and more advanced ways to flow-control inputs). My hope is that within
two month or so we will have a very stable version suitable for
production use in the most demanding environments.
I hope that this announcement will motivate some people to try rsyslog
out and provide feedback to the project. Just be warned that I will not
be at my desk next week: I am taking my now hopefully deserved break
after a long time of hard coding. I thought to delay the announcement,
but have decided it is probably better to get some head start. So please
do not be disappointed if I do not reply immediately on comments posted.
For comments and questions, I recommend using either the mailing list or
the web forum at
http://www.rsyslog.com
I hope that rsyslog will be useful. Any feedback is appreciated.
Best regards,
Rainer Gerhards
Adiscon
[*] I know that syslog-ng does disk buffering, but it does not support
it in the open source edition.