Mailing List Archive

actionqueue in front of tcp forward
I wanted to setup my rsyslog clients such that when the
rsyslog server is down and then comes back up, the clients
will resume forwarding messages to it.

I'm running into this same bug as another user:
http://bugzilla.adiscon.com/show_bug.cgi?id=45

According to that bug, the fix is in 3.12.1, which is what
I'm using on client and server.


But I still can't seem to get it to work. Prior to doing
this, I simply had this line:

*.* @@127.0.0.1:61514



Am I supposed to replace that line entirely with the
ActionQueue stanza, such as this one (the == is for clarity,
not actually part of my config):

==
$WorkDirectory /tmp/rsyslogwork
$ActionQueueType LinkedList
$ActionQueueFileName dqueue
$ActionQueueSaveOnShutdown on
$ActionQueueDiscardSeverity 7
$ActionResumeInterval 30
$ActionResumeRetryCount -1 # infinite retries on insert failure
& @@127.0.0.1:61514
==

If I do, rsyslog dies immediately on restart. So I changed
it so look like this instead:

==
*.* @@127.0.0.1:61514
$WorkDirectory /tmp/rsyslogwork
$ActionQueueType LinkedList
$ActionQueueFileName dqueue
$ActionQueueSaveOnShutdown on
$ActionQueueDiscardSeverity 7
$ActionResumeInterval 10
$ActionResumeRetryCount -1
& @@127.0.0.1:61514
==

This allows rsyslog to work again, but then it doesn't
actually do any resume. Just as the OP states in the bug, if
the rsyslog server is stopped, the client simply logs
messages locally. When the rsyslog server is started again,
the client once again logs locally as well as to the server.
But all the messages from the client between stop and start
on the server are not forwarded.

I checked action.c and it does have the code-fix listed in
the forum post tied to the bug.

I also understand that one message WILL be lost, which is a
TCP syslog protocol problem. So I sent multiple messages,
none of them came through.

johnn
actionqueue in front of tcp forward [ In reply to ]
I need to check for the restart problem (currently working on something
else). Please also follow this bug:

http://bugzilla.adiscon.com/show_bug.cgi?id=48

This is the "one message lost" problem, and I have created at least some
cure for it today (to be verified). Will check your bug report when I
verify it.

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Johnny Tan
> Sent: Wednesday, March 12, 2008 6:17 PM
> To: rsyslog-users
> Subject: [rsyslog] actionqueue in front of tcp forward
>
> I wanted to setup my rsyslog clients such that when the
> rsyslog server is down and then comes back up, the clients
> will resume forwarding messages to it.
>
> I'm running into this same bug as another user:
> http://bugzilla.adiscon.com/show_bug.cgi?id=45
>
> According to that bug, the fix is in 3.12.1, which is what
> I'm using on client and server.
>
>
> But I still can't seem to get it to work. Prior to doing
> this, I simply had this line:
>
> *.* @@127.0.0.1:61514
>
>
>
> Am I supposed to replace that line entirely with the
> ActionQueue stanza, such as this one (the == is for clarity,
> not actually part of my config):
>
> ==
> $WorkDirectory /tmp/rsyslogwork
> $ActionQueueType LinkedList
> $ActionQueueFileName dqueue
> $ActionQueueSaveOnShutdown on
> $ActionQueueDiscardSeverity 7
> $ActionResumeInterval 30
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> & @@127.0.0.1:61514
> ==
>
> If I do, rsyslog dies immediately on restart. So I changed
> it so look like this instead:
>
> ==
> *.* @@127.0.0.1:61514
> $WorkDirectory /tmp/rsyslogwork
> $ActionQueueType LinkedList
> $ActionQueueFileName dqueue
> $ActionQueueSaveOnShutdown on
> $ActionQueueDiscardSeverity 7
> $ActionResumeInterval 10
> $ActionResumeRetryCount -1
> & @@127.0.0.1:61514
> ==
>
> This allows rsyslog to work again, but then it doesn't
> actually do any resume. Just as the OP states in the bug, if
> the rsyslog server is stopped, the client simply logs
> messages locally. When the rsyslog server is started again,
> the client once again logs locally as well as to the server.
> But all the messages from the client between stop and start
> on the server are not forwarded.
>
> I checked action.c and it does have the code-fix listed in
> the forum post tied to the bug.
>
> I also understand that one message WILL be lost, which is a
> TCP syslog protocol problem. So I sent multiple messages,
> none of them came through.
>
> johnn
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
actionqueue in front of tcp forward [ In reply to ]
Argh, some config parsing seems to be wrong. There is a problem with the
config file. The right config format is this:

==
$WorkDirectory /tmp/rsyslogwork
$ActionQueueType LinkedList
$ActionQueueFileName dqueue
$ActionQueueSaveOnShutdown on
$ActionQueueDiscardSeverity 7
$ActionResumeInterval 10
$ActionResumeRetryCount -1
*.* @@127.0.0.1:61514
==

& in column one is to chain multiple actions together, but it can never
be at the start of the chain. Did you take it from some rsyslog
documentation? If so and you still know which, please let me know, I'd
like to fix that.

If I think correctly, the queued action is now chained to the regular
action, and probably never executed because the regular action fails ...
or something along these lines.

Please try the config above, I think it will work (at least it does in
my lab). And of course I'll look into the config parser and co ;)

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Johnny Tan
> Sent: Wednesday, March 12, 2008 6:17 PM
> To: rsyslog-users
> Subject: [rsyslog] actionqueue in front of tcp forward
>
> I wanted to setup my rsyslog clients such that when the
> rsyslog server is down and then comes back up, the clients
> will resume forwarding messages to it.
>
> I'm running into this same bug as another user:
> http://bugzilla.adiscon.com/show_bug.cgi?id=45
>
> According to that bug, the fix is in 3.12.1, which is what
> I'm using on client and server.
>
>
> But I still can't seem to get it to work. Prior to doing
> this, I simply had this line:
>
> *.* @@127.0.0.1:61514
>
>
>
> Am I supposed to replace that line entirely with the
> ActionQueue stanza, such as this one (the == is for clarity,
> not actually part of my config):
>
> ==
> $WorkDirectory /tmp/rsyslogwork
> $ActionQueueType LinkedList
> $ActionQueueFileName dqueue
> $ActionQueueSaveOnShutdown on
> $ActionQueueDiscardSeverity 7
> $ActionResumeInterval 30
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> & @@127.0.0.1:61514
> ==
>
> If I do, rsyslog dies immediately on restart. So I changed
> it so look like this instead:
>
> ==
> *.* @@127.0.0.1:61514
> $WorkDirectory /tmp/rsyslogwork
> $ActionQueueType LinkedList
> $ActionQueueFileName dqueue
> $ActionQueueSaveOnShutdown on
> $ActionQueueDiscardSeverity 7
> $ActionResumeInterval 10
> $ActionResumeRetryCount -1
> & @@127.0.0.1:61514
> ==
>
> This allows rsyslog to work again, but then it doesn't
> actually do any resume. Just as the OP states in the bug, if
> the rsyslog server is stopped, the client simply logs
> messages locally. When the rsyslog server is started again,
> the client once again logs locally as well as to the server.
> But all the messages from the client between stop and start
> on the server are not forwarded.
>
> I checked action.c and it does have the code-fix listed in
> the forum post tied to the bug.
>
> I also understand that one message WILL be lost, which is a
> TCP syslog protocol problem. So I sent multiple messages,
> none of them came through.
>
> johnn
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
actionqueue in front of tcp forward [ In reply to ]
Rainer Gerhards wrote:
> & in column one is to chain multiple actions together, but it can never
> be at the start of the chain. Did you take it from some rsyslog
> documentation? If so and you still know which, please let me know, I'd
> like to fix that.

No, I copied it from the bug report I listed. The OP must've
had a regular action that wasn't listed.

Anyway, I fixed it, restarted client rsyslog, but the
behavior is the same.

I don't need any special config on the server side, right?


Well, if it helps, here's the top of my client
/etc/rsyslog.conf, followed by the top of my server
/etc/rsyslog.conf:

==
client /etc/rsyslog.conf
==
# Modules

$ModLoad imuxsock.so
$ModLoad imklog.so
$ModLoad imfile.so

$WorkDirectory /tmp/rsyslogwork
$ActionQueueType LinkedList
$ActionQueueFileName dqueue
$ActionQueueSaveOnShutdown on
$ActionQueueDiscardSeverity 7
$ActionResumeInterval 10
$ActionResumeRetryCount -1
*.* @@127.0.0.1:61514

# Standard syslog configuration (for local logging)
[snip]
==



==
server /etc/rsyslog.conf
==
# Modules

$ModLoad imudp.so
$ModLoad imtcp.so
$ModLoad imuxsock.so
$ModLoad imklog.so
$ModLoad imfile.so


# TCP port
$InputTCPServerRun 61514


# Global Directives

$AllowedSender UDP, 127.0.0.1, 172.16.0.0/16, 192.168.32.0/24
$AllowedSender TCP, 127.0.0.1, 172.16.0.0/16, 192.168.32.0/24
$DirOwner root
$DirGroup root
$DirCreateMode 0755


# Templates
[snip]

# Standard syslog configuration
[snip]
==
actionqueue in front of tcp forward [ In reply to ]
Rainer Gerhards wrote:
> $WorkDirectory /tmp/rsyslogwork

Also, I should also point out that I NEVER see anything show
up in my WorkDirectory. That directory above is owned by
root:root and mode 0755. I turned off SELinux.

Nothing EVER shows up there.

johnn
actionqueue in front of tcp forward [ In reply to ]
That's fine, files are only created if the mem queue is filling up. By default thats after 8,000 msgs queued. And, no server needs no special config. More later, i am on a pda and can not really check out anything right now ;)

rainer

----- Urspr?ngliche Nachricht -----
Von: "Johnny Tan" <linuxweb at gmail.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 12.03.08 19:32
Betreff: Re: [rsyslog] actionqueue in front of tcp forward

Rainer Gerhards wrote:
> $WorkDirectory /tmp/rsyslogwork

Also, I should also point out that I NEVER see anything show
up in my WorkDirectory. That directory above is owned by
root:root and mode 0755. I turned off SELinux.

Nothing EVER shows up there.

johnn
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
actionqueue in front of tcp forward [ In reply to ]
Can you send me a ful debug output from the sender when it fails (maybe via mail) - if thats possible...

rainer

----- Urspr?ngliche Nachricht -----
Von: "Johnny Tan" <linuxweb at gmail.com>
An: "rsyslog-users" <rsyslog at lists.adiscon.com>
Gesendet: 12.03.08 19:28
Betreff: Re: [rsyslog] actionqueue in front of tcp forward

Rainer Gerhards wrote:
> & in column one is to chain multiple actions together, but it can never
> be at the start of the chain. Did you take it from some rsyslog
> documentation? If so and you still know which, please let me know, I'd
> like to fix that.

No, I copied it from the bug report I listed. The OP must've
had a regular action that wasn't listed.

Anyway, I fixed it, restarted client rsyslog, but the
behavior is the same.

I don't need any special config on the server side, right?


Well, if it helps, here's the top of my client
/etc/rsyslog.conf, followed by the top of my server
/etc/rsyslog.conf:

==
client /etc/rsyslog.conf
==
# Modules

$ModLoad imuxsock.so
$ModLoad imklog.so
$ModLoad imfile.so

$WorkDirectory /tmp/rsyslogwork
$ActionQueueType LinkedList
$ActionQueueFileName dqueue
$ActionQueueSaveOnShutdown on
$ActionQueueDiscardSeverity 7
$ActionResumeInterval 10
$ActionResumeRetryCount -1
*.* @@127.0.0.1:61514

# Standard syslog configuration (for local logging)
[snip]
==



==
server /etc/rsyslog.conf
==
# Modules

$ModLoad imudp.so
$ModLoad imtcp.so
$ModLoad imuxsock.so
$ModLoad imklog.so
$ModLoad imfile.so


# TCP port
$InputTCPServerRun 61514


# Global Directives

$AllowedSender UDP, 127.0.0.1, 172.16.0.0/16, 192.168.32.0/24
$AllowedSender TCP, 127.0.0.1, 172.16.0.0/16, 192.168.32.0/24
$DirOwner root
$DirGroup root
$DirCreateMode 0755


# Templates
[snip]

# Standard syslog configuration
[snip]
==
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
actionqueue in front of tcp forward [ In reply to ]
Rainer Gerhards wrote:
> Can you send me a ful debug output from the sender when
> it fails (maybe via mail) - if thats possible...

I sent it via YouSendIt.
Thanks for looking into this!

johnn
actionqueue in front of tcp forward [ In reply to ]
I have received the file and had a chance to look at it. It's quite
confusing. The send fails, but each retry succeeds... It looks like you
use stunnel. I currently doubt that stunnel accepts each send and only
reports the error after it can not connect to the remote side. So in
fact we continously run into the situation that exactly *that* one
message is lost. But I am still puzzled. Could you, for a test, run the
same without stunnel and tell me if the problem persists or goes away?

Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Johnny Tan
> Sent: Wednesday, March 12, 2008 8:08 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] actionqueue in front of tcp forward
>
> Rainer Gerhards wrote:
> > Can you send me a ful debug output from the sender when
> > it fails (maybe via mail) - if thats possible...
>
> I sent it via YouSendIt.
> Thanks for looking into this!
>
> johnn
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>
actionqueue in front of tcp forward [ In reply to ]
Rainer Gerhards wrote:
> I have received the file and had a chance to look at it. It's quite
> confusing. The send fails, but each retry succeeds... It looks like you
> use stunnel. I currently doubt that stunnel accepts each send and only
> reports the error after it can not connect to the remote side. So in
> fact we continously run into the situation that exactly *that* one
> message is lost. But I am still puzzled. Could you, for a test, run the
> same without stunnel and tell me if the problem persists or goes away?

Yes, I had the same thought about stunnel being the problem
and was running some tests.

It works! I now only lose the one message (known problem).
But if I go over stunnel, then I lose all messages during
the rsyslog server downtime.

I can do without the stunnel for now.

Thanks once again!
johnn
actionqueue in front of tcp forward [ In reply to ]
> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com
> [mailto:rsyslog-bounces at lists.adiscon.com] On Behalf Of Johnny Tan
> Sent: Wednesday, March 12, 2008 10:06 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] actionqueue in front of tcp forward
>
> Rainer Gerhards wrote:
> > I have received the file and had a chance to look at it. It's quite
> > confusing. The send fails, but each retry succeeds... It
> looks like you
> > use stunnel. I currently doubt that stunnel accepts each
> send and only
> > reports the error after it can not connect to the remote side. So in
> > fact we continously run into the situation that exactly *that* one
> > message is lost. But I am still puzzled. Could you, for a
> test, run the
> > same without stunnel and tell me if the problem persists or
> goes away?
>
> Yes, I had the same thought about stunnel being the problem
> and was running some tests.
>
> It works! I now only lose the one message (known problem).
> But if I go over stunnel, then I lose all messages during
> the rsyslog server downtime.

I have the strong feeling that it is time to do something against this
plain old ack-less syslog tcp protocol... Maybe I add a half-duplex mode
for starters. That's low, but quick to implement and ultra-reliable.
I'll also see that I get more serious with RFC 3195 re-enabling. I've
already done some basic thinking in regard to 3195 and the new syslog
engine and doing it ultra-reliable will require a little bit of work. So
there won't be an immediate cure - but defenitely the right route to
take.

How about half-duplex mode? Would that work for you? It means that each
message must be acked before the next one is sent, so tcp's streaming
features will almost be disabled. I'd expect a drop to at most 50% (more
probable 40%) of the performance compared to what we currently run
(half-duplex would obviously need to be an option...). So it would be a
large performance hit.

> I can do without the stunnel for now.

For encryption, you could also look into the GSSAPI modules. It's
contributed code, and I currently unfortunately have limited insight
into it. But varmojfekoj, the contributor, has done a great job.

> Thanks once again!

I am very intersted in real-life experience. After all, the engine is
still quite new. So I need feedback from the field to make it
ultra-solid - a lab is a lab, is a lab ;) So keep the thoughts flowing.

A side-note: we are rewriting phpLogCon, the web interface to syslog
data. Any chance you happen to have some interest in that? ;)

Rainer
> johnn
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>
actionqueue in front of tcp forward [ In reply to ]
Rainer Gerhards wrote:
> I have the strong feeling that it is time to do something against this
> plain old ack-less syslog tcp protocol... Maybe I add a half-duplex mode
> for starters. That's low, but quick to implement and ultra-reliable.
> I'll also see that I get more serious with RFC 3195 re-enabling. I've
> already done some basic thinking in regard to 3195 and the new syslog
> engine and doing it ultra-reliable will require a little bit of work. So
> there won't be an immediate cure - but defenitely the right route to
> take.
>
> How about half-duplex mode? Would that work for you? It means that each
> message must be acked before the next one is sent, so tcp's streaming
> features will almost be disabled. I'd expect a drop to at most 50% (more
> probable 40%) of the performance compared to what we currently run
> (half-duplex would obviously need to be an option...). So it would be a
> large performance hit.

Just curious: do either of the above solutions (3195 and
half-duplex) resolve the issue of losing all messages when
the server goes down and you are going over stunnel?

It sounds like the half-duplex does, I don't know enough
about the 3195 to know.


In answer to your question, I probably don't have as heavy a
load as some folks out there, so it's easy for me to say,
yes, half-duplex works for me, at least for the moment.


>> I can do without the stunnel for now.
>
> For encryption, you could also look into the GSSAPI modules. It's
> contributed code, and I currently unfortunately have limited insight
> into it. But varmojfekoj, the contributor, has done a great job.

Thanks! I'll have to look more closely at this in a staging
environment down the road, as I can't afford potentially
running into another stumbling block at the moment.


> A side-note: we are rewriting phpLogCon, the web interface to syslog
> data. Any chance you happen to have some interest in that? ;)

It'll be a couple weeks before I really start looking into
this, but yes, I was in fact planning to setup phpLogCon, too.

johnn
actionqueue in front of tcp forward [ In reply to ]
> Rainer Gerhards wrote:
> > I have the strong feeling that it is time to do something against
> this
> > plain old ack-less syslog tcp protocol... Maybe I add a half-duplex
> mode
> > for starters. That's low, but quick to implement and ultra-reliable.
> > I'll also see that I get more serious with RFC 3195 re-enabling.
I've
> > already done some basic thinking in regard to 3195 and the new
syslog
> > engine and doing it ultra-reliable will require a little bit of
work.
> So
> > there won't be an immediate cure - but defenitely the right route to
> > take.
> >
> > How about half-duplex mode? Would that work for you? It means that
> each
> > message must be acked before the next one is sent, so tcp's
streaming
> > features will almost be disabled. I'd expect a drop to at most 50%
> (more
> > probable 40%) of the performance compared to what we currently run
> > (half-duplex would obviously need to be an option...). So it would
be
> a
> > large performance hit.
>
> Just curious: do either of the above solutions (3195 and
> half-duplex) resolve the issue of losing all messages when
> the server goes down and you are going over stunnel?
>
> It sounds like the half-duplex does, I don't know enough
> about the 3195 to know.
>
>
> In answer to your question, I probably don't have as heavy a
> load as some folks out there, so it's easy for me to say,
> yes, half-duplex works for me, at least for the moment.

I had a somewhat sleepless night, which is good news because it usually
results in good ideas ;) This problem really bugged me, but I finally
made up my mind. Instead of wasting time on fixing broken plain tcp
syslog transport mode (e.g. by implementing half-duplex, which isn't
standard anyhow), I'll do "the right thing". I thought rfc 3195 is the
right thing. But it carries a lot of overhead that I really don't need.
And, most importantly, any standard additions takes ages to go through
the IETF (I know what I am talking about, have finally succeeded to get
a better syslog rfc though it in "just" 4 (or 5?) years -- and it is
still waiting to be published...).

So instead of adding on any of these existing protocols, I'll do a new,
lightweight but capable protocol specifically designed to solve the
shortcomings we currently experience. Please welcome RELP, the "reliable
event logging protocol" (name based on the quite successful selp [simple
event logging protocol] effort:
http://www.monitorware.com/en/workinprogress/selp.txt).

Relp will evolve over time. I hope to do something useful relatively
soon, and it will be extended as the project progresses. The ultimate
goal is to have a good, very reliable, protocol for rsyslog-to-rsyslog
communications. I'll don't care about the outside world, so I can do the
best of breed solution. For the rest of the world, rsyslog will of
course continue to support plain tcp syslog and will get better support
for rfc 3195. But if you wanna go hardcore on high-reliability,
high-performance event logging, relp will be your choice.

Technically, I'll split this off into rsyslog relp input and output
plugins AND an independent librelp, which provides core protocol
functionality (just in case somebody else wants to support it in the
long term, so this will not be tied to rsyslog).

>
> >> I can do without the stunnel for now.
> >
> > For encryption, you could also look into the GSSAPI modules. It's
> > contributed code, and I currently unfortunately have limited insight
> > into it. But varmojfekoj, the contributor, has done a great job.
>
> Thanks! I'll have to look more closely at this in a staging
> environment down the road, as I can't afford potentially
> running into another stumbling block at the moment.
>
>
> > A side-note: we are rewriting phpLogCon, the web interface to syslog
> > data. Any chance you happen to have some interest in that? ;)
>
> It'll be a couple weeks before I really start looking into
> this, but yes, I was in fact planning to setup phpLogCon, too.

Excellent. Please keep an eye on its beta announcements - various stages
of v2 will probably be available in a few weeks and I promise it will be
much better than what is currently there. In fact, it tries to become
the best-ever log web viewer. If there are some things you'd like to see
in such a viewer, it would be great to hear your opinion.

Rainer
actionqueue in front of tcp forward [ In reply to ]
>
> I had a somewhat sleepless night, which is good news because it
usually
> results in good ideas ;) This problem really bugged me, but I finally
> made up my mind. Instead of wasting time on fixing broken plain tcp
> syslog transport mode (e.g. by implementing half-duplex, which isn't
> standard anyhow), I'll do "the right thing". I thought rfc 3195 is the
> right thing. But it carries a lot of overhead that I really don't
need.
> And, most importantly, any standard additions takes ages to go through
> the IETF (I know what I am talking about, have finally succeeded to
get
> a better syslog rfc though it in "just" 4 (or 5?) years -- and it is
> still waiting to be published...).
>
> So instead of adding on any of these existing protocols, I'll do a
new,
> lightweight but capable protocol specifically designed to solve the
> shortcomings we currently experience. Please welcome RELP, the
"reliable
> event logging protocol" (name based on the quite successful selp
[simple
> event logging protocol] effort:
> http://www.monitorware.com/en/workinprogress/selp.txt).
>
> Relp will evolve over time. I hope to do something useful relatively
> soon, and it will be extended as the project progresses. The ultimate
> goal is to have a good, very reliable, protocol for rsyslog-to-rsyslog
> communications. I'll don't care about the outside world, so I can do
the
> best of breed solution. For the rest of the world, rsyslog will of
> course continue to support plain tcp syslog and will get better
support
> for rfc 3195. But if you wanna go hardcore on high-reliability,
> high-performance event logging, relp will be your choice.
>

Maybe in another 5 years RELP will replace rfc 3195 especially. Now that
rsyslog is part of official redhat release it should gain a lot of
momentum, the anouncement for RHEL5.2 beta contained rsyslog.

Regards