Mailing List Archive: I found that many logs vanished on the way of omfwd, so how to check it?

I found that many logs vanished on the way of omfwd, so how to check it?

Feb 12, 2020, 12:24 AM

Post #1 of 5 (702 views)

Hello, all,
I used omfwd to forward my logs to an endpoint. But I found that many of them have been vanished, but I don't know where to find them.

The status of sender is as below, and there are 40 senders. So the total numer is more than 12000.
Wed Jan 29 21:48:43 2020: global: origin=dynstats
Wed Jan 29 21:48:43 2020: imuxsock: origin=imuxsock submitted=351 ratelimit.discarded=0 ratelimit.numratelimiters=0
Wed Jan 29 21:48:43 2020: action-0-builtin:omfwd: origin=core.action processed=346 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 29 21:48:43 2020: resource-usage: origin=impstats utime=32025 stime=37956 maxrss=2184 minflt=566 majflt=0 inblock=0 oublock=40 nvcsw=520 nivcsw=2 openfiles=6
Wed Jan 29 21:48:43 2020: forward[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 29 21:48:43 2020: forward: origin=core.queue size=0 enqueued=351 full=0 discarded.full=0 discarded.nf=0 maxqsize=6
Wed Jan 29 21:48:43 2020: main Q: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0

On the receiver side.
Wed Jan 29 21:48:44 2020: global: origin=dynstats
Wed Jan 29 21:48:44 2020: imuxsock: origin=imuxsock submitted=2452 ratelimit.discarded=0 ratelimit.numratelimiters=0
Wed Jan 29 21:48:44 2020: action-0-omuxsock: origin=core.action processed=2947 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 29 21:48:44 2020: action-1-builtin:omfile: origin=core.action processed=2947 failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 29 21:48:44 2020: imptcp(*/514/IPv4): origin=imptcp submitted=2909 sessions.opened=51 sessions.openfailed=46 sessions.closed=0 bytes.received=45676 bytes.decompressed=458997
Wed Jan 29 21:48:44 2020: resource-usage: origin=impstats utime=185247 stime=366687 maxrss=5128 minflt=1434 majflt=0 inblock=0 oublock=1048 nvcsw=9145 nivcsw=3 openfiles=60
Wed Jan 29 21:48:44 2020: action-0-omuxsock queue: origin=core.queue size=0 enqueued=2947 full=0 discarded.full=0 discarded.nf=0 maxqsize=2285
Wed Jan 29 21:48:44 2020: action-1-builtin:omfile queue[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 29 21:48:44 2020: action-1-builtin:omfile queue: origin=core.queue size=0 enqueued=2947 full=0 discarded.full=0 discarded.nf=0 maxqsize=68
Wed Jan 29 21:48:44 2020: main Q: origin=core.queue size=0 enqueued=5361 full=0 discarded.full=0 discarded.nf=0 maxqsize=16
Wed Jan 29 21:48:44 2020: io-work-q: origin=imptcp enqueued=167 maxqsize=5

The sender's configuration is as below.
input(type="imuxsock" Socket="/dev/log" ruleset="forward")
ruleset(name="forward"
queue.type="fixedArray"
queue.size="100000"
queue.dequeueBatchSize="1000"
queue.workerThreads="4"
queue.filename="Forward"
queue.highwatermark="80000"
queue.lowwatermark="10000"
#queue.workerThreadMinimumMessages="60000"
) {
if prifilt("local5.*") then {
action(type="omfwd"
Protocol="tcp"
Target="imi"
Port="514"
ZipLevel="6"
compression.Mode="stream:always"
#compression.stream.flushOnTXEnd="off"
)
# action(type="omfile" file="/var/log/publog")
}
}

And the receiver's configuration is as below.
module(load="imptcp" threads="4")
input(type="imptcp" port="514" Compression.mode="stream:always")

So, how to check it? I did not find any failure yet.
Thank you very much

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: I found that many logs vanished on the way of omfwd, so how to check it? [ In reply to ]

rsyslog at lists

Feb 12, 2020, 12:59 AM

Post #2 of 5 (702 views)

Permalink

Maybe I overlook it, but where do you think the issue is?

Rainer

El mié., 12 feb. 2020 a las 9:25, lxy via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Hello, all,
> I used omfwd to forward my logs to an endpoint. But I found that many of them have been vanished, but I don't know where to find them.
>
>
>
> The status of sender is as below, and there are 40 senders. So the total numer is more than 12000.
> Wed Jan 29 21:48:43 2020: global: origin=dynstats
> Wed Jan 29 21:48:43 2020: imuxsock: origin=imuxsock submitted=351 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Wed Jan 29 21:48:43 2020: action-0-builtin:omfwd: origin=core.action processed=346 failed=0 suspended=0 suspended.duration=0 resumed=0
> Wed Jan 29 21:48:43 2020: resource-usage: origin=impstats utime=32025 stime=37956 maxrss=2184 minflt=566 majflt=0 inblock=0 oublock=40 nvcsw=520 nivcsw=2 openfiles=6
> Wed Jan 29 21:48:43 2020: forward[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
> Wed Jan 29 21:48:43 2020: forward: origin=core.queue size=0 enqueued=351 full=0 discarded.full=0 discarded.nf=0 maxqsize=6
> Wed Jan 29 21:48:43 2020: main Q: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>
>
> On the receiver side.
> Wed Jan 29 21:48:44 2020: global: origin=dynstats
> Wed Jan 29 21:48:44 2020: imuxsock: origin=imuxsock submitted=2452 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Wed Jan 29 21:48:44 2020: action-0-omuxsock: origin=core.action processed=2947 failed=0 suspended=0 suspended.duration=0 resumed=0
> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile: origin=core.action processed=2947 failed=0 suspended=0 suspended.duration=0 resumed=0
> Wed Jan 29 21:48:44 2020: imptcp(*/514/IPv4): origin=imptcp submitted=2909 sessions.opened=51 sessions.openfailed=46 sessions.closed=0 bytes.received=45676 bytes.decompressed=458997
> Wed Jan 29 21:48:44 2020: resource-usage: origin=impstats utime=185247 stime=366687 maxrss=5128 minflt=1434 majflt=0 inblock=0 oublock=1048 nvcsw=9145 nivcsw=3 openfiles=60
> Wed Jan 29 21:48:44 2020: action-0-omuxsock queue: origin=core.queue size=0 enqueued=2947 full=0 discarded.full=0 discarded.nf=0 maxqsize=2285
> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile queue[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile queue: origin=core.queue size=0 enqueued=2947 full=0 discarded.full=0 discarded.nf=0 maxqsize=68
> Wed Jan 29 21:48:44 2020: main Q: origin=core.queue size=0 enqueued=5361 full=0 discarded.full=0 discarded.nf=0 maxqsize=16
> Wed Jan 29 21:48:44 2020: io-work-q: origin=imptcp enqueued=167 maxqsize=5
>
>
> The sender's configuration is as below.
> input(type="imuxsock" Socket="/dev/log" ruleset="forward")
> ruleset(name="forward"
> queue.type="fixedArray"
> queue.size="100000"
> queue.dequeueBatchSize="1000"
> queue.workerThreads="4"
> queue.filename="Forward"
> queue.highwatermark="80000"
> queue.lowwatermark="10000"
> #queue.workerThreadMinimumMessages="60000"
> ) {
> if prifilt("local5.*") then {
> action(type="omfwd"
> Protocol="tcp"
> Target="imi"
> Port="514"
> ZipLevel="6"
> compression.Mode="stream:always"
> #compression.stream.flushOnTXEnd="off"
> )
> # action(type="omfile" file="/var/log/publog")
> }
> }
>
>
> And the receiver's configuration is as below.
> module(load="imptcp" threads="4")
> input(type="imptcp" port="514" Compression.mode="stream:always")
>
>
> So, how to check it? I did not find any failure yet.
> Thank you very much
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: I found that many logs vanished on the way of omfwd, so how to check it? [ In reply to ]

rsyslog at lists

Feb 12, 2020, 9:17 AM

Post #3 of 5 (701 views)

Permalink

you don't show any config on the receiver that would write logs to any file.

you don't show us the full config, so we don't know if the numbers in pstats are
reset after each report or not

David Lang

On Wed, 12 Feb 2020, lxy via rsyslog wrote:

> Date: Wed, 12 Feb 2020 16:24:51 +0800 (CST)
> From: lxy via rsyslog <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: lxy <lxyscls@163.com>
> Subject: [rsyslog] I found that many logs vanished on the way of omfwd,
> so how to check it?
>
> Hello, all,
> I used omfwd to forward my logs to an endpoint. But I found that many of them have been vanished, but I don't know where to find them.
>
>
>
> The status of sender is as below, and there are 40 senders. So the total numer is more than 12000.
> Wed Jan 29 21:48:43 2020: global: origin=dynstats
> Wed Jan 29 21:48:43 2020: imuxsock: origin=imuxsock submitted=351 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Wed Jan 29 21:48:43 2020: action-0-builtin:omfwd: origin=core.action processed=346 failed=0 suspended=0 suspended.duration=0 resumed=0
> Wed Jan 29 21:48:43 2020: resource-usage: origin=impstats utime=32025 stime=37956 maxrss=2184 minflt=566 majflt=0 inblock=0 oublock=40 nvcsw=520 nivcsw=2 openfiles=6
> Wed Jan 29 21:48:43 2020: forward[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
> Wed Jan 29 21:48:43 2020: forward: origin=core.queue size=0 enqueued=351 full=0 discarded.full=0 discarded.nf=0 maxqsize=6
> Wed Jan 29 21:48:43 2020: main Q: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>
>
> On the receiver side.
> Wed Jan 29 21:48:44 2020: global: origin=dynstats
> Wed Jan 29 21:48:44 2020: imuxsock: origin=imuxsock submitted=2452 ratelimit.discarded=0 ratelimit.numratelimiters=0
> Wed Jan 29 21:48:44 2020: action-0-omuxsock: origin=core.action processed=2947 failed=0 suspended=0 suspended.duration=0 resumed=0
> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile: origin=core.action processed=2947 failed=0 suspended=0 suspended.duration=0 resumed=0
> Wed Jan 29 21:48:44 2020: imptcp(*/514/IPv4): origin=imptcp submitted=2909 sessions.opened=51 sessions.openfailed=46 sessions.closed=0 bytes.received=45676 bytes.decompressed=458997
> Wed Jan 29 21:48:44 2020: resource-usage: origin=impstats utime=185247 stime=366687 maxrss=5128 minflt=1434 majflt=0 inblock=0 oublock=1048 nvcsw=9145 nivcsw=3 openfiles=60
> Wed Jan 29 21:48:44 2020: action-0-omuxsock queue: origin=core.queue size=0 enqueued=2947 full=0 discarded.full=0 discarded.nf=0 maxqsize=2285
> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile queue[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile queue: origin=core.queue size=0 enqueued=2947 full=0 discarded.full=0 discarded.nf=0 maxqsize=68
> Wed Jan 29 21:48:44 2020: main Q: origin=core.queue size=0 enqueued=5361 full=0 discarded.full=0 discarded.nf=0 maxqsize=16
> Wed Jan 29 21:48:44 2020: io-work-q: origin=imptcp enqueued=167 maxqsize=5
>
>
> The sender's configuration is as below.
> input(type="imuxsock" Socket="/dev/log" ruleset="forward")
> ruleset(name="forward"
> queue.type="fixedArray"
> queue.size="100000"
> queue.dequeueBatchSize="1000"
> queue.workerThreads="4"
> queue.filename="Forward"
> queue.highwatermark="80000"
> queue.lowwatermark="10000"
> #queue.workerThreadMinimumMessages="60000"
> ) {
> if prifilt("local5.*") then {
> action(type="omfwd"
> Protocol="tcp"
> Target="imi"
> Port="514"
> ZipLevel="6"
> compression.Mode="stream:always"
> #compression.stream.flushOnTXEnd="off"
> )
> # action(type="omfile" file="/var/log/publog")
> }
> }
>
>
> And the receiver's configuration is as below.
> module(load="imptcp" threads="4")
> input(type="imptcp" port="514" Compression.mode="stream:always")
>
>
> So, how to check it? I did not find any failure yet.
> Thank you very much
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: I found that many logs vanished on the way of omfwd, so how to check it? [ In reply to ]

rsyslog at lists

Feb 12, 2020, 6:11 PM

Post #4 of 5 (698 views)

Permalink

I am so sorry. I need to forward this reply to the mail list.
OK, because I think the submitted of imptcp could say something, so I have omitted some config.
And the result of pstats is accumulated, not reset.
If any other information can help, please tell me.

The full config of sender is as below.

module(load="imuxsock")
input(type="imuxsock" Socket="/dev/log" ruleset="forward")

module(load="impstats"
interval="3600"
severity="7"
log.syslog="off"
log.file="/var/log/rsyslog.stats")

ruleset(name="forward"
queue.type="fixedArray"
queue.size="100000"
queue.dequeueBatchSize="1000"
queue.workerThreads="4"
queue.filename="Forward"
queue.highwatermark="80000"
queue.lowwatermark="10000"
#queue.workerThreadMinimumMessages="60000"
) {
if prifilt("local5.*") then {
action(type="omfwd"
Protocol="tcp"
Target="imi"
Port="514"
ZipLevel="6"
compression.Mode="stream:always"
#compression.stream.flushOnTXEnd="off"
)
# action(type="omfile" file="/var/log/publog")
}
}
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

The full config of receiver.

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$AddUnixListenSocket /dev/log

module(load="imptcp" threads="4")
input(type="imptcp" port="514" Compression.mode="stream:always")

$ModLoad omuxsock
$OMUxSockSocket /var/tmp/log

module(load="impstats"
interval="3600"
severity="7"
log.syslog="off"
log.file="/var/log/rsyslog.stats")
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
template(name="RSYSLOG_SKSForwardFormat" type="string"
string="<%PRI%>%TIMESTAMP% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%")

$outchannel pure,/usr/admin/log/purelog,1073741824,/imi/lg.sh
$ActionQueueType FixedArray
$ActionQueueSize 400000
$ActionQueueWorkerThreads 4
$ActionQueueDequeueBatchSize 1000
$ActionQueueDiscardMark 320000
$ActionQueueDiscardSeverity 3
local5.* :omuxsock:;RSYSLOG_SKSForwardFormat

$ActionQueueType FixedArray
$ActionQueueSize 2400000
$ActionQueueWorkerThreads 4
$ActionQueueDequeueBatchSize 1000
$ActionQueueFileName Forward2
$ActionQueueMaxDiskSpace 10000000000
$ActionQueueHighWatermark 2000000
$ActionQueueLowWatermark 100000
$ActionQueueDiscardMark 1920000
$ActionQueueDiscardSeverity 8
local5.* :omfile:$pure

Thank you very much!

At 2020-02-13 01:17:15, "David Lang" <david@lang.hm> wrote:
>you don't show any config on the receiver that would write logs to any file.
>
>you don't show us the full config, so we don't know if the numbers in pstats are
>reset after each report or not
>
>David Lang
>
>On Wed, 12 Feb 2020, lxy via rsyslog wrote:
>
>> Date: Wed, 12 Feb 2020 16:24:51 +0800 (CST)
>> From: lxy via rsyslog <rsyslog@lists.adiscon.com>
>> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
>> Cc: lxy <lxyscls@163.com>
>> Subject: [rsyslog] I found that many logs vanished on the way of omfwd,
>> so how to check it?
>>
>> Hello, all,
>> I used omfwd to forward my logs to an endpoint. But I found that many of them have been vanished, but I don't know where to find them.
>>
>>
>>
>> The status of sender is as below, and there are 40 senders. So the total numer is more than 12000.
>> Wed Jan 29 21:48:43 2020: global: origin=dynstats
>> Wed Jan 29 21:48:43 2020: imuxsock: origin=imuxsock submitted=351 ratelimit.discarded=0 ratelimit.numratelimiters=0
>> Wed Jan 29 21:48:43 2020: action-0-builtin:omfwd: origin=core.action processed=346 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Wed Jan 29 21:48:43 2020: resource-usage: origin=impstats utime=32025 stime=37956 maxrss=2184 minflt=566 majflt=0 inblock=0 oublock=40 nvcsw=520 nivcsw=2 openfiles=6
>> Wed Jan 29 21:48:43 2020: forward[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>> Wed Jan 29 21:48:43 2020: forward: origin=core.queue size=0 enqueued=351 full=0 discarded.full=0 discarded.nf=0 maxqsize=6
>> Wed Jan 29 21:48:43 2020: main Q: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>>
>>
>> On the receiver side.
>> Wed Jan 29 21:48:44 2020: global: origin=dynstats
>> Wed Jan 29 21:48:44 2020: imuxsock: origin=imuxsock submitted=2452 ratelimit.discarded=0 ratelimit.numratelimiters=0
>> Wed Jan 29 21:48:44 2020: action-0-omuxsock: origin=core.action processed=2947 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile: origin=core.action processed=2947 failed=0 suspended=0 suspended.duration=0 resumed=0
>> Wed Jan 29 21:48:44 2020: imptcp(*/514/IPv4): origin=imptcp submitted=2909 sessions.opened=51 sessions.openfailed=46 sessions.closed=0 bytes.received=45676 bytes.decompressed=458997
>> Wed Jan 29 21:48:44 2020: resource-usage: origin=impstats utime=185247 stime=366687 maxrss=5128 minflt=1434 majflt=0 inblock=0 oublock=1048 nvcsw=9145 nivcsw=3 openfiles=60
>> Wed Jan 29 21:48:44 2020: action-0-omuxsock queue: origin=core.queue size=0 enqueued=2947 full=0 discarded.full=0 discarded.nf=0 maxqsize=2285
>> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile queue[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile queue: origin=core.queue size=0 enqueued=2947 full=0 discarded.full=0 discarded.nf=0 maxqsize=68
>> Wed Jan 29 21:48:44 2020: main Q: origin=core.queue size=0 enqueued=5361 full=0 discarded.full=0 discarded.nf=0 maxqsize=16
>> Wed Jan 29 21:48:44 2020: io-work-q: origin=imptcp enqueued=167 maxqsize=5
>>
>>
>> The sender's configuration is as below.
>> input(type="imuxsock" Socket="/dev/log" ruleset="forward")
>> ruleset(name="forward"
>> queue.type="fixedArray"
>> queue.size="100000"
>> queue.dequeueBatchSize="1000"
>> queue.workerThreads="4"
>> queue.filename="Forward"
>> queue.highwatermark="80000"
>> queue.lowwatermark="10000"
>> #queue.workerThreadMinimumMessages="60000"
>> ) {
>> if prifilt("local5.*") then {
>> action(type="omfwd"
>> Protocol="tcp"
>> Target="imi"
>> Port="514"
>> ZipLevel="6"
>> compression.Mode="stream:always"
>> #compression.stream.flushOnTXEnd="off"
>> )
>> # action(type="omfile" file="/var/log/publog")
>> }
>> }
>>
>>
>> And the receiver's configuration is as below.
>> module(load="imptcp" threads="4")
>> input(type="imptcp" port="514" Compression.mode="stream:always")
>>
>>
>> So, how to check it? I did not find any failure yet.
>> Thank you very much
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Re: I found that many logs vanished on the way of omfwd, so how to check it? [ In reply to ]

rsyslog at lists

Feb 16, 2020, 4:42 PM

Post #5 of 5 (693 views)

Permalink

Any idea, or any issue I can check?
The final outputs are the same. So I think logs are vanished on the air.

Thank you

At 2020-02-13 10:11:55, "lxy via rsyslog" <rsyslog@lists.adiscon.com> wrote:
>I am so sorry. I need to forward this reply to the mail list.
>OK, because I think the submitted of imptcp could say something, so I have omitted some config.
>And the result of pstats is accumulated, not reset.
>If any other information can help, please tell me.
>
>
>The full config of sender is as below.
>
>
>module(load="imuxsock")
>input(type="imuxsock" Socket="/dev/log" ruleset="forward")
>
>
>module(load="impstats"
> interval="3600"
> severity="7"
> log.syslog="off"
> log.file="/var/log/rsyslog.stats")
>
>
>ruleset(name="forward"
> queue.type="fixedArray"
> queue.size="100000"
> queue.dequeueBatchSize="1000"
> queue.workerThreads="4"
> queue.filename="Forward"
> queue.highwatermark="80000"
> queue.lowwatermark="10000"
>#queue.workerThreadMinimumMessages="60000"
> ) {
> if prifilt("local5.*") then {
> action(type="omfwd"
> Protocol="tcp"
> Target="imi"
> Port="514"
> ZipLevel="6"
> compression.Mode="stream:always"
>#compression.stream.flushOnTXEnd="off"
> )
># action(type="omfile" file="/var/log/publog")
> }
>}
>$WorkDirectory /var/lib/rsyslog
>$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>
>
>The full config of receiver.
>
>
>$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
>$AddUnixListenSocket /dev/log
>
>
>module(load="imptcp" threads="4")
>input(type="imptcp" port="514" Compression.mode="stream:always")
>
>
>$ModLoad omuxsock
>$OMUxSockSocket /var/tmp/log
>
>
>module(load="impstats"
> interval="3600"
> severity="7"
> log.syslog="off"
> log.file="/var/log/rsyslog.stats")
>$WorkDirectory /var/lib/rsyslog
>$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>template(name="RSYSLOG_SKSForwardFormat" type="string"
> string="<%PRI%>%TIMESTAMP% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%")
>
>
>$outchannel pure,/usr/admin/log/purelog,1073741824,/imi/lg.sh
>$ActionQueueType FixedArray
>$ActionQueueSize 400000
>$ActionQueueWorkerThreads 4
>$ActionQueueDequeueBatchSize 1000
>$ActionQueueDiscardMark 320000
>$ActionQueueDiscardSeverity 3
>local5.* :omuxsock:;RSYSLOG_SKSForwardFormat
>
>
>$ActionQueueType FixedArray
>$ActionQueueSize 2400000
>$ActionQueueWorkerThreads 4
>$ActionQueueDequeueBatchSize 1000
>$ActionQueueFileName Forward2
>$ActionQueueMaxDiskSpace 10000000000
>$ActionQueueHighWatermark 2000000
>$ActionQueueLowWatermark 100000
>$ActionQueueDiscardMark 1920000
>$ActionQueueDiscardSeverity 8
>local5.* :omfile:$pure
>
>
>Thank you very much!
>
>
>
>
>
>
>
>
>
>
>
>At 2020-02-13 01:17:15, "David Lang" <david@lang.hm> wrote:
>>you don't show any config on the receiver that would write logs to any file.
>>
>>you don't show us the full config, so we don't know if the numbers in pstats are
>>reset after each report or not
>>
>>David Lang
>>
>>On Wed, 12 Feb 2020, lxy via rsyslog wrote:
>>
>>> Date: Wed, 12 Feb 2020 16:24:51 +0800 (CST)
>>> From: lxy via rsyslog <rsyslog@lists.adiscon.com>
>>> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
>>> Cc: lxy <lxyscls@163.com>
>>> Subject: [rsyslog] I found that many logs vanished on the way of omfwd,
>>> so how to check it?
>>>
>>> Hello, all,
>>> I used omfwd to forward my logs to an endpoint. But I found that many of them have been vanished, but I don't know where to find them.
>>>
>>>
>>>
>>> The status of sender is as below, and there are 40 senders. So the total numer is more than 12000.
>>> Wed Jan 29 21:48:43 2020: global: origin=dynstats
>>> Wed Jan 29 21:48:43 2020: imuxsock: origin=imuxsock submitted=351 ratelimit.discarded=0 ratelimit.numratelimiters=0
>>> Wed Jan 29 21:48:43 2020: action-0-builtin:omfwd: origin=core.action processed=346 failed=0 suspended=0 suspended.duration=0 resumed=0
>>> Wed Jan 29 21:48:43 2020: resource-usage: origin=impstats utime=32025 stime=37956 maxrss=2184 minflt=566 majflt=0 inblock=0 oublock=40 nvcsw=520 nivcsw=2 openfiles=6
>>> Wed Jan 29 21:48:43 2020: forward[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>>> Wed Jan 29 21:48:43 2020: forward: origin=core.queue size=0 enqueued=351 full=0 discarded.full=0 discarded.nf=0 maxqsize=6
>>> Wed Jan 29 21:48:43 2020: main Q: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>>>
>>>
>>> On the receiver side.
>>> Wed Jan 29 21:48:44 2020: global: origin=dynstats
>>> Wed Jan 29 21:48:44 2020: imuxsock: origin=imuxsock submitted=2452 ratelimit.discarded=0 ratelimit.numratelimiters=0
>>> Wed Jan 29 21:48:44 2020: action-0-omuxsock: origin=core.action processed=2947 failed=0 suspended=0 suspended.duration=0 resumed=0
>>> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile: origin=core.action processed=2947 failed=0 suspended=0 suspended.duration=0 resumed=0
>>> Wed Jan 29 21:48:44 2020: imptcp(*/514/IPv4): origin=imptcp submitted=2909 sessions.opened=51 sessions.openfailed=46 sessions.closed=0 bytes.received=45676 bytes.decompressed=458997
>>> Wed Jan 29 21:48:44 2020: resource-usage: origin=impstats utime=185247 stime=366687 maxrss=5128 minflt=1434 majflt=0 inblock=0 oublock=1048 nvcsw=9145 nivcsw=3 openfiles=60
>>> Wed Jan 29 21:48:44 2020: action-0-omuxsock queue: origin=core.queue size=0 enqueued=2947 full=0 discarded.full=0 discarded.nf=0 maxqsize=2285
>>> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile queue[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
>>> Wed Jan 29 21:48:44 2020: action-1-builtin:omfile queue: origin=core.queue size=0 enqueued=2947 full=0 discarded.full=0 discarded.nf=0 maxqsize=68
>>> Wed Jan 29 21:48:44 2020: main Q: origin=core.queue size=0 enqueued=5361 full=0 discarded.full=0 discarded.nf=0 maxqsize=16
>>> Wed Jan 29 21:48:44 2020: io-work-q: origin=imptcp enqueued=167 maxqsize=5
>>>
>>>
>>> The sender's configuration is as below.
>>> input(type="imuxsock" Socket="/dev/log" ruleset="forward")
>>> ruleset(name="forward"
>>> queue.type="fixedArray"
>>> queue.size="100000"
>>> queue.dequeueBatchSize="1000"
>>> queue.workerThreads="4"
>>> queue.filename="Forward"
>>> queue.highwatermark="80000"
>>> queue.lowwatermark="10000"
>>> #queue.workerThreadMinimumMessages="60000"
>>> ) {
>>> if prifilt("local5.*") then {
>>> action(type="omfwd"
>>> Protocol="tcp"
>>> Target="imi"
>>> Port="514"
>>> ZipLevel="6"
>>> compression.Mode="stream:always"
>>> #compression.stream.flushOnTXEnd="off"
>>> )
>>> # action(type="omfile" file="/var/log/publog")
>>> }
>>> }
>>>
>>>
>>> And the receiver's configuration is as below.
>>> module(load="imptcp" threads="4")
>>> input(type="imptcp" port="514" Compression.mode="stream:always")
>>>
>>>
>>> So, how to check it? I did not find any failure yet.
>>> Thank you very much
>>>
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>
>_______________________________________________
>rsyslog mailing list
>http://lists.adiscon.net/mailman/listinfo/rsyslog
>http://www.rsyslog.com/professional-services/
>What's up with rsyslog? Follow https://twitter.com/rgerhards
>NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.