Mailing List Archive

Multiple instances are easy, but care needs to be taken to ensure they don’t collide. However, the first course of action to be to sanity check the existing configuration AND make sure that it is not an “old” version of rsyslog.

The list may be able to help if you post your entire configuration.

Regards,

> On Jun 16, 2022, at 09:21, vijay kumar via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> Hi Team,
>
> My rsyslog service is getting restarted very frequently and we understand
> it is due to race between the various threads, which causes one thread to
> free a message field while another tries to read/write it.
>
> log:
> ===
> ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address
> 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp 0x7f6079d74d88
> READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal)
> #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc)
> #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288
> #2 0x7f607d5066a5 in readjournal imjournal.c:497
> #3 0x55ae0523366e in thrdStarter ../threads.c:243
> #4 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>
> 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region
> [0x61e000000080,0x61e000000ac2)
> allocated by thread T3 (in:imjournal) here:
> #0 0x7f6085afcfe8 in __interceptor_realloc (/lib64/libasan.so.5+0xeffe8)
> #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b)
>
> Thread T3 (in:imjournal) created by T0 here:
> #0 0x7f6085a5fea3 in __interceptor_pthread_create
> (/lib64/libasan.so.5+0x52ea3)
> #1 0x55ae05234470 in thrdCreate ../threads.c:289
>
> SUMMARY: AddressSanitizer: heap-buffer-overflow
> (/lib64/libasan.so.5+0xb92fc)
> Shadow bytes around the buggy address:
> 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa
> 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Container overflow: fc
> Array cookie: ac
> Intra object redzone: bb
> ASan internal: fe
> Left alloca redzone: ca
> Right alloca redzone: cb
> =================================================================
> ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp 0x7f60706215f8
> READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg)
> #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26)
> #1 0x55ae0516d061 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x390061)
> #2 0x55ae0516e522 in MsgDup msg.c:1129
> #3 0x55ae05205d58 in execCall ruleset.c:290
> #4 0x55ae05205d58 in scriptExec ruleset.c:608
> #5 0x55ae05206532 in execIf ruleset.c:313
> #6 0x55ae05206532 in scriptExec ruleset.c:614
> #7 0x55ae05208868 in processBatch ruleset.c:660
> #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> #9 0x55ae051f100d in ConsumerReg queue.c:2145
> #10 0x55ae051e0804 in wtiWorker wti.c:428
> #11 0x55ae051d9dd5 in wtpWorker wtp.c:435
> #12 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>
> 0x60c0005d85c0 is located 0 bytes inside of 128-byte region
> [0x60c0005d85c0,0x60c0005d8640)
> freed by thread T6 (imudp(w1)) here:
> #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
> #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
>
> previously allocated by thread T6 (imudp(w1)) here:
> #0 0x7f6085afcba8 in __interceptor_malloc (/lib64/libasan.so.5+0xefba8)
> #1 0x55ae0516cff4 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x38fff4)
>
> Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here:
> #0 0x7f6085a5fea3 in __interceptor_pthread_create
> (/lib64/libasan.so.5+0x52ea3)
> #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
>
> Thread T6 (imudp(w1)) created by T0 here:
> #0 0x7f6085a5fea3 in __interceptor_pthread_create
> (/lib64/libasan.so.5+0x52ea3)
> #1 0x55ae05234470 in thrdCreate ../threads.c:289
>
> SUMMARY: AddressSanitizer: heap-use-after-free (/lib64/libasan.so.5+0x40b26)
> Shadow bytes around the buggy address:
> 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
> 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
> 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Container overflow: fc
> Array cookie: ac
> Intra object redzone: bb
> ASan internal: fe
> Left alloca redzone: ca
> Right alloca redzone: cb
> =================================================================
> ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp 0x7f6066965990
> WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local)
> #0 0x55ae0520a722 in propDestruct prop.c:63
> #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457
> #2 0x55ae05170299 in resolveDNS msg.c:522
> #3 0x55ae0517064a in getRcvFromIP msg.c:558
> #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317)
> #5 0x55ae0524aece in tplToString ../template.c:207
> #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114
> #7 0x55ae0522a4ab in processMsgMain ../action.c:1648
> #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825
> #9 0x55ae05205826 in execAct ruleset.c:209
> #10 0x55ae05205826 in scriptExec ruleset.c:599
> #11 0x55ae05208868 in processBatch ruleset.c:660
> #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> #13 0x55ae051f100d in ConsumerReg queue.c:2145
> #14 0x55ae051e0804 in wtiWorker wti.c:428
> #15 0x55ae051d9dd5 in wtpWorker wtp.c:435
> #16 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>
> 0x6040000285a0 is located 16 bytes inside of 48-byte region
> [0x604000028590,0x6040000285c0)
> freed by thread T16 (rs:qradar.local) here:
> #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
> #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
>
> previously allocated by thread T16 (rs:qradar.local) here:
> #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0)
> #1 0x55ae0520a439 in propConstruct prop.c:56
>
> Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here:
> #0 0x7f6085a5fea3 in __interceptor_pthread_create
> (/lib64/libasan.so.5+0x52ea3)
> #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
>
> SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in propDestruct
> Shadow bytes around the buggy address:
> 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd
> 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa
> 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa
> 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
> 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Container overflow: fc
> Array cookie: ac
> Intra object redzone: bb
> ASan internal: fe
> Left alloca redzone: ca
> Right alloca redzone: cb
> ==3035157==ABORTING
>
> =================================
> what is the possible solution ???
>
> would be to have multiple rsyslog instances, which is possible if the
> traffic is split between ports. If yes could you please suggest how to
> configure??
>
> Thanks & Regards
> Vijay Kumar Kanukula
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi John/Team,

Please find the attached configuration files and i am running this RHEL 8.6.

rsyslogd 8.2102.0-7.el8_6.1 (aka 2021.02) compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /var/run/rsyslogd.pid
Number of Bits in RainerScript integers: 64

Thanks & Regards
Vijay Kumar Kanukula

On Thu, 16 Jun 2022 at 19:57, John Chivian <jchivian@chivian.com> wrote:

> Multiple instances are easy, but care needs to be taken to ensure they
> don’t collide. However, the first course of action to be to sanity check
> the existing configuration AND make sure that it is not an “old” version of
> rsyslog.
>
> The list may be able to help if you post your entire configuration.
>
> Regards,
>
> > On Jun 16, 2022, at 09:21, vijay kumar via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
> >
> > Hi Team,
> >
> > My rsyslog service is getting restarted very frequently and we understand
> > it is due to race between the various threads, which causes one thread to
> > free a message field while another tries to read/write it.
> >
> > log:
> > ===
> > ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address
> > 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp 0x7f6079d74d88
> > READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal)
> > #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc)
> > #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288
> > #2 0x7f607d5066a5 in readjournal imjournal.c:497
> > #3 0x55ae0523366e in thrdStarter ../threads.c:243
> > #4 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> > #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >
> > 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region
> > [0x61e000000080,0x61e000000ac2)
> > allocated by thread T3 (in:imjournal) here:
> > #0 0x7f6085afcfe8 in __interceptor_realloc
> (/lib64/libasan.so.5+0xeffe8)
> > #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b)
> >
> > Thread T3 (in:imjournal) created by T0 here:
> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> > (/lib64/libasan.so.5+0x52ea3)
> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
> >
> > SUMMARY: AddressSanitizer: heap-buffer-overflow
> > (/lib64/libasan.so.5+0xb92fc)
> > Shadow bytes around the buggy address:
> > 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa
> > 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > Shadow byte legend (one shadow byte represents 8 application bytes):
> > Addressable: 00
> > Partially addressable: 01 02 03 04 05 06 07
> > Heap left redzone: fa
> > Freed heap region: fd
> > Stack left redzone: f1
> > Stack mid redzone: f2
> > Stack right redzone: f3
> > Stack after return: f5
> > Stack use after scope: f8
> > Global redzone: f9
> > Global init order: f6
> > Poisoned by user: f7
> > Container overflow: fc
> > Array cookie: ac
> > Intra object redzone: bb
> > ASan internal: fe
> > Left alloca redzone: ca
> > Right alloca redzone: cb
> > =================================================================
> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> > 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp 0x7f60706215f8
> > READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg)
> > #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26)
> > #1 0x55ae0516d061 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x390061)
> > #2 0x55ae0516e522 in MsgDup msg.c:1129
> > #3 0x55ae05205d58 in execCall ruleset.c:290
> > #4 0x55ae05205d58 in scriptExec ruleset.c:608
> > #5 0x55ae05206532 in execIf ruleset.c:313
> > #6 0x55ae05206532 in scriptExec ruleset.c:614
> > #7 0x55ae05208868 in processBatch ruleset.c:660
> > #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> > #9 0x55ae051f100d in ConsumerReg queue.c:2145
> > #10 0x55ae051e0804 in wtiWorker wti.c:428
> > #11 0x55ae051d9dd5 in wtpWorker wtp.c:435
> > #12 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> > #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >
> > 0x60c0005d85c0 is located 0 bytes inside of 128-byte region
> > [0x60c0005d85c0,0x60c0005d8640)
> > freed by thread T6 (imudp(w1)) here:
> > #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
> >
> > previously allocated by thread T6 (imudp(w1)) here:
> > #0 0x7f6085afcba8 in __interceptor_malloc
> (/lib64/libasan.so.5+0xefba8)
> > #1 0x55ae0516cff4 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x38fff4)
> >
> > Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here:
> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> > (/lib64/libasan.so.5+0x52ea3)
> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
> >
> > Thread T6 (imudp(w1)) created by T0 here:
> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> > (/lib64/libasan.so.5+0x52ea3)
> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
> >
> > SUMMARY: AddressSanitizer: heap-use-after-free
> (/lib64/libasan.so.5+0x40b26)
> > Shadow bytes around the buggy address:
> > 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> > 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> > 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
> > 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> > 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> > =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
> > 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> > 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> > 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> > Shadow byte legend (one shadow byte represents 8 application bytes):
> > Addressable: 00
> > Partially addressable: 01 02 03 04 05 06 07
> > Heap left redzone: fa
> > Freed heap region: fd
> > Stack left redzone: f1
> > Stack mid redzone: f2
> > Stack right redzone: f3
> > Stack after return: f5
> > Stack use after scope: f8
> > Global redzone: f9
> > Global init order: f6
> > Poisoned by user: f7
> > Container overflow: fc
> > Array cookie: ac
> > Intra object redzone: bb
> > ASan internal: fe
> > Left alloca redzone: ca
> > Right alloca redzone: cb
> > =================================================================
> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> > 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp 0x7f6066965990
> > WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local)
> > #0 0x55ae0520a722 in propDestruct prop.c:63
> > #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457
> > #2 0x55ae05170299 in resolveDNS msg.c:522
> > #3 0x55ae0517064a in getRcvFromIP msg.c:558
> > #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317)
> > #5 0x55ae0524aece in tplToString ../template.c:207
> > #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114
> > #7 0x55ae0522a4ab in processMsgMain ../action.c:1648
> > #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825
> > #9 0x55ae05205826 in execAct ruleset.c:209
> > #10 0x55ae05205826 in scriptExec ruleset.c:599
> > #11 0x55ae05208868 in processBatch ruleset.c:660
> > #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> > #13 0x55ae051f100d in ConsumerReg queue.c:2145
> > #14 0x55ae051e0804 in wtiWorker wti.c:428
> > #15 0x55ae051d9dd5 in wtpWorker wtp.c:435
> > #16 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> > #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >
> > 0x6040000285a0 is located 16 bytes inside of 48-byte region
> > [0x604000028590,0x6040000285c0)
> > freed by thread T16 (rs:qradar.local) here:
> > #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
> >
> > previously allocated by thread T16 (rs:qradar.local) here:
> > #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0)
> > #1 0x55ae0520a439 in propConstruct prop.c:56
> >
> > Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here:
> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> > (/lib64/libasan.so.5+0x52ea3)
> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
> >
> > SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in propDestruct
> > Shadow bytes around the buggy address:
> > 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> > 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd
> > 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa
> > 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> > =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa
> > 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> > 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
> > 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> > 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> > 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
> > Shadow byte legend (one shadow byte represents 8 application bytes):
> > Addressable: 00
> > Partially addressable: 01 02 03 04 05 06 07
> > Heap left redzone: fa
> > Freed heap region: fd
> > Stack left redzone: f1
> > Stack mid redzone: f2
> > Stack right redzone: f3
> > Stack after return: f5
> > Stack use after scope: f8
> > Global redzone: f9
> > Global init order: f6
> > Poisoned by user: f7
> > Container overflow: fc
> > Array cookie: ac
> > Intra object redzone: bb
> > ASan internal: fe
> > Left alloca redzone: ca
> > Right alloca redzone: cb
> > ==3035157==ABORTING
> >
> > =================================
> > what is the possible solution ???
> >
> > would be to have multiple rsyslog instances, which is possible if the
> > traffic is split between ports. If yes could you please suggest how to
> > configure??
> >
> > Thanks & Regards
> > Vijay Kumar Kanukula
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
>

On 16.06.2022 16:21, vijay kumar via rsyslog wrote:
> Hi Team,
>
> My rsyslog service is getting restarted very frequently and we understand
> it is due to race between the various threads, which causes one thread to
> free a message field while another tries to read/write it.
>
> [cut]
>
> would be to have multiple rsyslog instances, which is possible if the
> traffic is split between ports. If yes could you please suggest how to
> configure??

Multiple instances are multiple processes so there is no way for the
threads from two different processes to interact with each other.

That's what we have process-separation at OS-level for ;-)

The question is - what is your rsyslog version (and whether you're
hitting some well-known and already fixed bug). And what is your config.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On 16.06.2022 17:28, Mariusz Kruk via rsyslog wrote:
> On 16.06.2022 16:21, vijay kumar via rsyslog wrote:
>> Hi Team,
>>
>> My rsyslog service is getting restarted very frequently and we
>> understand
>> it is due to race between the various threads, which causes one
>> thread to
>> free a message field while another tries to read/write it.
>>
>> [cut]
>>
>>   would be to have multiple rsyslog instances, which is possible if the
>> traffic is split between ports. If yes could you please suggest how to
>> configure??
>
> Multiple instances are multiple processes so there is no way for the
> threads from two different processes to interact with each other.
>
> That's what we have process-separation at OS-level for ;-)
>
> The question is - what is your rsyslog version (and whether you're
> hitting some well-known and already fixed bug). And what is your config.

OK. From the config you posted I see that you do _not_ have multiple
rsyslog instances. You have multiple inputs defined within a single
instance. There's nothing unusual with that. I have several setups with
rsyslog listening on multiple ports or multiple IPs and nothing bad happens.

You didn't post the global /etc/rsyslogd.conf but I assume it's pretty
typical package-supplied config with the include directive at the end so
all the logic happens in those files you posted.

Anyway, what I can advise from experience, since you have many inputs,
many outputs and a bit of magic routing those inputs to outputs - since
you're separating the inputs by port, it's more convenient (and allows
you to set up separate queues to process each such pipeleine) to bind
each output with a specific ruleset processing just this type of events.

At the moment you're putting all events into main processing queue and
then you're calling various filters to match specific event types to
specific actions. That's a bit ineffective. And separating events into
dedicated queues gives you better accounting.

So instead of defining all inputs as "generic" ones and then calling
your filters like this:

## Input-to-Output Flows
#
# Process incoming events from UDP 514
if (re_match($inputname, "udp514"))
  then {
    call syslog.qradar
    call qradar.local
    stop
}

You can simply do

ruleset (name="process_qradar [... queue parameters ...]) {
   call syslog.qradar
   call qradar.local
   stop
}

input(type="imudp" port="514" name="udp514" ruleset="process_qradar")

You could also do single input and dynamically assign ruleset based on
the source IP but that's a completely different story.


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

it's important to realize how Rsyslog processes the config files.

It does not care what you have in what file.

if you start rsyslog with -o /path/to/file then the file will have the config as
rsyslog sees it.

When rsyslog starts, it goes through the file and finds all the module and input
commands (and a few others) and processes them. Then it sits and waits for log
events to happen. When a log event happens, it then processes all the actions
in the combined files for the ruleset that the log event happens in (unless you
bind a ruleset to an input as Mariusz describes, they all are in the default
ruleset)

so you may have put an input in a file along with the actions that you want to
take for that input, but that's not what rsyslog sees, the different files are
purely for administrative convieninece, rsyslog doesn't care if every line in
the config is in a separate file or if they are all in one file, it will do
exactly the same thing.

David Lang

On Thu, 16 Jun 2022, Mariusz Kruk via rsyslog wrote:

> Date: Thu, 16 Jun 2022 17:51:35 +0200
> From: Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: Mariusz Kruk <kruk@epsilon.eu.org>
> Subject: Re: [rsyslog] multiple rsyslog instances
>
>
> On 16.06.2022 17:28, Mariusz Kruk via rsyslog wrote:
>> On 16.06.2022 16:21, vijay kumar via rsyslog wrote:
>>> Hi Team,
>>>
>>> My rsyslog service is getting restarted very frequently and we
>>> understand
>>> it is due to race between the various threads, which causes one
>>> thread to
>>> free a message field while another tries to read/write it.
>>>
>>> [cut]
>>>
>>>   would be to have multiple rsyslog instances, which is possible if the
>>> traffic is split between ports. If yes could you please suggest how to
>>> configure??
>>
>> Multiple instances are multiple processes so there is no way for the
>> threads from two different processes to interact with each other.
>>
>> That's what we have process-separation at OS-level for ;-)
>>
>> The question is - what is your rsyslog version (and whether you're
>> hitting some well-known and already fixed bug). And what is your config.
>
> OK. From the config you posted I see that you do _not_ have multiple
> rsyslog instances. You have multiple inputs defined within a single
> instance. There's nothing unusual with that. I have several setups with
> rsyslog listening on multiple ports or multiple IPs and nothing bad happens.
>
> You didn't post the global /etc/rsyslogd.conf but I assume it's pretty
> typical package-supplied config with the include directive at the end so
> all the logic happens in those files you posted.
>
> Anyway, what I can advise from experience, since you have many inputs,
> many outputs and a bit of magic routing those inputs to outputs - since
> you're separating the inputs by port, it's more convenient (and allows
> you to set up separate queues to process each such pipeleine) to bind
> each output with a specific ruleset processing just this type of events.
>
> At the moment you're putting all events into main processing queue and
> then you're calling various filters to match specific event types to
> specific actions. That's a bit ineffective. And separating events into
> dedicated queues gives you better accounting.
>
> So instead of defining all inputs as "generic" ones and then calling
> your filters like this:
>
> ## Input-to-Output Flows
> #
> # Process incoming events from UDP 514
> if (re_match($inputname, "udp514"))
>   then {
>     call syslog.qradar
>     call qradar.local
>     stop
> }
>
> You can simply do
>
> ruleset (name="process_qradar [... queue parameters ...]) {
>    call syslog.qradar
>    call qradar.local
>    stop
> }
>
> input(type="imudp" port="514" name="udp514" ruleset="process_qradar")
>
> You could also do single input and dynamically assign ruleset based on
> the source IP but that's a completely different story.
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

it would be good to find the root cause. The best would be to use the
current 8.2206.0 version and see if it works. If so, all is fine. If
not, we should try to debug the issue.

Rainer

El jue, 16 jun 2022 a las 16:22, vijay kumar via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Hi Team,
>
> My rsyslog service is getting restarted very frequently and we understand
> it is due to race between the various threads, which causes one thread to
> free a message field while another tries to read/write it.
>
> log:
> ===
> ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address
> 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp 0x7f6079d74d88
> READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal)
> #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc)
> #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288
> #2 0x7f607d5066a5 in readjournal imjournal.c:497
> #3 0x55ae0523366e in thrdStarter ../threads.c:243
> #4 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>
> 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region
> [0x61e000000080,0x61e000000ac2)
> allocated by thread T3 (in:imjournal) here:
> #0 0x7f6085afcfe8 in __interceptor_realloc (/lib64/libasan.so.5+0xeffe8)
> #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b)
>
> Thread T3 (in:imjournal) created by T0 here:
> #0 0x7f6085a5fea3 in __interceptor_pthread_create
> (/lib64/libasan.so.5+0x52ea3)
> #1 0x55ae05234470 in thrdCreate ../threads.c:289
>
> SUMMARY: AddressSanitizer: heap-buffer-overflow
> (/lib64/libasan.so.5+0xb92fc)
> Shadow bytes around the buggy address:
> 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa
> 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Container overflow: fc
> Array cookie: ac
> Intra object redzone: bb
> ASan internal: fe
> Left alloca redzone: ca
> Right alloca redzone: cb
> =================================================================
> ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp 0x7f60706215f8
> READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg)
> #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26)
> #1 0x55ae0516d061 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x390061)
> #2 0x55ae0516e522 in MsgDup msg.c:1129
> #3 0x55ae05205d58 in execCall ruleset.c:290
> #4 0x55ae05205d58 in scriptExec ruleset.c:608
> #5 0x55ae05206532 in execIf ruleset.c:313
> #6 0x55ae05206532 in scriptExec ruleset.c:614
> #7 0x55ae05208868 in processBatch ruleset.c:660
> #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> #9 0x55ae051f100d in ConsumerReg queue.c:2145
> #10 0x55ae051e0804 in wtiWorker wti.c:428
> #11 0x55ae051d9dd5 in wtpWorker wtp.c:435
> #12 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>
> 0x60c0005d85c0 is located 0 bytes inside of 128-byte region
> [0x60c0005d85c0,0x60c0005d8640)
> freed by thread T6 (imudp(w1)) here:
> #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
> #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
>
> previously allocated by thread T6 (imudp(w1)) here:
> #0 0x7f6085afcba8 in __interceptor_malloc (/lib64/libasan.so.5+0xefba8)
> #1 0x55ae0516cff4 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x38fff4)
>
> Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here:
> #0 0x7f6085a5fea3 in __interceptor_pthread_create
> (/lib64/libasan.so.5+0x52ea3)
> #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
>
> Thread T6 (imudp(w1)) created by T0 here:
> #0 0x7f6085a5fea3 in __interceptor_pthread_create
> (/lib64/libasan.so.5+0x52ea3)
> #1 0x55ae05234470 in thrdCreate ../threads.c:289
>
> SUMMARY: AddressSanitizer: heap-use-after-free (/lib64/libasan.so.5+0x40b26)
> Shadow bytes around the buggy address:
> 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
> 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
> 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Container overflow: fc
> Array cookie: ac
> Intra object redzone: bb
> ASan internal: fe
> Left alloca redzone: ca
> Right alloca redzone: cb
> =================================================================
> ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp 0x7f6066965990
> WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local)
> #0 0x55ae0520a722 in propDestruct prop.c:63
> #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457
> #2 0x55ae05170299 in resolveDNS msg.c:522
> #3 0x55ae0517064a in getRcvFromIP msg.c:558
> #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317)
> #5 0x55ae0524aece in tplToString ../template.c:207
> #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114
> #7 0x55ae0522a4ab in processMsgMain ../action.c:1648
> #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825
> #9 0x55ae05205826 in execAct ruleset.c:209
> #10 0x55ae05205826 in scriptExec ruleset.c:599
> #11 0x55ae05208868 in processBatch ruleset.c:660
> #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> #13 0x55ae051f100d in ConsumerReg queue.c:2145
> #14 0x55ae051e0804 in wtiWorker wti.c:428
> #15 0x55ae051d9dd5 in wtpWorker wtp.c:435
> #16 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>
> 0x6040000285a0 is located 16 bytes inside of 48-byte region
> [0x604000028590,0x6040000285c0)
> freed by thread T16 (rs:qradar.local) here:
> #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
> #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
>
> previously allocated by thread T16 (rs:qradar.local) here:
> #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0)
> #1 0x55ae0520a439 in propConstruct prop.c:56
>
> Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here:
> #0 0x7f6085a5fea3 in __interceptor_pthread_create
> (/lib64/libasan.so.5+0x52ea3)
> #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
>
> SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in propDestruct
> Shadow bytes around the buggy address:
> 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd
> 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa
> 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa
> 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
> 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
> Shadow byte legend (one shadow byte represents 8 application bytes):
> Addressable: 00
> Partially addressable: 01 02 03 04 05 06 07
> Heap left redzone: fa
> Freed heap region: fd
> Stack left redzone: f1
> Stack mid redzone: f2
> Stack right redzone: f3
> Stack after return: f5
> Stack use after scope: f8
> Global redzone: f9
> Global init order: f6
> Poisoned by user: f7
> Container overflow: fc
> Array cookie: ac
> Intra object redzone: bb
> ASan internal: fe
> Left alloca redzone: ca
> Right alloca redzone: cb
> ==3035157==ABORTING
>
> =================================
> what is the possible solution ???
>
> would be to have multiple rsyslog instances, which is possible if the
> traffic is split between ports. If yes could you please suggest how to
> configure??
>
> Thanks & Regards
> Vijay Kumar Kanukula
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

HI Rainer/David/Marisuz,

Could you please help me with creating one input rule with a queue as
Marisuz suggested. I was failing to create a rule.

If I change anything in my 90-inputs.conf file do I need to make any
changes in 30-rules.conf???

@Rainer : I upgraded to the latest version of rsyslog and tested. But still
my rsyslog restarts continuously. It was not stable.

Thanks & Regards
Vijay Kumar Kanukula





On Fri, 17 Jun 2022 at 12:55, Rainer Gerhards <rgerhards@hq.adiscon.com>
wrote:

> it would be good to find the root cause. The best would be to use the
> current 8.2206.0 version and see if it works. If so, all is fine. If
> not, we should try to debug the issue.
>
> Rainer
>
> El jue, 16 jun 2022 a las 16:22, vijay kumar via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
> >
> > Hi Team,
> >
> > My rsyslog service is getting restarted very frequently and we understand
> > it is due to race between the various threads, which causes one thread to
> > free a message field while another tries to read/write it.
> >
> > log:
> > ===
> > ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address
> > 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp 0x7f6079d74d88
> > READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal)
> > #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc)
> > #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288
> > #2 0x7f607d5066a5 in readjournal imjournal.c:497
> > #3 0x55ae0523366e in thrdStarter ../threads.c:243
> > #4 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> > #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >
> > 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region
> > [0x61e000000080,0x61e000000ac2)
> > allocated by thread T3 (in:imjournal) here:
> > #0 0x7f6085afcfe8 in __interceptor_realloc
> (/lib64/libasan.so.5+0xeffe8)
> > #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b)
> >
> > Thread T3 (in:imjournal) created by T0 here:
> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> > (/lib64/libasan.so.5+0x52ea3)
> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
> >
> > SUMMARY: AddressSanitizer: heap-buffer-overflow
> > (/lib64/libasan.so.5+0xb92fc)
> > Shadow bytes around the buggy address:
> > 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa
> > 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > Shadow byte legend (one shadow byte represents 8 application bytes):
> > Addressable: 00
> > Partially addressable: 01 02 03 04 05 06 07
> > Heap left redzone: fa
> > Freed heap region: fd
> > Stack left redzone: f1
> > Stack mid redzone: f2
> > Stack right redzone: f3
> > Stack after return: f5
> > Stack use after scope: f8
> > Global redzone: f9
> > Global init order: f6
> > Poisoned by user: f7
> > Container overflow: fc
> > Array cookie: ac
> > Intra object redzone: bb
> > ASan internal: fe
> > Left alloca redzone: ca
> > Right alloca redzone: cb
> > =================================================================
> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> > 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp 0x7f60706215f8
> > READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg)
> > #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26)
> > #1 0x55ae0516d061 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x390061)
> > #2 0x55ae0516e522 in MsgDup msg.c:1129
> > #3 0x55ae05205d58 in execCall ruleset.c:290
> > #4 0x55ae05205d58 in scriptExec ruleset.c:608
> > #5 0x55ae05206532 in execIf ruleset.c:313
> > #6 0x55ae05206532 in scriptExec ruleset.c:614
> > #7 0x55ae05208868 in processBatch ruleset.c:660
> > #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> > #9 0x55ae051f100d in ConsumerReg queue.c:2145
> > #10 0x55ae051e0804 in wtiWorker wti.c:428
> > #11 0x55ae051d9dd5 in wtpWorker wtp.c:435
> > #12 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> > #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >
> > 0x60c0005d85c0 is located 0 bytes inside of 128-byte region
> > [0x60c0005d85c0,0x60c0005d8640)
> > freed by thread T6 (imudp(w1)) here:
> > #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
> >
> > previously allocated by thread T6 (imudp(w1)) here:
> > #0 0x7f6085afcba8 in __interceptor_malloc
> (/lib64/libasan.so.5+0xefba8)
> > #1 0x55ae0516cff4 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x38fff4)
> >
> > Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here:
> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> > (/lib64/libasan.so.5+0x52ea3)
> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
> >
> > Thread T6 (imudp(w1)) created by T0 here:
> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> > (/lib64/libasan.so.5+0x52ea3)
> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
> >
> > SUMMARY: AddressSanitizer: heap-use-after-free
> (/lib64/libasan.so.5+0x40b26)
> > Shadow bytes around the buggy address:
> > 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> > 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> > 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
> > 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> > 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> > =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
> > 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> > 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> > 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> > Shadow byte legend (one shadow byte represents 8 application bytes):
> > Addressable: 00
> > Partially addressable: 01 02 03 04 05 06 07
> > Heap left redzone: fa
> > Freed heap region: fd
> > Stack left redzone: f1
> > Stack mid redzone: f2
> > Stack right redzone: f3
> > Stack after return: f5
> > Stack use after scope: f8
> > Global redzone: f9
> > Global init order: f6
> > Poisoned by user: f7
> > Container overflow: fc
> > Array cookie: ac
> > Intra object redzone: bb
> > ASan internal: fe
> > Left alloca redzone: ca
> > Right alloca redzone: cb
> > =================================================================
> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> > 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp 0x7f6066965990
> > WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local)
> > #0 0x55ae0520a722 in propDestruct prop.c:63
> > #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457
> > #2 0x55ae05170299 in resolveDNS msg.c:522
> > #3 0x55ae0517064a in getRcvFromIP msg.c:558
> > #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317)
> > #5 0x55ae0524aece in tplToString ../template.c:207
> > #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114
> > #7 0x55ae0522a4ab in processMsgMain ../action.c:1648
> > #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825
> > #9 0x55ae05205826 in execAct ruleset.c:209
> > #10 0x55ae05205826 in scriptExec ruleset.c:599
> > #11 0x55ae05208868 in processBatch ruleset.c:660
> > #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> > #13 0x55ae051f100d in ConsumerReg queue.c:2145
> > #14 0x55ae051e0804 in wtiWorker wti.c:428
> > #15 0x55ae051d9dd5 in wtpWorker wtp.c:435
> > #16 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> > #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >
> > 0x6040000285a0 is located 16 bytes inside of 48-byte region
> > [0x604000028590,0x6040000285c0)
> > freed by thread T16 (rs:qradar.local) here:
> > #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
> >
> > previously allocated by thread T16 (rs:qradar.local) here:
> > #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0)
> > #1 0x55ae0520a439 in propConstruct prop.c:56
> >
> > Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here:
> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> > (/lib64/libasan.so.5+0x52ea3)
> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
> >
> > SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in propDestruct
> > Shadow bytes around the buggy address:
> > 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> > 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> > 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd
> > 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa
> > 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> > =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa
> > 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> > 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
> > 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> > 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> > 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
> > Shadow byte legend (one shadow byte represents 8 application bytes):
> > Addressable: 00
> > Partially addressable: 01 02 03 04 05 06 07
> > Heap left redzone: fa
> > Freed heap region: fd
> > Stack left redzone: f1
> > Stack mid redzone: f2
> > Stack right redzone: f3
> > Stack after return: f5
> > Stack use after scope: f8
> > Global redzone: f9
> > Global init order: f6
> > Poisoned by user: f7
> > Container overflow: fc
> > Array cookie: ac
> > Intra object redzone: bb
> > ASan internal: fe
> > Left alloca redzone: ca
> > Right alloca redzone: cb
> > ==3035157==ABORTING
> >
> > =================================
> > what is the possible solution ???
> >
> > would be to have multiple rsyslog instances, which is possible if the
> > traffic is split between ports. If yes could you please suggest how to
> > configure??
> >
> > Thanks & Regards
> > Vijay Kumar Kanukula
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

can you please post rsyslog -v output as well as the current ASAN report?

Thanks,
Rainer

El vie, 17 jun 2022 a las 10:12, vijay kumar
(<vijay.kanukula@gmail.com>) escribió:
>
> HI Rainer/David/Marisuz,
>
> Could you please help me with creating one input rule with a queue as Marisuz suggested. I was failing to create a rule.
>
> If I change anything in my 90-inputs.conf file do I need to make any changes in 30-rules.conf???
>
> @Rainer : I upgraded to the latest version of rsyslog and tested. But still my rsyslog restarts continuously. It was not stable.
>
> Thanks & Regards
> Vijay Kumar Kanukula
>
>
>
>
>
> On Fri, 17 Jun 2022 at 12:55, Rainer Gerhards <rgerhards@hq.adiscon.com> wrote:
>>
>> it would be good to find the root cause. The best would be to use the
>> current 8.2206.0 version and see if it works. If so, all is fine. If
>> not, we should try to debug the issue.
>>
>> Rainer
>>
>> El jue, 16 jun 2022 a las 16:22, vijay kumar via rsyslog
>> (<rsyslog@lists.adiscon.com>) escribió:
>> >
>> > Hi Team,
>> >
>> > My rsyslog service is getting restarted very frequently and we understand
>> > it is due to race between the various threads, which causes one thread to
>> > free a message field while another tries to read/write it.
>> >
>> > log:
>> > ===
>> > ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address
>> > 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp 0x7f6079d74d88
>> > READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal)
>> > #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc)
>> > #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288
>> > #2 0x7f607d5066a5 in readjournal imjournal.c:497
>> > #3 0x55ae0523366e in thrdStarter ../threads.c:243
>> > #4 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
>> > #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>> >
>> > 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region
>> > [0x61e000000080,0x61e000000ac2)
>> > allocated by thread T3 (in:imjournal) here:
>> > #0 0x7f6085afcfe8 in __interceptor_realloc (/lib64/libasan.so.5+0xeffe8)
>> > #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b)
>> >
>> > Thread T3 (in:imjournal) created by T0 here:
>> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>> > (/lib64/libasan.so.5+0x52ea3)
>> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
>> >
>> > SUMMARY: AddressSanitizer: heap-buffer-overflow
>> > (/lib64/libasan.so.5+0xb92fc)
>> > Shadow bytes around the buggy address:
>> > 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> > 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> > 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> > 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> > 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> > =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa
>> > 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> > 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> > 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> > 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> > 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> > Shadow byte legend (one shadow byte represents 8 application bytes):
>> > Addressable: 00
>> > Partially addressable: 01 02 03 04 05 06 07
>> > Heap left redzone: fa
>> > Freed heap region: fd
>> > Stack left redzone: f1
>> > Stack mid redzone: f2
>> > Stack right redzone: f3
>> > Stack after return: f5
>> > Stack use after scope: f8
>> > Global redzone: f9
>> > Global init order: f6
>> > Poisoned by user: f7
>> > Container overflow: fc
>> > Array cookie: ac
>> > Intra object redzone: bb
>> > ASan internal: fe
>> > Left alloca redzone: ca
>> > Right alloca redzone: cb
>> > =================================================================
>> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
>> > 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp 0x7f60706215f8
>> > READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg)
>> > #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26)
>> > #1 0x55ae0516d061 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x390061)
>> > #2 0x55ae0516e522 in MsgDup msg.c:1129
>> > #3 0x55ae05205d58 in execCall ruleset.c:290
>> > #4 0x55ae05205d58 in scriptExec ruleset.c:608
>> > #5 0x55ae05206532 in execIf ruleset.c:313
>> > #6 0x55ae05206532 in scriptExec ruleset.c:614
>> > #7 0x55ae05208868 in processBatch ruleset.c:660
>> > #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694
>> > #9 0x55ae051f100d in ConsumerReg queue.c:2145
>> > #10 0x55ae051e0804 in wtiWorker wti.c:428
>> > #11 0x55ae051d9dd5 in wtpWorker wtp.c:435
>> > #12 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
>> > #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>> >
>> > 0x60c0005d85c0 is located 0 bytes inside of 128-byte region
>> > [0x60c0005d85c0,0x60c0005d8640)
>> > freed by thread T6 (imudp(w1)) here:
>> > #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
>> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
>> >
>> > previously allocated by thread T6 (imudp(w1)) here:
>> > #0 0x7f6085afcba8 in __interceptor_malloc (/lib64/libasan.so.5+0xefba8)
>> > #1 0x55ae0516cff4 in msgSetFromSockinfo (/usr/sbin/rsyslogd+0x38fff4)
>> >
>> > Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here:
>> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>> > (/lib64/libasan.so.5+0x52ea3)
>> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
>> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
>> >
>> > Thread T6 (imudp(w1)) created by T0 here:
>> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>> > (/lib64/libasan.so.5+0x52ea3)
>> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
>> >
>> > SUMMARY: AddressSanitizer: heap-use-after-free (/lib64/libasan.so.5+0x40b26)
>> > Shadow bytes around the buggy address:
>> > 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>> > 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>> > 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
>> > 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>> > 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>> > =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
>> > 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>> > 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>> > 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> > 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> > 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>> > Shadow byte legend (one shadow byte represents 8 application bytes):
>> > Addressable: 00
>> > Partially addressable: 01 02 03 04 05 06 07
>> > Heap left redzone: fa
>> > Freed heap region: fd
>> > Stack left redzone: f1
>> > Stack mid redzone: f2
>> > Stack right redzone: f3
>> > Stack after return: f5
>> > Stack use after scope: f8
>> > Global redzone: f9
>> > Global init order: f6
>> > Poisoned by user: f7
>> > Container overflow: fc
>> > Array cookie: ac
>> > Intra object redzone: bb
>> > ASan internal: fe
>> > Left alloca redzone: ca
>> > Right alloca redzone: cb
>> > =================================================================
>> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
>> > 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp 0x7f6066965990
>> > WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local)
>> > #0 0x55ae0520a722 in propDestruct prop.c:63
>> > #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457
>> > #2 0x55ae05170299 in resolveDNS msg.c:522
>> > #3 0x55ae0517064a in getRcvFromIP msg.c:558
>> > #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317)
>> > #5 0x55ae0524aece in tplToString ../template.c:207
>> > #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114
>> > #7 0x55ae0522a4ab in processMsgMain ../action.c:1648
>> > #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825
>> > #9 0x55ae05205826 in execAct ruleset.c:209
>> > #10 0x55ae05205826 in scriptExec ruleset.c:599
>> > #11 0x55ae05208868 in processBatch ruleset.c:660
>> > #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694
>> > #13 0x55ae051f100d in ConsumerReg queue.c:2145
>> > #14 0x55ae051e0804 in wtiWorker wti.c:428
>> > #15 0x55ae051d9dd5 in wtpWorker wtp.c:435
>> > #16 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
>> > #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>> >
>> > 0x6040000285a0 is located 16 bytes inside of 48-byte region
>> > [0x604000028590,0x6040000285c0)
>> > freed by thread T16 (rs:qradar.local) here:
>> > #0 0x7f6085afc7e0 in __interceptor_free (/lib64/libasan.so.5+0xef7e0)
>> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
>> >
>> > previously allocated by thread T16 (rs:qradar.local) here:
>> > #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0)
>> > #1 0x55ae0520a439 in propConstruct prop.c:56
>> >
>> > Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here:
>> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>> > (/lib64/libasan.so.5+0x52ea3)
>> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
>> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
>> >
>> > SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in propDestruct
>> > Shadow bytes around the buggy address:
>> > 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> > 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
>> > 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd
>> > 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa
>> > 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
>> > =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa
>> > 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>> > 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
>> > 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
>> > 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
>> > 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
>> > Shadow byte legend (one shadow byte represents 8 application bytes):
>> > Addressable: 00
>> > Partially addressable: 01 02 03 04 05 06 07
>> > Heap left redzone: fa
>> > Freed heap region: fd
>> > Stack left redzone: f1
>> > Stack mid redzone: f2
>> > Stack right redzone: f3
>> > Stack after return: f5
>> > Stack use after scope: f8
>> > Global redzone: f9
>> > Global init order: f6
>> > Poisoned by user: f7
>> > Container overflow: fc
>> > Array cookie: ac
>> > Intra object redzone: bb
>> > ASan internal: fe
>> > Left alloca redzone: ca
>> > Right alloca redzone: cb
>> > ==3035157==ABORTING
>> >
>> > =================================
>> > what is the possible solution ???
>> >
>> > would be to have multiple rsyslog instances, which is possible if the
>> > traffic is split between ports. If yes could you please suggest how to
>> > configure??
>> >
>> > Thanks & Regards
>> > Vijay Kumar Kanukula
>> > _______________________________________________
>> > rsyslog mailing list
>> > https://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hello Rainer,

When we installed ASAN related RPMS to capture the logs rsyslog restart for
every 2 to 3 mins, it was purely unstable so we downgraded immediately.
Attaching ASAN logs captured day before yesterday.

Thanks & Regards
Vijay Kumar Kanukula

On Fri, 17 Jun 2022 at 13:44, Rainer Gerhards <rgerhards@hq.adiscon.com>
wrote:

> can you please post rsyslog -v output as well as the current ASAN report?
>
> Thanks,
> Rainer
>
> El vie, 17 jun 2022 a las 10:12, vijay kumar
> (<vijay.kanukula@gmail.com>) escribió:
> >
> > HI Rainer/David/Marisuz,
> >
> > Could you please help me with creating one input rule with a queue as
> Marisuz suggested. I was failing to create a rule.
> >
> > If I change anything in my 90-inputs.conf file do I need to make any
> changes in 30-rules.conf???
> >
> > @Rainer : I upgraded to the latest version of rsyslog and tested. But
> still my rsyslog restarts continuously. It was not stable.
> >
> > Thanks & Regards
> > Vijay Kumar Kanukula
> >
> >
> >
> >
> >
> > On Fri, 17 Jun 2022 at 12:55, Rainer Gerhards <rgerhards@hq.adiscon.com>
> wrote:
> >>
> >> it would be good to find the root cause. The best would be to use the
> >> current 8.2206.0 version and see if it works. If so, all is fine. If
> >> not, we should try to debug the issue.
> >>
> >> Rainer
> >>
> >> El jue, 16 jun 2022 a las 16:22, vijay kumar via rsyslog
> >> (<rsyslog@lists.adiscon.com>) escribió:
> >> >
> >> > Hi Team,
> >> >
> >> > My rsyslog service is getting restarted very frequently and we
> understand
> >> > it is due to race between the various threads, which causes one
> thread to
> >> > free a message field while another tries to read/write it.
> >> >
> >> > log:
> >> > ===
> >> > ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address
> >> > 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp
> 0x7f6079d74d88
> >> > READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal)
> >> > #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc)
> >> > #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288
> >> > #2 0x7f607d5066a5 in readjournal imjournal.c:497
> >> > #3 0x55ae0523366e in thrdStarter ../threads.c:243
> >> > #4 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> >> > #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >> >
> >> > 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region
> >> > [.0x61e000000080,0x61e000000ac2)
> >> > allocated by thread T3 (in:imjournal) here:
> >> > #0 0x7f6085afcfe8 in __interceptor_realloc
> (/lib64/libasan.so.5+0xeffe8)
> >> > #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b)
> >> >
> >> > Thread T3 (in:imjournal) created by T0 here:
> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> >> > (/lib64/libasan.so.5+0x52ea3)
> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
> >> >
> >> > SUMMARY: AddressSanitizer: heap-buffer-overflow
> >> > (/lib64/libasan.so.5+0xb92fc)
> >> > Shadow bytes around the buggy address:
> >> > 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> >> > 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> >> > 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> >> > 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> >> > 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> >> > =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa
> >> > 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> > 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> > 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> > 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> > 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> > Shadow byte legend (one shadow byte represents 8 application bytes):
> >> > Addressable: 00
> >> > Partially addressable: 01 02 03 04 05 06 07
> >> > Heap left redzone: fa
> >> > Freed heap region: fd
> >> > Stack left redzone: f1
> >> > Stack mid redzone: f2
> >> > Stack right redzone: f3
> >> > Stack after return: f5
> >> > Stack use after scope: f8
> >> > Global redzone: f9
> >> > Global init order: f6
> >> > Poisoned by user: f7
> >> > Container overflow: fc
> >> > Array cookie: ac
> >> > Intra object redzone: bb
> >> > ASan internal: fe
> >> > Left alloca redzone: ca
> >> > Right alloca redzone: cb
> >> > =================================================================
> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> >> > 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp
> 0x7f60706215f8
> >> > READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg)
> >> > #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26)
> >> > #1 0x55ae0516d061 in msgSetFromSockinfo
> (/usr/sbin/rsyslogd+0x390061)
> >> > #2 0x55ae0516e522 in MsgDup msg.c:1129
> >> > #3 0x55ae05205d58 in execCall ruleset.c:290
> >> > #4 0x55ae05205d58 in scriptExec ruleset.c:608
> >> > #5 0x55ae05206532 in execIf ruleset.c:313
> >> > #6 0x55ae05206532 in scriptExec ruleset.c:614
> >> > #7 0x55ae05208868 in processBatch ruleset.c:660
> >> > #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> >> > #9 0x55ae051f100d in ConsumerReg queue.c:2145
> >> > #10 0x55ae051e0804 in wtiWorker wti.c:428
> >> > #11 0x55ae051d9dd5 in wtpWorker wtp.c:435
> >> > #12 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> >> > #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >> >
> >> > 0x60c0005d85c0 is located 0 bytes inside of 128-byte region
> >> > [.0x60c0005d85c0,0x60c0005d8640)
> >> > freed by thread T6 (imudp(w1)) here:
> >> > #0 0x7f6085afc7e0 in __interceptor_free
> (/lib64/libasan.so.5+0xef7e0)
> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
> >> >
> >> > previously allocated by thread T6 (imudp(w1)) here:
> >> > #0 0x7f6085afcba8 in __interceptor_malloc
> (/lib64/libasan.so.5+0xefba8)
> >> > #1 0x55ae0516cff4 in msgSetFromSockinfo
> (/usr/sbin/rsyslogd+0x38fff4)
> >> >
> >> > Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here:
> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> >> > (/lib64/libasan.so.5+0x52ea3)
> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
> >> >
> >> > Thread T6 (imudp(w1)) created by T0 here:
> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> >> > (/lib64/libasan.so.5+0x52ea3)
> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
> >> >
> >> > SUMMARY: AddressSanitizer: heap-use-after-free
> (/lib64/libasan.so.5+0x40b26)
> >> > Shadow bytes around the buggy address:
> >> > 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> >> > 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> >> > 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
> >> > 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> >> > 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> >> > =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
> >> > 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> >> > 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> >> > 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> > 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> > 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> >> > Shadow byte legend (one shadow byte represents 8 application bytes):
> >> > Addressable: 00
> >> > Partially addressable: 01 02 03 04 05 06 07
> >> > Heap left redzone: fa
> >> > Freed heap region: fd
> >> > Stack left redzone: f1
> >> > Stack mid redzone: f2
> >> > Stack right redzone: f3
> >> > Stack after return: f5
> >> > Stack use after scope: f8
> >> > Global redzone: f9
> >> > Global init order: f6
> >> > Poisoned by user: f7
> >> > Container overflow: fc
> >> > Array cookie: ac
> >> > Intra object redzone: bb
> >> > ASan internal: fe
> >> > Left alloca redzone: ca
> >> > Right alloca redzone: cb
> >> > =================================================================
> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> >> > 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp
> 0x7f6066965990
> >> > WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local)
> >> > #0 0x55ae0520a722 in propDestruct prop.c:63
> >> > #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457
> >> > #2 0x55ae05170299 in resolveDNS msg.c:522
> >> > #3 0x55ae0517064a in getRcvFromIP msg.c:558
> >> > #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317)
> >> > #5 0x55ae0524aece in tplToString ../template.c:207
> >> > #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114
> >> > #7 0x55ae0522a4ab in processMsgMain ../action.c:1648
> >> > #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825
> >> > #9 0x55ae05205826 in execAct ruleset.c:209
> >> > #10 0x55ae05205826 in scriptExec ruleset.c:599
> >> > #11 0x55ae05208868 in processBatch ruleset.c:660
> >> > #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> >> > #13 0x55ae051f100d in ConsumerReg queue.c:2145
> >> > #14 0x55ae051e0804 in wtiWorker wti.c:428
> >> > #15 0x55ae051d9dd5 in wtpWorker wtp.c:435
> >> > #16 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
> >> > #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >> >
> >> > 0x6040000285a0 is located 16 bytes inside of 48-byte region
> >> > [.0x604000028590,0x6040000285c0)
> >> > freed by thread T16 (rs:qradar.local) here:
> >> > #0 0x7f6085afc7e0 in __interceptor_free
> (/lib64/libasan.so.5+0xef7e0)
> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
> >> >
> >> > previously allocated by thread T16 (rs:qradar.local) here:
> >> > #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0)
> >> > #1 0x55ae0520a439 in propConstruct prop.c:56
> >> >
> >> > Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here:
> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> >> > (/lib64/libasan.so.5+0x52ea3)
> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
> >> >
> >> > SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in
> propDestruct
> >> > Shadow bytes around the buggy address:
> >> > 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> > 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> >> > 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd
> >> > 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa
> >> > 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> >> > =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa
> >> > 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> >> > 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
> >> > 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> >> > 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> >> > 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
> >> > Shadow byte legend (one shadow byte represents 8 application bytes):
> >> > Addressable: 00
> >> > Partially addressable: 01 02 03 04 05 06 07
> >> > Heap left redzone: fa
> >> > Freed heap region: fd
> >> > Stack left redzone: f1
> >> > Stack mid redzone: f2
> >> > Stack right redzone: f3
> >> > Stack after return: f5
> >> > Stack use after scope: f8
> >> > Global redzone: f9
> >> > Global init order: f6
> >> > Poisoned by user: f7
> >> > Container overflow: fc
> >> > Array cookie: ac
> >> > Intra object redzone: bb
> >> > ASan internal: fe
> >> > Left alloca redzone: ca
> >> > Right alloca redzone: cb
> >> > ==3035157==ABORTING
> >> >
> >> > =================================
> >> > what is the possible solution ???
> >> >
> >> > would be to have multiple rsyslog instances, which is possible if the
> >> > traffic is split between ports. If yes could you please suggest how to
> >> > configure??
> >> >
> >> > Thanks & Regards
> >> > Vijay Kumar Kanukula
> >> > _______________________________________________
> >> > rsyslog mailing list
> >> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com/professional-services/
> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you DON'T LIKE THAT.
>

Hi Rainer,

Do you have any luck with asan logs???

Thanks & Regards
Vijay Kumar Kanukula

On Fri, 17 Jun 2022 at 14:11, vijay kumar <vijay.kanukula@gmail.com> wrote:

> Hello Rainer,
>
> When we installed ASAN related RPMS to capture the logs rsyslog restart
> for every 2 to 3 mins, it was purely unstable so we downgraded immediately.
> Attaching ASAN logs captured day before yesterday.
>
> Thanks & Regards
> Vijay Kumar Kanukula
>
> On Fri, 17 Jun 2022 at 13:44, Rainer Gerhards <rgerhards@hq.adiscon.com>
> wrote:
>
>> can you please post rsyslog -v output as well as the current ASAN report?
>>
>> Thanks,
>> Rainer
>>
>> El vie, 17 jun 2022 a las 10:12, vijay kumar
>> (<vijay.kanukula@gmail.com>) escribió:
>> >
>> > HI Rainer/David/Marisuz,
>> >
>> > Could you please help me with creating one input rule with a queue as
>> Marisuz suggested. I was failing to create a rule.
>> >
>> > If I change anything in my 90-inputs.conf file do I need to make any
>> changes in 30-rules.conf???
>> >
>> > @Rainer : I upgraded to the latest version of rsyslog and tested. But
>> still my rsyslog restarts continuously. It was not stable.
>> >
>> > Thanks & Regards
>> > Vijay Kumar Kanukula
>> >
>> >
>> >
>> >
>> >
>> > On Fri, 17 Jun 2022 at 12:55, Rainer Gerhards <rgerhards@hq.adiscon.com>
>> wrote:
>> >>
>> >> it would be good to find the root cause. The best would be to use the
>> >> current 8.2206.0 version and see if it works. If so, all is fine. If
>> >> not, we should try to debug the issue.
>> >>
>> >> Rainer
>> >>
>> >> El jue, 16 jun 2022 a las 16:22, vijay kumar via rsyslog
>> >> (<rsyslog@lists.adiscon.com>) escribió:
>> >> >
>> >> > Hi Team,
>> >> >
>> >> > My rsyslog service is getting restarted very frequently and we
>> understand
>> >> > it is due to race between the various threads, which causes one
>> thread to
>> >> > free a message field while another tries to read/write it.
>> >> >
>> >> > log:
>> >> > ===
>> >> > ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address
>> >> > 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp
>> 0x7f6079d74d88
>> >> > READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal)
>> >> > #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc)
>> >> > #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288
>> >> > #2 0x7f607d5066a5 in readjournal imjournal.c:497
>> >> > #3 0x55ae0523366e in thrdStarter ../threads.c:243
>> >> > #4 0x7f60855dd1ce in start_thread (/lib64/libpthread.so.0+0x81ce)
>> >> > #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>> >> >
>> >> > 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region
>> >> > [.0x61e000000080,0x61e000000ac2)
>> >> > allocated by thread T3 (in:imjournal) here:
>> >> > #0 0x7f6085afcfe8 in __interceptor_realloc
>> (/lib64/libasan.so.5+0xeffe8)
>> >> > #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b)
>> >> >
>> >> > Thread T3 (in:imjournal) created by T0 here:
>> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>> >> > (/lib64/libasan.so.5+0x52ea3)
>> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
>> >> >
>> >> > SUMMARY: AddressSanitizer: heap-buffer-overflow
>> >> > (/lib64/libasan.so.5+0xb92fc)
>> >> > Shadow bytes around the buggy address:
>> >> > 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> >> > 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> >> > 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> >> > 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> >> > 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> >> > =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa
>> >> > 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> >> > 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> >> > 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> >> > 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> >> > 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> >> > Shadow byte legend (one shadow byte represents 8 application bytes):
>> >> > Addressable: 00
>> >> > Partially addressable: 01 02 03 04 05 06 07
>> >> > Heap left redzone: fa
>> >> > Freed heap region: fd
>> >> > Stack left redzone: f1
>> >> > Stack mid redzone: f2
>> >> > Stack right redzone: f3
>> >> > Stack after return: f5
>> >> > Stack use after scope: f8
>> >> > Global redzone: f9
>> >> > Global init order: f6
>> >> > Poisoned by user: f7
>> >> > Container overflow: fc
>> >> > Array cookie: ac
>> >> > Intra object redzone: bb
>> >> > ASan internal: fe
>> >> > Left alloca redzone: ca
>> >> > Right alloca redzone: cb
>> >> > =================================================================
>> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
>> >> > 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp
>> 0x7f60706215f8
>> >> > READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg)
>> >> > #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26)
>> >> > #1 0x55ae0516d061 in msgSetFromSockinfo
>> (/usr/sbin/rsyslogd+0x390061)
>> >> > #2 0x55ae0516e522 in MsgDup msg.c:1129
>> >> > #3 0x55ae05205d58 in execCall ruleset.c:290
>> >> > #4 0x55ae05205d58 in scriptExec ruleset.c:608
>> >> > #5 0x55ae05206532 in execIf ruleset.c:313
>> >> > #6 0x55ae05206532 in scriptExec ruleset.c:614
>> >> > #7 0x55ae05208868 in processBatch ruleset.c:660
>> >> > #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694
>> >> > #9 0x55ae051f100d in ConsumerReg queue.c:2145
>> >> > #10 0x55ae051e0804 in wtiWorker wti.c:428
>> >> > #11 0x55ae051d9dd5 in wtpWorker wtp.c:435
>> >> > #12 0x7f60855dd1ce in start_thread
>> (/lib64/libpthread.so.0+0x81ce)
>> >> > #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>> >> >
>> >> > 0x60c0005d85c0 is located 0 bytes inside of 128-byte region
>> >> > [.0x60c0005d85c0,0x60c0005d8640)
>> >> > freed by thread T6 (imudp(w1)) here:
>> >> > #0 0x7f6085afc7e0 in __interceptor_free
>> (/lib64/libasan.so.5+0xef7e0)
>> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
>> >> >
>> >> > previously allocated by thread T6 (imudp(w1)) here:
>> >> > #0 0x7f6085afcba8 in __interceptor_malloc
>> (/lib64/libasan.so.5+0xefba8)
>> >> > #1 0x55ae0516cff4 in msgSetFromSockinfo
>> (/usr/sbin/rsyslogd+0x38fff4)
>> >> >
>> >> > Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here:
>> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>> >> > (/lib64/libasan.so.5+0x52ea3)
>> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
>> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
>> >> >
>> >> > Thread T6 (imudp(w1)) created by T0 here:
>> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>> >> > (/lib64/libasan.so.5+0x52ea3)
>> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
>> >> >
>> >> > SUMMARY: AddressSanitizer: heap-use-after-free
>> (/lib64/libasan.so.5+0x40b26)
>> >> > Shadow bytes around the buggy address:
>> >> > 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>> >> > 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>> >> > 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
>> >> > 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>> >> > 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>> >> > =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
>> >> > 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>> >> > 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>> >> > 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> >> > 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> >> > 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>> >> > Shadow byte legend (one shadow byte represents 8 application bytes):
>> >> > Addressable: 00
>> >> > Partially addressable: 01 02 03 04 05 06 07
>> >> > Heap left redzone: fa
>> >> > Freed heap region: fd
>> >> > Stack left redzone: f1
>> >> > Stack mid redzone: f2
>> >> > Stack right redzone: f3
>> >> > Stack after return: f5
>> >> > Stack use after scope: f8
>> >> > Global redzone: f9
>> >> > Global init order: f6
>> >> > Poisoned by user: f7
>> >> > Container overflow: fc
>> >> > Array cookie: ac
>> >> > Intra object redzone: bb
>> >> > ASan internal: fe
>> >> > Left alloca redzone: ca
>> >> > Right alloca redzone: cb
>> >> > =================================================================
>> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
>> >> > 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp
>> 0x7f6066965990
>> >> > WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local)
>> >> > #0 0x55ae0520a722 in propDestruct prop.c:63
>> >> > #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457
>> >> > #2 0x55ae05170299 in resolveDNS msg.c:522
>> >> > #3 0x55ae0517064a in getRcvFromIP msg.c:558
>> >> > #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317)
>> >> > #5 0x55ae0524aece in tplToString ../template.c:207
>> >> > #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114
>> >> > #7 0x55ae0522a4ab in processMsgMain ../action.c:1648
>> >> > #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825
>> >> > #9 0x55ae05205826 in execAct ruleset.c:209
>> >> > #10 0x55ae05205826 in scriptExec ruleset.c:599
>> >> > #11 0x55ae05208868 in processBatch ruleset.c:660
>> >> > #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694
>> >> > #13 0x55ae051f100d in ConsumerReg queue.c:2145
>> >> > #14 0x55ae051e0804 in wtiWorker wti.c:428
>> >> > #15 0x55ae051d9dd5 in wtpWorker wtp.c:435
>> >> > #16 0x7f60855dd1ce in start_thread
>> (/lib64/libpthread.so.0+0x81ce)
>> >> > #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>> >> >
>> >> > 0x6040000285a0 is located 16 bytes inside of 48-byte region
>> >> > [.0x604000028590,0x6040000285c0)
>> >> > freed by thread T16 (rs:qradar.local) here:
>> >> > #0 0x7f6085afc7e0 in __interceptor_free
>> (/lib64/libasan.so.5+0xef7e0)
>> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
>> >> >
>> >> > previously allocated by thread T16 (rs:qradar.local) here:
>> >> > #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0)
>> >> > #1 0x55ae0520a439 in propConstruct prop.c:56
>> >> >
>> >> > Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here:
>> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>> >> > (/lib64/libasan.so.5+0x52ea3)
>> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
>> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
>> >> >
>> >> > SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in
>> propDestruct
>> >> > Shadow bytes around the buggy address:
>> >> > 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>> >> > 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
>> >> > 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd
>> >> > 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa
>> >> > 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
>> >> > =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa
>> >> > 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>> >> > 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
>> >> > 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
>> >> > 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
>> >> > 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
>> >> > Shadow byte legend (one shadow byte represents 8 application bytes):
>> >> > Addressable: 00
>> >> > Partially addressable: 01 02 03 04 05 06 07
>> >> > Heap left redzone: fa
>> >> > Freed heap region: fd
>> >> > Stack left redzone: f1
>> >> > Stack mid redzone: f2
>> >> > Stack right redzone: f3
>> >> > Stack after return: f5
>> >> > Stack use after scope: f8
>> >> > Global redzone: f9
>> >> > Global init order: f6
>> >> > Poisoned by user: f7
>> >> > Container overflow: fc
>> >> > Array cookie: ac
>> >> > Intra object redzone: bb
>> >> > ASan internal: fe
>> >> > Left alloca redzone: ca
>> >> > Right alloca redzone: cb
>> >> > ==3035157==ABORTING
>> >> >
>> >> > =================================
>> >> > what is the possible solution ???
>> >> >
>> >> > would be to have multiple rsyslog instances, which is possible if
>> the
>> >> > traffic is split between ports. If yes could you please suggest how
>> to
>> >> > configure??
>> >> >
>> >> > Thanks & Regards
>> >> > Vijay Kumar Kanukula
>> >> > _______________________________________________
>> >> > rsyslog mailing list
>> >> > https://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> > http://www.rsyslog.com/professional-services/
>> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T LIKE THAT.
>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi,

sorry I did not get you, asan logs???


On Mon, Jun 20, 2022 at 1:24 PM vijay kumar via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Hi Rainer,
>
> Do you have any luck with asan logs???
>
> Thanks & Regards
> Vijay Kumar Kanukula
>
> On Fri, 17 Jun 2022 at 14:11, vijay kumar <vijay.kanukula@gmail.com>
> wrote:
>
> > Hello Rainer,
> >
> > When we installed ASAN related RPMS to capture the logs rsyslog restart
> > for every 2 to 3 mins, it was purely unstable so we downgraded
> immediately.
> > Attaching ASAN logs captured day before yesterday.
> >
> > Thanks & Regards
> > Vijay Kumar Kanukula
> >
> > On Fri, 17 Jun 2022 at 13:44, Rainer Gerhards <rgerhards@hq.adiscon.com>
> > wrote:
> >
> >> can you please post rsyslog -v output as well as the current ASAN
> report?
> >>
> >> Thanks,
> >> Rainer
> >>
> >> El vie, 17 jun 2022 a las 10:12, vijay kumar
> >> (<vijay.kanukula@gmail.com>) escribió:
> >> >
> >> > HI Rainer/David/Marisuz,
> >> >
> >> > Could you please help me with creating one input rule with a queue as
> >> Marisuz suggested. I was failing to create a rule.
> >> >
> >> > If I change anything in my 90-inputs.conf file do I need to make any
> >> changes in 30-rules.conf???
> >> >
> >> > @Rainer : I upgraded to the latest version of rsyslog and tested. But
> >> still my rsyslog restarts continuously. It was not stable.
> >> >
> >> > Thanks & Regards
> >> > Vijay Kumar Kanukula
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, 17 Jun 2022 at 12:55, Rainer Gerhards <
> rgerhards@hq.adiscon.com>
> >> wrote:
> >> >>
> >> >> it would be good to find the root cause. The best would be to use the
> >> >> current 8.2206.0 version and see if it works. If so, all is fine. If
> >> >> not, we should try to debug the issue.
> >> >>
> >> >> Rainer
> >> >>
> >> >> El jue, 16 jun 2022 a las 16:22, vijay kumar via rsyslog
> >> >> (<rsyslog@lists.adiscon.com>) escribió:
> >> >> >
> >> >> > Hi Team,
> >> >> >
> >> >> > My rsyslog service is getting restarted very frequently and we
> >> understand
> >> >> > it is due to race between the various threads, which causes one
> >> thread to
> >> >> > free a message field while another tries to read/write it.
> >> >> >
> >> >> > log:
> >> >> > ===
> >> >> > ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address
> >> >> > 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp
> >> 0x7f6079d74d88
> >> >> > READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal)
> >> >> > #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc)
> >> >> > #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288
> >> >> > #2 0x7f607d5066a5 in readjournal imjournal.c:497
> >> >> > #3 0x55ae0523366e in thrdStarter ../threads.c:243
> >> >> > #4 0x7f60855dd1ce in start_thread
> (/lib64/libpthread.so.0+0x81ce)
> >> >> > #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >> >> >
> >> >> > 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region
> >> >> > [.0x61e000000080,0x61e000000ac2)
> >> >> > allocated by thread T3 (in:imjournal) here:
> >> >> > #0 0x7f6085afcfe8 in __interceptor_realloc
> >> (/lib64/libasan.so.5+0xeffe8)
> >> >> > #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b)
> >> >> >
> >> >> > Thread T3 (in:imjournal) created by T0 here:
> >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> >> >> > (/lib64/libasan.so.5+0x52ea3)
> >> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
> >> >> >
> >> >> > SUMMARY: AddressSanitizer: heap-buffer-overflow
> >> >> > (/lib64/libasan.so.5+0xb92fc)
> >> >> > Shadow bytes around the buggy address:
> >> >> > 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> >> >> > 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> >> >> > 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> >> >> > 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> >> >> > 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> >> >> > =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa
> >> >> > 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> >> > 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> >> > 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> >> > 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> >> > 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> >> > Shadow byte legend (one shadow byte represents 8 application
> bytes):
> >> >> > Addressable: 00
> >> >> > Partially addressable: 01 02 03 04 05 06 07
> >> >> > Heap left redzone: fa
> >> >> > Freed heap region: fd
> >> >> > Stack left redzone: f1
> >> >> > Stack mid redzone: f2
> >> >> > Stack right redzone: f3
> >> >> > Stack after return: f5
> >> >> > Stack use after scope: f8
> >> >> > Global redzone: f9
> >> >> > Global init order: f6
> >> >> > Poisoned by user: f7
> >> >> > Container overflow: fc
> >> >> > Array cookie: ac
> >> >> > Intra object redzone: bb
> >> >> > ASan internal: fe
> >> >> > Left alloca redzone: ca
> >> >> > Right alloca redzone: cb
> >> >> > =================================================================
> >> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> >> >> > 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp
> >> 0x7f60706215f8
> >> >> > READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg)
> >> >> > #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26)
> >> >> > #1 0x55ae0516d061 in msgSetFromSockinfo
> >> (/usr/sbin/rsyslogd+0x390061)
> >> >> > #2 0x55ae0516e522 in MsgDup msg.c:1129
> >> >> > #3 0x55ae05205d58 in execCall ruleset.c:290
> >> >> > #4 0x55ae05205d58 in scriptExec ruleset.c:608
> >> >> > #5 0x55ae05206532 in execIf ruleset.c:313
> >> >> > #6 0x55ae05206532 in scriptExec ruleset.c:614
> >> >> > #7 0x55ae05208868 in processBatch ruleset.c:660
> >> >> > #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> >> >> > #9 0x55ae051f100d in ConsumerReg queue.c:2145
> >> >> > #10 0x55ae051e0804 in wtiWorker wti.c:428
> >> >> > #11 0x55ae051d9dd5 in wtpWorker wtp.c:435
> >> >> > #12 0x7f60855dd1ce in start_thread
> >> (/lib64/libpthread.so.0+0x81ce)
> >> >> > #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >> >> >
> >> >> > 0x60c0005d85c0 is located 0 bytes inside of 128-byte region
> >> >> > [.0x60c0005d85c0,0x60c0005d8640)
> >> >> > freed by thread T6 (imudp(w1)) here:
> >> >> > #0 0x7f6085afc7e0 in __interceptor_free
> >> (/lib64/libasan.so.5+0xef7e0)
> >> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
> >> >> >
> >> >> > previously allocated by thread T6 (imudp(w1)) here:
> >> >> > #0 0x7f6085afcba8 in __interceptor_malloc
> >> (/lib64/libasan.so.5+0xefba8)
> >> >> > #1 0x55ae0516cff4 in msgSetFromSockinfo
> >> (/usr/sbin/rsyslogd+0x38fff4)
> >> >> >
> >> >> > Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here:
> >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> >> >> > (/lib64/libasan.so.5+0x52ea3)
> >> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> >> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
> >> >> >
> >> >> > Thread T6 (imudp(w1)) created by T0 here:
> >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> >> >> > (/lib64/libasan.so.5+0x52ea3)
> >> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
> >> >> >
> >> >> > SUMMARY: AddressSanitizer: heap-use-after-free
> >> (/lib64/libasan.so.5+0x40b26)
> >> >> > Shadow bytes around the buggy address:
> >> >> > 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> >> >> > 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> >> >> > 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
> >> >> > 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> >> >> > 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> >> >> > =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
> >> >> > 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> >> >> > 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> >> >> > 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> >> > 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> >> > 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> >> >> > Shadow byte legend (one shadow byte represents 8 application
> bytes):
> >> >> > Addressable: 00
> >> >> > Partially addressable: 01 02 03 04 05 06 07
> >> >> > Heap left redzone: fa
> >> >> > Freed heap region: fd
> >> >> > Stack left redzone: f1
> >> >> > Stack mid redzone: f2
> >> >> > Stack right redzone: f3
> >> >> > Stack after return: f5
> >> >> > Stack use after scope: f8
> >> >> > Global redzone: f9
> >> >> > Global init order: f6
> >> >> > Poisoned by user: f7
> >> >> > Container overflow: fc
> >> >> > Array cookie: ac
> >> >> > Intra object redzone: bb
> >> >> > ASan internal: fe
> >> >> > Left alloca redzone: ca
> >> >> > Right alloca redzone: cb
> >> >> > =================================================================
> >> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
> >> >> > 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp
> >> 0x7f6066965990
> >> >> > WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local)
> >> >> > #0 0x55ae0520a722 in propDestruct prop.c:63
> >> >> > #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457
> >> >> > #2 0x55ae05170299 in resolveDNS msg.c:522
> >> >> > #3 0x55ae0517064a in getRcvFromIP msg.c:558
> >> >> > #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317)
> >> >> > #5 0x55ae0524aece in tplToString ../template.c:207
> >> >> > #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114
> >> >> > #7 0x55ae0522a4ab in processMsgMain ../action.c:1648
> >> >> > #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825
> >> >> > #9 0x55ae05205826 in execAct ruleset.c:209
> >> >> > #10 0x55ae05205826 in scriptExec ruleset.c:599
> >> >> > #11 0x55ae05208868 in processBatch ruleset.c:660
> >> >> > #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694
> >> >> > #13 0x55ae051f100d in ConsumerReg queue.c:2145
> >> >> > #14 0x55ae051e0804 in wtiWorker wti.c:428
> >> >> > #15 0x55ae051d9dd5 in wtpWorker wtp.c:435
> >> >> > #16 0x7f60855dd1ce in start_thread
> >> (/lib64/libpthread.so.0+0x81ce)
> >> >> > #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
> >> >> >
> >> >> > 0x6040000285a0 is located 16 bytes inside of 48-byte region
> >> >> > [.0x604000028590,0x6040000285c0)
> >> >> > freed by thread T16 (rs:qradar.local) here:
> >> >> > #0 0x7f6085afc7e0 in __interceptor_free
> >> (/lib64/libasan.so.5+0xef7e0)
> >> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
> >> >> >
> >> >> > previously allocated by thread T16 (rs:qradar.local) here:
> >> >> > #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0)
> >> >> > #1 0x55ae0520a439 in propConstruct prop.c:56
> >> >> >
> >> >> > Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here:
> >> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
> >> >> > (/lib64/libasan.so.5+0x52ea3)
> >> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
> >> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
> >> >> >
> >> >> > SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in
> >> propDestruct
> >> >> > Shadow bytes around the buggy address:
> >> >> > 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> >> >> > 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> >> >> > 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd
> >> >> > 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa
> >> >> > 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
> >> >> > =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa
> >> >> > 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
> >> >> > 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
> >> >> > 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> >> >> > 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
> >> >> > 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
> >> >> > Shadow byte legend (one shadow byte represents 8 application
> bytes):
> >> >> > Addressable: 00
> >> >> > Partially addressable: 01 02 03 04 05 06 07
> >> >> > Heap left redzone: fa
> >> >> > Freed heap region: fd
> >> >> > Stack left redzone: f1
> >> >> > Stack mid redzone: f2
> >> >> > Stack right redzone: f3
> >> >> > Stack after return: f5
> >> >> > Stack use after scope: f8
> >> >> > Global redzone: f9
> >> >> > Global init order: f6
> >> >> > Poisoned by user: f7
> >> >> > Container overflow: fc
> >> >> > Array cookie: ac
> >> >> > Intra object redzone: bb
> >> >> > ASan internal: fe
> >> >> > Left alloca redzone: ca
> >> >> > Right alloca redzone: cb
> >> >> > ==3035157==ABORTING
> >> >> >
> >> >> > =================================
> >> >> > what is the possible solution ???
> >> >> >
> >> >> > would be to have multiple rsyslog instances, which is possible if
> >> the
> >> >> > traffic is split between ports. If yes could you please suggest how
> >> to
> >> >> > configure??
> >> >> >
> >> >> > Thanks & Regards
> >> >> > Vijay Kumar Kanukula
> >> >> > _______________________________________________
> >> >> > rsyslog mailing list
> >> >> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >> > http://www.rsyslog.com/professional-services/
> >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if
> >> you DON'T LIKE THAT.
> >>
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi All,

still my rsyslog not stable, it is restarting with below information. Could
you please help us.

Jun 29 11:37:47 127.0.0.1 kernel: imudp(w0)[1327844]: segfault at
27f6584009f40 ip 00007f65a85f2cf7 sp 00007f65a2fc91a0 error 4 in
libc-2.28.so[7f65a8558000+1bc000]
Jun 29 11:37:47 127.0.0.1 kernel: Code: 8b 40 08 48 83 f8 0f 0f 86 b6 03 00
00 48 39 c2 0f 82 ad 03 00 00 49 8b 10 48 83 e2 f8 48 39 ca 0f 85 c5 06 00
00 49 8b 51 18 <4c> 39 4a 10 0f 85 df 04 00 00 49 39 69 10 0f 85 d5 04 00
00 a8 01
Jun 29 11:37:47 127.0.0.1 systemd[1]: Started Process Core Dump (PID
1395302/UID 0).
Jun 29 11:37:47 127.0.0.1 systemd-coredump[1395303]: Process 1327836
(rsyslogd) of user 0 dumped core.#012#012Stack trace of thread
1327844:#012#0 0x00007f65a85f2cf7 _int_malloc (libc.so.6)#012#1
0x00007f65a85f4842 malloc (libc.so.6)#012#2 0x000055d82dcb035a
msgSetFromSockinfo (rsyslogd)#012#3 0x00007f65a5c0dc01 rcvMainLoop
(imudp.so)#012#4 0x00007f65a5c0e200 wrkr (imudp.so)#012#5
0x00007f65a98ab1cf start_thread (libpthread.so.0)#012#6
0x00007f65a8591d83 __clone (libc.so.6)#012#012Stack trace of thread
1327837:#012#0 0x00007f65a98b4aa4 read (libpthread.so.0)#012#1
0x00007f65a66370ce readklog (imklog.so)#012#2 0x00007f65a663741d
klogLogKMsg (imklog.so)#012#3 0x00007f65a663658c runInput
(imklog.so)#012#4 0x000055d82dcda281 thrdStarter (rsyslogd)#012#5
0x00007f65a98ab1cf start_thread (libpthread.so.0)#012#6
0x00007f65a8591d83 __clone (libc.so.6)#012#012Stack trace of thread
1327836:#012#0 0x00007f65a867f331 pselect (libc.so.6)#012#1
0x000055d82dc860a9 mainloop (rsyslogd)#012#2 0x000055d82dc82e64 main
(rsyslogd)#012#3 0x00007f65a8592ca3 __libc_start_main (libc.so.6)#012#4
0x000055d82dc8322e _start (rsyslogd)#012#012Stack trace of thread
1327842:#012#0 0x00007f65a8687a27 epoll_wait (libc.so.6)#012#1
0x00007f65a5c0d8b8 rcvMainLoop (imudp.so)#012#2 0x00007f65a5c0e200 wrkr
(imudp.so)#012#3 0x00007f65a5c0e34f runInput (imudp.so)#012#4
0x000055d82dcda281 thrdStarter (rsyslogd)#012#5 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#6 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1327839:#012#0 0x00007f65a867cbd6
ppoll (libc.so.6)#012#1 0x00007f65a8dfc23f fd_wait_for_event
(libsystemd.so.0)#012#2 0x00007f65a8dd849b sd_journal_wait
(libsystemd.so.0)#012#3 0x00007f65a6226c40 runInput (imjournal.so)#012#4
0x000055d82dcda281 thrdStarter (rsyslogd)#012#5 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#6 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1327840:#012#0 0x00007f65a867f21f
__select (libc.so.6)#012#1 0x000055d82dcb9af7 srSleep (rsyslogd)#012#2
0x00007f65a6020960 runInput (impstats.so)#012#3 0x000055d82dcda281
thrdStarter (rsyslogd)#012#4 0x00007f65a98ab1cf start_thread
(libpthread.so.0)#012#5 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1327843:#012#0 0x00007f65a98b144c
pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00007f65a5e19344
wrkr (imptcp.so)#012#2 0x00007f65a98ab1cf start_thread
(libpthread.so.0)#012#3 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1327841:#012#0 0x00007f65a8687a27
epoll_wait (libc.so.6)#012#1 0x00007f65a5e1965c runInput (imptcp.so)#012#2
0x000055d82dcda281 thrdStarter (rsyslogd)#012#3 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#4 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1395226:#012#0 0x00007f65a98b179a
pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1
0x000055d82dcc5edc asyncWriterThread (rsyslogd)#012#2 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#3 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1395150:#012#0 0x00007f65a98b179a
pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1
0x000055d82dcc5edc asyncWriterThread (rsyslogd)#012#2 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#3 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1395071:#012#0 0x00007f65a98b179a
pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1
0x000055d82dcc5edc asyncWriterThread (rsyslogd)#012#2 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#3 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1395128:#012#0 0x00007f65a98b179a
pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1
0x000055d82dcc5edc asyncWriterThread (rsyslogd)#012#2 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#3 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1395236:#012#0 0x00007f65a98b179a
pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1
0x000055d82dcc5edc asyncWriterThread (rsyslogd)#012#2 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#3 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1327846:#012#0 0x00007f65a98b144c
pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x000055d82dcc8a2c
wtiWorker (rsyslogd)#012#2 0x000055d82dcc71bf wtpWorker (rsyslogd)#012#3
0x00007f65a98ab1cf start_thread (libpthread.so.0)#012#4
0x00007f65a8591d83 __clone (libc.so.6)#012#012Stack trace of thread
1395183:#012#0 0x00007f65a98b179a pthread_cond_timedwait@@GLIBC_2.3.2
(libpthread.so.0)#012#1 0x000055d82dcc5edc asyncWriterThread
(rsyslogd)#012#2 0x00007f65a98ab1cf start_thread (libpthread.so.0)#012#3
0x00007f65a8591d83 __clone (libc.so.6)#012#012Stack trace of thread
1395246:#012#0 0x00007f65a98b179a pthread_cond_timedwait@@GLIBC_2.3.2
(libpthread.so.0)#012#1 0x000055d82dcc5edc asyncWriterThread
(rsyslogd)#012#2 0x00007f65a98ab1cf start_thread (libpthread.so.0)#012#3
0x00007f65a8591d83 __clone (libc.so.6)#012#012Stack trace of thread
1328952:#012#0 0x00007f65a98b144c pthread_cond_wait@@GLIBC_2.3.2
(libpthread.so.0)#012#1 0x000055d82dcc8a2c wtiWorker (rsyslogd)#012#2
0x000055d82dcc71bf wtpWorker (rsyslogd)#012#3 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#4 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1395234:#012#0 0x00007f65a98b179a
pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1
0x000055d82dcc5edc asyncWriterThread (rsyslogd)#012#2 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#3 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1395160:#012#0 0x00007f65a98b179a
pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1
0x000055d82dcc5edc asyncWriterThread (rsyslogd)#012#2 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#3 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1395189:#012#0 0x00007f65a98b179a
pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1
0x000055d82dcc5edc asyncWriterThread (rsyslogd)#012#2 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#3 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1395282:#012#0 0x00007f65a98b179a
pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1
0x000055d82dcc5edc asyncWriterThread (rsyslogd)#012#2 0x00007f65a98ab1cf
start_thread (libpthread.so.0)#012#3 0x00007f65a8591d83 __clone
(libc.so.6)#012#012Stack trace of thread 1327854:#012#0 0x00007f65a98b144c
pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x000055d82dcc8a2c
wtiWorker (rsyslogd)#012#2 0x000055d82dcc71bf wtpWorker (rsyslogd)#012#3
0x00007f65a98ab1cf start_thread (libpthread.so.0)#012#4
0x00007f65a8591d83 __clone (libc.so.6)#012#012Stack trace of thread
1394253:#012#0 0x00007f65a98b179a pthread_cond_timedwait@@GLIBC_2.3.2
(libpthread.so.0)#012#1 0x000055d82dcc5edc asyncWriterThread
(rsyslogd)#012#2 0x00007f65a98ab1cf start_thread (libpthread.so.0)#012#3
0x00007f65a8591d83 __clone (libc.so.6)#012#012Stack trace of thread
1395283:#012#0 0x00007f65a98b179a pthread_cond_timedwait@@GLIBC_2.3.2
(libpthread.so.0)#012#1 0x000055d82dcc5edc asyncWriterThread
(rsyslogd)#012#2 0x00007f65a98ab1cf start_thread (libpthread.so.0)#012#3
0x00007f65a8591d83 __clone (libc.so.6)#012#012Stack trace of thread
1395286:#012#0 0x00007f65a98b179a pthread_cond_timedwait@@GLIBC_2.3.2
(libpthread.so.0)#012#1 0x000055d82dcc5edc asyncWriterThread
(rsyslogd)#012#2 0x00007f65a98ab1cf start_thread (libpthread.so.0)#012#3
0x00007f65a8591d83 __clone (libc.so.6)#012#012Stack trace of thread
1394917:#012#0 0x00007f65a98b179a pthread_cond_timedwait@@GLIBC_2.3.2
(libpthread.so.0)#012#1 0x000055d82dcc5edc asyncWriterThread
(rsyslogd)#012#2 0x00007f65a98ab1cf start_thread (libpthread.so.0)#012#3
0x00007f65a8591d83 __clone (libc.so.6)#012#012Stack trace of thread
1395290:#012#0 0x00007f65a98b179a pthread_cond_timedwait@@GLIBC_2.3.2
(libpthread.so.0)#012#1 0x000055d82dcc5edc asyncWriterThread
(rsyslogd)#012#2 0x00007f65a98ab1cf start_thread (libpthread.so.0)#012#3
0x00007f65a8591d83 __clone (libc.so.6)#012


Thanks & Regards
Vijay Kumar Kanukula

On Mon, 20 Jun 2022 at 22:54, vijay kumar <vijay.kanukula@gmail.com> wrote:

> Hi Rainer,
>
> Do you have any luck with asan logs???
>
> Thanks & Regards
> Vijay Kumar Kanukula
>
> On Fri, 17 Jun 2022 at 14:11, vijay kumar <vijay.kanukula@gmail.com>
> wrote:
>
>> Hello Rainer,
>>
>> When we installed ASAN related RPMS to capture the logs rsyslog restart
>> for every 2 to 3 mins, it was purely unstable so we downgraded immediately.
>> Attaching ASAN logs captured day before yesterday.
>>
>> Thanks & Regards
>> Vijay Kumar Kanukula
>>
>> On Fri, 17 Jun 2022 at 13:44, Rainer Gerhards <rgerhards@hq.adiscon.com>
>> wrote:
>>
>>> can you please post rsyslog -v output as well as the current ASAN report?
>>>
>>> Thanks,
>>> Rainer
>>>
>>> El vie, 17 jun 2022 a las 10:12, vijay kumar
>>> (<vijay.kanukula@gmail.com>) escribió:
>>> >
>>> > HI Rainer/David/Marisuz,
>>> >
>>> > Could you please help me with creating one input rule with a queue as
>>> Marisuz suggested. I was failing to create a rule.
>>> >
>>> > If I change anything in my 90-inputs.conf file do I need to make any
>>> changes in 30-rules.conf???
>>> >
>>> > @Rainer : I upgraded to the latest version of rsyslog and tested. But
>>> still my rsyslog restarts continuously. It was not stable.
>>> >
>>> > Thanks & Regards
>>> > Vijay Kumar Kanukula
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, 17 Jun 2022 at 12:55, Rainer Gerhards <
>>> rgerhards@hq.adiscon.com> wrote:
>>> >>
>>> >> it would be good to find the root cause. The best would be to use the
>>> >> current 8.2206.0 version and see if it works. If so, all is fine. If
>>> >> not, we should try to debug the issue.
>>> >>
>>> >> Rainer
>>> >>
>>> >> El jue, 16 jun 2022 a las 16:22, vijay kumar via rsyslog
>>> >> (<rsyslog@lists.adiscon.com>) escribió:
>>> >> >
>>> >> > Hi Team,
>>> >> >
>>> >> > My rsyslog service is getting restarted very frequently and we
>>> understand
>>> >> > it is due to race between the various threads, which causes one
>>> thread to
>>> >> > free a message field while another tries to read/write it.
>>> >> >
>>> >> > log:
>>> >> > ===
>>> >> > ==3035157==ERROR: AddressSanitizer: heap-buffer-overflow on address
>>> >> > 0x61e000000ac2 at pc 0x7f6085ac62fd bp 0x7f6079d755e0 sp
>>> 0x7f6079d74d88
>>> >> > READ of size 2627 at 0x61e000000ac2 thread T3 (in:imjournal)
>>> >> > #0 0x7f6085ac62fc (/lib64/libasan.so.5+0xb92fc)
>>> >> > #1 0x7f607d5066a5 in readJSONfromJournalMsg imjournal.c:288
>>> >> > #2 0x7f607d5066a5 in readjournal imjournal.c:497
>>> >> > #3 0x55ae0523366e in thrdStarter ../threads.c:243
>>> >> > #4 0x7f60855dd1ce in start_thread
>>> (/lib64/libpthread.so.0+0x81ce)
>>> >> > #5 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>>> >> >
>>> >> > 0x61e000000ac2 is located 0 bytes to the right of 2626-byte region
>>> >> > [.0x61e000000080,0x61e000000ac2)
>>> >> > allocated by thread T3 (in:imjournal) here:
>>> >> > #0 0x7f6085afcfe8 in __interceptor_realloc
>>> (/lib64/libasan.so.5+0xeffe8)
>>> >> > #1 0x7f6084af1a8b (/lib64/libsystemd.so.0+0x82a8b)
>>> >> >
>>> >> > Thread T3 (in:imjournal) created by T0 here:
>>> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>>> >> > (/lib64/libasan.so.5+0x52ea3)
>>> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
>>> >> >
>>> >> > SUMMARY: AddressSanitizer: heap-buffer-overflow
>>> >> > (/lib64/libasan.so.5+0xb92fc)
>>> >> > Shadow bytes around the buggy address:
>>> >> > 0x0c3c7fff8100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> >> > 0x0c3c7fff8110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> >> > 0x0c3c7fff8120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> >> > 0x0c3c7fff8130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> >> > 0x0c3c7fff8140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> >> > =>0x0c3c7fff8150: 00 00 00 00 00 00 00 00[02]fa fa fa fa fa fa fa
>>> >> > 0x0c3c7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>>> >> > 0x0c3c7fff8170: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>>> >> > 0x0c3c7fff8180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>>> >> > 0x0c3c7fff8190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>>> >> > 0x0c3c7fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>>> >> > Shadow byte legend (one shadow byte represents 8 application bytes):
>>> >> > Addressable: 00
>>> >> > Partially addressable: 01 02 03 04 05 06 07
>>> >> > Heap left redzone: fa
>>> >> > Freed heap region: fd
>>> >> > Stack left redzone: f1
>>> >> > Stack mid redzone: f2
>>> >> > Stack right redzone: f3
>>> >> > Stack after return: f5
>>> >> > Stack use after scope: f8
>>> >> > Global redzone: f9
>>> >> > Global init order: f6
>>> >> > Poisoned by user: f7
>>> >> > Container overflow: fc
>>> >> > Array cookie: ac
>>> >> > Intra object redzone: bb
>>> >> > ASan internal: fe
>>> >> > Left alloca redzone: ca
>>> >> > Right alloca redzone: cb
>>> >> > =================================================================
>>> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
>>> >> > 0x60c0005d85c0 at pc 0x7f6085a4db27 bp 0x7f6070621e50 sp
>>> 0x7f60706215f8
>>> >> > READ of size 128 at 0x60c0005d85c0 thread T9 (rs:main Q:Reg)
>>> >> > #0 0x7f6085a4db26 (/lib64/libasan.so.5+0x40b26)
>>> >> > #1 0x55ae0516d061 in msgSetFromSockinfo
>>> (/usr/sbin/rsyslogd+0x390061)
>>> >> > #2 0x55ae0516e522 in MsgDup msg.c:1129
>>> >> > #3 0x55ae05205d58 in execCall ruleset.c:290
>>> >> > #4 0x55ae05205d58 in scriptExec ruleset.c:608
>>> >> > #5 0x55ae05206532 in execIf ruleset.c:313
>>> >> > #6 0x55ae05206532 in scriptExec ruleset.c:614
>>> >> > #7 0x55ae05208868 in processBatch ruleset.c:660
>>> >> > #8 0x55ae050ae50c in msgConsumer rsyslogd.c:694
>>> >> > #9 0x55ae051f100d in ConsumerReg queue.c:2145
>>> >> > #10 0x55ae051e0804 in wtiWorker wti.c:428
>>> >> > #11 0x55ae051d9dd5 in wtpWorker wtp.c:435
>>> >> > #12 0x7f60855dd1ce in start_thread
>>> (/lib64/libpthread.so.0+0x81ce)
>>> >> > #13 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>>> >> >
>>> >> > 0x60c0005d85c0 is located 0 bytes inside of 128-byte region
>>> >> > [.0x60c0005d85c0,0x60c0005d8640)
>>> >> > freed by thread T6 (imudp(w1)) here:
>>> >> > #0 0x7f6085afc7e0 in __interceptor_free
>>> (/lib64/libasan.so.5+0xef7e0)
>>> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
>>> >> >
>>> >> > previously allocated by thread T6 (imudp(w1)) here:
>>> >> > #0 0x7f6085afcba8 in __interceptor_malloc
>>> (/lib64/libasan.so.5+0xefba8)
>>> >> > #1 0x55ae0516cff4 in msgSetFromSockinfo
>>> (/usr/sbin/rsyslogd+0x38fff4)
>>> >> >
>>> >> > Thread T9 (rs:main Q:Reg) created by T3 (in:imjournal) here:
>>> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>>> >> > (/lib64/libasan.so.5+0x52ea3)
>>> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
>>> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
>>> >> >
>>> >> > Thread T6 (imudp(w1)) created by T0 here:
>>> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>>> >> > (/lib64/libasan.so.5+0x52ea3)
>>> >> > #1 0x55ae05234470 in thrdCreate ../threads.c:289
>>> >> >
>>> >> > SUMMARY: AddressSanitizer: heap-use-after-free
>>> (/lib64/libasan.so.5+0x40b26)
>>> >> > Shadow bytes around the buggy address:
>>> >> > 0x0c18800b3060: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>>> >> > 0x0c18800b3070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>>> >> > 0x0c18800b3080: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
>>> >> > 0x0c18800b3090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>>> >> > 0x0c18800b30a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>>> >> > =>0x0c18800b30b0: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
>>> >> > 0x0c18800b30c0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>>> >> > 0x0c18800b30d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>>> >> > 0x0c18800b30e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>>> >> > 0x0c18800b30f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>>> >> > 0x0c18800b3100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>>> >> > Shadow byte legend (one shadow byte represents 8 application bytes):
>>> >> > Addressable: 00
>>> >> > Partially addressable: 01 02 03 04 05 06 07
>>> >> > Heap left redzone: fa
>>> >> > Freed heap region: fd
>>> >> > Stack left redzone: f1
>>> >> > Stack mid redzone: f2
>>> >> > Stack right redzone: f3
>>> >> > Stack after return: f5
>>> >> > Stack use after scope: f8
>>> >> > Global redzone: f9
>>> >> > Global init order: f6
>>> >> > Poisoned by user: f7
>>> >> > Container overflow: fc
>>> >> > Array cookie: ac
>>> >> > Intra object redzone: bb
>>> >> > ASan internal: fe
>>> >> > Left alloca redzone: ca
>>> >> > Right alloca redzone: cb
>>> >> > =================================================================
>>> >> > ==3035157==ERROR: AddressSanitizer: heap-use-after-free on address
>>> >> > 0x6040000285a0 at pc 0x55ae0520a723 bp 0x7f60669659a0 sp
>>> 0x7f6066965990
>>> >> > WRITE of size 4 at 0x6040000285a0 thread T16 (rs:qradar.local)
>>> >> > #0 0x55ae0520a722 in propDestruct prop.c:63
>>> >> > #1 0x55ae05170299 in MsgSetRcvFromIPWithoutAddRef msg.c:457
>>> >> > #2 0x55ae05170299 in resolveDNS msg.c:522
>>> >> > #3 0x55ae0517064a in getRcvFromIP msg.c:558
>>> >> > #4 0x55ae05179317 in MsgGetProp (/usr/sbin/rsyslogd+0x39c317)
>>> >> > #5 0x55ae0524aece in tplToString ../template.c:207
>>> >> > #6 0x55ae0522a4ab in prepareDoActionParams ../action.c:1114
>>> >> > #7 0x55ae0522a4ab in processMsgMain ../action.c:1648
>>> >> > #8 0x55ae0522c279 in doSubmitToActionQ ../action.c:1825
>>> >> > #9 0x55ae05205826 in execAct ruleset.c:209
>>> >> > #10 0x55ae05205826 in scriptExec ruleset.c:599
>>> >> > #11 0x55ae05208868 in processBatch ruleset.c:660
>>> >> > #12 0x55ae050ae50c in msgConsumer rsyslogd.c:694
>>> >> > #13 0x55ae051f100d in ConsumerReg queue.c:2145
>>> >> > #14 0x55ae051e0804 in wtiWorker wti.c:428
>>> >> > #15 0x55ae051d9dd5 in wtpWorker wtp.c:435
>>> >> > #16 0x7f60855dd1ce in start_thread
>>> (/lib64/libpthread.so.0+0x81ce)
>>> >> > #17 0x7f60835a1d82 in clone (/lib64/libc.so.6+0x39d82)
>>> >> >
>>> >> > 0x6040000285a0 is located 16 bytes inside of 48-byte region
>>> >> > [.0x604000028590,0x6040000285c0)
>>> >> > freed by thread T16 (rs:qradar.local) here:
>>> >> > #0 0x7f6085afc7e0 in __interceptor_free
>>> (/lib64/libasan.so.5+0xef7e0)
>>> >> > #1 0x55ae0515a3a5 in MsgSetRcvFromWithoutAddRef msg.c:471
>>> >> >
>>> >> > previously allocated by thread T16 (rs:qradar.local) here:
>>> >> > #0 0x7f6085afcdb0 in calloc (/lib64/libasan.so.5+0xefdb0)
>>> >> > #1 0x55ae0520a439 in propConstruct prop.c:56
>>> >> >
>>> >> > Thread T16 (rs:qradar.local) created by T9 (rs:main Q:Reg) here:
>>> >> > #0 0x7f6085a5fea3 in __interceptor_pthread_create
>>> >> > (/lib64/libasan.so.5+0x52ea3)
>>> >> > #1 0x55ae051dc9d8 in wtpStartWrkr wtp.c:497
>>> >> > #2 0x55ae051dc9d8 in wtpAdviseMaxWorkers wtp.c:570
>>> >> >
>>> >> > SUMMARY: AddressSanitizer: heap-use-after-free prop.c:63 in
>>> propDestruct
>>> >> > Shadow bytes around the buggy address:
>>> >> > 0x0c087fffd060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
>>> >> > 0x0c087fffd070: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
>>> >> > 0x0c087fffd080: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fd
>>> >> > 0x0c087fffd090: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd fa
>>> >> > 0x0c087fffd0a0: fa fa 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
>>> >> > =>0x0c087fffd0b0: fa fa fd fd[fd]fd fd fd fa fa fa fa fa fa fa fa
>>> >> > 0x0c087fffd0c0: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
>>> >> > 0x0c087fffd0d0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
>>> >> > 0x0c087fffd0e0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
>>> >> > 0x0c087fffd0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
>>> >> > 0x0c087fffd100: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
>>> >> > Shadow byte legend (one shadow byte represents 8 application bytes):
>>> >> > Addressable: 00
>>> >> > Partially addressable: 01 02 03 04 05 06 07
>>> >> > Heap left redzone: fa
>>> >> > Freed heap region: fd
>>> >> > Stack left redzone: f1
>>> >> > Stack mid redzone: f2
>>> >> > Stack right redzone: f3
>>> >> > Stack after return: f5
>>> >> > Stack use after scope: f8
>>> >> > Global redzone: f9
>>> >> > Global init order: f6
>>> >> > Poisoned by user: f7
>>> >> > Container overflow: fc
>>> >> > Array cookie: ac
>>> >> > Intra object redzone: bb
>>> >> > ASan internal: fe
>>> >> > Left alloca redzone: ca
>>> >> > Right alloca redzone: cb
>>> >> > ==3035157==ABORTING
>>> >> >
>>> >> > =================================
>>> >> > what is the possible solution ???
>>> >> >
>>> >> > would be to have multiple rsyslog instances, which is possible if
>>> the
>>> >> > traffic is split between ports. If yes could you please suggest how
>>> to
>>> >> > configure??
>>> >> >
>>> >> > Thanks & Regards
>>> >> > Vijay Kumar Kanukula
>>> >> > _______________________________________________
>>> >> > rsyslog mailing list
>>> >> > https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> >> > http://www.rsyslog.com/professional-services/
>>> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>> you DON'T LIKE THAT.
>>>
>>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.