Mailing List Archive: error starting up disk queue

We are getting an error when starting/restarting rsyslog on RHEL 7 "rsyslogd: error starting up disk queue, using pure in-memory mode [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2026 ]"

The system has the same config as another that is not getting the error and we have disabled SELINUX, rebooted and the error persists. /var/log has the same permissions as the system that doesn't generate the error. Any other tips we can use to see what is causing this?

$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
$FileCreateMode 0640
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
*.emerg :omusrmsg:*
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
*.* action(type="omfwd"
queue.type="linkedlist"
queue.filename="for_q1"
queue.spoolDirectory="/var/log"
queue.maxdiskspace="1g"
queue.timeoutenqueue="0"
action.resumeRetryCount="-1"
queue.saveOnShutdown="on"
target="192.168.2.2" port="516" protocol="tcp"
)

Here is a debug when the first error occurs:

'/var/log', maxFileSz 1048576, maxQSize 1000, lqsize 0, pqsize 0, child 0, full delay 450, light delay 700, deq batch size 128, high wtrmrk 900, low wtrmrk 700, discardmrk 980, max wrkr 1, min msgs f. wrkr 1000
7261.267377720:main thread : action 7 queue:Reg: finalizing construction of worker thread pool (numworkerThreads 1)
7261.267382385:main thread : action 7 queue:Reg/w0: finalizing construction of worker instance data (for 8 actions)
7261.267389892:main thread : action 7 queue:DAwpool: finalizing construction of worker thread pool (numworkerThreads 1)
7261.267393774:main thread : action 7 queue:DAwpool/w0: finalizing construction of worker instance data (for 8 actions)
7261.267399272:main thread : action 7 queue[DA]: starting queue
7261.267403577:main thread : action 7 queue[DA]: .qi file name is '/var/log/for_q1.qi', len 18
7261.267406858:main thread : action 7 queue[DA]: I am a child

7261.267421046:main thread : file stream /var/log/for_q1.qi params: flush interval 0, async write 0
7261.267431003:main thread : file '/var/log/for_q1.qi' opened as #6 with mode 384
7261.267443643:main thread : strm 0x563dea7ed240: opened file '/var/log/for_q1.qi' for READ as 6
7261.267449203:main thread : strm 0x563dea7ed240: file 6 read 0 bytes
7261.267452994:main thread : objDeserializePropBag error -2026 during header - trying to recover
7261.267458114:main thread : strm 0x563dea7ed240: file 6 read 0 bytes
7261.267461609:main thread : deserializer has possibly been able to re-sync and recover, state -2026
7261.267467354:main thread : strm 0x563dea7ed240: file 6(/var/log/for_q1.qi) closing, bDeleteOnClose 0
7261.267475936:main thread : action 7 queue[DA]: state -2026 reading .qi file - can not read persisted info (if any)
7261.267480108:main thread : Called LogMsg, msg: error starting up disk queue, using pure in-memory mode
7261.267544304:main thread : action 7 queue[DA]: shutdown: begin to destruct queue
7261.267548988:main thread : action 7 queue: error -2026 creating disk queue - giving up.
7261.267552314:main thread : action 7 queue: queue finished initialization
7261.267559662:main thread : Action builtin:omfwd[0x563dea7fce20]: queue 0x563dea7fe5d0 started
7261.267563608:main thread : Activating Ruleset Queue[(nil)] for Ruleset RSYSLOG_DefaultRuleset
7261.267567011:main thread : activateMainQueue: mainq cnf obj ptr is (nil)
7261.267571157:main thread : main Q: starting queue
7261.267579939:main thread : main Q: is NOT disk-assisted

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

this lists the workdirectory as /var/lib/rsyslog, what are the permissions
there? I usually make this something below /var/log

I would want the spool directory to be something more single-purpose than
/var/log

just double check the permissions on every directory you list in the config.

David Lang

On Tue, 31 Jan 2023, rsyslog--- via rsyslog wrote:

> Date: Tue, 31 Jan 2023 14:00:13 -0700
> From: rsyslog--- via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog@lists.adiscon.com
> Cc: rsyslog@mm.st
> Subject: [rsyslog] error starting up disk queue
>
> We are getting an error when starting/restarting rsyslog on RHEL 7 "rsyslogd: error starting up disk queue, using pure in-memory mode [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2026 ]"
>
> The system has the same config as another that is not getting the error and we have disabled SELINUX, rebooted and the error persists. /var/log has the same permissions as the system that doesn't generate the error. Any other tips we can use to see what is causing this?
>
> $WorkDirectory /var/lib/rsyslog
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
> $IncludeConfig /etc/rsyslog.d/*.conf
> $OmitLocalLogging on
> $IMJournalStateFile imjournal.state
> $FileCreateMode 0640
> *.info;mail.none;authpriv.none;cron.none /var/log/messages
> authpriv.* /var/log/secure
> mail.* -/var/log/maillog
> cron.* /var/log/cron
> *.emerg :omusrmsg:*
> uucp,news.crit /var/log/spooler
> local7.* /var/log/boot.log
> *.* action(type="omfwd"
> queue.type="linkedlist"
> queue.filename="for_q1"
> queue.spoolDirectory="/var/log"
> queue.maxdiskspace="1g"
> queue.timeoutenqueue="0"
> action.resumeRetryCount="-1"
> queue.saveOnShutdown="on"
> target="192.168.2.2" port="516" protocol="tcp"
> )
>
> Here is a debug when the first error occurs:
>
> '/var/log', maxFileSz 1048576, maxQSize 1000, lqsize 0, pqsize 0, child 0, full delay 450, light delay 700, deq batch size 128, high wtrmrk 900, low wtrmrk 700, discardmrk 980, max wrkr 1, min msgs f. wrkr 1000
> 7261.267377720:main thread : action 7 queue:Reg: finalizing construction of worker thread pool (numworkerThreads 1)
> 7261.267382385:main thread : action 7 queue:Reg/w0: finalizing construction of worker instance data (for 8 actions)
> 7261.267389892:main thread : action 7 queue:DAwpool: finalizing construction of worker thread pool (numworkerThreads 1)
> 7261.267393774:main thread : action 7 queue:DAwpool/w0: finalizing construction of worker instance data (for 8 actions)
> 7261.267399272:main thread : action 7 queue[DA]: starting queue
> 7261.267403577:main thread : action 7 queue[DA]: .qi file name is '/var/log/for_q1.qi', len 18
> 7261.267406858:main thread : action 7 queue[DA]: I am a child
>
> 7261.267421046:main thread : file stream /var/log/for_q1.qi params: flush interval 0, async write 0
> 7261.267431003:main thread : file '/var/log/for_q1.qi' opened as #6 with mode 384
> 7261.267443643:main thread : strm 0x563dea7ed240: opened file '/var/log/for_q1.qi' for READ as 6
> 7261.267449203:main thread : strm 0x563dea7ed240: file 6 read 0 bytes
> 7261.267452994:main thread : objDeserializePropBag error -2026 during header - trying to recover
> 7261.267458114:main thread : strm 0x563dea7ed240: file 6 read 0 bytes
> 7261.267461609:main thread : deserializer has possibly been able to re-sync and recover, state -2026
> 7261.267467354:main thread : strm 0x563dea7ed240: file 6(/var/log/for_q1.qi) closing, bDeleteOnClose 0
> 7261.267475936:main thread : action 7 queue[DA]: state -2026 reading .qi file - can not read persisted info (if any)
> 7261.267480108:main thread : Called LogMsg, msg: error starting up disk queue, using pure in-memory mode
> 7261.267544304:main thread : action 7 queue[DA]: shutdown: begin to destruct queue
> 7261.267548988:main thread : action 7 queue: error -2026 creating disk queue - giving up.
> 7261.267552314:main thread : action 7 queue: queue finished initialization
> 7261.267559662:main thread : Action builtin:omfwd[0x563dea7fce20]: queue 0x563dea7fe5d0 started
> 7261.267563608:main thread : Activating Ruleset Queue[(nil)] for Ruleset RSYSLOG_DefaultRuleset
> 7261.267567011:main thread : activateMainQueue: mainq cnf obj ptr is (nil)
> 7261.267571157:main thread : main Q: starting queue
> 7261.267579939:main thread : main Q: is NOT disk-assisted
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.