Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. RSyslog>
  3. users
1 2  View All
Repeated 111 to rsyslog UDS from nginx
rsyslog at lists
Sep 17, 2023, 9:39 AM
Post #1 of 33 (688 views)
Permalink
Hi,

ever since I started logging to a UDS from my nginx I get the occasional
111 in my nginx error logs.
As I literally don't have any other information or log entries I
honestly do not know how to debug this.
The thing is requests one second, or a few seconds later are processed
totally fine, so it cannot be a general problem, nor a access problem or
a permission problem.
I would be glad if anyone could help on fixing this

Examples:
error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket
error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
Connection refused) while logging to syslog, server:
unix:/var/cache/nginx/rsyslog.socket

configuration for this single use case, of course there is a lot more in
rsyslog.conf
$AddUnixListenSocket /var/cache/nginx/rsyslog.socket

$template app,"%msg:2:$%"

if $programname == "app" then {
   ^/usr/local/script/app_log.sh;app
   stop
}

The script app_log.sh does simply
echo "${@}" | /usr/bin/python <python_script>

Many thanks

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 17, 2023, 11:44 PM
Post #2 of 33 (687 views)
Permalink
Hello!

Does the timing match with rsyslog restarts (manual or logrotate-initiated)?

On Mon, 18 Sept 2023 at 00:39, TG Servers via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Hi,
>
> ever since I started logging to a UDS from my nginx I get the occasional
> 111 in my nginx error logs.
> As I literally don't have any other information or log entries I
> honestly do not know how to debug this.
> The thing is requests one second, or a few seconds later are processed
> totally fine, so it cannot be a general problem, nor a access problem or
> a permission problem.
> I would be glad if anyone could help on fixing this
>
> Examples:
> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
>
> configuration for this single use case, of course there is a lot more in
> rsyslog.conf
> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>
> $template app,"%msg:2:$%"
>
> if $programname == "app" then {
> ^/usr/local/script/app_log.sh;app
> stop
> }
>
> The script app_log.sh does simply
> echo "${@}" | /usr/bin/python <python_script>
>
> Many thanks
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.



--
Yury Bushmelev
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 17, 2023, 11:53 PM
Post #3 of 33 (687 views)
Permalink
Is this from a nginx text log? Any errors infos from rsyslog itself?

Rainer
PS: I do not see how this can be related to rsyslog, but you never
know. I do not yet understand the fault scenario TBH.

El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Hi,
>
> ever since I started logging to a UDS from my nginx I get the occasional
> 111 in my nginx error logs.
> As I literally don't have any other information or log entries I
> honestly do not know how to debug this.
> The thing is requests one second, or a few seconds later are processed
> totally fine, so it cannot be a general problem, nor a access problem or
> a permission problem.
> I would be glad if anyone could help on fixing this
>
> Examples:
> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
>
> configuration for this single use case, of course there is a lot more in
> rsyslog.conf
> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>
> $template app,"%msg:2:$%"
>
> if $programname == "app" then {
> ^/usr/local/script/app_log.sh;app
> stop
> }
>
> The script app_log.sh does simply
> echo "${@}" | /usr/bin/python <python_script>
>
> Many thanks
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 12:05 AM
Post #4 of 33 (687 views)
Permalink
Hi Yury,

no absolutely not, I cannot see any relation to things like that, that
is what leaves me a bit baffled here.
You can see this is all from one day, and if there is a 111 on 2:52:19
on 2:52:22 there is everything ok (just as an example)
Rsyslog restarts run between 0:10 and 0:15, depending on finish of
nightly dnf-update, logrotate runs at 0:00 via systemd timer, nginx is
not restarted automatically
There is no information in any log except in the nginx logs, and the
entries that are shown as failed are clearly missing in the target
analytics log
I cannot see any pattern...


On 18/09/2023 08:44, Yury Bushmelev via rsyslog wrote:
> Hello!
>
> Does the timing match with rsyslog restarts (manual or logrotate-initiated)?
>
> On Mon, 18 Sept 2023 at 00:39, TG Servers via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
>
>> Hi,
>>
>> ever since I started logging to a UDS from my nginx I get the occasional
>> 111 in my nginx error logs.
>> As I literally don't have any other information or log entries I
>> honestly do not know how to debug this.
>> The thing is requests one second, or a few seconds later are processed
>> totally fine, so it cannot be a general problem, nor a access problem or
>> a permission problem.
>> I would be glad if anyone could help on fixing this
>>
>> Examples:
>> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>>
>> configuration for this single use case, of course there is a lot more in
>> rsyslog.conf
>> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>>
>> $template app,"%msg:2:$%"
>>
>> if $programname == "app" then {
>> ^/usr/local/script/app_log.sh;app
>> stop
>> }
>>
>> The script app_log.sh does simply
>> echo "${@}" | /usr/bin/python <python_script>
>>
>> Many thanks
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>
>

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 12:09 AM
Post #5 of 33 (687 views)
Permalink
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 12:10 AM
Post #6 of 33 (687 views)
Permalink
Hi Rainer,

this is from nginx error log, yes.
No I cannot find any other errors, thats my problem
But it happens every single day, regularly...
as just written in another message re the question if it occurs with
rsyslog restart or logrotate :

no absolutely not, I cannot see any relation to things like that, that
is what leaves me a bit baffled here.
You can see this is all from one day, and if there is a 111 on 2:52:19
on 2:52:22 there is everything ok (just as an example)
Rsyslog restarts run between 0:10 and 0:15, depending on finish of
nightly dnf-update, logrotate runs at 0:00 via systemd timer, nginx is
not restarted automatically
There is no information in any log except in the nginx logs, and the
entries that are shown as failed are clearly missing in the target
analytics log
I cannot see any pattern...

On 18/09/2023 08:53, Rainer Gerhards wrote:
> Is this from a nginx text log? Any errors infos from rsyslog itself?
>
> Rainer
> PS: I do not see how this can be related to rsyslog, but you never
> know. I do not yet understand the fault scenario TBH.
>
> El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
>> Hi,
>>
>> ever since I started logging to a UDS from my nginx I get the occasional
>> 111 in my nginx error logs.
>> As I literally don't have any other information or log entries I
>> honestly do not know how to debug this.
>> The thing is requests one second, or a few seconds later are processed
>> totally fine, so it cannot be a general problem, nor a access problem or
>> a permission problem.
>> I would be glad if anyone could help on fixing this
>>
>> Examples:
>> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>>
>> configuration for this single use case, of course there is a lot more in
>> rsyslog.conf
>> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>>
>> $template app,"%msg:2:$%"
>>
>> if $programname == "app" then {
>> ^/usr/local/script/app_log.sh;app
>> stop
>> }
>>
>> The script app_log.sh does simply
>> echo "${@}" | /usr/bin/python <python_script>
>>
>> Many thanks
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 12:18 AM
Post #7 of 33 (687 views)
Permalink
Maybe a debug logs helps, but if rsyslog does not emit an error
message, it does not sound like it has some issue. I also don't see a
relation to the script. But to be sure, would it be possible to
temporarily remove it and see if that changes anything?

Rainer

El lun, 18 sept 2023 a las 9:09, TG Servers (<srvrs@prvtmail.net>) escribió:
>
> Hi Rainer,
>
> this is from nginx error log, yes.
> No I cannot find any other errors, thats my problem
> But it happens every single day, regularly...
> as just written in another message re the question if it occurs with rsyslog restart or logrotate :
>
> no absolutely not, I cannot see any relation to things like that, that is what leaves me a bit baffled here.
> You can see this is all from one day, and if there is a 111 on 2:52:19 on 2:52:22 there is everything ok (just as an example)
> Rsyslog restarts run between 0:10 and 0:15, depending on finish of nightly dnf-update, logrotate runs at 0:00 via systemd timer, nginx is not restarted automatically
> There is no information in any log except in the nginx logs, and the entries that are shown as failed are clearly missing in the target analytics log
> I cannot see any pattern...
>
>
> On 18/09/2023 08:53, Rainer Gerhards wrote:
>
> Is this from a nginx text log? Any errors infos from rsyslog itself?
>
> Rainer
> PS: I do not see how this can be related to rsyslog, but you never
> know. I do not yet understand the fault scenario TBH.
>
> El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
>
> Hi,
>
> ever since I started logging to a UDS from my nginx I get the occasional
> 111 in my nginx error logs.
> As I literally don't have any other information or log entries I
> honestly do not know how to debug this.
> The thing is requests one second, or a few seconds later are processed
> totally fine, so it cannot be a general problem, nor a access problem or
> a permission problem.
> I would be glad if anyone could help on fixing this
>
> Examples:
> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
>
> configuration for this single use case, of course there is a lot more in
> rsyslog.conf
> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>
> $template app,"%msg:2:$%"
>
> if $programname == "app" then {
> ^/usr/local/script/app_log.sh;app
> stop
> }
>
> The script app_log.sh does simply
> echo "${@}" | /usr/bin/python <python_script>
>
> Many thanks
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 12:21 AM
Post #8 of 33 (687 views)
Permalink
my thought is that if rsyslog is getting blocked (queues full) then it will stop
accepting new logs via unix sockets.

can you enable impstats and see if you have any reports of the main queue being
full during the times that this happens?

full configs for rsyslog could identify if there is anything you have there that
is likely to block for extended times.

David Lang


On Sun, 17 Sep 2023, TG Servers via rsyslog wrote:

> Hi,
>
> ever since I started logging to a UDS from my nginx I get the occasional
> 111 in my nginx error logs.
> As I literally don't have any other information or log entries I
> honestly do not know how to debug this.
> The thing is requests one second, or a few seconds later are processed
> totally fine, so it cannot be a general problem, nor a access problem or
> a permission problem.
> I would be glad if anyone could help on fixing this
>
> Examples:
> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
>
> configuration for this single use case, of course there is a lot more in
> rsyslog.conf
> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>
> $template app,"%msg:2:$%"
>
> if $programname == "app" then {
>    ^/usr/local/script/app_log.sh;app
>    stop
> }
>
> The script app_log.sh does simply
> echo "${@}" | /usr/bin/python <python_script>
>
> Many thanks
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 12:24 AM
Post #9 of 33 (687 views)
Permalink
what I did today in the morning was to put the socket in /run
input(type="imuxsock" socket="/run/logmat")

until now no errors so far but that does not mean a lot as there is not
much traffic right now.
you mean remove the python script or the whole script call? what I could
do is to echo the message to a file instead of piping it to the python
script, yes.
I will try it an let it run for today, or until I have an error again

On 18/09/2023 09:18, Rainer Gerhards wrote:
> Maybe a debug logs helps, but if rsyslog does not emit an error
> message, it does not sound like it has some issue. I also don't see a
> relation to the script. But to be sure, would it be possible to
> temporarily remove it and see if that changes anything?
>
> Rainer
>
> El lun, 18 sept 2023 a las 9:09, TG Servers (<srvrs@prvtmail.net>) escribió:
>> Hi Rainer,
>>
>> this is from nginx error log, yes.
>> No I cannot find any other errors, thats my problem
>> But it happens every single day, regularly...
>> as just written in another message re the question if it occurs with rsyslog restart or logrotate :
>>
>> no absolutely not, I cannot see any relation to things like that, that is what leaves me a bit baffled here.
>> You can see this is all from one day, and if there is a 111 on 2:52:19 on 2:52:22 there is everything ok (just as an example)
>> Rsyslog restarts run between 0:10 and 0:15, depending on finish of nightly dnf-update, logrotate runs at 0:00 via systemd timer, nginx is not restarted automatically
>> There is no information in any log except in the nginx logs, and the entries that are shown as failed are clearly missing in the target analytics log
>> I cannot see any pattern...
>>
>>
>> On 18/09/2023 08:53, Rainer Gerhards wrote:
>>
>> Is this from a nginx text log? Any errors infos from rsyslog itself?
>>
>> Rainer
>> PS: I do not see how this can be related to rsyslog, but you never
>> know. I do not yet understand the fault scenario TBH.
>>
>> El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog
>> (<rsyslog@lists.adiscon.com>) escribió:
>>
>> Hi,
>>
>> ever since I started logging to a UDS from my nginx I get the occasional
>> 111 in my nginx error logs.
>> As I literally don't have any other information or log entries I
>> honestly do not know how to debug this.
>> The thing is requests one second, or a few seconds later are processed
>> totally fine, so it cannot be a general problem, nor a access problem or
>> a permission problem.
>> I would be glad if anyone could help on fixing this
>>
>> Examples:
>> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
>> Connection refused) while logging to syslog, server:
>> unix:/var/cache/nginx/rsyslog.socket
>>
>> configuration for this single use case, of course there is a lot more in
>> rsyslog.conf
>> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>>
>> $template app,"%msg:2:$%"
>>
>> if $programname == "app" then {
>> ^/usr/local/script/app_log.sh;app
>> stop
>> }
>>
>> The script app_log.sh does simply
>> echo "${@}" | /usr/bin/python <python_script>
>>
>> Many thanks
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>
>>

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 6:08 AM
Post #10 of 33 (677 views)
Permalink
This email may contain proprietary information of BAE Systems and/or third parties.

Hi,

Has anyone got any thoughts/suggestions on this?

Cheers,

Sean.

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of David Lang via rsyslog
Sent: 18 September 2023 08:22
To: TG Servers via rsyslog <rsyslog@lists.adiscon.com>
Cc: David Lang <david@lang.hm>
Subject: Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx

----------------------------- PHISHING ALERT -----------------------------
This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click on a link or open an attachment.
For further information on how to spot and report a phishing email please access the Global Intranet, then select <Functions> / <IT>.

------------------------------------------------------------------------------------

my thought is that if rsyslog is getting blocked (queues full) then it will stop
accepting new logs via unix sockets.

can you enable impstats and see if you have any reports of the main queue being
full during the times that this happens?

full configs for rsyslog could identify if there is anything you have there that
is likely to block for extended times.

David Lang


On Sun, 17 Sep 2023, TG Servers via rsyslog wrote:

> Hi,
>
> ever since I started logging to a UDS from my nginx I get the occasional
> 111 in my nginx error logs.
> As I literally don't have any other information or log entries I
> honestly do not know how to debug this.
> The thing is requests one second, or a few seconds later are processed
> totally fine, so it cannot be a general problem, nor a access problem or
> a permission problem.
> I would be glad if anyone could help on fixing this
>
> Examples:
> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
>
> configuration for this single use case, of course there is a lot more in
> rsyslog.conf
> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>
> $template app,"%msg:2:$%"
>
> if $programname == "app" then {
>    ^/usr/local/script/app_log.sh;app
>    stop
> }
>
> The script app_log.sh does simply
> echo "${@}" | /usr/bin/python <python_script>
>
> Many thanks
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
********************************************************************
This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.
********************************************************************

BAE Systems may process information about you that may be subject to data protection
laws. For more information about how we use your personal information, how we protect
your information, our legal basis for using your information, your rights and who you can
contact, please refer to our Privacy Notice at www.baesystems.com/en/privacy
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 6:10 AM
Post #11 of 33 (677 views)
Permalink
This email may contain proprietary information of BAE Systems and/or third parties.

Sorry, replied to the wrong message.

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of Lennon, Sean (UK) via rsyslog
Sent: 18 September 2023 14:09
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Lennon, Sean (UK) <sean.lennon2@baesystems.com>
Subject: Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx

----------------------------- PHISHING ALERT ----------------------------- This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click on a link or open an attachment.
For further information on how to spot and report a phishing email please access the Global Intranet, then select <Functions> / <IT>.

------------------------------------------------------------------------------------





This email may contain proprietary information of BAE Systems and/or third parties.

Hi,

Has anyone got any thoughts/suggestions on this?

Cheers,

Sean.

-----Original Message-----
From: rsyslog <rsyslog-bounces@lists.adiscon.com> On Behalf Of David Lang via rsyslog
Sent: 18 September 2023 08:22
To: TG Servers via rsyslog <rsyslog@lists.adiscon.com>
Cc: David Lang <david@lang.hm>
Subject: Re: [rsyslog] Repeated 111 to rsyslog UDS from nginx

----------------------------- PHISHING ALERT ----------------------------- This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click on a link or open an attachment.
For further information on how to spot and report a phishing email please access the Global Intranet, then select <Functions> / <IT>.

------------------------------------------------------------------------------------

my thought is that if rsyslog is getting blocked (queues full) then it will stop accepting new logs via unix sockets.

can you enable impstats and see if you have any reports of the main queue being full during the times that this happens?

full configs for rsyslog could identify if there is anything you have there that is likely to block for extended times.

David Lang


On Sun, 17 Sep 2023, TG Servers via rsyslog wrote:

> Hi,
>
> ever since I started logging to a UDS from my nginx I get the
> occasional
> 111 in my nginx error logs.
> As I literally don't have any other information or log entries I
> honestly do not know how to debug this.
> The thing is requests one second, or a few seconds later are processed
> totally fine, so it cannot be a general problem, nor a access problem
> or a permission problem.
> I would be glad if anyone could help on fixing this
>
> Examples:
> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed (111:
> Connection refused) while logging to syslog, server:
> unix:/var/cache/nginx/rsyslog.socket
>
> configuration for this single use case, of course there is a lot more
> in rsyslog.conf $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>
> $template app,"%msg:2:$%"
>
> if $programname == "app" then {
>    ^/usr/local/script/app_log.sh;app
>    stop
> }
>
> The script app_log.sh does simply
> echo "${@}" | /usr/bin/python <python_script>
>
> Many thanks
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
********************************************************************
This email and any attachments are confidential to the intended recipient and may also be privileged. If you are not the intended recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or distribute its contents to any other person.
********************************************************************

BAE Systems may process information about you that may be subject to data protection laws. For more information about how we use your personal information, how we protect your information, our legal basis for using your information, your rights and who you can contact, please refer to our Privacy Notice at www.baesystems.com/en/privacy _______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 7:09 AM
Post #12 of 33 (677 views)
Permalink
so far not a single 111 today, I let this run the until late evening,
and if there is stil no 111 I will put back the python script in order
because right now there are 2 possibilities, I moved the socket as said,
and I skipped the script and just appended the message to a file
if either of the 2 things are responsible in the end I won't understand
it either :)
because the read socket is just piped to the script, so the script
should have absoletely no influence on the socket, but we will see what
happens instead of speculating now, nginx is also running on debug just
for the case

On 18/09/2023 09:24, TG Servers via rsyslog wrote:
> what I did today in the morning was to put the socket in /run
> input(type="imuxsock" socket="/run/logmat")
>
> until now no errors so far but that does not mean a lot as there is
> not much traffic right now.
> you mean remove the python script or the whole script call? what I
> could do is to echo the message to a file instead of piping it to the
> python script, yes.
> I will try it an let it run for today, or until I have an error again
>
> On 18/09/2023 09:18, Rainer Gerhards wrote:
>> Maybe a debug logs helps, but if rsyslog does not emit an error
>> message, it does not sound like it has some issue. I also don't see a
>> relation to the script. But to be sure, would it be possible to
>> temporarily remove it and see if that changes anything?
>>
>> Rainer
>>
>> El lun, 18 sept 2023 a las 9:09, TG Servers (<srvrs@prvtmail.net>)
>> escribió:
>>> Hi Rainer,
>>>
>>> this is from nginx error log, yes.
>>> No I cannot find any other errors, thats my problem
>>> But it happens every single day, regularly...
>>> as just written in another message re the question if it occurs with
>>> rsyslog restart or logrotate :
>>>
>>> no absolutely not, I cannot see any relation to things like that,
>>> that is what leaves me a bit baffled here.
>>> You can see this is all from one day, and if there is a 111 on
>>> 2:52:19 on 2:52:22 there is everything ok (just as an example)
>>> Rsyslog restarts run between 0:10 and 0:15, depending on finish of
>>> nightly dnf-update, logrotate runs at 0:00 via systemd timer, nginx
>>> is not restarted automatically
>>> There is no information in any log except in the nginx logs, and the
>>> entries that are shown as failed are clearly missing in the target
>>> analytics log
>>> I cannot see any pattern...
>>>
>>>
>>> On 18/09/2023 08:53, Rainer Gerhards wrote:
>>>
>>> Is this from a nginx text log? Any errors infos from rsyslog itself?
>>>
>>> Rainer
>>> PS: I do not see how this can be related to rsyslog, but you never
>>> know. I do not yet understand the fault scenario TBH.
>>>
>>> El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog
>>> (<rsyslog@lists.adiscon.com>) escribió:
>>>
>>> Hi,
>>>
>>> ever since I started logging to a UDS from my nginx I get the
>>> occasional
>>> 111 in my nginx error logs.
>>> As I literally don't have any other information or log entries I
>>> honestly do not know how to debug this.
>>> The thing is requests one second, or a few seconds later are processed
>>> totally fine, so it cannot be a general problem, nor a access
>>> problem or
>>> a permission problem.
>>> I would be glad if anyone could help on fixing this
>>>
>>> Examples:
>>> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed
>>> (111:
>>> Connection refused) while logging to syslog, server:
>>> unix:/var/cache/nginx/rsyslog.socket
>>>
>>> configuration for this single use case, of course there is a lot
>>> more in
>>> rsyslog.conf
>>> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>>>
>>> $template app,"%msg:2:$%"
>>>
>>> if $programname == "app" then {
>>>      ^/usr/local/script/app_log.sh;app
>>>      stop
>>> }
>>>
>>> The script app_log.sh does simply
>>> echo "${@}" | /usr/bin/python <python_script>
>>>
>>> Many thanks
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
>>>
>>>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 7:17 AM
Post #13 of 33 (677 views)
Permalink
> so far not a single 111 today, I let this run the until late evening,
> and if there is stil no 111 I will put back the python script in order
> because right now there are 2 possibilities, I moved the socket as said,
> and I skipped the script and just appended the message to a file
> if either of the 2 things are responsible in the end I won't understand
> it either :)

I don't know what the script does. But if it is slow, it may push back
to the main queue, making rsyslog unresponsive.

This is David's concern. Tomorrow, if you re-enable, you should also
enable impstats as David suggested.

Rainer
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 7:34 AM
Post #14 of 33 (677 views)
Permalink
I just wanted to add that in a further message as it came to my mind.
you were faster...
the script is definitely "slow", this is what I know for sure as it does
quite a lot of processing/analytics in the background, so even if you
trigger it from command line it can take half a sec or so.... I can't
change that, it needs to do what it does, I didn't write it
though it can handle manual fast F5 triggers in the browser without
issue and then it 111s when there are 2 requests incoming...
I thought rsyslog might handle that just well via the queue...
but then this might eventually really be the issue, and if it is, is
there anything to mitigate this from rsyslog side (in terms of own queue
for that socket or something in that direction)?
ok, will enable impstats, too when I switch back

Thanks,
Tom

On 18/09/2023 16:17, Rainer Gerhards wrote:
>> so far not a single 111 today, I let this run the until late evening,
>> and if there is stil no 111 I will put back the python script in order
>> because right now there are 2 possibilities, I moved the socket as said,
>> and I skipped the script and just appended the message to a file
>> if either of the 2 things are responsible in the end I won't understand
>> it either :)
> I don't know what the script does. But if it is slow, it may push back
> to the main queue, making rsyslog unresponsive.
>
> This is David's concern. Tomorrow, if you re-enable, you should also
> enable impstats as David suggested.
>
> Rainer

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 12:24 PM
Post #15 of 33 (664 views)
Permalink
rsyslog does not just pipe the socket to the script. It reads a message from the
socket, adds it to a queue (by default the main queue), then a separate thread
reads from the queue and sends the line to the script.

If the script takes too long to process the line, then a backlog of messages
build up in the queue.

When the queue is full, rsyslog stops reading the input (note that the OS can have
a buffer as well, depending on the input when that buffer fills, it will stall
the writer or throw away data). unix sockets block if the reader isn't able to
accept a message, which would result in the failure you are seeing

There are all sorts of options for the queue, you can make a dedicated queue for
that input, you can configure that queue to spill logs to disk when it gets too
full (reading them back later, which is a separate thread), throwing away logs
when it gets too full (depending on the log severity), etc.

David Lang


On Mon, 18 Sep 2023, TG Servers via rsyslog wrote:

> so far not a single 111 today, I let this run the until late evening,
> and if there is stil no 111 I will put back the python script in order
> because right now there are 2 possibilities, I moved the socket as said,
> and I skipped the script and just appended the message to a file
> if either of the 2 things are responsible in the end I won't understand
> it either :)
> because the read socket is just piped to the script, so the script
> should have absoletely no influence on the socket, but we will see what
> happens instead of speculating now, nginx is also running on debug just
> for the case
>
> On 18/09/2023 09:24, TG Servers via rsyslog wrote:
>> what I did today in the morning was to put the socket in /run
>> input(type="imuxsock" socket="/run/logmat")
>>
>> until now no errors so far but that does not mean a lot as there is
>> not much traffic right now.
>> you mean remove the python script or the whole script call? what I
>> could do is to echo the message to a file instead of piping it to the
>> python script, yes.
>> I will try it an let it run for today, or until I have an error again
>>
>> On 18/09/2023 09:18, Rainer Gerhards wrote:
>>> Maybe a debug logs helps, but if rsyslog does not emit an error
>>> message, it does not sound like it has some issue. I also don't see a
>>> relation to the script. But to be sure, would it be possible to
>>> temporarily remove it and see if that changes anything?
>>>
>>> Rainer
>>>
>>> El lun, 18 sept 2023 a las 9:09, TG Servers (<srvrs@prvtmail.net>)
>>> escribió:
>>>> Hi Rainer,
>>>>
>>>> this is from nginx error log, yes.
>>>> No I cannot find any other errors, thats my problem
>>>> But it happens every single day, regularly...
>>>> as just written in another message re the question if it occurs with
>>>> rsyslog restart or logrotate :
>>>>
>>>> no absolutely not, I cannot see any relation to things like that,
>>>> that is what leaves me a bit baffled here.
>>>> You can see this is all from one day, and if there is a 111 on
>>>> 2:52:19 on 2:52:22 there is everything ok (just as an example)
>>>> Rsyslog restarts run between 0:10 and 0:15, depending on finish of
>>>> nightly dnf-update, logrotate runs at 0:00 via systemd timer, nginx
>>>> is not restarted automatically
>>>> There is no information in any log except in the nginx logs, and the
>>>> entries that are shown as failed are clearly missing in the target
>>>> analytics log
>>>> I cannot see any pattern...
>>>>
>>>>
>>>> On 18/09/2023 08:53, Rainer Gerhards wrote:
>>>>
>>>> Is this from a nginx text log? Any errors infos from rsyslog itself?
>>>>
>>>> Rainer
>>>> PS: I do not see how this can be related to rsyslog, but you never
>>>> know. I do not yet understand the fault scenario TBH.
>>>>
>>>> El dom, 17 sept 2023 a las 18:39, TG Servers via rsyslog
>>>> (<rsyslog@lists.adiscon.com>) escribió:
>>>>
>>>> Hi,
>>>>
>>>> ever since I started logging to a UDS from my nginx I get the
>>>> occasional
>>>> 111 in my nginx error logs.
>>>> As I literally don't have any other information or log entries I
>>>> honestly do not know how to debug this.
>>>> The thing is requests one second, or a few seconds later are processed
>>>> totally fine, so it cannot be a general problem, nor a access
>>>> problem or
>>>> a permission problem.
>>>> I would be glad if anyone could help on fixing this
>>>>
>>>> Examples:
>>>> error.log:2023/09/17 02:41:59 [error] 346192#346192: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>> error.log:2023/09/17 02:41:59 [error] 346191#346191: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>> error.log:2023/09/17 02:52:19 [error] 346192#346192: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>> error.log:2023/09/17 04:09:44 [error] 346193#346193: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>> error.log:2023/09/17 04:09:45 [error] 346185#346185: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>> error.log:2023/09/17 04:20:20 [error] 346186#346186: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>> error.log:2023/09/17 06:20:01 [error] 346182#346182: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>> error.log:2023/09/17 08:32:35 [error] 346182#346182: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>> error.log:2023/09/17 08:32:35 [error] 346188#346188: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>> error.log:2023/09/17 09:34:34 [error] 346183#346183: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>> error.log:2023/09/17 09:34:35 [error] 346183#346183: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>> error.log:2023/09/17 16:00:25 [error] 346187#346187: send() failed
>>>> (111:
>>>> Connection refused) while logging to syslog, server:
>>>> unix:/var/cache/nginx/rsyslog.socket
>>>>
>>>> configuration for this single use case, of course there is a lot
>>>> more in
>>>> rsyslog.conf
>>>> $AddUnixListenSocket /var/cache/nginx/rsyslog.socket
>>>>
>>>> $template app,"%msg:2:$%"
>>>>
>>>> if $programname == "app" then {
>>>>      ^/usr/local/script/app_log.sh;app
>>>>      stop
>>>> }
>>>>
>>>> The script app_log.sh does simply
>>>> echo "${@}" | /usr/bin/python <python_script>
>>>>
>>>> Many thanks
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST if you DON'T LIKE THAT.
>>>>
>>>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 12:53 PM
Post #16 of 33 (664 views)
Permalink
I don't know what this is... I implemented a complete queue solution and
it occasionally happens when there is no request but one in sight, and
this one gets a 111 then, nothing in nginx debug log, no error to be
seen in rsyslog log
but one thing I realized, after a restart the first log message always,
reproducable gets a 111
the socket is not connected, nor listening, only after the first request
is logged/or not logged (which is logged with 111 in nginx) the socket
is connected and listening, so restarting rsyslog via systemd does not
connect/listen to/on the socket

the rsyslog debug log just tells us this :
6289.088037540:main thread    : imuxsock.c: imuxsock: Opened UNIX socket
'/run/logmat' (fd 6).

[root@xxx rsyslog.d]# systemctl restart rsyslog
[root@xxx rsyslog.d]# ss -x | grep logmat
[root@xxx rsyslog.d]# lsof /run/logmat
COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0 25300317
/run/logmat type=DGRAM (UNCONNECTED)

make a request from browser or curl

[root@xxx rsyslog.d]# lsof /run/logmat
COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0 25300317
/run/logmat type=DGRAM (CONNECTED)
[root@xxx rsyslog.d]# ss -x | grep logmat
u_dgr ESTAB 0 0 /run/logmat 25300317            * 0

On 18/09/2023 16:34, TG Servers via rsyslog wrote:
> I just wanted to add that in a further message as it came to my mind.
> you were faster...
> the script is definitely "slow", this is what I know for sure as it
> does quite a lot of processing/analytics in the background, so even if
> you trigger it from command line it can take half a sec or so.... I
> can't change that, it needs to do what it does, I didn't write it
> though it can handle manual fast F5 triggers in the browser without
> issue and then it 111s when there are 2 requests incoming...
> I thought rsyslog might handle that just well via the queue...
> but then this might eventually really be the issue, and if it is, is
> there anything to mitigate this from rsyslog side (in terms of own
> queue for that socket or something in that direction)?
> ok, will enable impstats, too when I switch back
>
> Thanks,
> Tom
>
> On 18/09/2023 16:17, Rainer Gerhards wrote:
>>> so far not a single 111 today, I let this run the until late evening,
>>> and if there is stil no 111 I will put back the python script in order
>>> because right now there are 2 possibilities, I moved the socket as
>>> said,
>>> and I skipped the script and just appended the message to a file
>>> if either of the 2 things are responsible in the end I won't understand
>>> it either :)
>> I don't know what the script does. But if it is slow, it may push back
>> to the main queue, making rsyslog unresponsive.
>>
>> This is David's concern. Tomorrow, if you re-enable, you should also
>> enable impstats as David suggested.
>>
>> Rainer
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 18, 2023, 1:00 PM
Post #17 of 33 (664 views)
Permalink
when rsyslog starts up, it generates various log messages, are they being sent
to the script?

it would really help to see the queue data from impstats

David Lang

On Mon, 18 Sep 2023, TG Servers via rsyslog wrote:

> I don't know what this is... I implemented a complete queue solution and
> it occasionally happens when there is no request but one in sight, and
> this one gets a 111 then, nothing in nginx debug log, no error to be
> seen in rsyslog log
> but one thing I realized, after a restart the first log message always,
> reproducable gets a 111
> the socket is not connected, nor listening, only after the first request
> is logged/or not logged (which is logged with 111 in nginx) the socket
> is connected and listening, so restarting rsyslog via systemd does not
> connect/listen to/on the socket
>
> the rsyslog debug log just tells us this :
> 6289.088037540:main thread    : imuxsock.c: imuxsock: Opened UNIX socket
> '/run/logmat' (fd 6).
>
> [root@xxx rsyslog.d]# systemctl restart rsyslog
> [root@xxx rsyslog.d]# ss -x | grep logmat
> [root@xxx rsyslog.d]# lsof /run/logmat
> COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
> rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0 25300317
> /run/logmat type=DGRAM (UNCONNECTED)
>
> make a request from browser or curl
>
> [root@xxx rsyslog.d]# lsof /run/logmat
> COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
> rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0 25300317
> /run/logmat type=DGRAM (CONNECTED)
> [root@xxx rsyslog.d]# ss -x | grep logmat
> u_dgr ESTAB 0 0 /run/logmat 25300317            * 0
>
> On 18/09/2023 16:34, TG Servers via rsyslog wrote:
>> I just wanted to add that in a further message as it came to my mind.
>> you were faster...
>> the script is definitely "slow", this is what I know for sure as it
>> does quite a lot of processing/analytics in the background, so even if
>> you trigger it from command line it can take half a sec or so.... I
>> can't change that, it needs to do what it does, I didn't write it
>> though it can handle manual fast F5 triggers in the browser without
>> issue and then it 111s when there are 2 requests incoming...
>> I thought rsyslog might handle that just well via the queue...
>> but then this might eventually really be the issue, and if it is, is
>> there anything to mitigate this from rsyslog side (in terms of own
>> queue for that socket or something in that direction)?
>> ok, will enable impstats, too when I switch back
>>
>> Thanks,
>> Tom
>>
>> On 18/09/2023 16:17, Rainer Gerhards wrote:
>>>> so far not a single 111 today, I let this run the until late evening,
>>>> and if there is stil no 111 I will put back the python script in order
>>>> because right now there are 2 possibilities, I moved the socket as
>>>> said,
>>>> and I skipped the script and just appended the message to a file
>>>> if either of the 2 things are responsible in the end I won't understand
>>>> it either :)
>>> I don't know what the script does. But if it is slow, it may push back
>>> to the main queue, making rsyslog unresponsive.
>>>
>>> This is David's concern. Tomorrow, if you re-enable, you should also
>>> enable impstats as David suggested.
>>>
>>> Rainer
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 20, 2023, 7:15 PM
Post #18 of 33 (660 views)
Permalink
the only way I was able to fix this was to use a dedicated socket
created via systemd and passed via systemd to rsyslog
since then it is working without any issues.
although I implemented a queue, too, this did not fix the problem as
long as the socket was handled by rsyslog itself
so this is "fixed" from my point of view, I know for the future now

On 18/09/2023 21:53, TG Servers via rsyslog wrote:
> I don't know what this is... I implemented a complete queue solution
> and it occasionally happens when there is no request but one in sight,
> and this one gets a 111 then, nothing in nginx debug log, no error to
> be seen in rsyslog log
> but one thing I realized, after a restart the first log message
> always, reproducable gets a 111
> the socket is not connected, nor listening, only after the first
> request is logged/or not logged (which is logged with 111 in nginx)
> the socket is connected and listening, so restarting rsyslog via
> systemd does not connect/listen to/on the socket
>
> the rsyslog debug log just tells us this :
> 6289.088037540:main thread    : imuxsock.c: imuxsock: Opened UNIX
> socket '/run/logmat' (fd 6).
>
> [root@xxx rsyslog.d]# systemctl restart rsyslog
> [root@xxx rsyslog.d]# ss -x | grep logmat
> [root@xxx rsyslog.d]# lsof /run/logmat
> COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
> rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0 25300317
> /run/logmat type=DGRAM (UNCONNECTED)
>
> make a request from browser or curl
>
> [root@xxx rsyslog.d]# lsof /run/logmat
> COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
> rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0 25300317
> /run/logmat type=DGRAM (CONNECTED)
> [root@xxx rsyslog.d]# ss -x | grep logmat
> u_dgr ESTAB 0 0 /run/logmat 25300317            * 0
>
> On 18/09/2023 16:34, TG Servers via rsyslog wrote:
>> I just wanted to add that in a further message as it came to my mind.
>> you were faster...
>> the script is definitely "slow", this is what I know for sure as it
>> does quite a lot of processing/analytics in the background, so even
>> if you trigger it from command line it can take half a sec or so....
>> I can't change that, it needs to do what it does, I didn't write it
>> though it can handle manual fast F5 triggers in the browser without
>> issue and then it 111s when there are 2 requests incoming...
>> I thought rsyslog might handle that just well via the queue...
>> but then this might eventually really be the issue, and if it is, is
>> there anything to mitigate this from rsyslog side (in terms of own
>> queue for that socket or something in that direction)?
>> ok, will enable impstats, too when I switch back
>>
>> Thanks,
>> Tom
>>
>> On 18/09/2023 16:17, Rainer Gerhards wrote:
>>>> so far not a single 111 today, I let this run the until late evening,
>>>> and if there is stil no 111 I will put back the python script in order
>>>> because right now there are 2 possibilities, I moved the socket as
>>>> said,
>>>> and I skipped the script and just appended the message to a file
>>>> if either of the 2 things are responsible in the end I won't
>>>> understand
>>>> it either :)
>>> I don't know what the script does. But if it is slow, it may push back
>>> to the main queue, making rsyslog unresponsive.
>>>
>>> This is David's concern. Tomorrow, if you re-enable, you should also
>>> enable impstats as David suggested.
>>>
>>> Rainer
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 20, 2023, 11:22 PM
Post #19 of 33 (660 views)
Permalink
now you have journald acting as a queue, so all messages from journald will end
up delayed when your script cannot keep up. You haven't solved the problem of
the slow script, you've just added another layer of buffer to fill up before you
notice.

with rsyslog you can set the queue size to whatever you want, and you can spill
logs to disk when your queue fills up.

but no matter what you do, if you have something that is processing logs slower
than they are being generated, eventually you will run out of queue space (in
memory or on disk) and have to stop accepting new messages, or start throwing
away messages you haven't processed yet

David Lang

On Thu, 21 Sep 2023, TG Servers via rsyslog wrote:

> the only way I was able to fix this was to use a dedicated socket
> created via systemd and passed via systemd to rsyslog
> since then it is working without any issues.
> although I implemented a queue, too, this did not fix the problem as
> long as the socket was handled by rsyslog itself
> so this is "fixed" from my point of view, I know for the future now
>
> On 18/09/2023 21:53, TG Servers via rsyslog wrote:
>> I don't know what this is... I implemented a complete queue solution
>> and it occasionally happens when there is no request but one in sight,
>> and this one gets a 111 then, nothing in nginx debug log, no error to
>> be seen in rsyslog log
>> but one thing I realized, after a restart the first log message
>> always, reproducable gets a 111
>> the socket is not connected, nor listening, only after the first
>> request is logged/or not logged (which is logged with 111 in nginx)
>> the socket is connected and listening, so restarting rsyslog via
>> systemd does not connect/listen to/on the socket
>>
>> the rsyslog debug log just tells us this :
>> 6289.088037540:main thread    : imuxsock.c: imuxsock: Opened UNIX
>> socket '/run/logmat' (fd 6).
>>
>> [root@xxx rsyslog.d]# systemctl restart rsyslog
>> [root@xxx rsyslog.d]# ss -x | grep logmat
>> [root@xxx rsyslog.d]# lsof /run/logmat
>> COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
>> rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0 25300317
>> /run/logmat type=DGRAM (UNCONNECTED)
>>
>> make a request from browser or curl
>>
>> [root@xxx rsyslog.d]# lsof /run/logmat
>> COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
>> rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0 25300317
>> /run/logmat type=DGRAM (CONNECTED)
>> [root@xxx rsyslog.d]# ss -x | grep logmat
>> u_dgr ESTAB 0 0 /run/logmat 25300317            * 0
>>
>> On 18/09/2023 16:34, TG Servers via rsyslog wrote:
>>> I just wanted to add that in a further message as it came to my mind.
>>> you were faster...
>>> the script is definitely "slow", this is what I know for sure as it
>>> does quite a lot of processing/analytics in the background, so even
>>> if you trigger it from command line it can take half a sec or so....
>>> I can't change that, it needs to do what it does, I didn't write it
>>> though it can handle manual fast F5 triggers in the browser without
>>> issue and then it 111s when there are 2 requests incoming...
>>> I thought rsyslog might handle that just well via the queue...
>>> but then this might eventually really be the issue, and if it is, is
>>> there anything to mitigate this from rsyslog side (in terms of own
>>> queue for that socket or something in that direction)?
>>> ok, will enable impstats, too when I switch back
>>>
>>> Thanks,
>>> Tom
>>>
>>> On 18/09/2023 16:17, Rainer Gerhards wrote:
>>>>> so far not a single 111 today, I let this run the until late evening,
>>>>> and if there is stil no 111 I will put back the python script in order
>>>>> because right now there are 2 possibilities, I moved the socket as
>>>>> said,
>>>>> and I skipped the script and just appended the message to a file
>>>>> if either of the 2 things are responsible in the end I won't
>>>>> understand
>>>>> it either :)
>>>> I don't know what the script does. But if it is slow, it may push back
>>>> to the main queue, making rsyslog unresponsive.
>>>>
>>>> This is David's concern. Tomorrow, if you re-enable, you should also
>>>> enable impstats as David suggested.
>>>>
>>>> Rainer
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 20, 2023, 11:34 PM
Post #20 of 33 (660 views)
Permalink
I guess it works because journal always throws messages away if it cannot
deliver them quickly. Luke a very short timeout+drop queue config in
rsyslog.

Rainer

Sent from phone, thus brief.

David Lang <david@lang.hm> schrieb am Do., 21. Sept. 2023, 08:23:

> now you have journald acting as a queue, so all messages from journald
> will end
> up delayed when your script cannot keep up. You haven't solved the problem
> of
> the slow script, you've just added another layer of buffer to fill up
> before you
> notice.
>
> with rsyslog you can set the queue size to whatever you want, and you can
> spill
> logs to disk when your queue fills up.
>
> but no matter what you do, if you have something that is processing logs
> slower
> than they are being generated, eventually you will run out of queue space
> (in
> memory or on disk) and have to stop accepting new messages, or start
> throwing
> away messages you haven't processed yet
>
> David Lang
>
> On Thu, 21 Sep 2023, TG Servers via rsyslog wrote:
>
> > the only way I was able to fix this was to use a dedicated socket
> > created via systemd and passed via systemd to rsyslog
> > since then it is working without any issues.
> > although I implemented a queue, too, this did not fix the problem as
> > long as the socket was handled by rsyslog itself
> > so this is "fixed" from my point of view, I know for the future now
> >
> > On 18/09/2023 21:53, TG Servers via rsyslog wrote:
> >> I don't know what this is... I implemented a complete queue solution
> >> and it occasionally happens when there is no request but one in sight,
> >> and this one gets a 111 then, nothing in nginx debug log, no error to
> >> be seen in rsyslog log
> >> but one thing I realized, after a restart the first log message
> >> always, reproducable gets a 111
> >> the socket is not connected, nor listening, only after the first
> >> request is logged/or not logged (which is logged with 111 in nginx)
> >> the socket is connected and listening, so restarting rsyslog via
> >> systemd does not connect/listen to/on the socket
> >>
> >> the rsyslog debug log just tells us this :
> >> 6289.088037540:main thread : imuxsock.c: imuxsock: Opened UNIX
> >> socket '/run/logmat' (fd 6).
> >>
> >> [root@xxx rsyslog.d]# systemctl restart rsyslog
> >> [root@xxx rsyslog.d]# ss -x | grep logmat
> >> [root@xxx rsyslog.d]# lsof /run/logmat
> >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> >> rsyslogd 2097140 root 6u unix 0x0000000000000000 0t0 25300317
> >> /run/logmat type=DGRAM (UNCONNECTED)
> >>
> >> make a request from browser or curl
> >>
> >> [root@xxx rsyslog.d]# lsof /run/logmat
> >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> >> rsyslogd 2097140 root 6u unix 0x0000000000000000 0t0 25300317
> >> /run/logmat type=DGRAM (CONNECTED)
> >> [root@xxx rsyslog.d]# ss -x | grep logmat
> >> u_dgr ESTAB 0 0 /run/logmat 25300317 * 0
> >>
> >> On 18/09/2023 16:34, TG Servers via rsyslog wrote:
> >>> I just wanted to add that in a further message as it came to my mind.
> >>> you were faster...
> >>> the script is definitely "slow", this is what I know for sure as it
> >>> does quite a lot of processing/analytics in the background, so even
> >>> if you trigger it from command line it can take half a sec or so....
> >>> I can't change that, it needs to do what it does, I didn't write it
> >>> though it can handle manual fast F5 triggers in the browser without
> >>> issue and then it 111s when there are 2 requests incoming...
> >>> I thought rsyslog might handle that just well via the queue...
> >>> but then this might eventually really be the issue, and if it is, is
> >>> there anything to mitigate this from rsyslog side (in terms of own
> >>> queue for that socket or something in that direction)?
> >>> ok, will enable impstats, too when I switch back
> >>>
> >>> Thanks,
> >>> Tom
> >>>
> >>> On 18/09/2023 16:17, Rainer Gerhards wrote:
> >>>>> so far not a single 111 today, I let this run the until late evening,
> >>>>> and if there is stil no 111 I will put back the python script in
> order
> >>>>> because right now there are 2 possibilities, I moved the socket as
> >>>>> said,
> >>>>> and I skipped the script and just appended the message to a file
> >>>>> if either of the 2 things are responsible in the end I won't
> >>>>> understand
> >>>>> it either :)
> >>>> I don't know what the script does. But if it is slow, it may push back
> >>>> to the main queue, making rsyslog unresponsive.
> >>>>
> >>>> This is David's concern. Tomorrow, if you re-enable, you should also
> >>>> enable impstats as David suggested.
> >>>>
> >>>> Rainer
> >>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >>> POST if you DON'T LIKE THAT.
> >>
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> >> if you DON'T LIKE THAT.
> >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> > LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 21, 2023, 12:14 AM
Post #21 of 33 (660 views)
Permalink
depends on the journald config. It can be configured to queue to disk, with
limits on disk size.

David Lang

On Thu, 21 Sep 2023, Rainer Gerhards wrote:

> I guess it works because journal always throws messages away if it cannot
> deliver them quickly. Luke a very short timeout+drop queue config in
> rsyslog.
>
> Rainer
>
> Sent from phone, thus brief.
>
> David Lang <david@lang.hm> schrieb am Do., 21. Sept. 2023, 08:23:
>
>> now you have journald acting as a queue, so all messages from journald
>> will end
>> up delayed when your script cannot keep up. You haven't solved the problem
>> of
>> the slow script, you've just added another layer of buffer to fill up
>> before you
>> notice.
>>
>> with rsyslog you can set the queue size to whatever you want, and you can
>> spill
>> logs to disk when your queue fills up.
>>
>> but no matter what you do, if you have something that is processing logs
>> slower
>> than they are being generated, eventually you will run out of queue space
>> (in
>> memory or on disk) and have to stop accepting new messages, or start
>> throwing
>> away messages you haven't processed yet
>>
>> David Lang
>>
>> On Thu, 21 Sep 2023, TG Servers via rsyslog wrote:
>>
>>> the only way I was able to fix this was to use a dedicated socket
>>> created via systemd and passed via systemd to rsyslog
>>> since then it is working without any issues.
>>> although I implemented a queue, too, this did not fix the problem as
>>> long as the socket was handled by rsyslog itself
>>> so this is "fixed" from my point of view, I know for the future now
>>>
>>> On 18/09/2023 21:53, TG Servers via rsyslog wrote:
>>>> I don't know what this is... I implemented a complete queue solution
>>>> and it occasionally happens when there is no request but one in sight,
>>>> and this one gets a 111 then, nothing in nginx debug log, no error to
>>>> be seen in rsyslog log
>>>> but one thing I realized, after a restart the first log message
>>>> always, reproducable gets a 111
>>>> the socket is not connected, nor listening, only after the first
>>>> request is logged/or not logged (which is logged with 111 in nginx)
>>>> the socket is connected and listening, so restarting rsyslog via
>>>> systemd does not connect/listen to/on the socket
>>>>
>>>> the rsyslog debug log just tells us this :
>>>> 6289.088037540:main thread : imuxsock.c: imuxsock: Opened UNIX
>>>> socket '/run/logmat' (fd 6).
>>>>
>>>> [root@xxx rsyslog.d]# systemctl restart rsyslog
>>>> [root@xxx rsyslog.d]# ss -x | grep logmat
>>>> [root@xxx rsyslog.d]# lsof /run/logmat
>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>>> rsyslogd 2097140 root 6u unix 0x0000000000000000 0t0 25300317
>>>> /run/logmat type=DGRAM (UNCONNECTED)
>>>>
>>>> make a request from browser or curl
>>>>
>>>> [root@xxx rsyslog.d]# lsof /run/logmat
>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>>> rsyslogd 2097140 root 6u unix 0x0000000000000000 0t0 25300317
>>>> /run/logmat type=DGRAM (CONNECTED)
>>>> [root@xxx rsyslog.d]# ss -x | grep logmat
>>>> u_dgr ESTAB 0 0 /run/logmat 25300317 * 0
>>>>
>>>> On 18/09/2023 16:34, TG Servers via rsyslog wrote:
>>>>> I just wanted to add that in a further message as it came to my mind.
>>>>> you were faster...
>>>>> the script is definitely "slow", this is what I know for sure as it
>>>>> does quite a lot of processing/analytics in the background, so even
>>>>> if you trigger it from command line it can take half a sec or so....
>>>>> I can't change that, it needs to do what it does, I didn't write it
>>>>> though it can handle manual fast F5 triggers in the browser without
>>>>> issue and then it 111s when there are 2 requests incoming...
>>>>> I thought rsyslog might handle that just well via the queue...
>>>>> but then this might eventually really be the issue, and if it is, is
>>>>> there anything to mitigate this from rsyslog side (in terms of own
>>>>> queue for that socket or something in that direction)?
>>>>> ok, will enable impstats, too when I switch back
>>>>>
>>>>> Thanks,
>>>>> Tom
>>>>>
>>>>> On 18/09/2023 16:17, Rainer Gerhards wrote:
>>>>>>> so far not a single 111 today, I let this run the until late evening,
>>>>>>> and if there is stil no 111 I will put back the python script in
>> order
>>>>>>> because right now there are 2 possibilities, I moved the socket as
>>>>>>> said,
>>>>>>> and I skipped the script and just appended the message to a file
>>>>>>> if either of the 2 things are responsible in the end I won't
>>>>>>> understand
>>>>>>> it either :)
>>>>>> I don't know what the script does. But if it is slow, it may push back
>>>>>> to the main queue, making rsyslog unresponsive.
>>>>>>
>>>>>> This is David's concern. Tomorrow, if you re-enable, you should also
>>>>>> enable impstats as David suggested.
>>>>>>
>>>>>> Rainer
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>> POST if you DON'T LIKE THAT.
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>> if you DON'T LIKE THAT.
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T
>>> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 21, 2023, 12:25 AM
Post #22 of 33 (660 views)
Permalink
 I did not get a single message from you David regarding that, that
confused me quite a bit as Rainer mentioned you already before, now I
know why :
450 4.7.25 Client host rejected: cannot find your hostname,
[66.167.xxx.xxx]; from=<david@lang.hm> to=<srvrs@prvtmail.net>
proto=ESMTP helo=<mail.lang.hm>
made an exception now

But to the point, I sadly don't get what both of you are telling me now.
This has nothing to do with journald. It just did not work when the
socket was created by rsyslog. If this is/was a rsyslog or a nginx
problem does not matter in the end to me, as this had to be fixed.

I am using a dedicated socket, completely aside from sysSock, and a
dedicated queue. sysSock ist not involved, nginx does not even log a
single line to journald, so how should journald act as a queue here, or
being negatively affected when it does not even receive a single message
of the process involved? It can't throw something away it does not have.
This socket is only used by and for this process and nothing else.
I also won't likely run out of queue space because this is not a process
that is 24/7 under a full load scenario. That might happen under an
attack maybe, but otherwise I don't see that happening
If I am seeing things wrong then I would be happy if it could be made
clear to me because as of now I do not see the problem.

Thanks,
Thomas


On 21/09/2023 08:34, Rainer Gerhards wrote:
> I guess it works because journal always throws messages away if it
> cannot deliver them quickly. Luke a very short timeout+drop queue
> config in rsyslog.
>
> Rainer
>
> Sent from phone, thus brief.
>
> David Lang <david@lang.hm> schrieb am Do., 21. Sept. 2023, 08:23:
>
> now you have journald acting as a queue, so all messages from
> journald will end
> up delayed when your script cannot keep up. You haven't solved the
> problem of
> the slow script, you've just added another layer of buffer to fill
> up before you
> notice.
>
> with rsyslog you can set the queue size to whatever you want, and
> you can spill
> logs to disk when your queue fills up.
>
> but no matter what you do, if you have something that is
> processing logs slower
> than they are being generated, eventually you will run out of
> queue space (in
> memory or on disk) and have to stop accepting new messages, or
> start throwing
> away messages you haven't processed yet
>
> David Lang
>
> On Thu, 21 Sep 2023, TG Servers via rsyslog wrote:
>
> > the only way I was able to fix this was to use a dedicated socket
> > created via systemd and passed via systemd to rsyslog
> > since then it is working without any issues.
> > although I implemented a queue, too, this did not fix the
> problem as
> > long as the socket was handled by rsyslog itself
> > so this is "fixed" from my point of view, I know for the future now
> >
> > On 18/09/2023 21:53, TG Servers via rsyslog wrote:
> >> I don't know what this is... I implemented a complete queue
> solution
> >> and it occasionally happens when there is no request but one in
> sight,
> >> and this one gets a 111 then, nothing in nginx debug log, no
> error to
> >> be seen in rsyslog log
> >> but one thing I realized, after a restart the first log message
> >> always, reproducable gets a 111
> >> the socket is not connected, nor listening, only after the first
> >> request is logged/or not logged (which is logged with 111 in
> nginx)
> >> the socket is connected and listening, so restarting rsyslog via
> >> systemd does not connect/listen to/on the socket
> >>
> >> the rsyslog debug log just tells us this :
> >> 6289.088037540:main thread    : imuxsock.c: imuxsock: Opened UNIX
> >> socket '/run/logmat' (fd 6).
> >>
> >> [root@xxx rsyslog.d]# systemctl restart rsyslog
> >> [root@xxx rsyslog.d]# ss -x | grep logmat
> >> [root@xxx rsyslog.d]# lsof /run/logmat
> >> COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF
> NODE NAME
> >> rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0
> 25300317
> >> /run/logmat type=DGRAM (UNCONNECTED)
> >>
> >> make a request from browser or curl
> >>
> >> [root@xxx rsyslog.d]# lsof /run/logmat
> >> COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF
> NODE NAME
> >> rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0
> 25300317
> >> /run/logmat type=DGRAM (CONNECTED)
> >> [root@xxx rsyslog.d]# ss -x | grep logmat
> >> u_dgr ESTAB 0 0 /run/logmat 25300317            * 0
> >>
> >> On 18/09/2023 16:34, TG Servers via rsyslog wrote:
> >>> I just wanted to add that in a further message as it came to
> my mind.
> >>> you were faster...
> >>> the script is definitely "slow", this is what I know for sure
> as it
> >>> does quite a lot of processing/analytics in the background, so
> even
> >>> if you trigger it from command line it can take half a sec or
> so....
> >>> I can't change that, it needs to do what it does, I didn't
> write it
> >>> though it can handle manual fast F5 triggers in the browser
> without
> >>> issue and then it 111s when there are 2 requests incoming...
> >>> I thought rsyslog might handle that just well via the queue...
> >>> but then this might eventually really be the issue, and if it
> is, is
> >>> there anything to mitigate this from rsyslog side (in terms of
> own
> >>> queue for that socket or something in that direction)?
> >>> ok, will enable impstats, too when I switch back
> >>>
> >>> Thanks,
> >>> Tom
> >>>
> >>> On 18/09/2023 16:17, Rainer Gerhards wrote:
> >>>>> so far not a single 111 today, I let this run the until late
> evening,
> >>>>> and if there is stil no 111 I will put back the python
> script in order
> >>>>> because right now there are 2 possibilities, I moved the
> socket as
> >>>>> said,
> >>>>> and I skipped the script and just appended the message to a file
> >>>>> if either of the 2 things are responsible in the end I won't
> >>>>> understand
> >>>>> it either :)
> >>>> I don't know what the script does. But if it is slow, it may
> push back
> >>>> to the main queue, making rsyslog unresponsive.
> >>>>
> >>>> This is David's concern. Tomorrow, if you re-enable, you
> should also
> >>>> enable impstats as David suggested.
> >>>>
> >>>> Rainer
> >>>
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >>> POST if you DON'T LIKE THAT.
> >>
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
> NOT POST
> >> if you DON'T LIKE THAT.
> >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> a myriad of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you DON'T
> > LIKE THAT.
>

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 21, 2023, 1:55 AM
Post #23 of 33 (660 views)
Permalink
if you are sending logs to journald and having journald send logs to syslog, you
are using journald as a queue for the delivery

when you were delivering directly to rsyslog, what was probably happening (we
don't know because you never enabled impstats to see) is that the logs were
arriving, but because your script takes so long to process each log message, the
queue was filling up, and when the queue is full, rsyslog cannot accept another
message, and that results in the error that you are reporting.

you did not configure rsyslog to use a separate queue for the logs from this
socket, so as they arrived they got added to the main queue along with all other
logs.

rsyslog has options to tell it to throw away logs when it's too busy (and even
can prioritize which ones it throws away). you can also configure it to write
logs to disk when the memory queue gets too full. But eventually you will run
out of disk space if you keep getting logs faster than you can process them.

David Lang



On Thu, 21 Sep 2023, TG Servers wrote:

>  I did not get a single message from you David regarding that, that confused
> me quite a bit as Rainer mentioned you already before, now I know why :
> 450 4.7.25 Client host rejected: cannot find your hostname, [66.167.xxx.xxx];
> from=<david@lang.hm> to=<srvrs@prvtmail.net> proto=ESMTP helo=<mail.lang.hm>
> made an exception now
>
> But to the point, I sadly don't get what both of you are telling me now. This
> has nothing to do with journald. It just did not work when the socket was
> created by rsyslog. If this is/was a rsyslog or a nginx problem does not
> matter in the end to me, as this had to be fixed.
>
> I am using a dedicated socket, completely aside from sysSock, and a dedicated
> queue. sysSock ist not involved, nginx does not even log a single line to
> journald, so how should journald act as a queue here, or being negatively
> affected when it does not even receive a single message of the process
> involved? It can't throw something away it does not have.
> This socket is only used by and for this process and nothing else.
> I also won't likely run out of queue space because this is not a process that
> is 24/7 under a full load scenario. That might happen under an attack maybe,
> but otherwise I don't see that happening
> If I am seeing things wrong then I would be happy if it could be made clear
> to me because as of now I do not see the problem.
>
> Thanks,
> Thomas
>
>
> On 21/09/2023 08:34, Rainer Gerhards wrote:
>> I guess it works because journal always throws messages away if it cannot
>> deliver them quickly. Luke a very short timeout+drop queue config in
>> rsyslog.
>>
>> Rainer
>>
>> Sent from phone, thus brief.
>>
>> David Lang <david@lang.hm> schrieb am Do., 21. Sept. 2023, 08:23:
>>
>> now you have journald acting as a queue, so all messages from
>> journald will end
>> up delayed when your script cannot keep up. You haven't solved the
>> problem of
>> the slow script, you've just added another layer of buffer to fill
>> up before you
>> notice.
>>
>> with rsyslog you can set the queue size to whatever you want, and
>> you can spill
>> logs to disk when your queue fills up.
>>
>> but no matter what you do, if you have something that is
>> processing logs slower
>> than they are being generated, eventually you will run out of
>> queue space (in
>> memory or on disk) and have to stop accepting new messages, or
>> start throwing
>> away messages you haven't processed yet
>>
>> David Lang
>>
>> On Thu, 21 Sep 2023, TG Servers via rsyslog wrote:
>>
>> > the only way I was able to fix this was to use a dedicated socket
>> > created via systemd and passed via systemd to rsyslog
>> > since then it is working without any issues.
>> > although I implemented a queue, too, this did not fix the
>> problem as
>> > long as the socket was handled by rsyslog itself
>> > so this is "fixed" from my point of view, I know for the future now
>> >
>> > On 18/09/2023 21:53, TG Servers via rsyslog wrote:
>> >> I don't know what this is... I implemented a complete queue
>> solution
>> >> and it occasionally happens when there is no request but one in
>> sight,
>> >> and this one gets a 111 then, nothing in nginx debug log, no
>> error to
>> >> be seen in rsyslog log
>> >> but one thing I realized, after a restart the first log message
>> >> always, reproducable gets a 111
>> >> the socket is not connected, nor listening, only after the first
>> >> request is logged/or not logged (which is logged with 111 in
>> nginx)
>> >> the socket is connected and listening, so restarting rsyslog via
>> >> systemd does not connect/listen to/on the socket
>> >>
>> >> the rsyslog debug log just tells us this :
>> >> 6289.088037540:main thread    : imuxsock.c: imuxsock: Opened UNIX
>> >> socket '/run/logmat' (fd 6).
>> >>
>> >> [root@xxx rsyslog.d]# systemctl restart rsyslog
>> >> [root@xxx rsyslog.d]# ss -x | grep logmat
>> >> [root@xxx rsyslog.d]# lsof /run/logmat
>> >> COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF
>> NODE NAME
>> >> rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0
>> 25300317
>> >> /run/logmat type=DGRAM (UNCONNECTED)
>> >>
>> >> make a request from browser or curl
>> >>
>> >> [root@xxx rsyslog.d]# lsof /run/logmat
>> >> COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF
>> NODE NAME
>> >> rsyslogd 2097140 root    6u  unix 0x0000000000000000      0t0
>> 25300317
>> >> /run/logmat type=DGRAM (CONNECTED)
>> >> [root@xxx rsyslog.d]# ss -x | grep logmat
>> >> u_dgr ESTAB 0 0 /run/logmat 25300317            * 0
>> >>
>> >> On 18/09/2023 16:34, TG Servers via rsyslog wrote:
>> >>> I just wanted to add that in a further message as it came to
>> my mind.
>> >>> you were faster...
>> >>> the script is definitely "slow", this is what I know for sure
>> as it
>> >>> does quite a lot of processing/analytics in the background, so
>> even
>> >>> if you trigger it from command line it can take half a sec or
>> so....
>> >>> I can't change that, it needs to do what it does, I didn't
>> write it
>> >>> though it can handle manual fast F5 triggers in the browser
>> without
>> >>> issue and then it 111s when there are 2 requests incoming...
>> >>> I thought rsyslog might handle that just well via the queue...
>> >>> but then this might eventually really be the issue, and if it
>> is, is
>> >>> there anything to mitigate this from rsyslog side (in terms of
>> own
>> >>> queue for that socket or something in that direction)?
>> >>> ok, will enable impstats, too when I switch back
>> >>>
>> >>> Thanks,
>> >>> Tom
>> >>>
>> >>> On 18/09/2023 16:17, Rainer Gerhards wrote:
>> >>>>> so far not a single 111 today, I let this run the until late
>> evening,
>> >>>>> and if there is stil no 111 I will put back the python
>> script in order
>> >>>>> because right now there are 2 possibilities, I moved the
>> socket as
>> >>>>> said,
>> >>>>> and I skipped the script and just appended the message to a file
>> >>>>> if either of the 2 things are responsible in the end I won't
>> >>>>> understand
>> >>>>> it either :)
>> >>>> I don't know what the script does. But if it is slow, it may
>> push back
>> >>>> to the main queue, making rsyslog unresponsive.
>> >>>>
>> >>>> This is David's concern. Tomorrow, if you re-enable, you
>> should also
>> >>>> enable impstats as David suggested.
>> >>>>
>> >>>> Rainer
>> >>>
>> >>> _______________________________________________
>> >>> rsyslog mailing list
>> >>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>> http://www.rsyslog.com/professional-services/
>> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> >>> POST if you DON'T LIKE THAT.
>> >>
>> >> _______________________________________________
>> >> rsyslog mailing list
>> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> http://www.rsyslog.com/professional-services/
>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>> NOT POST
>> >> if you DON'T LIKE THAT.
>> >
>> > _______________________________________________
>> > rsyslog mailing list
>> > https://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>> a myriad of
>> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T
>> > LIKE THAT.
>>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 21, 2023, 2:00 AM
Post #24 of 33 (660 views)
Permalink
On Thu, 21 Sep 2023, TG Servers wrote:

>  I did not get a single message from you David regarding that, that confused
> me quite a bit as Rainer mentioned you already before, now I know why :
> 450 4.7.25 Client host rejected: cannot find your hostname, [66.167.xxx.xxx];
> from=<david@lang.hm> to=<srvrs@prvtmail.net> proto=ESMTP helo=<mail.lang.hm>
> made an exception now

hmm, I haven't made any DNS changes or IP changes for a couple of years (since
the last time my ISP broke reverse DNS)

lookups against google DNS servers seem to work

dlang@dlang-mobile:~$ nslookup mail.lang.hm 8.8.8.8
Server: 8.8.8.8
Address: 8.8.8.8#53

Non-authoritative answer:
Name: mail.lang.hm
Address: 66.167.227.134

dlang@dlang-mobile:~$ nslookup 66.167.227.134
134.227.167.66.in-addr.arpa name = gw.lang.hm.

Authoritative answers can be found from:


what is the check that you are doing? I'm not running into problems with anyone
else that I know of.

David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Repeated 111 to rsyslog UDS from nginx [ In reply to ]
rsyslog at lists
Sep 21, 2023, 2:06 AM
Post #25 of 33 (660 views)
Permalink
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
1 2  View All

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal