Mailing List Archive

Why not log to local TCP socket?

Mariusz Kruk
Ekspert ds. Bezpiecze?stwa IT
COMP S.A.
Pion Cyberbezpiecze?stwa i Zarz?dzania Ryzykiem
e-mail: mariusz.kruk@comp.com.pl
e-mail: mariusz.kruk@safecomp.com
tel: +48 608 623 299

On 20.09.2020 18:13, Vitaly Repin via rsyslog wrote:
> What options do I have to log to rsyslog directly without having
> journald in between?
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hello,

Den mån 21 sep. 2020 kl 12:20 skrev Mariusz Kruk via rsyslog
<rsyslog@lists.adiscon.com>:

> Why not log to local TCP socket?

glibc does not allow that. Is there any library I can use to log via TCP socket?

Of course I can cherry pick the code from rsyslog implementation
(imptcp module). But is it the easiest/best approach?

I am a little bit surprised that I was not able to find a ready-to-use
non-glibc implementation of a syslog client library for C/C++.

> > What options do I have to log to rsyslog directly without having
> > journald in between?

--
WBR & WBW, Vitaly
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On 21.09.2020 12:32, Vitaly Repin wrote:
> Hello,
>
> Den mån 21 sep. 2020 kl 12:20 skrev Mariusz Kruk via rsyslog
> <rsyslog@lists.adiscon.com>:
>
>> Why not log to local TCP socket?
> glibc does not allow that. Is there any library I can use to log via TCP socket?
Well, there are many different logging frameworks. I suppose, since
you're looking into glibc sources, you're writing in C so I'd check, for
example log4c. Unless you're doing something very low-level, it's
usually useful to have a decent logging framework.
>
> Of course I can cherry pick the code from rsyslog implementation
> (imptcp module). But is it the easiest/best approach?
>
> I am a little bit surprised that I was not able to find a ready-to-use
> non-glibc implementation of a syslog client library for C/C++.

Maybe you're looking for a wrong thing. :-)

Logging framework is more than just a syslog client.


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hello,

Den mån 21 sep. 2020 kl 13:55 skrev Mariusz Kruk <mkr@safecomp.com>:

> >> Why not log to local TCP socket?
> > glibc does not allow that. Is there any library I can use to log via TCP socket?
> Well, there are many different logging frameworks. I suppose, since
> you're looking into glibc sources, you're writing in C so I'd check, for
> example log4c. Unless you're doing something very low-level, it's
> usually useful to have a decent logging framework.

Yes. But log4c uses glibc inside, looks like (Checked
src/appender_type_syslog.c)

I work in a project with its own logging framework. And I am in
position to change its implementation if needed.

> > Of course I can cherry pick the code from rsyslog implementation
> > (imptcp module). But is it the easiest/best approach?
> >
> > I am a little bit surprised that I was not able to find a ready-to-use
> > non-glibc implementation of a syslog client library for C/C++.
>
> Maybe you're looking for a wrong thing. :-)

> Logging framework is more than just a syslog client.

Yep. Usually they rely on glibc implementation to deliver syslog
logging functionality. Which logs through /dev/log only:
https://sourceware.org/git/?p=glibc.git;a=blob;f=misc/syslog.c;h=2cc63ef287a71fc6447774773adbffddaae514f3;hb=HEAD
(check openlog_internal function. _PATH_LOG is set to "/dev/log" in a
header file).

--
WBR & WBW, Vitaly
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On 21.09.2020 14:10, Vitaly Repin wrote:
> Why not log to local TCP socket?
>>> glibc does not allow that. Is there any library I can use to log via TCP socket?
>> Well, there are many different logging frameworks. I suppose, since
>> you're looking into glibc sources, you're writing in C so I'd check, for
>> example log4c. Unless you're doing something very low-level, it's
>> usually useful to have a decent logging framework.
> Yes. But log4c uses glibc inside, looks like (Checked
> src/appender_type_syslog.c)
>
> I work in a project with its own logging framework. And I am in
> position to change its implementation if needed.

Ehhh. Thought that if someone does "something like log4j", he would
indeed add remote syslog destination.

Anyway, implementing such thing should not be very hard given relative
simplicity of syslog RFC.

But indeed you're right, I assumed that there are more readily-available
C-based components for this as is usually the case with other languages.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Liblogging can do this.

Rainer

Sent from phone, thus brief.

Vitaly Repin via rsyslog <rsyslog@lists.adiscon.com> schrieb am Mo., 21.
Sep. 2020, 14:10:

> Hello,
>
> Den mån 21 sep. 2020 kl 13:55 skrev Mariusz Kruk <mkr@safecomp.com>:
>
> > >> Why not log to local TCP socket?
> > > glibc does not allow that. Is there any library I can use to log via
> TCP socket?
> > Well, there are many different logging frameworks. I suppose, since
> > you're looking into glibc sources, you're writing in C so I'd check, for
> > example log4c. Unless you're doing something very low-level, it's
> > usually useful to have a decent logging framework.
>
> Yes. But log4c uses glibc inside, looks like (Checked
> src/appender_type_syslog.c)
>
> I work in a project with its own logging framework. And I am in
> position to change its implementation if needed.
>
> > > Of course I can cherry pick the code from rsyslog implementation
> > > (imptcp module). But is it the easiest/best approach?
> > >
> > > I am a little bit surprised that I was not able to find a ready-to-use
> > > non-glibc implementation of a syslog client library for C/C++.
> >
> > Maybe you're looking for a wrong thing. :-)
>
> > Logging framework is more than just a syslog client.
>
> Yep. Usually they rely on glibc implementation to deliver syslog
> logging functionality. Which logs through /dev/log only:
>
> https://sourceware.org/git/?p=glibc.git;a=blob;f=misc/syslog.c;h=2cc63ef287a71fc6447774773adbffddaae514f3;hb=HEAD
> (check openlog_internal function. _PATH_LOG is set to "/dev/log" in a
> header file).
>
> --
> WBR & WBW, Vitaly
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hello,

Den mån 21 sep. 2020 kl 17:26 skrev Rainer Gerhards <rgerhards@hq.adiscon.com>:

> Liblogging can do this.

Thanks! It looks exactly as an answer I was looking for!

I am testing it right now. Look great. And has support for RFC 3195 (RELP) also!

Do I understand it right from the COPYING that it is licensed under BSD license?

--
WBR & WBW, Vitaly
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

> Den mån 21 sep. 2020 kl 17:26 skrev Rainer Gerhards <rgerhards@hq.adiscon.com>:
>
> > Liblogging can do this.
>
> Thanks! It looks exactly as an answer I was looking for!
>
> I am testing it right now. Look great. And has support for RFC 3195 (RELP) also!

Note: RFC3195 is NOT RELP. 3195 is a very early and (for practical
reasons) now-abandoned syslog standard by IETF. RELP is a
"proprietary" "standard" but well-alive ;-)
>
> Do I understand it right from the COPYING that it is licensed under BSD license?

Yes!

HTH
Rainer
>
> --
> WBR & WBW, Vitaly
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hello,

Den tis 22 sep. 2020 kl 09:52 skrev Rainer Gerhards <rgerhards@hq.adiscon.com>:
> Note: RFC3195 is NOT RELP. 3195 is a very early and (for practical
> reasons) now-abandoned syslog standard by IETF. RELP is a
> "proprietary" "standard" but well-alive ;-)

Aha! Thanks for clarification. So, if I want to send messages through
RELP, I have to use librelp?

--
WBR & WBW, Vitaly
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

> > Note: RFC3195 is NOT RELP. 3195 is a very early and (for practical
> > reasons) now-abandoned syslog standard by IETF. RELP is a
> > "proprietary" "standard" but well-alive ;-)
>
> Aha! Thanks for clarification. So, if I want to send messages through
> RELP, I have to use librelp?

I checked to be sure: currently there is no RELP channel driver. The
base idea was that liblogging-stdlog provides a route to
file/journal/unix socket and rsyslog picks up things from the unix
socket and then forwards it via RELP. This is superior to a channel
driver, because this makes operation async, which usually is what you
want.

If you need some more info on the overall idea, let me know. I'd do a
small prep (maybe we can jump into a short conf call, looks like you
are in EU time zones which should make that fairly easy. If we take
that route, I'd like to record so others can benefit as well. Comments
appreciated.

Rainer
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hello,

Den tis 22 sep. 2020 kl 10:59 skrev Rainer Gerhards <rgerhards@hq.adiscon.com>:

> I checked to be sure: currently there is no RELP channel driver. The
> base idea was that liblogging-stdlog provides a route to
> file/journal/unix socket and rsyslog picks up things from the unix
> socket and then forwards it via RELP. This is superior to a channel
> driver, because this makes operation async, which usually is what you
> want.
>
> If you need some more info on the overall idea, let me know. I'd do a
> small prep (maybe we can jump into a short conf call, looks like you
> are in EU time zones which should make that fairly easy. If we take
> that route, I'd like to record so others can benefit as well. Comments
> appreciated.

This sounds absolutely fantastic! I will reach you in a private mail.

It would be really great to know more about reliable logging via rsyslogd.
And I believe it will be very useful knowledge to many software developers.

The fact that logging through syslog is not always reliable is not a
well-known fact, I am afraid.

Do I understand it right that if I do not want to ever experience a message loss
if I log through rsyslogd (e.g., if I log financial transactions or
implement smth like flight recorder)
I have to design the system in the following way:

1. Apps log directly to rsyslog through unix socket
2. SysSock.FlowControl is enabled in imuxsock / rsyslogd
3. rsyslogd forwards the data to another rsyslog server through RELP protocol

?

If I can keep logs in the same node where the app is running is it
enough to implement just the first two steps
and let rsyslog write logs using the omfile module? Or can it start to
drop messages after a certain logging rate is reached?

I tested with loggen utility from syslog-ng package the local
configuration (app logs to rsyslogd through UNIX socket
and rsyslogd writes to the files directly). In this configuration
there was always a logging rate after which rsyslog started to drop
messages.
And this makes perfect sense for async operations, I believe. If there
is no way to stop the log source, rsyslog has no other option than to
drop logs.
Disk writing speed is limited.

But if I activate SysSock.FlowControl, will it save the party (for the
cost of blocking log producing processes sometimes)?

And I had an idea to test the 3rd setup: app sends log directly to
rsyslog through RELP. But it looks like this use case is not what RELP
author had in mind when designing RELP ? :-)

--
WBR & WBW, Vitaly
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.