Mailing List Archive

rsyslog does not change it's config during a run (with the exception of the
table_lookup() function, which may work for you)

syslog can filter on anything it knows about, so if you fetch from journald with
imjournal so that you can see all the metadata that journald takes the time to
lookup, you can filter on any of it. If you have journald write to a socket for
rsyslog to read, journald doesn't send that metadata (and refuses to consider
doing so, I've asked)

does this answer your questions?

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hello David,Indeed the information you have provided is useful. Another follow up question.
In case of using imjournal to filter systemd-journald log that is being forwarded to rsyslog-v8, what property I can use in rsyslog config for such log filtering? What if the journald log entry contain custom field that I want to use as the filtering condition?

Thanks,Patrick

On Friday, January 3, 2020, 5:36:16 p.m. PST, David Lang <david@lang.hm> wrote:

rsyslog does not change it's config during a run (with the exception of the
table_lookup() function, which may work for you)

syslog can filter on anything it knows about, so if you fetch from journald with
imjournal so that you can see all the metadata that journald takes the time to
lookup, you can filter on any of it. If you have journald write to a socket for
rsyslog to read, journald doesn't send that metadata (and refuses to consider
doing so, I've asked)

does this answer your questions?

David Lang

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

log the message with the template RSYSLOG_DebugFormat and it will show you every
value it has parsed. You may need to use mmnormalize or mmjsonparse (the first
is a batter long-term option as it is far more flexible) to extract the values
from json if they aren't already parsed. look for a json string in the $!
section, every value there can be individually addressed.

David Lang


On Sun, 2 Feb 2020,
Patrick Leung wrote:

> Date: Sun, 2 Feb 2020 09:30:55 +0000 (UTC)
> From: Patrick Leung <leungsk38@yahoo.com>
> To: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>,
> David Lang <david@lang.hm>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> Hello David,Indeed the information you have provided is useful. Another follow up question.
> In case of using imjournal to filter systemd-journald log that is being forwarded to rsyslog-v8, what property I can use in rsyslog config for such log filtering? What if the journald log entry contain custom field that I want to use as the filtering condition?
>
> Thanks,Patrick
>
> On Friday, January 3, 2020, 5:36:16 p.m. PST, David Lang <david@lang.hm> wrote:
>
> rsyslog does not change it's config during a run (with the exception of the
> table_lookup() function, which may work for you)
>
> syslog can filter on anything it knows about, so if you fetch from journald with
> imjournal so that you can see all the metadata that journald takes the time to
> lookup, you can filter on any of it. If you have journald write to a socket for
> rsyslog to read, journald doesn't send that metadata (and refuses to consider
> doing so, I've asked)
>
> does this answer your questions?
>
> David Lang
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Thank you for your clarification once again. If I understand it right, the use of RSYSLOG_DebugFormat template gives me the generated syslog file that contains structured journald log content, filtering is handled prior to output log file generation.
What I would like to achieve:forward journald log to rsyslog, and use rsyslog's filtering capabilities to filter journald master log file to individual log file base on matched ruleset in rsyslog. I may want to filter journald log base on systemd units and container services to their individual log files.
1) Is it possible to load both imuxsock and imjournal modules when starting rsyslog,   and have these modules process journald log and generate individual log file   base on the matching ruleset?2) If imjournal is loaded, I am not crystal clear on rsyslog config syntax   to set up the ruleset for structured journald log entry filtering.   Essentially I am looking for the equivalent of, for example, "journalctl CONTAINER_ID=<id> > /path/to/container_id_log".3) $programname property (which is part of TAG stemmed from MSG syslog fornat AFAIK) is available for ruleset matching when imuxsock is used?
Thanks again for help.
Thanks,Patrick
On Sunday, February 2, 2020, 04:59:50 a.m. EST, David Lang <david@lang.hm> wrote:

log the message with the template RSYSLOG_DebugFormat and it will show you every
value it has parsed. You may need to use mmnormalize or mmjsonparse (the first
is a batter long-term option as it is far more flexible) to extract the values
from json if they aren't already parsed. look for a json string in the $!
section, every value there can be individually addressed.

David Lang


  On Sun, 2 Feb 2020,
Patrick Leung wrote:

> Date: Sun, 2 Feb 2020 09:30:55 +0000 (UTC)
> From: Patrick Leung <leungsk38@yahoo.com>
> To: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>,
>    David Lang <david@lang.hm>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> Hello David,Indeed the information you have provided is useful. Another follow up question.
> In case of using imjournal to filter systemd-journald log that is being forwarded to rsyslog-v8, what property I can use in rsyslog config for such log filtering? What if the journald log entry contain custom field that I want to use as the filtering condition?
>
> Thanks,Patrick
>
>    On Friday, January 3, 2020, 5:36:16 p.m. PST, David Lang <david@lang.hm> wrote:
>
> rsyslog does not change it's config during a run (with the exception of the
> table_lookup() function, which may work for you)
>
> syslog can filter on anything it knows about, so if you fetch from journald with
> imjournal so that you can see all the metadata that journald takes the time to
> lookup, you can filter on any of it. If you have journald write to a socket for
> rsyslog to read, journald doesn't send that metadata (and refuses to consider
> doing so, I've asked)
>
> does this answer your questions?
>
> David Lang
>

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On Mon, 3 Feb 2020, Patrick Leung wrote:

> Thank you for your clarification once again. If I understand it right, the
> use of RSYSLOG_DebugFormat template gives me the generated syslog file that
> contains structured journald log content, filtering is handled prior to
> output log file generation.

not quite, filtering happens how you define it in the config file.

the debug format shows you what the contents of all the variables are at the
point in the config file that you write the log with that format.

This lets you see the contents to understand what your filters have to work
with.

filters can be the traditional facility.severity (i.e. mail.info) format, or
they can be much more powerful if..then formats

> What I would like to achieve:forward journald log to rsyslog, and use
> rsyslog's filtering capabilities to filter journald master log file to
> individual log file base on matched ruleset in rsyslog. I may want to filter
> journald log base on systemd units and container services to their individual
> log files.

that's trivail

> 1) Is it possible to load both imuxsock and imjournal modules when starting
> rsyslog,

yes, but if you have journald deliver the messages to imuxsock and have rsyslog
fetch the messages via imjournal you will get two copies of every message


>?and have these modules process journald log and generate individual
> log file? ?base on the matching ruleset?

yes, rsyslog runs every log message through the ruleset(s) in the config file

> 2) If imjournal is loaded, I am not
> crystal clear on rsyslog config syntax? ?to set up the ruleset for structured
> journald log entry filtering.? ?Essentially I am looking for the equivalent
> of, for example, "journalctl CONTAINER_ID=<id> > /path/to/container_id_log".

you would do something like

if $!CONTAINER_ID == "id" then /path/to/file

note that you can also use dynamic file names so that you can use variables in
the path, so you could end up writing to /path/to/ID/file (look for dynafile)

> 3) $programname property (which is part of TAG stemmed from MSG syslog fornat
> AFAIK) is available for ruleset matching when imuxsock is used?

programname is available with both imuxsock and imjournal

David Lang

> Thanks again for help.
> Thanks,Patrick
> On Sunday, February 2, 2020, 04:59:50 a.m. EST, David Lang <david@lang.hm> wrote:
>
> log the message with the template RSYSLOG_DebugFormat and it will show you every
> value it has parsed. You may need to use mmnormalize or mmjsonparse (the first
> is a batter long-term option as it is far more flexible) to extract the values
> from json if they aren't already parsed. look for a json string in the $!
> section, every value there can be individually addressed.
>
> David Lang
>
>
> ? On Sun, 2 Feb 2020,
> Patrick Leung wrote:
>
>> Date: Sun, 2 Feb 2020 09:30:55 +0000 (UTC)
>> From: Patrick Leung <leungsk38@yahoo.com>
>> To: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>,
>> ? ? David Lang <david@lang.hm>
>> Subject: Re: [rsyslog] rsyslog journald filtering
>>
>> Hello David,Indeed the information you have provided is useful. Another follow up question.
>> In case of using imjournal to filter systemd-journald log that is being forwarded to rsyslog-v8, what property I can use in rsyslog config for such log filtering? What if the journald log entry contain custom field that I want to use as the filtering condition?
>>
>> Thanks,Patrick
>>
>> ? ? On Friday, January 3, 2020, 5:36:16 p.m. PST, David Lang <david@lang.hm> wrote:
>>
>> rsyslog does not change it's config during a run (with the exception of the
>> table_lookup() function, which may work for you)
>>
>> syslog can filter on anything it knows about, so if you fetch from journald with
>> imjournal so that you can see all the metadata that journald takes the time to
>> lookup, you can filter on any of it. If you have journald write to a socket for
>> rsyslog to read, journald doesn't send that metadata (and refuses to consider
>> doing so, I've asked)
>>
>> does this answer your questions?
>>
>> David Lang
>>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Thank you again for the information. Again, if I load both imuxsock and imjournal, how do I associate filters to each input module?e.g. rsyslog.confif ( filter1_condition ) then { action(xxx) } 
if ( filter2_condition ) then { action(yyy) }Associate "filter1" to process messages received by imuxsock and "filter2" to process messages received by imjournalimuxsock supports ruleset binding; but not imjournal.
On Monday, February 3, 2020, 04:12:04 p.m. EST, David Lang <david@lang.hm> wrote:

On Mon, 3 Feb 2020, Patrick Leung wrote:

>  Thank you for your clarification once again. If I understand it right, the
>  use of RSYSLOG_DebugFormat template gives me the generated syslog file that
>  contains structured journald log content, filtering is handled prior to
>  output log file generation.

not quite, filtering happens how you define it in the config file.

the debug format shows you what the contents of all the variables are at the
point in the config file that you write the log with that format.

This lets you see the contents to understand what your filters have to work
with.

filters can be the traditional facility.severity (i.e. mail.info) format, or
they can be much more powerful if..then formats

> What I would like to achieve:forward journald log to rsyslog, and use
> rsyslog's filtering capabilities to filter journald master log file to
> individual log file base on matched ruleset in rsyslog. I may want to filter
> journald log base on systemd units and container services to their individual
> log files.

that's trivail

> 1) Is it possible to load both imuxsock and imjournal modules when starting
> rsyslog,

yes, but if you have journald deliver the messages to imuxsock and have rsyslog
fetch the messages via imjournal you will get two copies of every message


> and have these modules process journald log and generate individual
> log file   base on the matching ruleset?

yes, rsyslog runs every log message through the ruleset(s) in the config file

> 2) If imjournal is loaded, I am not
> crystal clear on rsyslog config syntax   to set up the ruleset for structured
> journald log entry filtering.   Essentially I am looking for the equivalent
> of, for example, "journalctl CONTAINER_ID=<id> > /path/to/container_id_log".

you would do something like

if $!CONTAINER_ID == "id" then /path/to/file

note that you can also use dynamic file names so that you can use variables in
the path, so you could end up writing to /path/to/ID/file (look for dynafile)

> 3) $programname property (which is part of TAG stemmed from MSG syslog fornat
> AFAIK) is available for ruleset matching when imuxsock is used?

programname is available with both imuxsock and imjournal

David Lang

> Thanks again for help.
> Thanks,Patrick
>    On Sunday, February 2, 2020, 04:59:50 a.m. EST, David Lang <david@lang.hm> wrote:
>
> log the message with the template RSYSLOG_DebugFormat and it will show you every
> value it has parsed. You may need to use mmnormalize or mmjsonparse (the first
> is a batter long-term option as it is far more flexible) to extract the values
> from json if they aren't already parsed. look for a json string in the $!
> section, every value there can be individually addressed.
>
> David Lang
>
>
>   On Sun, 2 Feb 2020,
> Patrick Leung wrote:
>
>> Date: Sun, 2 Feb 2020 09:30:55 +0000 (UTC)
>> From: Patrick Leung <leungsk38@yahoo.com>
>> To: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>,
>>     David Lang <david@lang.hm>
>> Subject: Re: [rsyslog] rsyslog journald filtering
>>
>> Hello David,Indeed the information you have provided is useful. Another follow up question.
>> In case of using imjournal to filter systemd-journald log that is being forwarded to rsyslog-v8, what property I can use in rsyslog config for such log filtering? What if the journald log entry contain custom field that I want to use as the filtering condition?
>>
>> Thanks,Patrick
>>
>>     On Friday, January 3, 2020, 5:36:16 p.m. PST, David Lang <david@lang.hm> wrote:
>>
>> rsyslog does not change it's config during a run (with the exception of the
>> table_lookup() function, which may work for you)
>>
>> syslog can filter on anything it knows about, so if you fetch from journald with
>> imjournal so that you can see all the metadata that journald takes the time to
>> lookup, you can filter on any of it. If you have journald write to a socket for
>> rsyslog to read, journald doesn't send that metadata (and refuses to consider
>> doing so, I've asked)
>>
>> does this answer your questions?
>>
>> David Lang
>>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On Wed, 5 Feb 2020, Patrick Leung wrote:

> Thank you again for the information. Again, if I load both imuxsock and imjournal, how do I associate filters to each input module?

as you note, imjournal doesn't support binding to a ruleset, so logs from it
just go into the default ruleset.

if you log with the template RSYSLOG_DebugFormat you may find something you can
filter on (I suspect that $input will work, but I haven't tested it)

so something like
if $input == 'imjournal' and <other filter> then { action() }

David Lang

>e.g. rsyslog.confif ( filter1_condition ) then { action(xxx) }?
> if ( filter2_condition ) then { action(yyy) }Associate "filter1" to process messages received by imuxsock and "filter2" to process messages received by imjournalimuxsock supports ruleset binding; but not imjournal.
> On Monday, February 3, 2020, 04:12:04 p.m. EST, David Lang <david@lang.hm> wrote:
>
> On Mon, 3 Feb 2020, Patrick Leung wrote:
>
>> ? Thank you for your clarification once again. If I understand it right, the
>> ? use of RSYSLOG_DebugFormat template gives me the generated syslog file that
>> ? contains structured journald log content, filtering is handled prior to
>> ? output log file generation.
>
> not quite, filtering happens how you define it in the config file.
>
> the debug format shows you what the contents of all the variables are at the
> point in the config file that you write the log with that format.
>
> This lets you see the contents to understand what your filters have to work
> with.
>
> filters can be the traditional facility.severity (i.e. mail.info) format, or
> they can be much more powerful if..then formats
>
>> What I would like to achieve:forward journald log to rsyslog, and use
>> rsyslog's filtering capabilities to filter journald master log file to
>> individual log file base on matched ruleset in rsyslog. I may want to filter
>> journald log base on systemd units and container services to their individual
>> log files.
>
> that's trivail
>
>> 1) Is it possible to load both imuxsock and imjournal modules when starting
>> rsyslog,
>
> yes, but if you have journald deliver the messages to imuxsock and have rsyslog
> fetch the messages via imjournal you will get two copies of every message
>
>
>> ?and have these modules process journald log and generate individual
>> log file? ?base on the matching ruleset?
>
> yes, rsyslog runs every log message through the ruleset(s) in the config file
>
>> 2) If imjournal is loaded, I am not
>> crystal clear on rsyslog config syntax? ?to set up the ruleset for structured
>> journald log entry filtering.? ?Essentially I am looking for the equivalent
>> of, for example, "journalctl CONTAINER_ID=<id> > /path/to/container_id_log".
>
> you would do something like
>
> if $!CONTAINER_ID == "id" then /path/to/file
>
> note that you can also use dynamic file names so that you can use variables in
> the path, so you could end up writing to /path/to/ID/file (look for dynafile)
>
>> 3) $programname property (which is part of TAG stemmed from MSG syslog fornat
>> AFAIK) is available for ruleset matching when imuxsock is used?
>
> programname is available with both imuxsock and imjournal
>
> David Lang
>
>> Thanks again for help.
>> Thanks,Patrick
>> ? ? On Sunday, February 2, 2020, 04:59:50 a.m. EST, David Lang <david@lang.hm> wrote:
>>
>> log the message with the template RSYSLOG_DebugFormat and it will show you every
>> value it has parsed. You may need to use mmnormalize or mmjsonparse (the first
>> is a batter long-term option as it is far more flexible) to extract the values
>> from json if they aren't already parsed. look for a json string in the $!
>> section, every value there can be individually addressed.
>>
>> David Lang
>>
>>
>> ? On Sun, 2 Feb 2020,
>> Patrick Leung wrote:
>>
>>> Date: Sun, 2 Feb 2020 09:30:55 +0000 (UTC)
>>> From: Patrick Leung <leungsk38@yahoo.com>
>>> To: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>,
>>> ? ? David Lang <david@lang.hm>
>>> Subject: Re: [rsyslog] rsyslog journald filtering
>>>
>>> Hello David,Indeed the information you have provided is useful. Another follow up question.
>>> In case of using imjournal to filter systemd-journald log that is being forwarded to rsyslog-v8, what property I can use in rsyslog config for such log filtering? What if the journald log entry contain custom field that I want to use as the filtering condition?
>>>
>>> Thanks,Patrick
>>>
>>> ? ? On Friday, January 3, 2020, 5:36:16 p.m. PST, David Lang <david@lang.hm> wrote:
>>>
>>> rsyslog does not change it's config during a run (with the exception of the
>>> table_lookup() function, which may work for you)
>>>
>>> syslog can filter on anything it knows about, so if you fetch from journald with
>>> imjournal so that you can see all the metadata that journald takes the time to
>>> lookup, you can filter on any of it. If you have journald write to a socket for
>>> rsyslog to read, journald doesn't send that metadata (and refuses to consider
>>> doing so, I've asked)
>>>
>>> does this answer your questions?
>>>
>>> David Lang
>>>
>>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

I am reaching out again for some help to use imjournal to fetch journald log entries while imuxsock is also used. I am having a systemd system, rsyslog.service has "Requires=syslog.socket" commented out. rsyslog_v8.1903 is used.
The /etc/rsyslog.conf I have:
module(load="imuxsock"       SysSock.Use="on"       SysSock.Name="/run/systemd/journal/syslog")
module(load="builtin:omfile"       dirCreateMode="0755"       fileCreateMode="0640")
module(load="mmjsonparse")action(type="mmjsonparse")
$FileOwner syslog$FileGroup syslog$FileCreateMode 0640$DirCreateMode 0755$Umask 0022$PrivDropToUser syslog$PrivDropToGroup syslog
include(file="/etc/rsyslog.d/custom-syslog.conf mode="required")
Where, /etc/rsyslog.d/custom-syslog.conf defines the following:module(load="imjournal" PersistStateInterval="100" StateFile="/path/to/file")if ( $inputname == "imuxsock" ) then {    action(type="omfile" file="/var/log/imuxsock_log")else if ( $inputname == "imjournal" ) then {    action(type="omfile" file="/var/log/imjournal_log")}
I understand that the filtering of "$inputname" may or may not work as David's response. I want to know the proper configuration to enable the use of imjournal.
Thanks,Patrick

On Wednesday, February 5, 2020, 04:19:24 p.m. EST, David Lang <david@lang.hm> wrote:

On Wed, 5 Feb 2020, Patrick Leung wrote:

> Thank you again for the information. Again, if I load both imuxsock and imjournal, how do I associate filters to each input module?

as you note, imjournal doesn't support binding to a ruleset, so logs from it
just go into the default ruleset.

if you log with the template RSYSLOG_DebugFormat you may find something you can
filter on (I suspect that $input will work, but I haven't tested it)

so something like
if $input == 'imjournal' and <other filter> then { action() }

David Lang

>e.g. rsyslog.confif ( filter1_condition ) then { action(xxx) } 
> if ( filter2_condition ) then { action(yyy) }Associate "filter1" to process messages received by imuxsock and "filter2" to process messages received by imjournalimuxsock supports ruleset binding; but not imjournal.
>    On Monday, February 3, 2020, 04:12:04 p.m. EST, David Lang <david@lang.hm> wrote:
>
> On Mon, 3 Feb 2020, Patrick Leung wrote:
>
>>   Thank you for your clarification once again. If I understand it right, the
>>   use of RSYSLOG_DebugFormat template gives me the generated syslog file that
>>   contains structured journald log content, filtering is handled prior to
>>   output log file generation.
>
> not quite, filtering happens how you define it in the config file.
>
> the debug format shows you what the contents of all the variables are at the
> point in the config file that you write the log with that format.
>
> This lets you see the contents to understand what your filters have to work
> with.
>
> filters can be the traditional facility.severity (i.e. mail.info) format, or
> they can be much more powerful if..then formats
>
>> What I would like to achieve:forward journald log to rsyslog, and use
>> rsyslog's filtering capabilities to filter journald master log file to
>> individual log file base on matched ruleset in rsyslog. I may want to filter
>> journald log base on systemd units and container services to their individual
>> log files.
>
> that's trivail
>
>> 1) Is it possible to load both imuxsock and imjournal modules when starting
>> rsyslog,
>
> yes, but if you have journald deliver the messages to imuxsock and have rsyslog
> fetch the messages via imjournal you will get two copies of every message
>
>
>>  and have these modules process journald log and generate individual
>> log file   base on the matching ruleset?
>
> yes, rsyslog runs every log message through the ruleset(s) in the config file
>
>> 2) If imjournal is loaded, I am not
>> crystal clear on rsyslog config syntax   to set up the ruleset for structured
>> journald log entry filtering.   Essentially I am looking for the equivalent
>> of, for example, "journalctl CONTAINER_ID=<id> > /path/to/container_id_log".
>
> you would do something like
>
> if $!CONTAINER_ID == "id" then /path/to/file
>
> note that you can also use dynamic file names so that you can use variables in
> the path, so you could end up writing to /path/to/ID/file (look for dynafile)
>
>> 3) $programname property (which is part of TAG stemmed from MSG syslog fornat
>> AFAIK) is available for ruleset matching when imuxsock is used?
>
> programname is available with both imuxsock and imjournal
>
> David Lang
>
>> Thanks again for help.
>> Thanks,Patrick
>>     On Sunday, February 2, 2020, 04:59:50 a.m. EST, David Lang <david@lang.hm> wrote:
>>
>> log the message with the template RSYSLOG_DebugFormat and it will show you every
>> value it has parsed. You may need to use mmnormalize or mmjsonparse (the first
>> is a batter long-term option as it is far more flexible) to extract the values
>> from json if they aren't already parsed. look for a json string in the $!
>> section, every value there can be individually addressed.
>>
>> David Lang
>>
>>
>>   On Sun, 2 Feb 2020,
>> Patrick Leung wrote:
>>
>>> Date: Sun, 2 Feb 2020 09:30:55 +0000 (UTC)
>>> From: Patrick Leung <leungsk38@yahoo.com>
>>> To: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>,
>>>     David Lang <david@lang.hm>
>>> Subject: Re: [rsyslog] rsyslog journald filtering
>>>
>>> Hello David,Indeed the information you have provided is useful. Another follow up question.
>>> In case of using imjournal to filter systemd-journald log that is being forwarded to rsyslog-v8, what property I can use in rsyslog config for such log filtering? What if the journald log entry contain custom field that I want to use as the filtering condition?
>>>
>>> Thanks,Patrick
>>>
>>>     On Friday, January 3, 2020, 5:36:16 p.m. PST, David Lang <david@lang.hm> wrote:
>>>
>>> rsyslog does not change it's config during a run (with the exception of the
>>> table_lookup() function, which may work for you)
>>>
>>> syslog can filter on anything it knows about, so if you fetch from journald with
>>> imjournal so that you can see all the metadata that journald takes the time to
>>> lookup, you can filter on any of it. If you have journald write to a socket for
>>> rsyslog to read, journald doesn't send that metadata (and refuses to consider
>>> doing so, I've asked)
>>>
>>> does this answer your questions?
>>>
>>> David Lang
>>>
>>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

as you can see your mail client is wrapping the config whic makes it very hard
to read.

if you write a log file with the template RSYSLOG_DebugFormat it will show you
the contents of all the variables so you can see exactly what you have to work
with.

please post a sample of such a log.

David Lang

On Fri, 14 Feb 2020, Patrick Leung wrote:

> Date: Fri, 14 Feb 2020 01:51:42 +0000 (UTC)
> From: Patrick Leung <leungsk38@yahoo.com>
> To: David Lang <david@lang.hm>
> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> I am reaching out again for some help to use imjournal to fetch journald log entries while imuxsock is also used. I am having a systemd system, rsyslog.service has "Requires=syslog.socket" commented out. rsyslog_v8.1903 is used.
> The /etc/rsyslog.conf I have:
> module(load="imuxsock"? ? ? ?SysSock.Use="on"? ? ? ?SysSock.Name="/run/systemd/journal/syslog")
> module(load="builtin:omfile"? ? ? ?dirCreateMode="0755"? ? ? ?fileCreateMode="0640")
> module(load="mmjsonparse")action(type="mmjsonparse")
> $FileOwner syslog$FileGroup syslog$FileCreateMode 0640$DirCreateMode 0755$Umask 0022$PrivDropToUser syslog$PrivDropToGroup syslog
> include(file="/etc/rsyslog.d/custom-syslog.conf mode="required")
> Where, /etc/rsyslog.d/custom-syslog.conf defines the following:module(load="imjournal" PersistStateInterval="100" StateFile="/path/to/file")if ( $inputname == "imuxsock" ) then {? ? action(type="omfile" file="/var/log/imuxsock_log")else if ( $inputname == "imjournal" ) then {? ? action(type="omfile" file="/var/log/imjournal_log")}
> I understand that the filtering of "$inputname" may or may not work as David's response. I want to know the proper configuration to enable the use of imjournal.
> Thanks,Patrick
>
> On Wednesday, February 5, 2020, 04:19:24 p.m. EST, David Lang <david@lang.hm> wrote:
>
> On Wed, 5 Feb 2020, Patrick Leung wrote:
>
>> Thank you again for the information. Again, if I load both imuxsock and imjournal, how do I associate filters to each input module?
>
> as you note, imjournal doesn't support binding to a ruleset, so logs from it
> just go into the default ruleset.
>
> if you log with the template RSYSLOG_DebugFormat you may find something you can
> filter on (I suspect that $input will work, but I haven't tested it)
>
> so something like
> if $input == 'imjournal' and <other filter> then { action() }
>
> David Lang
>
>> e.g. rsyslog.confif ( filter1_condition ) then { action(xxx) }?
>> if ( filter2_condition ) then { action(yyy) }Associate "filter1" to process messages received by imuxsock and "filter2" to process messages received by imjournalimuxsock supports ruleset binding; but not imjournal.
>> ? ? On Monday, February 3, 2020, 04:12:04 p.m. EST, David Lang <david@lang.hm> wrote:
>>
>> On Mon, 3 Feb 2020, Patrick Leung wrote:
>>
>>> ? Thank you for your clarification once again. If I understand it right, the
>>> ? use of RSYSLOG_DebugFormat template gives me the generated syslog file that
>>> ? contains structured journald log content, filtering is handled prior to
>>> ? output log file generation.
>>
>> not quite, filtering happens how you define it in the config file.
>>
>> the debug format shows you what the contents of all the variables are at the
>> point in the config file that you write the log with that format.
>>
>> This lets you see the contents to understand what your filters have to work
>> with.
>>
>> filters can be the traditional facility.severity (i.e. mail.info) format, or
>> they can be much more powerful if..then formats
>>
>>> What I would like to achieve:forward journald log to rsyslog, and use
>>> rsyslog's filtering capabilities to filter journald master log file to
>>> individual log file base on matched ruleset in rsyslog. I may want to filter
>>> journald log base on systemd units and container services to their individual
>>> log files.
>>
>> that's trivail
>>
>>> 1) Is it possible to load both imuxsock and imjournal modules when starting
>>> rsyslog,
>>
>> yes, but if you have journald deliver the messages to imuxsock and have rsyslog
>> fetch the messages via imjournal you will get two copies of every message
>>
>>
>>> ?and have these modules process journald log and generate individual
>>> log file? ?base on the matching ruleset?
>>
>> yes, rsyslog runs every log message through the ruleset(s) in the config file
>>
>>> 2) If imjournal is loaded, I am not
>>> crystal clear on rsyslog config syntax? ?to set up the ruleset for structured
>>> journald log entry filtering.? ?Essentially I am looking for the equivalent
>>> of, for example, "journalctl CONTAINER_ID=<id> > /path/to/container_id_log".
>>
>> you would do something like
>>
>> if $!CONTAINER_ID == "id" then /path/to/file
>>
>> note that you can also use dynamic file names so that you can use variables in
>> the path, so you could end up writing to /path/to/ID/file (look for dynafile)
>>
>>> 3) $programname property (which is part of TAG stemmed from MSG syslog fornat
>>> AFAIK) is available for ruleset matching when imuxsock is used?
>>
>> programname is available with both imuxsock and imjournal
>>
>> David Lang
>>
>>> Thanks again for help.
>>> Thanks,Patrick
>>> ? ? On Sunday, February 2, 2020, 04:59:50 a.m. EST, David Lang <david@lang.hm> wrote:
>>>
>>> log the message with the template RSYSLOG_DebugFormat and it will show you every
>>> value it has parsed. You may need to use mmnormalize or mmjsonparse (the first
>>> is a batter long-term option as it is far more flexible) to extract the values
>>> from json if they aren't already parsed. look for a json string in the $!
>>> section, every value there can be individually addressed.
>>>
>>> David Lang
>>>
>>>
>>> ? On Sun, 2 Feb 2020,
>>> Patrick Leung wrote:
>>>
>>>> Date: Sun, 2 Feb 2020 09:30:55 +0000 (UTC)
>>>> From: Patrick Leung <leungsk38@yahoo.com>
>>>> To: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>,
>>>> ? ? David Lang <david@lang.hm>
>>>> Subject: Re: [rsyslog] rsyslog journald filtering
>>>>
>>>> Hello David,Indeed the information you have provided is useful. Another follow up question.
>>>> In case of using imjournal to filter systemd-journald log that is being forwarded to rsyslog-v8, what property I can use in rsyslog config for such log filtering? What if the journald log entry contain custom field that I want to use as the filtering condition?
>>>>
>>>> Thanks,Patrick
>>>>
>>>> ? ? On Friday, January 3, 2020, 5:36:16 p.m. PST, David Lang <david@lang.hm> wrote:
>>>>
>>>> rsyslog does not change it's config during a run (with the exception of the
>>>> table_lookup() function, which may work for you)
>>>>
>>>> syslog can filter on anything it knows about, so if you fetch from journald with
>>>> imjournal so that you can see all the metadata that journald takes the time to
>>>> lookup, you can filter on any of it. If you have journald write to a socket for
>>>> rsyslog to read, journald doesn't send that metadata (and refuses to consider
>>>> doing so, I've asked)
>>>>
>>>> does this answer your questions?
>>>>
>>>> David Lang
>>>>
>>>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi,
Sorry about the illegible config file content. Please allow me to try again. My system is using rsyslog_8.1903 from Yocto meta-openembedded/warrior that has systemd-journald_v241 forward to syslog. "/etc/rsyslog.conf" includes a custom rsyslog configuration "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration files.

Given these configuration files, I obtain ONLY logging entries from imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.

Debug line with all properties:
FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd', PROCID: '1', MSGID: '-',
TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
msg: ' Started machine-id persistence.'
escaped msg: ' Started machine-id persistence.'
inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started machine-id persistence.'
$!:{ "msg": "Started machine-id persistence." }
$.:
$/:

I would like to craft a rsyslog configuration such that both imuxsock and imjournal are used, and I will have a filtering base on the meta data in structured journald log fetched from imjournal.

I hope the given information clarifies my approach.


Thanks,
Patrick
==============================
[/etc/rsyslog.conf]
module(load="imuxsock"
  SysSock.Use="on"
  SysSock.Name="/run/systemd/journal/syslog")

module(load="builtin:omfile"
  dirCreateMode="0755"
  fileCreateMode="0640")

module(load="mmjsonparse")
action(type="mmjsonparse")

# Set the default permissions
$FileOwner syslog
$FileGroup syslog
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

include(file="/etc/rsyslog.d/*.conf" mode="required")

==============================

[/etc/rsyslog.d/custom-rsyslog.conf]
module(load="imjournal" PersisStateInterval="100")
action(type="mmjsonparse")
if ( $inputname == "imuxsock" ) then {
    action(type="omfile" file="/var/log/syslog/imuxsock.log" template="RSYSLOG_DebugFormat")
else if ( $inputname == "imjournal" ) then {
    action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
}
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

On Wed, 19 Feb 2020, Patrick Leung wrote:

> Hi,
> Sorry about the illegible config file content. Please allow me to try again. My system is using rsyslog_8.1903 from Yocto meta-openembedded/warrior that has systemd-journald_v241 forward to syslog. "/etc/rsyslog.conf" includes a custom rsyslog configuration "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration files.
>
> Given these configuration files, I obtain ONLY logging entries from imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.
>
> Debug line with all properties:
> FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
> syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd', PROCID: '1', MSGID: '-',
> TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
> msg: ' Started machine-id persistence.'
> escaped msg: ' Started machine-id persistence.'
> inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started machine-id persistence.'
> $!:{ "msg": "Started machine-id persistence." }
> $.:
> $/:
>
> I would like to craft a rsyslog configuration such that both imuxsock and imjournal are used, and I will have a filtering base on the meta data in structured journald log fetched from imjournal.

Ok, now setup similar thing for a log you get from imjournal and look at the
different properties that you get.

I would suggest that you look at what inputname is for each one, that will
probably work for your filtering.

David Lang

> I hope the given information clarifies my approach.
>
>
> Thanks,
> Patrick
> ==============================
> [/etc/rsyslog.conf]
> module(load="imuxsock"
>   SysSock.Use="on"
>   SysSock.Name="/run/systemd/journal/syslog")
>
> module(load="builtin:omfile"
>   dirCreateMode="0755"
>   fileCreateMode="0640")
>
> module(load="mmjsonparse")
> action(type="mmjsonparse")
>
> # Set the default permissions
> $FileOwner syslog
> $FileGroup syslog
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> $PrivDropToUser syslog
> $PrivDropToGroup syslog
>
> include(file="/etc/rsyslog.d/*.conf" mode="required")
>
> ==============================
>
> [/etc/rsyslog.d/custom-rsyslog.conf]
> module(load="imjournal" PersisStateInterval="100")
> action(type="mmjsonparse")
> if ( $inputname == "imuxsock" ) then {
>     action(type="omfile" file="/var/log/syslog/imuxsock.log" template="RSYSLOG_DebugFormat")
> else if ( $inputname == "imjournal" ) then {
>     action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
> }
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi,
Looks like the "/etc/rsyslog.d/custom-rsyslog.conf" in my previous email response has the "else-if" clause missed (somehow). It checks for "imuxsock" and then "imjournal" and logs accordingly as you suggested:
if ( $inputname == "imuxsock") then {
    action(type="omfile" ...)
} else if ( $inputname == "imjournal" ) then {
    action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
}

The problem I've been facing is that there is NO imjournal logs being filed.
I tried also having rsyslog.conf that loads and uses imjournal/mmjsonparse only (no *.conf include), NO logs have been filed as well. I am trying to figure out why there is no imjournal log, then the next step is to examine the meta-data and build the filters.

I assume imjournal is working in rsyslog_v8.1903, and (even though it is not project supported AFAIK) there is still development happening in imjournal plug-in (https://github.com/rsyslog/rsyslog/commits/master/plugins/imjournal/imjournal.c).

Thanks,
Patrick


On Wednesday, February 19, 2020, 05:47:21 a.m. EST, David Lang <david@lang.hm> wrote:





On Wed, 19 Feb 2020, Patrick Leung wrote:

> Hi,
> Sorry about the illegible config file content. Please allow me to try again. My system is using rsyslog_8.1903 from Yocto meta-openembedded/warrior that has systemd-journald_v241 forward to syslog. "/etc/rsyslog.conf" includes a custom rsyslog configuration "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration files.
>
> Given these configuration files, I obtain ONLY logging entries from imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.
>
> Debug line with all properties:
> FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
> syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd', PROCID: '1', MSGID: '-',
> TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
> msg: ' Started machine-id persistence.'
> escaped msg: ' Started machine-id persistence.'
> inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started machine-id persistence.'
> $!:{ "msg": "Started machine-id persistence." }
> $.:
> $/:
>
> I would like to craft a rsyslog configuration such that both imuxsock and imjournal are used, and I will have a filtering base on the meta data in structured journald log fetched from imjournal.

Ok, now setup  similar thing for a log you get from imjournal and look at the
different properties that you get.

I would suggest that you look at what inputname is for each one, that will
probably work for your filtering.

David Lang


> I hope the given information clarifies my approach.
>
>
> Thanks,
> Patrick
> ==============================
> [/etc/rsyslog.conf]
> module(load="imuxsock"
>   SysSock.Use="on"
>   SysSock.Name="/run/systemd/journal/syslog")
>
> module(load="builtin:omfile"
>   dirCreateMode="0755"
>   fileCreateMode="0640")
>
> module(load="mmjsonparse")
> action(type="mmjsonparse")
>
> # Set the default permissions
> $FileOwner syslog
> $FileGroup syslog
> $FileCreateMode 0640
> $DirCreateMode 0755
> $Umask 0022
> $PrivDropToUser syslog
> $PrivDropToGroup syslog
>
> include(file="/etc/rsyslog.d/*.conf" mode="required")
>
> ==============================
>
> [/etc/rsyslog.d/custom-rsyslog.conf]
> module(load="imjournal" PersisStateInterval="100")
> action(type="mmjsonparse")
> if ( $inputname == "imuxsock" ) then {
>     action(type="omfile" file="/var/log/syslog/imuxsock.log" template="RSYSLOG_DebugFormat")
> else if ( $inputname == "imjournal" ) then {
>     action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
> }
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

setup a short test where you turn of imuxsock and only have imjournal, then log
everything with the debug format so that you can see what an imjournal message
looks like.

David Lang

On Wed, 19 Feb 2020, Patrick Leung wrote:

> Date: Wed, 19 Feb 2020 17:46:59 +0000 (UTC)
> From: Patrick Leung <leungsk38@yahoo.com>
> To: David Lang <david@lang.hm>
> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> Hi,
> Looks like the "/etc/rsyslog.d/custom-rsyslog.conf" in my previous email response has the "else-if" clause missed (somehow). It checks for "imuxsock" and then "imjournal" and logs accordingly as you suggested:
> if ( $inputname == "imuxsock") then {
>     action(type="omfile" ...)
> } else if ( $inputname == "imjournal" ) then {
>     action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
> }
>
> The problem I've been facing is that there is NO imjournal logs being filed.
> I tried also having rsyslog.conf that loads and uses imjournal/mmjsonparse only (no *.conf include), NO logs have been filed as well. I am trying to figure out why there is no imjournal log, then the next step is to examine the meta-data and build the filters.
>
> I assume imjournal is working in rsyslog_v8.1903, and (even though it is not project supported AFAIK) there is still development happening in imjournal plug-in (https://github.com/rsyslog/rsyslog/commits/master/plugins/imjournal/imjournal.c).
>
> Thanks,
> Patrick
>
>
> On Wednesday, February 19, 2020, 05:47:21 a.m. EST, David Lang <david@lang.hm> wrote:
>
>
>
>
>
> On Wed, 19 Feb 2020, Patrick Leung wrote:
>
>> Hi,
>> Sorry about the illegible config file content. Please allow me to try again. My system is using rsyslog_8.1903 from Yocto meta-openembedded/warrior that has systemd-journald_v241 forward to syslog. "/etc/rsyslog.conf" includes a custom rsyslog configuration "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration files.
>>
>> Given these configuration files, I obtain ONLY logging entries from imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.
>>
>> Debug line with all properties:
>> FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
>> syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd', PROCID: '1', MSGID: '-',
>> TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
>> msg: ' Started machine-id persistence.'
>> escaped msg: ' Started machine-id persistence.'
>> inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started machine-id persistence.'
>> $!:{ "msg": "Started machine-id persistence." }
>> $.:
>> $/:
>>
>> I would like to craft a rsyslog configuration such that both imuxsock and imjournal are used, and I will have a filtering base on the meta data in structured journald log fetched from imjournal.
>
> Ok, now setup  similar thing for a log you get from imjournal and look at the
> different properties that you get.
>
> I would suggest that you look at what inputname is for each one, that will
> probably work for your filtering.
>
> David Lang
>
>
>> I hope the given information clarifies my approach.
>>
>>
>> Thanks,
>> Patrick
>> ==============================
>> [/etc/rsyslog.conf]
>> module(load="imuxsock"
>>   SysSock.Use="on"
>>   SysSock.Name="/run/systemd/journal/syslog")
>>
>> module(load="builtin:omfile"
>>   dirCreateMode="0755"
>>   fileCreateMode="0640")
>>
>> module(load="mmjsonparse")
>> action(type="mmjsonparse")
>>
>> # Set the default permissions
>> $FileOwner syslog
>> $FileGroup syslog
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>> $PrivDropToUser syslog
>> $PrivDropToGroup syslog
>>
>> include(file="/etc/rsyslog.d/*.conf" mode="required")
>>
>> ==============================
>>
>> [/etc/rsyslog.d/custom-rsyslog.conf]
>> module(load="imjournal" PersisStateInterval="100")
>> action(type="mmjsonparse")
>> if ( $inputname == "imuxsock" ) then {
>>     action(type="omfile" file="/var/log/syslog/imuxsock.log" template="RSYSLOG_DebugFormat")
>> else if ( $inputname == "imjournal" ) then {
>>     action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
>> }
>>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi David and Patrick,


Thanks a lot for your inputs and thoughts on how to resolve the custom token issue that we're having with rsyslog. Let me try your suggestions and get back to you.


Rgds,

Amila.
________________________________
From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of David Lang via rsyslog <rsyslog@lists.adiscon.com>
Sent: 19 February 2020 23:55
To: Patrick Leung <leungsk38@yahoo.com>
Cc: David Lang <david@lang.hm>; Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] rsyslog journald filtering

setup a short test where you turn of imuxsock and only have imjournal, then log
everything with the debug format so that you can see what an imjournal message
looks like.

David Lang

On Wed, 19 Feb 2020, Patrick Leung wrote:

> Date: Wed, 19 Feb 2020 17:46:59 +0000 (UTC)
> From: Patrick Leung <leungsk38@yahoo.com>
> To: David Lang <david@lang.hm>
> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> Hi,
> Looks like the "/etc/rsyslog.d/custom-rsyslog.conf" in my previous email response has the "else-if" clause missed (somehow). It checks for "imuxsock" and then "imjournal" and logs accordingly as you suggested:
> if ( $inputname == "imuxsock") then {
> action(type="omfile" ...)
> } else if ( $inputname == "imjournal" ) then {
> action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
> }
>
> The problem I've been facing is that there is NO imjournal logs being filed.
> I tried also having rsyslog.conf that loads and uses imjournal/mmjsonparse only (no *.conf include), NO logs have been filed as well. I am trying to figure out why there is no imjournal log, then the next step is to examine the meta-data and build the filters.
>
> I assume imjournal is working in rsyslog_v8.1903, and (even though it is not project supported AFAIK) there is still development happening in imjournal plug-in (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frsyslog%2Frsyslog%2Fcommits%2Fmaster%2Fplugins%2Fimjournal%2Fimjournal.c&amp;data=01%7C01%7Camila.dias%40quantei.com%7C158928a97ab941c8c52c08d7b5691780%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=xkUbxP0D7cqxwXEuUZuunG6vXk4yrS75rmO112kU0Jw%3D&amp;reserved=0).
>
> Thanks,
> Patrick
>
>
> On Wednesday, February 19, 2020, 05:47:21 a.m. EST, David Lang <david@lang.hm> wrote:
>
>
>
>
>
> On Wed, 19 Feb 2020, Patrick Leung wrote:
>
>> Hi,
>> Sorry about the illegible config file content. Please allow me to try again. My system is using rsyslog_8.1903 from Yocto meta-openembedded/warrior that has systemd-journald_v241 forward to syslog. "/etc/rsyslog.conf" includes a custom rsyslog configuration "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration files.
>>
>> Given these configuration files, I obtain ONLY logging entries from imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.
>>
>> Debug line with all properties:
>> FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
>> syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd', PROCID: '1', MSGID: '-',
>> TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
>> msg: ' Started machine-id persistence.'
>> escaped msg: ' Started machine-id persistence.'
>> inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started machine-id persistence.'
>> $!:{ "msg": "Started machine-id persistence." }
>> $.:
>> $/:
>>
>> I would like to craft a rsyslog configuration such that both imuxsock and imjournal are used, and I will have a filtering base on the meta data in structured journald log fetched from imjournal.
>
> Ok, now setup similar thing for a log you get from imjournal and look at the
> different properties that you get.
>
> I would suggest that you look at what inputname is for each one, that will
> probably work for your filtering.
>
> David Lang
>
>
>> I hope the given information clarifies my approach.
>>
>>
>> Thanks,
>> Patrick
>> ==============================
>> [/etc/rsyslog.conf]
>> module(load="imuxsock"
>> SysSock.Use="on"
>> SysSock.Name="/run/systemd/journal/syslog")
>>
>> module(load="builtin:omfile"
>> dirCreateMode="0755"
>> fileCreateMode="0640")
>>
>> module(load="mmjsonparse")
>> action(type="mmjsonparse")
>>
>> # Set the default permissions
>> $FileOwner syslog
>> $FileGroup syslog
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>> $PrivDropToUser syslog
>> $PrivDropToGroup syslog
>>
>> include(file="/etc/rsyslog.d/*.conf" mode="required")
>>
>> ==============================
>>
>> [/etc/rsyslog.d/custom-rsyslog.conf]
>> module(load="imjournal" PersisStateInterval="100")
>> action(type="mmjsonparse")
>> if ( $inputname == "imuxsock" ) then {
>> action(type="omfile" file="/var/log/syslog/imuxsock.log" template="RSYSLOG_DebugFormat")
>> else if ( $inputname == "imjournal" ) then {
>> action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
>> }
>>
>
_______________________________________________
rsyslog mailing list
https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Camila.dias%40quantei.com%7C158928a97ab941c8c52c08d7b5691780%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=z9gvfQVZGNns3fwz2rvzZEZvDWZJtGeku%2BqtLGedELo%3D&amp;reserved=0
https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Camila.dias%40quantei.com%7C158928a97ab941c8c52c08d7b5691780%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=Y8c0DG8WCzoDcSp%2F5El8sOnK7Zy9K3B%2BTF6wPW0Y6CI%3D&amp;reserved=0
What's up with rsyslog? Follow https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Camila.dias%40quantei.com%7C158928a97ab941c8c52c08d7b5691780%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=7nymt5UeHTrvgXH%2BQ0exwYpXptzB7ZpX3dR7DE%2BOvck%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi All,

I got the variable / input name that sets up the custom token through a similar debug example thanks to Patrick and that variable / input is called STRUCTURED-DATA. Since I'm not that familiar with custom templates I did the standard template with %STRUCTURED-DATA% as the log file name and it shows our cloud app instances belonging us and our customers separately with each log file labeled with the custom token ID.

the template I used is this: $template RemoteSource, "/var/log/cloud-app-name/%FROMHOST-IP%/%STRUCTURED-DATA%.log"
*.* ?RemoteSource


But what I actually want is for the variable / input name STRUCTURED-DATA to display the custom token just inside the /var/log/cloud-app-name/%FROMHOST-IP.log

Any suggestion on how to setup the template ?


Rgds,

Amila.
________________________________
From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Amila Dias via rsyslog <rsyslog@lists.adiscon.com>
Sent: 20 February 2020 06:59
To: David Lang via rsyslog <rsyslog@lists.adiscon.com>; Patrick Leung <leungsk38@yahoo.com>
Cc: Amila Dias <Amila.Dias@quantei.com>
Subject: Re: [rsyslog] rsyslog journald filtering

Hi David and Patrick,


Thanks a lot for your inputs and thoughts on how to resolve the custom token issue that we're having with rsyslog. Let me try your suggestions and get back to you.


Rgds,

Amila.
________________________________
From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of David Lang via rsyslog <rsyslog@lists.adiscon.com>
Sent: 19 February 2020 23:55
To: Patrick Leung <leungsk38@yahoo.com>
Cc: David Lang <david@lang.hm>; Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] rsyslog journald filtering

setup a short test where you turn of imuxsock and only have imjournal, then log
everything with the debug format so that you can see what an imjournal message
looks like.

David Lang

On Wed, 19 Feb 2020, Patrick Leung wrote:

> Date: Wed, 19 Feb 2020 17:46:59 +0000 (UTC)
> From: Patrick Leung <leungsk38@yahoo.com>
> To: David Lang <david@lang.hm>
> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> Hi,
> Looks like the "/etc/rsyslog.d/custom-rsyslog.conf" in my previous email response has the "else-if" clause missed (somehow). It checks for "imuxsock" and then "imjournal" and logs accordingly as you suggested:
> if ( $inputname == "imuxsock") then {
> action(type="omfile" ...)
> } else if ( $inputname == "imjournal" ) then {
> action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
> }
>
> The problem I've been facing is that there is NO imjournal logs being filed.
> I tried also having rsyslog.conf that loads and uses imjournal/mmjsonparse only (no *.conf include), NO logs have been filed as well. I am trying to figure out why there is no imjournal log, then the next step is to examine the meta-data and build the filters.
>
> I assume imjournal is working in rsyslog_v8.1903, and (even though it is not project supported AFAIK) there is still development happening in imjournal plug-in (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frsyslog%2Frsyslog%2Fcommits%2Fmaster%2Fplugins%2Fimjournal%2Fimjournal.c&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=z5%2FcB4g56xNJy8dqLKr8Gt4%2FeEMejFzgwzTkqkb%2BtjU%3D&amp;reserved=0).
>
> Thanks,
> Patrick
>
>
> On Wednesday, February 19, 2020, 05:47:21 a.m. EST, David Lang <david@lang.hm> wrote:
>
>
>
>
>
> On Wed, 19 Feb 2020, Patrick Leung wrote:
>
>> Hi,
>> Sorry about the illegible config file content. Please allow me to try again. My system is using rsyslog_8.1903 from Yocto meta-openembedded/warrior that has systemd-journald_v241 forward to syslog. "/etc/rsyslog.conf" includes a custom rsyslog configuration "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration files.
>>
>> Given these configuration files, I obtain ONLY logging entries from imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.
>>
>> Debug line with all properties:
>> FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
>> syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd', PROCID: '1', MSGID: '-',
>> TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
>> msg: ' Started machine-id persistence.'
>> escaped msg: ' Started machine-id persistence.'
>> inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started machine-id persistence.'
>> $!:{ "msg": "Started machine-id persistence." }
>> $.:
>> $/:
>>
>> I would like to craft a rsyslog configuration such that both imuxsock and imjournal are used, and I will have a filtering base on the meta data in structured journald log fetched from imjournal.
>
> Ok, now setup similar thing for a log you get from imjournal and look at the
> different properties that you get.
>
> I would suggest that you look at what inputname is for each one, that will
> probably work for your filtering.
>
> David Lang
>
>
>> I hope the given information clarifies my approach.
>>
>>
>> Thanks,
>> Patrick
>> ==============================
>> [/etc/rsyslog.conf]
>> module(load="imuxsock"
>> SysSock.Use="on"
>> SysSock.Name="/run/systemd/journal/syslog")
>>
>> module(load="builtin:omfile"
>> dirCreateMode="0755"
>> fileCreateMode="0640")
>>
>> module(load="mmjsonparse")
>> action(type="mmjsonparse")
>>
>> # Set the default permissions
>> $FileOwner syslog
>> $FileGroup syslog
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>> $PrivDropToUser syslog
>> $PrivDropToGroup syslog
>>
>> include(file="/etc/rsyslog.d/*.conf" mode="required")
>>
>> ==============================
>>
>> [/etc/rsyslog.d/custom-rsyslog.conf]
>> module(load="imjournal" PersisStateInterval="100")
>> action(type="mmjsonparse")
>> if ( $inputname == "imuxsock" ) then {
>> action(type="omfile" file="/var/log/syslog/imuxsock.log" template="RSYSLOG_DebugFormat")
>> else if ( $inputname == "imjournal" ) then {
>> action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
>> }
>>
>
_______________________________________________
rsyslog mailing list
https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=KkXVmpZvKrTSIOhm0wy%2Fd5lGJ%2BWHk8qpXC3sMI2MjWo%3D&amp;reserved=0
https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=1Ikd%2FhAFjTYk7RSvOyOHcDm8iZhSj2tE%2BxZfH2F6Gmw%3D&amp;reserved=0
What's up with rsyslog? Follow https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=xeyZwsdNp3ltTFSlTcJPeqJPLQfHrD0XBUaPpOq%2FscU%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=KkXVmpZvKrTSIOhm0wy%2Fd5lGJ%2BWHk8qpXC3sMI2MjWo%3D&amp;reserved=0
https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=1Ikd%2FhAFjTYk7RSvOyOHcDm8iZhSj2tE%2BxZfH2F6Gmw%3D&amp;reserved=0
What's up with rsyslog? Follow https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=xeyZwsdNp3ltTFSlTcJPeqJPLQfHrD0XBUaPpOq%2FscU%3D&amp;reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

please craft an example log and what you want the result to look like.

David Lang

On Thu, 20 Feb 2020, Amila Dias wrote:

> Date: Thu, 20 Feb 2020 11:19:56 +0000
> From: Amila Dias <Amila.Dias@quantei.com>
> To: Amila Dias via rsyslog <rsyslog@lists.adiscon.com>,
> Patrick Leung <leungsk38@yahoo.com>, David Lang <david@lang.hm>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> Hi All,
>
> I got the variable / input name that sets up the custom token through a similar debug example thanks to Patrick and that variable / input is called STRUCTURED-DATA. Since I'm not that familiar with custom templates I did the standard template with %STRUCTURED-DATA% as the log file name and it shows our cloud app instances belonging us and our customers separately with each log file labeled with the custom token ID.
>
> the template I used is this: $template RemoteSource, "/var/log/cloud-app-name/%FROMHOST-IP%/%STRUCTURED-DATA%.log"
> *.* ?RemoteSource
>
>
> But what I actually want is for the variable / input name STRUCTURED-DATA to display the custom token just inside the /var/log/cloud-app-name/%FROMHOST-IP.log
>
> Any suggestion on how to setup the template ?
>
>
> Rgds,
>
> Amila.
> ________________________________
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of Amila Dias via rsyslog <rsyslog@lists.adiscon.com>
> Sent: 20 February 2020 06:59
> To: David Lang via rsyslog <rsyslog@lists.adiscon.com>; Patrick Leung <leungsk38@yahoo.com>
> Cc: Amila Dias <Amila.Dias@quantei.com>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> Hi David and Patrick,
>
>
> Thanks a lot for your inputs and thoughts on how to resolve the custom token issue that we're having with rsyslog. Let me try your suggestions and get back to you.
>
>
> Rgds,
>
> Amila.
> ________________________________
> From: rsyslog <rsyslog-bounces@lists.adiscon.com> on behalf of David Lang via rsyslog <rsyslog@lists.adiscon.com>
> Sent: 19 February 2020 23:55
> To: Patrick Leung <leungsk38@yahoo.com>
> Cc: David Lang <david@lang.hm>; Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> setup a short test where you turn of imuxsock and only have imjournal, then log
> everything with the debug format so that you can see what an imjournal message
> looks like.
>
> David Lang
>
> On Wed, 19 Feb 2020, Patrick Leung wrote:
>
>> Date: Wed, 19 Feb 2020 17:46:59 +0000 (UTC)
>> From: Patrick Leung <leungsk38@yahoo.com>
>> To: David Lang <david@lang.hm>
>> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] rsyslog journald filtering
>>
>> Hi,
>> Looks like the "/etc/rsyslog.d/custom-rsyslog.conf" in my previous email response has the "else-if" clause missed (somehow). It checks for "imuxsock" and then "imjournal" and logs accordingly as you suggested:
>> if ( $inputname == "imuxsock") then {
>> action(type="omfile" ...)
>> } else if ( $inputname == "imjournal" ) then {
>> action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
>> }
>>
>> The problem I've been facing is that there is NO imjournal logs being filed.
>> I tried also having rsyslog.conf that loads and uses imjournal/mmjsonparse only (no *.conf include), NO logs have been filed as well. I am trying to figure out why there is no imjournal log, then the next step is to examine the meta-data and build the filters.
>>
>> I assume imjournal is working in rsyslog_v8.1903, and (even though it is not project supported AFAIK) there is still development happening in imjournal plug-in (https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frsyslog%2Frsyslog%2Fcommits%2Fmaster%2Fplugins%2Fimjournal%2Fimjournal.c&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=z5%2FcB4g56xNJy8dqLKr8Gt4%2FeEMejFzgwzTkqkb%2BtjU%3D&amp;reserved=0).
>>
>> Thanks,
>> Patrick
>>
>>
>> On Wednesday, February 19, 2020, 05:47:21 a.m. EST, David Lang <david@lang.hm> wrote:
>>
>>
>>
>>
>>
>> On Wed, 19 Feb 2020, Patrick Leung wrote:
>>
>>> Hi,
>>> Sorry about the illegible config file content. Please allow me to try again. My system is using rsyslog_8.1903 from Yocto meta-openembedded/warrior that has systemd-journald_v241 forward to syslog. "/etc/rsyslog.conf" includes a custom rsyslog configuration "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration files.
>>>
>>> Given these configuration files, I obtain ONLY logging entries from imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.
>>>
>>> Debug line with all properties:
>>> FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
>>> syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd', PROCID: '1', MSGID: '-',
>>> TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
>>> msg: ' Started machine-id persistence.'
>>> escaped msg: ' Started machine-id persistence.'
>>> inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started machine-id persistence.'
>>> $!:{ "msg": "Started machine-id persistence." }
>>> $.:
>>> $/:
>>>
>>> I would like to craft a rsyslog configuration such that both imuxsock and imjournal are used, and I will have a filtering base on the meta data in structured journald log fetched from imjournal.
>>
>> Ok, now setup similar thing for a log you get from imjournal and look at the
>> different properties that you get.
>>
>> I would suggest that you look at what inputname is for each one, that will
>> probably work for your filtering.
>>
>> David Lang
>>
>>
>>> I hope the given information clarifies my approach.
>>>
>>>
>>> Thanks,
>>> Patrick
>>> ==============================
>>> [/etc/rsyslog.conf]
>>> module(load="imuxsock"
>>> SysSock.Use="on"
>>> SysSock.Name="/run/systemd/journal/syslog")
>>>
>>> module(load="builtin:omfile"
>>> dirCreateMode="0755"
>>> fileCreateMode="0640")
>>>
>>> module(load="mmjsonparse")
>>> action(type="mmjsonparse")
>>>
>>> # Set the default permissions
>>> $FileOwner syslog
>>> $FileGroup syslog
>>> $FileCreateMode 0640
>>> $DirCreateMode 0755
>>> $Umask 0022
>>> $PrivDropToUser syslog
>>> $PrivDropToGroup syslog
>>>
>>> include(file="/etc/rsyslog.d/*.conf" mode="required")
>>>
>>> ==============================
>>>
>>> [/etc/rsyslog.d/custom-rsyslog.conf]
>>> module(load="imjournal" PersisStateInterval="100")
>>> action(type="mmjsonparse")
>>> if ( $inputname == "imuxsock" ) then {
>>> action(type="omfile" file="/var/log/syslog/imuxsock.log" template="RSYSLOG_DebugFormat")
>>> else if ( $inputname == "imjournal" ) then {
>>> action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
>>> }
>>>
>>
> _______________________________________________
> rsyslog mailing list
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=KkXVmpZvKrTSIOhm0wy%2Fd5lGJ%2BWHk8qpXC3sMI2MjWo%3D&amp;reserved=0
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=1Ikd%2FhAFjTYk7RSvOyOHcDm8iZhSj2tE%2BxZfH2F6Gmw%3D&amp;reserved=0
> What's up with rsyslog? Follow https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=xeyZwsdNp3ltTFSlTcJPeqJPLQfHrD0XBUaPpOq%2FscU%3D&amp;reserved=0
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=KkXVmpZvKrTSIOhm0wy%2Fd5lGJ%2BWHk8qpXC3sMI2MjWo%3D&amp;reserved=0
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=1Ikd%2FhAFjTYk7RSvOyOHcDm8iZhSj2tE%2BxZfH2F6Gmw%3D&amp;reserved=0
> What's up with rsyslog? Follow https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&amp;data=01%7C01%7Camila.dias%40quantei.com%7Ca033b92ef6b24fba8ea008d7b5a44d7d%7C75398b6e83a24926ba37e090ca2d2539%7C0&amp;sdata=xeyZwsdNp3ltTFSlTcJPeqJPLQfHrD0XBUaPpOq%2FscU%3D&amp;reserved=0
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

A quick question if you don't mind:

I would like to double check the required build changes/configuration in rsyslog to build a working imjournal. My distro is Yocto warrior-based (systemd-v241), the rsyslog_v8.1903.0 recipe has "imjournal" package option enabled (i.e. PACKAGECONFIG += "imjournal").

Is enabling of "libsystemd" in rsyslog required in order to have imjournal plugin working?

Thanks,
Patrick

On Wednesday, February 19, 2020, 01:25:21 p.m. EST, David Lang <david@lang.hm> wrote:





setup a short test where you turn of imuxsock and only have imjournal, then log
everything with the debug format so that you can see what an imjournal message
looks like.

David Lang

  On Wed, 19 Feb 2020, Patrick Leung wrote:

> Date: Wed, 19 Feb 2020 17:46:59 +0000 (UTC)
> From: Patrick Leung <leungsk38@yahoo.com>
> To: David Lang <david@lang.hm>
> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> Hi,
> Looks like the "/etc/rsyslog.d/custom-rsyslog.conf" in my previous email response has the "else-if" clause missed (somehow). It checks for "imuxsock" and then "imjournal" and logs accordingly as you suggested:
> if ( $inputname == "imuxsock") then {
>     action(type="omfile" ...)
> } else if ( $inputname == "imjournal" ) then {
>     action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
> }
>
> The problem I've been facing is that there is NO imjournal logs being filed.
> I tried also having rsyslog.conf that loads and uses imjournal/mmjsonparse only (no *.conf include), NO logs have been filed as well. I am trying to figure out why there is no imjournal log, then the next step is to examine the meta-data and build the filters.
>
> I assume imjournal is working in rsyslog_v8.1903, and (even though it is not project supported AFAIK) there is still development happening in imjournal plug-in (https://github.com/rsyslog/rsyslog/commits/master/plugins/imjournal/imjournal.c).
>
> Thanks,
> Patrick
>
>
> On Wednesday, February 19, 2020, 05:47:21 a.m. EST, David Lang <david@lang.hm> wrote:
>
>
>
>
>
> On Wed, 19 Feb 2020, Patrick Leung wrote:
>
>> Hi,
>> Sorry about the illegible config file content. Please allow me to try again. My system is using rsyslog_8.1903 from Yocto meta-openembedded/warrior that has systemd-journald_v241 forward to syslog. "/etc/rsyslog.conf" includes a custom rsyslog configuration "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration files.
>>
>> Given these configuration files, I obtain ONLY logging entries from imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.
>>
>> Debug line with all properties:
>> FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
>> syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd', PROCID: '1', MSGID: '-',
>> TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
>> msg: ' Started machine-id persistence.'
>> escaped msg: ' Started machine-id persistence.'
>> inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started machine-id persistence.'
>> $!:{ "msg": "Started machine-id persistence." }
>> $.:
>> $/:
>>
>> I would like to craft a rsyslog configuration such that both imuxsock and imjournal are used, and I will have a filtering base on the meta data in structured journald log fetched from imjournal.
>
> Ok, now setup  similar thing for a log you get from imjournal and look at the
> different properties that you get.
>
> I would suggest that you look at what inputname is for each one, that will
> probably work for your filtering.
>
> David Lang
>
>
>> I hope the given information clarifies my approach.
>>
>>
>> Thanks,
>> Patrick
>> ==============================
>> [/etc/rsyslog.conf]
>> module(load="imuxsock"
>>   SysSock.Use="on"
>>   SysSock.Name="/run/systemd/journal/syslog")
>>
>> module(load="builtin:omfile"
>>   dirCreateMode="0755"
>>   fileCreateMode="0640")
>>
>> module(load="mmjsonparse")
>> action(type="mmjsonparse")
>>
>> # Set the default permissions
>> $FileOwner syslog
>> $FileGroup syslog
>> $FileCreateMode 0640
>> $DirCreateMode 0755
>> $Umask 0022
>> $PrivDropToUser syslog
>> $PrivDropToGroup syslog
>>
>> include(file="/etc/rsyslog.d/*.conf" mode="required")
>>
>> ==============================
>>
>> [/etc/rsyslog.d/custom-rsyslog.conf]
>> module(load="imjournal" PersisStateInterval="100")
>> action(type="mmjsonparse")
>> if ( $inputname == "imuxsock" ) then {
>>     action(type="omfile" file="/var/log/syslog/imuxsock.log" template="RSYSLOG_DebugFormat")
>> else if ( $inputname == "imjournal" ) then {
>>     action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
>> }
>>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

I would expect it tobe, it's been a while since I built from scratch, and I
wasn't building for a systemd enabled distro at that point.

David Lang

On Thu, 20 Feb 2020, Patrick Leung wrote:

> Date: Thu, 20 Feb 2020 23:14:09 +0000 (UTC)
> From: Patrick Leung <leungsk38@yahoo.com>
> To: David Lang <david@lang.hm>
> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> A quick question if you don't mind:
>
> I would like to double check the required build changes/configuration in rsyslog to build a working imjournal. My distro is Yocto warrior-based (systemd-v241), the rsyslog_v8.1903.0 recipe has "imjournal" package option enabled (i.e. PACKAGECONFIG += "imjournal").
>
> Is enabling of "libsystemd" in rsyslog required in order to have imjournal plugin working?
>
> Thanks,
> Patrick
>
> On Wednesday, February 19, 2020, 01:25:21 p.m. EST, David Lang <david@lang.hm> wrote:
>
>
>
>
>
> setup a short test where you turn of imuxsock and only have imjournal, then log
> everything with the debug format so that you can see what an imjournal message
> looks like.
>
> David Lang
>
>   On Wed, 19 Feb 2020, Patrick Leung wrote:
>
>> Date: Wed, 19 Feb 2020 17:46:59 +0000 (UTC)
>> From: Patrick Leung <leungsk38@yahoo.com>
>> To: David Lang <david@lang.hm>
>> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] rsyslog journald filtering
>>
>> Hi,
>> Looks like the "/etc/rsyslog.d/custom-rsyslog.conf" in my previous email response has the "else-if" clause missed (somehow). It checks for "imuxsock" and then "imjournal" and logs accordingly as you suggested:
>> if ( $inputname == "imuxsock") then {
>>     action(type="omfile" ...)
>> } else if ( $inputname == "imjournal" ) then {
>>     action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
>> }
>>
>> The problem I've been facing is that there is NO imjournal logs being filed.
>> I tried also having rsyslog.conf that loads and uses imjournal/mmjsonparse only (no *.conf include), NO logs have been filed as well. I am trying to figure out why there is no imjournal log, then the next step is to examine the meta-data and build the filters.
>>
>> I assume imjournal is working in rsyslog_v8.1903, and (even though it is not project supported AFAIK) there is still development happening in imjournal plug-in (https://github.com/rsyslog/rsyslog/commits/master/plugins/imjournal/imjournal.c).
>>
>> Thanks,
>> Patrick
>>
>>
>> On Wednesday, February 19, 2020, 05:47:21 a.m. EST, David Lang <david@lang.hm> wrote:
>>
>>
>>
>>
>>
>> On Wed, 19 Feb 2020, Patrick Leung wrote:
>>
>>> Hi,
>>> Sorry about the illegible config file content. Please allow me to try again. My system is using rsyslog_8.1903 from Yocto meta-openembedded/warrior that has systemd-journald_v241 forward to syslog. "/etc/rsyslog.conf" includes a custom rsyslog configuration "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration files.
>>>
>>> Given these configuration files, I obtain ONLY logging entries from imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.
>>>
>>> Debug line with all properties:
>>> FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
>>> syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd', PROCID: '1', MSGID: '-',
>>> TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
>>> msg: ' Started machine-id persistence.'
>>> escaped msg: ' Started machine-id persistence.'
>>> inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started machine-id persistence.'
>>> $!:{ "msg": "Started machine-id persistence." }
>>> $.:
>>> $/:
>>>
>>> I would like to craft a rsyslog configuration such that both imuxsock and imjournal are used, and I will have a filtering base on the meta data in structured journald log fetched from imjournal.
>>
>> Ok, now setup  similar thing for a log you get from imjournal and look at the
>> different properties that you get.
>>
>> I would suggest that you look at what inputname is for each one, that will
>> probably work for your filtering.
>>
>> David Lang
>>
>>
>>> I hope the given information clarifies my approach.
>>>
>>>
>>> Thanks,
>>> Patrick
>>> ==============================
>>> [/etc/rsyslog.conf]
>>> module(load="imuxsock"
>>>   SysSock.Use="on"
>>>   SysSock.Name="/run/systemd/journal/syslog")
>>>
>>> module(load="builtin:omfile"
>>>   dirCreateMode="0755"
>>>   fileCreateMode="0640")
>>>
>>> module(load="mmjsonparse")
>>> action(type="mmjsonparse")
>>>
>>> # Set the default permissions
>>> $FileOwner syslog
>>> $FileGroup syslog
>>> $FileCreateMode 0640
>>> $DirCreateMode 0755
>>> $Umask 0022
>>> $PrivDropToUser syslog
>>> $PrivDropToGroup syslog
>>>
>>> include(file="/etc/rsyslog.d/*.conf" mode="required")
>>>
>>> ==============================
>>>
>>> [/etc/rsyslog.d/custom-rsyslog.conf]
>>> module(load="imjournal" PersisStateInterval="100")
>>> action(type="mmjsonparse")
>>> if ( $inputname == "imuxsock" ) then {
>>>     action(type="omfile" file="/var/log/syslog/imuxsock.log" template="RSYSLOG_DebugFormat")
>>> else if ( $inputname == "imjournal" ) then {
>>>     action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
>>> }
>>>
>>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

yup - you need libsystemd to access the journal database.

Rainer

El vie., 21 feb. 2020 a las 1:12, David Lang via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> I would expect it tobe, it's been a while since I built from scratch, and I
> wasn't building for a systemd enabled distro at that point.
>
> David Lang
>
> On Thu, 20 Feb 2020, Patrick Leung wrote:
>
> > Date: Thu, 20 Feb 2020 23:14:09 +0000 (UTC)
> > From: Patrick Leung <leungsk38@yahoo.com>
> > To: David Lang <david@lang.hm>
> > Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> > Subject: Re: [rsyslog] rsyslog journald filtering
> >
> > A quick question if you don't mind:
> >
> > I would like to double check the required build changes/configuration in rsyslog to build a working imjournal. My distro is Yocto warrior-based (systemd-v241), the rsyslog_v8.1903.0 recipe has "imjournal" package option enabled (i.e. PACKAGECONFIG += "imjournal").
> >
> > Is enabling of "libsystemd" in rsyslog required in order to have imjournal plugin working?
> >
> > Thanks,
> > Patrick
> >
> > On Wednesday, February 19, 2020, 01:25:21 p.m. EST, David Lang <david@lang.hm> wrote:
> >
> >
> >
> >
> >
> > setup a short test where you turn of imuxsock and only have imjournal, then log
> > everything with the debug format so that you can see what an imjournal message
> > looks like.
> >
> > David Lang
> >
> > On Wed, 19 Feb 2020, Patrick Leung wrote:
> >
> >> Date: Wed, 19 Feb 2020 17:46:59 +0000 (UTC)
> >> From: Patrick Leung <leungsk38@yahoo.com>
> >> To: David Lang <david@lang.hm>
> >> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> >> Subject: Re: [rsyslog] rsyslog journald filtering
> >>
> >> Hi,
> >> Looks like the "/etc/rsyslog.d/custom-rsyslog.conf" in my previous email response has the "else-if" clause missed (somehow). It checks for "imuxsock" and then "imjournal" and logs accordingly as you suggested:
> >> if ( $inputname == "imuxsock") then {
> >> action(type="omfile" ...)
> >> } else if ( $inputname == "imjournal" ) then {
> >> action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
> >> }
> >>
> >> The problem I've been facing is that there is NO imjournal logs being filed.
> >> I tried also having rsyslog.conf that loads and uses imjournal/mmjsonparse only (no *.conf include), NO logs have been filed as well. I am trying to figure out why there is no imjournal log, then the next step is to examine the meta-data and build the filters.
> >>
> >> I assume imjournal is working in rsyslog_v8.1903, and (even though it is not project supported AFAIK) there is still development happening in imjournal plug-in (https://github.com/rsyslog/rsyslog/commits/master/plugins/imjournal/imjournal.c).
> >>
> >> Thanks,
> >> Patrick
> >>
> >>
> >> On Wednesday, February 19, 2020, 05:47:21 a.m. EST, David Lang <david@lang.hm> wrote:
> >>
> >>
> >>
> >>
> >>
> >> On Wed, 19 Feb 2020, Patrick Leung wrote:
> >>
> >>> Hi,
> >>> Sorry about the illegible config file content. Please allow me to try again. My system is using rsyslog_8.1903 from Yocto meta-openembedded/warrior that has systemd-journald_v241 forward to syslog. "/etc/rsyslog.conf" includes a custom rsyslog configuration "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration files.
> >>>
> >>> Given these configuration files, I obtain ONLY logging entries from imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.
> >>>
> >>> Debug line with all properties:
> >>> FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
> >>> syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd', PROCID: '1', MSGID: '-',
> >>> TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
> >>> msg: ' Started machine-id persistence.'
> >>> escaped msg: ' Started machine-id persistence.'
> >>> inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started machine-id persistence.'
> >>> $!:{ "msg": "Started machine-id persistence." }
> >>> $.:
> >>> $/:
> >>>
> >>> I would like to craft a rsyslog configuration such that both imuxsock and imjournal are used, and I will have a filtering base on the meta data in structured journald log fetched from imjournal.
> >>
> >> Ok, now setup similar thing for a log you get from imjournal and look at the
> >> different properties that you get.
> >>
> >> I would suggest that you look at what inputname is for each one, that will
> >> probably work for your filtering.
> >>
> >> David Lang
> >>
> >>
> >>> I hope the given information clarifies my approach.
> >>>
> >>>
> >>> Thanks,
> >>> Patrick
> >>> ==============================
> >>> [/etc/rsyslog.conf]
> >>> module(load="imuxsock"
> >>> SysSock.Use="on"
> >>> SysSock.Name="/run/systemd/journal/syslog")
> >>>
> >>> module(load="builtin:omfile"
> >>> dirCreateMode="0755"
> >>> fileCreateMode="0640")
> >>>
> >>> module(load="mmjsonparse")
> >>> action(type="mmjsonparse")
> >>>
> >>> # Set the default permissions
> >>> $FileOwner syslog
> >>> $FileGroup syslog
> >>> $FileCreateMode 0640
> >>> $DirCreateMode 0755
> >>> $Umask 0022
> >>> $PrivDropToUser syslog
> >>> $PrivDropToGroup syslog
> >>>
> >>> include(file="/etc/rsyslog.d/*.conf" mode="required")
> >>>
> >>> ==============================
> >>>
> >>> [/etc/rsyslog.d/custom-rsyslog.conf]
> >>> module(load="imjournal" PersisStateInterval="100")
> >>> action(type="mmjsonparse")
> >>> if ( $inputname == "imuxsock" ) then {
> >>> action(type="omfile" file="/var/log/syslog/imuxsock.log" template="RSYSLOG_DebugFormat")
> >>> else if ( $inputname == "imjournal" ) then {
> >>> action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
> >>> }
> >>>
> >>
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Thanks for all the help thus far.
In this case, the "--enable-imjournal" and "--enable-libsystemd" need to be declared when building rsyslog w/ imjournal?

Thanks
Patrick


On Thursday, February 20, 2020, 04:12:17 p.m. PST, David Lang <david@lang.hm> wrote:





I would expect it tobe, it's been a while since I built from scratch, and I
wasn't building for a systemd enabled distro at that point.

David Lang

On Thu, 20 Feb 2020, Patrick Leung wrote:

> Date: Thu, 20 Feb 2020 23:14:09 +0000 (UTC)
> From: Patrick Leung <leungsk38@yahoo.com>
> To: David Lang <david@lang.hm>
> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog journald filtering
>
> A quick question if you don't mind:
>
> I would like to double check the required build changes/configuration in rsyslog to build a working imjournal. My distro is Yocto warrior-based (systemd-v241), the rsyslog_v8.1903.0 recipe has "imjournal" package option enabled (i.e. PACKAGECONFIG += "imjournal").
>
> Is enabling of "libsystemd" in rsyslog required in order to have imjournal plugin working?
>
> Thanks,
> Patrick
>
> On Wednesday, February 19, 2020, 01:25:21 p.m. EST, David Lang <david@lang.hm> wrote:
>
>
>
>
>
> setup a short test where you turn of imuxsock and only have imjournal, then log
> everything with the debug format so that you can see what an imjournal message
> looks like.
>
> David Lang
>
>   On Wed, 19 Feb 2020, Patrick Leung wrote:
>
>> Date: Wed, 19 Feb 2020 17:46:59 +0000 (UTC)
>> From: Patrick Leung <leungsk38@yahoo.com>
>> To: David Lang <david@lang.hm>
>> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] rsyslog journald filtering
>>
>> Hi,
>> Looks like the "/etc/rsyslog.d/custom-rsyslog.conf" in my previous email response has the "else-if" clause missed (somehow). It checks for "imuxsock" and then "imjournal" and logs accordingly as you suggested:
>> if ( $inputname == "imuxsock") then {
>>     action(type="omfile" ...)
>> } else if ( $inputname == "imjournal" ) then {
>>     action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
>> }
>>
>> The problem I've been facing is that there is NO imjournal logs being filed.
>> I tried also having rsyslog.conf that loads and uses imjournal/mmjsonparse only (no *.conf include), NO logs have been filed as well. I am trying to figure out why there is no imjournal log, then the next step is to examine the meta-data and build the filters.
>>
>> I assume imjournal is working in rsyslog_v8.1903, and (even though it is not project supported AFAIK) there is still development happening in imjournal plug-in (https://github.com/rsyslog/rsyslog/commits/master/plugins/imjournal/imjournal.c).
>>
>> Thanks,
>> Patrick
>>
>>
>> On Wednesday, February 19, 2020, 05:47:21 a.m. EST, David Lang <david@lang.hm> wrote:
>>
>>
>>
>>
>>
>> On Wed, 19 Feb 2020, Patrick Leung wrote:
>>
>>> Hi,
>>> Sorry about the illegible config file content. Please allow me to try again. My system is using rsyslog_8.1903 from Yocto meta-openembedded/warrior that has systemd-journald_v241 forward to syslog. "/etc/rsyslog.conf" includes a custom rsyslog configuration "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration files.
>>>
>>> Given these configuration files, I obtain ONLY logging entries from imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.
>>>
>>> Debug line with all properties:
>>> FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
>>> syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd', PROCID: '1', MSGID: '-',
>>> TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
>>> msg: ' Started machine-id persistence.'
>>> escaped msg: ' Started machine-id persistence.'
>>> inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started machine-id persistence.'
>>> $!:{ "msg": "Started machine-id persistence." }
>>> $.:
>>> $/:
>>>
>>> I would like to craft a rsyslog configuration such that both imuxsock and imjournal are used, and I will have a filtering base on the meta data in structured journald log fetched from imjournal.
>>
>> Ok, now setup  similar thing for a log you get from imjournal and look at the
>> different properties that you get.
>>
>> I would suggest that you look at what inputname is for each one, that will
>> probably work for your filtering.
>>
>> David Lang
>>
>>
>>> I hope the given information clarifies my approach.
>>>
>>>
>>> Thanks,
>>> Patrick
>>> ==============================
>>> [/etc/rsyslog.conf]
>>> module(load="imuxsock"
>>>   SysSock.Use="on"
>>>   SysSock.Name="/run/systemd/journal/syslog")
>>>
>>> module(load="builtin:omfile"
>>>   dirCreateMode="0755"
>>>   fileCreateMode="0640")
>>>
>>> module(load="mmjsonparse")
>>> action(type="mmjsonparse")
>>>
>>> # Set the default permissions
>>> $FileOwner syslog
>>> $FileGroup syslog
>>> $FileCreateMode 0640
>>> $DirCreateMode 0755
>>> $Umask 0022
>>> $PrivDropToUser syslog
>>> $PrivDropToGroup syslog
>>>
>>> include(file="/etc/rsyslog.d/*.conf" mode="required")
>>>
>>> ==============================
>>>
>>> [/etc/rsyslog.d/custom-rsyslog.conf]
>>> module(load="imjournal" PersisStateInterval="100")
>>> action(type="mmjsonparse")
>>> if ( $inputname == "imuxsock" ) then {
>>>     action(type="omfile" file="/var/log/syslog/imuxsock.log" template="RSYSLOG_DebugFormat")
>>> else if ( $inputname == "imjournal" ) then {
>>>     action(type="omfile" file="/var/log/syslog/imjournal.log" template="RSYSLOG_DebugFormat")
>>> }
>>>
>>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Yes

Sent from phone, thus brief.

Patrick Leung via rsyslog <rsyslog@lists.adiscon.com> schrieb am Fr., 21.
Feb. 2020, 21:16:

> Thanks for all the help thus far.
> In this case, the "--enable-imjournal" and "--enable-libsystemd" need to
> be declared when building rsyslog w/ imjournal?
>
> Thanks
> Patrick
>
>
> On Thursday, February 20, 2020, 04:12:17 p.m. PST, David Lang <
> david@lang.hm> wrote:
>
>
>
>
>
> I would expect it tobe, it's been a while since I built from scratch, and
> I
> wasn't building for a systemd enabled distro at that point.
>
> David Lang
>
> On Thu, 20 Feb 2020, Patrick Leung wrote:
>
> > Date: Thu, 20 Feb 2020 23:14:09 +0000 (UTC)
> > From: Patrick Leung <leungsk38@yahoo.com>
> > To: David Lang <david@lang.hm>
> > Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> > Subject: Re: [rsyslog] rsyslog journald filtering
> >
> > A quick question if you don't mind:
> >
> > I would like to double check the required build changes/configuration in
> rsyslog to build a working imjournal. My distro is Yocto warrior-based
> (systemd-v241), the rsyslog_v8.1903.0 recipe has "imjournal" package option
> enabled (i.e. PACKAGECONFIG += "imjournal").
> >
> > Is enabling of "libsystemd" in rsyslog required in order to have
> imjournal plugin working?
> >
> > Thanks,
> > Patrick
> >
> > On Wednesday, February 19, 2020, 01:25:21 p.m. EST, David Lang <
> david@lang.hm> wrote:
> >
> >
> >
> >
> >
> > setup a short test where you turn of imuxsock and only have imjournal,
> then log
> > everything with the debug format so that you can see what an imjournal
> message
> > looks like.
> >
> > David Lang
> >
> > On Wed, 19 Feb 2020, Patrick Leung wrote:
> >
> >> Date: Wed, 19 Feb 2020 17:46:59 +0000 (UTC)
> >> From: Patrick Leung <leungsk38@yahoo.com>
> >> To: David Lang <david@lang.hm>
> >> Cc: Patrick Leung via rsyslog <rsyslog@lists.adiscon.com>
> >> Subject: Re: [rsyslog] rsyslog journald filtering
> >>
> >> Hi,
> >> Looks like the "/etc/rsyslog.d/custom-rsyslog.conf" in my previous
> email response has the "else-if" clause missed (somehow). It checks for
> "imuxsock" and then "imjournal" and logs accordingly as you suggested:
> >> if ( $inputname == "imuxsock") then {
> >> action(type="omfile" ...)
> >> } else if ( $inputname == "imjournal" ) then {
> >> action(type="omfile" file="/var/log/syslog/imjournal.log"
> template="RSYSLOG_DebugFormat")
> >> }
> >>
> >> The problem I've been facing is that there is NO imjournal logs being
> filed.
> >> I tried also having rsyslog.conf that loads and uses
> imjournal/mmjsonparse only (no *.conf include), NO logs have been filed as
> well. I am trying to figure out why there is no imjournal log, then the
> next step is to examine the meta-data and build the filters.
> >>
> >> I assume imjournal is working in rsyslog_v8.1903, and (even though it
> is not project supported AFAIK) there is still development happening in
> imjournal plug-in (
> https://github.com/rsyslog/rsyslog/commits/master/plugins/imjournal/imjournal.c
> ).
> >>
> >> Thanks,
> >> Patrick
> >>
> >>
> >> On Wednesday, February 19, 2020, 05:47:21 a.m. EST, David Lang <
> david@lang.hm> wrote:
> >>
> >>
> >>
> >>
> >>
> >> On Wed, 19 Feb 2020, Patrick Leung wrote:
> >>
> >>> Hi,
> >>> Sorry about the illegible config file content. Please allow me to try
> again. My system is using rsyslog_8.1903 from Yocto
> meta-openembedded/warrior that has systemd-journald_v241 forward to syslog.
> "/etc/rsyslog.conf" includes a custom rsyslog configuration
> "/etc/rsyslog.d/custom-rsyslog.conf". Please see below the configuration
> files.
> >>>
> >>> Given these configuration files, I obtain ONLY logging entries from
> imuxsock. The "RSYSLOG_DebugFormat" gives me the syslog message, e.g.
> >>>
> >>> Debug line with all properties:
> >>> FROMHOST: 'xxx', fromhost-ip: '127.0.0.1', HOSTNAME: 'xxx', PRI: 30,
> >>> syslogtag 'systemd[1]:', programname: 'systemd', APP-NAME: 'systemd',
> PROCID: '1', MSGID: '-',
> >>> TIMESTAMP: 'Feb 18 16:17:16', STRUCTURED-DATA: '-',
> >>> msg: ' Started machine-id persistence.'
> >>> escaped msg: ' Started machine-id persistence.'
> >>> inputname: imuxsock rawmsg: '<30>Feb 18 16:17:16 systemd[1]: Started
> machine-id persistence.'
> >>> $!:{ "msg": "Started machine-id persistence." }
> >>> $.:
> >>> $/:
> >>>
> >>> I would like to craft a rsyslog configuration such that both imuxsock
> and imjournal are used, and I will have a filtering base on the meta data
> in structured journald log fetched from imjournal.
> >>
> >> Ok, now setup similar thing for a log you get from imjournal and look
> at the
> >> different properties that you get.
> >>
> >> I would suggest that you look at what inputname is for each one, that
> will
> >> probably work for your filtering.
> >>
> >> David Lang
> >>
> >>
> >>> I hope the given information clarifies my approach.
> >>>
> >>>
> >>> Thanks,
> >>> Patrick
> >>> ==============================
> >>> [/etc/rsyslog.conf]
> >>> module(load="imuxsock"
> >>> SysSock.Use="on"
> >>> SysSock.Name="/run/systemd/journal/syslog")
> >>>
> >>> module(load="builtin:omfile"
> >>> dirCreateMode="0755"
> >>> fileCreateMode="0640")
> >>>
> >>> module(load="mmjsonparse")
> >>> action(type="mmjsonparse")
> >>>
> >>> # Set the default permissions
> >>> $FileOwner syslog
> >>> $FileGroup syslog
> >>> $FileCreateMode 0640
> >>> $DirCreateMode 0755
> >>> $Umask 0022
> >>> $PrivDropToUser syslog
> >>> $PrivDropToGroup syslog
> >>>
> >>> include(file="/etc/rsyslog.d/*.conf" mode="required")
> >>>
> >>> ==============================
> >>>
> >>> [/etc/rsyslog.d/custom-rsyslog.conf]
> >>> module(load="imjournal" PersisStateInterval="100")
> >>> action(type="mmjsonparse")
> >>> if ( $inputname == "imuxsock" ) then {
> >>> action(type="omfile" file="/var/log/syslog/imuxsock.log"
> template="RSYSLOG_DebugFormat")
> >>> else if ( $inputname == "imjournal" ) then {
> >>> action(type="omfile" file="/var/log/syslog/imjournal.log"
> template="RSYSLOG_DebugFormat")
> >>> }
> >>>
> >>
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.