Mailing List Archive

please post your full configs, it's likely that there are other things in the
config that are causing issues.

note that when you are within an if statement, you don't need to do the &, just
a bare stop will apply to everything that the if matched.

David Lang

On Tue, 19 Mar 2024, Kees de Jong via rsyslog wrote:

> Date: Tue, 19 Mar 2024 13:09:01 +0000
> From: Kees de Jong via rsyslog <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: Kees de Jong <kees.dejong@surf.nl>
> Subject: [rsyslog] Bash history --> rsyslog relay --> Elastic
>
> Hi,
>
>
> I want to collect Bash history and forward it to my Elastic cluster. I
> have a central rsyslog server in my cluster, which receives logs
> already and forwards some to another rsyslog server.
>
> I use the Bash `shopt -s syslog_history` option on Red Hat. On the
> nodes in my cluster I forward the logs to the central rsyslog
> logserver. Which looks like this:
>
> if $programname == '-bash' or $programname == 'bash' and $msg contains
> 'HISTORY:' then {
> @x.x.x.x:514
> & stop
> }
>
> The central logserver has a Fleet Agent running, which runs an UDP
> listener integration on 127.0.0.1 on UDP 10514. This integration has
> syslog parsing enabled.
>
> I can send data from a node to the central rsyslog server. In tcpdump I
> see the messages send from another node. On the central rsyslog server
> I have the following setup:
>
> if $programname == '-bash' or $programname == 'bash' and $msg contains
> 'HISTORY:' then {
> @127.0.0.1:10514
> & stop
> }
>
> Which forwards the logs to the local Elastic Fleet Agent. However, logs
> send from a node in my cluster is not forwarded by the central rsylog
> server to the local Fleet port. When I enable the Bash syslog option on
> the central log server it does forward the local Bash history to the
> Elastic cluster. But it doesn't forward the data received from the
> nodes in my cluster.
>
> What am I missing? Other logs are forwarded to another syslog server
> just fine. So the setup should work I guess? That forwarding is done
> like this:
>
> if ($syslogfacility-text == ['auth','authpriv'] and ($programname
> contains 'ssh')) then @y.y.y.y:514
>
> So maybe the filter for receiving relayed Bash history messages is
> wrong? Since it does work fine for locally generated logs. And other
> type of logs are forwarded just fine.
>
>
>

Hi David,


I've attached the full config with some redacted parts such as IPs. I
hope that helps.


--
Kees de Jong | Supercomputing | https://www.surf.nl/en/about-surf
OpenPGP fingerprint: 0x0E45C98AB51428E6

On Tue, 2024-03-19 at 07:39 -0700, David Lang wrote:
> please post your full configs, it's likely that there are other
> things in the
> config that are causing issues.
>
> note that when you are within an if statement, you don't need to do
> the &, just
> a bare stop will apply to everything that the if matched.
>
> David Lang
>
> On Tue, 19 Mar 2024, Kees de Jong via rsyslog wrote:
>
> > Date: Tue, 19 Mar 2024 13:09:01 +0000
> > From: Kees de Jong via rsyslog <rsyslog@lists.adiscon.com>
> > To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> > Cc: Kees de Jong <kees.dejong@surf.nl>
> > Subject: [rsyslog] Bash history --> rsyslog relay --> Elastic
> >
> > Hi,
> >
> >
> > I want to collect Bash history and forward it to my Elastic
> > cluster. I
> > have a central rsyslog server in my cluster, which receives logs
> > already and forwards some to another rsyslog server.
> >
> > I use the Bash `shopt -s syslog_history` option on Red Hat. On the
> > nodes in my cluster I forward the logs to the central rsyslog
> > logserver. Which looks like this:
> >
> > if $programname == '-bash' or $programname == 'bash' and $msg
> > contains
> > 'HISTORY:' then {
> >    @x.x.x.x:514
> >     & stop
> > }
> >
> > The central logserver has a Fleet Agent running, which runs an UDP
> > listener integration on 127.0.0.1 on UDP 10514. This integration
> > has
> > syslog parsing enabled.
> >
> > I can send data from a node to the central rsyslog server. In
> > tcpdump I
> > see the messages send from another node. On the central rsyslog
> > server
> > I have the following setup:
> >
> > if $programname == '-bash' or $programname == 'bash' and $msg
> > contains
> > 'HISTORY:' then {
> >    @127.0.0.1:10514
> >     & stop
> > }
> >
> > Which forwards the logs to the local Elastic Fleet Agent. However,
> > logs
> > send from a node in my cluster is not forwarded by the central
> > rsylog
> > server to the local Fleet port. When I enable the Bash syslog
> > option on
> > the central log server it does forward the local Bash history to
> > the
> > Elastic cluster. But it doesn't forward the data received from the
> > nodes in my cluster.
> >
> > What am I missing? Other logs are forwarded to another syslog
> > server
> > just fine. So the setup should work I guess? That forwarding is
> > done
> > like this:
> >
> > if ($syslogfacility-text == ['auth','authpriv'] and ($programname
> > contains 'ssh')) then @y.y.y.y:514
> >
> > So maybe the filter for receiving relayed Bash history messages is
> > wrong? Since it does work fine for locally generated logs. And
> > other
> > type of logs are forwarded just fine.
> >
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> POST if you DON'T LIKE THAT.

Hi,


I've reset the rsyslog configuration on the rsyslog server to the Red
Hat default and confirmed it with `rpmverify`, I'm running version
rsyslog-8.2102.0-7.el8_6.1.x86_64.

Then I only added the config to receive and process the Bash history
logs:

cat /etc/rsyslog.d/00-a.conf
if $programname == '-bash' or $programname == 'bash' and $msg contains
'HISTORY:' then {
@127.0.0.1:10514
/var/log/bash-history.log
& stop
}

The default /etc/rsyslog.conf (without comments) is shown below:

# rg -v '^#' /etc/rsyslog.conf
2:
6:
8:
9:module(load="imuxsock" # provides support for local system
logging (e.g. via logger command)
10: SysSock.Use="off") # Turn off message reception via local log
socket;
11: # local messages are retrieved through
imjournal now.
12:module(load="imjournal" # provides access to the systemd
journal
13: StateFile="imjournal.state") # File to store the position in
the journal
16:
21:
26:
28:
30:global(workDirectory="/var/lib/rsyslog")
31:
33:module(load="builtin:omfile"
Template="RSYSLOG_TraditionalFileFormat")
34:
36:include(file="/etc/rsyslog.d/*.conf" mode="optional")
37:
39:
43:
46:*.info;mail.none;authpriv.none;cron.none
/var/log/messages
47:
49:authpriv.*
/var/log/secure
50:
52:mail.* -
/var/log/maillog
53:
54:
56:cron.*
/var/log/cron
57:
59:*.emerg :omusrmsg:*
60:
62:uucp,news.crit
/var/log/spooler
63:
65:local7.*
/var/log/boot.log
66:
67:

As you can see, I both forward the logs to a local UDP socket and a
local file. I also commented out one of them each time, to make sure
there is no issue with either the socket or file setup.

When I run a `tcpdump` on the rsyslog server, I do see that the
messages are coming in from the client:

10:49:00.007063 IP (tos 0x0, ttl 64, id 30789, offset 0, flags [DF],
proto UDP (17), length 98)
x.x.x.x.50145 > y.y.y.y.syslog: SYSLOG, length: 70
Facility user (1), Severity info (6)
Msg: Mar 25 10:48:59 rsyslogclient -bash[398711]: HISTORY:
PID=398711 UID=0 ls

However, the rsyslog server does not process these logs. The receiving
server forwards the logs to a local socket which is setup with
`netcat`, it does not show any output of messages. The local log file
remains empty as well. It's writable by anyone (-rw-rw-rw- 1 root root
0 Mar 25 10:52 /var/log/bash-history.log)

I've ran the same setup also on a VM with Fedora with version rsyslog-
8.2310.0-1.fc39.x86_64, with that version and setup it works fine.
Could it be that there is a syntax change somewhere? Or maybe something
else is wrong here? If it could be a syntax issue, what change could I
test?

Just to summarize:
* The client is able to forward the Bash history messages through
syslog.
* The server receives these messages as well.
* However, for some reason it does not match with the filter and thus
no action is taken by rsyslog to forward it to either a local socket or
a local file.



--
Kees de Jong | Supercomputing | https://www.surf.nl/en/about-surf
OpenPGP fingerprint: 0x0E45C98AB51428E6

On Wed, 2024-03-20 at 08:54 +0000, Kees de Jong via rsyslog wrote:
> Hi David,
>
>
> I've attached the full config with some redacted parts such as IPs. I
> hope that helps.
>
>

on the receiving system, log the messages with the template RSYSLOG_DebugFormat
and give us a sample message.

note that there are other reasons why you may see a log message with tcpdump but
rsyslog will not process it, including if there is not a route back to the
sender, or if there are firewall rules blocking that port. It's possible that
SELinux or systemd are also preventing rsyslog from accessing that port.

David Lang

On Mon, 25 Mar 2024, Kees de Jong via rsyslog wrote:

> Date: Mon, 25 Mar 2024 10:25:23 +0000
> From: Kees de Jong via rsyslog <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: Kees de Jong <kees.dejong@surf.nl>
> Subject: Re: [rsyslog] Bash history --> rsyslog relay --> Elastic
>
> Hi,
>
>
> I've reset the rsyslog configuration on the rsyslog server to the Red
> Hat default and confirmed it with `rpmverify`, I'm running version
> rsyslog-8.2102.0-7.el8_6.1.x86_64.
>
> Then I only added the config to receive and process the Bash history
> logs:
>
> cat /etc/rsyslog.d/00-a.conf
> if $programname == '-bash' or $programname == 'bash' and $msg contains
> 'HISTORY:' then {
> @127.0.0.1:10514
> /var/log/bash-history.log
> & stop
> }
>
> The default /etc/rsyslog.conf (without comments) is shown below:
>
> # rg -v '^#' /etc/rsyslog.conf
> 2:
> 6:
> 8:
> 9:module(load="imuxsock" # provides support for local system
> logging (e.g. via logger command)
> 10: SysSock.Use="off") # Turn off message reception via local log
> socket;
> 11: # local messages are retrieved through
> imjournal now.
> 12:module(load="imjournal" # provides access to the systemd
> journal
> 13: StateFile="imjournal.state") # File to store the position in
> the journal
> 16:
> 21:
> 26:
> 28:
> 30:global(workDirectory="/var/lib/rsyslog")
> 31:
> 33:module(load="builtin:omfile"
> Template="RSYSLOG_TraditionalFileFormat")
> 34:
> 36:include(file="/etc/rsyslog.d/*.conf" mode="optional")
> 37:
> 39:
> 43:
> 46:*.info;mail.none;authpriv.none;cron.none
> /var/log/messages
> 47:
> 49:authpriv.*
> /var/log/secure
> 50:
> 52:mail.* -
> /var/log/maillog
> 53:
> 54:
> 56:cron.*
> /var/log/cron
> 57:
> 59:*.emerg :omusrmsg:*
> 60:
> 62:uucp,news.crit
> /var/log/spooler
> 63:
> 65:local7.*
> /var/log/boot.log
> 66:
> 67:
>
> As you can see, I both forward the logs to a local UDP socket and a
> local file. I also commented out one of them each time, to make sure
> there is no issue with either the socket or file setup.
>
> When I run a `tcpdump` on the rsyslog server, I do see that the
> messages are coming in from the client:
>
> 10:49:00.007063 IP (tos 0x0, ttl 64, id 30789, offset 0, flags [DF],
> proto UDP (17), length 98)
> x.x.x.x.50145 > y.y.y.y.syslog: SYSLOG, length: 70
> Facility user (1), Severity info (6)
> Msg: Mar 25 10:48:59 rsyslogclient -bash[398711]: HISTORY:
> PID=398711 UID=0 ls
>
> However, the rsyslog server does not process these logs. The receiving
> server forwards the logs to a local socket which is setup with
> `netcat`, it does not show any output of messages. The local log file
> remains empty as well. It's writable by anyone (-rw-rw-rw- 1 root root
> 0 Mar 25 10:52 /var/log/bash-history.log)
>
> I've ran the same setup also on a VM with Fedora with version rsyslog-
> 8.2310.0-1.fc39.x86_64, with that version and setup it works fine.
> Could it be that there is a syntax change somewhere? Or maybe something
> else is wrong here? If it could be a syntax issue, what change could I
> test?
>
> Just to summarize:
> * The client is able to forward the Bash history messages through
> syslog.
> * The server receives these messages as well.
> * However, for some reason it does not match with the filter and thus
> no action is taken by rsyslog to forward it to either a local socket or
> a local file.
>
>
>
>

Hi David,



SELinux is disabled on all hosts. Other logs do get through from remote
hosts and are stored on a local disk. In this test setup I only enabled
the config for Bash history. I did this to exclude any configuration
that might intercept or drop the Bash history logs.

But when I include the other configs again, they log just fine locally.
So I think we can exclude systemd or a firewall in that case. Also,
when I use netcat, I can send over messages over those ports and see
them also with tcpdump.

> on the receiving system, log the messages with the template
RSYSLOG_DebugFormat and give us a sample message.

What exactly do you mean by this? Could you please elaborate on that?
At the moment no messages are logged for the Bash history. How can I
use this template to enhance the debugging? Thanks!



--
Kees de Jong | Supercomputing | https://www.surf.nl/en/about-surf
OpenPGP fingerprint: 0x0E45C98AB51428E6

On Mon, 2024-03-25 at 04:16 -0700, David Lang wrote:
> on the receiving system, log the messages with the template
> RSYSLOG_DebugFormat
> and give us a sample message.
>
> note that there are other reasons why you may see a log message with
> tcpdump but
> rsyslog will not process it, including if there is not a route back
> to the
> sender, or if there are firewall rules blocking that port. It's
> possible that
> SELinux or systemd are also preventing rsyslog from accessing that
> port.
>
> David Lang
>
>   On Mon, 25 Mar 2024, Kees de Jong via rsyslog wrote:
>
> > Date: Mon, 25 Mar 2024 10:25:23 +0000
> > From: Kees de Jong via rsyslog <rsyslog@lists.adiscon.com>
> > To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> > Cc: Kees de Jong <kees.dejong@surf.nl>
> > Subject: Re: [rsyslog] Bash history --> rsyslog relay --> Elastic
> >
> > Hi,
> >
> >
> > I've reset the rsyslog configuration on the rsyslog server to the
> > Red
> > Hat default and confirmed it with `rpmverify`, I'm running version
> > rsyslog-8.2102.0-7.el8_6.1.x86_64.
> >
> > Then I only added the config to receive and process the Bash
> > history
> > logs:
> >
> > cat /etc/rsyslog.d/00-a.conf
> > if $programname == '-bash' or $programname == 'bash' and $msg
> > contains
> > 'HISTORY:' then {
> >    @127.0.0.1:10514
> >    /var/log/bash-history.log
> >    & stop
> > }
> >
> > The default /etc/rsyslog.conf (without comments) is shown below:
> >
> > # rg -v '^#' /etc/rsyslog.conf
> > 2:
> > 6:
> > 8:
> > 9:module(load="imuxsock"          # provides support for local
> > system
> > logging (e.g. via logger command)
> > 10:       SysSock.Use="off") # Turn off message reception via local
> > log
> > socket;
> > 11:                       # local messages are retrieved through
> > imjournal now.
> > 12:module(load="imjournal"          # provides access to the
> > systemd
> > journal
> > 13:       StateFile="imjournal.state") # File to store the position
> > in
> > the journal
> > 16:
> > 21:
> > 26:
> > 28:
> > 30:global(workDirectory="/var/lib/rsyslog")
> > 31:
> > 33:module(load="builtin:omfile"
> > Template="RSYSLOG_TraditionalFileFormat")
> > 34:
> > 36:include(file="/etc/rsyslog.d/*.conf" mode="optional")
> > 37:
> > 39:
> > 43:
> > 46:*.info;mail.none;authpriv.none;cron.none
> > /var/log/messages
> > 47:
> > 49:authpriv.*
> > /var/log/secure
> > 50:
> > 52:mail.*                                                  -
> > /var/log/maillog
> > 53:
> > 54:
> > 56:cron.*
> > /var/log/cron
> > 57:
> > 59:*.emerg                                                
> > :omusrmsg:*
> > 60:
> > 62:uucp,news.crit
> > /var/log/spooler
> > 63:
> > 65:local7.*
> > /var/log/boot.log
> > 66:
> > 67:
> >
> > As you can see, I both forward the logs to a local UDP socket and a
> > local file. I also commented out one of them each time, to make
> > sure
> > there is no issue with either the socket or file setup.
> >
> > When I run a `tcpdump` on the rsyslog server, I do see that the
> > messages are coming in from the client:
> >
> > 10:49:00.007063 IP (tos 0x0, ttl 64, id 30789, offset 0, flags
> > [DF],
> > proto UDP (17), length 98)
> >    x.x.x.x.50145 > y.y.y.y.syslog: SYSLOG, length: 70
> >        Facility user (1), Severity info (6)
> >        Msg: Mar 25 10:48:59 rsyslogclient -bash[398711]: HISTORY:
> > PID=398711 UID=0 ls
> >
> > However, the rsyslog server does not process these logs. The
> > receiving
> > server forwards the logs to a local socket which is setup with
> > `netcat`, it does not show any output of messages. The local log
> > file
> > remains empty as well. It's writable by anyone (-rw-rw-rw- 1 root
> > root
> > 0 Mar 25 10:52 /var/log/bash-history.log)
> >
> > I've ran the same setup also on a VM with Fedora with version
> > rsyslog-
> > 8.2310.0-1.fc39.x86_64, with that version and setup it works fine.
> > Could it be that there is a syntax change somewhere? Or maybe
> > something
> > else is wrong here? If it could be a syntax issue, what change
> > could I
> > test?
> >
> > Just to summarize:
> > * The client is able to forward the Bash history messages through
> > syslog.
> > * The server receives these messages as well.
> > * However, for some reason it does not match with the filter and
> > thus
> > no action is taken by rsyslog to forward it to either a local
> > socket or
> > a local file.
> >
> >
> >
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> POST if you DON'T LIKE THAT.

Ok, the fact that you are getting other logs remotely does eliminate the
permission/network problems.

That just means that the filters you are applying to find the bash logs are not
matching the log contents.

To figure this out, you need to figure out what is actually being sent (since
it's not what you think is being sent or it would match your filters). To figure
this out, we need to figure out exactly what is being sent.

The best way to do this is to configure the reciving syslog server to log all
logs using the template RSYSLOG_DebugFormat, a line like the following would do
it

/var/log/debuglog;RSYSLOG_DebugFormat

then find a sample of the log message you are looking for in this log, and you
should then be able to see both the rawmsg of exactly what is arriving, and the
various properties showing how it was parsed apart. At that point you should be
able to adjust your filters to match the log message.

You can also dump the log message via tcpdump and analyse that, but that
requires manually figuring out how the log is being parsed.

I don't believe that you have shown a sample of what the log message looks like
(if you did, I apologize for missing it, please re-post it)

when the problem isn't network/permissions, >90% of the time the problem is that
the log isn't being parsed the way you think it is, so the filter doesn't match.

David Lang

On Mon, 25 Mar 2024, Kees de Jong via rsyslog wrote:

> Hi David,
>
>
>
> SELinux is disabled on all hosts. Other logs do get through from remote
> hosts and are stored on a local disk. In this test setup I only enabled
> the config for Bash history. I did this to exclude any configuration
> that might intercept or drop the Bash history logs.
>
> But when I include the other configs again, they log just fine locally.
> So I think we can exclude systemd or a firewall in that case. Also,
> when I use netcat, I can send over messages over those ports and see
> them also with tcpdump.
>
>> on the receiving system, log the messages with the template
> RSYSLOG_DebugFormat and give us a sample message.
>
> What exactly do you mean by this? Could you please elaborate on that?
> At the moment no messages are logged for the Bash history. How can I
> use this template to enhance the debugging? Thanks!
>
>
>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi,


Just for future generations. It was my mistake. In the rsyslog chain of
forwarding there was a mix of UDP and TCP. The destination listened on
a different protocol. So I saw the messages, they just weren't coming
through because rsyslog listened on another protocol. Sorry for the
late update. Thanks for the details though, it helped me troubleshoot
rsylog better.


--
Kees de Jong | Supercomputing | https://www.surf.nl/en/about-surf
OpenPGP fingerprint: 0x0E45C98AB51428E6

On Mon, 2024-03-25 at 13:36 -0700, David Lang wrote:
> Ok, the fact that you are getting other logs remotely does eliminate
> the
> permission/network problems.
>
> That just means that the filters you are applying to find the bash
> logs are not
> matching the log contents.
>
> To figure this out, you need to figure out what is actually being
> sent (since
> it's not what you think is being sent or it would match your
> filters). To figure
> this out, we need to figure out exactly what is being sent.
>
> The best way to do this is to configure the reciving syslog server to
> log all
> logs using the template RSYSLOG_DebugFormat, a line like the
> following would do
> it
>
> /var/log/debuglog;RSYSLOG_DebugFormat
>
> then find a sample of the log message you are looking for in this
> log, and you
> should then be able to see both the rawmsg of exactly what is
> arriving, and the
> various properties showing how it was parsed apart. At that point you
> should be
> able to adjust your filters to match the log message.
>
> You can also dump the log message via tcpdump and analyse that, but
> that
> requires manually figuring out how the log is being parsed.
>
> I don't believe that you have shown a sample of what the log message
> looks like
> (if you did, I apologize for missing it, please re-post it)
>
> when the problem isn't network/permissions, >90% of the time the
> problem is that
> the log isn't being parsed the way you think it is, so the filter
> doesn't match.
>
> David Lang
>
> On Mon, 25 Mar 2024, Kees de Jong via rsyslog wrote:
>
> > Hi David,
> >
> >
> >
> > SELinux is disabled on all hosts. Other logs do get through from
> > remote
> > hosts and are stored on a local disk. In this test setup I only
> > enabled
> > the config for Bash history. I did this to exclude any
> > configuration
> > that might intercept or drop the Bash history logs.
> >
> > But when I include the other configs again, they log just fine
> > locally.
> > So I think we can exclude systemd or a firewall in that case. Also,
> > when I use netcat, I can send over messages over those ports and
> > see
> > them also with tcpdump.
> >
> > > on the receiving system, log the messages with the template
> > RSYSLOG_DebugFormat and give us a sample message.
> >
> > What exactly do you mean by this? Could you please elaborate on
> > that?
> > At the moment no messages are logged for the Bash history. How can
> > I
> > use this template to enhance the debugging? Thanks!
> >
> >
> >
> >