Mailing List Archive

rsyslog ignoring random incoming tcp-messages
Hello,

I have a rather strange problem with an rsyslog-machine.

I configured it to process incoming logs over network via tcp port 514 using imtcp.

The incoming logs on that connection are simply stored in a specific folder in a log-file.
This setup works 90% of the time but sometimes logs get "lost".

I have no idea what causes it. I set up a tcpdump on the receiving machine and I can confirm that the packages are indeed being delivered.
But for some reason rsyslog does not notice them. I can find all other log-messages mentioned in the rsyslog-debug log, where they are noticed and processed correctly.

The ones that don't make it to the logfiles are no where to be found. I cannot find any difference in these files, so I would expect them to be handled identically.

Has anyone of you ever had similar problems and found a solution? Its not a firewall or network problem as the packages are definitely reaching the client.

I am grateful for any support you guys can provide

With kind regards

--
Florian Seifer
Berater secunet Anwendungen, Managed Security Services
Division Operational Services
secunet Security Networks AG

Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
florian.seifer@secunet.com
Kurf?rstenstra?e 58, 45138 Essen, Germany
www.secunet.com

-----------------------------------------------------------------------
Sitz: Kurf?rstenstra?e 58, 45138 Essen
Amtsgericht Essen HRB 13615
Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
Aufsichtsratsvorsitzender: Ralf Wintergerst

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog ignoring random incoming tcp-messages [ In reply to ]
config?

El mié., 11 nov. 2020 a las 9:11, Seifer, Florian via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Hello,
>
> I have a rather strange problem with an rsyslog-machine.
>
> I configured it to process incoming logs over network via tcp port 514 using imtcp.
>
> The incoming logs on that connection are simply stored in a specific folder in a log-file.
> This setup works 90% of the time but sometimes logs get "lost".
>
> I have no idea what causes it. I set up a tcpdump on the receiving machine and I can confirm that the packages are indeed being delivered.
> But for some reason rsyslog does not notice them. I can find all other log-messages mentioned in the rsyslog-debug log, where they are noticed and processed correctly.
>
> The ones that don't make it to the logfiles are no where to be found. I cannot find any difference in these files, so I would expect them to be handled identically.
>
> Has anyone of you ever had similar problems and found a solution? Its not a firewall or network problem as the packages are definitely reaching the client.
>
> I am grateful for any support you guys can provide
>
> With kind regards
>
> --
> Florian Seifer
> Berater secunet Anwendungen, Managed Security Services
> Division Operational Services
> secunet Security Networks AG
>
> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> florian.seifer@secunet.com
> Kurfürstenstraße 58, 45138 Essen, Germany
> www.secunet.com
>
> -----------------------------------------------------------------------
> Sitz: Kurfürstenstraße 58, 45138 Essen
> Amtsgericht Essen HRB 13615
> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> Aufsichtsratsvorsitzender: Ralf Wintergerst
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog ignoring random incoming tcp-messages [ In reply to ]
# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
SysSock.Use="off") # Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
module(load="imjournal" # provides access to the systemd journal
StateFile="imjournal.state") # File to store the position in the journal
#module(load="imklog") # reads kernel messages (the same are read from journald)
#module(load"immark") # provides --MARK-- message capability

# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
module(load="imtcp") # needs to be done just once








#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
global(workDirectory="/var/lib/rsyslog")

# Use default timestamp format
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")

# Include all config files in /etc/rsyslog.d/
include(file="/etc/rsyslog.d/*.conf" mode="optional")

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog


# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log






# ### sample forwarding rule ###
#action(type="omfwd"
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#queue.filename="fwdRule1" # unique name prefix for spool files
#queue.maxdiskspace="1g" # 1gb space limit (use as much as possible)
#queue.saveonshutdown="on" # save messages to disk on shutdown
#queue.type="LinkedList" # run asynchronously
#action.resumeRetryCount="-1" # infinite retries if host is down
# Remote Logging (we use TCP for reliable delivery)
# remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
#Target="remote_host" Port="XXX" Protocol="tcp")


# Diese Regel erstellt pro Hostname ein eigenes Verzeichnis
$template RemoteHost,"/var/log/remote/%hostname%/%programname%.log"

# Der Regelsatz remote wendet auf alle eingehenden Logs das Template RemoteHost an:

$RuleSet remote
*.* ?RemoteHost

# Nun binden wir den Regelsatz "remote" an alle syslog Meldungen von Außen.

input(type="imtcp" port="514" ruleset="remote" supportoctetcountedframing="off")

$EscapeControlCharactersOnReceive on

$RepeatedMsgReduction off

$SystemLogRateLimitInterval 0

$DebugLevel 2

$DebugFile /var/log/rsyslog/debug.log

--
Florian Seifer
Berater secunet Anwendungen, Managed Security Services
Division Operational Services
secunet Security Networks AG

Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
florian.seifer@secunet.com
Kurfürstenstraße 58, 45138 Essen, Germany
www.secunet.com

-----------------------------------------------------------------------
Sitz: Kurfürstenstraße 58, 45138 Essen
Amtsgericht Essen HRB 13615
Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
Aufsichtsratsvorsitzender: Ralf Wintergerst

-----Ursprüngliche Nachricht-----
Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
Gesendet: Mittwoch, 11. November 2020 09:13
An: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Seifer, Florian <florian.seifer@secunet.com>
Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages

config?

El mié., 11 nov. 2020 a las 9:11, Seifer, Florian via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Hello,
>
> I have a rather strange problem with an rsyslog-machine.
>
> I configured it to process incoming logs over network via tcp port 514 using imtcp.
>
> The incoming logs on that connection are simply stored in a specific folder in a log-file.
> This setup works 90% of the time but sometimes logs get "lost".
>
> I have no idea what causes it. I set up a tcpdump on the receiving machine and I can confirm that the packages are indeed being delivered.
> But for some reason rsyslog does not notice them. I can find all other log-messages mentioned in the rsyslog-debug log, where they are noticed and processed correctly.
>
> The ones that don't make it to the logfiles are no where to be found. I cannot find any difference in these files, so I would expect them to be handled identically.
>
> Has anyone of you ever had similar problems and found a solution? Its not a firewall or network problem as the packages are definitely reaching the client.
>
> I am grateful for any support you guys can provide
>
> With kind regards
>
> --
> Florian Seifer
> Berater secunet Anwendungen, Managed Security Services
> Division Operational Services
> secunet Security Networks AG
>
> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> florian.seifer@secunet.com
> Kurfürstenstraße 58, 45138 Essen, Germany
> www.secunet.com
>
> -----------------------------------------------------------------------
> Sitz: Kurfürstenstraße 58, 45138 Essen
> Amtsgericht Essen HRB 13615
> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> Aufsichtsratsvorsitzender: Ralf Wintergerst
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog ignoring random incoming tcp-messages [ In reply to ]
looks good. my gut feeling is that some senders do not properly
terminate the syslog frame with '\n'. Can you check you captures for a
case where a message is missing and post all bytes of the last seen
and missing messages in the correct sequence?

Rainer

El mié., 11 nov. 2020 a las 9:16, Seifer, Florian
(<florian.seifer@secunet.com>) escribió:
>
> # rsyslog configuration file
>
> # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
> # or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
> # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
>
> #### MODULES ####
>
> module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
> SysSock.Use="off") # Turn off message reception via local log socket;
> # local messages are retrieved through imjournal now.
> module(load="imjournal" # provides access to the systemd journal
> StateFile="imjournal.state") # File to store the position in the journal
> #module(load="imklog") # reads kernel messages (the same are read from journald)
> #module(load"immark") # provides --MARK-- message capability
>
> # Provides UDP syslog reception
> # for parameters see http://www.rsyslog.com/doc/imudp.html
> module(load="imudp") # needs to be done just once
> input(type="imudp" port="514")
>
> # Provides TCP syslog reception
> # for parameters see http://www.rsyslog.com/doc/imtcp.html
> module(load="imtcp") # needs to be done just once
>
>
>
>
>
>
>
>
> #### GLOBAL DIRECTIVES ####
>
> # Where to place auxiliary files
> global(workDirectory="/var/lib/rsyslog")
>
> # Use default timestamp format
> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
>
> # Include all config files in /etc/rsyslog.d/
> include(file="/etc/rsyslog.d/*.conf" mode="optional")
>
> #### RULES ####
>
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.* /dev/console
>
> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>
> # The authpriv file has restricted access.
> authpriv.* /var/log/secure
>
> # Log all the mail messages in one place.
> mail.* -/var/log/maillog
>
>
> # Log cron stuff
> cron.* /var/log/cron
>
> # Everybody gets emergency messages
> *.emerg :omusrmsg:*
>
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit /var/log/spooler
>
> # Save boot messages also to boot.log
> local7.* /var/log/boot.log
>
>
>
>
>
>
> # ### sample forwarding rule ###
> #action(type="omfwd"
> # An on-disk queue is created for this action. If the remote host is
> # down, messages are spooled to disk and sent when it is up again.
> #queue.filename="fwdRule1" # unique name prefix for spool files
> #queue.maxdiskspace="1g" # 1gb space limit (use as much as possible)
> #queue.saveonshutdown="on" # save messages to disk on shutdown
> #queue.type="LinkedList" # run asynchronously
> #action.resumeRetryCount="-1" # infinite retries if host is down
> # Remote Logging (we use TCP for reliable delivery)
> # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
> #Target="remote_host" Port="XXX" Protocol="tcp")
>
>
> # Diese Regel erstellt pro Hostname ein eigenes Verzeichnis
> $template RemoteHost,"/var/log/remote/%hostname%/%programname%.log"
>
> # Der Regelsatz remote wendet auf alle eingehenden Logs das Template RemoteHost an:
>
> $RuleSet remote
> *.* ?RemoteHost
>
> # Nun binden wir den Regelsatz "remote" an alle syslog Meldungen von Außen.
>
> input(type="imtcp" port="514" ruleset="remote" supportoctetcountedframing="off")
>
> $EscapeControlCharactersOnReceive on
>
> $RepeatedMsgReduction off
>
> $SystemLogRateLimitInterval 0
>
> $DebugLevel 2
>
> $DebugFile /var/log/rsyslog/debug.log
>
> --
> Florian Seifer
> Berater secunet Anwendungen, Managed Security Services
> Division Operational Services
> secunet Security Networks AG
>
> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> florian.seifer@secunet.com
> Kurfürstenstraße 58, 45138 Essen, Germany
> www.secunet.com
>
> -----------------------------------------------------------------------
> Sitz: Kurfürstenstraße 58, 45138 Essen
> Amtsgericht Essen HRB 13615
> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> Aufsichtsratsvorsitzender: Ralf Wintergerst
>
> -----Ursprüngliche Nachricht-----
> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Gesendet: Mittwoch, 11. November 2020 09:13
> An: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Seifer, Florian <florian.seifer@secunet.com>
> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>
> config?
>
> El mié., 11 nov. 2020 a las 9:11, Seifer, Florian via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
> >
> > Hello,
> >
> > I have a rather strange problem with an rsyslog-machine.
> >
> > I configured it to process incoming logs over network via tcp port 514 using imtcp.
> >
> > The incoming logs on that connection are simply stored in a specific folder in a log-file.
> > This setup works 90% of the time but sometimes logs get "lost".
> >
> > I have no idea what causes it. I set up a tcpdump on the receiving machine and I can confirm that the packages are indeed being delivered.
> > But for some reason rsyslog does not notice them. I can find all other log-messages mentioned in the rsyslog-debug log, where they are noticed and processed correctly.
> >
> > The ones that don't make it to the logfiles are no where to be found. I cannot find any difference in these files, so I would expect them to be handled identically.
> >
> > Has anyone of you ever had similar problems and found a solution? Its not a firewall or network problem as the packages are definitely reaching the client.
> >
> > I am grateful for any support you guys can provide
> >
> > With kind regards
> >
> > --
> > Florian Seifer
> > Berater secunet Anwendungen, Managed Security Services
> > Division Operational Services
> > secunet Security Networks AG
> >
> > Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> > florian.seifer@secunet.com
> > Kurfürstenstraße 58, 45138 Essen, Germany
> > www.secunet.com
> >
> > -----------------------------------------------------------------------
> > Sitz: Kurfürstenstraße 58, 45138 Essen
> > Amtsgericht Essen HRB 13615
> > Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> > Aufsichtsratsvorsitzender: Ralf Wintergerst
> >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog ignoring random incoming tcp-messages [ In reply to ]
Hello Rainer,

I am not terribly firm with tcpdump, how would I go about doing that?

Also I forgot to mention, the logs are always coming from the same Host with exactly the same format.
( A VmWare logInsight-Server if that’s relevant) so I doubt the messages are suddenly changing form.


-----Ursprüngliche Nachricht-----
Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
Gesendet: Mittwoch, 11. November 2020 09:26
An: Seifer, Florian <florian.seifer@secunet.com>
Cc: rsyslog-users <rsyslog@lists.adiscon.com>
Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages

looks good. my gut feeling is that some senders do not properly
terminate the syslog frame with '\n'. Can you check you captures for a
case where a message is missing and post all bytes of the last seen
and missing messages in the correct sequence?

Rainer

El mié., 11 nov. 2020 a las 9:16, Seifer, Florian
(<florian.seifer@secunet.com>) escribió:
>
> # rsyslog configuration file
>
> # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
> # or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
> # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
>
> #### MODULES ####
>
> module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
> SysSock.Use="off") # Turn off message reception via local log socket;
> # local messages are retrieved through imjournal now.
> module(load="imjournal" # provides access to the systemd journal
> StateFile="imjournal.state") # File to store the position in the journal
> #module(load="imklog") # reads kernel messages (the same are read from journald)
> #module(load"immark") # provides --MARK-- message capability
>
> # Provides UDP syslog reception
> # for parameters see http://www.rsyslog.com/doc/imudp.html
> module(load="imudp") # needs to be done just once
> input(type="imudp" port="514")
>
> # Provides TCP syslog reception
> # for parameters see http://www.rsyslog.com/doc/imtcp.html
> module(load="imtcp") # needs to be done just once
>
>
>
>
>
>
>
>
> #### GLOBAL DIRECTIVES ####
>
> # Where to place auxiliary files
> global(workDirectory="/var/lib/rsyslog")
>
> # Use default timestamp format
> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
>
> # Include all config files in /etc/rsyslog.d/
> include(file="/etc/rsyslog.d/*.conf" mode="optional")
>
> #### RULES ####
>
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.* /dev/console
>
> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>
> # The authpriv file has restricted access.
> authpriv.* /var/log/secure
>
> # Log all the mail messages in one place.
> mail.* -/var/log/maillog
>
>
> # Log cron stuff
> cron.* /var/log/cron
>
> # Everybody gets emergency messages
> *.emerg :omusrmsg:*
>
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit /var/log/spooler
>
> # Save boot messages also to boot.log
> local7.* /var/log/boot.log
>
>
>
>
>
>
> # ### sample forwarding rule ###
> #action(type="omfwd"
> # An on-disk queue is created for this action. If the remote host is
> # down, messages are spooled to disk and sent when it is up again.
> #queue.filename="fwdRule1" # unique name prefix for spool files
> #queue.maxdiskspace="1g" # 1gb space limit (use as much as possible)
> #queue.saveonshutdown="on" # save messages to disk on shutdown
> #queue.type="LinkedList" # run asynchronously
> #action.resumeRetryCount="-1" # infinite retries if host is down
> # Remote Logging (we use TCP for reliable delivery)
> # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
> #Target="remote_host" Port="XXX" Protocol="tcp")
>
>
> # Diese Regel erstellt pro Hostname ein eigenes Verzeichnis
> $template RemoteHost,"/var/log/remote/%hostname%/%programname%.log"
>
> # Der Regelsatz remote wendet auf alle eingehenden Logs das Template RemoteHost an:
>
> $RuleSet remote
> *.* ?RemoteHost
>
> # Nun binden wir den Regelsatz "remote" an alle syslog Meldungen von Außen.
>
> input(type="imtcp" port="514" ruleset="remote" supportoctetcountedframing="off")
>
> $EscapeControlCharactersOnReceive on
>
> $RepeatedMsgReduction off
>
> $SystemLogRateLimitInterval 0
>
> $DebugLevel 2
>
> $DebugFile /var/log/rsyslog/debug.log
>
> --
> Florian Seifer
> Berater secunet Anwendungen, Managed Security Services
> Division Operational Services
> secunet Security Networks AG
>
> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> florian.seifer@secunet.com
> Kurfürstenstraße 58, 45138 Essen, Germany
> www.secunet.com
>
> -----------------------------------------------------------------------
> Sitz: Kurfürstenstraße 58, 45138 Essen
> Amtsgericht Essen HRB 13615
> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> Aufsichtsratsvorsitzender: Ralf Wintergerst
>
> -----Ursprüngliche Nachricht-----
> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Gesendet: Mittwoch, 11. November 2020 09:13
> An: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Seifer, Florian <florian.seifer@secunet.com>
> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>
> config?
>
> El mié., 11 nov. 2020 a las 9:11, Seifer, Florian via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
> >
> > Hello,
> >
> > I have a rather strange problem with an rsyslog-machine.
> >
> > I configured it to process incoming logs over network via tcp port 514 using imtcp.
> >
> > The incoming logs on that connection are simply stored in a specific folder in a log-file.
> > This setup works 90% of the time but sometimes logs get "lost".
> >
> > I have no idea what causes it. I set up a tcpdump on the receiving machine and I can confirm that the packages are indeed being delivered.
> > But for some reason rsyslog does not notice them. I can find all other log-messages mentioned in the rsyslog-debug log, where they are noticed and processed correctly.
> >
> > The ones that don't make it to the logfiles are no where to be found. I cannot find any difference in these files, so I would expect them to be handled identically.
> >
> > Has anyone of you ever had similar problems and found a solution? Its not a firewall or network problem as the packages are definitely reaching the client.
> >
> > I am grateful for any support you guys can provide
> >
> > With kind regards
> >
> > --
> > Florian Seifer
> > Berater secunet Anwendungen, Managed Security Services
> > Division Operational Services
> > secunet Security Networks AG
> >
> > Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> > florian.seifer@secunet.com
> > Kurfürstenstraße 58, 45138 Essen, Germany
> > www.secunet.com
> >
> > -----------------------------------------------------------------------
> > Sitz: Kurfürstenstraße 58, 45138 Essen
> > Amtsgericht Essen HRB 13615
> > Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> > Aufsichtsratsvorsitzender: Ralf Wintergerst
> >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog ignoring random incoming tcp-messages [ In reply to ]
Maybe you can just use wireshark -- a screen shot from the gui is
probably sufficient.

Rainer

El mié., 11 nov. 2020 a las 9:31, Seifer, Florian
(<florian.seifer@secunet.com>) escribió:
>
> Hello Rainer,
>
> I am not terribly firm with tcpdump, how would I go about doing that?
>
> Also I forgot to mention, the logs are always coming from the same Host with exactly the same format.
> ( A VmWare logInsight-Server if that’s relevant) so I doubt the messages are suddenly changing form.
>
>
> -----Ursprüngliche Nachricht-----
> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Gesendet: Mittwoch, 11. November 2020 09:26
> An: Seifer, Florian <florian.seifer@secunet.com>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>
> looks good. my gut feeling is that some senders do not properly
> terminate the syslog frame with '\n'. Can you check you captures for a
> case where a message is missing and post all bytes of the last seen
> and missing messages in the correct sequence?
>
> Rainer
>
> El mié., 11 nov. 2020 a las 9:16, Seifer, Florian
> (<florian.seifer@secunet.com>) escribió:
> >
> > # rsyslog configuration file
> >
> > # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
> > # or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
> > # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
> >
> > #### MODULES ####
> >
> > module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
> > SysSock.Use="off") # Turn off message reception via local log socket;
> > # local messages are retrieved through imjournal now.
> > module(load="imjournal" # provides access to the systemd journal
> > StateFile="imjournal.state") # File to store the position in the journal
> > #module(load="imklog") # reads kernel messages (the same are read from journald)
> > #module(load"immark") # provides --MARK-- message capability
> >
> > # Provides UDP syslog reception
> > # for parameters see http://www.rsyslog.com/doc/imudp.html
> > module(load="imudp") # needs to be done just once
> > input(type="imudp" port="514")
> >
> > # Provides TCP syslog reception
> > # for parameters see http://www.rsyslog.com/doc/imtcp.html
> > module(load="imtcp") # needs to be done just once
> >
> >
> >
> >
> >
> >
> >
> >
> > #### GLOBAL DIRECTIVES ####
> >
> > # Where to place auxiliary files
> > global(workDirectory="/var/lib/rsyslog")
> >
> > # Use default timestamp format
> > module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
> >
> > # Include all config files in /etc/rsyslog.d/
> > include(file="/etc/rsyslog.d/*.conf" mode="optional")
> >
> > #### RULES ####
> >
> > # Log all kernel messages to the console.
> > # Logging much else clutters up the screen.
> > #kern.* /dev/console
> >
> > # Log anything (except mail) of level info or higher.
> > # Don't log private authentication messages!
> > *.info;mail.none;authpriv.none;cron.none /var/log/messages
> >
> > # The authpriv file has restricted access.
> > authpriv.* /var/log/secure
> >
> > # Log all the mail messages in one place.
> > mail.* -/var/log/maillog
> >
> >
> > # Log cron stuff
> > cron.* /var/log/cron
> >
> > # Everybody gets emergency messages
> > *.emerg :omusrmsg:*
> >
> > # Save news errors of level crit and higher in a special file.
> > uucp,news.crit /var/log/spooler
> >
> > # Save boot messages also to boot.log
> > local7.* /var/log/boot.log
> >
> >
> >
> >
> >
> >
> > # ### sample forwarding rule ###
> > #action(type="omfwd"
> > # An on-disk queue is created for this action. If the remote host is
> > # down, messages are spooled to disk and sent when it is up again.
> > #queue.filename="fwdRule1" # unique name prefix for spool files
> > #queue.maxdiskspace="1g" # 1gb space limit (use as much as possible)
> > #queue.saveonshutdown="on" # save messages to disk on shutdown
> > #queue.type="LinkedList" # run asynchronously
> > #action.resumeRetryCount="-1" # infinite retries if host is down
> > # Remote Logging (we use TCP for reliable delivery)
> > # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
> > #Target="remote_host" Port="XXX" Protocol="tcp")
> >
> >
> > # Diese Regel erstellt pro Hostname ein eigenes Verzeichnis
> > $template RemoteHost,"/var/log/remote/%hostname%/%programname%.log"
> >
> > # Der Regelsatz remote wendet auf alle eingehenden Logs das Template RemoteHost an:
> >
> > $RuleSet remote
> > *.* ?RemoteHost
> >
> > # Nun binden wir den Regelsatz "remote" an alle syslog Meldungen von Außen.
> >
> > input(type="imtcp" port="514" ruleset="remote" supportoctetcountedframing="off")
> >
> > $EscapeControlCharactersOnReceive on
> >
> > $RepeatedMsgReduction off
> >
> > $SystemLogRateLimitInterval 0
> >
> > $DebugLevel 2
> >
> > $DebugFile /var/log/rsyslog/debug.log
> >
> > --
> > Florian Seifer
> > Berater secunet Anwendungen, Managed Security Services
> > Division Operational Services
> > secunet Security Networks AG
> >
> > Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> > florian.seifer@secunet.com
> > Kurfürstenstraße 58, 45138 Essen, Germany
> > www.secunet.com
> >
> > -----------------------------------------------------------------------
> > Sitz: Kurfürstenstraße 58, 45138 Essen
> > Amtsgericht Essen HRB 13615
> > Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> > Aufsichtsratsvorsitzender: Ralf Wintergerst
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > Gesendet: Mittwoch, 11. November 2020 09:13
> > An: rsyslog-users <rsyslog@lists.adiscon.com>
> > Cc: Seifer, Florian <florian.seifer@secunet.com>
> > Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
> >
> > config?
> >
> > El mié., 11 nov. 2020 a las 9:11, Seifer, Florian via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribió:
> > >
> > > Hello,
> > >
> > > I have a rather strange problem with an rsyslog-machine.
> > >
> > > I configured it to process incoming logs over network via tcp port 514 using imtcp.
> > >
> > > The incoming logs on that connection are simply stored in a specific folder in a log-file.
> > > This setup works 90% of the time but sometimes logs get "lost".
> > >
> > > I have no idea what causes it. I set up a tcpdump on the receiving machine and I can confirm that the packages are indeed being delivered.
> > > But for some reason rsyslog does not notice them. I can find all other log-messages mentioned in the rsyslog-debug log, where they are noticed and processed correctly.
> > >
> > > The ones that don't make it to the logfiles are no where to be found. I cannot find any difference in these files, so I would expect them to be handled identically.
> > >
> > > Has anyone of you ever had similar problems and found a solution? Its not a firewall or network problem as the packages are definitely reaching the client.
> > >
> > > I am grateful for any support you guys can provide
> > >
> > > With kind regards
> > >
> > > --
> > > Florian Seifer
> > > Berater secunet Anwendungen, Managed Security Services
> > > Division Operational Services
> > > secunet Security Networks AG
> > >
> > > Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> > > florian.seifer@secunet.com
> > > Kurfürstenstraße 58, 45138 Essen, Germany
> > > www.secunet.com
> > >
> > > -----------------------------------------------------------------------
> > > Sitz: Kurfürstenstraße 58, 45138 Essen
> > > Amtsgericht Essen HRB 13615
> > > Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> > > Aufsichtsratsvorsitzender: Ralf Wintergerst
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog ignoring random incoming tcp-messages [ In reply to ]
Unfortunatly I cant install a GUI on the system.

I will try to come up with something

--
Florian Seifer
Berater secunet Anwendungen, Managed Security Services
Division Operational Services
secunet Security Networks AG

Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
florian.seifer@secunet.com
Kurfürstenstraße 58, 45138 Essen, Germany
www.secunet.com

-----------------------------------------------------------------------
Sitz: Kurfürstenstraße 58, 45138 Essen
Amtsgericht Essen HRB 13615
Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
Aufsichtsratsvorsitzender: Ralf Wintergerst

-----Ursprüngliche Nachricht-----
Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
Gesendet: Mittwoch, 11. November 2020 09:34
An: Seifer, Florian <florian.seifer@secunet.com>
Cc: rsyslog-users <rsyslog@lists.adiscon.com>
Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages

Maybe you can just use wireshark -- a screen shot from the gui is
probably sufficient.

Rainer

El mié., 11 nov. 2020 a las 9:31, Seifer, Florian
(<florian.seifer@secunet.com>) escribió:
>
> Hello Rainer,
>
> I am not terribly firm with tcpdump, how would I go about doing that?
>
> Also I forgot to mention, the logs are always coming from the same Host with exactly the same format.
> ( A VmWare logInsight-Server if that’s relevant) so I doubt the messages are suddenly changing form.
>
>
> -----Ursprüngliche Nachricht-----
> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Gesendet: Mittwoch, 11. November 2020 09:26
> An: Seifer, Florian <florian.seifer@secunet.com>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>
> looks good. my gut feeling is that some senders do not properly
> terminate the syslog frame with '\n'. Can you check you captures for a
> case where a message is missing and post all bytes of the last seen
> and missing messages in the correct sequence?
>
> Rainer
>
> El mié., 11 nov. 2020 a las 9:16, Seifer, Florian
> (<florian.seifer@secunet.com>) escribió:
> >
> > # rsyslog configuration file
> >
> > # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
> > # or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
> > # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
> >
> > #### MODULES ####
> >
> > module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
> > SysSock.Use="off") # Turn off message reception via local log socket;
> > # local messages are retrieved through imjournal now.
> > module(load="imjournal" # provides access to the systemd journal
> > StateFile="imjournal.state") # File to store the position in the journal
> > #module(load="imklog") # reads kernel messages (the same are read from journald)
> > #module(load"immark") # provides --MARK-- message capability
> >
> > # Provides UDP syslog reception
> > # for parameters see http://www.rsyslog.com/doc/imudp.html
> > module(load="imudp") # needs to be done just once
> > input(type="imudp" port="514")
> >
> > # Provides TCP syslog reception
> > # for parameters see http://www.rsyslog.com/doc/imtcp.html
> > module(load="imtcp") # needs to be done just once
> >
> >
> >
> >
> >
> >
> >
> >
> > #### GLOBAL DIRECTIVES ####
> >
> > # Where to place auxiliary files
> > global(workDirectory="/var/lib/rsyslog")
> >
> > # Use default timestamp format
> > module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
> >
> > # Include all config files in /etc/rsyslog.d/
> > include(file="/etc/rsyslog.d/*.conf" mode="optional")
> >
> > #### RULES ####
> >
> > # Log all kernel messages to the console.
> > # Logging much else clutters up the screen.
> > #kern.* /dev/console
> >
> > # Log anything (except mail) of level info or higher.
> > # Don't log private authentication messages!
> > *.info;mail.none;authpriv.none;cron.none /var/log/messages
> >
> > # The authpriv file has restricted access.
> > authpriv.* /var/log/secure
> >
> > # Log all the mail messages in one place.
> > mail.* -/var/log/maillog
> >
> >
> > # Log cron stuff
> > cron.* /var/log/cron
> >
> > # Everybody gets emergency messages
> > *.emerg :omusrmsg:*
> >
> > # Save news errors of level crit and higher in a special file.
> > uucp,news.crit /var/log/spooler
> >
> > # Save boot messages also to boot.log
> > local7.* /var/log/boot.log
> >
> >
> >
> >
> >
> >
> > # ### sample forwarding rule ###
> > #action(type="omfwd"
> > # An on-disk queue is created for this action. If the remote host is
> > # down, messages are spooled to disk and sent when it is up again.
> > #queue.filename="fwdRule1" # unique name prefix for spool files
> > #queue.maxdiskspace="1g" # 1gb space limit (use as much as possible)
> > #queue.saveonshutdown="on" # save messages to disk on shutdown
> > #queue.type="LinkedList" # run asynchronously
> > #action.resumeRetryCount="-1" # infinite retries if host is down
> > # Remote Logging (we use TCP for reliable delivery)
> > # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
> > #Target="remote_host" Port="XXX" Protocol="tcp")
> >
> >
> > # Diese Regel erstellt pro Hostname ein eigenes Verzeichnis
> > $template RemoteHost,"/var/log/remote/%hostname%/%programname%.log"
> >
> > # Der Regelsatz remote wendet auf alle eingehenden Logs das Template RemoteHost an:
> >
> > $RuleSet remote
> > *.* ?RemoteHost
> >
> > # Nun binden wir den Regelsatz "remote" an alle syslog Meldungen von Außen.
> >
> > input(type="imtcp" port="514" ruleset="remote" supportoctetcountedframing="off")
> >
> > $EscapeControlCharactersOnReceive on
> >
> > $RepeatedMsgReduction off
> >
> > $SystemLogRateLimitInterval 0
> >
> > $DebugLevel 2
> >
> > $DebugFile /var/log/rsyslog/debug.log
> >
> > --
> > Florian Seifer
> > Berater secunet Anwendungen, Managed Security Services
> > Division Operational Services
> > secunet Security Networks AG
> >
> > Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> > florian.seifer@secunet.com
> > Kurfürstenstraße 58, 45138 Essen, Germany
> > www.secunet.com
> >
> > -----------------------------------------------------------------------
> > Sitz: Kurfürstenstraße 58, 45138 Essen
> > Amtsgericht Essen HRB 13615
> > Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> > Aufsichtsratsvorsitzender: Ralf Wintergerst
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > Gesendet: Mittwoch, 11. November 2020 09:13
> > An: rsyslog-users <rsyslog@lists.adiscon.com>
> > Cc: Seifer, Florian <florian.seifer@secunet.com>
> > Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
> >
> > config?
> >
> > El mié., 11 nov. 2020 a las 9:11, Seifer, Florian via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribió:
> > >
> > > Hello,
> > >
> > > I have a rather strange problem with an rsyslog-machine.
> > >
> > > I configured it to process incoming logs over network via tcp port 514 using imtcp.
> > >
> > > The incoming logs on that connection are simply stored in a specific folder in a log-file.
> > > This setup works 90% of the time but sometimes logs get "lost".
> > >
> > > I have no idea what causes it. I set up a tcpdump on the receiving machine and I can confirm that the packages are indeed being delivered.
> > > But for some reason rsyslog does not notice them. I can find all other log-messages mentioned in the rsyslog-debug log, where they are noticed and processed correctly.
> > >
> > > The ones that don't make it to the logfiles are no where to be found. I cannot find any difference in these files, so I would expect them to be handled identically.
> > >
> > > Has anyone of you ever had similar problems and found a solution? Its not a firewall or network problem as the packages are definitely reaching the client.
> > >
> > > I am grateful for any support you guys can provide
> > >
> > > With kind regards
> > >
> > > --
> > > Florian Seifer
> > > Berater secunet Anwendungen, Managed Security Services
> > > Division Operational Services
> > > secunet Security Networks AG
> > >
> > > Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> > > florian.seifer@secunet.com
> > > Kurfürstenstraße 58, 45138 Essen, Germany
> > > www.secunet.com
> > >
> > > -----------------------------------------------------------------------
> > > Sitz: Kurfürstenstraße 58, 45138 Essen
> > > Amtsgericht Essen HRB 13615
> > > Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> > > Aufsichtsratsvorsitzender: Ralf Wintergerst
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog ignoring random incoming tcp-messages [ In reply to ]
Ok I got it:

0x0000: 4500 0130 ad1e 4000 3d06 8571 0a35 cd0d E..0..@.=..q.5..
0x0010: 0a35 28c1 c26c 0202 acec aacd c2a8 f13b .5(..l.........;
0x0020: 8018 001d 92c6 0000 0101 080a d80f e141 ...............A
0x0030: 8125 ea9a 3230 3230 2d31 312d 3131 5430 .%..2020-11-11T0
0x0040: 393a 3134 3a35 362e 3436 365a 2031 302e 9:14:56.466Z.10.
0x0050: 3533 2e32 3035 2e32 202d 202d 202d 202d 53.205.2.-.-.-.-
0x0060: 2044 6965 7320 6973 7420 6569 6e65 2076 .Dies.ist.eine.v
0x0070: 6f6e 204c 6f67 2049 6e73 6967 6874 2065 on.Log.Insight.e
0x0080: 7273 7465 6c6c 7465 2075 6e64 2061 6e20 rstellte.und.an.
0x0090: 6c6f 6777 6174 6368 2e73 6563 756e 6574 logwatch.secunet
0x00a0: 2e64 653a 3531 3420 6d69 7420 5241 5720 .de:514.mit.RAW.
0x00b0: 5443 5020 2875 6e64 206d 6974 2054 6167 TCP.(und.mit.Tag
0x00c0: 7320 686f 7374 3d7a 6b7a 2920 6765 7365 s.host=zkz).gese
0x00d0: 6e64 6574 6520 5465 7374 6e61 6368 7269 ndete.Testnachri
0x00e0: 6368 7420 66c3 bc72 2064 6965 2045 7265 cht.f..r.die.Ere
0x00f0: 6967 6e69 7377 6569 7465 726c 6569 7475 ignisweiterleitu
0x0100: 6e67 206d 6974 2064 656d 204e 616d 656e ng.mit.dem.Namen
0x0110: 2027 416c 6172 6d61 6e6c 6167 6520 5363 .'Alarmanlage.Sc
0x0120: 6861 7266 2f55 6e73 6368 6172 6627 2e0a harf/Unscharf'..

Above is a test-packet I fired to test my new tcpdump options.

Now I have to wait for another log to disappear

--
Florian Seifer
Berater secunet Anwendungen, Managed Security Services
Division Operational Services
secunet Security Networks AG

Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
florian.seifer@secunet.com
Kurfürstenstraße 58, 45138 Essen, Germany
www.secunet.com

-----------------------------------------------------------------------
Sitz: Kurfürstenstraße 58, 45138 Essen
Amtsgericht Essen HRB 13615
Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
Aufsichtsratsvorsitzender: Ralf Wintergerst

-----Ursprüngliche Nachricht-----
Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
Gesendet: Mittwoch, 11. November 2020 09:34
An: Seifer, Florian <florian.seifer@secunet.com>
Cc: rsyslog-users <rsyslog@lists.adiscon.com>
Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages

Maybe you can just use wireshark -- a screen shot from the gui is
probably sufficient.

Rainer

El mié., 11 nov. 2020 a las 9:31, Seifer, Florian
(<florian.seifer@secunet.com>) escribió:
>
> Hello Rainer,
>
> I am not terribly firm with tcpdump, how would I go about doing that?
>
> Also I forgot to mention, the logs are always coming from the same Host with exactly the same format.
> ( A VmWare logInsight-Server if that’s relevant) so I doubt the messages are suddenly changing form.
>
>
> -----Ursprüngliche Nachricht-----
> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Gesendet: Mittwoch, 11. November 2020 09:26
> An: Seifer, Florian <florian.seifer@secunet.com>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>
> looks good. my gut feeling is that some senders do not properly
> terminate the syslog frame with '\n'. Can you check you captures for a
> case where a message is missing and post all bytes of the last seen
> and missing messages in the correct sequence?
>
> Rainer
>
> El mié., 11 nov. 2020 a las 9:16, Seifer, Florian
> (<florian.seifer@secunet.com>) escribió:
> >
> > # rsyslog configuration file
> >
> > # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
> > # or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
> > # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
> >
> > #### MODULES ####
> >
> > module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
> > SysSock.Use="off") # Turn off message reception via local log socket;
> > # local messages are retrieved through imjournal now.
> > module(load="imjournal" # provides access to the systemd journal
> > StateFile="imjournal.state") # File to store the position in the journal
> > #module(load="imklog") # reads kernel messages (the same are read from journald)
> > #module(load"immark") # provides --MARK-- message capability
> >
> > # Provides UDP syslog reception
> > # for parameters see http://www.rsyslog.com/doc/imudp.html
> > module(load="imudp") # needs to be done just once
> > input(type="imudp" port="514")
> >
> > # Provides TCP syslog reception
> > # for parameters see http://www.rsyslog.com/doc/imtcp.html
> > module(load="imtcp") # needs to be done just once
> >
> >
> >
> >
> >
> >
> >
> >
> > #### GLOBAL DIRECTIVES ####
> >
> > # Where to place auxiliary files
> > global(workDirectory="/var/lib/rsyslog")
> >
> > # Use default timestamp format
> > module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
> >
> > # Include all config files in /etc/rsyslog.d/
> > include(file="/etc/rsyslog.d/*.conf" mode="optional")
> >
> > #### RULES ####
> >
> > # Log all kernel messages to the console.
> > # Logging much else clutters up the screen.
> > #kern.* /dev/console
> >
> > # Log anything (except mail) of level info or higher.
> > # Don't log private authentication messages!
> > *.info;mail.none;authpriv.none;cron.none /var/log/messages
> >
> > # The authpriv file has restricted access.
> > authpriv.* /var/log/secure
> >
> > # Log all the mail messages in one place.
> > mail.* -/var/log/maillog
> >
> >
> > # Log cron stuff
> > cron.* /var/log/cron
> >
> > # Everybody gets emergency messages
> > *.emerg :omusrmsg:*
> >
> > # Save news errors of level crit and higher in a special file.
> > uucp,news.crit /var/log/spooler
> >
> > # Save boot messages also to boot.log
> > local7.* /var/log/boot.log
> >
> >
> >
> >
> >
> >
> > # ### sample forwarding rule ###
> > #action(type="omfwd"
> > # An on-disk queue is created for this action. If the remote host is
> > # down, messages are spooled to disk and sent when it is up again.
> > #queue.filename="fwdRule1" # unique name prefix for spool files
> > #queue.maxdiskspace="1g" # 1gb space limit (use as much as possible)
> > #queue.saveonshutdown="on" # save messages to disk on shutdown
> > #queue.type="LinkedList" # run asynchronously
> > #action.resumeRetryCount="-1" # infinite retries if host is down
> > # Remote Logging (we use TCP for reliable delivery)
> > # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
> > #Target="remote_host" Port="XXX" Protocol="tcp")
> >
> >
> > # Diese Regel erstellt pro Hostname ein eigenes Verzeichnis
> > $template RemoteHost,"/var/log/remote/%hostname%/%programname%.log"
> >
> > # Der Regelsatz remote wendet auf alle eingehenden Logs das Template RemoteHost an:
> >
> > $RuleSet remote
> > *.* ?RemoteHost
> >
> > # Nun binden wir den Regelsatz "remote" an alle syslog Meldungen von Außen.
> >
> > input(type="imtcp" port="514" ruleset="remote" supportoctetcountedframing="off")
> >
> > $EscapeControlCharactersOnReceive on
> >
> > $RepeatedMsgReduction off
> >
> > $SystemLogRateLimitInterval 0
> >
> > $DebugLevel 2
> >
> > $DebugFile /var/log/rsyslog/debug.log
> >
> > --
> > Florian Seifer
> > Berater secunet Anwendungen, Managed Security Services
> > Division Operational Services
> > secunet Security Networks AG
> >
> > Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> > florian.seifer@secunet.com
> > Kurfürstenstraße 58, 45138 Essen, Germany
> > www.secunet.com
> >
> > -----------------------------------------------------------------------
> > Sitz: Kurfürstenstraße 58, 45138 Essen
> > Amtsgericht Essen HRB 13615
> > Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> > Aufsichtsratsvorsitzender: Ralf Wintergerst
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > Gesendet: Mittwoch, 11. November 2020 09:13
> > An: rsyslog-users <rsyslog@lists.adiscon.com>
> > Cc: Seifer, Florian <florian.seifer@secunet.com>
> > Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
> >
> > config?
> >
> > El mié., 11 nov. 2020 a las 9:11, Seifer, Florian via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribió:
> > >
> > > Hello,
> > >
> > > I have a rather strange problem with an rsyslog-machine.
> > >
> > > I configured it to process incoming logs over network via tcp port 514 using imtcp.
> > >
> > > The incoming logs on that connection are simply stored in a specific folder in a log-file.
> > > This setup works 90% of the time but sometimes logs get "lost".
> > >
> > > I have no idea what causes it. I set up a tcpdump on the receiving machine and I can confirm that the packages are indeed being delivered.
> > > But for some reason rsyslog does not notice them. I can find all other log-messages mentioned in the rsyslog-debug log, where they are noticed and processed correctly.
> > >
> > > The ones that don't make it to the logfiles are no where to be found. I cannot find any difference in these files, so I would expect them to be handled identically.
> > >
> > > Has anyone of you ever had similar problems and found a solution? Its not a firewall or network problem as the packages are definitely reaching the client.
> > >
> > > I am grateful for any support you guys can provide
> > >
> > > With kind regards
> > >
> > > --
> > > Florian Seifer
> > > Berater secunet Anwendungen, Managed Security Services
> > > Division Operational Services
> > > secunet Security Networks AG
> > >
> > > Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> > > florian.seifer@secunet.com
> > > Kurfürstenstraße 58, 45138 Essen, Germany
> > > www.secunet.com
> > >
> > > -----------------------------------------------------------------------
> > > Sitz: Kurfürstenstraße 58, 45138 Essen
> > > Amtsgericht Essen HRB 13615
> > > Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> > > Aufsichtsratsvorsitzender: Ralf Wintergerst
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog ignoring random incoming tcp-messages [ In reply to ]
looks good! Looking forward to the real sample. You can also check
yourself if the frame is terminated with LF (hex 0a). In the sample it
is.

Rainer

El mié., 11 nov. 2020 a las 10:18, Seifer, Florian
(<florian.seifer@secunet.com>) escribió:
>
> Ok I got it:
>
> 0x0000: 4500 0130 ad1e 4000 3d06 8571 0a35 cd0d E..0..@.=..q.5..
> 0x0010: 0a35 28c1 c26c 0202 acec aacd c2a8 f13b .5(..l.........;
> 0x0020: 8018 001d 92c6 0000 0101 080a d80f e141 ...............A
> 0x0030: 8125 ea9a 3230 3230 2d31 312d 3131 5430 .%..2020-11-11T0
> 0x0040: 393a 3134 3a35 362e 3436 365a 2031 302e 9:14:56.466Z.10.
> 0x0050: 3533 2e32 3035 2e32 202d 202d 202d 202d 53.205.2.-.-.-.-
> 0x0060: 2044 6965 7320 6973 7420 6569 6e65 2076 .Dies.ist.eine.v
> 0x0070: 6f6e 204c 6f67 2049 6e73 6967 6874 2065 on.Log.Insight.e
> 0x0080: 7273 7465 6c6c 7465 2075 6e64 2061 6e20 rstellte.und.an.
> 0x0090: 6c6f 6777 6174 6368 2e73 6563 756e 6574 logwatch.secunet
> 0x00a0: 2e64 653a 3531 3420 6d69 7420 5241 5720 .de:514.mit.RAW.
> 0x00b0: 5443 5020 2875 6e64 206d 6974 2054 6167 TCP.(und.mit.Tag
> 0x00c0: 7320 686f 7374 3d7a 6b7a 2920 6765 7365 s.host=zkz).gese
> 0x00d0: 6e64 6574 6520 5465 7374 6e61 6368 7269 ndete.Testnachri
> 0x00e0: 6368 7420 66c3 bc72 2064 6965 2045 7265 cht.f..r.die.Ere
> 0x00f0: 6967 6e69 7377 6569 7465 726c 6569 7475 ignisweiterleitu
> 0x0100: 6e67 206d 6974 2064 656d 204e 616d 656e ng.mit.dem.Namen
> 0x0110: 2027 416c 6172 6d61 6e6c 6167 6520 5363 .'Alarmanlage.Sc
> 0x0120: 6861 7266 2f55 6e73 6368 6172 6627 2e0a harf/Unscharf'..
>
> Above is a test-packet I fired to test my new tcpdump options.
>
> Now I have to wait for another log to disappear
>
> --
> Florian Seifer
> Berater secunet Anwendungen, Managed Security Services
> Division Operational Services
> secunet Security Networks AG
>
> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> florian.seifer@secunet.com
> Kurfürstenstraße 58, 45138 Essen, Germany
> www.secunet.com
>
> -----------------------------------------------------------------------
> Sitz: Kurfürstenstraße 58, 45138 Essen
> Amtsgericht Essen HRB 13615
> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> Aufsichtsratsvorsitzender: Ralf Wintergerst
>
> -----Ursprüngliche Nachricht-----
> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
> Gesendet: Mittwoch, 11. November 2020 09:34
> An: Seifer, Florian <florian.seifer@secunet.com>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>
> Maybe you can just use wireshark -- a screen shot from the gui is
> probably sufficient.
>
> Rainer
>
> El mié., 11 nov. 2020 a las 9:31, Seifer, Florian
> (<florian.seifer@secunet.com>) escribió:
> >
> > Hello Rainer,
> >
> > I am not terribly firm with tcpdump, how would I go about doing that?
> >
> > Also I forgot to mention, the logs are always coming from the same Host with exactly the same format.
> > ( A VmWare logInsight-Server if that’s relevant) so I doubt the messages are suddenly changing form.
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > Gesendet: Mittwoch, 11. November 2020 09:26
> > An: Seifer, Florian <florian.seifer@secunet.com>
> > Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> > Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
> >
> > looks good. my gut feeling is that some senders do not properly
> > terminate the syslog frame with '\n'. Can you check you captures for a
> > case where a message is missing and post all bytes of the last seen
> > and missing messages in the correct sequence?
> >
> > Rainer
> >
> > El mié., 11 nov. 2020 a las 9:16, Seifer, Florian
> > (<florian.seifer@secunet.com>) escribió:
> > >
> > > # rsyslog configuration file
> > >
> > > # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
> > > # or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
> > > # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
> > >
> > > #### MODULES ####
> > >
> > > module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
> > > SysSock.Use="off") # Turn off message reception via local log socket;
> > > # local messages are retrieved through imjournal now.
> > > module(load="imjournal" # provides access to the systemd journal
> > > StateFile="imjournal.state") # File to store the position in the journal
> > > #module(load="imklog") # reads kernel messages (the same are read from journald)
> > > #module(load"immark") # provides --MARK-- message capability
> > >
> > > # Provides UDP syslog reception
> > > # for parameters see http://www.rsyslog.com/doc/imudp.html
> > > module(load="imudp") # needs to be done just once
> > > input(type="imudp" port="514")
> > >
> > > # Provides TCP syslog reception
> > > # for parameters see http://www.rsyslog.com/doc/imtcp.html
> > > module(load="imtcp") # needs to be done just once
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > #### GLOBAL DIRECTIVES ####
> > >
> > > # Where to place auxiliary files
> > > global(workDirectory="/var/lib/rsyslog")
> > >
> > > # Use default timestamp format
> > > module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
> > >
> > > # Include all config files in /etc/rsyslog.d/
> > > include(file="/etc/rsyslog.d/*.conf" mode="optional")
> > >
> > > #### RULES ####
> > >
> > > # Log all kernel messages to the console.
> > > # Logging much else clutters up the screen.
> > > #kern.* /dev/console
> > >
> > > # Log anything (except mail) of level info or higher.
> > > # Don't log private authentication messages!
> > > *.info;mail.none;authpriv.none;cron.none /var/log/messages
> > >
> > > # The authpriv file has restricted access.
> > > authpriv.* /var/log/secure
> > >
> > > # Log all the mail messages in one place.
> > > mail.* -/var/log/maillog
> > >
> > >
> > > # Log cron stuff
> > > cron.* /var/log/cron
> > >
> > > # Everybody gets emergency messages
> > > *.emerg :omusrmsg:*
> > >
> > > # Save news errors of level crit and higher in a special file.
> > > uucp,news.crit /var/log/spooler
> > >
> > > # Save boot messages also to boot.log
> > > local7.* /var/log/boot.log
> > >
> > >
> > >
> > >
> > >
> > >
> > > # ### sample forwarding rule ###
> > > #action(type="omfwd"
> > > # An on-disk queue is created for this action. If the remote host is
> > > # down, messages are spooled to disk and sent when it is up again.
> > > #queue.filename="fwdRule1" # unique name prefix for spool files
> > > #queue.maxdiskspace="1g" # 1gb space limit (use as much as possible)
> > > #queue.saveonshutdown="on" # save messages to disk on shutdown
> > > #queue.type="LinkedList" # run asynchronously
> > > #action.resumeRetryCount="-1" # infinite retries if host is down
> > > # Remote Logging (we use TCP for reliable delivery)
> > > # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
> > > #Target="remote_host" Port="XXX" Protocol="tcp")
> > >
> > >
> > > # Diese Regel erstellt pro Hostname ein eigenes Verzeichnis
> > > $template RemoteHost,"/var/log/remote/%hostname%/%programname%.log"
> > >
> > > # Der Regelsatz remote wendet auf alle eingehenden Logs das Template RemoteHost an:
> > >
> > > $RuleSet remote
> > > *.* ?RemoteHost
> > >
> > > # Nun binden wir den Regelsatz "remote" an alle syslog Meldungen von Außen.
> > >
> > > input(type="imtcp" port="514" ruleset="remote" supportoctetcountedframing="off")
> > >
> > > $EscapeControlCharactersOnReceive on
> > >
> > > $RepeatedMsgReduction off
> > >
> > > $SystemLogRateLimitInterval 0
> > >
> > > $DebugLevel 2
> > >
> > > $DebugFile /var/log/rsyslog/debug.log
> > >
> > > --
> > > Florian Seifer
> > > Berater secunet Anwendungen, Managed Security Services
> > > Division Operational Services
> > > secunet Security Networks AG
> > >
> > > Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> > > florian.seifer@secunet.com
> > > Kurfürstenstraße 58, 45138 Essen, Germany
> > > www.secunet.com
> > >
> > > -----------------------------------------------------------------------
> > > Sitz: Kurfürstenstraße 58, 45138 Essen
> > > Amtsgericht Essen HRB 13615
> > > Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> > > Aufsichtsratsvorsitzender: Ralf Wintergerst
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
> > > Gesendet: Mittwoch, 11. November 2020 09:13
> > > An: rsyslog-users <rsyslog@lists.adiscon.com>
> > > Cc: Seifer, Florian <florian.seifer@secunet.com>
> > > Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
> > >
> > > config?
> > >
> > > El mié., 11 nov. 2020 a las 9:11, Seifer, Florian via rsyslog
> > > (<rsyslog@lists.adiscon.com>) escribió:
> > > >
> > > > Hello,
> > > >
> > > > I have a rather strange problem with an rsyslog-machine.
> > > >
> > > > I configured it to process incoming logs over network via tcp port 514 using imtcp.
> > > >
> > > > The incoming logs on that connection are simply stored in a specific folder in a log-file.
> > > > This setup works 90% of the time but sometimes logs get "lost".
> > > >
> > > > I have no idea what causes it. I set up a tcpdump on the receiving machine and I can confirm that the packages are indeed being delivered.
> > > > But for some reason rsyslog does not notice them. I can find all other log-messages mentioned in the rsyslog-debug log, where they are noticed and processed correctly.
> > > >
> > > > The ones that don't make it to the logfiles are no where to be found. I cannot find any difference in these files, so I would expect them to be handled identically.
> > > >
> > > > Has anyone of you ever had similar problems and found a solution? Its not a firewall or network problem as the packages are definitely reaching the client.
> > > >
> > > > I am grateful for any support you guys can provide
> > > >
> > > > With kind regards
> > > >
> > > > --
> > > > Florian Seifer
> > > > Berater secunet Anwendungen, Managed Security Services
> > > > Division Operational Services
> > > > secunet Security Networks AG
> > > >
> > > > Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
> > > > florian.seifer@secunet.com
> > > > Kurfürstenstraße 58, 45138 Essen, Germany
> > > > www.secunet.com
> > > >
> > > > -----------------------------------------------------------------------
> > > > Sitz: Kurfürstenstraße 58, 45138 Essen
> > > > Amtsgericht Essen HRB 13615
> > > > Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
> > > > Aufsichtsratsvorsitzender: Ralf Wintergerst
> > > >
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog ignoring random incoming tcp-messages [ In reply to ]
Hi,

I think an strace (-s 8192) on the receiver host can be useful.

Benoit



Le 11/11/2020 à 10:20, Rainer Gerhards via rsyslog a écrit :
> looks good! Looking forward to the real sample. You can also check
> yourself if the frame is terminated with LF (hex 0a). In the sample it
> is.
>
> Rainer
>
> El mié., 11 nov. 2020 a las 10:18, Seifer, Florian
> (<florian.seifer@secunet.com>) escribió:
>>
>> Ok I got it:
>>
>> 0x0000: 4500 0130 ad1e 4000 3d06 8571 0a35 cd0d E..0..@.=..q.5..
>> 0x0010: 0a35 28c1 c26c 0202 acec aacd c2a8 f13b .5(..l.........;
>> 0x0020: 8018 001d 92c6 0000 0101 080a d80f e141 ...............A
>> 0x0030: 8125 ea9a 3230 3230 2d31 312d 3131 5430 .%..2020-11-11T0
>> 0x0040: 393a 3134 3a35 362e 3436 365a 2031 302e 9:14:56.466Z.10.
>> 0x0050: 3533 2e32 3035 2e32 202d 202d 202d 202d 53.205.2.-.-.-.-
>> 0x0060: 2044 6965 7320 6973 7420 6569 6e65 2076 .Dies.ist.eine.v
>> 0x0070: 6f6e 204c 6f67 2049 6e73 6967 6874 2065 on.Log.Insight.e
>> 0x0080: 7273 7465 6c6c 7465 2075 6e64 2061 6e20 rstellte.und.an.
>> 0x0090: 6c6f 6777 6174 6368 2e73 6563 756e 6574 logwatch.secunet
>> 0x00a0: 2e64 653a 3531 3420 6d69 7420 5241 5720 .de:514.mit.RAW.
>> 0x00b0: 5443 5020 2875 6e64 206d 6974 2054 6167 TCP.(und.mit.Tag
>> 0x00c0: 7320 686f 7374 3d7a 6b7a 2920 6765 7365 s.host=zkz).gese
>> 0x00d0: 6e64 6574 6520 5465 7374 6e61 6368 7269 ndete.Testnachri
>> 0x00e0: 6368 7420 66c3 bc72 2064 6965 2045 7265 cht.f..r.die.Ere
>> 0x00f0: 6967 6e69 7377 6569 7465 726c 6569 7475 ignisweiterleitu
>> 0x0100: 6e67 206d 6974 2064 656d 204e 616d 656e ng.mit.dem.Namen
>> 0x0110: 2027 416c 6172 6d61 6e6c 6167 6520 5363 .'Alarmanlage.Sc
>> 0x0120: 6861 7266 2f55 6e73 6368 6172 6627 2e0a harf/Unscharf'..
>>
>> Above is a test-packet I fired to test my new tcpdump options.
>>
>> Now I have to wait for another log to disappear
>>
>> --
>> Florian Seifer
>> Berater secunet Anwendungen, Managed Security Services
>> Division Operational Services
>> secunet Security Networks AG
>>
>> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
>> florian.seifer@secunet.com
>> Kurfürstenstraße 58, 45138 Essen, Germany
>> www.secunet.com
>>
>> -----------------------------------------------------------------------
>> Sitz: Kurfürstenstraße 58, 45138 Essen
>> Amtsgericht Essen HRB 13615
>> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
>> Aufsichtsratsvorsitzender: Ralf Wintergerst
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
>> Gesendet: Mittwoch, 11. November 2020 09:34
>> An: Seifer, Florian <florian.seifer@secunet.com>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>>
>> Maybe you can just use wireshark -- a screen shot from the gui is
>> probably sufficient.
>>
>> Rainer
>>
>> El mié., 11 nov. 2020 a las 9:31, Seifer, Florian
>> (<florian.seifer@secunet.com>) escribió:
>>>
>>> Hello Rainer,
>>>
>>> I am not terribly firm with tcpdump, how would I go about doing that?
>>>
>>> Also I forgot to mention, the logs are always coming from the same Host with exactly the same format.
>>> ( A VmWare logInsight-Server if that’s relevant) so I doubt the messages are suddenly changing form.
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
>>> Gesendet: Mittwoch, 11. November 2020 09:26
>>> An: Seifer, Florian <florian.seifer@secunet.com>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>>>
>>> looks good. my gut feeling is that some senders do not properly
>>> terminate the syslog frame with '\n'. Can you check you captures for a
>>> case where a message is missing and post all bytes of the last seen
>>> and missing messages in the correct sequence?
>>>
>>> Rainer
>>>
>>> El mié., 11 nov. 2020 a las 9:16, Seifer, Florian
>>> (<florian.seifer@secunet.com>) escribió:
>>>>
>>>> # rsyslog configuration file
>>>>
>>>> # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
>>>> # or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
>>>> # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
>>>>
>>>> #### MODULES ####
>>>>
>>>> module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
>>>> SysSock.Use="off") # Turn off message reception via local log socket;
>>>> # local messages are retrieved through imjournal now.
>>>> module(load="imjournal" # provides access to the systemd journal
>>>> StateFile="imjournal.state") # File to store the position in the journal
>>>> #module(load="imklog") # reads kernel messages (the same are read from journald)
>>>> #module(load"immark") # provides --MARK-- message capability
>>>>
>>>> # Provides UDP syslog reception
>>>> # for parameters see http://www.rsyslog.com/doc/imudp.html
>>>> module(load="imudp") # needs to be done just once
>>>> input(type="imudp" port="514")
>>>>
>>>> # Provides TCP syslog reception
>>>> # for parameters see http://www.rsyslog.com/doc/imtcp.html
>>>> module(load="imtcp") # needs to be done just once
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> #### GLOBAL DIRECTIVES ####
>>>>
>>>> # Where to place auxiliary files
>>>> global(workDirectory="/var/lib/rsyslog")
>>>>
>>>> # Use default timestamp format
>>>> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
>>>>
>>>> # Include all config files in /etc/rsyslog.d/
>>>> include(file="/etc/rsyslog.d/*.conf" mode="optional")
>>>>
>>>> #### RULES ####
>>>>
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> #kern.* /dev/console
>>>>
>>>> # Log anything (except mail) of level info or higher.
>>>> # Don't log private authentication messages!
>>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>>>
>>>> # The authpriv file has restricted access.
>>>> authpriv.* /var/log/secure
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.* -/var/log/maillog
>>>>
>>>>
>>>> # Log cron stuff
>>>> cron.* /var/log/cron
>>>>
>>>> # Everybody gets emergency messages
>>>> *.emerg :omusrmsg:*
>>>>
>>>> # Save news errors of level crit and higher in a special file.
>>>> uucp,news.crit /var/log/spooler
>>>>
>>>> # Save boot messages also to boot.log
>>>> local7.* /var/log/boot.log
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> # ### sample forwarding rule ###
>>>> #action(type="omfwd"
>>>> # An on-disk queue is created for this action. If the remote host is
>>>> # down, messages are spooled to disk and sent when it is up again.
>>>> #queue.filename="fwdRule1" # unique name prefix for spool files
>>>> #queue.maxdiskspace="1g" # 1gb space limit (use as much as possible)
>>>> #queue.saveonshutdown="on" # save messages to disk on shutdown
>>>> #queue.type="LinkedList" # run asynchronously
>>>> #action.resumeRetryCount="-1" # infinite retries if host is down
>>>> # Remote Logging (we use TCP for reliable delivery)
>>>> # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
>>>> #Target="remote_host" Port="XXX" Protocol="tcp")
>>>>
>>>>
>>>> # Diese Regel erstellt pro Hostname ein eigenes Verzeichnis
>>>> $template RemoteHost,"/var/log/remote/%hostname%/%programname%.log"
>>>>
>>>> # Der Regelsatz remote wendet auf alle eingehenden Logs das Template RemoteHost an:
>>>>
>>>> $RuleSet remote
>>>> *.* ?RemoteHost
>>>>
>>>> # Nun binden wir den Regelsatz "remote" an alle syslog Meldungen von Außen.
>>>>
>>>> input(type="imtcp" port="514" ruleset="remote" supportoctetcountedframing="off")
>>>>
>>>> $EscapeControlCharactersOnReceive on
>>>>
>>>> $RepeatedMsgReduction off
>>>>
>>>> $SystemLogRateLimitInterval 0
>>>>
>>>> $DebugLevel 2
>>>>
>>>> $DebugFile /var/log/rsyslog/debug.log
>>>>
>>>> --
>>>> Florian Seifer
>>>> Berater secunet Anwendungen, Managed Security Services
>>>> Division Operational Services
>>>> secunet Security Networks AG
>>>>
>>>> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
>>>> florian.seifer@secunet.com
>>>> Kurfürstenstraße 58, 45138 Essen, Germany
>>>> www.secunet.com
>>>>
>>>> -----------------------------------------------------------------------
>>>> Sitz: Kurfürstenstraße 58, 45138 Essen
>>>> Amtsgericht Essen HRB 13615
>>>> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
>>>> Aufsichtsratsvorsitzender: Ralf Wintergerst
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
>>>> Gesendet: Mittwoch, 11. November 2020 09:13
>>>> An: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Cc: Seifer, Florian <florian.seifer@secunet.com>
>>>> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>>>>
>>>> config?
>>>>
>>>> El mié., 11 nov. 2020 a las 9:11, Seifer, Florian via rsyslog
>>>> (<rsyslog@lists.adiscon.com>) escribió:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I have a rather strange problem with an rsyslog-machine.
>>>>>
>>>>> I configured it to process incoming logs over network via tcp port 514 using imtcp.
>>>>>
>>>>> The incoming logs on that connection are simply stored in a specific folder in a log-file.
>>>>> This setup works 90% of the time but sometimes logs get "lost".
>>>>>
>>>>> I have no idea what causes it. I set up a tcpdump on the receiving machine and I can confirm that the packages are indeed being delivered.
>>>>> But for some reason rsyslog does not notice them. I can find all other log-messages mentioned in the rsyslog-debug log, where they are noticed and processed correctly.
>>>>>
>>>>> The ones that don't make it to the logfiles are no where to be found. I cannot find any difference in these files, so I would expect them to be handled identically.
>>>>>
>>>>> Has anyone of you ever had similar problems and found a solution? Its not a firewall or network problem as the packages are definitely reaching the client.
>>>>>
>>>>> I am grateful for any support you guys can provide
>>>>>
>>>>> With kind regards
>>>>>
>>>>> --
>>>>> Florian Seifer
>>>>> Berater secunet Anwendungen, Managed Security Services
>>>>> Division Operational Services
>>>>> secunet Security Networks AG
>>>>>
>>>>> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
>>>>> florian.seifer@secunet.com
>>>>> Kurfürstenstraße 58, 45138 Essen, Germany
>>>>> www.secunet.com
>>>>>
>>>>> -----------------------------------------------------------------------
>>>>> Sitz: Kurfürstenstraße 58, 45138 Essen
>>>>> Amtsgericht Essen HRB 13615
>>>>> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
>>>>> Aufsichtsratsvorsitzender: Ralf Wintergerst
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>

--
Benoit DOLEZ
GSM: +33 6 21 05 91 69 mailto:bdolez@ant-computing.com
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog ignoring random incoming tcp-messages [ In reply to ]
Strace of what?

Rsyslog?

--
Florian Seifer
Berater secunet Anwendungen, Managed Security Services
Division Operational Services
secunet Security Networks AG

Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
florian.seifer@secunet.com
Kurfürstenstraße 58, 45138 Essen, Germany
www.secunet.com

-----------------------------------------------------------------------
Sitz: Kurfürstenstraße 58, 45138 Essen
Amtsgericht Essen HRB 13615
Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
Aufsichtsratsvorsitzender: Ralf Wintergerst

-----Ursprüngliche Nachricht-----
Von: rsyslog <rsyslog-bounces@lists.adiscon.com> Im Auftrag von Benoit DOLEZ via rsyslog
Gesendet: Mittwoch, 11. November 2020 10:54
An: rsyslog@lists.adiscon.com
Cc: Benoit DOLEZ <bdolez@ant-computing.com>
Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages

Hi,

I think an strace (-s 8192) on the receiver host can be useful.

Benoit



Le 11/11/2020 à 10:20, Rainer Gerhards via rsyslog a écrit :
> looks good! Looking forward to the real sample. You can also check
> yourself if the frame is terminated with LF (hex 0a). In the sample it
> is.
>
> Rainer
>
> El mié., 11 nov. 2020 a las 10:18, Seifer, Florian
> (<florian.seifer@secunet.com>) escribió:
>>
>> Ok I got it:
>>
>> 0x0000: 4500 0130 ad1e 4000 3d06 8571 0a35 cd0d E..0..@.=..q.5..
>> 0x0010: 0a35 28c1 c26c 0202 acec aacd c2a8 f13b .5(..l.........;
>> 0x0020: 8018 001d 92c6 0000 0101 080a d80f e141 ...............A
>> 0x0030: 8125 ea9a 3230 3230 2d31 312d 3131 5430 .%..2020-11-11T0
>> 0x0040: 393a 3134 3a35 362e 3436 365a 2031 302e 9:14:56.466Z.10.
>> 0x0050: 3533 2e32 3035 2e32 202d 202d 202d 202d 53.205.2.-.-.-.-
>> 0x0060: 2044 6965 7320 6973 7420 6569 6e65 2076 .Dies.ist.eine.v
>> 0x0070: 6f6e 204c 6f67 2049 6e73 6967 6874 2065 on.Log.Insight.e
>> 0x0080: 7273 7465 6c6c 7465 2075 6e64 2061 6e20 rstellte.und.an.
>> 0x0090: 6c6f 6777 6174 6368 2e73 6563 756e 6574 logwatch.secunet
>> 0x00a0: 2e64 653a 3531 3420 6d69 7420 5241 5720 .de:514.mit.RAW.
>> 0x00b0: 5443 5020 2875 6e64 206d 6974 2054 6167 TCP.(und.mit.Tag
>> 0x00c0: 7320 686f 7374 3d7a 6b7a 2920 6765 7365 s.host=zkz).gese
>> 0x00d0: 6e64 6574 6520 5465 7374 6e61 6368 7269 ndete.Testnachri
>> 0x00e0: 6368 7420 66c3 bc72 2064 6965 2045 7265 cht.f..r.die.Ere
>> 0x00f0: 6967 6e69 7377 6569 7465 726c 6569 7475 ignisweiterleitu
>> 0x0100: 6e67 206d 6974 2064 656d 204e 616d 656e ng.mit.dem.Namen
>> 0x0110: 2027 416c 6172 6d61 6e6c 6167 6520 5363 .'Alarmanlage.Sc
>> 0x0120: 6861 7266 2f55 6e73 6368 6172 6627 2e0a harf/Unscharf'..
>>
>> Above is a test-packet I fired to test my new tcpdump options.
>>
>> Now I have to wait for another log to disappear
>>
>> --
>> Florian Seifer
>> Berater secunet Anwendungen, Managed Security Services
>> Division Operational Services
>> secunet Security Networks AG
>>
>> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
>> florian.seifer@secunet.com
>> Kurfürstenstraße 58, 45138 Essen, Germany
>> www.secunet.com
>>
>> -----------------------------------------------------------------------
>> Sitz: Kurfürstenstraße 58, 45138 Essen
>> Amtsgericht Essen HRB 13615
>> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
>> Aufsichtsratsvorsitzender: Ralf Wintergerst
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
>> Gesendet: Mittwoch, 11. November 2020 09:34
>> An: Seifer, Florian <florian.seifer@secunet.com>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>>
>> Maybe you can just use wireshark -- a screen shot from the gui is
>> probably sufficient.
>>
>> Rainer
>>
>> El mié., 11 nov. 2020 a las 9:31, Seifer, Florian
>> (<florian.seifer@secunet.com>) escribió:
>>>
>>> Hello Rainer,
>>>
>>> I am not terribly firm with tcpdump, how would I go about doing that?
>>>
>>> Also I forgot to mention, the logs are always coming from the same Host with exactly the same format.
>>> ( A VmWare logInsight-Server if that’s relevant) so I doubt the messages are suddenly changing form.
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
>>> Gesendet: Mittwoch, 11. November 2020 09:26
>>> An: Seifer, Florian <florian.seifer@secunet.com>
>>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>>> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>>>
>>> looks good. my gut feeling is that some senders do not properly
>>> terminate the syslog frame with '\n'. Can you check you captures for a
>>> case where a message is missing and post all bytes of the last seen
>>> and missing messages in the correct sequence?
>>>
>>> Rainer
>>>
>>> El mié., 11 nov. 2020 a las 9:16, Seifer, Florian
>>> (<florian.seifer@secunet.com>) escribió:
>>>>
>>>> # rsyslog configuration file
>>>>
>>>> # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
>>>> # or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
>>>> # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
>>>>
>>>> #### MODULES ####
>>>>
>>>> module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
>>>> SysSock.Use="off") # Turn off message reception via local log socket;
>>>> # local messages are retrieved through imjournal now.
>>>> module(load="imjournal" # provides access to the systemd journal
>>>> StateFile="imjournal.state") # File to store the position in the journal
>>>> #module(load="imklog") # reads kernel messages (the same are read from journald)
>>>> #module(load"immark") # provides --MARK-- message capability
>>>>
>>>> # Provides UDP syslog reception
>>>> # for parameters see http://www.rsyslog.com/doc/imudp.html
>>>> module(load="imudp") # needs to be done just once
>>>> input(type="imudp" port="514")
>>>>
>>>> # Provides TCP syslog reception
>>>> # for parameters see http://www.rsyslog.com/doc/imtcp.html
>>>> module(load="imtcp") # needs to be done just once
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> #### GLOBAL DIRECTIVES ####
>>>>
>>>> # Where to place auxiliary files
>>>> global(workDirectory="/var/lib/rsyslog")
>>>>
>>>> # Use default timestamp format
>>>> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
>>>>
>>>> # Include all config files in /etc/rsyslog.d/
>>>> include(file="/etc/rsyslog.d/*.conf" mode="optional")
>>>>
>>>> #### RULES ####
>>>>
>>>> # Log all kernel messages to the console.
>>>> # Logging much else clutters up the screen.
>>>> #kern.* /dev/console
>>>>
>>>> # Log anything (except mail) of level info or higher.
>>>> # Don't log private authentication messages!
>>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>>>>
>>>> # The authpriv file has restricted access.
>>>> authpriv.* /var/log/secure
>>>>
>>>> # Log all the mail messages in one place.
>>>> mail.* -/var/log/maillog
>>>>
>>>>
>>>> # Log cron stuff
>>>> cron.* /var/log/cron
>>>>
>>>> # Everybody gets emergency messages
>>>> *.emerg :omusrmsg:*
>>>>
>>>> # Save news errors of level crit and higher in a special file.
>>>> uucp,news.crit /var/log/spooler
>>>>
>>>> # Save boot messages also to boot.log
>>>> local7.* /var/log/boot.log
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> # ### sample forwarding rule ###
>>>> #action(type="omfwd"
>>>> # An on-disk queue is created for this action. If the remote host is
>>>> # down, messages are spooled to disk and sent when it is up again.
>>>> #queue.filename="fwdRule1" # unique name prefix for spool files
>>>> #queue.maxdiskspace="1g" # 1gb space limit (use as much as possible)
>>>> #queue.saveonshutdown="on" # save messages to disk on shutdown
>>>> #queue.type="LinkedList" # run asynchronously
>>>> #action.resumeRetryCount="-1" # infinite retries if host is down
>>>> # Remote Logging (we use TCP for reliable delivery)
>>>> # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
>>>> #Target="remote_host" Port="XXX" Protocol="tcp")
>>>>
>>>>
>>>> # Diese Regel erstellt pro Hostname ein eigenes Verzeichnis
>>>> $template RemoteHost,"/var/log/remote/%hostname%/%programname%.log"
>>>>
>>>> # Der Regelsatz remote wendet auf alle eingehenden Logs das Template RemoteHost an:
>>>>
>>>> $RuleSet remote
>>>> *.* ?RemoteHost
>>>>
>>>> # Nun binden wir den Regelsatz "remote" an alle syslog Meldungen von Außen.
>>>>
>>>> input(type="imtcp" port="514" ruleset="remote" supportoctetcountedframing="off")
>>>>
>>>> $EscapeControlCharactersOnReceive on
>>>>
>>>> $RepeatedMsgReduction off
>>>>
>>>> $SystemLogRateLimitInterval 0
>>>>
>>>> $DebugLevel 2
>>>>
>>>> $DebugFile /var/log/rsyslog/debug.log
>>>>
>>>> --
>>>> Florian Seifer
>>>> Berater secunet Anwendungen, Managed Security Services
>>>> Division Operational Services
>>>> secunet Security Networks AG
>>>>
>>>> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
>>>> florian.seifer@secunet.com
>>>> Kurfürstenstraße 58, 45138 Essen, Germany
>>>> www.secunet.com
>>>>
>>>> -----------------------------------------------------------------------
>>>> Sitz: Kurfürstenstraße 58, 45138 Essen
>>>> Amtsgericht Essen HRB 13615
>>>> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
>>>> Aufsichtsratsvorsitzender: Ralf Wintergerst
>>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Rainer Gerhards <rgerhards@hq.adiscon.com>
>>>> Gesendet: Mittwoch, 11. November 2020 09:13
>>>> An: rsyslog-users <rsyslog@lists.adiscon.com>
>>>> Cc: Seifer, Florian <florian.seifer@secunet.com>
>>>> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages
>>>>
>>>> config?
>>>>
>>>> El mié., 11 nov. 2020 a las 9:11, Seifer, Florian via rsyslog
>>>> (<rsyslog@lists.adiscon.com>) escribió:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I have a rather strange problem with an rsyslog-machine.
>>>>>
>>>>> I configured it to process incoming logs over network via tcp port 514 using imtcp.
>>>>>
>>>>> The incoming logs on that connection are simply stored in a specific folder in a log-file.
>>>>> This setup works 90% of the time but sometimes logs get "lost".
>>>>>
>>>>> I have no idea what causes it. I set up a tcpdump on the receiving machine and I can confirm that the packages are indeed being delivered.
>>>>> But for some reason rsyslog does not notice them. I can find all other log-messages mentioned in the rsyslog-debug log, where they are noticed and processed correctly.
>>>>>
>>>>> The ones that don't make it to the logfiles are no where to be found. I cannot find any difference in these files, so I would expect them to be handled identically.
>>>>>
>>>>> Has anyone of you ever had similar problems and found a solution? Its not a firewall or network problem as the packages are definitely reaching the client.
>>>>>
>>>>> I am grateful for any support you guys can provide
>>>>>
>>>>> With kind regards
>>>>>
>>>>> --
>>>>> Florian Seifer
>>>>> Berater secunet Anwendungen, Managed Security Services
>>>>> Division Operational Services
>>>>> secunet Security Networks AG
>>>>>
>>>>> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259
>>>>> florian.seifer@secunet.com
>>>>> Kurfürstenstraße 58, 45138 Essen, Germany
>>>>> www.secunet.com
>>>>>
>>>>> -----------------------------------------------------------------------
>>>>> Sitz: Kurfürstenstraße 58, 45138 Essen
>>>>> Amtsgericht Essen HRB 13615
>>>>> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines
>>>>> Aufsichtsratsvorsitzender: Ralf Wintergerst
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>

--
Benoit DOLEZ
GSM: +33 6 21 05 91 69 mailto:bdolez@ant-computing.com
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: rsyslog ignoring random incoming tcp-messages [ In reply to ]
Yes, strace of the rsyslogd process on the receiver host (the one who
lose data).

Example :
# strace -r -tt -T -s 8192 -o /tmp/rsyslog.strace -p $(pidof rsyslogd)

Benoit

Le 11/11/2020 à 11:00, Seifer, Florian a écrit :
> Strace of what?
>
> Rsyslog?
>

--
Benoit DOLEZ
GSM: +33 6 21 05 91 69 mailto:bdolez@ant-computing.com
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.