Mailing List Archive

Rsyslog agent 7.0 - Increase Processed files Speed
Good Morning All
We have a lot of log files to process and send to siem, the csv files are
about 1 GB dimension and are download on filesystem every 15 minutes

How can we increase the line (then files) processed for seconds?

Follow the screenshot:
[image: image.png]

The Lines count seems not very performed.

Are there some parameters that we can change in order to process faster?

The second question is related to the processed file: Where can we find the
filename of the processed files?

This would we useful to move them in other path or deleted them


Thanks in advance


*Fabio Danìa*

Information & Communication Technology

Authentication & Network services



*On behalf of** FCA Item*

Corso Luigi Settembrini 167, Ingresso 19

10135 Torino – ITALY
Re: Rsyslog agent 7.0 - Increase Processed files Speed [ In reply to ]
Dear Fabio,

Please see my comments below:

> -----Ursprüngliche Nachricht-----
> Von: rsyslog <rsyslog-bounces@lists.adiscon.com> Im Auftrag von Fabio
> Dania via rsyslog
> Gesendet: Freitag, 16. April 2021 19:41
> An: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Fabio Dania <fabio.dania@external.fcagroup.com>
> Betreff: [rsyslog] Rsyslog agent 7.0 - Increase Processed files Speed
>
> Good Morning All
> We have a lot of log files to process and send to siem, the csv files are
> about
> 1 GB dimension and are download on filesystem every 15 minutes
>
> How can we increase the line (then files) processed for seconds?

For highest performance, keep Overrun Prevention Delay (ms) Setting to 0 -
this will remove any delay from the processing.
Then I recommend to enable "Delay writing Last File Position" and set it to
100 which means the last processed file position will only be written every
100 lines or after a timeout (If all lines are processed).

> The Lines count seems not very performed.

The count does not reflect the lines processed but the actual byte position
of the last processed line within the file.

> The second question is related to the processed file: Where can we find
> the
> filename of the processed files?

If you are using registry configuration mode, they can be find in the
registry under:
HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Adiscon\RSyslogAgentData\Services\{GUIDOFSERVICE}

Each processed file will have its own node there with nLastPos:Low and
nLastPos:High as DWORD values.
nLastPos:Low is the file position if the file does not exceed 4GB.



Best regards,
Andre Lorbach
--
Adiscon GmbH
Mozartstr. 21
97950 Großrinderfeld, Germany
Ph. +49-9349-9298530
Geschäftsführer/President: Rainer Gerhards Reg.-Gericht Mannheim, HRB
560610
Ust.-IDNr.: DE 81 22 04 622
Web: www.adiscon.com - Mail: info@adiscon.com

Informations regarding your data privacy policy can be found here:
https://www.adiscon.com/data-privacy-policy/

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient or have received this e-mail in error please
notify the sender immediately and delete this e-mail. Any unauthorized
copying, disclosure or distribution of the material in this e-mail is
strictly forbidden.

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese E-Mail. Das unerlaubte Kopieren und die unbefugte
Weitergabe dieser E-Mail sind nicht gestattet.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: Rsyslog agent 7.0 - Increase Processed files Speed [ In reply to ]
Thanks so Much Andre
We have implemented the suggests parameters.

Now we have about 20 MB per minute: is it a good performance for rsyslog
agent or is it possibile improve the speed?
Our csv files are around 1 GB and aren't a few




*Fabio Danìa*

Information & Communication Technology

Authentication & Network services



*On behalf of** FCA Item*

Corso Luigi Settembrini 167, Ingresso 19

10135 Torino – ITALY


On Mon, Apr 19, 2021 at 9:13 AM Andre Lorbach <alorbach@adiscon.com> wrote:

> Dear Fabio,
>
> Please see my comments below:
>
> > -----Ursprüngliche Nachricht-----
> > Von: rsyslog <rsyslog-bounces@lists.adiscon.com> Im Auftrag von Fabio
> > Dania via rsyslog
> > Gesendet: Freitag, 16. April 2021 19:41
> > An: rsyslog-users <rsyslog@lists.adiscon.com>
> > Cc: Fabio Dania <fabio.dania@external.fcagroup.com>
> > Betreff: [rsyslog] Rsyslog agent 7.0 - Increase Processed files Speed
> >
> > Good Morning All
> > We have a lot of log files to process and send to siem, the csv files are
> > about
> > 1 GB dimension and are download on filesystem every 15 minutes
> >
> > How can we increase the line (then files) processed for seconds?
>
> For highest performance, keep Overrun Prevention Delay (ms) Setting to 0 -
> this will remove any delay from the processing.
> Then I recommend to enable "Delay writing Last File Position" and set it to
> 100 which means the last processed file position will only be written every
> 100 lines or after a timeout (If all lines are processed).
>
> > The Lines count seems not very performed.
>
> The count does not reflect the lines processed but the actual byte position
> of the last processed line within the file.
>
> > The second question is related to the processed file: Where can we find
> > the
> > filename of the processed files?
>
> If you are using registry configuration mode, they can be find in the
> registry under:
>
> HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Adiscon\RSyslogAgentData\Services\{GUIDOFSERVICE}
>
> Each processed file will have its own node there with nLastPos:Low and
> nLastPos:High as DWORD values.
> nLastPos:Low is the file position if the file does not exceed 4GB.
>
>
>
> Best regards,
> Andre Lorbach
> --
> Adiscon GmbH
> Mozartstr. 21
> 97950 Großrinderfeld, Germany
> Ph. +49-9349-9298530
> Geschäftsführer/President: Rainer Gerhards Reg.-Gericht Mannheim, HRB
> 560610
> Ust.-IDNr.: DE 81 22 04 622
> Web: www.adiscon.com - Mail: info@adiscon.com
>
> Informations regarding your data privacy policy can be found here:
> https://www.adiscon.com/data-privacy-policy/
>
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient or have received this e-mail in error please
> notify the sender immediately and delete this e-mail. Any unauthorized
> copying, disclosure or distribution of the material in this e-mail is
> strictly forbidden.
>
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
> vernichten Sie diese E-Mail. Das unerlaubte Kopieren und die unbefugte
> Weitergabe dieser E-Mail sind nicht gestattet.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.