Mailing List Archive

does

set $.newmsg = replace(...);

and using the $.newmsg variable inside the template work? On first
look it should...

HTH
Rainer

El jue, 17 dic 2020 a las 18:34, Salvatore Totaro via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> Hi,
>
> I have a msg property made in this way:
>
> 1.5470373000e+05,nan,1.5060465000e+05,nan,2.1682000000e+02,nan,2.4761550000e+04,nan,1.6121420000e+04,nan
>
> In order to insert this row in my postgres db I use this template:
>
> template(name="energy_value" type="list" option.sql="on") {
> constant(value="INSERT INTO energyvalue (datetime, made_by, Nh_T1, Nh_T2, Ph_im_T1, Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2, Qh_ex_T1, Qh_ex_T2 ) values ('")
> property(name="timereported" dateformat="pgsql" date.inUTC="on")
> constant(value="','")
> property(name="hostname")
> constant(value="',")
> property(name="msg")
> constant(value=")")
> }
>
> but postgresql has problem with nan value. The solution is to replace nan token with ‘nan’::double precision
>
> I tried to use replace function but with no luck. What is the right way to replace a regex like (N|nA|aN|n) with ‘nan’::double precision?
>
> thanks
> Salvatore
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hello,

I tried with this:
set $.newmsg = replace("nan nan nan","nan","'nan'::double precision");
template(name="RMM_energy_value" type="list" option.sql="on") {
constant(value="INSERT INTO energyvalue (datetime, made_by, Nh_T1, Nh_T2, Ph_im_T1, Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2, Qh_ex_T1, Qh_ex_T2 ) values ('")
property(name="timereported" dateformat="pgsql" date.inUTC="on")
constant(value="','")
property(name="hostname")
constant(value="',")
property(name="$.newmsg")
constant(value=")")
}

and with:
set $.newmsg = replace($msg,"nan","'nan'::double precision”);

and with:
set $.newmsg = “hello”;

but postgresql db receives this:
INSERT INTO energyvalue (datetime, made_by, Nh_T1, Nh_T2, Ph_im_T1, Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2, Qh_ex_T1, Qh_ex_T2 ) values ('2020-12-17 18:46:42','18137709’,)

Seems that $.newmsg is empty.

What I’m doing wrong?

thanks
Salvatore

> On 17 Dec 2020, at 18:34, Salvatore Totaro via rsyslog <rsyslog@lists.adiscon.com> wrote:
>
> Hi,
>
> I have a msg property made in this way:
>
> 1.5470373000e+05,nan,1.5060465000e+05,nan,2.1682000000e+02,nan,2.4761550000e+04,nan,1.6121420000e+04,nan
>
> In order to insert this row in my postgres db I use this template:
>
> template(name="energy_value" type="list" option.sql="on") {
> constant(value="INSERT INTO energyvalue (datetime, made_by, Nh_T1, Nh_T2, Ph_im_T1, Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2, Qh_ex_T1, Qh_ex_T2 ) values ('")
> property(name="timereported" dateformat="pgsql" date.inUTC="on")
> constant(value="','")
> property(name="hostname")
> constant(value="',")
> property(name="msg")
> constant(value=")")
> }
>
> but postgresql has problem with nan value. The solution is to replace nan token with ‘nan’::double precision
>
> I tried to use replace function but with no luck. What is the right way to replace a regex like (N|nA|aN|n) with ‘nan’::double precision?
>
> thanks
> Salvatore
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Full config pls.

Sent from phone, thus brief.

Salvatore Totaro via rsyslog <rsyslog@lists.adiscon.com> schrieb am Do.,
17. Dez. 2020, 19:55:

> Hello,
>
> I tried with this:
> set $.newmsg = replace("nan nan nan","nan","'nan'::double precision");
> template(name="RMM_energy_value" type="list" option.sql="on") {
> constant(value="INSERT INTO energyvalue (datetime, made_by, Nh_T1,
> Nh_T2, Ph_im_T1, Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2,
> Qh_ex_T1, Qh_ex_T2 ) values ('")
> property(name="timereported" dateformat="pgsql" date.inUTC="on")
> constant(value="','")
> property(name="hostname")
> constant(value="',")
> property(name="$.newmsg")
> constant(value=")")
> }
>
> and with:
> set $.newmsg = replace($msg,"nan","'nan'::double precision”);
>
> and with:
> set $.newmsg = “hello”;
>
> but postgresql db receives this:
> INSERT INTO energyvalue (datetime, made_by, Nh_T1, Nh_T2, Ph_im_T1,
> Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2, Qh_ex_T1, Qh_ex_T2 )
> values ('2020-12-17 18:46:42','18137709’,)
>
> Seems that $.newmsg is empty.
>
> What I’m doing wrong?
>
> thanks
> Salvatore
>
> > On 17 Dec 2020, at 18:34, Salvatore Totaro via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
> >
> > Hi,
> >
> > I have a msg property made in this way:
> >
> >
> 1.5470373000e+05,nan,1.5060465000e+05,nan,2.1682000000e+02,nan,2.4761550000e+04,nan,1.6121420000e+04,nan
> >
> > In order to insert this row in my postgres db I use this template:
> >
> > template(name="energy_value" type="list" option.sql="on") {
> > constant(value="INSERT INTO energyvalue (datetime, made_by, Nh_T1,
> Nh_T2, Ph_im_T1, Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2,
> Qh_ex_T1, Qh_ex_T2 ) values ('")
> > property(name="timereported" dateformat="pgsql" date.inUTC="on")
> > constant(value="','")
> > property(name="hostname")
> > constant(value="',")
> > property(name="msg")
> > constant(value=")")
> > }
> >
> > but postgresql has problem with nan value. The solution is to replace
> nan token with ‘nan’::double precision
> >
> > I tried to use replace function but with no luck. What is the right way
> to replace a regex like (N|nA|aN|n) with ‘nan’::double precision?
> >
> > thanks
> > Salvatore
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

after doing the set, write a logfile with the template RSYSLOG_DebugFormat so
you can see the contents of every variable.

David Lang

On Thu, 17 Dec 2020, Salvatore Totaro via rsyslog wrote:

> Date: Thu, 17 Dec 2020 19:55:18 +0100
> From: Salvatore Totaro via rsyslog <rsyslog@lists.adiscon.com>
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Salvatore Totaro <tsalva@tiscali.it>
> Subject: Re: [rsyslog] how to replace msg content
>
> Hello,
>
> I tried with this:
> set $.newmsg = replace("nan nan nan","nan","'nan'::double precision");
> template(name="RMM_energy_value" type="list" option.sql="on") {
> constant(value="INSERT INTO energyvalue (datetime, made_by, Nh_T1, Nh_T2, Ph_im_T1, Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2, Qh_ex_T1, Qh_ex_T2 ) values ('")
> property(name="timereported" dateformat="pgsql" date.inUTC="on")
> constant(value="','")
> property(name="hostname")
> constant(value="',")
> property(name="$.newmsg")
> constant(value=")")
> }
>
> and with:
> set $.newmsg = replace($msg,"nan","'nan'::double precision”);
>
> and with:
> set $.newmsg = “hello”;
>
> but postgresql db receives this:
> INSERT INTO energyvalue (datetime, made_by, Nh_T1, Nh_T2, Ph_im_T1, Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2, Qh_ex_T1, Qh_ex_T2 ) values ('2020-12-17 18:46:42','18137709’,)
>
> Seems that $.newmsg is empty.
>
> What I’m doing wrong?
>
> thanks
> Salvatore
>
>> On 17 Dec 2020, at 18:34, Salvatore Totaro via rsyslog <rsyslog@lists.adiscon.com> wrote:
>>
>> Hi,
>>
>> I have a msg property made in this way:
>>
>> 1.5470373000e+05,nan,1.5060465000e+05,nan,2.1682000000e+02,nan,2.4761550000e+04,nan,1.6121420000e+04,nan
>>
>> In order to insert this row in my postgres db I use this template:
>>
>> template(name="energy_value" type="list" option.sql="on") {
>> constant(value="INSERT INTO energyvalue (datetime, made_by, Nh_T1, Nh_T2, Ph_im_T1, Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2, Qh_ex_T1, Qh_ex_T2 ) values ('")
>> property(name="timereported" dateformat="pgsql" date.inUTC="on")
>> constant(value="','")
>> property(name="hostname")
>> constant(value="',")
>> property(name="msg")
>> constant(value=")")
>> }
>>
>> but postgresql has problem with nan value. The solution is to replace nan token with ‘nan’::double precision
>>
>> I tried to use replace function but with no luck. What is the right way to replace a regex like (N|nA|aN|n) with ‘nan’::double precision?
>>
>> thanks
>> Salvatore
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Hi,

I found my mistake. Placing the replace statement in the ruleset area the new property is well setup.

This way:

ruleset(name="RMM_energy_rules"){
set $.msg_with_nan_replaced = replace($msg,"nan”,”’nan’::double precision");
action(type="ompgsql" server=“localhost" user="rmmlog" pass=“xxxxxxxxxxx" db="rmmdb"
template="RMM_energy_value"
queue.size="10000" queue.type="linkedList"
queue.workerthreads="5"
queue.workerthreadMinimumMessages="1000"
queue.timeoutWorkerthreadShutdown="1000"
queue.timeoutEnqueue="10000" )
}

Now the problem is that postgres db receives:
INSERT INTO energyvalue (datetime, made_by, Nh_T1, Nh_T2, Ph_im_T1, Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2, Qh_ex_T1, Qh_ex_T2 ) values ('2020-12-18 10:18:40','18137709', 3.3247528000e+05,\'nan\'::double precision,3.1351968000e+05,\'nan\'::double precision,0.0000000000e+00,\'nan\'::double precision,1.9279240000e+04,\'nan\'::double precision,9.0233560000e+04,\'nan\'::double precision)

I tried ‘drop’, ’space’ and ‘escape’ in property(name="$.msg_with_nan_replaced" controlcharacters="drop”) but the result is the same.

However this issue is not so important because using null instead of nan the insert is committed. Seems that null is not a IEEE 754 complaint value.

thanks!
Salvatore


> On 17 Dec 2020, at 19:13, Rainer Gerhards <rgerhards@hq.adiscon.com> wrote:
>
> does
>
> set $.newmsg = replace(...);
>
> and using the $.newmsg variable inside the template work? On first
> look it should...
>
> HTH
> Rainer
>
> El jue, 17 dic 2020 a las 18:34, Salvatore Totaro via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
>>
>> Hi,
>>
>> I have a msg property made in this way:
>>
>> 1.5470373000e+05,nan,1.5060465000e+05,nan,2.1682000000e+02,nan,2.4761550000e+04,nan,1.6121420000e+04,nan
>>
>> In order to insert this row in my postgres db I use this template:
>>
>> template(name="energy_value" type="list" option.sql="on") {
>> constant(value="INSERT INTO energyvalue (datetime, made_by, Nh_T1, Nh_T2, Ph_im_T1, Ph_im_T2, Ph_ex_T1, Ph_ex_T2, Qh_im_T1, Qh_im_T2, Qh_ex_T1, Qh_ex_T2 ) values ('")
>> property(name="timereported" dateformat="pgsql" date.inUTC="on")
>> constant(value="','")
>> property(name="hostname")
>> constant(value="',")
>> property(name="msg")
>> constant(value=")")
>> }
>>
>> but postgresql has problem with nan value. The solution is to replace nan token with ‘nan’::double precision
>>
>> I tried to use replace function but with no luck. What is the right way to replace a regex like (N|nA|aN|n) with ‘nan’::double precision?
>>
>> thanks
>> Salvatore
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.