Mailing List Archive: nprobe fails writing to elastic search

nprobe fails writing to elastic search

Mar 23, 2016, 1:10 AM

Post #1 of 4 (1244 views)

We are using nprobe to write to elastic search various http request we
monitor within the network.

>From time to time we see that some of the http request that we monitor is
not written into elastic search, we do see this in the flows file the
nprobe generate. When looking in the elastic search log file we can see the
following errors dealing with invalid char either in the http_url or in the
http_ua.

Has someone have seen this problem or have an idea on how to overcome this
issue?

[2016-03-23 07:46:40,362][DEBUG][action.bulk ] [Poltergeist]
[nprobe127-2016.03.23][0] failed to execute bulk item (index) index
{[nprobe127-2016.03.23][nProbe][AVOicJRnGkpqroZghzZT],
source[{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"184.87.179.64","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"20:E5:2A:0F:89:FC","L4_SRC_PORT":51090,"L4_DST_PORT":80,"IN_BYTES":52,"OUT_BYTES":0,"IN_PKTS":1,"OUT_PKTS":0,"FIRST_SWITCHED":1458719137,"LAST_SWITCHED":1458719137,"L7_PROTO_NAME":"Unknown","PROTOCOL":6,"HTTP_URL":"

ï¿½[ï¿½","HTTP_RET_CODE":0,"HTTP_REFERER":"","HTTP_UA":"","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"NL","DST_IP_CITY":"Amsterdam","@version":"1","@timestamp":"2016-03-23T07:45:37Z",
"EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}

MapperParsingException[failed to parse [HTTP_URL]]; nested:
JsonParseException[.Illegal unquoted character ((CTRL-CHAR, code 14)): has
to be escaped using backslash to be included in string value

at [Source:
org.elasticsearch.common.io.stream.InputStreamStreamInput@34879213; line:
1, column: 327]];

[2016-03-23 07:54:30,195][DEBUG][action.bulk ] [Poltergeist]
[nprobe127-2016.03.23][0] failed to execute bulk item (index) index
{[nprobe127-2016.03.23][nProbe][AVOid7-zGkpqroZghzj4],
source[.{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"10.0.45.2","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"00:13:23:04:41:0F","L4_SRC_PORT":51140,"L4_DST_PORT":80,"IN_BYTES":470,"OUT_BYTES":7350,"IN_PKTS":3,"OUT_PKTS":5,"FIRST_SWITCHED":1458719669,"LAST_SWITCHED":1458719669,"L7_PROTO_NAME":"HTTP","PROTOCOL":6,"HTTP_URL":"
10.0.45.2/topmenu.jsï¿½","HTTP_RET_CODE":200,"HTTP_REFERER":"
10.0.45.2/viewer/avstream_vca.shtml?streamid=first&inch=1","HTTP_UA":"Mozilla/5.0
(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/49.0.2623.75
Safari/537.36ï¿½","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"","DST_IP_CITY":"","@version":"1","@timestamp":"2016-03-23T07:54:29Z",
"EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}

MapperParsingException[failed to parse [HTTP_URL]]; nested:
JsonParseException[Invalid UTF-8 middle byte 0x7f

Thanks

Ohad

Re: nprobe fails writing to elastic search [ In reply to ]

deri at ntop

Mar 23, 2016, 1:30 AM

Post #2 of 4 (1224 views)

Permalink

Ohad
have you tried the latest build: it should fix this issue

Luca

On 03/23/2016 09:10 AM, Ohad Kleinman wrote:
>
> We are using nprobe to write to elastic search various http request we
> monitor within the network.
>
> From time to time we see that some of the http request that we monitor
> is not written into elastic search, we do see this in the flows file
> the nprobe generate. When looking in the elastic search log file we
> can see the following errors dealing with invalid char either in the
> http_url or in the http_ua.
>
> Has someone have seen this problem or have an idea on how to overcome
> this issue?
>
>
>
> [2016-03-23 07:46:40,362][DEBUG][action.bulk ]
> [Poltergeist] [nprobe127-2016.03.23][0] failed to execute bulk item
> (index) index {[nprobe127-2016.03.23][nProbe][AVOicJRnGkpqroZghzZT],
> source[{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"184.87.179.64","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"20:E5:2A:0F:89:FC","L4_SRC_PORT":51090,"L4_DST_PORT":80,"IN_BYTES":52,"OUT_BYTES":0,"IN_PKTS":1,"OUT_PKTS":0,"FIRST_SWITCHED":1458719137,"LAST_SWITCHED":1458719137,"L7_PROTO_NAME":"Unknown","PROTOCOL":6,"HTTP_URL":"
>
> ï¿½[ï¿½","HTTP_RET_CODE":0,"HTTP_REFERER":"","HTTP_UA":"","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"NL","DST_IP_CITY":"Amsterdam","@version":"1","@timestamp":"2016-03-23T07:45:37Z",
> "EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}
>
> MapperParsingException[failed to parse [HTTP_URL]]; nested:
> JsonParseException[.Illegal unquoted character ((CTRL-CHAR, code 14)):
> has to be escaped using backslash to be included in string value
>
> at [Source:
> org.elasticsearch.common.io.stream.InputStreamStreamInput@34879213;
> line: 1, column: 327]];
>
>
>
> [2016-03-23 07:54:30,195][DEBUG][action.bulk ]
> [Poltergeist] [nprobe127-2016.03.23][0] failed to execute bulk item
> (index) index {[nprobe127-2016.03.23][nProbe][AVOid7-zGkpqroZghzj4],
> source[{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"10.0.45.2","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"00:13:23:04:41:0F","L4_SRC_PORT":51140,"L4_DST_PORT":80,"IN_BYTES":470,"OUT_BYTES":7350,"IN_PKTS":3,"OUT_PKTS":5,"FIRST_SWITCHED":1458719669,"LAST_SWITCHED":1458719669,"L7_PROTO_NAME":"HTTP","PROTOCOL":6,"HTTP_URL":"10.0.45.2/topmenu.js
> <http://10.0.45.2/topmenu.js>ï¿½","HTTP_RET_CODE":200,"HTTP_REFERER":"10.0.45.2/viewer/avstream_vca.shtml?streamid=first&inch=1
> <http://10.0.45.2/viewer/avstream_vca.shtml?streamid=first&inch=1>","HTTP_UA":"Mozilla/5.0
> (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
> Chrome/49.0.2623.75
> Safari/537.36ï¿½","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"","DST_IP_CITY":"","@version":"1","@timestamp":"2016-03-23T07:54:29Z",
> "EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}
>
> MapperParsingException[failed to parse [HTTP_URL]]; nested:
> JsonParseException[Invalid UTF-8 middle byte 0x7f
>
>
>
> Thanks
>
>
>
> Ohad
>
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: nprobe fails writing to elastic search [ In reply to ]

ohadk at vglnt

Mar 23, 2016, 1:45 AM

Post #3 of 4 (1222 views)

Permalink

Luca,

Will update with latest build and confirm if it was resolved.

Ohad

*From:* ntop-misc-bounces@listgateway.unipi.it [mailto:
ntop-misc-bounces@listgateway.unipi.it] *On Behalf Of *Luca Deri
*Sent:* Wednesday, March 23, 2016 10:30 AM
*To:* ntop-misc@listgateway.unipi.it
*Subject:* Re: [Ntop-misc] nprobe fails writing to elastic search

Ohad
have you tried the latest build: it should fix this issue

Luca

On 03/23/2016 09:10 AM, Ohad Kleinman wrote:

We are using nprobe to write to elastic search various http request we
monitor within the network.

>From time to time we see that some of the http request that we monitor is
not written into elastic search, we do see this in the flows file the
nprobe generate. When looking in the elastic search log file we can see the
following errors dealing with invalid char either in the http_url or in the
http_ua.

Has someone have seen this problem or have an idea on how to overcome this
issue?

[2016-03-23 07:46:40,362][DEBUG][action.bulk ] [Poltergeist]
[nprobe127-2016.03.23][0] failed to execute bulk item (index) index
{[nprobe127-2016.03.23][nProbe][AVOicJRnGkpqroZghzZT],
source[{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"184.87.179.64","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"20:E5:2A:0F:89:FC","L4_SRC_PORT":51090,"L4_DST_PORT":80,"IN_BYTES":52,"OUT_BYTES":0,"IN_PKTS":1,"OUT_PKTS":0,"FIRST_SWITCHED":1458719137,"LAST_SWITCHED":1458719137,"L7_PROTO_NAME":"Unknown","PROTOCOL":6,"HTTP_URL":"

ï¿½[ï¿½","HTTP_RET_CODE":0,"HTTP_REFERER":"","HTTP_UA":"","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"NL","DST_IP_CITY":"Amsterdam","@version":"1","@timestamp":"2016-03-23T07:45:37Z",
"EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}

MapperParsingException[failed to parse [HTTP_URL]]; nested:
JsonParseException[.Illegal unquoted character ((CTRL-CHAR, code 14)): has
to be escaped using backslash to be included in string value

at [Source:
org.elasticsearch.common.io.stream.InputStreamStreamInput@34879213; line:
1, column: 327]];

[2016-03-23 07:54:30,195][DEBUG][action.bulk ] [Poltergeist]
[nprobe127-2016.03.23][0] failed to execute bulk item (index) index
{[nprobe127-2016.03.23][nProbe][AVOid7-zGkpqroZghzj4],
source[.{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"10.0.45.2","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"00:13:23:04:41:0F","L4_SRC_PORT":51140,"L4_DST_PORT":80,"IN_BYTES":470,"OUT_BYTES":7350,"IN_PKTS":3,"OUT_PKTS":5,"FIRST_SWITCHED":1458719669,"LAST_SWITCHED":1458719669,"L7_PROTO_NAME":"HTTP","PROTOCOL":6,"HTTP_URL":"
10.0.45.2/topmenu.jsï¿½","HTTP_RET_CODE":200,"HTTP_REFERER":"
10.0.45.2/viewer/avstream_vca.shtml?streamid=first&inch=1","HTTP_UA":"Mozilla/5.0
(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/49.0.2623.75
Safari/537.36ï¿½","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"","DST_IP_CITY":"","@version":"1","@timestamp":"2016-03-23T07:54:29Z",
"EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}

MapperParsingException[failed to parse [HTTP_URL]]; nested:
JsonParseException[Invalid UTF-8 middle byte 0x7f

Thanks

Ohad

_______________________________________________

Ntop-misc mailing list

Ntop-misc@listgateway.unipi.it

http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: nprobe fails writing to elastic search [ In reply to ]

ohadk at vglnt

Mar 23, 2016, 2:01 AM

Post #4 of 4 (1224 views)

Permalink

Luca,

After the update to latest version problem resolved.

Thank you.

Ohad

*From:* ntop-misc-bounces@listgateway.unipi.it [mailto:
ntop-misc-bounces@listgateway.unipi.it] *On Behalf Of *Luca Deri
*Sent:* Wednesday, March 23, 2016 10:30 AM
*To:* ntop-misc@listgateway.unipi.it
*Subject:* Re: [Ntop-misc] nprobe fails writing to elastic search

Ohad
have you tried the latest build: it should fix this issue

Luca

On 03/23/2016 09:10 AM, Ohad Kleinman wrote:

We are using nprobe to write to elastic search various http request we
monitor within the network.

>From time to time we see that some of the http request that we monitor is
not written into elastic search, we do see this in the flows file the
nprobe generate. When looking in the elastic search log file we can see the
following errors dealing with invalid char either in the http_url or in the
http_ua.

Has someone have seen this problem or have an idea on how to overcome this
issue?

[2016-03-23 07:46:40,362][DEBUG][action.bulk ] [Poltergeist]
[nprobe127-2016.03.23][0] failed to execute bulk item (index) index
{[nprobe127-2016.03.23][nProbe][AVOicJRnGkpqroZghzZT],
source[{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"184.87.179.64","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"20:E5:2A:0F:89:FC","L4_SRC_PORT":51090,"L4_DST_PORT":80,"IN_BYTES":52,"OUT_BYTES":0,"IN_PKTS":1,"OUT_PKTS":0,"FIRST_SWITCHED":1458719137,"LAST_SWITCHED":1458719137,"L7_PROTO_NAME":"Unknown","PROTOCOL":6,"HTTP_URL":"

ï¿½[ï¿½","HTTP_RET_CODE":0,"HTTP_REFERER":"","HTTP_UA":"","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"NL","DST_IP_CITY":"Amsterdam","@version":"1","@timestamp":"2016-03-23T07:45:37Z",
"EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}

MapperParsingException[failed to parse [HTTP_URL]]; nested:
JsonParseException[.Illegal unquoted character ((CTRL-CHAR, code 14)): has
to be escaped using backslash to be included in string value

at [Source:
org.elasticsearch.common.io.stream.InputStreamStreamInput@34879213; line:
1, column: 327]];

[2016-03-23 07:54:30,195][DEBUG][action.bulk ] [Poltergeist]
[nprobe127-2016.03.23][0] failed to execute bulk item (index) index
{[nprobe127-2016.03.23][nProbe][AVOid7-zGkpqroZghzj4],
source[.{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"10.0.45.2","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"00:13:23:04:41:0F","L4_SRC_PORT":51140,"L4_DST_PORT":80,"IN_BYTES":470,"OUT_BYTES":7350,"IN_PKTS":3,"OUT_PKTS":5,"FIRST_SWITCHED":1458719669,"LAST_SWITCHED":1458719669,"L7_PROTO_NAME":"HTTP","PROTOCOL":6,"HTTP_URL":"
10.0.45.2/topmenu.jsï¿½","HTTP_RET_CODE":200,"HTTP_REFERER":"
10.0.45.2/viewer/avstream_vca.shtml?streamid=first&inch=1","HTTP_UA":"Mozilla/5.0
(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/49.0.2623.75
Safari/537.36ï¿½","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"","DST_IP_CITY":"","@version":"1","@timestamp":"2016-03-23T07:54:29Z",
"EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}

MapperParsingException[failed to parse [HTTP_URL]]; nested:
JsonParseException[Invalid UTF-8 middle byte 0x7f

Thanks

Ohad

_______________________________________________

Ntop-misc mailing list

Ntop-misc@listgateway.unipi.it

http://listgateway.unipi.it/mailman/listinfo/ntop-misc