We are using nprobe to write to elastic search various http request we
monitor within the network.
>From time to time we see that some of the http request that we monitor is
not written into elastic search, we do see this in the flows file the
nprobe generate. When looking in the elastic search log file we can see the
following errors dealing with invalid char either in the http_url or in the
http_ua.
Has someone have seen this problem or have an idea on how to overcome this
issue?
[2016-03-23 07:46:40,362][DEBUG][action.bulk ] [Poltergeist]
[nprobe127-2016.03.23][0] failed to execute bulk item (index) index
{[nprobe127-2016.03.23][nProbe][AVOicJRnGkpqroZghzZT],
source[{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"184.87.179.64","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"20:E5:2A:0F:89:FC","L4_SRC_PORT":51090,"L4_DST_PORT":80,"IN_BYTES":52,"OUT_BYTES":0,"IN_PKTS":1,"OUT_PKTS":0,"FIRST_SWITCHED":1458719137,"LAST_SWITCHED":1458719137,"L7_PROTO_NAME":"Unknown","PROTOCOL":6,"HTTP_URL":"
�[�","HTTP_RET_CODE":0,"HTTP_REFERER":"","HTTP_UA":"","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"NL","DST_IP_CITY":"Amsterdam","@version":"1","@timestamp":"2016-03-23T07:45:37Z",
"EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}
MapperParsingException[failed to parse [HTTP_URL]]; nested:
JsonParseException[.Illegal unquoted character ((CTRL-CHAR, code 14)): has
to be escaped using backslash to be included in string value
at [Source:
org.elasticsearch.common.io.stream.InputStreamStreamInput@34879213; line:
1, column: 327]];
[2016-03-23 07:54:30,195][DEBUG][action.bulk ] [Poltergeist]
[nprobe127-2016.03.23][0] failed to execute bulk item (index) index
{[nprobe127-2016.03.23][nProbe][AVOid7-zGkpqroZghzj4],
source[.{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"10.0.45.2","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"00:13:23:04:41:0F","L4_SRC_PORT":51140,"L4_DST_PORT":80,"IN_BYTES":470,"OUT_BYTES":7350,"IN_PKTS":3,"OUT_PKTS":5,"FIRST_SWITCHED":1458719669,"LAST_SWITCHED":1458719669,"L7_PROTO_NAME":"HTTP","PROTOCOL":6,"HTTP_URL":"
10.0.45.2/topmenu.js�","HTTP_RET_CODE":200,"HTTP_REFERER":"
10.0.45.2/viewer/avstream_vca.shtml?streamid=first&inch=1","HTTP_UA":"Mozilla/5.0
(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/49.0.2623.75
Safari/537.36�","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"","DST_IP_CITY":"","@version":"1","@timestamp":"2016-03-23T07:54:29Z",
"EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}
MapperParsingException[failed to parse [HTTP_URL]]; nested:
JsonParseException[Invalid UTF-8 middle byte 0x7f
Thanks
Ohad
monitor within the network.
>From time to time we see that some of the http request that we monitor is
not written into elastic search, we do see this in the flows file the
nprobe generate. When looking in the elastic search log file we can see the
following errors dealing with invalid char either in the http_url or in the
http_ua.
Has someone have seen this problem or have an idea on how to overcome this
issue?
[2016-03-23 07:46:40,362][DEBUG][action.bulk ] [Poltergeist]
[nprobe127-2016.03.23][0] failed to execute bulk item (index) index
{[nprobe127-2016.03.23][nProbe][AVOicJRnGkpqroZghzZT],
source[{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"184.87.179.64","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"20:E5:2A:0F:89:FC","L4_SRC_PORT":51090,"L4_DST_PORT":80,"IN_BYTES":52,"OUT_BYTES":0,"IN_PKTS":1,"OUT_PKTS":0,"FIRST_SWITCHED":1458719137,"LAST_SWITCHED":1458719137,"L7_PROTO_NAME":"Unknown","PROTOCOL":6,"HTTP_URL":"
�[�","HTTP_RET_CODE":0,"HTTP_REFERER":"","HTTP_UA":"","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"NL","DST_IP_CITY":"Amsterdam","@version":"1","@timestamp":"2016-03-23T07:45:37Z",
"EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}
MapperParsingException[failed to parse [HTTP_URL]]; nested:
JsonParseException[.Illegal unquoted character ((CTRL-CHAR, code 14)): has
to be escaped using backslash to be included in string value
at [Source:
org.elasticsearch.common.io.stream.InputStreamStreamInput@34879213; line:
1, column: 327]];
[2016-03-23 07:54:30,195][DEBUG][action.bulk ] [Poltergeist]
[nprobe127-2016.03.23][0] failed to execute bulk item (index) index
{[nprobe127-2016.03.23][nProbe][AVOid7-zGkpqroZghzj4],
source[.{"IPV4_SRC_ADDR":"10.0.97.2","IPV4_DST_ADDR":"10.0.45.2","IN_SRC_MAC":"44:37:E6:EF:6B:27","OUT_DST_MAC":"00:13:23:04:41:0F","L4_SRC_PORT":51140,"L4_DST_PORT":80,"IN_BYTES":470,"OUT_BYTES":7350,"IN_PKTS":3,"OUT_PKTS":5,"FIRST_SWITCHED":1458719669,"LAST_SWITCHED":1458719669,"L7_PROTO_NAME":"HTTP","PROTOCOL":6,"HTTP_URL":"
10.0.45.2/topmenu.js�","HTTP_RET_CODE":200,"HTTP_REFERER":"
10.0.45.2/viewer/avstream_vca.shtml?streamid=first&inch=1","HTTP_UA":"Mozilla/5.0
(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/49.0.2623.75
Safari/537.36�","SRC_IP_COUNTRY":"","SRC_IP_CITY":"","DST_IP_COUNTRY":"","DST_IP_CITY":"","@version":"1","@timestamp":"2016-03-23T07:54:29Z",
"EXPORTER_IPV4_ADDRESS":"127.0.0.1"}]}
MapperParsingException[failed to parse [HTTP_URL]]; nested:
JsonParseException[Invalid UTF-8 middle byte 0x7f
Thanks
Ohad