Mailing List Archive: OUT_BYTES value always zero in flow records when I use --disable-cache

OUT_BYTES value always zero in flow records when I use --disable-cache

Jan 4, 2018, 11:33 AM

Post #1 of 4 (683 views)

I am running nprobe as a collector, received flows from a few routers and
forwarding to Kafka. I was doing some ingest performance testing and wanted
to increase the rate of flow records/sec so I added --disable-cache to my
configuration. This results in nearly 10X more records per second output
from nprobe into Kafka which I wanted for testing. However, while testing
throughput, I noticed that the OUT_BYTES field in the JSON records
published to Kafka where always zero values now. If I remove
--disable-cache, the OUT_BYTES values return to as expected (non-zero
values)? Seems quite strange. Can anyone explain what is happening here?

Welcome to nProbe v.8.2.171214 (r5982) for x86_64-unknown-linux-gnu
with native PF_RING acceleration.
Copyright 2002-17 ntop.org

Build OS: CentOS Linux release 7.4.1708 (Core)
SystemID: 68A2B43E76056A7E
GIT rev: 8.2-stable:81448aa287d3968208054e69cdef2a10aa91758a:20171214

--disable-cache << Adding this config makes OUT_BYTES become zero for all
flow records
--interface=none
--collector=none
--collector-port=2055
--verbose=1
--lifetime-timeout=120
--idle-timeout=30
--queue-timeout=30
--flow-version=9
--hash-size=256000
--kafka-add-timestamp
--kafka="kafka01:9092;netflow-raw;1"
-T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV4_SRC_MASK
%IPV4_DST_MASK %IPV4_NEXT_HOP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES
%FIRST_SWITCHED %LAST_SWITCHED %TCP_FLAGS %PROTOCOL %SRC_TOS %DIRECTION
%EXPORTER_IPV4_ADDRESS %L4_SRC_PORT_MAP %L4_DST_PORT_MAP %INPUT_SNMP
%OUTPUT_SNMP %FLOW_ID"

Re: OUT_BYTES value always zero in flow records when I use --disable-cache [ In reply to ]

mainardi at ntop

Jan 4, 2018, 2:27 PM

Post #2 of 4 (683 views)

Permalink

Your NetFlow exporter outputs each bi-directional flow as two separate data records -- one for the client-to-server direction and the other for the server-to-client direction. Once the direction is given, one counter for the bytes (i.e., IN_BYTES) is necessary.

By default, nProbe automatically figures out and merges the two directions of collected flows, turning the server-to-client IN_BYTES into the flow OUT_BYTES.

If you --disable-cache, you will prevent nProbe from doing any merge. In this case, nProbe will act as a transparent proxy and will immediately output each data record as-is, leaving the OUT_BYTES at zero.
Contrarily, if you don't --disable-cache, nProbe will automatically merge the two data records into the same flow and then will only output one flow with non-zero OUT_BYTES and IN_BYTES.

Simone

> On 4 Jan 2018, at 20:33, Mark Petronic <markpetronic@gmail.com> wrote:
>
> I am running nprobe as a collector, received flows from a few routers and forwarding to Kafka. I was doing some ingest performance testing and wanted to increase the rate of flow records/sec so I added --disable-cache to my configuration. This results in nearly 10X more records per second output from nprobe into Kafka which I wanted for testing. However, while testing throughput, I noticed that the OUT_BYTES field in the JSON records published to Kafka where always zero values now. If I remove --disable-cache, the OUT_BYTES values return to as expected (non-zero values)? Seems quite strange. Can anyone explain what is happening here?
>
>
> Welcome to nProbe v.8.2.171214 (r5982) for x86_64-unknown-linux-gnu
> with native PF_RING acceleration.
> Copyright 2002-17 ntop.org <http://ntop.org/>
>
> Build OS: CentOS Linux release 7.4.1708 (Core)
> SystemID: 68A2B43E76056A7E
> GIT rev: 8.2-stable:81448aa287d3968208054e69cdef2a10aa91758a:20171214
>
>
> --disable-cache << Adding this config makes OUT_BYTES become zero for all flow records
> --interface=none
> --collector=none
> --collector-port=2055
> --verbose=1
> --lifetime-timeout=120
> --idle-timeout=30
> --queue-timeout=30
> --flow-version=9
> --hash-size=256000
> --kafka-add-timestamp
> --kafka="kafka01:9092;netflow-raw;1"
> -T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV4_SRC_MASK %IPV4_DST_MASK %IPV4_NEXT_HOP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %TCP_FLAGS %PROTOCOL %SRC_TOS %DIRECTION %EXPORTER_IPV4_ADDRESS %L4_SRC_PORT_MAP %L4_DST_PORT_MAP %INPUT_SNMP %OUTPUT_SNMP %FLOW_ID"
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: OUT_BYTES value always zero in flow records when I use --disable-cache [ In reply to ]

markpetronic at gmail

Jan 5, 2018, 5:54 AM

Post #3 of 4 (682 views)

Permalink

Thank you, Simone, for the quick response. This very much clarifies it for
me.

I have to say, as a new user of nprobe and noob to netflow, I am very
impressed with the timeliness and level of responses I get from this forum.
Very appreciated and helpful in me ramping up.

As a follow up related to this answer, how does nprobe internally treat UDP
data in aggregations? For example, a UDP DNS request and corresponding
response. The 4-tuples would match up regarding src and dst ips and ports
(at least for the single request and response for one DNS request) but this
is a connectionless protocol. Does nprobe attempt to "finesse" these types
of paired packets into a common aggregation bin if the arrive within the
window of idle timeout or simply only treat TCP connections as ones that
can be aggregated? In my world, many of the tenant networks that we will be
monitoring have a lot of small TCP transactions and DNS UDP dominates much
of the flow volume - at least as a percentage of flow records (not
necessarily byte volume).

Thanks... Mark

On Thu, Jan 4, 2018 at 5:27 PM, Simone Mainardi <mainardi@ntop.org> wrote:

> Your NetFlow exporter outputs each bi-directional flow as two separate
> data records -- one for the client-to-server direction and the other for
> the server-to-client direction. Once the direction is given, one counter
> for the bytes (i.e., IN_BYTES) is necessary.
>
> By default, nProbe automatically figures out and merges the two directions
> of collected flows, turning the server-to-client IN_BYTES into the flow
> OUT_BYTES.
>
> If you --disable-cache, you will prevent nProbe from doing any merge. In
> this case, nProbe will act as a transparent proxy and will immediately
> output each data record as-is, leaving the OUT_BYTES at zero.
> Contrarily, if you don't --disable-cache, nProbe will automatically merge
> the two data records into the same flow and then will only output one flow
> with non-zero OUT_BYTES and IN_BYTES.
>
>
> Simone
>
> On 4 Jan 2018, at 20:33, Mark Petronic <markpetronic@gmail.com> wrote:
>
> I am running nprobe as a collector, received flows from a few routers and
> forwarding to Kafka. I was doing some ingest performance testing and wanted
> to increase the rate of flow records/sec so I added --disable-cache to my
> configuration. This results in nearly 10X more records per second output
> from nprobe into Kafka which I wanted for testing. However, while testing
> throughput, I noticed that the OUT_BYTES field in the JSON records
> published to Kafka where always zero values now. If I remove
> --disable-cache, the OUT_BYTES values return to as expected (non-zero
> values)? Seems quite strange. Can anyone explain what is happening here?
>
>
> Welcome to nProbe v.8.2.171214 (r5982) for x86_64-unknown-linux-gnu
> with native PF_RING acceleration.
> Copyright 2002-17 ntop.org
>
> Build OS: CentOS Linux release 7.4.1708 (Core)
> SystemID: 68A2B43E76056A7E
> GIT rev: 8.2-stable:81448aa287d3968208054e69cdef2a
> 10aa91758a:20171214
>
>
> --disable-cache << Adding this config makes OUT_BYTES become zero for all
> flow records
> --interface=none
> --collector=none
> --collector-port=2055
> --verbose=1
> --lifetime-timeout=120
> --idle-timeout=30
> --queue-timeout=30
> --flow-version=9
> --hash-size=256000
> --kafka-add-timestamp
> --kafka="kafka01:9092;netflow-raw;1"
> -T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV4_SRC_MASK
> %IPV4_DST_MASK %IPV4_NEXT_HOP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES
> %FIRST_SWITCHED %LAST_SWITCHED %TCP_FLAGS %PROTOCOL %SRC_TOS %DIRECTION
> %EXPORTER_IPV4_ADDRESS %L4_SRC_PORT_MAP %L4_DST_PORT_MAP %INPUT_SNMP
> %OUTPUT_SNMP %FLOW_ID"
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>

Re: OUT_BYTES value always zero in flow records when I use --disable-cache [ In reply to ]

mainardi at ntop

Jan 5, 2018, 6:42 AM

Post #4 of 4 (682 views)

Permalink

Dear Mark,

Also the layer-4 protocol is used when doing aggregations. As UDP is connectionless, there's no way for nProbe to determine the flow has terminated (there's no FIN packet) apart from using the idle timeout.

Simone

> On 5 Jan 2018, at 14:54, Mark Petronic <markpetronic@gmail.com> wrote:
>
> Thank you, Simone, for the quick response. This very much clarifies it for me.
>
> I have to say, as a new user of nprobe and noob to netflow, I am very impressed with the timeliness and level of responses I get from this forum. Very appreciated and helpful in me ramping up.
>
> As a follow up related to this answer, how does nprobe internally treat UDP data in aggregations? For example, a UDP DNS request and corresponding response. The 4-tuples would match up regarding src and dst ips and ports (at least for the single request and response for one DNS request) but this is a connectionless protocol. Does nprobe attempt to "finesse" these types of paired packets into a common aggregation bin if the arrive within the window of idle timeout or simply only treat TCP connections as ones that can be aggregated? In my world, many of the tenant networks that we will be monitoring have a lot of small TCP transactions and DNS UDP dominates much of the flow volume - at least as a percentage of flow records (not necessarily byte volume).
>
> Thanks... Mark
>
> On Thu, Jan 4, 2018 at 5:27 PM, Simone Mainardi <mainardi@ntop.org <mailto:mainardi@ntop.org>> wrote:
> Your NetFlow exporter outputs each bi-directional flow as two separate data records -- one for the client-to-server direction and the other for the server-to-client direction. Once the direction is given, one counter for the bytes (i.e., IN_BYTES) is necessary.
>
> By default, nProbe automatically figures out and merges the two directions of collected flows, turning the server-to-client IN_BYTES into the flow OUT_BYTES.
>
> If you --disable-cache, you will prevent nProbe from doing any merge. In this case, nProbe will act as a transparent proxy and will immediately output each data record as-is, leaving the OUT_BYTES at zero.
> Contrarily, if you don't --disable-cache, nProbe will automatically merge the two data records into the same flow and then will only output one flow with non-zero OUT_BYTES and IN_BYTES.
>
>
> Simone
>
>> On 4 Jan 2018, at 20:33, Mark Petronic <markpetronic@gmail.com <mailto:markpetronic@gmail.com>> wrote:
>>
>> I am running nprobe as a collector, received flows from a few routers and forwarding to Kafka. I was doing some ingest performance testing and wanted to increase the rate of flow records/sec so I added --disable-cache to my configuration. This results in nearly 10X more records per second output from nprobe into Kafka which I wanted for testing. However, while testing throughput, I noticed that the OUT_BYTES field in the JSON records published to Kafka where always zero values now. If I remove --disable-cache, the OUT_BYTES values return to as expected (non-zero values)? Seems quite strange. Can anyone explain what is happening here?
>>
>>
>> Welcome to nProbe v.8.2.171214 (r5982) for x86_64-unknown-linux-gnu
>> with native PF_RING acceleration.
>> Copyright 2002-17 ntop.org <http://ntop.org/>
>>
>> Build OS: CentOS Linux release 7.4.1708 (Core)
>> SystemID: 68A2B43E76056A7E
>> GIT rev: 8.2-stable:81448aa287d3968208054e69cdef2a10aa91758a:20171214
>>
>>
>> --disable-cache << Adding this config makes OUT_BYTES become zero for all flow records
>> --interface=none
>> --collector=none
>> --collector-port=2055
>> --verbose=1
>> --lifetime-timeout=120
>> --idle-timeout=30
>> --queue-timeout=30
>> --flow-version=9
>> --hash-size=256000
>> --kafka-add-timestamp
>> --kafka="kafka01:9092;netflow-raw;1"
>> -T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV4_SRC_MASK %IPV4_DST_MASK %IPV4_NEXT_HOP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %TCP_FLAGS %PROTOCOL %SRC_TOS %DIRECTION %EXPORTER_IPV4_ADDRESS %L4_SRC_PORT_MAP %L4_DST_PORT_MAP %INPUT_SNMP %OUTPUT_SNMP %FLOW_ID"
>>
>> _______________________________________________
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc