Mailing List Archive

Duplicate flow entries
Hi everyone,

Starting with ntopng, I have a small issue initially setting it up.

I use port mirroring on a switch to replicate all ports to port 5 where a
dedicated ntopng interface 'listens' (Official package on raspbian 10).
On that same switch I have my Internet gateway (Unifi USG3P) connected to
port 1. This same device also acts as a DHCP/DNS server.

When mirroring all ports BUT port 1, I receive alerts about thousands of
DNS queries not being answered. I did confirm that with a pcap dump.

So I went and started to mirror port 1 along with others, and the missing
traffic (DNS replies) started to be collected.
The issue is that with that configuration, all flows are listed twice in
ntop. Internal hosts are showing normally and with "@1" at the end of the
hostname.

Is there a way for ntop to discard this duplicated traffic in the
accounting of ntopng? It makes sense to me that it is detected as a host's
traffic will be seen on its own switch port and then in many cases on port
1.

Many thanks.
Re: Duplicate flow entries [ In reply to ]
Hi,

> On 21 May 2020, at 14:55, David van Ginneken <david@van-ginneken.org> wrote:
>
> Hi everyone,
>
> Starting with ntopng, I have a small issue initially setting it up.
>
> I use port mirroring on a switch to replicate all ports to port 5 where a dedicated ntopng interface 'listens' (Official package on raspbian 10).
> On that same switch I have my Internet gateway (Unifi USG3P) connected to port 1. This same device also acts as a DHCP/DNS server.
>
> When mirroring all ports BUT port 1, I receive alerts about thousands of DNS queries not being answered. I did confirm that with a pcap dump.

When you monitor just port 1, apart from the DNS queries unanswered alerts, do you get bi-directional traffic if you look at the flows page? Do you see the @1?

>
> So I went and started to mirror port 1 along with others, and the missing traffic (DNS replies) started to be collected.
> The issue is that with that configuration, all flows are listed twice in ntop. Internal hosts are showing normally and with "@1" at the end of the hostname.

@1 means VLAN=1 so VLAN-tagged packets are received from the mirror port. VLAN depend on your switch configuration. If you can disregard VLANs you can use option --ignore-vlans

>
> Is there a way for ntop to discard this duplicated traffic in the accounting of ntopng?

I am not sure the traffic is duplicated. It could be that ntopng is keeping the two directions of every flow separated due to the VLAN. Let's continue the investigation depending on your responses.

Simone

> It makes sense to me that it is detected as a host's traffic will be seen on its own switch port and then in many cases on port 1.
>
> Many thanks.
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop

_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Duplicate flow entries [ In reply to ]
Hi,

> On 21 May 2020, at 14:55, David van Ginneken <david@van-ginneken.org> wrote:
>
> Hi everyone,
>
> Starting with ntopng, I have a small issue initially setting it up.
>
> I use port mirroring on a switch to replicate all ports to port 5 where a dedicated ntopng interface 'listens' (Official package on raspbian 10).
> On that same switch I have my Internet gateway (Unifi USG3P) connected to port 1. This same device also acts as a DHCP/DNS server.
>
> When mirroring all ports BUT port 1, I receive alerts about thousands of DNS queries not being answered. I did confirm that with a pcap dump.

When you monitor just port 1, apart from the DNS queries unanswered alerts, do you get bi-directional traffic if you look at the flows page? Do you see the @1?

>
> So I went and started to mirror port 1 along with others, and the missing traffic (DNS replies) started to be collected.
> The issue is that with that configuration, all flows are listed twice in ntop. Internal hosts are showing normally and with "@1" at the end of the hostname.

@1 means VLAN=1 so VLAN-tagged packets are received from the mirror port. VLAN depend on your switch configuration. If you can disregard VLANs you can use option --ignore-vlans

>
> Is there a way for ntop to discard this duplicated traffic in the accounting of ntopng?

I am not sure the traffic is duplicated. It could be that ntopng is keeping the two directions of every flow separated due to the VLAN. Let's continue the investigation depending on your responses.

Simone

> It makes sense to me that it is detected as a host's traffic will be seen on its own switch port and then in many cases on port 1.
>
> Many thanks.
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop

_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Duplicate flow entries [ In reply to ]
Hi Simone,

Thanks for the advice. The --ignore-vlans option seems to help as I do not
see the duplicates anymore.
I do have vlans on my network but it is not a problem for me not to have
this separated in the display.

Now I still get odd alerts about HTTP requests not being answered. I'll
investigate a bit further but it seems VERY similar to the issue Aaron and
Emanuele are discussing in parallel.
And, on top of that, it seems Aaron is also using a Unifi device (not the
same model but I think the OSes are). Could this be a pointer to the root
cause of our issues?

Thanks again.


Le jeu. 21 mai 2020 à 23:18, Simone Mainardi <mainardi@ntop.org> a écrit :

> Hi,
>
> > On 21 May 2020, at 14:55, David van Ginneken <david@van-ginneken.org>
> wrote:
> >
> > Hi everyone,
> >
> > Starting with ntopng, I have a small issue initially setting it up.
> >
> > I use port mirroring on a switch to replicate all ports to port 5 where
> a dedicated ntopng interface 'listens' (Official package on raspbian 10).
> > On that same switch I have my Internet gateway (Unifi USG3P) connected
> to port 1. This same device also acts as a DHCP/DNS server.
> >
> > When mirroring all ports BUT port 1, I receive alerts about thousands of
> DNS queries not being answered. I did confirm that with a pcap dump.
>
> When you monitor just port 1, apart from the DNS queries unanswered
> alerts, do you get bi-directional traffic if you look at the flows page? Do
> you see the @1?
>

> >
> > So I went and started to mirror port 1 along with others, and the
> missing traffic (DNS replies) started to be collected.
> > The issue is that with that configuration, all flows are listed twice in
> ntop. Internal hosts are showing normally and with "@1" at the end of the
> hostname.
>
> @1 means VLAN=1 so VLAN-tagged packets are received from the mirror port.
> VLAN depend on your switch configuration. If you can disregard VLANs you
> can use option --ignore-vlans
>
> >
> > Is there a way for ntop to discard this duplicated traffic in the
> accounting of ntopng?
>
> I am not sure the traffic is duplicated. It could be that ntopng is
> keeping the two directions of every flow separated due to the VLAN. Let's
> continue the investigation depending on your responses.
>
> Simone
>
> > It makes sense to me that it is detected as a host's traffic will be
> seen on its own switch port and then in many cases on port 1.
> >
> > Many thanks.
> > _______________________________________________
> > Ntop mailing list
> > Ntop@listgateway.unipi.it
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
Re: Duplicate flow entries [ In reply to ]
Hi Simone,

Thanks for the advice. The --ignore-vlans option seems to help as I do not
see the duplicates anymore.
I do have vlans on my network but it is not a problem for me not to have
this separated in the display.

Now I still get odd alerts about HTTP requests not being answered. I'll
investigate a bit further but it seems VERY similar to the issue Aaron and
Emanuele are discussing in parallel.
And, on top of that, it seems Aaron is also using a Unifi device (not the
same model but I think the OSes are). Could this be a pointer to the root
cause of our issues?

Thanks again.


Le jeu. 21 mai 2020 à 23:18, Simone Mainardi <mainardi@ntop.org> a écrit :

> Hi,
>
> > On 21 May 2020, at 14:55, David van Ginneken <david@van-ginneken.org>
> wrote:
> >
> > Hi everyone,
> >
> > Starting with ntopng, I have a small issue initially setting it up.
> >
> > I use port mirroring on a switch to replicate all ports to port 5 where
> a dedicated ntopng interface 'listens' (Official package on raspbian 10).
> > On that same switch I have my Internet gateway (Unifi USG3P) connected
> to port 1. This same device also acts as a DHCP/DNS server.
> >
> > When mirroring all ports BUT port 1, I receive alerts about thousands of
> DNS queries not being answered. I did confirm that with a pcap dump.
>
> When you monitor just port 1, apart from the DNS queries unanswered
> alerts, do you get bi-directional traffic if you look at the flows page? Do
> you see the @1?
>

> >
> > So I went and started to mirror port 1 along with others, and the
> missing traffic (DNS replies) started to be collected.
> > The issue is that with that configuration, all flows are listed twice in
> ntop. Internal hosts are showing normally and with "@1" at the end of the
> hostname.
>
> @1 means VLAN=1 so VLAN-tagged packets are received from the mirror port.
> VLAN depend on your switch configuration. If you can disregard VLANs you
> can use option --ignore-vlans
>
> >
> > Is there a way for ntop to discard this duplicated traffic in the
> accounting of ntopng?
>
> I am not sure the traffic is duplicated. It could be that ntopng is
> keeping the two directions of every flow separated due to the VLAN. Let's
> continue the investigation depending on your responses.
>
> Simone
>
> > It makes sense to me that it is detected as a host's traffic will be
> seen on its own switch port and then in many cases on port 1.
> >
> > Many thanks.
> > _______________________________________________
> > Ntop mailing list
> > Ntop@listgateway.unipi.it
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
Re: Duplicate flow entries [ In reply to ]
Hi David,

This should be the same issue experienced by Aaron due to frame padding.
Please check out the other thread.

Regards,

Emanuele

On 5/24/20 7:54 PM, David van Ginneken wrote:
> Hi Simone,
>
> Thanks for the advice. The --ignore-vlans option seems to help as I do
> not see the duplicates anymore.
> I do have vlans on my network but it is not a problem for me not to
> have this separated in the display.
>
> Now I still get odd alerts about HTTP requests not being answered.
> I'll investigate a bit further but it seems VERY similar to the issue
> Aaron and Emanuele are discussing in parallel.
> And, on top of that, it seems Aaron is also using a Unifi device (not
> the same model but I think the OSes are). Could this be a pointer to
> the root cause of our issues?
>
> Thanks again.
>
>
> Le jeu. 21 mai 2020 à 23:18, Simone Mainardi <mainardi@ntop.org
> <mailto:mainardi@ntop.org>> a écrit :
>
> Hi,
>
> > On 21 May 2020, at 14:55, David van Ginneken
> <david@van-ginneken.org <mailto:david@van-ginneken.org>> wrote:
> >
> > Hi everyone,
> >
> > Starting with ntopng, I have a small issue initially setting it up.
> >
> > I use port mirroring on a switch to replicate all ports to port
> 5 where a dedicated ntopng interface 'listens' (Official package
> on raspbian 10).
> > On that same switch I have my Internet gateway (Unifi USG3P)
> connected to port 1. This same device also acts as a DHCP/DNS server.
> >
> > When mirroring all ports BUT port 1, I receive alerts about
> thousands of DNS queries not being answered. I did confirm that
> with a pcap dump.
>
> When you monitor just port 1, apart from the DNS queries
> unanswered alerts, do you get bi-directional traffic if you look
> at the flows page? Do you see the @1?
>
>
> >
> > So I went and started to mirror port 1 along with others, and
> the missing traffic (DNS replies) started to be collected.
> > The issue is that with that configuration, all flows are listed
> twice in ntop. Internal hosts are showing normally and with "@1"
> at the end of the hostname.
>
> @1 means VLAN=1 so VLAN-tagged packets are received from the
> mirror port. VLAN depend on your switch configuration. If you can
> disregard VLANs you can use option --ignore-vlans
>
> >
> > Is there a way for ntop to discard this duplicated traffic in
> the accounting of ntopng?
>
> I am not sure the traffic is duplicated. It could be that ntopng
> is keeping the two directions of every flow separated due to the
> VLAN. Let's continue the investigation depending on your responses.
>
> Simone
>
> >   It makes sense to me that it is detected as a host's traffic
> will be seen on its own switch port and then in many cases on port 1.
> >
> > Many thanks.
> > _______________________________________________
> > Ntop mailing list
> > Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Duplicate flow entries [ In reply to ]
*Hi Emanuele,*

*I do confirm this was the same issue. I updated the package and alerts
went away.*


*Many thanks.*

Le lun. 25 mai 2020 à 12:23, Emanuele Faranda <faranda@ntop.org> a écrit :

> Hi David,
>
> This should be the same issue experienced by Aaron due to frame padding.
> Please check out the other thread.
>
> Regards,
>
> Emanuele
> On 5/24/20 7:54 PM, David van Ginneken wrote:
>
> Hi Simone,
>
> Thanks for the advice. The --ignore-vlans option seems to help as I do not
> see the duplicates anymore.
> I do have vlans on my network but it is not a problem for me not to have
> this separated in the display.
>
> Now I still get odd alerts about HTTP requests not being answered. I'll
> investigate a bit further but it seems VERY similar to the issue Aaron and
> Emanuele are discussing in parallel.
> And, on top of that, it seems Aaron is also using a Unifi device (not the
> same model but I think the OSes are). Could this be a pointer to the root
> cause of our issues?
>
> Thanks again.
>
>
> Le jeu. 21 mai 2020 à 23:18, Simone Mainardi <mainardi@ntop.org> a écrit :
>
>> Hi,
>>
>> > On 21 May 2020, at 14:55, David van Ginneken <david@van-ginneken.org>
>> wrote:
>> >
>> > Hi everyone,
>> >
>> > Starting with ntopng, I have a small issue initially setting it up.
>> >
>> > I use port mirroring on a switch to replicate all ports to port 5 where
>> a dedicated ntopng interface 'listens' (Official package on raspbian 10).
>> > On that same switch I have my Internet gateway (Unifi USG3P) connected
>> to port 1. This same device also acts as a DHCP/DNS server.
>> >
>> > When mirroring all ports BUT port 1, I receive alerts about thousands
>> of DNS queries not being answered. I did confirm that with a pcap dump.
>>
>> When you monitor just port 1, apart from the DNS queries unanswered
>> alerts, do you get bi-directional traffic if you look at the flows page? Do
>> you see the @1?
>>
>
>> >
>> > So I went and started to mirror port 1 along with others, and the
>> missing traffic (DNS replies) started to be collected.
>> > The issue is that with that configuration, all flows are listed twice
>> in ntop. Internal hosts are showing normally and with "@1" at the end of
>> the hostname.
>>
>> @1 means VLAN=1 so VLAN-tagged packets are received from the mirror port.
>> VLAN depend on your switch configuration. If you can disregard VLANs you
>> can use option --ignore-vlans
>>
>> >
>> > Is there a way for ntop to discard this duplicated traffic in the
>> accounting of ntopng?
>>
>> I am not sure the traffic is duplicated. It could be that ntopng is
>> keeping the two directions of every flow separated due to the VLAN. Let's
>> continue the investigation depending on your responses.
>>
>> Simone
>>
>> > It makes sense to me that it is detected as a host's traffic will be
>> seen on its own switch port and then in many cases on port 1.
>> >
>> > Many thanks.
>> > _______________________________________________
>> > Ntop mailing list
>> > Ntop@listgateway.unipi.it
>> > http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>
> _______________________________________________
> Ntop mailing listNtop@listgateway.unipi.ithttp://listgateway.unipi.it/mailman/listinfo/ntop
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Duplicate flow entries [ In reply to ]
*Hi Emanuele,*

*I do confirm this was the same issue. I updated the package and alerts
went away.*


*Many thanks.*

Le lun. 25 mai 2020 à 12:23, Emanuele Faranda <faranda@ntop.org> a écrit :

> Hi David,
>
> This should be the same issue experienced by Aaron due to frame padding.
> Please check out the other thread.
>
> Regards,
>
> Emanuele
> On 5/24/20 7:54 PM, David van Ginneken wrote:
>
> Hi Simone,
>
> Thanks for the advice. The --ignore-vlans option seems to help as I do not
> see the duplicates anymore.
> I do have vlans on my network but it is not a problem for me not to have
> this separated in the display.
>
> Now I still get odd alerts about HTTP requests not being answered. I'll
> investigate a bit further but it seems VERY similar to the issue Aaron and
> Emanuele are discussing in parallel.
> And, on top of that, it seems Aaron is also using a Unifi device (not the
> same model but I think the OSes are). Could this be a pointer to the root
> cause of our issues?
>
> Thanks again.
>
>
> Le jeu. 21 mai 2020 à 23:18, Simone Mainardi <mainardi@ntop.org> a écrit :
>
>> Hi,
>>
>> > On 21 May 2020, at 14:55, David van Ginneken <david@van-ginneken.org>
>> wrote:
>> >
>> > Hi everyone,
>> >
>> > Starting with ntopng, I have a small issue initially setting it up.
>> >
>> > I use port mirroring on a switch to replicate all ports to port 5 where
>> a dedicated ntopng interface 'listens' (Official package on raspbian 10).
>> > On that same switch I have my Internet gateway (Unifi USG3P) connected
>> to port 1. This same device also acts as a DHCP/DNS server.
>> >
>> > When mirroring all ports BUT port 1, I receive alerts about thousands
>> of DNS queries not being answered. I did confirm that with a pcap dump.
>>
>> When you monitor just port 1, apart from the DNS queries unanswered
>> alerts, do you get bi-directional traffic if you look at the flows page? Do
>> you see the @1?
>>
>
>> >
>> > So I went and started to mirror port 1 along with others, and the
>> missing traffic (DNS replies) started to be collected.
>> > The issue is that with that configuration, all flows are listed twice
>> in ntop. Internal hosts are showing normally and with "@1" at the end of
>> the hostname.
>>
>> @1 means VLAN=1 so VLAN-tagged packets are received from the mirror port.
>> VLAN depend on your switch configuration. If you can disregard VLANs you
>> can use option --ignore-vlans
>>
>> >
>> > Is there a way for ntop to discard this duplicated traffic in the
>> accounting of ntopng?
>>
>> I am not sure the traffic is duplicated. It could be that ntopng is
>> keeping the two directions of every flow separated due to the VLAN. Let's
>> continue the investigation depending on your responses.
>>
>> Simone
>>
>> > It makes sense to me that it is detected as a host's traffic will be
>> seen on its own switch port and then in many cases on port 1.
>> >
>> > Many thanks.
>> > _______________________________________________
>> > Ntop mailing list
>> > Ntop@listgateway.unipi.it
>> > http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>
> _______________________________________________
> Ntop mailing listNtop@listgateway.unipi.ithttp://listgateway.unipi.it/mailman/listinfo/ntop
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Duplicate flow entries [ In reply to ]
Hi David,

Great, thank you for reporting.

Regards,

Emanuele

On 5/27/20 2:17 PM, David van Ginneken wrote:
> /Hi Emanuele,/
> /
> /
> /I do confirm this was the same issue. I updated the package and
> alerts went away./
> /
> /
> /Many thanks.
> /
>
> Le lun. 25 mai 2020 à 12:23, Emanuele Faranda <faranda@ntop.org
> <mailto:faranda@ntop.org>> a écrit :
>
> Hi David,
>
> This should be the same issue experienced by Aaron due to frame
> padding. Please check out the other thread.
>
> Regards,
>
> Emanuele
>
> On 5/24/20 7:54 PM, David van Ginneken wrote:
>> Hi Simone,
>>
>> Thanks for the advice. The --ignore-vlans option seems to help as
>> I do not see the duplicates anymore.
>> I do have vlans on my network but it is not a problem for me not
>> to have this separated in the display.
>>
>> Now I still get odd alerts about HTTP requests not being
>> answered. I'll investigate a bit further but it seems VERY
>> similar to the issue Aaron and Emanuele are discussing in parallel.
>> And, on top of that, it seems Aaron is also using a Unifi device
>> (not the same model but I think the OSes are). Could this be a
>> pointer to the root cause of our issues?
>>
>> Thanks again.
>>
>>
>> Le jeu. 21 mai 2020 à 23:18, Simone Mainardi <mainardi@ntop.org
>> <mailto:mainardi@ntop.org>> a écrit :
>>
>> Hi,
>>
>> > On 21 May 2020, at 14:55, David van Ginneken
>> <david@van-ginneken.org <mailto:david@van-ginneken.org>> wrote:
>> >
>> > Hi everyone,
>> >
>> > Starting with ntopng, I have a small issue initially
>> setting it up.
>> >
>> > I use port mirroring on a switch to replicate all ports to
>> port 5 where a dedicated ntopng interface 'listens' (Official
>> package on raspbian 10).
>> > On that same switch I have my Internet gateway (Unifi
>> USG3P) connected to port 1. This same device also acts as a
>> DHCP/DNS server.
>> >
>> > When mirroring all ports BUT port 1, I receive alerts about
>> thousands of DNS queries not being answered. I did confirm
>> that with a pcap dump.
>>
>> When you monitor just port 1, apart from the DNS queries
>> unanswered alerts, do you get bi-directional traffic if you
>> look at the flows page? Do you see the @1?
>>
>>
>> >
>> > So I went and started to mirror port 1 along with others,
>> and the missing traffic (DNS replies) started to be collected.
>> > The issue is that with that configuration, all flows are
>> listed twice in ntop. Internal hosts are showing normally and
>> with "@1" at the end of the hostname.
>>
>> @1 means VLAN=1 so VLAN-tagged packets are received from the
>> mirror port. VLAN depend on your switch configuration. If you
>> can disregard VLANs you can use option --ignore-vlans
>>
>> >
>> > Is there a way for ntop to discard this duplicated traffic
>> in the accounting of ntopng?
>>
>> I am not sure the traffic is duplicated. It could be that
>> ntopng is keeping the two directions of every flow separated
>> due to the VLAN. Let's continue the investigation depending
>> on your responses.
>>
>> Simone
>>
>> >   It makes sense to me that it is detected as a host's
>> traffic will be seen on its own switch port and then in many
>> cases on port 1.
>> >
>> > Many thanks.
>> > _______________________________________________
>> > Ntop mailing list
>> > Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>> > http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop