Mailing List Archive

As a sidenote, a second router that runs my LookingGlass and is used to
inject blackhole routes, is also not receiving the full table from the
BGP02.
This can be observed at http://lg.as48972.net/

Jan Hugo Prins



On 11/03/2016 04:36 PM, jan hugo prins wrote:
> Hi everyone,
>
> I have a strange issue.
> I have 2 external BGP routers. They both have 2 uplinks to my transit
> providers and this works fine.
> Until a while ago, when I wanted to take one router out for maintenance,
> I would shut it's transit links, remove "default-information originate
> always" from OSPF, set higher costs on the uplink and / or set
> "max-metric router-lsa administrative" wait for most of the traffic to
> shift to the second router and do my maintenance.
>
> Because the router would always have an iBGP link to the second router
> it would receive all routes from this router and be fine in the case of
> some traffic coming in anyway. But today I wanted to upgrade Quagga to
> version 1.10 and I saw something really strange:
>
> bgp02.as48972.net# sh ip bgp sum
> BGP router identifier 95.130.232.3, local AS number 48972
> RIB entries 1120940, using 120 MiB of memory
> Peers 29, using 129 KiB of memory
> Peer groups 3, using 96 bytes of memory
>
> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down
> State/P
> fxRcd
> 77.222.66.177 4 16243 1434500 11053 0 0 0 01w0d16h 609362
> 78.152.49.109 4 5580 3162545 66219 0 0 0 01w0d16h 609733
> 95.130.232.2 4 48972 0 0 0 0 0 never Idle
> (Admin)
> 95.130.232.4 4 48972 10983 58016 0 0 0 02:28:03 0
> 95.130.232.6 4 48972 2536718 65789 0 0 0 02:21:53 137022
> 193.108.98.2 4 20828 208151 11056 0 0 0 2d09h49m 137235
> 193.108.98.3 4 20828 204274 11056 0 0 0 2d09h39m 137227
>
> bgp03.as48972.net(config-router)# do sh ip bgp sum
> BGP router identifier 95.130.232.6, local AS number 48972
> RIB entries 311015, using 33 MiB of memory
> Peers 21, using 94 KiB of memory
> Peer groups 3, using 96 bytes of memory
>
> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down
> State/P
> fxRcd
> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21 Idle
> (Admin)
> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14 Idle
> (Admin)
> 95.130.232.2 4 48972 0 0 0 0 0 never Idle
> (Admin)
> 95.130.232.3 4 48972 7643 867546 0 0 0 02:21:10 39002
> 95.130.232.4 4 48972 837 558695 0 0 0 02:27:21 0
> 193.108.98.2 4 20828 68276 883 0 0 0 03:30:44 137235
> 193.108.98.3 4 20828 64031 883 0 0 0 03:30:35 137227
>
> I would expect that the BGP03 would get at least 580.000 routes from the
> BGP02, but this doesn't happen for some strange reason.
> Also resetting the iBGP link between the BGP02 and BGP03 does not fix
> the problem.
>
> Anyone any idea?
>
> jan Hugo Prins
>
>
>
> _______________________________________________
> Quagga-users mailing list
> Quagga-users@lists.quagga.net
> https://lists.quagga.net/mailman/listinfo/quagga-users


_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

Hello,

you could do a tcpdump on the LookingGlass route collector to debug what
has changed between versions. Are the missing prefixes sent? If so what is
the difference (next-hop...)?

tcpdump -nni any "host ip and port 179" -s0 -w /tmp/file.cap

Ngrep and wireshark should help.

2016-11-03 17:08 GMT+01:00 jan hugo prins <jhp@jhprins.org>:

> As a sidenote, a second router that runs my LookingGlass and is used to
> inject blackhole routes, is also not receiving the full table from the
> BGP02.
> This can be observed at http://lg.as48972.net/
>
> Jan Hugo Prins
>
>
>
> On 11/03/2016 04:36 PM, jan hugo prins wrote:
> > Hi everyone,
> >
> > I have a strange issue.
> > I have 2 external BGP routers. They both have 2 uplinks to my transit
> > providers and this works fine.
> > Until a while ago, when I wanted to take one router out for maintenance,
> > I would shut it's transit links, remove "default-information originate
> > always" from OSPF, set higher costs on the uplink and / or set
> > "max-metric router-lsa administrative" wait for most of the traffic to
> > shift to the second router and do my maintenance.
> >
> > Because the router would always have an iBGP link to the second router
> > it would receive all routes from this router and be fine in the case of
> > some traffic coming in anyway. But today I wanted to upgrade Quagga to
> > version 1.10 and I saw something really strange:
> >
> > bgp02.as48972.net# sh ip bgp sum
> > BGP router identifier 95.130.232.3, local AS number 48972
> > RIB entries 1120940, using 120 MiB of memory
> > Peers 29, using 129 KiB of memory
> > Peer groups 3, using 96 bytes of memory
> >
> > Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down
> > State/P
> > fxRcd
> > 77.222.66.177 4 16243 1434500 11053 0 0 0 01w0d16h
> 609362
> > 78.152.49.109 4 5580 3162545 66219 0 0 0 01w0d16h
> 609733
> > 95.130.232.2 4 48972 0 0 0 0 0 never Idle
> > (Admin)
> > 95.130.232.4 4 48972 10983 58016 0 0 0 02:28:03
> 0
> > 95.130.232.6 4 48972 2536718 65789 0 0 0 02:21:53
> 137022
> > 193.108.98.2 4 20828 208151 11056 0 0 0 2d09h49m
> 137235
> > 193.108.98.3 4 20828 204274 11056 0 0 0 2d09h39m
> 137227
> >
> > bgp03.as48972.net(config-router)# do sh ip bgp sum
> > BGP router identifier 95.130.232.6, local AS number 48972
> > RIB entries 311015, using 33 MiB of memory
> > Peers 21, using 94 KiB of memory
> > Peer groups 3, using 96 bytes of memory
> >
> > Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down
> > State/P
> > fxRcd
> > 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21 Idle
> > (Admin)
> > 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14 Idle
> > (Admin)
> > 95.130.232.2 4 48972 0 0 0 0 0 never Idle
> > (Admin)
> > 95.130.232.3 4 48972 7643 867546 0 0 0 02:21:10
> 39002
> > 95.130.232.4 4 48972 837 558695 0 0 0 02:27:21
> 0
> > 193.108.98.2 4 20828 68276 883 0 0 0 03:30:44
> 137235
> > 193.108.98.3 4 20828 64031 883 0 0 0 03:30:35
> 137227
> >
> > I would expect that the BGP03 would get at least 580.000 routes from the
> > BGP02, but this doesn't happen for some strange reason.
> > Also resetting the iBGP link between the BGP02 and BGP03 does not fix
> > the problem.
> >
> > Anyone any idea?
> >
> > jan Hugo Prins
> >
> >
> >
> > _______________________________________________
> > Quagga-users mailing list
> > Quagga-users@lists.quagga.net
> > https://lists.quagga.net/mailman/listinfo/quagga-users
>
>
> _______________________________________________
> Quagga-users mailing list
> Quagga-users@lists.quagga.net
> https://lists.quagga.net/mailman/listinfo/quagga-users
>

Some extra side note:

A random prefix on the BGP02:

bgp02.as48972.net# sh ip bgp 23.226.189.0/24
BGP routing table entry for 23.226.189.0/24
Paths: (3 available, best #2, table Default-IP-Routing-Table)
Advertised to non peer-group peers:
95.130.232.4 95.130.232.6
5580 5580 1299 4809 38197
78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6)
Origin IGP, metric 0, localpref 90, valid, internal
Last update: Thu Nov 3 16:09:45 2016

16243 47886 3356 4809 38197
77.222.66.177 from 77.222.66.177 (87.249.109.240)
Origin IGP, localpref 90, valid, external, best
Last update: Thu Oct 27 00:56:04 2016

5580 5580 1299 4809 38197
78.152.49.109 from 78.152.49.109 (80.94.64.124)
Origin IGP, metric 0, localpref 90, valid, external
Community: 1299:30000 5580:21220
Last update: Mon Oct 31 17:51:58 2016

Is available on the BGP03:

bgp03.as48972.net# sh ip bgp 23.226.189.0/24
BGP routing table entry for 23.226.189.0/24
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Advertised to non peer-group peers:
95.130.232.3 95.130.232.4
16243 47886 3356 4809 38197
77.222.66.181 from 77.222.66.181 (87.249.109.241)
Origin IGP, localpref 90, valid, external
Last update: Thu Nov 3 16:09:38 2016

5580 5580 1299 4809 38197
78.152.40.92 from 78.152.40.92 (80.94.64.124)
Origin IGP, metric 0, localpref 90, valid, external, best
Community: 1299:30000 5580:21220
Last update: Thu Nov 3 16:09:21 2016

But even though the BGP02 claims to have it advertised to both the bgp03
and the routeserver / lookingglass:

[root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24
[root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24

It did not.

Jan Hugo Prins



On 11/03/2016 05:08 PM, jan hugo prins wrote:
> As a sidenote, a second router that runs my LookingGlass and is used to
> inject blackhole routes, is also not receiving the full table from the
> BGP02.
> This can be observed at http://lg.as48972.net/
>
> Jan Hugo Prins
>
>
>
> On 11/03/2016 04:36 PM, jan hugo prins wrote:
>> Hi everyone,
>>
>> I have a strange issue.
>> I have 2 external BGP routers. They both have 2 uplinks to my transit
>> providers and this works fine.
>> Until a while ago, when I wanted to take one router out for maintenance,
>> I would shut it's transit links, remove "default-information originate
>> always" from OSPF, set higher costs on the uplink and / or set
>> "max-metric router-lsa administrative" wait for most of the traffic to
>> shift to the second router and do my maintenance.
>>
>> Because the router would always have an iBGP link to the second router
>> it would receive all routes from this router and be fine in the case of
>> some traffic coming in anyway. But today I wanted to upgrade Quagga to
>> version 1.10 and I saw something really strange:
>>
>> bgp02.as48972.net# sh ip bgp sum
>> BGP router identifier 95.130.232.3, local AS number 48972
>> RIB entries 1120940, using 120 MiB of memory
>> Peers 29, using 129 KiB of memory
>> Peer groups 3, using 96 bytes of memory
>>
>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down
>> State/P
>> fxRcd
>> 77.222.66.177 4 16243 1434500 11053 0 0 0 01w0d16h 609362
>> 78.152.49.109 4 5580 3162545 66219 0 0 0 01w0d16h 609733
>> 95.130.232.2 4 48972 0 0 0 0 0 never Idle
>> (Admin)
>> 95.130.232.4 4 48972 10983 58016 0 0 0 02:28:03 0
>> 95.130.232.6 4 48972 2536718 65789 0 0 0 02:21:53 137022
>> 193.108.98.2 4 20828 208151 11056 0 0 0 2d09h49m 137235
>> 193.108.98.3 4 20828 204274 11056 0 0 0 2d09h39m 137227
>>
>> bgp03.as48972.net(config-router)# do sh ip bgp sum
>> BGP router identifier 95.130.232.6, local AS number 48972
>> RIB entries 311015, using 33 MiB of memory
>> Peers 21, using 94 KiB of memory
>> Peer groups 3, using 96 bytes of memory
>>
>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down
>> State/P
>> fxRcd
>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21 Idle
>> (Admin)
>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14 Idle
>> (Admin)
>> 95.130.232.2 4 48972 0 0 0 0 0 never Idle
>> (Admin)
>> 95.130.232.3 4 48972 7643 867546 0 0 0 02:21:10 39002
>> 95.130.232.4 4 48972 837 558695 0 0 0 02:27:21 0
>> 193.108.98.2 4 20828 68276 883 0 0 0 03:30:44 137235
>> 193.108.98.3 4 20828 64031 883 0 0 0 03:30:35 137227
>>
>> I would expect that the BGP03 would get at least 580.000 routes from the
>> BGP02, but this doesn't happen for some strange reason.
>> Also resetting the iBGP link between the BGP02 and BGP03 does not fix
>> the problem.
>>
>> Anyone any idea?
>>
>> jan Hugo Prins
>>
>>
>>
>> _______________________________________________
>> Quagga-users mailing list
>> Quagga-users@lists.quagga.net
>> https://lists.quagga.net/mailman/listinfo/quagga-users
>
> _______________________________________________
> Quagga-users mailing list
> Quagga-users@lists.quagga.net
> https://lists.quagga.net/mailman/listinfo/quagga-users


_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

Jan,

can you clarify if there are any route-maps, filter-list, prefix-lists
or anything else?
Also, are you doing any “neighbor x.x.x.x next-hop-self” or
peer groups?

(I’m not sure interested in config which would match, but also filters
etc on the relevant peers in case they are misbehaving)

My testing on filter-lists etc is a bit lacking, so I would assume the
bug most likely to be there.

- Martin


On 3 Nov 2016, at 9:41, jan hugo prins wrote:

> Some extra side note:
>
> A random prefix on the BGP02:
>
> bgp02.as48972.net# sh ip bgp 23.226.189.0/24
> BGP routing table entry for 23.226.189.0/24
> Paths: (3 available, best #2, table Default-IP-Routing-Table)
> Advertised to non peer-group peers:
> 95.130.232.4 95.130.232.6
> 5580 5580 1299 4809 38197
> 78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6)
> Origin IGP, metric 0, localpref 90, valid, internal
> Last update: Thu Nov 3 16:09:45 2016
>
> 16243 47886 3356 4809 38197
> 77.222.66.177 from 77.222.66.177 (87.249.109.240)
> Origin IGP, localpref 90, valid, external, best
> Last update: Thu Oct 27 00:56:04 2016
>
> 5580 5580 1299 4809 38197
> 78.152.49.109 from 78.152.49.109 (80.94.64.124)
> Origin IGP, metric 0, localpref 90, valid, external
> Community: 1299:30000 5580:21220
> Last update: Mon Oct 31 17:51:58 2016
>
> Is available on the BGP03:
>
> bgp03.as48972.net# sh ip bgp 23.226.189.0/24
> BGP routing table entry for 23.226.189.0/24
> Paths: (2 available, best #2, table Default-IP-Routing-Table)
> Advertised to non peer-group peers:
> 95.130.232.3 95.130.232.4
> 16243 47886 3356 4809 38197
> 77.222.66.181 from 77.222.66.181 (87.249.109.241)
> Origin IGP, localpref 90, valid, external
> Last update: Thu Nov 3 16:09:38 2016
>
> 5580 5580 1299 4809 38197
> 78.152.40.92 from 78.152.40.92 (80.94.64.124)
> Origin IGP, metric 0, localpref 90, valid, external, best
> Community: 1299:30000 5580:21220
> Last update: Thu Nov 3 16:09:21 2016
>
> But even though the BGP02 claims to have it advertised to both the
> bgp03
> and the routeserver / lookingglass:
>
> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24
> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24
>
> It did not.
>
> Jan Hugo Prins
>
> On 11/03/2016 05:08 PM, jan hugo prins wrote:
>> As a sidenote, a second router that runs my LookingGlass and is used
>> to
>> inject blackhole routes, is also not receiving the full table from
>> the
>> BGP02.
>> This can be observed at http://lg.as48972.net/
>>
>> Jan Hugo Prins
>>
>>
>>
>> On 11/03/2016 04:36 PM, jan hugo prins wrote:
>>> Hi everyone,
>>>
>>> I have a strange issue.
>>> I have 2 external BGP routers. They both have 2 uplinks to my
>>> transit
>>> providers and this works fine.
>>> Until a while ago, when I wanted to take one router out for
>>> maintenance,
>>> I would shut it's transit links, remove "default-information
>>> originate
>>> always" from OSPF, set higher costs on the uplink and / or set
>>> "max-metric router-lsa administrative" wait for most of the traffic
>>> to
>>> shift to the second router and do my maintenance.
>>>
>>> Because the router would always have an iBGP link to the second
>>> router
>>> it would receive all routes from this router and be fine in the case
>>> of
>>> some traffic coming in anyway. But today I wanted to upgrade Quagga
>>> to
>>> version 1.10 and I saw something really strange:
>>>
>>> bgp02.as48972.net# sh ip bgp sum
>>> BGP router identifier 95.130.232.3, local AS number 48972
>>> RIB entries 1120940, using 120 MiB of memory
>>> Peers 29, using 129 KiB of memory
>>> Peer groups 3, using 96 bytes of memory
>>>
>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>> Up/Down
>>> State/P
>>> fxRcd
>>> 77.222.66.177 4 16243 1434500 11053 0 0 0 01w0d16h
>>> 609362
>>> 78.152.49.109 4 5580 3162545 66219 0 0 0 01w0d16h
>>> 609733
>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>> Idle
>>> (Admin)
>>> 95.130.232.4 4 48972 10983 58016 0 0 0 02:28:03
>>> 0
>>> 95.130.232.6 4 48972 2536718 65789 0 0 0 02:21:53
>>> 137022
>>> 193.108.98.2 4 20828 208151 11056 0 0 0 2d09h49m
>>> 137235
>>> 193.108.98.3 4 20828 204274 11056 0 0 0 2d09h39m
>>> 137227
>>>
>>> bgp03.as48972.net(config-router)# do sh ip bgp sum
>>> BGP router identifier 95.130.232.6, local AS number 48972
>>> RIB entries 311015, using 33 MiB of memory
>>> Peers 21, using 94 KiB of memory
>>> Peer groups 3, using 96 bytes of memory
>>>
>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>> Up/Down
>>> State/P
>>> fxRcd
>>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21
>>> Idle
>>> (Admin)
>>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14
>>> Idle
>>> (Admin)
>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>> Idle
>>> (Admin)
>>> 95.130.232.3 4 48972 7643 867546 0 0 0 02:21:10
>>> 39002
>>> 95.130.232.4 4 48972 837 558695 0 0 0 02:27:21
>>> 0
>>> 193.108.98.2 4 20828 68276 883 0 0 0 03:30:44
>>> 137235
>>> 193.108.98.3 4 20828 64031 883 0 0 0 03:30:35
>>> 137227
>>>
>>> I would expect that the BGP03 would get at least 580.000 routes from
>>> the
>>> BGP02, but this doesn't happen for some strange reason.
>>> Also resetting the iBGP link between the BGP02 and BGP03 does not
>>> fix
>>> the problem.
>>>
>>> Anyone any idea?
>>>
>>> jan Hugo Prins
>>>
>>>
>>>
>>> _______________________________________________
>>> Quagga-users mailing list
>>> Quagga-users@lists.quagga.net
>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>
>> _______________________________________________
>> Quagga-users mailing list
>> Quagga-users@lists.quagga.net
>> https://lists.quagga.net/mailman/listinfo/quagga-users
>
> _______________________________________________
> Quagga-users mailing list
> Quagga-users@lists.quagga.net
> https://lists.quagga.net/mailman/listinfo/quagga-users

_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

I'm happy to provide you with the complete config of my 3 routers.
I will send them to you personaly after stripping the passwords.

Jan Hugo


On 11/04/2016 12:32 AM, Martin Winter wrote:
> Jan,
>
> can you clarify if there are any route-maps, filter-list, prefix-lists
> or anything else?
> Also, are you doing any “neighbor x.x.x.x next-hop-self” or
> peer groups?
>
> (I’m not sure interested in config which would match, but also filters
> etc on the relevant peers in case they are misbehaving)
>
> My testing on filter-lists etc is a bit lacking, so I would assume the
> bug most likely to be there.
>
> - Martin
>
>
> On 3 Nov 2016, at 9:41, jan hugo prins wrote:
>
>> Some extra side note:
>>
>> A random prefix on the BGP02:
>>
>> bgp02.as48972.net# sh ip bgp 23.226.189.0/24
>> BGP routing table entry for 23.226.189.0/24
>> Paths: (3 available, best #2, table Default-IP-Routing-Table)
>> Advertised to non peer-group peers:
>> 95.130.232.4 95.130.232.6
>> 5580 5580 1299 4809 38197
>> 78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6)
>> Origin IGP, metric 0, localpref 90, valid, internal
>> Last update: Thu Nov 3 16:09:45 2016
>>
>> 16243 47886 3356 4809 38197
>> 77.222.66.177 from 77.222.66.177 (87.249.109.240)
>> Origin IGP, localpref 90, valid, external, best
>> Last update: Thu Oct 27 00:56:04 2016
>>
>> 5580 5580 1299 4809 38197
>> 78.152.49.109 from 78.152.49.109 (80.94.64.124)
>> Origin IGP, metric 0, localpref 90, valid, external
>> Community: 1299:30000 5580:21220
>> Last update: Mon Oct 31 17:51:58 2016
>>
>> Is available on the BGP03:
>>
>> bgp03.as48972.net# sh ip bgp 23.226.189.0/24
>> BGP routing table entry for 23.226.189.0/24
>> Paths: (2 available, best #2, table Default-IP-Routing-Table)
>> Advertised to non peer-group peers:
>> 95.130.232.3 95.130.232.4
>> 16243 47886 3356 4809 38197
>> 77.222.66.181 from 77.222.66.181 (87.249.109.241)
>> Origin IGP, localpref 90, valid, external
>> Last update: Thu Nov 3 16:09:38 2016
>>
>> 5580 5580 1299 4809 38197
>> 78.152.40.92 from 78.152.40.92 (80.94.64.124)
>> Origin IGP, metric 0, localpref 90, valid, external, best
>> Community: 1299:30000 5580:21220
>> Last update: Thu Nov 3 16:09:21 2016
>>
>> But even though the BGP02 claims to have it advertised to both the bgp03
>> and the routeserver / lookingglass:
>>
>> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24
>> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24
>>
>> It did not.
>>
>> Jan Hugo Prins
>>
>> On 11/03/2016 05:08 PM, jan hugo prins wrote:
>>> As a sidenote, a second router that runs my LookingGlass and is used to
>>> inject blackhole routes, is also not receiving the full table from the
>>> BGP02.
>>> This can be observed at http://lg.as48972.net/
>>>
>>> Jan Hugo Prins
>>>
>>>
>>>
>>> On 11/03/2016 04:36 PM, jan hugo prins wrote:
>>>> Hi everyone,
>>>>
>>>> I have a strange issue.
>>>> I have 2 external BGP routers. They both have 2 uplinks to my transit
>>>> providers and this works fine.
>>>> Until a while ago, when I wanted to take one router out for
>>>> maintenance,
>>>> I would shut it's transit links, remove "default-information originate
>>>> always" from OSPF, set higher costs on the uplink and / or set
>>>> "max-metric router-lsa administrative" wait for most of the
>>>> traffic to
>>>> shift to the second router and do my maintenance.
>>>>
>>>> Because the router would always have an iBGP link to the second router
>>>> it would receive all routes from this router and be fine in the
>>>> case of
>>>> some traffic coming in anyway. But today I wanted to upgrade Quagga to
>>>> version 1.10 and I saw something really strange:
>>>>
>>>> bgp02.as48972.net# sh ip bgp sum
>>>> BGP router identifier 95.130.232.3, local AS number 48972
>>>> RIB entries 1120940, using 120 MiB of memory
>>>> Peers 29, using 129 KiB of memory
>>>> Peer groups 3, using 96 bytes of memory
>>>>
>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>> Up/Down
>>>> State/P
>>>> fxRcd
>>>> 77.222.66.177 4 16243 1434500 11053 0 0 0
>>>> 01w0d16h 609362
>>>> 78.152.49.109 4 5580 3162545 66219 0 0 0
>>>> 01w0d16h 609733
>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>> Idle
>>>> (Admin)
>>>> 95.130.232.4 4 48972 10983 58016 0 0 0
>>>> 02:28:03 0
>>>> 95.130.232.6 4 48972 2536718 65789 0 0 0
>>>> 02:21:53 137022
>>>> 193.108.98.2 4 20828 208151 11056 0 0 0
>>>> 2d09h49m 137235
>>>> 193.108.98.3 4 20828 204274 11056 0 0 0
>>>> 2d09h39m 137227
>>>>
>>>> bgp03.as48972.net(config-router)# do sh ip bgp sum
>>>> BGP router identifier 95.130.232.6, local AS number 48972
>>>> RIB entries 311015, using 33 MiB of memory
>>>> Peers 21, using 94 KiB of memory
>>>> Peer groups 3, using 96 bytes of memory
>>>>
>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>> Up/Down
>>>> State/P
>>>> fxRcd
>>>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21
>>>> Idle
>>>> (Admin)
>>>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14
>>>> Idle
>>>> (Admin)
>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>> Idle
>>>> (Admin)
>>>> 95.130.232.3 4 48972 7643 867546 0 0 0
>>>> 02:21:10 39002
>>>> 95.130.232.4 4 48972 837 558695 0 0 0
>>>> 02:27:21 0
>>>> 193.108.98.2 4 20828 68276 883 0 0 0
>>>> 03:30:44 137235
>>>> 193.108.98.3 4 20828 64031 883 0 0 0
>>>> 03:30:35 137227
>>>>
>>>> I would expect that the BGP03 would get at least 580.000 routes
>>>> from the
>>>> BGP02, but this doesn't happen for some strange reason.
>>>> Also resetting the iBGP link between the BGP02 and BGP03 does not fix
>>>> the problem.
>>>>
>>>> Anyone any idea?
>>>>
>>>> jan Hugo Prins
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Quagga-users mailing list
>>>> Quagga-users@lists.quagga.net
>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>
>>> _______________________________________________
>>> Quagga-users mailing list
>>> Quagga-users@lists.quagga.net
>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>
>> _______________________________________________
>> Quagga-users mailing list
>> Quagga-users@lists.quagga.net
>> https://lists.quagga.net/mailman/listinfo/quagga-users
>
> _______________________________________________
> Quagga-users mailing list
> Quagga-users@lists.quagga.net
> https://lists.quagga.net/mailman/listinfo/quagga-users


_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

This sounds like the problem I saw last Jan.

https://lists.quagga.net/pipermail/quagga-users/2016-January/014248.html



On 11/3/2016 8:31 PM, jan hugo prins wrote:
> I'm happy to provide you with the complete config of my 3 routers.
> I will send them to you personaly after stripping the passwords.
>
> Jan Hugo
>
>
> On 11/04/2016 12:32 AM, Martin Winter wrote:
>> Jan,
>>
>> can you clarify if there are any route-maps, filter-list, prefix-lists
>> or anything else?
>> Also, are you doing any “neighbor x.x.x.x next-hop-self” or
>> peer groups?
>>
>> (I’m not sure interested in config which would match, but also filters
>> etc on the relevant peers in case they are misbehaving)
>>
>> My testing on filter-lists etc is a bit lacking, so I would assume the
>> bug most likely to be there.
>>
>> - Martin
>>
>>
>> On 3 Nov 2016, at 9:41, jan hugo prins wrote:
>>
>>> Some extra side note:
>>>
>>> A random prefix on the BGP02:
>>>
>>> bgp02.as48972.net# sh ip bgp 23.226.189.0/24
>>> BGP routing table entry for 23.226.189.0/24
>>> Paths: (3 available, best #2, table Default-IP-Routing-Table)
>>> Advertised to non peer-group peers:
>>> 95.130.232.4 95.130.232.6
>>> 5580 5580 1299 4809 38197
>>> 78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6)
>>> Origin IGP, metric 0, localpref 90, valid, internal
>>> Last update: Thu Nov 3 16:09:45 2016
>>>
>>> 16243 47886 3356 4809 38197
>>> 77.222.66.177 from 77.222.66.177 (87.249.109.240)
>>> Origin IGP, localpref 90, valid, external, best
>>> Last update: Thu Oct 27 00:56:04 2016
>>>
>>> 5580 5580 1299 4809 38197
>>> 78.152.49.109 from 78.152.49.109 (80.94.64.124)
>>> Origin IGP, metric 0, localpref 90, valid, external
>>> Community: 1299:30000 5580:21220
>>> Last update: Mon Oct 31 17:51:58 2016
>>>
>>> Is available on the BGP03:
>>>
>>> bgp03.as48972.net# sh ip bgp 23.226.189.0/24
>>> BGP routing table entry for 23.226.189.0/24
>>> Paths: (2 available, best #2, table Default-IP-Routing-Table)
>>> Advertised to non peer-group peers:
>>> 95.130.232.3 95.130.232.4
>>> 16243 47886 3356 4809 38197
>>> 77.222.66.181 from 77.222.66.181 (87.249.109.241)
>>> Origin IGP, localpref 90, valid, external
>>> Last update: Thu Nov 3 16:09:38 2016
>>>
>>> 5580 5580 1299 4809 38197
>>> 78.152.40.92 from 78.152.40.92 (80.94.64.124)
>>> Origin IGP, metric 0, localpref 90, valid, external, best
>>> Community: 1299:30000 5580:21220
>>> Last update: Thu Nov 3 16:09:21 2016
>>>
>>> But even though the BGP02 claims to have it advertised to both the bgp03
>>> and the routeserver / lookingglass:
>>>
>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24
>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24
>>>
>>> It did not.
>>>
>>> Jan Hugo Prins
>>>
>>> On 11/03/2016 05:08 PM, jan hugo prins wrote:
>>>> As a sidenote, a second router that runs my LookingGlass and is used to
>>>> inject blackhole routes, is also not receiving the full table from the
>>>> BGP02.
>>>> This can be observed at http://lg.as48972.net/
>>>>
>>>> Jan Hugo Prins
>>>>
>>>>
>>>>
>>>> On 11/03/2016 04:36 PM, jan hugo prins wrote:
>>>>> Hi everyone,
>>>>>
>>>>> I have a strange issue.
>>>>> I have 2 external BGP routers. They both have 2 uplinks to my transit
>>>>> providers and this works fine.
>>>>> Until a while ago, when I wanted to take one router out for
>>>>> maintenance,
>>>>> I would shut it's transit links, remove "default-information originate
>>>>> always" from OSPF, set higher costs on the uplink and / or set
>>>>> "max-metric router-lsa administrative" wait for most of the
>>>>> traffic to
>>>>> shift to the second router and do my maintenance.
>>>>>
>>>>> Because the router would always have an iBGP link to the second router
>>>>> it would receive all routes from this router and be fine in the
>>>>> case of
>>>>> some traffic coming in anyway. But today I wanted to upgrade Quagga to
>>>>> version 1.10 and I saw something really strange:
>>>>>
>>>>> bgp02.as48972.net# sh ip bgp sum
>>>>> BGP router identifier 95.130.232.3, local AS number 48972
>>>>> RIB entries 1120940, using 120 MiB of memory
>>>>> Peers 29, using 129 KiB of memory
>>>>> Peer groups 3, using 96 bytes of memory
>>>>>
>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>> Up/Down
>>>>> State/P
>>>>> fxRcd
>>>>> 77.222.66.177 4 16243 1434500 11053 0 0 0
>>>>> 01w0d16h 609362
>>>>> 78.152.49.109 4 5580 3162545 66219 0 0 0
>>>>> 01w0d16h 609733
>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>> Idle
>>>>> (Admin)
>>>>> 95.130.232.4 4 48972 10983 58016 0 0 0
>>>>> 02:28:03 0
>>>>> 95.130.232.6 4 48972 2536718 65789 0 0 0
>>>>> 02:21:53 137022
>>>>> 193.108.98.2 4 20828 208151 11056 0 0 0
>>>>> 2d09h49m 137235
>>>>> 193.108.98.3 4 20828 204274 11056 0 0 0
>>>>> 2d09h39m 137227
>>>>>
>>>>> bgp03.as48972.net(config-router)# do sh ip bgp sum
>>>>> BGP router identifier 95.130.232.6, local AS number 48972
>>>>> RIB entries 311015, using 33 MiB of memory
>>>>> Peers 21, using 94 KiB of memory
>>>>> Peer groups 3, using 96 bytes of memory
>>>>>
>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>> Up/Down
>>>>> State/P
>>>>> fxRcd
>>>>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21
>>>>> Idle
>>>>> (Admin)
>>>>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14
>>>>> Idle
>>>>> (Admin)
>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>> Idle
>>>>> (Admin)
>>>>> 95.130.232.3 4 48972 7643 867546 0 0 0
>>>>> 02:21:10 39002
>>>>> 95.130.232.4 4 48972 837 558695 0 0 0
>>>>> 02:27:21 0
>>>>> 193.108.98.2 4 20828 68276 883 0 0 0
>>>>> 03:30:44 137235
>>>>> 193.108.98.3 4 20828 64031 883 0 0 0
>>>>> 03:30:35 137227
>>>>>
>>>>> I would expect that the BGP03 would get at least 580.000 routes
>>>>> from the
>>>>> BGP02, but this doesn't happen for some strange reason.
>>>>> Also resetting the iBGP link between the BGP02 and BGP03 does not fix
>>>>> the problem.
>>>>>
>>>>> Anyone any idea?
>>>>>
>>>>> jan Hugo Prins
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Quagga-users mailing list
>>>>> Quagga-users@lists.quagga.net
>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>
>>>> _______________________________________________
>>>> Quagga-users mailing list
>>>> Quagga-users@lists.quagga.net
>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>
>>> _______________________________________________
>>> Quagga-users mailing list
>>> Quagga-users@lists.quagga.net
>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>
>> _______________________________________________
>> Quagga-users mailing list
>> Quagga-users@lists.quagga.net
>> https://lists.quagga.net/mailman/listinfo/quagga-users
>
>
> _______________________________________________
> Quagga-users mailing list
> Quagga-users@lists.quagga.net
> https://lists.quagga.net/mailman/listinfo/quagga-users
>
>


--
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/

_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

Looks indeed like exactly the same problem.

Jan Hugo Prins


On 11/15/2016 10:40 PM, Mike Tancsa wrote:
> This sounds like the problem I saw last Jan.
>
> https://lists.quagga.net/pipermail/quagga-users/2016-January/014248.html
>
>
>
> On 11/3/2016 8:31 PM, jan hugo prins wrote:
>> I'm happy to provide you with the complete config of my 3 routers.
>> I will send them to you personaly after stripping the passwords.
>>
>> Jan Hugo
>>
>>
>> On 11/04/2016 12:32 AM, Martin Winter wrote:
>>> Jan,
>>>
>>> can you clarify if there are any route-maps, filter-list, prefix-lists
>>> or anything else?
>>> Also, are you doing any “neighbor x.x.x.x next-hop-self” or
>>> peer groups?
>>>
>>> (I’m not sure interested in config which would match, but also filters
>>> etc on the relevant peers in case they are misbehaving)
>>>
>>> My testing on filter-lists etc is a bit lacking, so I would assume the
>>> bug most likely to be there.
>>>
>>> - Martin
>>>
>>>
>>> On 3 Nov 2016, at 9:41, jan hugo prins wrote:
>>>
>>>> Some extra side note:
>>>>
>>>> A random prefix on the BGP02:
>>>>
>>>> bgp02.as48972.net# sh ip bgp 23.226.189.0/24
>>>> BGP routing table entry for 23.226.189.0/24
>>>> Paths: (3 available, best #2, table Default-IP-Routing-Table)
>>>> Advertised to non peer-group peers:
>>>> 95.130.232.4 95.130.232.6
>>>> 5580 5580 1299 4809 38197
>>>> 78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6)
>>>> Origin IGP, metric 0, localpref 90, valid, internal
>>>> Last update: Thu Nov 3 16:09:45 2016
>>>>
>>>> 16243 47886 3356 4809 38197
>>>> 77.222.66.177 from 77.222.66.177 (87.249.109.240)
>>>> Origin IGP, localpref 90, valid, external, best
>>>> Last update: Thu Oct 27 00:56:04 2016
>>>>
>>>> 5580 5580 1299 4809 38197
>>>> 78.152.49.109 from 78.152.49.109 (80.94.64.124)
>>>> Origin IGP, metric 0, localpref 90, valid, external
>>>> Community: 1299:30000 5580:21220
>>>> Last update: Mon Oct 31 17:51:58 2016
>>>>
>>>> Is available on the BGP03:
>>>>
>>>> bgp03.as48972.net# sh ip bgp 23.226.189.0/24
>>>> BGP routing table entry for 23.226.189.0/24
>>>> Paths: (2 available, best #2, table Default-IP-Routing-Table)
>>>> Advertised to non peer-group peers:
>>>> 95.130.232.3 95.130.232.4
>>>> 16243 47886 3356 4809 38197
>>>> 77.222.66.181 from 77.222.66.181 (87.249.109.241)
>>>> Origin IGP, localpref 90, valid, external
>>>> Last update: Thu Nov 3 16:09:38 2016
>>>>
>>>> 5580 5580 1299 4809 38197
>>>> 78.152.40.92 from 78.152.40.92 (80.94.64.124)
>>>> Origin IGP, metric 0, localpref 90, valid, external, best
>>>> Community: 1299:30000 5580:21220
>>>> Last update: Thu Nov 3 16:09:21 2016
>>>>
>>>> But even though the BGP02 claims to have it advertised to both the bgp03
>>>> and the routeserver / lookingglass:
>>>>
>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24
>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24
>>>>
>>>> It did not.
>>>>
>>>> Jan Hugo Prins
>>>>
>>>> On 11/03/2016 05:08 PM, jan hugo prins wrote:
>>>>> As a sidenote, a second router that runs my LookingGlass and is used to
>>>>> inject blackhole routes, is also not receiving the full table from the
>>>>> BGP02.
>>>>> This can be observed at http://lg.as48972.net/
>>>>>
>>>>> Jan Hugo Prins
>>>>>
>>>>>
>>>>>
>>>>> On 11/03/2016 04:36 PM, jan hugo prins wrote:
>>>>>> Hi everyone,
>>>>>>
>>>>>> I have a strange issue.
>>>>>> I have 2 external BGP routers. They both have 2 uplinks to my transit
>>>>>> providers and this works fine.
>>>>>> Until a while ago, when I wanted to take one router out for
>>>>>> maintenance,
>>>>>> I would shut it's transit links, remove "default-information originate
>>>>>> always" from OSPF, set higher costs on the uplink and / or set
>>>>>> "max-metric router-lsa administrative" wait for most of the
>>>>>> traffic to
>>>>>> shift to the second router and do my maintenance.
>>>>>>
>>>>>> Because the router would always have an iBGP link to the second router
>>>>>> it would receive all routes from this router and be fine in the
>>>>>> case of
>>>>>> some traffic coming in anyway. But today I wanted to upgrade Quagga to
>>>>>> version 1.10 and I saw something really strange:
>>>>>>
>>>>>> bgp02.as48972.net# sh ip bgp sum
>>>>>> BGP router identifier 95.130.232.3, local AS number 48972
>>>>>> RIB entries 1120940, using 120 MiB of memory
>>>>>> Peers 29, using 129 KiB of memory
>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>
>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>> Up/Down
>>>>>> State/P
>>>>>> fxRcd
>>>>>> 77.222.66.177 4 16243 1434500 11053 0 0 0
>>>>>> 01w0d16h 609362
>>>>>> 78.152.49.109 4 5580 3162545 66219 0 0 0
>>>>>> 01w0d16h 609733
>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>> Idle
>>>>>> (Admin)
>>>>>> 95.130.232.4 4 48972 10983 58016 0 0 0
>>>>>> 02:28:03 0
>>>>>> 95.130.232.6 4 48972 2536718 65789 0 0 0
>>>>>> 02:21:53 137022
>>>>>> 193.108.98.2 4 20828 208151 11056 0 0 0
>>>>>> 2d09h49m 137235
>>>>>> 193.108.98.3 4 20828 204274 11056 0 0 0
>>>>>> 2d09h39m 137227
>>>>>>
>>>>>> bgp03.as48972.net(config-router)# do sh ip bgp sum
>>>>>> BGP router identifier 95.130.232.6, local AS number 48972
>>>>>> RIB entries 311015, using 33 MiB of memory
>>>>>> Peers 21, using 94 KiB of memory
>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>
>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>> Up/Down
>>>>>> State/P
>>>>>> fxRcd
>>>>>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21
>>>>>> Idle
>>>>>> (Admin)
>>>>>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14
>>>>>> Idle
>>>>>> (Admin)
>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>> Idle
>>>>>> (Admin)
>>>>>> 95.130.232.3 4 48972 7643 867546 0 0 0
>>>>>> 02:21:10 39002
>>>>>> 95.130.232.4 4 48972 837 558695 0 0 0
>>>>>> 02:27:21 0
>>>>>> 193.108.98.2 4 20828 68276 883 0 0 0
>>>>>> 03:30:44 137235
>>>>>> 193.108.98.3 4 20828 64031 883 0 0 0
>>>>>> 03:30:35 137227
>>>>>>
>>>>>> I would expect that the BGP03 would get at least 580.000 routes
>>>>>> from the
>>>>>> BGP02, but this doesn't happen for some strange reason.
>>>>>> Also resetting the iBGP link between the BGP02 and BGP03 does not fix
>>>>>> the problem.
>>>>>>
>>>>>> Anyone any idea?
>>>>>>
>>>>>> jan Hugo Prins
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Quagga-users mailing list
>>>>>> Quagga-users@lists.quagga.net
>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>> _______________________________________________
>>>>> Quagga-users mailing list
>>>>> Quagga-users@lists.quagga.net
>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>> _______________________________________________
>>>> Quagga-users mailing list
>>>> Quagga-users@lists.quagga.net
>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>> _______________________________________________
>>> Quagga-users mailing list
>>> Quagga-users@lists.quagga.net
>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>
>> _______________________________________________
>> Quagga-users mailing list
>> Quagga-users@lists.quagga.net
>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>
>>
>


_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

Hi all

I also saw something very similar in January:

https://lists.quagga.net/pipermail/quagga-users/2016-February/014275.html

I've not managed to reproduce this - in lab testing I couldn't make it
fail, I could break it on production but that isn't the best place to be
troubleshooting and debugging!

It seemed to be related to certain prefixes that just weren't advertised
between routers within our network; but there wasn't any real rhyme or
reason to what these were (comparing attributes etc).

(Apologies if this goes twice to the list, I replied first time from the
wrong address)

Paul.


On 15/11/2016 21:40, Mike Tancsa wrote:
> This sounds like the problem I saw last Jan.
>
> https://lists.quagga.net/pipermail/quagga-users/2016-January/014248.html
>
>
>
> On 11/3/2016 8:31 PM, jan hugo prins wrote:
>> I'm happy to provide you with the complete config of my 3 routers.
>> I will send them to you personaly after stripping the passwords.
>>
>> Jan Hugo
>>
>>
>> On 11/04/2016 12:32 AM, Martin Winter wrote:
>>> Jan,
>>>
>>> can you clarify if there are any route-maps, filter-list, prefix-lists
>>> or anything else?
>>> Also, are you doing any “neighbor x.x.x.x next-hop-self” or
>>> peer groups?
>>>
>>> (I’m not sure interested in config which would match, but also filters
>>> etc on the relevant peers in case they are misbehaving)
>>>
>>> My testing on filter-lists etc is a bit lacking, so I would assume the
>>> bug most likely to be there.
>>>
>>> - Martin
>>>
>>>
>>> On 3 Nov 2016, at 9:41, jan hugo prins wrote:
>>>
>>>> Some extra side note:
>>>>
>>>> A random prefix on the BGP02:
>>>>
>>>> bgp02.as48972.net# sh ip bgp 23.226.189.0/24
>>>> BGP routing table entry for 23.226.189.0/24
>>>> Paths: (3 available, best #2, table Default-IP-Routing-Table)
>>>> Advertised to non peer-group peers:
>>>> 95.130.232.4 95.130.232.6
>>>> 5580 5580 1299 4809 38197
>>>> 78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6)
>>>> Origin IGP, metric 0, localpref 90, valid, internal
>>>> Last update: Thu Nov 3 16:09:45 2016
>>>>
>>>> 16243 47886 3356 4809 38197
>>>> 77.222.66.177 from 77.222.66.177 (87.249.109.240)
>>>> Origin IGP, localpref 90, valid, external, best
>>>> Last update: Thu Oct 27 00:56:04 2016
>>>>
>>>> 5580 5580 1299 4809 38197
>>>> 78.152.49.109 from 78.152.49.109 (80.94.64.124)
>>>> Origin IGP, metric 0, localpref 90, valid, external
>>>> Community: 1299:30000 5580:21220
>>>> Last update: Mon Oct 31 17:51:58 2016
>>>>
>>>> Is available on the BGP03:
>>>>
>>>> bgp03.as48972.net# sh ip bgp 23.226.189.0/24
>>>> BGP routing table entry for 23.226.189.0/24
>>>> Paths: (2 available, best #2, table Default-IP-Routing-Table)
>>>> Advertised to non peer-group peers:
>>>> 95.130.232.3 95.130.232.4
>>>> 16243 47886 3356 4809 38197
>>>> 77.222.66.181 from 77.222.66.181 (87.249.109.241)
>>>> Origin IGP, localpref 90, valid, external
>>>> Last update: Thu Nov 3 16:09:38 2016
>>>>
>>>> 5580 5580 1299 4809 38197
>>>> 78.152.40.92 from 78.152.40.92 (80.94.64.124)
>>>> Origin IGP, metric 0, localpref 90, valid, external, best
>>>> Community: 1299:30000 5580:21220
>>>> Last update: Thu Nov 3 16:09:21 2016
>>>>
>>>> But even though the BGP02 claims to have it advertised to both the bgp03
>>>> and the routeserver / lookingglass:
>>>>
>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24
>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24
>>>>
>>>> It did not.
>>>>
>>>> Jan Hugo Prins
>>>>
>>>> On 11/03/2016 05:08 PM, jan hugo prins wrote:
>>>>> As a sidenote, a second router that runs my LookingGlass and is used to
>>>>> inject blackhole routes, is also not receiving the full table from the
>>>>> BGP02.
>>>>> This can be observed at http://lg.as48972.net/
>>>>>
>>>>> Jan Hugo Prins
>>>>>
>>>>>
>>>>>
>>>>> On 11/03/2016 04:36 PM, jan hugo prins wrote:
>>>>>> Hi everyone,
>>>>>>
>>>>>> I have a strange issue.
>>>>>> I have 2 external BGP routers. They both have 2 uplinks to my transit
>>>>>> providers and this works fine.
>>>>>> Until a while ago, when I wanted to take one router out for
>>>>>> maintenance,
>>>>>> I would shut it's transit links, remove "default-information originate
>>>>>> always" from OSPF, set higher costs on the uplink and / or set
>>>>>> "max-metric router-lsa administrative" wait for most of the
>>>>>> traffic to
>>>>>> shift to the second router and do my maintenance.
>>>>>>
>>>>>> Because the router would always have an iBGP link to the second router
>>>>>> it would receive all routes from this router and be fine in the
>>>>>> case of
>>>>>> some traffic coming in anyway. But today I wanted to upgrade Quagga to
>>>>>> version 1.10 and I saw something really strange:
>>>>>>
>>>>>> bgp02.as48972.net# sh ip bgp sum
>>>>>> BGP router identifier 95.130.232.3, local AS number 48972
>>>>>> RIB entries 1120940, using 120 MiB of memory
>>>>>> Peers 29, using 129 KiB of memory
>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>
>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>> Up/Down
>>>>>> State/P
>>>>>> fxRcd
>>>>>> 77.222.66.177 4 16243 1434500 11053 0 0 0
>>>>>> 01w0d16h 609362
>>>>>> 78.152.49.109 4 5580 3162545 66219 0 0 0
>>>>>> 01w0d16h 609733
>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>> Idle
>>>>>> (Admin)
>>>>>> 95.130.232.4 4 48972 10983 58016 0 0 0
>>>>>> 02:28:03 0
>>>>>> 95.130.232.6 4 48972 2536718 65789 0 0 0
>>>>>> 02:21:53 137022
>>>>>> 193.108.98.2 4 20828 208151 11056 0 0 0
>>>>>> 2d09h49m 137235
>>>>>> 193.108.98.3 4 20828 204274 11056 0 0 0
>>>>>> 2d09h39m 137227
>>>>>>
>>>>>> bgp03.as48972.net(config-router)# do sh ip bgp sum
>>>>>> BGP router identifier 95.130.232.6, local AS number 48972
>>>>>> RIB entries 311015, using 33 MiB of memory
>>>>>> Peers 21, using 94 KiB of memory
>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>
>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>> Up/Down
>>>>>> State/P
>>>>>> fxRcd
>>>>>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21
>>>>>> Idle
>>>>>> (Admin)
>>>>>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14
>>>>>> Idle
>>>>>> (Admin)
>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>> Idle
>>>>>> (Admin)
>>>>>> 95.130.232.3 4 48972 7643 867546 0 0 0
>>>>>> 02:21:10 39002
>>>>>> 95.130.232.4 4 48972 837 558695 0 0 0
>>>>>> 02:27:21 0
>>>>>> 193.108.98.2 4 20828 68276 883 0 0 0
>>>>>> 03:30:44 137235
>>>>>> 193.108.98.3 4 20828 64031 883 0 0 0
>>>>>> 03:30:35 137227
>>>>>>
>>>>>> I would expect that the BGP03 would get at least 580.000 routes
>>>>>> from the
>>>>>> BGP02, but this doesn't happen for some strange reason.
>>>>>> Also resetting the iBGP link between the BGP02 and BGP03 does not fix
>>>>>> the problem.
>>>>>>
>>>>>> Anyone any idea?
>>>>>>
>>>>>> jan Hugo Prins
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Quagga-users mailing list
>>>>>> Quagga-users@lists.quagga.net
>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>> _______________________________________________
>>>>> Quagga-users mailing list
>>>>> Quagga-users@lists.quagga.net
>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>> _______________________________________________
>>>> Quagga-users mailing list
>>>> Quagga-users@lists.quagga.net
>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>> _______________________________________________
>>> Quagga-users mailing list
>>> Quagga-users@lists.quagga.net
>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>
>> _______________________________________________
>> Quagga-users mailing list
>> Quagga-users@lists.quagga.net
>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>
>>
>


_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

The other thing I noticed, is that over time, the problem iBGP peer
sends less and less routes to its peers. So at start up time, it will
advertise (x) prefixes to its iBGP peers. Over the course of a day, it
will go to (x-y) and eventually to almost nothing until I do a soft
clear out and it will return to (x). But (x) is still a number lower
than it should be. I have artificially set the local pref to a higher
value, so it should be passing far more routes to the iBGP peers than
just (x).

---Mike

On 11/16/2016 5:00 AM, Paul Thornton wrote:
> Hi all
>
> I also saw something very similar in January:
>
> https://lists.quagga.net/pipermail/quagga-users/2016-February/014275.html
>
> I've not managed to reproduce this - in lab testing I couldn't make it
> fail, I could break it on production but that isn't the best place to be
> troubleshooting and debugging!
>
> It seemed to be related to certain prefixes that just weren't advertised
> between routers within our network; but there wasn't any real rhyme or
> reason to what these were (comparing attributes etc).
>
> (Apologies if this goes twice to the list, I replied first time from the
> wrong address)
>
> Paul.
>
>
> On 15/11/2016 21:40, Mike Tancsa wrote:
>> This sounds like the problem I saw last Jan.
>>
>> https://lists.quagga.net/pipermail/quagga-users/2016-January/014248.html
>>
>>
>>
>> On 11/3/2016 8:31 PM, jan hugo prins wrote:
>>> I'm happy to provide you with the complete config of my 3 routers.
>>> I will send them to you personaly after stripping the passwords.
>>>
>>> Jan Hugo
>>>
>>>
>>> On 11/04/2016 12:32 AM, Martin Winter wrote:
>>>> Jan,
>>>>
>>>> can you clarify if there are any route-maps, filter-list, prefix-lists
>>>> or anything else?
>>>> Also, are you doing any “neighbor x.x.x.x next-hop-self” or
>>>> peer groups?
>>>>
>>>> (I’m not sure interested in config which would match, but also filters
>>>> etc on the relevant peers in case they are misbehaving)
>>>>
>>>> My testing on filter-lists etc is a bit lacking, so I would assume the
>>>> bug most likely to be there.
>>>>
>>>> - Martin
>>>>
>>>>
>>>> On 3 Nov 2016, at 9:41, jan hugo prins wrote:
>>>>
>>>>> Some extra side note:
>>>>>
>>>>> A random prefix on the BGP02:
>>>>>
>>>>> bgp02.as48972.net# sh ip bgp 23.226.189.0/24
>>>>> BGP routing table entry for 23.226.189.0/24
>>>>> Paths: (3 available, best #2, table Default-IP-Routing-Table)
>>>>> Advertised to non peer-group peers:
>>>>> 95.130.232.4 95.130.232.6
>>>>> 5580 5580 1299 4809 38197
>>>>> 78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6)
>>>>> Origin IGP, metric 0, localpref 90, valid, internal
>>>>> Last update: Thu Nov 3 16:09:45 2016
>>>>>
>>>>> 16243 47886 3356 4809 38197
>>>>> 77.222.66.177 from 77.222.66.177 (87.249.109.240)
>>>>> Origin IGP, localpref 90, valid, external, best
>>>>> Last update: Thu Oct 27 00:56:04 2016
>>>>>
>>>>> 5580 5580 1299 4809 38197
>>>>> 78.152.49.109 from 78.152.49.109 (80.94.64.124)
>>>>> Origin IGP, metric 0, localpref 90, valid, external
>>>>> Community: 1299:30000 5580:21220
>>>>> Last update: Mon Oct 31 17:51:58 2016
>>>>>
>>>>> Is available on the BGP03:
>>>>>
>>>>> bgp03.as48972.net# sh ip bgp 23.226.189.0/24
>>>>> BGP routing table entry for 23.226.189.0/24
>>>>> Paths: (2 available, best #2, table Default-IP-Routing-Table)
>>>>> Advertised to non peer-group peers:
>>>>> 95.130.232.3 95.130.232.4
>>>>> 16243 47886 3356 4809 38197
>>>>> 77.222.66.181 from 77.222.66.181 (87.249.109.241)
>>>>> Origin IGP, localpref 90, valid, external
>>>>> Last update: Thu Nov 3 16:09:38 2016
>>>>>
>>>>> 5580 5580 1299 4809 38197
>>>>> 78.152.40.92 from 78.152.40.92 (80.94.64.124)
>>>>> Origin IGP, metric 0, localpref 90, valid, external, best
>>>>> Community: 1299:30000 5580:21220
>>>>> Last update: Thu Nov 3 16:09:21 2016
>>>>>
>>>>> But even though the BGP02 claims to have it advertised to both the
>>>>> bgp03
>>>>> and the routeserver / lookingglass:
>>>>>
>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24
>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24
>>>>>
>>>>> It did not.
>>>>>
>>>>> Jan Hugo Prins
>>>>>
>>>>> On 11/03/2016 05:08 PM, jan hugo prins wrote:
>>>>>> As a sidenote, a second router that runs my LookingGlass and is
>>>>>> used to
>>>>>> inject blackhole routes, is also not receiving the full table from
>>>>>> the
>>>>>> BGP02.
>>>>>> This can be observed at http://lg.as48972.net/
>>>>>>
>>>>>> Jan Hugo Prins
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 11/03/2016 04:36 PM, jan hugo prins wrote:
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I have a strange issue.
>>>>>>> I have 2 external BGP routers. They both have 2 uplinks to my
>>>>>>> transit
>>>>>>> providers and this works fine.
>>>>>>> Until a while ago, when I wanted to take one router out for
>>>>>>> maintenance,
>>>>>>> I would shut it's transit links, remove "default-information
>>>>>>> originate
>>>>>>> always" from OSPF, set higher costs on the uplink and / or set
>>>>>>> "max-metric router-lsa administrative" wait for most of the
>>>>>>> traffic to
>>>>>>> shift to the second router and do my maintenance.
>>>>>>>
>>>>>>> Because the router would always have an iBGP link to the second
>>>>>>> router
>>>>>>> it would receive all routes from this router and be fine in the
>>>>>>> case of
>>>>>>> some traffic coming in anyway. But today I wanted to upgrade
>>>>>>> Quagga to
>>>>>>> version 1.10 and I saw something really strange:
>>>>>>>
>>>>>>> bgp02.as48972.net# sh ip bgp sum
>>>>>>> BGP router identifier 95.130.232.3, local AS number 48972
>>>>>>> RIB entries 1120940, using 120 MiB of memory
>>>>>>> Peers 29, using 129 KiB of memory
>>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>>
>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>>> Up/Down
>>>>>>> State/P
>>>>>>> fxRcd
>>>>>>> 77.222.66.177 4 16243 1434500 11053 0 0 0
>>>>>>> 01w0d16h 609362
>>>>>>> 78.152.49.109 4 5580 3162545 66219 0 0 0
>>>>>>> 01w0d16h 609733
>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>>> Idle
>>>>>>> (Admin)
>>>>>>> 95.130.232.4 4 48972 10983 58016 0 0 0
>>>>>>> 02:28:03 0
>>>>>>> 95.130.232.6 4 48972 2536718 65789 0 0 0
>>>>>>> 02:21:53 137022
>>>>>>> 193.108.98.2 4 20828 208151 11056 0 0 0
>>>>>>> 2d09h49m 137235
>>>>>>> 193.108.98.3 4 20828 204274 11056 0 0 0
>>>>>>> 2d09h39m 137227
>>>>>>>
>>>>>>> bgp03.as48972.net(config-router)# do sh ip bgp sum
>>>>>>> BGP router identifier 95.130.232.6, local AS number 48972
>>>>>>> RIB entries 311015, using 33 MiB of memory
>>>>>>> Peers 21, using 94 KiB of memory
>>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>>
>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>>> Up/Down
>>>>>>> State/P
>>>>>>> fxRcd
>>>>>>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21
>>>>>>> Idle
>>>>>>> (Admin)
>>>>>>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14
>>>>>>> Idle
>>>>>>> (Admin)
>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>>> Idle
>>>>>>> (Admin)
>>>>>>> 95.130.232.3 4 48972 7643 867546 0 0 0
>>>>>>> 02:21:10 39002
>>>>>>> 95.130.232.4 4 48972 837 558695 0 0 0
>>>>>>> 02:27:21 0
>>>>>>> 193.108.98.2 4 20828 68276 883 0 0 0
>>>>>>> 03:30:44 137235
>>>>>>> 193.108.98.3 4 20828 64031 883 0 0 0
>>>>>>> 03:30:35 137227
>>>>>>>
>>>>>>> I would expect that the BGP03 would get at least 580.000 routes
>>>>>>> from the
>>>>>>> BGP02, but this doesn't happen for some strange reason.
>>>>>>> Also resetting the iBGP link between the BGP02 and BGP03 does not
>>>>>>> fix
>>>>>>> the problem.
>>>>>>>
>>>>>>> Anyone any idea?
>>>>>>>
>>>>>>> jan Hugo Prins
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Quagga-users mailing list
>>>>>>> Quagga-users@lists.quagga.net
>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>>> _______________________________________________
>>>>>> Quagga-users mailing list
>>>>>> Quagga-users@lists.quagga.net
>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>> _______________________________________________
>>>>> Quagga-users mailing list
>>>>> Quagga-users@lists.quagga.net
>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>> _______________________________________________
>>>> Quagga-users mailing list
>>>> Quagga-users@lists.quagga.net
>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>
>>> _______________________________________________
>>> Quagga-users mailing list
>>> Quagga-users@lists.quagga.net
>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>
>>>
>>
>
>
> _______________________________________________
> Quagga-users mailing list
> Quagga-users@lists.quagga.net
> https://lists.quagga.net/mailman/listinfo/quagga-users
>
>


--
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/

_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

xx.yy.zz.118 4 11647 513123 433373 0 0 0 13w1d08h 18134
and then after a soft clear
19910

I dont see this on my other ibgp peers. The problem one has a few dozen
ebgp peers as well as some route maps and filter lists. But then again,
other routers (with fewer ebgp peers) have similar configs. How can I
tell if I am running into memory pressure ?


> sh mem
Temporary memory : 3
String vector : 2
Vector : 2311
Vector index : 2311
Link List : 86
Link Node : 263
Thread : 391
Thread master : 1
Thread stats : 23
VTY : 2
Interface : 10
Connected : 16
Buffer : 3
Buffer data : 1
Stream : 294
Stream data : 294
Stream FIFO : 96
Prefix : 24
Hash : 1352
Hash Bucket : 1133638
Hash Index : 1352
Route table : 50
Route node : 1
Prefix List : 117
Prefix List Entry : 5802
Prefix List Str : 117
Route map : 6
Route map name : 6
Route map index : 9
Route map rule : 19
Route map rule str : 19
Route map compiled : 22
Command desc : 27898
Socket union : 240
Logging : 1
Zclient : 2
Work queue : 65
Work queue name string : 65
Priority queue : 2
Priority queue data : 2
Host config : 3
VRF : 1
VRF name : 1
VRF bit-map : 17
-----------------------------
Nexthop : 14
-----------------------------
BGP instance : 1
BGP listen socket details : 2
BGP peer : 96
BGP peer hostname : 96
Peer group : 5
Peer description : 78
Peer password string : 13
BGP attribute : 629265
BGP extra attributes : 629265
BGP aspath : 200384
BGP aspath seg : 200698
BGP aspath segment data : 200698
BGP aspath str : 200384
-----------------------------
BGP table : 49
BGP node : 1186260
BGP route : 2592777
BGP ancillary route info : 2438838
BGP connected : 6
BGP static : 3
BGP adv attr : 303272
BGP adv : 1654505
BGP synchronise : 1344
BGP adj in : 121250
BGP adj out : 2272503
-----------------------------
BGP AS list : 15
BGP AS filter : 129
BGP AS filter str : 129
-----------------------------
community : 676
community val : 676
community str : 676
-----------------------------
extcommunity : 6
extcommunity val : 6
extcommunity str : 6
-----------------------------
community-list : 2
community-list name : 2
community-list entry : 2
community-list handler : 1
-----------------------------
BGP transit attr : 10
BGP transit val : 10
-----------------------------
BGP nexthop : 14
BGP regexp : 129
BGP own address : 4
-----------------------------

Temporary memory : 18
String vector : 2
Vector : 800
Vector index : 800
Link List : 49
Link Node : 49
Thread : 10
Thread master : 1
Thread stats : 8
VTY : 2
Interface : 17
Connected : 16
Buffer : 3
Buffer data : 1
Stream : 4
Stream data : 4
Prefix : 24
Hash : 1
Hash Bucket : 8
Hash Index : 1
Route table : 26
Route node : 1186369
Command desc : 8948
Logging : 1
Work queue : 2
Work queue name string : 1
Priority queue : 2
Priority queue data : 2
Host config : 2
VRF : 1
VRF name : 1
VRF bit-map : 37
-----------------------------
ZEBRA VRF : 1
Nexthop : 645208
RIB : 645207
Static route : 21
RIB destination : 645201
RIB table info : 4
-----------------------------

On 11/16/2016 9:22 AM, Mike Tancsa wrote:
> The other thing I noticed, is that over time, the problem iBGP peer
> sends less and less routes to its peers. So at start up time, it will
> advertise (x) prefixes to its iBGP peers. Over the course of a day, it
> will go to (x-y) and eventually to almost nothing until I do a soft
> clear out and it will return to (x). But (x) is still a number lower
> than it should be. I have artificially set the local pref to a higher
> value, so it should be passing far more routes to the iBGP peers than
> just (x).
>
> ---Mike
>
> On 11/16/2016 5:00 AM, Paul Thornton wrote:
>> Hi all
>>
>> I also saw something very similar in January:
>>
>> https://lists.quagga.net/pipermail/quagga-users/2016-February/014275.html
>>
>> I've not managed to reproduce this - in lab testing I couldn't make it
>> fail, I could break it on production but that isn't the best place to be
>> troubleshooting and debugging!
>>
>> It seemed to be related to certain prefixes that just weren't advertised
>> between routers within our network; but there wasn't any real rhyme or
>> reason to what these were (comparing attributes etc).
>>
>> (Apologies if this goes twice to the list, I replied first time from the
>> wrong address)
>>
>> Paul.
>>
>>
>> On 15/11/2016 21:40, Mike Tancsa wrote:
>>> This sounds like the problem I saw last Jan.
>>>
>>> https://lists.quagga.net/pipermail/quagga-users/2016-January/014248.html
>>>
>>>
>>>
>>> On 11/3/2016 8:31 PM, jan hugo prins wrote:
>>>> I'm happy to provide you with the complete config of my 3 routers.
>>>> I will send them to you personaly after stripping the passwords.
>>>>
>>>> Jan Hugo
>>>>
>>>>
>>>> On 11/04/2016 12:32 AM, Martin Winter wrote:
>>>>> Jan,
>>>>>
>>>>> can you clarify if there are any route-maps, filter-list, prefix-lists
>>>>> or anything else?
>>>>> Also, are you doing any “neighbor x.x.x.x next-hop-self” or
>>>>> peer groups?
>>>>>
>>>>> (I’m not sure interested in config which would match, but also filters
>>>>> etc on the relevant peers in case they are misbehaving)
>>>>>
>>>>> My testing on filter-lists etc is a bit lacking, so I would assume the
>>>>> bug most likely to be there.
>>>>>
>>>>> - Martin
>>>>>
>>>>>
>>>>> On 3 Nov 2016, at 9:41, jan hugo prins wrote:
>>>>>
>>>>>> Some extra side note:
>>>>>>
>>>>>> A random prefix on the BGP02:
>>>>>>
>>>>>> bgp02.as48972.net# sh ip bgp 23.226.189.0/24
>>>>>> BGP routing table entry for 23.226.189.0/24
>>>>>> Paths: (3 available, best #2, table Default-IP-Routing-Table)
>>>>>> Advertised to non peer-group peers:
>>>>>> 95.130.232.4 95.130.232.6
>>>>>> 5580 5580 1299 4809 38197
>>>>>> 78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6)
>>>>>> Origin IGP, metric 0, localpref 90, valid, internal
>>>>>> Last update: Thu Nov 3 16:09:45 2016
>>>>>>
>>>>>> 16243 47886 3356 4809 38197
>>>>>> 77.222.66.177 from 77.222.66.177 (87.249.109.240)
>>>>>> Origin IGP, localpref 90, valid, external, best
>>>>>> Last update: Thu Oct 27 00:56:04 2016
>>>>>>
>>>>>> 5580 5580 1299 4809 38197
>>>>>> 78.152.49.109 from 78.152.49.109 (80.94.64.124)
>>>>>> Origin IGP, metric 0, localpref 90, valid, external
>>>>>> Community: 1299:30000 5580:21220
>>>>>> Last update: Mon Oct 31 17:51:58 2016
>>>>>>
>>>>>> Is available on the BGP03:
>>>>>>
>>>>>> bgp03.as48972.net# sh ip bgp 23.226.189.0/24
>>>>>> BGP routing table entry for 23.226.189.0/24
>>>>>> Paths: (2 available, best #2, table Default-IP-Routing-Table)
>>>>>> Advertised to non peer-group peers:
>>>>>> 95.130.232.3 95.130.232.4
>>>>>> 16243 47886 3356 4809 38197
>>>>>> 77.222.66.181 from 77.222.66.181 (87.249.109.241)
>>>>>> Origin IGP, localpref 90, valid, external
>>>>>> Last update: Thu Nov 3 16:09:38 2016
>>>>>>
>>>>>> 5580 5580 1299 4809 38197
>>>>>> 78.152.40.92 from 78.152.40.92 (80.94.64.124)
>>>>>> Origin IGP, metric 0, localpref 90, valid, external, best
>>>>>> Community: 1299:30000 5580:21220
>>>>>> Last update: Thu Nov 3 16:09:21 2016
>>>>>>
>>>>>> But even though the BGP02 claims to have it advertised to both the
>>>>>> bgp03
>>>>>> and the routeserver / lookingglass:
>>>>>>
>>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24
>>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24
>>>>>>
>>>>>> It did not.
>>>>>>
>>>>>> Jan Hugo Prins
>>>>>>
>>>>>> On 11/03/2016 05:08 PM, jan hugo prins wrote:
>>>>>>> As a sidenote, a second router that runs my LookingGlass and is
>>>>>>> used to
>>>>>>> inject blackhole routes, is also not receiving the full table from
>>>>>>> the
>>>>>>> BGP02.
>>>>>>> This can be observed at http://lg.as48972.net/
>>>>>>>
>>>>>>> Jan Hugo Prins
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 11/03/2016 04:36 PM, jan hugo prins wrote:
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> I have a strange issue.
>>>>>>>> I have 2 external BGP routers. They both have 2 uplinks to my
>>>>>>>> transit
>>>>>>>> providers and this works fine.
>>>>>>>> Until a while ago, when I wanted to take one router out for
>>>>>>>> maintenance,
>>>>>>>> I would shut it's transit links, remove "default-information
>>>>>>>> originate
>>>>>>>> always" from OSPF, set higher costs on the uplink and / or set
>>>>>>>> "max-metric router-lsa administrative" wait for most of the
>>>>>>>> traffic to
>>>>>>>> shift to the second router and do my maintenance.
>>>>>>>>
>>>>>>>> Because the router would always have an iBGP link to the second
>>>>>>>> router
>>>>>>>> it would receive all routes from this router and be fine in the
>>>>>>>> case of
>>>>>>>> some traffic coming in anyway. But today I wanted to upgrade
>>>>>>>> Quagga to
>>>>>>>> version 1.10 and I saw something really strange:
>>>>>>>>
>>>>>>>> bgp02.as48972.net# sh ip bgp sum
>>>>>>>> BGP router identifier 95.130.232.3, local AS number 48972
>>>>>>>> RIB entries 1120940, using 120 MiB of memory
>>>>>>>> Peers 29, using 129 KiB of memory
>>>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>>>
>>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>>>> Up/Down
>>>>>>>> State/P
>>>>>>>> fxRcd
>>>>>>>> 77.222.66.177 4 16243 1434500 11053 0 0 0
>>>>>>>> 01w0d16h 609362
>>>>>>>> 78.152.49.109 4 5580 3162545 66219 0 0 0
>>>>>>>> 01w0d16h 609733
>>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 95.130.232.4 4 48972 10983 58016 0 0 0
>>>>>>>> 02:28:03 0
>>>>>>>> 95.130.232.6 4 48972 2536718 65789 0 0 0
>>>>>>>> 02:21:53 137022
>>>>>>>> 193.108.98.2 4 20828 208151 11056 0 0 0
>>>>>>>> 2d09h49m 137235
>>>>>>>> 193.108.98.3 4 20828 204274 11056 0 0 0
>>>>>>>> 2d09h39m 137227
>>>>>>>>
>>>>>>>> bgp03.as48972.net(config-router)# do sh ip bgp sum
>>>>>>>> BGP router identifier 95.130.232.6, local AS number 48972
>>>>>>>> RIB entries 311015, using 33 MiB of memory
>>>>>>>> Peers 21, using 94 KiB of memory
>>>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>>>
>>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>>>> Up/Down
>>>>>>>> State/P
>>>>>>>> fxRcd
>>>>>>>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 95.130.232.3 4 48972 7643 867546 0 0 0
>>>>>>>> 02:21:10 39002
>>>>>>>> 95.130.232.4 4 48972 837 558695 0 0 0
>>>>>>>> 02:27:21 0
>>>>>>>> 193.108.98.2 4 20828 68276 883 0 0 0
>>>>>>>> 03:30:44 137235
>>>>>>>> 193.108.98.3 4 20828 64031 883 0 0 0
>>>>>>>> 03:30:35 137227
>>>>>>>>
>>>>>>>> I would expect that the BGP03 would get at least 580.000 routes
>>>>>>>> from the
>>>>>>>> BGP02, but this doesn't happen for some strange reason.
>>>>>>>> Also resetting the iBGP link between the BGP02 and BGP03 does not
>>>>>>>> fix
>>>>>>>> the problem.
>>>>>>>>
>>>>>>>> Anyone any idea?
>>>>>>>>
>>>>>>>> jan Hugo Prins
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Quagga-users mailing list
>>>>>>>> Quagga-users@lists.quagga.net
>>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>>>> _______________________________________________
>>>>>>> Quagga-users mailing list
>>>>>>> Quagga-users@lists.quagga.net
>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>>> _______________________________________________
>>>>>> Quagga-users mailing list
>>>>>> Quagga-users@lists.quagga.net
>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>> _______________________________________________
>>>>> Quagga-users mailing list
>>>>> Quagga-users@lists.quagga.net
>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>
>>>> _______________________________________________
>>>> Quagga-users mailing list
>>>> Quagga-users@lists.quagga.net
>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> Quagga-users mailing list
>> Quagga-users@lists.quagga.net
>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>
>>
>
>


--
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/

_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

I have seen this as well. What I noticed is that initially a set of x
routes is pushed from router A to router B. After that point you would
expect that a route coming is on router A would be pushed as update to
it's iBGP peers. But what I notice is that this doesn't happen (debug
bgp updates out). After the initial connection only updates about
unreachable networks are send to the iBGP peers.

I don't know what happened yesterday evening because it looks like the
iBGP session between my 2 frontend routers has been reset.



But you can see that the number of routes is going down again on the
iBGP session.



This while the number of routes from my upsteams is rather stable:



I'm really curious what kind of bug this is.

Jan Hugo




On 11/16/2016 03:22 PM, Mike Tancsa wrote:
> The other thing I noticed, is that over time, the problem iBGP peer
> sends less and less routes to its peers. So at start up time, it will
> advertise (x) prefixes to its iBGP peers. Over the course of a day, it
> will go to (x-y) and eventually to almost nothing until I do a soft
> clear out and it will return to (x). But (x) is still a number lower
> than it should be. I have artificially set the local pref to a higher
> value, so it should be passing far more routes to the iBGP peers than
> just (x).
>
> ---Mike
>
> On 11/16/2016 5:00 AM, Paul Thornton wrote:
>> Hi all
>>
>> I also saw something very similar in January:
>>
>> https://lists.quagga.net/pipermail/quagga-users/2016-February/014275.html
>>
>> I've not managed to reproduce this - in lab testing I couldn't make it
>> fail, I could break it on production but that isn't the best place to be
>> troubleshooting and debugging!
>>
>> It seemed to be related to certain prefixes that just weren't advertised
>> between routers within our network; but there wasn't any real rhyme or
>> reason to what these were (comparing attributes etc).
>>
>> (Apologies if this goes twice to the list, I replied first time from the
>> wrong address)
>>
>> Paul.
>>
>>
>> On 15/11/2016 21:40, Mike Tancsa wrote:
>>> This sounds like the problem I saw last Jan.
>>>
>>> https://lists.quagga.net/pipermail/quagga-users/2016-January/014248.html
>>>
>>>
>>>
>>> On 11/3/2016 8:31 PM, jan hugo prins wrote:
>>>> I'm happy to provide you with the complete config of my 3 routers.
>>>> I will send them to you personaly after stripping the passwords.
>>>>
>>>> Jan Hugo
>>>>
>>>>
>>>> On 11/04/2016 12:32 AM, Martin Winter wrote:
>>>>> Jan,
>>>>>
>>>>> can you clarify if there are any route-maps, filter-list, prefix-lists
>>>>> or anything else?
>>>>> Also, are you doing any “neighbor x.x.x.x next-hop-self” or
>>>>> peer groups?
>>>>>
>>>>> (I’m not sure interested in config which would match, but also filters
>>>>> etc on the relevant peers in case they are misbehaving)
>>>>>
>>>>> My testing on filter-lists etc is a bit lacking, so I would assume the
>>>>> bug most likely to be there.
>>>>>
>>>>> - Martin
>>>>>
>>>>>
>>>>> On 3 Nov 2016, at 9:41, jan hugo prins wrote:
>>>>>
>>>>>> Some extra side note:
>>>>>>
>>>>>> A random prefix on the BGP02:
>>>>>>
>>>>>> bgp02.as48972.net# sh ip bgp 23.226.189.0/24
>>>>>> BGP routing table entry for 23.226.189.0/24
>>>>>> Paths: (3 available, best #2, table Default-IP-Routing-Table)
>>>>>> Advertised to non peer-group peers:
>>>>>> 95.130.232.4 95.130.232.6
>>>>>> 5580 5580 1299 4809 38197
>>>>>> 78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6)
>>>>>> Origin IGP, metric 0, localpref 90, valid, internal
>>>>>> Last update: Thu Nov 3 16:09:45 2016
>>>>>>
>>>>>> 16243 47886 3356 4809 38197
>>>>>> 77.222.66.177 from 77.222.66.177 (87.249.109.240)
>>>>>> Origin IGP, localpref 90, valid, external, best
>>>>>> Last update: Thu Oct 27 00:56:04 2016
>>>>>>
>>>>>> 5580 5580 1299 4809 38197
>>>>>> 78.152.49.109 from 78.152.49.109 (80.94.64.124)
>>>>>> Origin IGP, metric 0, localpref 90, valid, external
>>>>>> Community: 1299:30000 5580:21220
>>>>>> Last update: Mon Oct 31 17:51:58 2016
>>>>>>
>>>>>> Is available on the BGP03:
>>>>>>
>>>>>> bgp03.as48972.net# sh ip bgp 23.226.189.0/24
>>>>>> BGP routing table entry for 23.226.189.0/24
>>>>>> Paths: (2 available, best #2, table Default-IP-Routing-Table)
>>>>>> Advertised to non peer-group peers:
>>>>>> 95.130.232.3 95.130.232.4
>>>>>> 16243 47886 3356 4809 38197
>>>>>> 77.222.66.181 from 77.222.66.181 (87.249.109.241)
>>>>>> Origin IGP, localpref 90, valid, external
>>>>>> Last update: Thu Nov 3 16:09:38 2016
>>>>>>
>>>>>> 5580 5580 1299 4809 38197
>>>>>> 78.152.40.92 from 78.152.40.92 (80.94.64.124)
>>>>>> Origin IGP, metric 0, localpref 90, valid, external, best
>>>>>> Community: 1299:30000 5580:21220
>>>>>> Last update: Thu Nov 3 16:09:21 2016
>>>>>>
>>>>>> But even though the BGP02 claims to have it advertised to both the
>>>>>> bgp03
>>>>>> and the routeserver / lookingglass:
>>>>>>
>>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24
>>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24
>>>>>>
>>>>>> It did not.
>>>>>>
>>>>>> Jan Hugo Prins
>>>>>>
>>>>>> On 11/03/2016 05:08 PM, jan hugo prins wrote:
>>>>>>> As a sidenote, a second router that runs my LookingGlass and is
>>>>>>> used to
>>>>>>> inject blackhole routes, is also not receiving the full table from
>>>>>>> the
>>>>>>> BGP02.
>>>>>>> This can be observed at http://lg.as48972.net/
>>>>>>>
>>>>>>> Jan Hugo Prins
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 11/03/2016 04:36 PM, jan hugo prins wrote:
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> I have a strange issue.
>>>>>>>> I have 2 external BGP routers. They both have 2 uplinks to my
>>>>>>>> transit
>>>>>>>> providers and this works fine.
>>>>>>>> Until a while ago, when I wanted to take one router out for
>>>>>>>> maintenance,
>>>>>>>> I would shut it's transit links, remove "default-information
>>>>>>>> originate
>>>>>>>> always" from OSPF, set higher costs on the uplink and / or set
>>>>>>>> "max-metric router-lsa administrative" wait for most of the
>>>>>>>> traffic to
>>>>>>>> shift to the second router and do my maintenance.
>>>>>>>>
>>>>>>>> Because the router would always have an iBGP link to the second
>>>>>>>> router
>>>>>>>> it would receive all routes from this router and be fine in the
>>>>>>>> case of
>>>>>>>> some traffic coming in anyway. But today I wanted to upgrade
>>>>>>>> Quagga to
>>>>>>>> version 1.10 and I saw something really strange:
>>>>>>>>
>>>>>>>> bgp02.as48972.net# sh ip bgp sum
>>>>>>>> BGP router identifier 95.130.232.3, local AS number 48972
>>>>>>>> RIB entries 1120940, using 120 MiB of memory
>>>>>>>> Peers 29, using 129 KiB of memory
>>>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>>>
>>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>>>> Up/Down
>>>>>>>> State/P
>>>>>>>> fxRcd
>>>>>>>> 77.222.66.177 4 16243 1434500 11053 0 0 0
>>>>>>>> 01w0d16h 609362
>>>>>>>> 78.152.49.109 4 5580 3162545 66219 0 0 0
>>>>>>>> 01w0d16h 609733
>>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 95.130.232.4 4 48972 10983 58016 0 0 0
>>>>>>>> 02:28:03 0
>>>>>>>> 95.130.232.6 4 48972 2536718 65789 0 0 0
>>>>>>>> 02:21:53 137022
>>>>>>>> 193.108.98.2 4 20828 208151 11056 0 0 0
>>>>>>>> 2d09h49m 137235
>>>>>>>> 193.108.98.3 4 20828 204274 11056 0 0 0
>>>>>>>> 2d09h39m 137227
>>>>>>>>
>>>>>>>> bgp03.as48972.net(config-router)# do sh ip bgp sum
>>>>>>>> BGP router identifier 95.130.232.6, local AS number 48972
>>>>>>>> RIB entries 311015, using 33 MiB of memory
>>>>>>>> Peers 21, using 94 KiB of memory
>>>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>>>
>>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>>>> Up/Down
>>>>>>>> State/P
>>>>>>>> fxRcd
>>>>>>>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>>>> Idle
>>>>>>>> (Admin)
>>>>>>>> 95.130.232.3 4 48972 7643 867546 0 0 0
>>>>>>>> 02:21:10 39002
>>>>>>>> 95.130.232.4 4 48972 837 558695 0 0 0
>>>>>>>> 02:27:21 0
>>>>>>>> 193.108.98.2 4 20828 68276 883 0 0 0
>>>>>>>> 03:30:44 137235
>>>>>>>> 193.108.98.3 4 20828 64031 883 0 0 0
>>>>>>>> 03:30:35 137227
>>>>>>>>
>>>>>>>> I would expect that the BGP03 would get at least 580.000 routes
>>>>>>>> from the
>>>>>>>> BGP02, but this doesn't happen for some strange reason.
>>>>>>>> Also resetting the iBGP link between the BGP02 and BGP03 does not
>>>>>>>> fix
>>>>>>>> the problem.
>>>>>>>>
>>>>>>>> Anyone any idea?
>>>>>>>>
>>>>>>>> jan Hugo Prins
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Quagga-users mailing list
>>>>>>>> Quagga-users@lists.quagga.net
>>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>>>> _______________________________________________
>>>>>>> Quagga-users mailing list
>>>>>>> Quagga-users@lists.quagga.net
>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>>> _______________________________________________
>>>>>> Quagga-users mailing list
>>>>>> Quagga-users@lists.quagga.net
>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>> _______________________________________________
>>>>> Quagga-users mailing list
>>>>> Quagga-users@lists.quagga.net
>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>> _______________________________________________
>>>> Quagga-users mailing list
>>>> Quagga-users@lists.quagga.net
>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>
>>>>
>>
>> _______________________________________________
>> Quagga-users mailing list
>> Quagga-users@lists.quagga.net
>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>
>>
>

I don't think this is a memory issue. At least not system base memory.
My router has 32G of memory, can't imagine that I filled that memory.
I actually have stats that tell me I don't live under any kind of memory
pressure, except when it is something inside the daemon.

Jan Hugo


On 11/16/2016 09:29 PM, Mike Tancsa wrote:
> xx.yy.zz.118 4 11647 513123 433373 0 0 0 13w1d08h 18134
> and then after a soft clear
> 19910
>
> I dont see this on my other ibgp peers. The problem one has a few dozen
> ebgp peers as well as some route maps and filter lists. But then again,
> other routers (with fewer ebgp peers) have similar configs. How can I
> tell if I am running into memory pressure ?
>
>
>> sh mem
> Temporary memory : 3
> String vector : 2
> Vector : 2311
> Vector index : 2311
> Link List : 86
> Link Node : 263
> Thread : 391
> Thread master : 1
> Thread stats : 23
> VTY : 2
> Interface : 10
> Connected : 16
> Buffer : 3
> Buffer data : 1
> Stream : 294
> Stream data : 294
> Stream FIFO : 96
> Prefix : 24
> Hash : 1352
> Hash Bucket : 1133638
> Hash Index : 1352
> Route table : 50
> Route node : 1
> Prefix List : 117
> Prefix List Entry : 5802
> Prefix List Str : 117
> Route map : 6
> Route map name : 6
> Route map index : 9
> Route map rule : 19
> Route map rule str : 19
> Route map compiled : 22
> Command desc : 27898
> Socket union : 240
> Logging : 1
> Zclient : 2
> Work queue : 65
> Work queue name string : 65
> Priority queue : 2
> Priority queue data : 2
> Host config : 3
> VRF : 1
> VRF name : 1
> VRF bit-map : 17
> -----------------------------
> Nexthop : 14
> -----------------------------
> BGP instance : 1
> BGP listen socket details : 2
> BGP peer : 96
> BGP peer hostname : 96
> Peer group : 5
> Peer description : 78
> Peer password string : 13
> BGP attribute : 629265
> BGP extra attributes : 629265
> BGP aspath : 200384
> BGP aspath seg : 200698
> BGP aspath segment data : 200698
> BGP aspath str : 200384
> -----------------------------
> BGP table : 49
> BGP node : 1186260
> BGP route : 2592777
> BGP ancillary route info : 2438838
> BGP connected : 6
> BGP static : 3
> BGP adv attr : 303272
> BGP adv : 1654505
> BGP synchronise : 1344
> BGP adj in : 121250
> BGP adj out : 2272503
> -----------------------------
> BGP AS list : 15
> BGP AS filter : 129
> BGP AS filter str : 129
> -----------------------------
> community : 676
> community val : 676
> community str : 676
> -----------------------------
> extcommunity : 6
> extcommunity val : 6
> extcommunity str : 6
> -----------------------------
> community-list : 2
> community-list name : 2
> community-list entry : 2
> community-list handler : 1
> -----------------------------
> BGP transit attr : 10
> BGP transit val : 10
> -----------------------------
> BGP nexthop : 14
> BGP regexp : 129
> BGP own address : 4
> -----------------------------
>
> Temporary memory : 18
> String vector : 2
> Vector : 800
> Vector index : 800
> Link List : 49
> Link Node : 49
> Thread : 10
> Thread master : 1
> Thread stats : 8
> VTY : 2
> Interface : 17
> Connected : 16
> Buffer : 3
> Buffer data : 1
> Stream : 4
> Stream data : 4
> Prefix : 24
> Hash : 1
> Hash Bucket : 8
> Hash Index : 1
> Route table : 26
> Route node : 1186369
> Command desc : 8948
> Logging : 1
> Work queue : 2
> Work queue name string : 1
> Priority queue : 2
> Priority queue data : 2
> Host config : 2
> VRF : 1
> VRF name : 1
> VRF bit-map : 37
> -----------------------------
> ZEBRA VRF : 1
> Nexthop : 645208
> RIB : 645207
> Static route : 21
> RIB destination : 645201
> RIB table info : 4
> -----------------------------
>
> On 11/16/2016 9:22 AM, Mike Tancsa wrote:
>> The other thing I noticed, is that over time, the problem iBGP peer
>> sends less and less routes to its peers. So at start up time, it will
>> advertise (x) prefixes to its iBGP peers. Over the course of a day, it
>> will go to (x-y) and eventually to almost nothing until I do a soft
>> clear out and it will return to (x). But (x) is still a number lower
>> than it should be. I have artificially set the local pref to a higher
>> value, so it should be passing far more routes to the iBGP peers than
>> just (x).
>>
>> ---Mike
>>
>> On 11/16/2016 5:00 AM, Paul Thornton wrote:
>>> Hi all
>>>
>>> I also saw something very similar in January:
>>>
>>> https://lists.quagga.net/pipermail/quagga-users/2016-February/014275.html
>>>
>>> I've not managed to reproduce this - in lab testing I couldn't make it
>>> fail, I could break it on production but that isn't the best place to be
>>> troubleshooting and debugging!
>>>
>>> It seemed to be related to certain prefixes that just weren't advertised
>>> between routers within our network; but there wasn't any real rhyme or
>>> reason to what these were (comparing attributes etc).
>>>
>>> (Apologies if this goes twice to the list, I replied first time from the
>>> wrong address)
>>>
>>> Paul.
>>>
>>>
>>> On 15/11/2016 21:40, Mike Tancsa wrote:
>>>> This sounds like the problem I saw last Jan.
>>>>
>>>> https://lists.quagga.net/pipermail/quagga-users/2016-January/014248.html
>>>>
>>>>
>>>>
>>>> On 11/3/2016 8:31 PM, jan hugo prins wrote:
>>>>> I'm happy to provide you with the complete config of my 3 routers.
>>>>> I will send them to you personaly after stripping the passwords.
>>>>>
>>>>> Jan Hugo
>>>>>
>>>>>
>>>>> On 11/04/2016 12:32 AM, Martin Winter wrote:
>>>>>> Jan,
>>>>>>
>>>>>> can you clarify if there are any route-maps, filter-list, prefix-lists
>>>>>> or anything else?
>>>>>> Also, are you doing any “neighbor x.x.x.x next-hop-self” or
>>>>>> peer groups?
>>>>>>
>>>>>> (I’m not sure interested in config which would match, but also filters
>>>>>> etc on the relevant peers in case they are misbehaving)
>>>>>>
>>>>>> My testing on filter-lists etc is a bit lacking, so I would assume the
>>>>>> bug most likely to be there.
>>>>>>
>>>>>> - Martin
>>>>>>
>>>>>>
>>>>>> On 3 Nov 2016, at 9:41, jan hugo prins wrote:
>>>>>>
>>>>>>> Some extra side note:
>>>>>>>
>>>>>>> A random prefix on the BGP02:
>>>>>>>
>>>>>>> bgp02.as48972.net# sh ip bgp 23.226.189.0/24
>>>>>>> BGP routing table entry for 23.226.189.0/24
>>>>>>> Paths: (3 available, best #2, table Default-IP-Routing-Table)
>>>>>>> Advertised to non peer-group peers:
>>>>>>> 95.130.232.4 95.130.232.6
>>>>>>> 5580 5580 1299 4809 38197
>>>>>>> 78.152.40.92 (metric 11) from 95.130.232.6 (95.130.232.6)
>>>>>>> Origin IGP, metric 0, localpref 90, valid, internal
>>>>>>> Last update: Thu Nov 3 16:09:45 2016
>>>>>>>
>>>>>>> 16243 47886 3356 4809 38197
>>>>>>> 77.222.66.177 from 77.222.66.177 (87.249.109.240)
>>>>>>> Origin IGP, localpref 90, valid, external, best
>>>>>>> Last update: Thu Oct 27 00:56:04 2016
>>>>>>>
>>>>>>> 5580 5580 1299 4809 38197
>>>>>>> 78.152.49.109 from 78.152.49.109 (80.94.64.124)
>>>>>>> Origin IGP, metric 0, localpref 90, valid, external
>>>>>>> Community: 1299:30000 5580:21220
>>>>>>> Last update: Mon Oct 31 17:51:58 2016
>>>>>>>
>>>>>>> Is available on the BGP03:
>>>>>>>
>>>>>>> bgp03.as48972.net# sh ip bgp 23.226.189.0/24
>>>>>>> BGP routing table entry for 23.226.189.0/24
>>>>>>> Paths: (2 available, best #2, table Default-IP-Routing-Table)
>>>>>>> Advertised to non peer-group peers:
>>>>>>> 95.130.232.3 95.130.232.4
>>>>>>> 16243 47886 3356 4809 38197
>>>>>>> 77.222.66.181 from 77.222.66.181 (87.249.109.241)
>>>>>>> Origin IGP, localpref 90, valid, external
>>>>>>> Last update: Thu Nov 3 16:09:38 2016
>>>>>>>
>>>>>>> 5580 5580 1299 4809 38197
>>>>>>> 78.152.40.92 from 78.152.40.92 (80.94.64.124)
>>>>>>> Origin IGP, metric 0, localpref 90, valid, external, best
>>>>>>> Community: 1299:30000 5580:21220
>>>>>>> Last update: Thu Nov 3 16:09:21 2016
>>>>>>>
>>>>>>> But even though the BGP02 claims to have it advertised to both the
>>>>>>> bgp03
>>>>>>> and the routeserver / lookingglass:
>>>>>>>
>>>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.4 adv |grep 23.226.189.0/24
>>>>>>> [root@bgp02 ~]# show ip bgp nei 95.130.232.6 adv |grep 23.226.189.0/24
>>>>>>>
>>>>>>> It did not.
>>>>>>>
>>>>>>> Jan Hugo Prins
>>>>>>>
>>>>>>> On 11/03/2016 05:08 PM, jan hugo prins wrote:
>>>>>>>> As a sidenote, a second router that runs my LookingGlass and is
>>>>>>>> used to
>>>>>>>> inject blackhole routes, is also not receiving the full table from
>>>>>>>> the
>>>>>>>> BGP02.
>>>>>>>> This can be observed at http://lg.as48972.net/
>>>>>>>>
>>>>>>>> Jan Hugo Prins
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 11/03/2016 04:36 PM, jan hugo prins wrote:
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> I have a strange issue.
>>>>>>>>> I have 2 external BGP routers. They both have 2 uplinks to my
>>>>>>>>> transit
>>>>>>>>> providers and this works fine.
>>>>>>>>> Until a while ago, when I wanted to take one router out for
>>>>>>>>> maintenance,
>>>>>>>>> I would shut it's transit links, remove "default-information
>>>>>>>>> originate
>>>>>>>>> always" from OSPF, set higher costs on the uplink and / or set
>>>>>>>>> "max-metric router-lsa administrative" wait for most of the
>>>>>>>>> traffic to
>>>>>>>>> shift to the second router and do my maintenance.
>>>>>>>>>
>>>>>>>>> Because the router would always have an iBGP link to the second
>>>>>>>>> router
>>>>>>>>> it would receive all routes from this router and be fine in the
>>>>>>>>> case of
>>>>>>>>> some traffic coming in anyway. But today I wanted to upgrade
>>>>>>>>> Quagga to
>>>>>>>>> version 1.10 and I saw something really strange:
>>>>>>>>>
>>>>>>>>> bgp02.as48972.net# sh ip bgp sum
>>>>>>>>> BGP router identifier 95.130.232.3, local AS number 48972
>>>>>>>>> RIB entries 1120940, using 120 MiB of memory
>>>>>>>>> Peers 29, using 129 KiB of memory
>>>>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>>>>
>>>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>>>>> Up/Down
>>>>>>>>> State/P
>>>>>>>>> fxRcd
>>>>>>>>> 77.222.66.177 4 16243 1434500 11053 0 0 0
>>>>>>>>> 01w0d16h 609362
>>>>>>>>> 78.152.49.109 4 5580 3162545 66219 0 0 0
>>>>>>>>> 01w0d16h 609733
>>>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>>>>> Idle
>>>>>>>>> (Admin)
>>>>>>>>> 95.130.232.4 4 48972 10983 58016 0 0 0
>>>>>>>>> 02:28:03 0
>>>>>>>>> 95.130.232.6 4 48972 2536718 65789 0 0 0
>>>>>>>>> 02:21:53 137022
>>>>>>>>> 193.108.98.2 4 20828 208151 11056 0 0 0
>>>>>>>>> 2d09h49m 137235
>>>>>>>>> 193.108.98.3 4 20828 204274 11056 0 0 0
>>>>>>>>> 2d09h39m 137227
>>>>>>>>>
>>>>>>>>> bgp03.as48972.net(config-router)# do sh ip bgp sum
>>>>>>>>> BGP router identifier 95.130.232.6, local AS number 48972
>>>>>>>>> RIB entries 311015, using 33 MiB of memory
>>>>>>>>> Peers 21, using 94 KiB of memory
>>>>>>>>> Peer groups 3, using 96 bytes of memory
>>>>>>>>>
>>>>>>>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
>>>>>>>>> Up/Down
>>>>>>>>> State/P
>>>>>>>>> fxRcd
>>>>>>>>> 77.222.66.181 4 16243 298044 882 0 0 0 00:01:21
>>>>>>>>> Idle
>>>>>>>>> (Admin)
>>>>>>>>> 78.152.40.92 4 5580 559289 5234 0 0 0 00:01:14
>>>>>>>>> Idle
>>>>>>>>> (Admin)
>>>>>>>>> 95.130.232.2 4 48972 0 0 0 0 0 never
>>>>>>>>> Idle
>>>>>>>>> (Admin)
>>>>>>>>> 95.130.232.3 4 48972 7643 867546 0 0 0
>>>>>>>>> 02:21:10 39002
>>>>>>>>> 95.130.232.4 4 48972 837 558695 0 0 0
>>>>>>>>> 02:27:21 0
>>>>>>>>> 193.108.98.2 4 20828 68276 883 0 0 0
>>>>>>>>> 03:30:44 137235
>>>>>>>>> 193.108.98.3 4 20828 64031 883 0 0 0
>>>>>>>>> 03:30:35 137227
>>>>>>>>>
>>>>>>>>> I would expect that the BGP03 would get at least 580.000 routes
>>>>>>>>> from the
>>>>>>>>> BGP02, but this doesn't happen for some strange reason.
>>>>>>>>> Also resetting the iBGP link between the BGP02 and BGP03 does not
>>>>>>>>> fix
>>>>>>>>> the problem.
>>>>>>>>>
>>>>>>>>> Anyone any idea?
>>>>>>>>>
>>>>>>>>> jan Hugo Prins
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Quagga-users mailing list
>>>>>>>>> Quagga-users@lists.quagga.net
>>>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>>>>> _______________________________________________
>>>>>>>> Quagga-users mailing list
>>>>>>>> Quagga-users@lists.quagga.net
>>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>>>> _______________________________________________
>>>>>>> Quagga-users mailing list
>>>>>>> Quagga-users@lists.quagga.net
>>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>>> _______________________________________________
>>>>>> Quagga-users mailing list
>>>>>> Quagga-users@lists.quagga.net
>>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>> _______________________________________________
>>>>> Quagga-users mailing list
>>>>> Quagga-users@lists.quagga.net
>>>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>>>
>>>>>
>>>
>>> _______________________________________________
>>> Quagga-users mailing list
>>> Quagga-users@lists.quagga.net
>>> https://lists.quagga.net/mailman/listinfo/quagga-users
>>>
>>>
>>
>


_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

On 11/16/2016 5:34 PM, jan hugo prins wrote:
> I don't think this is a memory issue. At least not system base memory.
> My router has 32G of memory, can't imagine that I filled that memory.
> I actually have stats that tell me I don't live under any kind of memory
> pressure, except when it is something inside the daemon.

Yeah, I was thinking more some internal allocation too. Thinking back
to when it originally happened, I am wondering if it started when I
added a few new peers. On your box where you see the issue, do you have
many peers configured ?

I also see the problem crop up right away when my external peers go
down. There was a scheduled outage at the IX the other night, so all the
eBGP learned routes were withdrawn. When the eBGP peers came back up
back up, almost none of the routes that were learned from my eBGP peers
were propagated to my iBGP peers. I had to do a soft clear out on the
peer group for my iBGP peers to see the routes, and even then, it really
is just a subset of what should be sent. tcpdump confirmed, they were
never pushed out the physical interface.

---Mike


--
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/

_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

Hi,

I have indeed some peering links on this router (10 I think)
I have found last night and today that the problem is gone when I shut
one specific peering session.
The moment I shut that peering session all updates come in on the other
iBGP peers.
When I keep that session up the iBGP peers don't receive any updates
(except for withdrawls).
When I shut a different eBGP peering session, the result is not the
same. I'm now trying to get in touch with that peering partner to find
out what kind of router they are using and what version of the BGP
software on that router. Debugging logs and PCAP file are send to Martin.

Jan Hugo Prins


On 11/17/2016 04:23 PM, Mike Tancsa wrote:
> On 11/16/2016 5:34 PM, jan hugo prins wrote:
>> I don't think this is a memory issue. At least not system base memory.
>> My router has 32G of memory, can't imagine that I filled that memory.
>> I actually have stats that tell me I don't live under any kind of memory
>> pressure, except when it is something inside the daemon.
> Yeah, I was thinking more some internal allocation too. Thinking back
> to when it originally happened, I am wondering if it started when I
> added a few new peers. On your box where you see the issue, do you have
> many peers configured ?
>
> I also see the problem crop up right away when my external peers go
> down. There was a scheduled outage at the IX the other night, so all the
> eBGP learned routes were withdrawn. When the eBGP peers came back up
> back up, almost none of the routes that were learned from my eBGP peers
> were propagated to my iBGP peers. I had to do a soft clear out on the
> peer group for my iBGP peers to see the routes, and even then, it really
> is just a subset of what should be sent. tcpdump confirmed, they were
> never pushed out the physical interface.
>
> ---Mike
>
>


_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

On 11/17/2016 11:26 AM, jan hugo prins wrote:
> Hi,
>
> I have indeed some peering links on this router (10 I think)
> I have found last night and today that the problem is gone when I shut
> one specific peering session.
> The moment I shut that peering session all updates come in on the other
> iBGP peers.

How strange. Do they have an excessive amount of attributes /
communities to them ? Uniq capabilities advertised ?

---Mike
--
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada http://www.tancsa.com/

_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

I have gone through the pcap and I can't find anything strange.
I'm happy to share the pcap with you if you want to have a look at it.

I have this peering session now down for almost 2 days and everything is
stable.
I really don't know what could be the cause if this strange behavior.

I heard more people are having issues like this or had issues like this.
Could we make a little list to see if we can find common denominators:

LIke this:

Router that stops forwarding to iBGP:
Version: Quagga 0.99.24.1
Number of Transit sessions (Full table): 2
Number of Transit session (Partial table 137.000 routes): 2
Number of peering sessions 11
Number of iBGP sessions: 2
Identified problem session: Remote router Cisco.
Number of incoming routes on peering session: 51
Number of outgoing routes on peering session: 2

I really don't know if anyone else has been able to identify the session
that creates the problem?

Jan Hugo Prins

On 11/17/2016 06:13 PM, Mike Tancsa wrote:
> On 11/17/2016 11:26 AM, jan hugo prins wrote:
>> Hi,
>>
>> I have indeed some peering links on this router (10 I think)
>> I have found last night and today that the problem is gone when I shut
>> one specific peering session.
>> The moment I shut that peering session all updates come in on the other
>> iBGP peers.
> How strange. Do they have an excessive amount of attributes /
> communities to them ? Uniq capabilities advertised ?
>
> ---Mike



_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users