Mailing List Archive: network is unreachable in instance

network is unreachable in instance

Jun 18, 2018, 3:48 AM

Post #1 of 3 (400 views)

I have created question on ask.opestack, but profile is still not approved
by moderators. (https://ask.openstack.org/en/question/114803/network-is-
unreachable-in-instance/) Someone in IRC suggested to send the mail here.

Problem seems to be similar to, but I am not able to follow the bug to fix
the issue - https://ask.openstack.org/en/question/80462/network-is-
unreachable-in-instance/

We have 3 controller nodes connected to 12 compute nodes. Eth3 port on all
the compute node is down.

[Thu Jun 14 14:38:57] root@:~# node-ping-check

Ping check of all service-ip, vm-ip, data-ip and floating-ip, ping
status result after ip, 1/0 = alive/dead

command start time 2018-06-14 14:39:00

node service-ip vm-ip data-ip floating-ip
---------------------------------------------------------------------
h002 10.1.12.102 1 10.2.12.102 1 10.3.13.102 1 10.4.13.102 1
(controller 1)
h003 10.1.12.103 1 10.2.12.103 1 10.3.13.103 1 10.4.13.103 1
(controller 2)
h004 10.1.12.104 1 10.2.12.104 1 10.3.13.104 1 10.4.13.104 1
(controller 3 + Network node)
h005 10.1.12.105 1 10.2.12.105 1 10.3.13.105 1 10.4.13.105 0
h006 10.1.12.106 1 10.2.12.106 1 10.3.13.106 1 10.4.13.106 0
h007 10.1.12.107 1 10.2.12.107 1 10.3.13.107 1 10.4.13.107 0
h008 10.1.12.108 1 10.2.12.108 1 10.3.13.108 1 10.4.13.108 0
h009 10.1.12.109 1 10.2.12.109 1 10.3.13.109 1 10.4.13.109 0
h010 10.1.12.110 1 10.2.12.110 1 10.3.13.110 1 10.4.13.110 0
h011 10.1.14.111 1 10.2.14.111 1 10.3.15.111 1 10.4.15.111 0
h012 10.1.14.112 1 10.2.14.112 1 10.3.15.112 1 10.4.15.112 0
h013 10.1.14.113 1 10.2.14.113 1 10.3.15.113 1 10.4.15.113 0
h014 10.1.14.114 1 10.2.14.114 1 10.3.15.114 1 10.4.15.114 0
h015 10.1.14.115 1 10.2.14.115 1 10.3.15.115 1 10.4.15.115 0
h016 10.1.14.116 0 10.2.14.116 0 10.3.15.116 0 10.4.15.116 0 (
Down because of hardware issue)
h017 10.1.14.117 1 10.2.14.117 1 10.3.15.117 1 10.4.15.117 0

[Thu Jun 14 14:39:39] root@:~#

To bring eth3 on Network node followed following steps:

ifconfig eth3 down
ifconfig eth3 0.0.0.0 up

ovs-vsctl del-br br-ex
ovs-vsctl add-br br-ex
ovs-vsctl add-port br-ex eth3
ifconfig br-ex 10.4.13.104 netmask 255.255.255.0
route add -net 10.4.0.0 netmask 255.255.0.0 gateway 10.4.13.1 dev br-ex

On Network node:

[2018-06-18 11:38:24] root@h004:~$ ip netns exec
qdhcp-eb7573a0-9d91-491c-b150-90f0135778ad ping 12.12.0.4
PING 12.12.0.4 (12.12.0.4) 56(84) bytes of data.
64 bytes from 12.12.0.4: icmp_seq=1 ttl=64 time=1.98 ms
64 bytes from 12.12.0.4: icmp_seq=2 ttl=64 time=0.529 ms
^C
--- 12.12.0.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.529/1.254/1.980/0.726 ms
[2018-06-18 11:38:33] root@h004:~$ ip netns exec
qdhcp-eb7573a0-9d91-491c-b150-90f0135778ad ssh cirros@12.12.0.4
^C
[SSH Not working]

[2018-06-18 11:45:51] root@h004:~$ ip netns exec
qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc
traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
connect: Network is unreachable
[2018-06-18 11:58:01] root@h004:~$ ip netns exec
qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc
traceroute 12.12.0.4
traceroute to 12.12.0.4 (12.12.0.4), 30 hops max, 60 byte packets

On Compute node:
[2018-06-18 11:56:12] root@h011:~$ ip netns exec
qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc
traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
connect: Network is unreachable

[2018-06-18 11:59:20] root@h011:~$ ip netns exec
qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc
ping 12.12.0.4
PING 12.12.0.4 (12.12.0.4) 56(84) bytes of data.
64 bytes from 12.12.0.4: icmp_seq=1 ttl=64 time=1.61 ms
64 bytes from 12.12.0.4: icmp_seq=2 ttl=64 time=0.273 ms
^C
--- 12.12.0.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.273/0.943/1.614/0.671 ms
[2018-06-18 12:02:21] root@h011:~$ ip netns exec
qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc
ping x.x.170.61 (floating ip)
connect: Network is unreachable

--
?
P *SAVE PAPER – Please do not print this e-mail unless absolutely
necessary.*

Re: network is unreachable in instance [ In reply to ]

skaplons at redhat

Jun 18, 2018, 4:28 AM

Post #2 of 3 (399 views)

Permalink

Hi,

> Wiadomo?? napisana przez Jay See <jayachander.it@gmail.com> w dniu 18.06.2018, o godz. 12:48:
>
>
> I have created question on ask.opestack, but profile is still not approved by moderators. (https://ask.openstack.org/en/question/114803/network-is-unreachable-in-instance/) Someone in IRC suggested to send the mail here.
>
>
> Problem seems to be similar to, but I am not able to follow the bug to fix the issue - https://ask.openstack.org/en/question/80462/network-is-unreachable-in-instance/
>
> We have 3 controller nodes connected to 12 compute nodes. Eth3 port on all the compute node is down.
>
> [Thu Jun 14 14:38:57] root@:~# node-ping-check
>
> Ping check of all service-ip, vm-ip, data-ip and floating-ip, ping
> status result after ip, 1/0 = alive/dead
>
> command start time 2018-06-14 14:39:00
>
> node service-ip vm-ip data-ip floating-ip
> ---------------------------------------------------------------------
> h002 10.1.12.102 1 10.2.12.102 1 10.3.13.102 1 10.4.13.102 1 (controller 1)
> h003 10.1.12.103 1 10.2.12.103 1 10.3.13.103 1 10.4.13.103 1 (controller 2)
> h004 10.1.12.104 1 10.2.12.104 1 10.3.13.104 1 10.4.13.104 1 (controller 3 + Network node)
> h005 10.1.12.105 1 10.2.12.105 1 10.3.13.105 1 10.4.13.105 0
> h006 10.1.12.106 1 10.2.12.106 1 10.3.13.106 1 10.4.13.106 0
> h007 10.1.12.107 1 10.2.12.107 1 10.3.13.107 1 10.4.13.107 0
> h008 10.1.12.108 1 10.2.12.108 1 10.3.13.108 1 10.4.13.108 0
> h009 10.1.12.109 1 10.2.12.109 1 10.3.13.109 1 10.4.13.109 0
> h010 10.1.12.110 1 10.2.12.110 1 10.3.13.110 1 10.4.13.110 0
> h011 10.1.14.111 1 10.2.14.111 1 10.3.15.111 1 10.4.15.111 0
> h012 10.1.14.112 1 10.2.14.112 1 10.3.15.112 1 10.4.15.112 0
> h013 10.1.14.113 1 10.2.14.113 1 10.3.15.113 1 10.4.15.113 0
> h014 10.1.14.114 1 10.2.14.114 1 10.3.15.114 1 10.4.15.114 0
> h015 10.1.14.115 1 10.2.14.115 1 10.3.15.115 1 10.4.15.115 0
> h016 10.1.14.116 0 10.2.14.116 0 10.3.15.116 0 10.4.15.116 0 ( Down because of hardware issue)
> h017 10.1.14.117 1 10.2.14.117 1 10.3.15.117 1 10.4.15.117 0
>
> [Thu Jun 14 14:39:39] root@:~#
>
>
> To bring eth3 on Network node followed following steps:
>
> ifconfig eth3 down
> ifconfig eth3 0.0.0.0 up
>
> ovs-vsctl del-br br-ex
> ovs-vsctl add-br br-ex

When You removed and added again br-ex it has no any openflow rules configured. So there will be no traffic passed via this bridge at all please try to restart neutron-openvswitch-agent and check if rules are recreated.
If that will help, You can apply patch https://review.openstack.org/#/c/573696/ and check if that will help :)

> ovs-vsctl add-port br-ex eth3
> ifconfig br-ex 10.4.13.104 netmask 255.255.255.0
> route add -net 10.4.0.0 netmask 255.255.0.0 gateway 10.4.13.1 dev br-ex
>
>
> On Network node:
>
> [2018-06-18 11:38:24] root@h004:~$ ip netns exec qdhcp-eb7573a0-9d91-491c-b150-90f0135778ad ping 12.12.0.4
> PING 12.12.0.4 (12.12.0.4) 56(84) bytes of data.
> 64 bytes from 12.12.0.4: icmp_seq=1 ttl=64 time=1.98 ms
> 64 bytes from 12.12.0.4: icmp_seq=2 ttl=64 time=0.529 ms
> ^C
> --- 12.12.0.4 ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 1001ms
> rtt min/avg/max/mdev = 0.529/1.254/1.980/0.726 ms
> [2018-06-18 11:38:33] root@h004:~$ ip netns exec qdhcp-eb7573a0-9d91-491c-b150-90f0135778ad ssh cirros@12.12.0.4
> ^C
> [SSH Not working]
>
> [2018-06-18 11:45:51] root@h004:~$ ip netns exec qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc traceroute 8.8.8.8
> traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
> connect: Network is unreachable
> [2018-06-18 11:58:01] root@h004:~$ ip netns exec qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc traceroute 12.12.0.4
> traceroute to 12.12.0.4 (12.12.0.4), 30 hops max, 60 byte packets
>
>
> On Compute node:
> [2018-06-18 11:56:12] root@h011:~$ ip netns exec qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc traceroute 8.8.8.8
> traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
> connect: Network is unreachable
>
> [2018-06-18 11:59:20] root@h011:~$ ip netns exec qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc ping 12.12.0.4
> PING 12.12.0.4 (12.12.0.4) 56(84) bytes of data.
> 64 bytes from 12.12.0.4: icmp_seq=1 ttl=64 time=1.61 ms
> 64 bytes from 12.12.0.4: icmp_seq=2 ttl=64 time=0.273 ms
> ^C
> --- 12.12.0.4 ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 1001ms
> rtt min/avg/max/mdev = 0.273/0.943/1.614/0.671 ms
> [2018-06-18 12:02:21] root@h011:~$ ip netns exec qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc ping x.x.170.61 (floating ip)
> connect: Network is unreachable
>
>
>
> --
> ?P SAVE PAPER – Please do not print this e-mail unless absolutely necessary.
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

—
Slawek Kaplonski
Senior software engineer
Red Hat

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: network is unreachable in instance [ In reply to ]

jayachander.it at gmail

Jun 18, 2018, 9:14 AM

Post #3 of 3 (398 views)

Permalink

Hai,

I think , I am not effected by this bug. As we have around 220 vm max. But
as suggested in bug, do I need to set this on all nodes. (1. controller 2.
Network and 3. Compute) or just network node?
As br-ex bridge is been used , I am supposed to use only br-ex while
setting.

ovs-vsctl set bridge <bridge> other-config:mac-table-size=50000

Regarding the MTU link suggested by Slawomir Kaplonski in IRC discussion,
we are using 1500 MTU consistently in all those locations.

PS: I am not expert in openstack system, just started working on existing
system

~ Jayachander.

On Mon, Jun 18, 2018 at 1:28 PM, Slawomir Kaplonski <skaplons@redhat.com>
wrote:

> Hi,
>
> > Wiadomo?? napisana przez Jay See <jayachander.it@gmail.com> w dniu
> 18.06.2018, o godz. 12:48:
> >
> >
> > I have created question on ask.opestack, but profile is still not
> approved by moderators. (https://ask.openstack.org/en/
> question/114803/network-is-unreachable-in-instance/) Someone in IRC
> suggested to send the mail here.
> >
> >
> > Problem seems to be similar to, but I am not able to follow the bug to
> fix the issue - https://ask.openstack.org/en/question/80462/network-is-
> unreachable-in-instance/
> >
> > We have 3 controller nodes connected to 12 compute nodes. Eth3 port on
> all the compute node is down.
> >
> > [Thu Jun 14 14:38:57] root@:~# node-ping-check
> >
> > Ping check of all service-ip, vm-ip, data-ip and floating-ip, ping
> > status result after ip, 1/0 = alive/dead
> >
> > command start time 2018-06-14 14:39:00
> >
> > node service-ip vm-ip data-ip floating-ip
> > ---------------------------------------------------------------------
> > h002 10.1.12.102 1 10.2.12.102 1 10.3.13.102 1 10.4.13.102 1
> (controller 1)
> > h003 10.1.12.103 1 10.2.12.103 1 10.3.13.103 1 10.4.13.103 1
> (controller 2)
> > h004 10.1.12.104 1 10.2.12.104 1 10.3.13.104 1 10.4.13.104 1
> (controller 3 + Network node)
> > h005 10.1.12.105 1 10.2.12.105 1 10.3.13.105 1 10.4.13.105 0
> > h006 10.1.12.106 1 10.2.12.106 1 10.3.13.106 1 10.4.13.106 0
> > h007 10.1.12.107 1 10.2.12.107 1 10.3.13.107 1 10.4.13.107 0
> > h008 10.1.12.108 1 10.2.12.108 1 10.3.13.108 1 10.4.13.108 0
> > h009 10.1.12.109 1 10.2.12.109 1 10.3.13.109 1 10.4.13.109 0
> > h010 10.1.12.110 1 10.2.12.110 1 10.3.13.110 1 10.4.13.110 0
> > h011 10.1.14.111 1 10.2.14.111 1 10.3.15.111 1 10.4.15.111 0
> > h012 10.1.14.112 1 10.2.14.112 1 10.3.15.112 1 10.4.15.112 0
> > h013 10.1.14.113 1 10.2.14.113 1 10.3.15.113 1 10.4.15.113 0
> > h014 10.1.14.114 1 10.2.14.114 1 10.3.15.114 1 10.4.15.114 0
> > h015 10.1.14.115 1 10.2.14.115 1 10.3.15.115 1 10.4.15.115 0
> > h016 10.1.14.116 0 10.2.14.116 0 10.3.15.116 0 10.4.15.116 0 (
> Down because of hardware issue)
> > h017 10.1.14.117 1 10.2.14.117 1 10.3.15.117 1 10.4.15.117 0
> >
> > [Thu Jun 14 14:39:39] root@:~#
> >
> >
> > To bring eth3 on Network node followed following steps:
> >
> > ifconfig eth3 down
> > ifconfig eth3 0.0.0.0 up
> >
> > ovs-vsctl del-br br-ex
> > ovs-vsctl add-br br-ex
>
> When You removed and added again br-ex it has no any openflow rules
> configured. So there will be no traffic passed via this bridge at all
> please try to restart neutron-openvswitch-agent and check if rules are
> recreated.
> If that will help, You can apply patch https://review.openstack.org/#
> /c/573696/ and check if that will help :)
>
> > ovs-vsctl add-port br-ex eth3
> > ifconfig br-ex 10.4.13.104 netmask 255.255.255.0
> > route add -net 10.4.0.0 netmask 255.255.0.0 gateway 10.4.13.1 dev br-ex
> >
> >
> > On Network node:
> >
> > [2018-06-18 11:38:24] root@h004:~$ ip netns exec
> qdhcp-eb7573a0-9d91-491c-b150-90f0135778ad ping 12.12.0.4
> > PING 12.12.0.4 (12.12.0.4) 56(84) bytes of data.
> > 64 bytes from 12.12.0.4: icmp_seq=1 ttl=64 time=1.98 ms
> > 64 bytes from 12.12.0.4: icmp_seq=2 ttl=64 time=0.529 ms
> > ^C
> > --- 12.12.0.4 ping statistics ---
> > 2 packets transmitted, 2 received, 0% packet loss, time 1001ms
> > rtt min/avg/max/mdev = 0.529/1.254/1.980/0.726 ms
> > [2018-06-18 11:38:33] root@h004:~$ ip netns exec
> qdhcp-eb7573a0-9d91-491c-b150-90f0135778ad ssh cirros@12.12.0.4
> > ^C
> > [SSH Not working]
> >
> > [2018-06-18 11:45:51] root@h004:~$ ip netns exec
> qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc traceroute 8.8.8.8
> > traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
> > connect: Network is unreachable
> > [2018-06-18 11:58:01] root@h004:~$ ip netns exec
> qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc traceroute 12.12.0.4
> > traceroute to 12.12.0.4 (12.12.0.4), 30 hops max, 60 byte packets
> >
> >
> > On Compute node:
> > [2018-06-18 11:56:12] root@h011:~$ ip netns exec
> qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc traceroute 8.8.8.8
> > traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
> > connect: Network is unreachable
> >
> > [2018-06-18 11:59:20] root@h011:~$ ip netns exec
> qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc ping 12.12.0.4
> > PING 12.12.0.4 (12.12.0.4) 56(84) bytes of data.
> > 64 bytes from 12.12.0.4: icmp_seq=1 ttl=64 time=1.61 ms
> > 64 bytes from 12.12.0.4: icmp_seq=2 ttl=64 time=0.273 ms
> > ^C
> > --- 12.12.0.4 ping statistics ---
> > 2 packets transmitted, 2 received, 0% packet loss, time 1001ms
> > rtt min/avg/max/mdev = 0.273/0.943/1.614/0.671 ms
> > [2018-06-18 12:02:21] root@h011:~$ ip netns exec
> qrouter-d770b552-1979-4063-ae79-c6ff402c8cbc ping x.x.170.61 (floating
> ip)
> > connect: Network is unreachable
> >
> >
> >
> > --
> > ?P SAVE PAPER – Please do not print this e-mail unless absolutely
> necessary.
> > _______________________________________________
> > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> > Post to : openstack@lists.openstack.org
> > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
> —
> Slawek Kaplonski
> Senior software engineer
> Red Hat
>
>

--
?
P *SAVE PAPER – Please do not print this e-mail unless absolutely
necessary.*