Mailing List Archive

I'm having the same problem when I delete the DHCP port openstack recreates the port and everything works fine again ...

Marcio Prado
Analista de TI - Infraestrutura e Redes
Fone: +55 (35) 9.9821-3561
www.marcioprado.eti.br
Em 29/10/2018 17:50, Torin Woltjer <torin.woltjer@granddial.com> escreveu:

Recently installed a new compute node, but noticed none of the instances that I put on it will successfully receive network addresses from DHCP. This seems to work on all other compute nodes however. When listening for DHCP requests on the vxlan of the compute node, I notice that while I can see the DHCP requests on the new compute node, I do not see them anywhere else. If I manually assign an address to the interface on the instance I am able to ping in and out. Running dhclient -v on an instance on a working compute node successfully gets a DHCP response, on the new compute node there is no response, I also discovered that the instance on the new compute node cannot ping the DHCP ports at 172.16.1.2 & 172.16.1.3 yet can ping the gateway at 172.16.1.1.
The setup is neutron-linuxbridge on Openstack Queens.
Torin Woltjer Grand Dial Communications - A ZK Tech Inc. Company 616.776.1066 ext. 2006http://www.granddial.com"]http://www.granddial.com"]www.granddial.com

I deleted both DHCP ports and they recreated as you said. However, instances are still unable to get network addresses automatically.

Torin Woltjer

Grand Dial Communications - A ZK Tech Inc. Company

616.776.1066 ext. 2006
www.granddial.com

----------------------------------------
From: Marcio Prado <marcioprado@marcioprado.eti.br>
Sent: 10/29/18 6:23 PM
To: torin.woltjer@granddial.com
Subject: Re: [Openstack] DHCP not accessible on new compute node.
The door is recreated automatically. The problem like I said is not in
DHCP, but for some reason, erasing and waiting for OpenStack to
re-create the port often solves the problem.

Please, if you can find out the problem in fact, let me know. I'm very
interested to know.

You can delete the door without fear. OpenStack will recreate in a short
time.

Interestingly, I created a brand new selfservice network and DHCP doesn't work on that either. I've followed the instructions in the minimal setup (excluding the controllers as they're already set up) but the new node has no access to the DHCP agent in neutron it seems. Is there a likely component that I've overlooked?

Torin Woltjer

Grand Dial Communications - A ZK Tech Inc. Company

616.776.1066 ext. 2006
www.granddial.com

----------------------------------------
From: "Torin Woltjer" <torin.woltjer@granddial.com>
Sent: 10/30/18 10:48 AM
To: <marcioprado@marcioprado.eti.br>, "openstack@lists.openstack.org" <openstack@lists.openstack.org>
Subject: Re: [Openstack] DHCP not accessible on new compute node.
I deleted both DHCP ports and they recreated as you said. However, instances are still unable to get network addresses automatically.

Torin Woltjer

Grand Dial Communications - A ZK Tech Inc. Company

616.776.1066 ext. 2006
www.granddial.com

----------------------------------------
From: Marcio Prado <marcioprado@marcioprado.eti.br>
Sent: 10/29/18 6:23 PM
To: torin.woltjer@granddial.com
Subject: Re: [Openstack] DHCP not accessible on new compute node.
The door is recreated automatically. The problem like I said is not in
DHCP, but for some reason, erasing and waiting for OpenStack to
re-create the port often solves the problem.

Please, if you can find out the problem in fact, let me know. I'm very
interested to know.

You can delete the door without fear. OpenStack will recreate in a short
time.

I believe you have not forgotten anything. This should probably be bug
...

As my cloud is not production, but rather masters research. I migrate
the VM live to a node that is working, restart it, after that I migrate
back to the original node that was not working and it keeps running ...


Em 30-10-2018 17:50, Torin Woltjer escreveu:
> Interestingly, I created a brand new selfservice network and DHCP
> doesn't work on that either. I've followed the instructions in the
> minimal setup (excluding the controllers as they're already set up)
> but the new node has no access to the DHCP agent in neutron it seems.
> Is there a likely component that I've overlooked?
>
> _TORIN WOLTJER_
>
> GRAND DIAL COMMUNICATIONS - A ZK TECH INC. COMPANY
>
> 616.776.1066 EXT. 2006
> _WWW.GRANDDIAL.COM [1]_
>
> -------------------------
>
> FROM: "Torin Woltjer" <torin.woltjer@granddial.com>
> SENT: 10/30/18 10:48 AM
> TO: <marcioprado@marcioprado.eti.br>, "openstack@lists.openstack.org"
> <openstack@lists.openstack.org>
> SUBJECT: Re: [Openstack] DHCP not accessible on new compute node.
>
> I deleted both DHCP ports and they recreated as you said. However,
> instances are still unable to get network addresses automatically.
>
> _TORIN WOLTJER_
>
> GRAND DIAL COMMUNICATIONS - A ZK TECH INC. COMPANY
>
> 616.776.1066 EXT. 2006
> _ [1] [1]WWW.GRANDDIAL.COM [1]_
>
> -------------------------
>
> FROM: Marcio Prado <marcioprado@marcioprado.eti.br>
> SENT: 10/29/18 6:23 PM
> TO: torin.woltjer@granddial.com
> SUBJECT: Re: [Openstack] DHCP not accessible on new compute node.
> The door is recreated automatically. The problem like I said is not in
> DHCP, but for some reason, erasing and waiting for OpenStack to
> re-create the port often solves the problem.
>
> Please, if you can find out the problem in fact, let me know. I'm very
> interested to know.
>
> You can delete the door without fear. OpenStack will recreate in a
> short
> time.
>
> Links:
> ------
> [1] http://www.granddial.com

--
Marcio Prado
Analista de TI - Infraestrutura e Redes
Fone: (35) 9.9821-3561
www.marcioprado.eti.br

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

I've completely wiped the node and reinstalled it, and the problem still persists. I can't ping instances on other compute nodes, or ping the DHCP ports. Instances don't get addresses or metadata when started on this node.

From: Marcio Prado <marcioprado@marcioprado.eti.br>
Sent: 11/1/18 9:51 AM
To: torin.woltjer@granddial.com
Cc: openstack@lists.openstack.org
Subject: Re: [Openstack] DHCP not accessible on new compute node.
I believe you have not forgotten anything. This should probably be bug
...

As my cloud is not production, but rather masters research. I migrate
the VM live to a node that is working, restart it, after that I migrate
back to the original node that was not working and it keeps running ...

Em 30-10-2018 17:50, Torin Woltjer escreveu:
> Interestingly, I created a brand new selfservice network and DHCP
> doesn't work on that either. I've followed the instructions in the
> minimal setup (excluding the controllers as they're already set up)
> but the new node has no access to the DHCP agent in neutron it seems.
> Is there a likely component that I've overlooked?
>
> _TORIN WOLTJER_
>
> GRAND DIAL COMMUNICATIONS - A ZK TECH INC. COMPANY
>
> 616.776.1066 EXT. 2006
> _WWW.GRANDDIAL.COM [1]_
>
> -------------------------
>
> FROM: "Torin Woltjer"
> SENT: 10/30/18 10:48 AM
> TO: , "openstack@lists.openstack.org"
>
> SUBJECT: Re: [Openstack] DHCP not accessible on new compute node.
>
> I deleted both DHCP ports and they recreated as you said. However,
> instances are still unable to get network addresses automatically.
>
> _TORIN WOLTJER_
>
> GRAND DIAL COMMUNICATIONS - A ZK TECH INC. COMPANY
>
> 616.776.1066 EXT. 2006
> _ [1] [1]WWW.GRANDDIAL.COM [1]_
>
> -------------------------
>
> FROM: Marcio Prado
> SENT: 10/29/18 6:23 PM
> TO: torin.woltjer@granddial.com
> SUBJECT: Re: [Openstack] DHCP not accessible on new compute node.
> The door is recreated automatically. The problem like I said is not in
> DHCP, but for some reason, erasing and waiting for OpenStack to
> re-create the port often solves the problem.
>
> Please, if you can find out the problem in fact, let me know. I'm very
> interested to know.
>
> You can delete the door without fear. OpenStack will recreate in a
> short
> time.
>
> Links:
> ------
> [1] http://www.granddial.com

--
Marcio Prado
Analista de TI - Infraestrutura e Redes
Fone: (35) 9.9821-3561
www.marcioprado.eti.br

So I did further ping tests and explored differences between my working compute nodes and my non-working compute node. Firstly, it seems that the VXLAN is working between the nonworking compute node and controller nodes. After manually setting IP addresses, I can ping from an instance on the non working node to 172.16.1.1 (neutron gateway); when running tcpdump I can see icmp on:
-compute's bridge interface
-compute's vxlan interface
-controller's vxlan interface
-controller's bridge interface
-controller's qrouter namespace

This behavior is expected and is the same for instances on the working compute nodes. However if I try to ping 172.16.1.2 (neutron dhcp) from an instance on the nonworking compute node, pings do not flow. If I use tcpdump to listen for pings I cannot hear any, even listening on the compute node itself; this includes listening on the vxlan, bridge, and the tap device directly. Once I try to ping in reverse, from the dhcp netns on the controller to the instance on the non-working compute node, pings begin to flow. The same is true for pings between the instance on the nonworking compute and an instance on the working compute. Pings do not flow, until the working instance pings. Once pings are flowing between the nonworking instance and neutron DHCP; I run dhclient on the instance and start listening for DHCP requests with tcpdump, and I hear them on:
-compute's bridge interface
-compute's vxlan interface
They don't make it to the controller node.

I've re-enabled l2-population on the controller's and rebooted them just in case, but the problem persists. A diff of /etc/ on all compute nodes shows that all openstack and networking related configuration is effectively identical. The last difference between the non-working compute node and the working compute nodes as far as I can tell, is that the new node has a different network card. The working nodes use "Broadcom Limited NetXtreme II BCM57712 10 Gigabit Ethernet" and the nonworking node uses a "NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter".

Are there any known issues with neutron and this brand of network adapter? I looked at the capabilities on both adapters and here are the differences:

Broadcom NetXen
tx-tcp-ecn-segmentation: on tx-tcp-ecn-segmentation: off [fixed]
rx-vlan-offload: on [fixed] rx-vlan-offload: off [fixed]
receive-hashing: on receive-hashing: off [fixed]
rx-vlan-filter: on rx-vlan-filter: off [fixed]
tx-gre-segmentation: on tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: on tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: on tx-ipxip4-segmentation: off [fixed]
tx-udp_tnl-segmentation: on tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: on tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: on tx-gso-partial: off [fixed]
loopback: off loopback: off [fixed]
rx-udp_tunnel-port-offload: on rx-udp_tunnel-port-offload: off [fixed]

I've just done this and the problem is still there.

Torin Woltjer

Grand Dial Communications - A ZK Tech Inc. Company

616.776.1066 ext. 2006
www.granddial.com

----------------------------------------
From: Marcio Prado <marcioprado@marcioprado.eti.br>
Sent: 11/2/18 5:08 PM
To: torin.woltjer@granddial.com
Subject: Re: [Openstack] DHCP not accessible on new compute node.
Clone the hd of a server and restore to what is not working.

then only change the required settings ... ip, hostname, etc.

Marcio Prado
Analista de TI - Infraestrutura e Redes
Fone: (35) 9.9821-3561
www.marcioprado.eti.br

Em 02/11/2018 16:27, Torin Woltjer <torin.woltjer@granddial.com> escreveu:I've completely wiped the node and reinstalled it, and the problem still persists. I can't ping instances on other compute nodes, or ping the DHCP ports. Instances don't get addresses or metadata when started on this node.

From: Marcio Prado <marcioprado@marcioprado.eti.br>
Sent: 11/1/18 9:51 AM
To: torin.woltjer@granddial.com
Cc: openstack@lists.openstack.org
Subject: Re: [Openstack] DHCP not accessible on new compute node.
I believe you have not forgotten anything. This should probably be bug
...

As my cloud is not production, but rather masters research. I migrate
the VM live to a node that is working, restart it, after that I migrate
back to the original node that was not working and it keeps running ...

Em 30-10-2018 17:50, Torin Woltjer escreveu:
> Interestingly, I created a brand new selfservice network and DHCP
> doesn't work on that either. I've followed the instructions in the
> minimal setup (excluding the controllers as they're already set up)
> but the new node has no access to the DHCP agent in neutron it seems.
> Is there a likely component that I've overlooked?
>
> _TORIN WOLTJER_
>
> GRAND DIAL COMMUNICATIONS - A ZK TECH INC. COMPANY
>
> 616.776.1066 EXT. 2006
> _WWW.GRANDDIAL.COM [1]_
>
> -------------------------
>
> FROM: "Torin Woltjer"
> SENT: 10/30/18 10:48 AM
> TO: , "openstack@lists.openstack.org"
>
> SUBJECT: Re: [Openstack] DHCP not accessible on new compute node.
>
> I deleted both DHCP ports and they recreated as you said. However,
> instances are still unable to get network addresses automatically.
>
> _TORIN WOLTJER_
>
> GRAND DIAL COMMUNICATIONS - A ZK TECH INC. COMPANY
>
> 616.776.1066 EXT. 2006
> _ [1] [1]WWW.GRANDDIAL.COM [1]_
>
> -------------------------
>
> FROM: Marcio Prado
> SENT: 10/29/18 6:23 PM
> TO: torin.woltjer@granddial.com
> SUBJECT: Re: [Openstack] DHCP not accessible on new compute node.
> The door is recreated automatically. The problem like I said is not in
> DHCP, but for some reason, erasing and waiting for OpenStack to
> re-create the port often solves the problem.
>
> Please, if you can find out the problem in fact, let me know. I'm very
> interested to know.
>
> You can delete the door without fear. OpenStack will recreate in a
> short
> time.
>
> Links:
> ------
> [1] http://www.granddial.com

--
Marcio Prado
Analista de TI - Infraestrutura e Redes
Fone: (35) 9.9821-3561
www.marcioprado.eti.br

Wow! It seems meaningless, but changing the exchange door.

Marcio Prado
Analista de TI - Infraestrutura e Redes
Fone: (35) 9.9821-3561
www.marcioprado.eti.br
Em 15/11/2018 13:33, Torin Woltjer <torin.woltjer@granddial.com> escreveu:

I&#39;ve just done this and the problem is still there.
Torin Woltjer Grand Dial Communications - A ZK Tech Inc. Company 616.776.1066 ext. 2006http://www.granddial.com"]http://www.granddial.com"]www.granddial.com

---

From: Marcio Prado <marcioprado&#64;marcioprado.eti.br>
Sent: 11/2/18 5:08 PM
To: torin.woltjer&#64;granddial.com
Subject: Re: [Openstack] DHCP not accessible on new compute node.Clone the hd of a server and restore to what is not working.
then only change the required settings ... ip, hostname, etc.
Marcio Prado
Analista de TI - Infraestrutura e Redes
Fone: (35) 9.9821-3561
http://www.marcioprado.eti.br"]http://www.marcioprado.eti.br"]www.marcioprado.eti.br
Em 02/11/2018 16:27, Torin Woltjer <torin.woltjer&#64;granddial.com> escreveu:

I&#39;ve completely wiped the node and reinstalled it, and the problem still persists. I can&#39;t ping instances on other compute nodes, or ping the DHCP ports. Instances don&#39;t get addresses or metadata when started on this node.
From: Marcio Prado <marcioprado&#64;marcioprado.eti.br>
Sent: 11/1/18 9:51 AM
To: torin.woltjer&#64;granddial.com
Cc: openstack&#64;lists.openstack.org
Subject: Re: [Openstack] DHCP not accessible on new compute node.I believe you have not forgotten anything. This should probably be bug...
As my cloud is not production, but rather masters research. I migratethe VM live to a node that is working, restart it, after that I migrateback to the original node that was not working and it keeps running ...

Em 30-10-2018 17:50, Torin Woltjer escreveu:> Interestingly, I created a brand new selfservice network and DHCP> doesn&#39;t work on that either. I&#39;ve followed the instructions in the> minimal setup (excluding the controllers as they&#39;re already set up)> but the new node has no access to the DHCP agent in neutron it seems.> Is there a likely component that I&#39;ve overlooked?>> _TORIN WOLTJER_>> GRAND DIAL COMMUNICATIONS - A ZK TECH INC. COMPANY>> 616.776.1066 EXT. 2006> _WWW.GRANDDIAL.COM [1]_>> ------------------------->> FROM: &#34;Torin Woltjer&#34;> SENT: 10/30/18 10:48 AM> TO: , &#34;openstack&#64;lists.openstack.org&#34;>> SUBJECT: Re: [Openstack] DHCP not accessible on new compute node.>> I deleted both DHCP ports and they recreated as you said. However,> instances are still unable to get network addresses automatically.>> _TORIN WOLTJER_>> GRAND DIAL COMMUNICATIONS - A ZK TECH INC. COMPANY>> 616.776.1066 EXT. 2006> _ [1] [1]WWW.GRANDDIAL.COM [1]_>> ------------------------->> FROM: Marcio Prado> SENT: 10/29/18 6:23 PM> TO: torin.woltjer&#64;granddial.com> SUBJECT: Re: [Openstack] DHCP not accessible on new compute node.> The door is recreated automatically. The problem like I said is not in> DHCP, but for some reason, erasing and waiting for OpenStack to> re-create the port often solves the problem.>> Please, if you can find out the problem in fact, let me know. I&#39;m very> interested to know.>> You can delete the door without fear. OpenStack will recreate in a> short> time.>> Links:> ------> [1] http://www.granddial.com"]http://www.granddial.com"]http://www.granddial.com"]http://www.granddial.com"]http://www.granddial.com
--Marcio PradoAnalista de TI - Infraestrutura e RedesFone: (35) 9.9821-3561http://www.marcioprado.eti.br"]http://www.marcioprado.eti.br"]http://www.marcioprado.eti.br"]http://www.marcioprado.eti.br"]www.marcioprado.eti.br