Mailing List Archive: [Pike][Neutron] L3 metering with DVR doesn't work

[Pike][Neutron] L3 metering with DVR doesn't work

Oct 25, 2018, 5:06 AM

Post #1 of 3 (1339 views)

Hello,

I'm trying to set up metering for neutron in Pike. I tested it with a
centralized router and it works, but when I try with a distributed router it
doesn't record any usage samples. I have one compute node and one
network node
and I've created an instance with a floating ip.

openstack router show public-router2
+-------------------------+----------------------------------------------------+
| Field                   |
Value                                              |
+-------------------------+----------------------------------------------------+
| admin_state_up          |
UP                                                 |
| availability_zone_hints
|                                                    |
| availability_zones      |
nova                                               |
| created_at              |
2018-10-05T12:07:32Z                               |
| description |                                                    |
| distributed             |
True                                               |
| external_gateway_info   | {"network_id":
"b96473ce-                          |
|                         | 94f6-464f-a703-5285fb8ff3d3", "enable_snat":
true, |
|                         | "external_fixed_ips":
[{"subnet_id":               |
|                         |
"6c08c3d9-7df1-4bec-b847-19f80b9d1764",            |
|                         | "ip_address":
"192.168.252.102"}]}                 |
| flavor_id               |
None                                               |
| ha                      |
False                                              |
| id                      |
37c1794b-58d1-4d0d-b34b-944ca411b86b               |
| name                    |
public-router2                                     |
| project_id              |
fe203109e67f4e39b066c9529f9fc35d                   |
| revision_number         |
5                                                  |
| routes |                                                    |
| status                  |
ACTIVE                                             |
| tags |                                                    |
| updated_at              |
2018-10-05T12:09:36Z                               |
+-------------------------+----------------------------------------------------+

openstack network agent list
+-----------+------------+-----------+-------------------+-------+-------+--------------+
| ID        | Agent Type | Host      | Availability Zone | Alive | State
| Binary       |
+-----------+------------+-----------+-------------------+-------+-------+--------------+
| 14b9ea75- | L3 agent   | compute1. | nova | :-)   | UP    | neutron-l3-a |
| 1dc1-4e37 |            | localdoma | |       |       | gent         |
| -a2b0-508 |            | in        | |       |       |              |
| 3d336916d |            |           | |       |       |              |
| 26139ec1- | Metering   | compute1. | None | :-)   | UP    | neutron-     |
| f4f9-4bb3 | agent      | localdoma | |       |       | metering-    |
| -aebb-c35 |            | in        | |       |       | agent        |
| 3a36ed79c |            |           | |       |       |              |
| 2a54971f- | DHCP agent | network1. | nova | :-)   | UP    | neutron-     |
| 9849-4ed2 |            | localdoma | |       |       | dhcp-agent   |
| -b009-00e |            | in        | |       |       |              |
| 45eb4d255 |            |           | |       |       |              |
| 443c0b49- | Open       | compute1. | None | :-)   | UP    | neutron-     |
| 4484-44d2 | vSwitch    | localdoma | |       |       | openvswitch- |
| -a704-32a | agent      | in        | |       |       | agent        |
| 92ffe6982 |            |           | |       |       |              |
| 5d00a219 | L3 agent   | network1. | nova | :-)   | UP    | neutron-vpn- |
| -abce-    |            | localdoma | |       |       | agent        |
| 48ca-     |            | in        | |       |       |              |
| ba1e-d962 |            |           | |       |       |              |
| 01bd7de3 |            |           | |       |       |              |
| bc3458b4 | Open       | network1. | None | :-)   | UP    | neutron-     |
| -250e-    | vSwitch    | localdoma | |       |       | openvswitch- |
| 4adf-90e0 | agent      | in        | |       |       | agent        |
| -110a1a7f |            |           | |       |       |              |
| 6ccb      |            |           | |       |       |              |
| c29f9da8- | Metering   | network1. | None | :-)   | UP    | neutron-     |
| ca58-4a11 | agent      | localdoma | |       |       | metering-    |
| -b500-a25 |            | in        | |       |       | agent        |
| 3f820808e |            |           | |       |       |              |
| cdce667d- | Metadata   | network1. | None | :-)   | UP    | neutron-     |
| faa4      | agent      | localdoma | |       |       | metadata-    |
| -49ed-    |            | in        | |       |       | agent        |
| 83ee-e0e5 |            |           | |       |       |              |
| a352d482 |            |           | |       |       |              |
| cf5ae104- | Metadata   | compute1. | None | :-)   | UP    | neutron-     |
| 49d7-4c85 | agent      | localdoma | |       |       | metadata-    |
| -a252-cc5 |            | in        | |       |       | agent        |
| 9a9a12789 |            |           | |       |       |              |
+-----------+------------+-----------+-------------------+-------+-------+--------------+

If I check the node on which my distributed router is running it tells
me that
it's running on the network node:

neutron l3-agent-list-hosting-router 37c1794b-58d1-4d0d-b34b-944ca411b86b
+--------------------------------------+----------------------+----------------+-------+----------+
| id                                   | host                 |
admin_state_up | alive | ha_state |
+--------------------------------------+----------------------+----------------+-------+----------+
| 5d00a219-abce-48ca-ba1e-d96201bd7de3 | network1.localdomain |
True           | :-)   |          |
+--------------------------------------+----------------------+----------------+-------+----------+

If I check the iptable rules for the router on the compute and network
nodes by running:

ip netns exec qrouter-37c1794b-58d1-4d0d-b34b-944ca411b86b iptables -nv -L

I see that compute1 records the traffic while network1 doesn't. Also, I
did some
debugging and found out that the metering agent on compute1 receives an
empty
list of routers when querying the routers that it should monitor.

Source:

https://github.com/openstack/neutron/blob/stable/pike/neutron/services/metering/agents/metering_agent.py#L177-L189

https://github.com/openstack/neutron/blob/stable/pike/neutron/db/metering/metering_rpc.py#L33-L57

I think this is because the router is running on network1. Why is it
running on
network1 and why does it seem that the l3 agent on compute1 does the actual
routing?

Thanks,
Alex

Re: [Pike][Neutron] L3 metering with DVR doesn't work [ In reply to ]

haleyb.dev at gmail

Oct 26, 2018, 11:49 AM

Post #2 of 3 (1336 views)

Permalink

On 10/25/2018 08:06 AM, Alexandru Sorodoc wrote:
> Hello,
>
> I'm trying to set up metering for neutron in Pike. I tested it with a
> centralized router and it works, but when I try with a distributed router it
> doesn't record any usage samples. I have one compute node and one
> network node
> and I've created an instance with a floating ip.

The metering agent isn't very well maintained, and I don't see any open
bugs similar to this issue. The only thing I can remember is this
abandoned change regarding traffic counters for DVR routers -
https://review.openstack.org/#/c/486493/ but there was no follow-on from
the author.

The best thing to do would be to try and reproduce it on the master
branch (or Rocky) and file a bug.

> I think this is because the router is running on network1. Why is it
> running on
> network1 and why does it seem that the l3 agent on compute1 does the
actual
> routing?

The compute node will do all the routing when a floating IP is
associated, the router on network1 is for default snat traffic when
there is no floating IP and the instance tries to communicate out the
external network.

-Brian

>
> openstack router show public-router2
> +-------------------------+----------------------------------------------------+
> | Field                   |
> Value                                              |
> +-------------------------+----------------------------------------------------+
> | admin_state_up          |
> UP                                                 |
> | availability_zone_hints
> |                                                    |
> | availability_zones      |
> nova                                               |
> | created_at              |
> 2018-10-05T12:07:32Z                               |
> | description |                                                    |
> | distributed             |
> True                                               |
> | external_gateway_info   | {"network_id":
> "b96473ce-                          |
> |                         | 94f6-464f-a703-5285fb8ff3d3", "enable_snat":
> true, |
> |                         | "external_fixed_ips":
> [{"subnet_id":               |
> |                         |
> "6c08c3d9-7df1-4bec-b847-19f80b9d1764",            |
> |                         | "ip_address":
> "192.168.252.102"}]}                 |
> | flavor_id               |
> None                                               |
> | ha                      |
> False                                              |
> | id                      |
> 37c1794b-58d1-4d0d-b34b-944ca411b86b               |
> | name                    |
> public-router2                                     |
> | project_id              |
> fe203109e67f4e39b066c9529f9fc35d                   |
> | revision_number         |
> 5                                                  |
> | routes |                                                    |
> | status                  |
> ACTIVE                                             |
> | tags |                                                    |
> | updated_at              |
> 2018-10-05T12:09:36Z                               |
> +-------------------------+----------------------------------------------------+
>
> openstack network agent list
> +-----------+------------+-----------+-------------------+-------+-------+--------------+
> | ID        | Agent Type | Host      | Availability Zone | Alive | State
> | Binary       |
> +-----------+------------+-----------+-------------------+-------+-------+--------------+
> | 14b9ea75- | L3 agent   | compute1. | nova | :-)   | UP    | neutron-l3-a |
> | 1dc1-4e37 |            | localdoma | |       |       | gent         |
> | -a2b0-508 |            | in        | |       |       |              |
> | 3d336916d |            |           | |       |       |              |
> | 26139ec1- | Metering   | compute1. | None | :-)   | UP    | neutron-     |
> | f4f9-4bb3 | agent      | localdoma | |       |       | metering-    |
> | -aebb-c35 |            | in        | |       |       | agent        |
> | 3a36ed79c |            |           | |       |       |              |
> | 2a54971f- | DHCP agent | network1. | nova | :-)   | UP    | neutron-     |
> | 9849-4ed2 |            | localdoma | |       |       | dhcp-agent   |
> | -b009-00e |            | in        | |       |       |              |
> | 45eb4d255 |            |           | |       |       |              |
> | 443c0b49- | Open       | compute1. | None | :-)   | UP    | neutron-     |
> | 4484-44d2 | vSwitch    | localdoma | |       |       | openvswitch- |
> | -a704-32a | agent      | in        | |       |       | agent        |
> | 92ffe6982 |            |           | |       |       |              |
> | 5d00a219 | L3 agent   | network1. | nova | :-)   | UP    | neutron-vpn- |
> | -abce-    |            | localdoma | |       |       | agent        |
> | 48ca-     |            | in        | |       |       |              |
> | ba1e-d962 |            |           | |       |       |              |
> | 01bd7de3 |            |           | |       |       |              |
> | bc3458b4 | Open       | network1. | None | :-)   | UP    | neutron-     |
> | -250e-    | vSwitch    | localdoma | |       |       | openvswitch- |
> | 4adf-90e0 | agent      | in        | |       |       | agent        |
> | -110a1a7f |            |           | |       |       |              |
> | 6ccb      |            |           | |       |       |              |
> | c29f9da8- | Metering   | network1. | None | :-)   | UP    | neutron-     |
> | ca58-4a11 | agent      | localdoma | |       |       | metering-    |
> | -b500-a25 |            | in        | |       |       | agent        |
> | 3f820808e |            |           | |       |       |              |
> | cdce667d- | Metadata   | network1. | None | :-)   | UP    | neutron-     |
> | faa4      | agent      | localdoma | |       |       | metadata-    |
> | -49ed-    |            | in        | |       |       | agent        |
> | 83ee-e0e5 |            |           | |       |       |              |
> | a352d482 |            |           | |       |       |              |
> | cf5ae104- | Metadata   | compute1. | None | :-)   | UP    | neutron-     |
> | 49d7-4c85 | agent      | localdoma | |       |       | metadata-    |
> | -a252-cc5 |            | in        | |       |       | agent        |
> | 9a9a12789 |            |           | |       |       |              |
> +-----------+------------+-----------+-------------------+-------+-------+--------------+
>
> If I check the node on which my distributed router is running it tells
> me that
> it's running on the network node:
>
> neutron l3-agent-list-hosting-router 37c1794b-58d1-4d0d-b34b-944ca411b86b
> +--------------------------------------+----------------------+----------------+-------+----------+
> | id                                   | host                 |
> admin_state_up | alive | ha_state |
> +--------------------------------------+----------------------+----------------+-------+----------+
> | 5d00a219-abce-48ca-ba1e-d96201bd7de3 | network1.localdomain |
> True           | :-)   |          |
> +--------------------------------------+----------------------+----------------+-------+----------+
>
> If I check the iptable rules for the router on the compute and network
> nodes by running:
>
> ip netns exec qrouter-37c1794b-58d1-4d0d-b34b-944ca411b86b iptables -nv -L
>
> I see that compute1 records the traffic while network1 doesn't. Also, I
> did some
> debugging and found out that the metering agent on compute1 receives an
> empty
> list of routers when querying the routers that it should monitor.
>
> Source:
>
> https://github.com/openstack/neutron/blob/stable/pike/neutron/services/metering/agents/metering_agent.py#L177-L189
>
> https://github.com/openstack/neutron/blob/stable/pike/neutron/db/metering/metering_rpc.py#L33-L57
>
> I think this is because the router is running on network1. Why is it
> running on
> network1 and why does it seem that the l3 agent on compute1 does the actual
> routing?
>
> Thanks,
> Alex
>
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>

_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Pike][Neutron] L3 metering with DVR doesn't work [ In reply to ]

alex at privacysystems

Nov 30, 2018, 6:24 AM

Post #3 of 3 (1323 views)

Permalink

Hello Brian,

Thanks for the info.

I looked into the code for the metering agent and discovered the following:

1. The metering agent on compute nodes doesn't get notified about DVR
routers running on the node.

2. For DVR routers it tries to meter the rfp- interface(for floating
ips) on the snat- namespace. This is wrong. The rfp- interface lies on
the qrouter- namespace on compute nodes. Also, there is a qg- interface
on the snat- namespace on network nodes (for performing NAT) which
should be metered too.

3. There is a race condition whereby the metering agent is notified
about a router before its namespaces are created. The agent ends up not
adding the metering rules for those namespaces and this leads to
unrecorded traffic.

I addressed those issues in a change:
https://review.openstack.org/#/c/621165/. Any feedback is appreciated.

Best regards,
Alex

On 26/10/2018 21:49, Brian Haley wrote:
> On 10/25/2018 08:06 AM, Alexandru Sorodoc wrote:
>> Hello,
>>
>> I'm trying to set up metering for neutron in Pike. I tested it with a
>> centralized router and it works, but when I try with a distributed
>> router it
>> doesn't record any usage samples. I have one compute node and one
>> network node
>> and I've created an instance with a floating ip.
>
> The metering agent isn't very well maintained, and I don't see any
> open bugs similar to this issue. The only thing I can remember is
> this abandoned change regarding traffic counters for DVR routers -
> https://review.openstack.org/#/c/486493/ but there was no follow-on
> from the author.
>
> The best thing to do would be to try and reproduce it on the master
> branch (or Rocky) and file a bug.
>
> > I think this is because the router is running on network1. Why is it
> > running on
> > network1 and why does it seem that the l3 agent on compute1 does the
> actual
> > routing?
>
> The compute node will do all the routing when a floating IP is
> associated, the router on network1 is for default snat traffic when
> there is no floating IP and the instance tries to communicate out the
> external network.
>
> -Brian
>
>>
>> openstack router show public-router2
>> +-------------------------+----------------------------------------------------+
>>
>> | Field                   |
>> Value                                              |
>> +-------------------------+----------------------------------------------------+
>>
>> | admin_state_up          |
>> UP                                                 |
>> | availability_zone_hints
>> |                                                    |
>> | availability_zones      |
>> nova                                               |
>> | created_at              |
>> 2018-10-05T12:07:32Z                               |
>> | description |                                                    |
>> | distributed             |
>> True                                               |
>> | external_gateway_info   | {"network_id":
>> "b96473ce-                          |
>> |                         | 94f6-464f-a703-5285fb8ff3d3",
>> "enable_snat": true, |
>> |                         | "external_fixed_ips":
>> [{"subnet_id":               |
>> |                         |
>> "6c08c3d9-7df1-4bec-b847-19f80b9d1764",            |
>> |                         | "ip_address":
>> "192.168.252.102"}]}                 |
>> | flavor_id               |
>> None                                               |
>> | ha                      |
>> False                                              |
>> | id                      |
>> 37c1794b-58d1-4d0d-b34b-944ca411b86b               |
>> | name                    |
>> public-router2                                     |
>> | project_id              |
>> fe203109e67f4e39b066c9529f9fc35d                   |
>> | revision_number         |
>> 5                                                  |
>> | routes |                                                    |
>> | status                  |
>> ACTIVE                                             |
>> | tags |                                                    |
>> | updated_at              |
>> 2018-10-05T12:09:36Z                               |
>> +-------------------------+----------------------------------------------------+
>>
>>
>> openstack network agent list
>> +-----------+------------+-----------+-------------------+-------+-------+--------------+
>>
>> | ID        | Agent Type | Host      | Availability Zone | Alive |
>> State | Binary       |
>> +-----------+------------+-----------+-------------------+-------+-------+--------------+
>>
>> | 14b9ea75- | L3 agent   | compute1. | nova | :-)   | UP    |
>> neutron-l3-a |
>> | 1dc1-4e37 |            | localdoma | |       |       | gent         |
>> | -a2b0-508 |            | in        | |       | |              |
>> | 3d336916d |            |           | |       | |              |
>> | 26139ec1- | Metering   | compute1. | None | :-)   | UP    |
>> neutron-     |
>> | f4f9-4bb3 | agent      | localdoma | |       |       | metering-    |
>> | -aebb-c35 |            | in        | |       |       | agent        |
>> | 3a36ed79c |            |           | |       | |              |
>> | 2a54971f- | DHCP agent | network1. | nova | :-)   | UP    |
>> neutron-     |
>> | 9849-4ed2 |            | localdoma | |       |       | dhcp-agent   |
>> | -b009-00e |            | in        | |       | |              |
>> | 45eb4d255 |            |           | |       | |              |
>> | 443c0b49- | Open       | compute1. | None | :-)   | UP    |
>> neutron-     |
>> | 4484-44d2 | vSwitch    | localdoma | |       |       | openvswitch- |
>> | -a704-32a | agent      | in        | |       |       | agent        |
>> | 92ffe6982 |            |           | |       | |              |
>> | 5d00a219 | L3 agent   | network1. | nova | :-)   | UP    |
>> neutron-vpn- |
>> | -abce-    |            | localdoma | |       |       | agent        |
>> | 48ca-     |            | in        | |       | |              |
>> | ba1e-d962 |            |           | |       | |              |
>> | 01bd7de3 |            |           | |       | |              |
>> | bc3458b4 | Open       | network1. | None | :-)   | UP    |
>> neutron-     |
>> | -250e-    | vSwitch    | localdoma | |       |       | openvswitch- |
>> | 4adf-90e0 | agent      | in        | |       |       | agent        |
>> | -110a1a7f |            |           | |       | |              |
>> | 6ccb      |            |           | |       | |              |
>> | c29f9da8- | Metering   | network1. | None | :-)   | UP    |
>> neutron-     |
>> | ca58-4a11 | agent      | localdoma | |       |       | metering-    |
>> | -b500-a25 |            | in        | |       |       | agent        |
>> | 3f820808e |            |           | |       | |              |
>> | cdce667d- | Metadata   | network1. | None | :-)   | UP    |
>> neutron-     |
>> | faa4      | agent      | localdoma | |       |       | metadata-    |
>> | -49ed-    |            | in        | |       |       | agent        |
>> | 83ee-e0e5 |            |           | |       | |              |
>> | a352d482 |            |           | |       | |              |
>> | cf5ae104- | Metadata   | compute1. | None | :-)   | UP    |
>> neutron-     |
>> | 49d7-4c85 | agent      | localdoma | |       |       | metadata-    |
>> | -a252-cc5 |            | in        | |       |       | agent        |
>> | 9a9a12789 |            |           | |       | |              |
>> +-----------+------------+-----------+-------------------+-------+-------+--------------+
>>
>>
>> If I check the node on which my distributed router is running it
>> tells me that
>> it's running on the network node:
>>
>> neutron l3-agent-list-hosting-router
>> 37c1794b-58d1-4d0d-b34b-944ca411b86b
>> +--------------------------------------+----------------------+----------------+-------+----------+
>>
>> | id                                   | host                 |
>> admin_state_up | alive | ha_state |
>> +--------------------------------------+----------------------+----------------+-------+----------+
>>
>> | 5d00a219-abce-48ca-ba1e-d96201bd7de3 | network1.localdomain |
>> True           | :-)   |          |
>> +--------------------------------------+----------------------+----------------+-------+----------+
>>
>>
>> If I check the iptable rules for the router on the compute and
>> network nodes by running:
>>
>> ip netns exec qrouter-37c1794b-58d1-4d0d-b34b-944ca411b86b iptables
>> -nv -L
>>
>> I see that compute1 records the traffic while network1 doesn't. Also,
>> I did some
>> debugging and found out that the metering agent on compute1 receives
>> an empty
>> list of routers when querying the routers that it should monitor.
>>
>> Source:
>>
>> https://github.com/openstack/neutron/blob/stable/pike/neutron/services/metering/agents/metering_agent.py#L177-L189
>>
>>
>> https://github.com/openstack/neutron/blob/stable/pike/neutron/db/metering/metering_rpc.py#L33-L57
>>
>>
>> I think this is because the router is running on network1. Why is it
>> running on
>> network1 and why does it seem that the l3 agent on compute1 does the
>> actual
>> routing?
>>
>> Thanks,
>> Alex
>>
>>
>>
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack@lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>