Hi Gary, All,
Some comments inline.
Thanks,
~Sumit.
From: netstack-bounces+snaiksat=cisco.com@lists.launchpad.net
[mailto:netstack-bounces+snaiksat=cisco.com@lists.launchpad.net] On
Behalf Of Gary Kotton
Sent: Friday, May 11, 2012 1:29 AM
To: Maru Newby
Cc: Christopher Wright; netstack@lists.launchpad.net
Subject: Re: [Netstack]
ScalableAgents(
https://blueprints.launchpad.net/quantum/+spec/scalable-a gent-comms)
Hi,
Thanks for the input and comments. This is really great.
I would like to propose the following staged development (enable to
stabilize, test, and then optimize):
1. Stage 1 - have the agent detect a change, initially by polling. When
the agent detects and update then it will contact the plugin for a
detailed update about the network.
<Sumit> Currently the agents work by first polling the Quantum DB to
detect changes in the network/port state, and then accordingly react
locally (to check for the presence of a tap device, etc.). As I
understand, you are proposing to switch the sequence of this logic,
i.e., the agent first detects a change in the local state (e.g., a new
tap device has been created) and then communicates with the Quantum
plugin to obtain more context for this change. If this is the thought, I
believe it is reasonable and will eliminate the overhead from polling
the DB. The premise here of course is that the agent is able to locally
detect all the changes that it needs to react to, and in the basic case
of the Linux Bridge plugin, I don't think there is anything beyond
creation of tap devices, so this should work. If there are other state
changes introduced by the Quantum plugin that the agent needs to react
to, then the agent would not know about these in the absence of a
notification mechanism.</Sumit>
2. Stage 2 - Event driven support. One option is to have the operating
system notify the agent (as suggested by Darragh) another is to have the
VIF driver notify the agent. I am in favour of the latter. The VIF
driver is essentially creating the new tap device or deleting the
existing tap device. It seems logical that this would drive the update
on the agent.
<Sumit> I would actually prefer the former approach. It's better to
decouple the components to the extent possible so as to be able to
update/reuse them independently. I doubt that we will get any
significant performance advantage out of making the VIF driver aware of
the agents and having them communicate with those explicitly. However,
it does introduce a stronger coupling and we should probably avoid
it.</Sumit>
What I would like to do is a quick POC of the above and then write a
detailed design of the flow so that we can all review. If it "compiles
and runs" on paper then it will speed up the development, testing and
deployment. It will also enable us to document for future reference.
This will also save time with review and the ping/pong with the -1's.
<Sumit> Great, thanks for doing this! </Sumit>
Have a good weekend and thanks for the inputs and comments. Hopefully
next week I'll have an update on the progress.
Thanks
Gary
On 05/11/2012 12:30 AM, Maru Newby wrote:
Thanks Darragh! That should cover kvm. And apparently it's possible to
be notified of vif changes from xen/xcp too, a more xen-savvy co-worker
is tracking down details.
Gary, it sounds like it will be possible to have the agent notified
directly of device changes. What are you thoughts as to modifying your
proposal to take this into account?
Cheers,
Maru
On 2012-05-10, at 2:06 PM, Darragh OReilly wrote:
maybe udev events rules/actions could be installed for add/remove tap
device events
http://www.reactivated.net/writing_udev_rules.html#external-run ________________________________
From: Maru Newby <mnewby@internap.com>
To: gkotton@redhat.com
Cc: Christopher Wright <chrisw@redhat.com>; netstack@lists.launchpad.net
Sent: Thursday, 10 May 2012, 18:38
Subject: Re: [Netstack] Scalable
Agents(
https://blueprints.launchpad.net/quantum/+spec/scalable-agent-com ms)
Hi Gary,
I appreciate the effort you've put into condensing the options.
I agree with your suggestion that option 1 is a good starting point.
How will the agent discover changes to tap devices? Can an agent
register for events from linux/kvm or xen, or would the agent just poll?
For all I know agents may do this already, so I apologize if this is a
silly question.
Regarding option 2, I still see no reason to have the vif driver talk to
the agent directly. Ensuring a single point of contact between quantum
clients (of which the vif driver is one) and quantum, namely the rest
interface, limits complexity and will be easier to maintain and test.
If and when performance or other concerns require direct vif driver to
agent communication, we can go down that road, but as of now it's
answering a question that hasn't been asked. YAGNI.
I would also argue that even RPC communication between the plugin and
agent is gold-plating. The problem at hand is that database polling
doesn't scale well. The simple answer is for the plugin and agent to
communicate directly rather than through a database intermediary.
Adding RPC to the mix is an implementation detail, pure and simple, and
is not cost-free. RPC introduces queue dependency that can be
problematic to debug and as we've seen in nova can cause performance
issues all its own.
I'm all for leaving us open going forward to introduce an RPC
dependency, but I think the most important thing is to create a clean
communication interface between plugin and agent. The initial
implementation can be something simple (and relatively dependency free)
like secured http. The semantics for implementing and debugging http
communication are well-known to all of us. If and when RPC becomes
necessary, it will be straightforward to plug in a new transport driver.
Let's keep it simple - distributed computing is complicated enough!
Cheers,
Maru
On 2012-05-10, at 8:22 AM, Gary Kotton wrote:
Hi,
Below is a table that lists a number of options, a short description,
their advantages and disadvantages. Hopefully this can give an idea of
the scope and complexity.
Option
Description
Advantages
Disadvantages
1 .Agent driving data retrieval from plugin
The agent maintains a list of tap devices. If there is a new tap device
then the agent will request the network information for this tap device
from the quantum plugin. In the case of the open source ovs and lb
(linuxbridge) plugins this is tap + 11 letters of the attachment id.
The agent will send an RCP update about the delta to the plugin. The
plugin will answer accordingly. For example if one or more tap devices
are detected then these are sent to the plugin. For each new tap device
the plugin will sent the network information (tags etc) and set the
database attachment as up. For deletion they will be removed (or set as
down).
Simple
Self contained in Quantum
If there is more than 1 attachment ID with the same prefix of 11
characters then this will not work (this currently is a bug)
The agent will still have to poll the network interfaces.
2 .VIF driver driving retrieval from plugin
The VIF driver updates the plugin about a change, which inturn updates
the relevant agent (this was described in the link
https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd )
Event driven.
No polling
VIF driver and agents will need to share communication channels
3. Plugin broadcasting
When the plugin receives a change it broadcasts the change to all of the
registered agents
Relatively simple
Lots of unnecessary messages to agents that do not need to deal with the
traffic
I think that option #1 is a good start. This can later be optimized to
option #2.
Thanks
Gary
On 05/10/2012 10:05 AM, Gary Kotton wrote:
On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:
Hi Gary,
Thanks for initiating this. A couple of comments/questions -
1. Do we really need the VIF driver to communicate the agent's identity;
I am referring to the agent ID being sent by the VIF driver in the
message? In general, I am not sure if there is a need to have the VIF
driver send messages/notifications in the first place, but I perhaps
it's being included as a capability in the framework?
At the moment the open source plugins are not aware of the agents. The
agents poll the data base for updates. The agent ID enables a agent to
regsiter with the plugin, this in trrun enables the plugin to send a
update to the specific agent. The update is initiated by the VIF driver.
In my opinion this does the following:
1. updates the agents as soon as possible regarding a network change
2. limits traffic on the network
3. removes the database interface from the agents
2. One model I was thinking of (which is kind of inline with the
existing agent implementations), is where the agents are smart, and they
know what to do in response to changes in the state of the logical
Quantum resources. In such cases, the Quantum plugin need not have to
keep track of sending a message to a particular agent. Instead, can we
have broadcast messages from the plugin to all the agents? If the plugin
has to unicast messages to specific agents, then it needs to maintain a
lot more state/topology information which should not be mandated for
this sole reason.
I too thought about this option. In a sense the above proposal is an
optimization of what you mention. This comes at the cost of complexity.
The broadcast option is nice when the number of agents is small. When
this is large, then for each network update there will be
NUMBER_OF_AGENT messages sent for each update. The advantage of what you
mention is that the code is self contained in Quantum.
It may be better to start with the broadcast and then deal with the
optimizations afterwards.
Thanks
Gary
Thanks,
~Sumit.
-----Original Message-----
From: netstack-bounces+snaiksat=cisco.com@lists.launchpad.net
[mailto:netstack-bounces+snaiksat=cisco.com@lists.launchpad.net] On
Behalf Of Gary Kotton
Sent: Wednesday, May 09, 2012 4:27 AM
To:<netstack@lists.launchpad.net> <mailto:netstack@lists.launchpad.net>
Subject: [Netstack] Scalable
Agents(
https://blueprints.launchpad.net/quantum/+spec/scalable-agent- comms)
Hi,
I have added a very high level description on how to address the
issue.
This can be seen at:
https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd ZKgOlpvg/edit
Comments will be greatly appreciated.
Questions:
1. Do we want agents to be backward compatible (that is, still
maintain
the polling code)
2. The generation of the Agent ID
3. Any other ideas or thoughts about the matter?
I'd like to go ahead with a POC and implement this.
Thanks
Gary
--
Mailing list:
https://launchpad.net/~netstack <
https://launchpad.net/%7Enetstack>
Post to : netstack@lists.launchpad.net
Unsubscribe :
https://launchpad.net/~netstack <
https://launchpad.net/%7Enetstack>
More help :
https://help.launchpad.net/ListHelp --
Mailing list:
https://launchpad.net/~netstack <
https://launchpad.net/%7Enetstack>
Post to : netstack@lists.launchpad.net
Unsubscribe :
https://launchpad.net/~netstack <
https://launchpad.net/%7Enetstack>
More help :
https://help.launchpad.net/ListHelp --
Mailing list:
https://launchpad.net/~netstack <
https://launchpad.net/%7Enetstack>
Post to : netstack@lists.launchpad.net
Unsubscribe :
https://launchpad.net/~netstack <
https://launchpad.net/%7Enetstack>
More help :
https://help.launchpad.net/ListHelp