Mailing List Archive

Safe, fast BGP table dumps in quagga
Hello Quagga developers

We are running quagga with about 60 sessions. We dump bgp tables once a
day. Recently, we have noticed that bgpd drops its sessions while it's
writing bgp tables to the disk. The reason is that table dump takes
close to 2 minutes in our case....

To solve the problem of table dumps locking the bgpd process, I propose
to divide the table dump procedure into two parts:
(1) Make an in-memory copy of the current routing table (RIB). Store the
time-stamp of the copy operation for later use.
(2) Dump the in-memory copy in small chunks to the file. So, the process
will dump a few entries in one go, then come up for air and do other
higher priority stuff like keepalive and update processing, and then
again dive back into table dumping.
The time-stamps dumped along with the table entries is the one stored in
step (1).

The advantage: it's simple enough to implement, and should solve our
problem. Step (1) should not lock bgpd long enough for it to miss the
keepalives. All the routing table entries will have a single time-stamp
which will make it easier to re-construct RIB at any instance by
applying updates to the dumped routing table entries...

The disadvantage: the extra memory and a bit of extra processing....

I am sending this e-mail to u for the following reasons:
(1) Are there any other problems/pitfalls that u guys can think of?
(2) If I make the extensions, would there be any interest in making this
a part of the quagga distribution? If so, how do u want me to proceed?
Alternatively, if u guys want to do it, what time frame would it take u
do it?

thx,
aman
Re: Safe, fast BGP table dumps in quagga [ In reply to ]
Aman Shaikh wrote:

> Hello Quagga developers
>
> We are running quagga with about 60 sessions. We dump bgp tables once
> a day. Recently, we have noticed that bgpd drops its sessions while
> it's writing bgp tables to the disk. The reason is that table dump
> takes close to 2 minutes in our case....
>
> To solve the problem of table dumps locking the bgpd process, I
> propose to divide the table dump procedure into two parts:
> (1) Make an in-memory copy of the current routing table (RIB). Store
> the time-stamp of the copy operation for later use.

Just fork. All systems where quagga works is copy on write so there is
virtually minimal overhead in that. In btw - gated dumps this way since
times forgotten.

> (2) Dump the in-memory copy in small chunks to the file. So, the
> process will dump a few entries in one go, then come up for air and do
> other higher priority stuff like keepalive and update processing, and
> then again dive back into table dumping.
> The time-stamps dumped along with the table entries is the one stored
> in step (1).

No need for that. See above.

>
> The advantage: it's simple enough to implement, and should solve our
> problem. Step (1) should not lock bgpd long enough for it to miss the
> keepalives. All the routing table entries will have a single
> time-stamp which will make it easier to re-construct RIB at any
> instance by applying updates to the dumped routing table entries...
>
> The disadvantage: the extra memory and a bit of extra processing....
>
> I am sending this e-mail to u for the following reasons:
> (1) Are there any other problems/pitfalls that u guys can think of?
> (2) If I make the extensions, would there be any interest in making
> this a part of the quagga distribution? If so, how do u want me to
> proceed? Alternatively, if u guys want to do it, what time frame would
> it take u do it?
>
> thx,
> aman
>
> _______________________________________________
> Quagga-dev mailing list
> Quagga-dev@lists.quagga.net
> http://lists.quagga.net/mailman/listinfo/quagga-dev
>
>
Re: Safe, fast BGP table dumps in quagga [ In reply to ]
Aman Shaikh wrote:

> Hello Quagga developers
>
> We are running quagga with about 60 sessions. We dump bgp tables once
> a day. Recently, we have noticed that bgpd drops its sessions while
> it's writing bgp tables to the disk. The reason is that table dump
> takes close to 2 minutes in our case....
>
> To solve the problem of table dumps locking the bgpd process, I
> propose to divide the table dump procedure into two parts:
> (1) Make an in-memory copy of the current routing table (RIB). Store
> the time-stamp of the copy operation for later use.

It is a very good idea ;-D

>
> (2) Dump the in-memory copy in small chunks to the file. So, the
> process will dump a few entries in one go, then come up for air and do
> other higher priority stuff like keepalive and update processing, and
> then again dive back into table dumping.
> The time-stamps dumped along with the table entries is the one stored
> in step (1).

It think that you could use /proc/BGP-PID/mem in order to read this
duplicated memory of the process with a small tool. For example, gdb
uses this feature to dump the structures of a process.

man procfs
[...]
/proc/curproc/mem the complete virtual address space of the process
[...]

The syntax of the tool could be:
$ bgpdump -P BGP-PID -f /usr/local/sbin/bgpd

For example, the execution flow of bgpdump could be:
1/ bgpdump sends a signal to BGP-PID
2/ bgpd duplicated its memory
3/ bgpdump map the symbols and the adresses of bgpd with /proc/PID/mem
4/ bgpdump reads the duplicated data
5/ bgpdump sends a signal to BGP-PID that can free its duplicated structure
6/ bgpdump dumps this data into a file


> (2) If I make the extensions, would there be any interest in making
> this a part of the quagga distribution? If so, how do u want me to
> proceed? Alternatively, if u guys want to do it, what time frame would
> it take u do it?

If this extension works, I am ready to review it for the Quaggq project.

Regards,
Vincent
Re: Safe, fast BGP table dumps in quagga [ In reply to ]
For example, the execution flow of bgpdump could be:
1/ bgpdump sends a signal to BGP-PID
2/ bgpd duplicated its memory
3/ bgpdump map the symbols and the adresses of bgpd with /proc/PID/mem
4/ bgpdump reads the duplicated data
5/ bgpdump sends a signal to BGP-PID that can free its duplicated structure
6/ bgpdump dumps this data into a file

That looks cool, but /proc is not POSIX, so perhaps instead one should
use standards-based shared memory mapping (or do it the zebra way and
have a general mechanism with multiple backends, like how various
kernel communications are handled).

--
Greg Troxel <gdt@ir.bbn.com>
Re: Safe, fast BGP table dumps in quagga [ In reply to ]
I am curious as to why this needs to eb so complicated? If the command
results in the bgpd calling fork(), the parent can immediately go back to
processing as normal, and the child could then take its time to roam
through its copy of the memory and output whatever is needed, and then
terminate. It's arguably a very simple change to the existing code. What
am I missing?

- Marc

On Thu, 8 Jan 2004, Greg Troxel wrote:

> For example, the execution flow of bgpdump could be:
> 1/ bgpdump sends a signal to BGP-PID
> 2/ bgpd duplicated its memory
> 3/ bgpdump map the symbols and the adresses of bgpd with /proc/PID/mem
> 4/ bgpdump reads the duplicated data
> 5/ bgpdump sends a signal to BGP-PID that can free its duplicated structure
> 6/ bgpdump dumps this data into a file
>
> That looks cool, but /proc is not POSIX, so perhaps instead one should
> use standards-based shared memory mapping (or do it the zebra way and
> have a general mechanism with multiple backends, like how various
> kernel communications are handled).
>
> --
> Greg Troxel <gdt@ir.bbn.com>
> _______________________________________________
> Quagga-dev mailing list
> Quagga-dev@lists.quagga.net
> http://lists.quagga.net/mailman/listinfo/quagga-dev
>
Re: Safe, fast BGP table dumps in quagga [ In reply to ]
Marc Evans wrote:

>I am curious as to why this needs to eb so complicated? If the command
>results in the bgpd calling fork(), the parent can immediately go back to
>processing as normal, and the child could then take its time to roam
>through its copy of the memory and output whatever is needed, and then
>terminate. It's arguably a very simple change to the existing code. What
>am I missing?
>
>
That is exactly what I said. Gated has been doing EXACTLY this since
times forgotten. Why reinvent the wheel?

>- Marc
>
>On Thu, 8 Jan 2004, Greg Troxel wrote:
>
>
>
>> For example, the execution flow of bgpdump could be:
>> 1/ bgpdump sends a signal to BGP-PID
>> 2/ bgpd duplicated its memory
>> 3/ bgpdump map the symbols and the adresses of bgpd with /proc/PID/mem
>> 4/ bgpdump reads the duplicated data
>> 5/ bgpdump sends a signal to BGP-PID that can free its duplicated structure
>> 6/ bgpdump dumps this data into a file
>>
>>That looks cool, but /proc is not POSIX, so perhaps instead one should
>>use standards-based shared memory mapping (or do it the zebra way and
>>have a general mechanism with multiple backends, like how various
>>kernel communications are handled).
>>
>>--
>> Greg Troxel <gdt@ir.bbn.com>
>>_______________________________________________
>>Quagga-dev mailing list
>>Quagga-dev@lists.quagga.net
>>http://lists.quagga.net/mailman/listinfo/quagga-dev
>>
>>
>>
>_______________________________________________
>Quagga-dev mailing list
>Quagga-dev@lists.quagga.net
>http://lists.quagga.net/mailman/listinfo/quagga-dev
>
>
>
>
Re: Safe, fast BGP table dumps in quagga [ In reply to ]
Aman Shaikh wrote:

>Thx.
>
>I thought about forking. I have only one concern. What happens to the
>socket descriptors for TCP connections in the child process. Should I
>close them in the child process? Would it have impact on the BGP sessions?
>
>
Methinks that you should be able to close them safely.

Question is what to do later.

If you want to make the information visible back on the CLI you will
need to make sure that this task does not consume all resources so that
you do not bring yourself back to square one. Still, it is much easier
then trying to reference a live table which is being accessed by the bgp
connections and possibly cheaper in terms of resources. Alternatively
you can introduce a modifier to some commands that forces them running
after a fork and dumping to file (methinks that will be very messy, but
it may be the exact soltution to your needs).

Brgds,

A

>aman
>
>On Thu, 8 Jan 2004, Anton Ivanov wrote:
>
>
>
>
>
>
>
Re: Safe, fast BGP table dumps in quagga [ In reply to ]
On Thu, 8 Jan 2004, Anton Ivanov wrote:

> Just fork. All systems where quagga works is copy on write so there
> is virtually minimal overhead in that.

that would be the best approach.

regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
warning: do not ever send email to spam@dishone.st
Fortune:
Did you know that if you took all the economists in the world and lined
them up end to end, they'd still point in the wrong direction?
Re: Safe, fast BGP table dumps in quagga [ In reply to ]
On Thu, 8 Jan 2004, Greg Troxel wrote:

> That looks cool, but /proc is not POSIX, so perhaps instead one
> should use standards-based shared memory mapping (or do it the
> zebra way and have a general mechanism with multiple backends, like
> how various kernel communications are handled).

I agree with Marc, its way over complex :) just fork().

But re: portable shared memory, I think shm_open() is probably the
way to go (if ever there were a need for shared memory - dont
envisage any need at present.).

regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
warning: do not ever send email to spam@dishone.st
Fortune:
No problem is so formidable that you can't just walk away from it.
-- C. Schulz