Stephen Hemminger wrote (on 30-Nov-2009 at 19:44)
...
> Unfortunately, pthread mutex and locking primitives are slow. That is
> why almost all the databases end up rolling their own. Part of the
> problem is the glibc clash between being portable and being fast.
>
> What is the proposed design for using threads? I am concerned that if
> it just changes the bottleneck from being a single thread, to having a
> single lock there will be little performance gain.
The objective is to separate out:
1. "Routeing Engine" -- managing the RIB(s) and calculating updates.
2. "BGP Engine" -- Finite State Machine and all BGP session I/O
3. CLI -- all I/O
in order to:
1. ensure that BGP sessions are kept alive, even when the
Routeing Engine gets very busy.
2. ensure that the CLI is responsive, even when the Routeing Engine
gets very busy. (Though when a command is actually executed, it
will need the attention of the Routeing Engine).
The problem is that, especially when running many hundreds of Route Server
peers, bgpd has so much work to do dealing with RIB(s) etc. that it drops
the ball on BGP sessions. That is, BGP sessions hit holdtime expired
time-outs either at the Quagga end or at the other end.
The longer term objective is to be able to cope with up to 1,000 Route
Server clients. So the problem isn't going to get any easier.
On a multi-processor there will also be a small advantage with some of the
work being done in the BGP Engine.
But the objective is not to try to get a performance gain, but to make sure
that no matter how busy the Routeing Engine gets, things don't break down.
Once the threading infrastructure is there, and in the much longer term, one
could look at multi-threading the Routeing Engine -- for use on
multi-processor machines.
The Routeing Engine and the BGP Engine share a common view of prefixes and
sets of attributes. But otherwise communicate via queues of messages.
Dividing these into separate processes would incur greater communication
costs.
I think there's enough common state between the proposed threads to make
threading the appropriate approach. I think there's enough separation
between the proposed sets to mean that they won't be forever banging mutexes
and getting in each other's way.
Do you have typical timings for locking/unlocking a mutex when there is no
contention ? I've just done a little experiment: locking and unlocking a
mutex in a loop, checking the return codes for both operations:
while (n)
{
--n ;
ret = pthread_mutex_lock(&my_mutex) ;
if (ret != 0)
break ;
ret = pthread_mutex_unlock(&my_mutex) ;
if (ret != 0)
break ;
} ;
on Phenom II 3GHz this took 19.4nS for each time round the loop. Replacing
the two functions by ret = 0, and the loop took 3.15nS. Those were with
-O0. With -O2, it was 17.6nS round the loop. (For n=1,000,000,000.)
For comparison I tried calloc(1, 128) and free(). That ran 60nS per cycle
(for n=100,000,000).
Chris
_______________________________________________
Quagga-dev mailing list
Quagga-dev@lists.quagga.net
http://lists.quagga.net/mailman/listinfo/quagga-dev