Mailing List Archive: Layer 2 ethernet redundancy?

Layer 2 ethernet redundancy?

Jan 21, 2004, 6:52 AM

Post #1 of 5 (1622 views)

Hi there,
As I'm not having much joy from other sources I'm appealing to
the wisdom of the list.

I'm looking into a scenario where we have a single Juniper (m20)
providing traffic aggregation & tunnel endpoints for customers.
Specifically I'm interested in ideas on how to best achieve
redundancy.

There are two separate uplink routers, each via a separate GE PIC.
On the incoming side we have multiple 'customer routers' connecting
into two layer2 switches (Cisco 6500).

Presently there is a separate FE PIC interface into each switch.
We run BGP between the m20 and each of the routers.

See an attempt at ascii diagram below:

customers - Router 1 --- l2switch1 --- fe-0/0/0 ip x.x.x.5
customers - Router 2 --- l2switch1
|||
customers - Router 3 --- l2switch2 --- fe-1/0/0 ip x.x.x.6
customers - Router 4 --- l2switch2

The two switches are trunked and all traffic is within the same vlan.

Currently we have configured a separate IP address on each of the FE ports.
For BGP 'redundancy' mode we have following neighbour setup:

router1 <-> m20 x.x.x.5
router2 <-> m20 x.x.x.5
router3 <-> m20 x.x.x.6
router4 <-> m20 x.x.x.6

While the solution seems to 'work' I get the feeling it is more
by accident rather than by design.

There has been a suggestion recently to use 802.3ad link aggregation instead.
Ie both fe interfaces become part of a virtual ae0 interface with just the
one IP address.
Unfortuantely the proposed design does this just on the m20 end and does not
configure anything in the switches for 802.3ad.
While a test seems to work we did notice that the virtual ethernet address
bounces around the l2switch ports (to/from the trunking port and direct port)

I have read through many of the Junos documents but am yet to find any
good recommendations on how to best implement redundancy in this environment.

Requirement really is to eliminate single point of failure.
Currently if a l2switch fails we lose "comms" from m20 to two routers but
since customer side can in theory reroute through other router group it should
be ok.
Basically our active interface into the vlan fails, which subsequently fails
two of the BGP sessions.
The second interface 'takes over' the connection into local lan and the BGP
sessions using the remaining interface as the local address keep going and
our traffic reroutes through the remaining connected routers.

Can anyone offer any suggestions on any of the following:
- where to find examples of 'best practice'?
- how to improve/optimise the above discussed items?

Please be gentle.

Thanks,

Tony

Layer 2 ethernet redundancy? [ In reply to ]

dek at hades

Jan 21, 2004, 8:38 PM

Post #2 of 5 (1527 views)

Permalink

Hi Tony,

A few notes, more of a general kind rather than strict guidelines for your
situation:

1) Your M20 *is* a single point of failure, so I presume it has dual REs
installed.

2) From the past experience, we have found that box redundancy *usually*
makes sense when it is deployed in geographically diverse locations (i.e.
different POPs, connected to compose a redundant topology). In cases of a
single-location deployments, it makes more sense to deploy a single bigger
box, but equip it with redundant everything. 6500 can be made fairly
redundant - different 10/100(/1000) cards for connections to your M20,
different 10/100(/1000) cards (maybe separate from previous ones) to connect
to your customers.

I'm not sure if it is a good idea to run 803.2ad to two different boxes.

Hope this helps, even if not much.

Regards,

-- D.K.

On Wed, Jan 21, 2004 at 10:52:37PM +1100, Tony Frank wrote:

[skip]

> Can anyone offer any suggestions on any of the following:
> - where to find examples of 'best practice'?
> - how to improve/optimise the above discussed items?
---end quoted text---

Layer 2 ethernet redundancy? [ In reply to ]

tfrank at optushome

Jan 21, 2004, 9:50 PM

Post #3 of 5 (1523 views)

Permalink

Hi,

On Thu, Jan 22, 2004 at 12:38:50PM +1100, Dmitri Kalintsev wrote:

> 1) Your M20 *is* a single point of failure, so I presume it has dual REs
> installed.
> 2) From the past experience, we have found that box redundancy *usually*
> makes sense when it is deployed in geographically diverse locations (i.e.
> different POPs, connected to compose a redundant topology). In cases of a
> single-location deployments, it makes more sense to deploy a single bigger
> box, but equip it with redundant everything. 6500 can be made fairly
> redundant - different 10/100(/1000) cards for connections to your M20,
> different 10/100(/1000) cards (maybe separate from previous ones) to connect
> to your customers.

Currently m20 is "very" redundant - dual re, ssb, dual PIC for downstream
separate dual PIC for upstream (all on separate FPC)

The entire setup is duplicated in another geographical location so if the m20
is down, traffic can reroute.

The specific concerns I guess are more to do with the practice of having the
two interfaces into separate switches carrying the same logical vlan.

eg other routers have only single interface, so it's simple to use the
interface IP as the BGP neighbour address - if either interface or router
fails the path is gone (and BGP goes away)

On m20 we have two interfaces with two IP.
Currently we spread the BGP sessions across the two interface IP addresses.
Another point that was not mentioned originally is that each 'customer' has
two routers, one on each l2 switch.

As I mentioned, it works but I am keen on examples of how it is done elsewhere.

> I'm not sure if it is a good idea to run 803.2ad to two different boxes.

Some of us agree with that, others suggest "if it works, use it" :)

thanks for the feedback,

Tony

Layer 2 ethernet redundancy? [ In reply to ]

jeff at reflected

Jan 21, 2004, 11:53 PM

Post #4 of 5 (1536 views)

Permalink

On Wed, 2004-01-21 at 21:50, Tony Frank wrote:
> The specific concerns I guess are more to do with the practice of having the
> two interfaces into separate switches carrying the same logical vlan.
>
> eg other routers have only single interface, so it's simple to use the
> interface IP as the BGP neighbour address - if either interface or router
> fails the path is gone (and BGP goes away)
>
> On m20 we have two interfaces with two IP.
> Currently we spread the BGP sessions across the two interface IP addresses.
> Another point that was not mentioned originally is that each 'customer' has
> two routers, one on each l2 switch.

Perhaps you can explain more comprehensively how your BGP sessions are
setup from your m20 to your routers. I don't understand your problem as
well as I'd like, but I'm willing to bet that layer 2 redundancy on your
m20 is not what you actually need.

Seems to me like your "routers" 1 - 4 should be speaking BGP to your
m20s on loopback addresses which participate in your IGP, for example
OSPF / ISIS / RIP2. In this instance, it doesn't matter if one of your
FE interfaces fails, because you'll have two or more layer 3 subnets on
different physical interfaces participating in your IGP, and any of
those can and will carry the route to your loopback address.

This is what your IGP is for. Do you have one now? If not, get one. Pick
OSPF because it works with a lot of devices, unless you are already more
familar with another IGP that you prefer over OSPF. No holy war, please.

When you follow-up, include example AS numbers for "routers" 1 - 4. If
you are using a route-reflector or confederation, send details on that
as well. You will probably be asking more specific questions that will
be best answered if the list readership understands your topology.

--
Jeff at Reflected Networks

Layer 2 ethernet redundancy? [ In reply to ]

tfrank at optushome

Jan 22, 2004, 3:21 AM

Post #5 of 5 (1538 views)

Permalink

Hi,

On Wed, Jan 21, 2004 at 11:27:08PM -0500, Jeff Wheeler wrote:
> Perhaps you can explain more comprehensively how your BGP sessions are
> setup from your m20 to your routers. I don't understand your problem as
> well as I'd like, but I'm willing to bet that layer 2 redundancy on your
> m20 is not what you actually need.
>
> Seems to me like your "routers" 1 - 4 should be speaking BGP to your
> m20s on loopback addresses which participate in your IGP, for example
> OSPF / ISIS / RIP2. In this instance, it doesn't matter if one of your
> FE interfaces fails, because you'll have two or more layer 3 subnets on
> different physical interfaces participating in your IGP, and any of
> those can and will carry the route to your loopback address.
>
> This is what your IGP is for. Do you have one now? If not, get one. Pick
> OSPF because it works with a lot of devices, unless you are already more
> familar with another IGP that you prefer over OSPF. No holy war, please.
>
> When you follow-up, include example AS numbers for "routers" 1 - 4. If
> you are using a route-reflector or confederation, send details on that
> as well. You will probably be asking more specific questions that will
> be best answered if the list readership understands your topology.

Ok.
No IGP in use as all the routers are on the same ethernet vlan.
Each "customer router" has only single interface, so if it fails router
is off air anyhow - no loopbacks used as such.
To provide redundancy we have two routers per "customer", one on each l2 switch.

If a router fails (except for m20) then there's a second l3 path available.
If a switch fails, there's a second l2 path available.
On m20, if a single PIC or FPC fails, we have a second PIC/FPC
If entire m20 dies there's a duplicated setup in another physical location with
WAN linking sites (that bit seems to be fine)

Possibly OSPF would be a better choice instead of BGP - something to try..
The routers are all under the same administrative domain in that same people
that configure the m20 will configure the other routers.
Our concern with OSPF was in relation to route leaks although I'm not sure
if that is still an issue.

Maybe a sample config will help explain the m20 side..

On the m20:

interfaces {
fe-0/3/1 {
unit 0 {
family inet address 192.168.0.6/24;
}
}
fe-1/3/1 {
unit 0 {
family inet address 192.168.0.7/24;
}
}
}

(in a vrf (type vrf), but I dont think it matters)
routing-options {
aggregate {
route 176.16.0.0/16;
}
autonomous-system 65502;
}
protocols {
bgp {
export export_pol;
group custa_routers {
type external;
peer-as 65501;
local-as 65502;
neighbour 192.168.0.2 {
description "cust a r1";
local-address 192.168.0.6;
}
neighbour 192.168.0.3 {
description "cust a r2";
local-address 192.168.0.7;
}
}
}
}

I think better approach is probably to use a loopback on the m20 to
terminate all BGP sessions.
The problem then is having the loopback routed properly in failure
scenario (which is where IGP such as OSPF comes in)
If we were to put in OSPF I think that would alleviate the need for BGP.

May be the way to go.

thanks,

Tony