Mailing List Archive

Advisory - tunneling of IP at exchange points.
-- PLEASE NOTE: If you are replying to this, consider pruning the list
-- of cc's rather than crossposting replies wildly! Thanks.


The LINX and several of its members have recently had to take action
against an ISP that was using GRE tunneling between exchange points
to appropriate the capacity of other ISPs.

Although the offending party is now unlikely to repeat this abuse,
we feel that this is an important vulnerabilty which could
potentially be newly exploited by others elsewhere. We have created
an advisory which describes details of the vulnerability, and how to
detect and prevent it.

This advisory can be found at http://www.linx.net/tunnel-advisory.txt

Any questions are best referred to the authors listed in the
advisory text.

I would like to gratefully acknowledge the efforts of Lyndon
Levesley and Alex Bligh at GX Networks among others in tracking down
the abuse and creating this document.

Keith Mitchell

Chairman
London InterNet Exchange keith@linx.org
Geneva House, 3 Park Road
Peterborough PE1 2UX
United Kingdom
Phone: +44 1733 705000 (fax 353929)


Paul

--
Paul Thornton, Network Engineer, London Internet Exchange Ltd.
Tel: 07000 783797 Mobile: +44 467 372205
Re: Advisory - tunneling of IP at exchange points. [ In reply to ]
On Tue, 25 Nov 1997 14:47:22 +0000 (GMT)
Paul Thornton <prt@linx.net> wrote:

>
> The LINX and several of its members have recently had to take action
> against an ISP that was using GRE tunneling between exchange points
> to appropriate the capacity of other ISPs.
>

Hmm unfortuntely for us GRF owners it seems that filterd cannot deal
with filter this. Joy! I wonder how many months for a fix!?

Neil.
--
Neil J. McRae. Alive and Kicking. Domino: In the glow of the night.
neil@DOMINO.ORG NetBSD/sparc: 100% SpF (Solaris protection Factor)
Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>
Re: Advisory - tunneling of IP at exchange points. [ In reply to ]
>>>>> On Tue, 25 Nov 1997 at around 15:53:28,
>>>>> "NJM" == Neil J. McRae penned:

NJM> On Tue, 25 Nov 1997 14:47:22 +0000 (GMT)
NJM> Paul Thornton <prt@linx.net> wrote:

+> The LINX and several of its members have recently had to take action
+> against an ISP that was using GRE tunneling between exchange points
+> to appropriate the capacity of other ISPs.

NJM> Hmm unfortuntely for us GRF owners it seems that filterd cannot deal
NJM> with filter this. Joy! I wonder how many months for a fix!?

Neil,

With a bit of effort, you could

a) allow valid traffic sourced from a NAP address
b) deny any other traffic with a NAP source addr

couldn't you ?

e.g.

[ inbound at ME ]

(in pseudo ACL :)

! Allow ping, trace etc. to work in and out
permit src=192.41.177.0/24 proto=(icmp, echo-request OR echo-reply OR
unreachable, ttl-exceed ... etc.)
! oh, and BGP
permit src=192.41.177.0/24 proto=(tcp, 179)
! horrible way to allow people to traceroute in from their NAP routers
permit src=192.41.177.0/24 proto=(udp, port>30000)
!
! Some other stuff I can't be bothered to think of here
!
deny src=192.41.177.0/24

As, in general, you shouldn't see many types of traffic into you
with a source address of a NAP router. I know it's possible that
people might want to telnet to one of your SMTP ports from their
Mae-East router but it ain't very likely ;)

[. I'm assuming that the problem is you can't say "deny proto=0x2f" or
similar ? ]

NJM> Neil.

Cheers,

Lyndon


--
Penis Envy is a total Phallusy.
Re: Advisory - tunneling of IP at exchange points. [ In reply to ]
Maybe I'm missing something, but couldn't you block this with routing
as well? The attack seems to be based on the fact that your NAP routers have
routes to other NAP LANs.

Let's say you connect to just MAE-E and MAE-W. At MAE-E, add a route
for the MAE-W network to null0. Do the opposite at MAE-W. While this may
not
work for everyone, is should work for the majority. It may also be more
pleasant then adding filters to a high speed interface.

Jeff Swinton

At 03:53 PM 11/25/97 +0000, Neil J. McRae wrote:
>On Tue, 25 Nov 1997 14:47:22 +0000 (GMT)
> Paul Thornton <prt@linx.net> wrote:
>
>>
>> The LINX and several of its members have recently had to take action
>> against an ISP that was using GRE tunneling between exchange points
>> to appropriate the capacity of other ISPs.
>>
>
>Hmm unfortuntely for us GRF owners it seems that filterd cannot deal
>with filter this. Joy! I wonder how many months for a fix!?
>
>Neil.
>--
>Neil J. McRae. Alive and Kicking. Domino: In the glow of the night.
>neil@DOMINO.ORG NetBSD/sparc: 100% SpF (Solaris protection Factor)
> Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>
>
>
>
Re: Advisory - tunneling of IP at exchange points. [ In reply to ]
>>>>> On Tue, 25 Nov 1997 at around 11:44:17,
>>>>> "JS" == Jeff Swinton penned:

JS> Maybe I'm missing something, but couldn't you block this with routing
JS> as well? The attack seems to be based on the fact that your NAP routers have
JS> routes to other NAP LANs.

JS> Let's say you connect to just MAE-E and MAE-W. At MAE-E, add a route
JS> for the MAE-W network to null0. Do the opposite at MAE-W. While this may
JS> not
JS> work for everyone, is should work for the majority. It may also be more
JS> pleasant then adding filters to a high speed interface.

No - this would involve much more work than that.

Take the case of

(ME peers)---[ME router]======[MW router]------(MW peers)

all sitting inside the same AS. (put as many routers as you like in
between them or in other parts of your network - it still holds)

The next hop that "MW router" sees for a ME peer's route would be
the address of that peer *on the ME LAN*.

In general, any router that speaks iBGP needs to know a route to
every exit point of every other iBGP router. You /could/ do this
differently I suppose but it would be a ridiculous amount of work and
it would make debugging problems somewhat harder.

JS> Jeff Swinton

Cheers,

Lyndon Levesley
GX Networks


--
Penis Envy is a total Phallusy.
Re: Advisory - tunneling of IP at exchange points. [ In reply to ]
As I said, this solution isn't for everyone. Some people do set a next-hop
self somewhere within their network, I would bet the majority.
If this is the case for you, you can at least prevent people you don't peer
with
from doing it. Blackhole the NAP LANs, and add valid statics for the
people you peer with.

Jeff Swinton

At 05:03 PM 11/25/97 +0000, Lyndon Levesley wrote:
>
>>>>>> On Tue, 25 Nov 1997 at around 11:44:17,
>>>>>> "JS" == Jeff Swinton penned:
>
> JS> Maybe I'm missing something, but couldn't you block this with routing
> JS> as well? The attack seems to be based on the fact that your NAP
routers have
> JS> routes to other NAP LANs.
>
> JS> Let's say you connect to just MAE-E and MAE-W. At MAE-E, add a route
> JS> for the MAE-W network to null0. Do the opposite at MAE-W. While
this may
> JS> not
> JS> work for everyone, is should work for the majority. It may also be more
> JS> pleasant then adding filters to a high speed interface.
>
>No - this would involve much more work than that.
>
>Take the case of
>
>(ME peers)---[ME router]======[MW router]------(MW peers)
>
>all sitting inside the same AS. (put as many routers as you like in
>between them or in other parts of your network - it still holds)
>
> The next hop that "MW router" sees for a ME peer's route would be
>the address of that peer *on the ME LAN*.
>
> In general, any router that speaks iBGP needs to know a route to
>every exit point of every other iBGP router. You /could/ do this
>differently I suppose but it would be a ridiculous amount of work and
>it would make debugging problems somewhat harder.
>
> JS> Jeff Swinton
>
>Cheers,
>
>Lyndon Levesley
>GX Networks
>
>
>--
>Penis Envy is a total Phallusy.
>
>
>
Re: Advisory - tunneling of IP at exchange points. [ In reply to ]
Jeff,

> As I said, this solution isn't for everyone. Some people do set a next-hop
> self somewhere within their network, I would bet the majority.
> If this is the case for you, you can at least prevent people you don't peer
> with
> from doing it. Blackhole the NAP LANs, and add valid statics for the
> people you peer with.

This is possible but has other disadvantages. For instance, say you are
a non-peer of MCI, but reach MCI with an egress interface of your
IXP router (onto a 3rd party router). Let's say they reach MCI
through another IXP router (hardly unlikely). Then all traceroutes
from your IXP router will "*" out as MCI will have blackholed the
source address your traceroute has come from. Also it does not fix
the problem if it is a peer abusing your network (but very few
solutions do). What we tried to do was look at methods which only
hit the "illegal" traffic and had as little colateral damage as
possible.

However the most important thing probably isn't the suggested solutions,
just a heads up that it's out there, happening (or more accurately
was happening in at least one case). Once people know this, I'm
sure they'll all have their own imaginative and creative solutions
to fix this in a manner which best suits their network.

--
Alex Bligh
GX Networks (formerly Xara Networks)
Re: Advisory - tunneling of IP at exchange points. [ In reply to ]
Paul Thornton wrote:
>
We have created
> an advisory which describes details of the vulnerability, and how to
> detect and prevent it.

Paul, well written. It is important to remember that many
of the exchange points include dedicated multicast routers
which are utilizing GRE or IPinIP tunnels to route multicast
to participating ISPs located on the respective exchange.
Assumung the tunnels are routed appropriatly by prior agreement
these tunnels would fall into the "legitimate" use category
and should not be filtered.

- John
Re: Advisory - tunneling of IP at exchange points. [ In reply to ]
I've received a swathe of private email pointing out other and more
complex methods of this exploit. The reason we were not keen to go
into detail about things like this is that the advisory was a "heads
up" to a large public forum and it would be nice if people would not
mention more detailed exploits on such a public forum as NANOG :)

I can think of at least three methods to perform this exploit in
such a way as to invalidate the suggested, or in some cases *any*,
filtering and restrict detection to netflow and the like. My personal
view is that people that are clueful enough to do so probably won't
so I'm not keen to educate people in this way ;)

Cheers,

Lyndon Levesley
GX Networks (formerly Xara Networks)



--
Penis Envy is a total Phallusy.
Re: Advisory - tunneling of IP at exchange points. [ In reply to ]
It's kind of nice to see nanog used in the same form/fashion that CERT,FIRST
and CIAC notification lists are used for. Its a heads up that the bad code
will get widespread deployment quite fast and even worse, it gives folks who
had not thought about being bad hats the idea that they might make a
difference.

negative feedback is still feedback. :(

--
--bill
RE: Advisory - tunneling of IP at exchange points. [ In reply to ]
Thanks Paul, wonderful job. Side-note (taken from the exploit write-up
http://www.linx.net/tunnel-advisory.txt):

>// Adding "log-input" to the end of each access-list line will log
>// the hardware address of the sender for good measure. IOS 11.1
>// and upwards only (from memory)

We find log-input to very unreliable and often producing wrong
information. It indeed operates differently across the 11.1 train (no
comment on 11.2 offered) I think 11.1.15 breaks it badly. Albeit
improperly worded and not well defined in print on CCO, please reference
cisco BUGid CSCdj40503 prior to trusting log-input for any valid info.

Best regards,

David Van Allen - FASTNET(tm) / You Tools Corporation
dave@fast.net (888)321-FAST(3278) http://www.fast.net
FASTNET - Business and Personal Internet Solutions



> -----Original Message-----
> From: Paul Thornton [SMTP:prt@linx.net]
> Sent: Tuesday, November 25, 1997 9:47 AM
> To: nanog@merit.edu
> Cc: eof@ripe.net; se-gix@sunet.se; mae-east-tech@uu.net;
> membership@linx.net; ops@linx.net
> Subject: Advisory - tunneling of IP at exchange points.
>
> -- PLEASE NOTE: If you are replying to this, consider pruning the list
> -- of cc's rather than crossposting replies wildly!
> Thanks.
>
[snip]

> The LINX and several of its members have recently had to take action
> against an ISP that was using GRE tunneling between exchange points
> to appropriate the capacity of other ISPs.
>
> Keith Mitchell
>
> Chairman
> London InterNet Exchange keith@linx.org
> Geneva House, 3 Park Road
> Peterborough PE1 2UX
> United Kingdom
> Phone: +44 1733 705000 (fax 353929)
>
>
> Paul
>
> --
> Paul Thornton, Network Engineer, London Internet Exchange Ltd.
> Tel: 07000 783797 Mobile: +44 467 372205
RE: Advisory - tunneling of IP at exchange points. [ In reply to ]
On Wed, 26 Nov 1997, Dave Van Allen wrote:

==>We find log-input to very unreliable and often producing wrong
==>information. It indeed operates differently across the 11.1 train (no
==>comment on 11.2 offered) I think 11.1.15 breaks it badly. Albeit
==>improperly worded and not well defined in print on CCO, please reference
==>cisco BUGid CSCdj40503 prior to trusting log-input for any valid info.

CSCdj40503 simply fixes a problem where packets are not logged under
certain conditions. It doesn't change any information.

I've never seen a problem with log-input reporting bad information; if you
have and can reproduce, please document and contact your normal support
channels to fix this valuable tool.

/cah
RE: Advisory - tunneling of IP at exchange points. [ In reply to ]
Hi, Craig, that's all been done, cisco verified the troubles and
anomalies here on-site, all info has been transmitted to them now for
months. The BUGid description is worded incorrectly as I said, so you
can't really glean meaningful info from just the description. I opened
that ticket and it has been very frustrating indeed. You're right, it's
a valuable tool however it is verifiably broken in certain instances. I
was just cautioning anyone who needed to count on it.

Happy holidays!

Best regards,

David Van Allen - FASTNET(tm) / You Tools Corporation
dave@fast.net (888)321-FAST(3278) http://www.fast.net
FASTNET - Business and Personal Internet Solutions



> -----Original Message-----
> From: Craig A. Huegen [SMTP:chuegen@quadrunner.com]
> Sent: Wednesday, November 26, 1997 10:35 AM
> To: Dave Van Allen
> Cc: 'Paul Thornton'; nanog@merit.edu
> Subject: RE: Advisory - tunneling of IP at exchange points.
>
> On Wed, 26 Nov 1997, Dave Van Allen wrote:
>
> ==>We find log-input to very unreliable and often producing wrong
> ==>information. It indeed operates differently across the 11.1 train
> (no
> ==>comment on 11.2 offered) I think 11.1.15 breaks it badly. Albeit
> ==>improperly worded and not well defined in print on CCO, please
> reference
> ==>cisco BUGid CSCdj40503 prior to trusting log-input for any valid
> info.
>
> CSCdj40503 simply fixes a problem where packets are not logged under
> certain conditions. It doesn't change any information.
>
> I've never seen a problem with log-input reporting bad information; if
> you
> have and can reproduce, please document and contact your normal
> support
> channels to fix this valuable tool.
>
> /cah