Mailing List Archive

Ingress filtering on transits, peers, and IX ports
We recently received an email notice from a group of security
researchers who are looking at the feasibility of attacks using spoofed
traffic. Their methodology, in broad strokes, was to send traffic to
our DNS servers with a source IP that looked like it came from our
network. Their attacks were successful, and they included a summary of
what they found. So this message has started an internal conversation
on what traffic we should be filtering and how.

This security test was not about BCP 38 for ingress traffic from our
customers, nor was it about BGP ingress filtering. This tested our
ingress filtering from the rest of the Internet.

It seems to me like we should be filtering traffic with spoofed IPs on
our transit, IX, and peering links. I have done this filtering in the
enterprise world extensively, and it's very helpful to keep out bad
traffic. BCP 84 also discusses ingress filtering for SP's briefly and
seems to advocate for it.

We have about 15 IP blocks allocated to us. With a network as small as
ours, I chose to go with a static ACL approach to start the
conversation. I crafted a static ACL, blocking all ingress traffic
sourced from any of our assigned IP ranges. I made sure to include:

* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)

The questions I wanted to ask the SP community are:

* What traffic filtering do you do on your transits, on IX ports, and
your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like?
What makes it easy or difficult to update?
* How did you test your filters when they were implemented?

Thanks a lot,

-Brian
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.

RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)

Here’s a Cisco brief on the topic:


https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding



I think all router vendors support this feature. Here’s a similar article by Juniper:

https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html


-mel beckman

On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:

?We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.

This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.

It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.

We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:

* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)

The questions I wanted to ask the SP community are:

* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
* How did you test your filters when they were implemented?

Thanks a lot,

-Brian
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
Matt Harris|Infrastructure Lead Engineer
816-256-5446|Direct
Looking for something?
Helpdesk Portal|Email Support|Billing Portal
We build and deliver end-to-end IT solutions.
On Tue, Oct 13, 2020 at 5:22 PM Mel Beckman <mel@beckman.org> wrote:

> You can also use Unicast Reverse Path Forwarding. RPF is more efficient
> than ACLs, and has the added advantage of not requiring maintenance. In a
> nutshell, if your router has a route to a prefix in its local RIB, then
> incoming packets from a border interface having a matching source IP will
> be dropped.
>
> RPF has knobs and dials to make it work for various ISP environments.
> Implement it carefully (as is be standing next to the router involved :
>

I received one of the aforementioned messages as well, and my response was
that perhaps the best overall step towards protection at scale from the
issue they raise would be for SPs to implement URPF facing stubby,
single-homed networks. This is effectively the low-hanging fruit and
doesn't require too much additional labor in terms of maintaining
additional ACLs or prefix lists. In the case of multi-homed networks,
things are less straight forward, but multi-homed networks make up a
minority even if we exclude consumer internet connections.

Take care,
Matt
RE: Ingress filtering on transits, peers, and IX ports [ In reply to ]
That’s an interesting suggestion



There are 2 modes for uRPF. Loose and strict.



Which one would you recommend in this scenario and why?



There are many ways to solve this and definitely uRPF is one layer of defense. But, probably not the best alone. I advocate a 3 layers approach.



I’m curious to hear/read which uRPF would you recommend for this particular case.



Thanks

Jean St-Laurent





From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Mel Beckman
Sent: Tuesday, October 13, 2020 6:22 PM
To: Brian Knight <ml@knight-networks.com>
Cc: nanog@nanog.org
Subject: Re: Ingress filtering on transits, peers, and IX ports



You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.



RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)



Here’s a Cisco brief on the topic:



https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding





I think all router vendors support this feature. Here’s a similar article by Juniper:



https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html




-mel beckman




On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote:



?We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.



This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.



It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.



We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:



* Permit entries for P-t-P WAN subnets on peering links

* Permit entries for IP assignments to customers running multi-homed BGP

* The "permit ipv4 any any" at the end :)



The questions I wanted to ask the SP community are:



* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?

* How is it accomplished? Through static ACL or some flavor of uRPF?

* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?

* How did you test your filters when they were implemented?



Thanks a lot,



-Brian
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
Hi Mel,

My understanding of uRPF is:

* Strict mode will permit a packet only if there is a route for the
source IP in the RIB, and that route points to the interface where the
packet was received

* Loose mode will permit a packet if there is a route for the source IP
in the RIB. It does not matter where the route is pointed.

Strict mode won't work for us, because with our multi-homed transits and
IX peers, we will almost certainly drop a legitimate packet because the
best route is through another transit.

Loose mode won't work for us, because all of our own prefixes are in our
RIB, and thus the uRPF check on a transit would never block anything.

Or am I missing something?

Thanks,

-Brian

On 2020-10-13 17:22, Mel Beckman wrote:

> You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
>
> RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
>
> Here's a Cisco brief on the topic:
>
> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>
> I think all router vendors support this feature. Here's a similar article by Juniper:
>
> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>
> -mel beckman
>
>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
>
>> We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
>
>> This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
>
>> It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
>
>> We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
>
>> * Permit entries for P-t-P WAN subnets on peering links
>
>> * Permit entries for IP assignments to customers running multi-homed BGP
>
>> * The "permit ipv4 any any" at the end :)
>
>> The questions I wanted to ask the SP community are:
>
>> * What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
>
>> * How is it accomplished? Through static ACL or some flavor of uRPF?
>
>> * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
>
>> * How did you test your filters when they were implemented?
>
>> Thanks a lot,
>
>> -Brian
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
Hi, Brian

Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated
by: RFC8704 Enhanced Feasible-Path uRPF).

Ingress Access Lists require typically manual maintenance, but are
the most bulletproof when done properly; typically, ingress access
lists are best fit between the edge and the ISP when the
configuration is not too dynamic if strict RPF is not an option,
between ISPs if the number of used prefixes is low, or as an
additional layer of protection


Ingress filters Best Practices
https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
*Don’t accept BOGON ASNs
*Don’t accept BOGON prefixes
*Don’t accept your own prefix
*Don’t accept default (unless you requested it)
*Don’t accept prefixes that are too specific
*Don’t accept if AS Path is too long
*Create filters based on Internet Routing Registries

And also BGP Best Current Practices by Philip Smith
http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf

Regards.

El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
(<nanog@nanog.org>) escribió:
>
> Hi Mel,
>
> My understanding of uRPF is:
>
> * Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
>
> * Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
>
> Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
>
> Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
>
> Or am I missing something?
>
> Thanks,
>
> -Brian
>
> On 2020-10-13 17:22, Mel Beckman wrote:
>
> You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
>
> RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
>
> Here's a Cisco brief on the topic:
>
>
> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>
>
>
>
>
> I think all router vendors support this feature. Here's a similar article by Juniper:
>
> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>
>
> -mel beckman
>
> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
>
> We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
>
> This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
>
> It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
>
> We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
>
> * Permit entries for P-t-P WAN subnets on peering links
>
> * Permit entries for IP assignments to customers running multi-homed BGP
>
> * The "permit ipv4 any any" at the end :)
>
> The questions I wanted to ask the SP community are:
>
> * What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
>
> * How is it accomplished? Through static ACL or some flavor of uRPF?
>
> * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
>
> * How did you test your filters when they were implemented?
>
> Thanks a lot,
>
> -Brian
>
>
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
Specifically with regards to “Don’t accept your own prefix”, this poses an interesting challenge for the original notice sent out by the security researchers.

It is not uncommon today for various content networks (and others) to operate multiple “island networks” with a single ASN. For example, AS65001 operates in Los Angeles and New York, with no internal connectivity between them - only connectivity via the Internet (loop prevention disablement is done by every tier 1 provider upon request). Los Angeles advertises 192.168.10.0/24 and New York advertises 192.168.20.0/24 to the Internet, and both have relevant anti-spoofing filters (e.g. Los Angeles has an ingress filter to drop traffic with source IP of 192.168.10.0/24, and New York has an ingress filter to drop traffic with a source IP of 192.168.20.0/24). Traffic between Los Angeles and New York is not filtered as they need to legitimately pass traffic to each other over the Internet. This triggers a false positive detection by the system in question that sent the original notification email.

After discussing this with the people running this project and highlighting that this will be generating false positive data as part of their research (and probably quite a lot, this practice is fairly common), the response was to establish some form of tunnel between the AS islands over the Internet. Not realistic for a bunch of the content companies who practice this design pushing tens/hundreds of Gbps over the Internet (if not more). There seemed to be no interest in the discussion that the data being generated by this system is arguably flawed.

Tl;dr - definitely don’t accept your own prefix from the site it originated from, or other sites that have internal connectivity. But also don’t assume that an AS has a full-mesh of internal connectivity behind it and shouldn’t accept its own prefixes for any reason.

Sent from my iPhone

> On Oct 13, 2020, at 7:54 PM, Marcos Manoni <marcosmanoni@gmail.com> wrote:
>
> ?Hi, Brian
>
> Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated
> by: RFC8704 Enhanced Feasible-Path uRPF).
>
> Ingress Access Lists require typically manual maintenance, but are
> the most bulletproof when done properly; typically, ingress access
> lists are best fit between the edge and the ISP when the
> configuration is not too dynamic if strict RPF is not an option,
> between ISPs if the number of used prefixes is low, or as an
> additional layer of protection
>
>
> Ingress filters Best Practices
> https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
> *Don’t accept BOGON ASNs
> *Don’t accept BOGON prefixes
> *Don’t accept your own prefix
> *Don’t accept default (unless you requested it)
> *Don’t accept prefixes that are too specific
> *Don’t accept if AS Path is too long
> *Create filters based on Internet Routing Registries
>
> And also BGP Best Current Practices by Philip Smith
> http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
>
> Regards.
>
>> El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
>> (<nanog@nanog.org>) escribió:
>>
>> Hi Mel,
>>
>> My understanding of uRPF is:
>>
>> * Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
>>
>> * Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
>>
>> Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
>>
>> Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
>>
>> Or am I missing something?
>>
>> Thanks,
>>
>> -Brian
>>
>> On 2020-10-13 17:22, Mel Beckman wrote:
>>
>> You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
>>
>> RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
>>
>> Here's a Cisco brief on the topic:
>>
>>
>> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>>
>>
>>
>>
>>
>> I think all router vendors support this feature. Here's a similar article by Juniper:
>>
>> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>>
>>
>> -mel beckman
>>
>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
>>
>> We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
>>
>> This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
>>
>> It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
>>
>> We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
>>
>> * Permit entries for P-t-P WAN subnets on peering links
>>
>> * Permit entries for IP assignments to customers running multi-homed BGP
>>
>> * The "permit ipv4 any any" at the end :)
>>
>> The questions I wanted to ask the SP community are:
>>
>> * What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
>>
>> * How is it accomplished? Through static ACL or some flavor of uRPF?
>>
>> * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
>>
>> * How did you test your filters when they were implemented?
>>
>> Thanks a lot,
>>
>> -Brian
>>
>>
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
Aside from the BCPs currently being discussed for ingress filtering, I
would be very interested in seeing what this traffic looked like from the
perspective of your DNS servers' logs.

I assume you're talking about customer facing recursive/caching resolvers,
and not authoritative-only nameservers.

Considering that one can run an instance of an anycasted recursive
nameserver, under heavy load for a very large number of clients, on a $600
1U server these days... I wonder what exactly the threat model is.

Reverse amplification of DNS traffic returning to the spoofed IPs for DoS
purposes, such as to cause the nameserver to DoS a single /32 endpoint IP
being targeted, as in common online gaming disputes?

What volume of pps or Mbps would appear as spurious traffic as a result of
this attack?



On Tue, Oct 13, 2020 at 3:14 PM Brian Knight via NANOG <nanog@nanog.org>
wrote:

> We recently received an email notice from a group of security
> researchers who are looking at the feasibility of attacks using spoofed
> traffic. Their methodology, in broad strokes, was to send traffic to
> our DNS servers with a source IP that looked like it came from our
> network. Their attacks were successful, and they included a summary of
> what they found. So this message has started an internal conversation
> on what traffic we should be filtering and how.
>
> This security test was not about BCP 38 for ingress traffic from our
> customers, nor was it about BGP ingress filtering. This tested our
> ingress filtering from the rest of the Internet.
>
> It seems to me like we should be filtering traffic with spoofed IPs on
> our transit, IX, and peering links. I have done this filtering in the
> enterprise world extensively, and it's very helpful to keep out bad
> traffic. BCP 84 also discusses ingress filtering for SP's briefly and
> seems to advocate for it.
>
> We have about 15 IP blocks allocated to us. With a network as small as
> ours, I chose to go with a static ACL approach to start the
> conversation. I crafted a static ACL, blocking all ingress traffic
> sourced from any of our assigned IP ranges. I made sure to include:
>
> * Permit entries for P-t-P WAN subnets on peering links
> * Permit entries for IP assignments to customers running multi-homed BGP
> * The "permit ipv4 any any" at the end :)
>
> The questions I wanted to ask the SP community are:
>
> * What traffic filtering do you do on your transits, on IX ports, and
> your direct peering links?
> * How is it accomplished? Through static ACL or some flavor of uRPF?
> * If you use static ACLs, what is the administrative overhead like?
> What makes it easy or difficult to update?
> * How did you test your filters when they were implemented?
>
> Thanks a lot,
>
> -Brian
>
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
> Considering that one can run an instance of an anycasted recursive
> nameserver, under heavy load for a very large number of clients, on a $600
> 1U server these days... I wonder what exactly the threat model is.

A customer forwarded one of these notices to us - looked like it's about
recursive DNS cache poisoning. It's been a while since I looked
closely, but I thought modern recursive DNS software was pretty
resistant to that, and anyway, the real answer to that is DNSSEC.

I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email
from some group I've never heard of (and haven't AFAIK engaged the
community about their "new" attack, scans, or notices)... seems more
like shameless self promotion.

--
Chris Adams <cma@cmadams.net>
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
If I had a dollar for every 'scary security alert' email received in a NOC
email inbox from a 'security researcher group' that is the results of a
port scan, or some small subset of trojan infected residential endpoint
computers attempting outbound connections on ($common_service_port), or
similar...



On Tue, Oct 13, 2020 at 7:50 PM Chris Adams <cma@cmadams.net> wrote:

> Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
> > Considering that one can run an instance of an anycasted recursive
> > nameserver, under heavy load for a very large number of clients, on a
> $600
> > 1U server these days... I wonder what exactly the threat model is.
>
> A customer forwarded one of these notices to us - looked like it's about
> recursive DNS cache poisoning. It's been a while since I looked
> closely, but I thought modern recursive DNS software was pretty
> resistant to that, and anyway, the real answer to that is DNSSEC.
>
> I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email
> from some group I've never heard of (and haven't AFAIK engaged the
> community about their "new" attack, scans, or notices)... seems more
> like shameless self promotion.
>
> --
> Chris Adams <cma@cmadams.net>
>
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
On 10/13/20 8:04 PM, Eric Kuhnke wrote:
> If I had a dollar for every 'scary security alert' email received in a
> NOC email inbox from a 'security researcher group' that is the results
> of a port scan, or some small subset of trojan infected residential
> endpoint computers attempting outbound connections on
> ($common_service_port), or similar...
>


I get stupid automated "abuse" notices all the time about being an evil
haxx0r, which is actually just having egress proxy enabled on GGC.

The most crazy email I've had so far was saying that I "breached Section
4 of the Terms and Conditions of the domain" and that my as (the AS the
GGC nodes are behind) is "to immediately cease and desist" followed by a
bunch of BS about how their IP addresses are restricted and no crawl
rights have been granted blah blah blah.
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
Brian Knight via NANOG wrote on 13/10/2020 23:49:
> Strict mode won't work for us, because with our multi-homed transits and
> IX peers, we will almost certainly drop a legitimate packet because the
> best route is through another transit.

there's no "almost" about it: strict mode is unfeasible for both transit
and IX ports.

Nick
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
On 10/13/20 9:40 PM, Nikolas Geyer wrote:
> Tl;dr - definitely don’t accept your own prefix from the site it originated from, or other sites that have internal connectivity. But also don’t assume that an AS has a full-mesh of internal connectivity behind it and shouldn’t accept its own prefixes for any reason.

While I can understand some reasons why people don't do it, I believe
the proper thing to do in this case is have multiple ASNs - one for each
island.

They obviously have distinct routing policy and thus qualify at least
under ARIN policy AFAIK. With AS4, we don't have any imminent shortage
of ASNs and don't need to be particularly stingy about allocating them
as long as a need is met.
--
Brandon Martin
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
Hi Marcos,

Thanks for your reply. But I am looking for guidance on traffic
filtering, not BGP prefix filtering.

I have looked at BCP 84, and it's a good overview of the methods
available to an ISP. My questions are more operational in nature and
are not covered by the document. Of the choices presented in BCP 84,
what do folks really use? If it's an ACL, what challenges have there
been with updates? etc.

-Brian


On 2020-10-13 18:52, Marcos Manoni wrote:
> Hi, Brian
>
> Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated
> by: RFC8704 Enhanced Feasible-Path uRPF).
>
> Ingress Access Lists require typically manual maintenance, but are
> the most bulletproof when done properly; typically, ingress access
> lists are best fit between the edge and the ISP when the
> configuration is not too dynamic if strict RPF is not an option,
> between ISPs if the number of used prefixes is low, or as an
> additional layer of protection
>
>
> Ingress filters Best Practices
> https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
> *Don’t accept BOGON ASNs
> *Don’t accept BOGON prefixes
> *Don’t accept your own prefix
> *Don’t accept default (unless you requested it)
> *Don’t accept prefixes that are too specific
> *Don’t accept if AS Path is too long
> *Create filters based on Internet Routing Registries
>
> And also BGP Best Current Practices by Philip Smith
> http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
>
> Regards.
>
> El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
> (<nanog@nanog.org>) escribió:
>>
>> Hi Mel,
>>
>> My understanding of uRPF is:
>>
>> * Strict mode will permit a packet only if there is a route for the
>> source IP in the RIB, and that route points to the interface where the
>> packet was received
>>
>> * Loose mode will permit a packet if there is a route for the source
>> IP in the RIB. It does not matter where the route is pointed.
>>
>> Strict mode won't work for us, because with our multi-homed transits
>> and IX peers, we will almost certainly drop a legitimate packet
>> because the best route is through another transit.
>>
>> Loose mode won't work for us, because all of our own prefixes are in
>> our RIB, and thus the uRPF check on a transit would never block
>> anything.
>>
>> Or am I missing something?
>>
>> Thanks,
>>
>> -Brian
>>
>> On 2020-10-13 17:22, Mel Beckman wrote:
>>
>> You can also use Unicast Reverse Path Forwarding. RPF is more
>> efficient than ACLs, and has the added advantage of not requiring
>> maintenance. In a nutshell, if your router has a route to a prefix in
>> its local RIB, then incoming packets from a border interface having a
>> matching source IP will be dropped.
>>
>> RPF has knobs and dials to make it work for various ISP environments.
>> Implement it carefully (as is be standing next to the router involved
>> :)
>>
>> Here's a Cisco brief on the topic:
>>
>>
>> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>>
>>
>>
>>
>>
>> I think all router vendors support this feature. Here's a similar
>> article by Juniper:
>>
>> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>>
>>
>> -mel beckman
>>
>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org>
>> wrote:
>>
>> We recently received an email notice from a group of security
>> researchers who are looking at the feasibility of attacks using
>> spoofed traffic. Their methodology, in broad strokes, was to send
>> traffic to our DNS servers with a source IP that looked like it came
>> from our network. Their attacks were successful, and they included a
>> summary of what they found. So this message has started an internal
>> conversation on what traffic we should be filtering and how.
>>
>> This security test was not about BCP 38 for ingress traffic from our
>> customers, nor was it about BGP ingress filtering. This tested our
>> ingress filtering from the rest of the Internet.
>>
>> It seems to me like we should be filtering traffic with spoofed IPs on
>> our transit, IX, and peering links. I have done this filtering in the
>> enterprise world extensively, and it's very helpful to keep out bad
>> traffic. BCP 84 also discusses ingress filtering for SP's briefly and
>> seems to advocate for it.
>>
>> We have about 15 IP blocks allocated to us. With a network as small
>> as ours, I chose to go with a static ACL approach to start the
>> conversation. I crafted a static ACL, blocking all ingress traffic
>> sourced from any of our assigned IP ranges. I made sure to include:
>>
>> * Permit entries for P-t-P WAN subnets on peering links
>>
>> * Permit entries for IP assignments to customers running multi-homed
>> BGP
>>
>> * The "permit ipv4 any any" at the end :)
>>
>> The questions I wanted to ask the SP community are:
>>
>> * What traffic filtering do you do on your transits, on IX ports, and
>> your direct peering links?
>>
>> * How is it accomplished? Through static ACL or some flavor of uRPF?
>>
>> * If you use static ACLs, what is the administrative overhead like?
>> What makes it easy or difficult to update?
>>
>> * How did you test your filters when they were implemented?
>>
>> Thanks a lot,
>>
>> -Brian
>>
>>
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
Hi Eric,

I shot a message over the folk who did the testing for more info about
their test. If I'm able to find anything useful in our logs from their
detail, I'll post it to the list.

The message referenced DNS cache poisoning and DDOS amplification, so it
seemed fairly general and more focused on whether ASes accepted spoofed
traffic. They also referenced the new NXNSAttack, which I did not know
about previously.

Thanks,

-Brian

On 2020-10-13 20:49, Eric Kuhnke wrote:

> Aside from the BCPs currently being discussed for ingress filtering, I would be very interested in seeing what this traffic looked like from the perspective of your DNS servers' logs.
>
> I assume you're talking about customer facing recursive/caching resolvers, and not authoritative-only nameservers.
>
> Considering that one can run an instance of an anycasted recursive nameserver, under heavy load for a very large number of clients, on a $600 1U server these days... I wonder what exactly the threat model is.
>
> Reverse amplification of DNS traffic returning to the spoofed IPs for DoS purposes, such as to cause the nameserver to DoS a single /32 endpoint IP being targeted, as in common online gaming disputes?
>
> What volume of pps or Mbps would appear as spurious traffic as a result of this attack?
>
> On Tue, Oct 13, 2020 at 3:14 PM Brian Knight via NANOG <nanog@nanog.org> wrote:
>
>> We recently received an email notice from a group of security
>> researchers who are looking at the feasibility of attacks using spoofed
>> traffic. Their methodology, in broad strokes, was to send traffic to
>> our DNS servers with a source IP that looked like it came from our
>> network. Their attacks were successful, and they included a summary of
>> what they found. So this message has started an internal conversation
>> on what traffic we should be filtering and how.
>>
>> This security test was not about BCP 38 for ingress traffic from our
>> customers, nor was it about BGP ingress filtering. This tested our
>> ingress filtering from the rest of the Internet.
>>
>> It seems to me like we should be filtering traffic with spoofed IPs on
>> our transit, IX, and peering links. I have done this filtering in the
>> enterprise world extensively, and it's very helpful to keep out bad
>> traffic. BCP 84 also discusses ingress filtering for SP's briefly and
>> seems to advocate for it.
>>
>> We have about 15 IP blocks allocated to us. With a network as small as
>> ours, I chose to go with a static ACL approach to start the
>> conversation. I crafted a static ACL, blocking all ingress traffic
>> sourced from any of our assigned IP ranges. I made sure to include:
>>
>> * Permit entries for P-t-P WAN subnets on peering links
>> * Permit entries for IP assignments to customers running multi-homed BGP
>> * The "permit ipv4 any any" at the end :)
>>
>> The questions I wanted to ask the SP community are:
>>
>> * What traffic filtering do you do on your transits, on IX ports, and
>> your direct peering links?
>> * How is it accomplished? Through static ACL or some flavor of uRPF?
>> * If you use static ACLs, what is the administrative overhead like?
>> What makes it easy or difficult to update?
>> * How did you test your filters when they were implemented?
>>
>> Thanks a lot,
>>
>> -Brian
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
> On Oct 13, 2020, at 8:49 PM, Chris Adams <cma@cmadams.net> wrote:
>
> Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
>> Considering that one can run an instance of an anycasted recursive
>> nameserver, under heavy load for a very large number of clients, on a $600
>> 1U server these days... I wonder what exactly the threat model is.
>
> A customer forwarded one of these notices to us - looked like it's about
> recursive DNS cache poisoning.

In part, yes. More generally, it's about allowing spoofed-source packets in your front door, appearing to be from your own network, and what a bad actor could do with them. The probes from the experiment were harmless. But if there were malicious intent, this access could facilitate cache poisoning, depending on your DNS server configuration.

> It's been a while since I looked
> closely, but I thought modern recursive DNS software was pretty
> resistant to that, and anyway, the real answer to that is DNSSEC.

It is. But DNSSEC requires support both on the side of the resolver (validation enabled) and the authoritative server (zone signed). Adoption is still far from universal. There are efforts to improve that, but it can't be your only hope, in its current state.

But, perhaps more importantly, cache poisoning is not the only concern. Other vulnerable DNS (for example) configurations might be exploited by a single packet being received and acted on as "trusted".

> I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email

Crafting wording in an alert email such that it should both be taken seriously and it doesn't sound too dramatic is hard. We have gotten many positive responses. But we've also gotten some *meh*. In the end we made a choice about whether individual reach-out was important and worth the effort, ahead of future publication and presentation. We decided that it was. Many operators have agreed with us. But I get that not everyone will feel the same about it.

> from some group I've never heard of (and haven't AFAIK engaged the
> community about their "new" attack, scans, or notices)

I suppose it depends on your definition of "engage the community". I think that's what we're doing right now. We're also no stranger to NANOG (though perhaps more of a lurker on the mailing list). But community is a much broader term. And anyway, there is some order to this whole thing, and broader announcements will come later.

Cheers,
Casey
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
Great for customer-facing interfaces, though.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Nick Hilliard" <nick@foobar.org>
To: "Brian Knight" <ml@knight-networks.com>
Cc: nanog@nanog.org
Sent: Wednesday, October 14, 2020 3:12:22 AM
Subject: Re: Ingress filtering on transits, peers, and IX ports

Brian Knight via NANOG wrote on 13/10/2020 23:49:
> Strict mode won't work for us, because with our multi-homed transits and
> IX peers, we will almost certainly drop a legitimate packet because the
> best route is through another transit.

there's no "almost" about it: strict mode is unfeasible for both transit
and IX ports.

Nick
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
On Tue, Oct 13, 2020 at 05:49:42PM -0500, Brian Knight via NANOG wrote:
> Hi Mel,
>
> My understanding of uRPF is:
>
> * Strict mode will permit a packet only if there is a route for the
> source IP in the RIB, and that route points to the interface where the
> packet was received
>
> * Loose mode will permit a packet if there is a route for the source IP
> in the RIB. It does not matter where the route is pointed.
>
> Strict mode won't work for us, because with our multi-homed transits and
> IX peers, we will almost certainly drop a legitimate packet because the
> best route is through another transit.
>
> Loose mode won't work for us, because all of our own prefixes are in our
> RIB, and thus the uRPF check on a transit would never block anything.

You'll be surprised at the garbage you would drop that you can't return.

- Jared
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
I too would like to know more about their methodology and actual
tangibles ideally in the form of PCAPs.


On 10/14/20 4:56 PM, Brian Knight via NANOG wrote:
> Hi Eric,
>
> I shot a message over the folk who did the testing for more info about
> their test.  If I'm able to find anything useful in our logs from their
> detail, I'll post it to the list.
>
> The message referenced DNS cache poisoning and DDOS amplification, so it
> seemed fairly general and more focused on whether ASes accepted spoofed
> traffic.  They also referenced the new NXNSAttack, which I did not know
> about previously.
>
> Thanks,
>
> -Brian
>
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
Hi Bryan,

> On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
>
> I too would like to know more about their methodology

We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.

Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.

> and actual tangibles ideally in the form of PCAPs.

What do you mean by "tangibles in the form of PCAPs"?

Casey
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
"What do you mean by "tangibles in the form of PCAPs””

I think Brian means packet captures, in PCAP format, so that we can observe example testing via Wireshark or some other protocol analyzer.

-mel


> On Oct 14, 2020, at 11:52 AM, Casey Deccio <casey@deccio.net> wrote:
>
> Hi Bryan,
>
>> On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
>>
>> I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.
>
>> and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
I think he means packet captures from an example, voluntarily-tested
recursive nameserver subject to this attack.


On Wed, Oct 14, 2020 at 11:53 AM Casey Deccio <casey@deccio.net> wrote:

> Hi Bryan,
>
> > On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
> >
> > I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be
> available in a few weeks. Happy to post it here if folks are interested.
> Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet.
> Sorry.
>
> > and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
So I have put together what I think is a reasonable and complete ACL.
From my time in the enterprise world, I know that a good ingress ACL
filters out traffic sourcing from:

* Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc
(well-documented in
https://team-cymru.com/community-services/bogon-reference/bogon-reference-http/)
* RIR-assigned blocks I am announcing to the rest of the world

However, I recognized a SP-specific case where we could receive
legitimate traffic sourcing from our own IP blocks: customers running
multi-homed BGP where we have assigned PA space to them. So I added
"permit" statements for traffic sourcing from these blocks.

Also, we have direct peering links that are numbered within our assigned
prefixes. So we can use the same ACL with these peer interfaces and
continue to have BGP work, I added "permit" statements for these
point-to-point subnets.

So the order of the statements is:

* Permit where source is direct peer PtP networks
* Permit where source is BGP customer PA prefix
* Deny where source is bogon
* Deny where source is our advertised prefixes
* Permit all other traffic

I considered BGP customer PI prefixes to be out of scope for ingress
filtering, since the customer is likely to be multi-homing. Should we
consider filtering them?

The Team Cymru Secure IOS Template
[https://www.cymru.com/Documents/secure-ios-template.html] also
references an ICMP fragment drop entry on the ingress ACL. I think
that's good for an enterprise network, but as an SP, I'm very hesitant
to include this. Is this included in anyone else's transit / peer / IX
ACL?

Is there anything else that I'm not thinking of?

Thanks,

-Brian


On 2020-10-14 09:25, Brian Knight via NANOG wrote:
> Hi Marcos,
>
> Thanks for your reply. But I am looking for guidance on traffic
> filtering, not BGP prefix filtering.
>
> I have looked at BCP 84, and it's a good overview of the methods
> available to an ISP. My questions are more operational in nature and
> are not covered by the document. Of the choices presented in BCP 84,
> what do folks really use? If it's an ACL, what challenges have there
> been with updates? etc.
>
> -Brian
>
>
> On 2020-10-13 18:52, Marcos Manoni wrote:
>> Hi, Brian
>>
>> Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated
>> by: RFC8704 Enhanced Feasible-Path uRPF).
>>
>> Ingress Access Lists require typically manual maintenance, but are
>> the most bulletproof when done properly; typically, ingress access
>> lists are best fit between the edge and the ISP when the
>> configuration is not too dynamic if strict RPF is not an option,
>> between ISPs if the number of used prefixes is low, or as an
>> additional layer of protection
>>
>>
>> Ingress filters Best Practices
>> https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
>> *Don’t accept BOGON ASNs
>> *Don’t accept BOGON prefixes
>> *Don’t accept your own prefix
>> *Don’t accept default (unless you requested it)
>> *Don’t accept prefixes that are too specific
>> *Don’t accept if AS Path is too long
>> *Create filters based on Internet Routing Registries
>>
>> And also BGP Best Current Practices by Philip Smith
>> http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
>>
>> Regards.
>>
>> El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
>> (<nanog@nanog.org>) escribió:
>>>
>>> Hi Mel,
>>>
>>> My understanding of uRPF is:
>>>
>>> * Strict mode will permit a packet only if there is a route for the
>>> source IP in the RIB, and that route points to the interface where
>>> the packet was received
>>>
>>> * Loose mode will permit a packet if there is a route for the source
>>> IP in the RIB. It does not matter where the route is pointed.
>>>
>>> Strict mode won't work for us, because with our multi-homed transits
>>> and IX peers, we will almost certainly drop a legitimate packet
>>> because the best route is through another transit.
>>>
>>> Loose mode won't work for us, because all of our own prefixes are in
>>> our RIB, and thus the uRPF check on a transit would never block
>>> anything.
>>>
>>> Or am I missing something?
>>>
>>> Thanks,
>>>
>>> -Brian
>>>
>>> On 2020-10-13 17:22, Mel Beckman wrote:
>>>
>>> You can also use Unicast Reverse Path Forwarding. RPF is more
>>> efficient than ACLs, and has the added advantage of not requiring
>>> maintenance. In a nutshell, if your router has a route to a prefix in
>>> its local RIB, then incoming packets from a border interface having a
>>> matching source IP will be dropped.
>>>
>>> RPF has knobs and dials to make it work for various ISP environments.
>>> Implement it carefully (as is be standing next to the router involved
>>> :)
>>>
>>> Here's a Cisco brief on the topic:
>>>
>>>
>>> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>>>
>>>
>>>
>>>
>>>
>>> I think all router vendors support this feature. Here's a similar
>>> article by Juniper:
>>>
>>> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>>>
>>>
>>> -mel beckman
>>>
>>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org>
>>> wrote:
>>>
>>> We recently received an email notice from a group of security
>>> researchers who are looking at the feasibility of attacks using
>>> spoofed traffic. Their methodology, in broad strokes, was to send
>>> traffic to our DNS servers with a source IP that looked like it came
>>> from our network. Their attacks were successful, and they included a
>>> summary of what they found. So this message has started an internal
>>> conversation on what traffic we should be filtering and how.
>>>
>>> This security test was not about BCP 38 for ingress traffic from our
>>> customers, nor was it about BGP ingress filtering. This tested our
>>> ingress filtering from the rest of the Internet.
>>>
>>> It seems to me like we should be filtering traffic with spoofed IPs
>>> on our transit, IX, and peering links. I have done this filtering in
>>> the enterprise world extensively, and it's very helpful to keep out
>>> bad traffic. BCP 84 also discusses ingress filtering for SP's
>>> briefly and seems to advocate for it.
>>>
>>> We have about 15 IP blocks allocated to us. With a network as small
>>> as ours, I chose to go with a static ACL approach to start the
>>> conversation. I crafted a static ACL, blocking all ingress traffic
>>> sourced from any of our assigned IP ranges. I made sure to include:
>>>
>>> * Permit entries for P-t-P WAN subnets on peering links
>>>
>>> * Permit entries for IP assignments to customers running multi-homed
>>> BGP
>>>
>>> * The "permit ipv4 any any" at the end :)
>>>
>>> The questions I wanted to ask the SP community are:
>>>
>>> * What traffic filtering do you do on your transits, on IX ports, and
>>> your direct peering links?
>>>
>>> * How is it accomplished? Through static ACL or some flavor of uRPF?
>>>
>>> * If you use static ACLs, what is the administrative overhead like?
>>> What makes it easy or difficult to update?
>>>
>>> * How did you test your filters when they were implemented?
>>>
>>> Thanks a lot,
>>>
>>> -Brian
>>>
>>>
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
> On 15 Oct 2020, at 04:09, Casey Deccio <casey@deccio.net> wrote:
>
>
>> On Oct 13, 2020, at 8:49 PM, Chris Adams <cma@cmadams.net> wrote:
>>
>> Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
>>> Considering that one can run an instance of an anycasted recursive
>>> nameserver, under heavy load for a very large number of clients, on a $600
>>> 1U server these days... I wonder what exactly the threat model is.
>>
>> A customer forwarded one of these notices to us - looked like it's about
>> recursive DNS cache poisoning.
>
> In part, yes. More generally, it's about allowing spoofed-source packets in your front door, appearing to be from your own network, and what a bad actor could do with them. The probes from the experiment were harmless. But if there were malicious intent, this access could facilitate cache poisoning, depending on your DNS server configuration.
>
>> It's been a while since I looked
>> closely, but I thought modern recursive DNS software was pretty
>> resistant to that, and anyway, the real answer to that is DNSSEC.
>
> It is. But DNSSEC requires support both on the side of the resolver (validation enabled) and the authoritative server (zone signed). Adoption is still far from universal. There are efforts to improve that, but it can't be your only hope, in its current state.
>
> But, perhaps more importantly, cache poisoning is not the only concern. Other vulnerable DNS (for example) configurations might be exploited by a single packet being received and acted on as "trusted”.

I know Casey knows this, but to the rest of you, this is why TSIG, SIG(0) and DNS COOKIE where invented. IP addresses, especially IP addresses on UDP packets, are not trustworthy. If you are not using one or more of these when sending a query you should be updating your DNS software. If you are a ISP that is purchasing CPEs you should be requiring that the CPE uses these by default.

If you are purchasing other equipment it also should be using it by default. Fixing security issues requires everyone to play their part. +10% (and growing) of the worlds authoritative DNS servers support DNS COOKIE. I don’t have measurements for recursive servers. For it to be of benefit the clients also need to be sending requests with an DNS COOKIE option present. Many recursive servers send this option by default as well. The option was design to allow independent upgrade of servers and clients to occur. The only coordination required is within a anycast server cluster where all the servers need to support the option and use a common shared secret and algorithm. Clients will workaround shared secret / algorithm misconfigurations.

Mark

>> I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email
>
> Crafting wording in an alert email such that it should both be taken seriously and it doesn't sound too dramatic is hard. We have gotten many positive responses. But we've also gotten some *meh*. In the end we made a choice about whether individual reach-out was important and worth the effort, ahead of future publication and presentation. We decided that it was. Many operators have agreed with us. But I get that not everyone will feel the same about it.
>
>> from some group I've never heard of (and haven't AFAIK engaged the
>> community about their "new" attack, scans, or notices)
>
> I suppose it depends on your definition of "engage the community". I think that's what we're doing right now. We're also no stranger to NANOG (though perhaps more of a lurker on the mailing list). But community is a much broader term. And anyway, there is some order to this whole thing, and broader announcements will come later.
>
> Cheers,
> Casey

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Re: Ingress filtering on transits, peers, and IX ports [ In reply to ]
All DNS resolvers discovered on our network belong to customers. Our own
resolvers, running unbound, were not discovered.

While filtering same AS on ingress could help those customers (but only one
was a open relay), filtering bogons is something the customer can also do.
Or the software can be fixed. Do we really expect the ISP to implement
firewalls instead of customers upgrading software?

I also note that apparently our own ISPs (transits) do not filter bogons
either.

The above is a principal question. I am going to filter bogons, it just is
not very high on my long list of stuff to do.

Regards

Baldur


ons. 14. okt. 2020 20.53 skrev Casey Deccio <casey@deccio.net>:

> Hi Bryan,
>
> > On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
> >
> > I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be
> available in a few weeks. Happy to post it here if folks are interested.
> Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet.
> Sorry.
>
> > and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey

1 2  View All