Mailing List Archive

You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.

RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)

Here’s a Cisco brief on the topic:


https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding



I think all router vendors support this feature. Here’s a similar article by Juniper:

https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html


-mel beckman

On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:

?We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.

This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.

It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.

We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:

* Permit entries for P-t-P WAN subnets on peering links
* Permit entries for IP assignments to customers running multi-homed BGP
* The "permit ipv4 any any" at the end :)

The questions I wanted to ask the SP community are:

* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
* How is it accomplished? Through static ACL or some flavor of uRPF?
* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
* How did you test your filters when they were implemented?

Thanks a lot,

-Brian

Matt Harris|Infrastructure Lead Engineer
816-256-5446|Direct
Looking for something?
Helpdesk Portal|Email Support|Billing Portal
We build and deliver end-to-end IT solutions.
On Tue, Oct 13, 2020 at 5:22 PM Mel Beckman <mel@beckman.org> wrote:

> You can also use Unicast Reverse Path Forwarding. RPF is more efficient
> than ACLs, and has the added advantage of not requiring maintenance. In a
> nutshell, if your router has a route to a prefix in its local RIB, then
> incoming packets from a border interface having a matching source IP will
> be dropped.
>
> RPF has knobs and dials to make it work for various ISP environments.
> Implement it carefully (as is be standing next to the router involved :
>

I received one of the aforementioned messages as well, and my response was
that perhaps the best overall step towards protection at scale from the
issue they raise would be for SPs to implement URPF facing stubby,
single-homed networks. This is effectively the low-hanging fruit and
doesn't require too much additional labor in terms of maintaining
additional ACLs or prefix lists. In the case of multi-homed networks,
things are less straight forward, but multi-homed networks make up a
minority even if we exclude consumer internet connections.

Take care,
Matt

That’s an interesting suggestion



There are 2 modes for uRPF. Loose and strict.



Which one would you recommend in this scenario and why?



There are many ways to solve this and definitely uRPF is one layer of defense. But, probably not the best alone. I advocate a 3 layers approach.



I’m curious to hear/read which uRPF would you recommend for this particular case.



Thanks

Jean St-Laurent





From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Mel Beckman
Sent: Tuesday, October 13, 2020 6:22 PM
To: Brian Knight <ml@knight-networks.com>
Cc: nanog@nanog.org
Subject: Re: Ingress filtering on transits, peers, and IX ports



You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.



RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)



Here’s a Cisco brief on the topic:



https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding





I think all router vendors support this feature. Here’s a similar article by Juniper:



https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html




-mel beckman




On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote:



?We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.



This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.



It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.



We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:



* Permit entries for P-t-P WAN subnets on peering links

* Permit entries for IP assignments to customers running multi-homed BGP

* The "permit ipv4 any any" at the end :)



The questions I wanted to ask the SP community are:



* What traffic filtering do you do on your transits, on IX ports, and your direct peering links?

* How is it accomplished? Through static ACL or some flavor of uRPF?

* If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?

* How did you test your filters when they were implemented?



Thanks a lot,



-Brian

Hi Mel,

My understanding of uRPF is:

* Strict mode will permit a packet only if there is a route for the
source IP in the RIB, and that route points to the interface where the
packet was received

* Loose mode will permit a packet if there is a route for the source IP
in the RIB. It does not matter where the route is pointed.

Strict mode won't work for us, because with our multi-homed transits and
IX peers, we will almost certainly drop a legitimate packet because the
best route is through another transit.

Loose mode won't work for us, because all of our own prefixes are in our
RIB, and thus the uRPF check on a transit would never block anything.

Or am I missing something?

Thanks,

-Brian

On 2020-10-13 17:22, Mel Beckman wrote:

> You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
>
> RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
>
> Here's a Cisco brief on the topic:
>
> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>
> I think all router vendors support this feature. Here's a similar article by Juniper:
>
> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>
> -mel beckman
>
>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
>
>> We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
>
>> This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
>
>> It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
>
>> We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
>
>> * Permit entries for P-t-P WAN subnets on peering links
>
>> * Permit entries for IP assignments to customers running multi-homed BGP
>
>> * The "permit ipv4 any any" at the end :)
>
>> The questions I wanted to ask the SP community are:
>
>> * What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
>
>> * How is it accomplished? Through static ACL or some flavor of uRPF?
>
>> * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
>
>> * How did you test your filters when they were implemented?
>
>> Thanks a lot,
>
>> -Brian

Hi, Brian

Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated
by: RFC8704 Enhanced Feasible-Path uRPF).

Ingress Access Lists require typically manual maintenance, but are
the most bulletproof when done properly; typically, ingress access
lists are best fit between the edge and the ISP when the
configuration is not too dynamic if strict RPF is not an option,
between ISPs if the number of used prefixes is low, or as an
additional layer of protection


Ingress filters Best Practices
https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
*Don’t accept BOGON ASNs
*Don’t accept BOGON prefixes
*Don’t accept your own prefix
*Don’t accept default (unless you requested it)
*Don’t accept prefixes that are too specific
*Don’t accept if AS Path is too long
*Create filters based on Internet Routing Registries

And also BGP Best Current Practices by Philip Smith
http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf

Regards.

El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
(<nanog@nanog.org>) escribió:
>
> Hi Mel,
>
> My understanding of uRPF is:
>
> * Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
>
> * Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
>
> Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
>
> Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
>
> Or am I missing something?
>
> Thanks,
>
> -Brian
>
> On 2020-10-13 17:22, Mel Beckman wrote:
>
> You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
>
> RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
>
> Here's a Cisco brief on the topic:
>
>
> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>
>
>
>
>
> I think all router vendors support this feature. Here's a similar article by Juniper:
>
> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>
>
> -mel beckman
>
> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
>
> We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
>
> This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
>
> It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
>
> We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
>
> * Permit entries for P-t-P WAN subnets on peering links
>
> * Permit entries for IP assignments to customers running multi-homed BGP
>
> * The "permit ipv4 any any" at the end :)
>
> The questions I wanted to ask the SP community are:
>
> * What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
>
> * How is it accomplished? Through static ACL or some flavor of uRPF?
>
> * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
>
> * How did you test your filters when they were implemented?
>
> Thanks a lot,
>
> -Brian
>
>

Specifically with regards to “Don’t accept your own prefix”, this poses an interesting challenge for the original notice sent out by the security researchers.

It is not uncommon today for various content networks (and others) to operate multiple “island networks” with a single ASN. For example, AS65001 operates in Los Angeles and New York, with no internal connectivity between them - only connectivity via the Internet (loop prevention disablement is done by every tier 1 provider upon request). Los Angeles advertises 192.168.10.0/24 and New York advertises 192.168.20.0/24 to the Internet, and both have relevant anti-spoofing filters (e.g. Los Angeles has an ingress filter to drop traffic with source IP of 192.168.10.0/24, and New York has an ingress filter to drop traffic with a source IP of 192.168.20.0/24). Traffic between Los Angeles and New York is not filtered as they need to legitimately pass traffic to each other over the Internet. This triggers a false positive detection by the system in question that sent the original notification email.

After discussing this with the people running this project and highlighting that this will be generating false positive data as part of their research (and probably quite a lot, this practice is fairly common), the response was to establish some form of tunnel between the AS islands over the Internet. Not realistic for a bunch of the content companies who practice this design pushing tens/hundreds of Gbps over the Internet (if not more). There seemed to be no interest in the discussion that the data being generated by this system is arguably flawed.

Tl;dr - definitely don’t accept your own prefix from the site it originated from, or other sites that have internal connectivity. But also don’t assume that an AS has a full-mesh of internal connectivity behind it and shouldn’t accept its own prefixes for any reason.

Sent from my iPhone

> On Oct 13, 2020, at 7:54 PM, Marcos Manoni <marcosmanoni@gmail.com> wrote:
>
> ?Hi, Brian
>
> Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated
> by: RFC8704 Enhanced Feasible-Path uRPF).
>
> Ingress Access Lists require typically manual maintenance, but are
> the most bulletproof when done properly; typically, ingress access
> lists are best fit between the edge and the ISP when the
> configuration is not too dynamic if strict RPF is not an option,
> between ISPs if the number of used prefixes is low, or as an
> additional layer of protection
>
>
> Ingress filters Best Practices
> https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
> *Don’t accept BOGON ASNs
> *Don’t accept BOGON prefixes
> *Don’t accept your own prefix
> *Don’t accept default (unless you requested it)
> *Don’t accept prefixes that are too specific
> *Don’t accept if AS Path is too long
> *Create filters based on Internet Routing Registries
>
> And also BGP Best Current Practices by Philip Smith
> http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
>
> Regards.
>
>> El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
>> (<nanog@nanog.org>) escribió:
>>
>> Hi Mel,
>>
>> My understanding of uRPF is:
>>
>> * Strict mode will permit a packet only if there is a route for the source IP in the RIB, and that route points to the interface where the packet was received
>>
>> * Loose mode will permit a packet if there is a route for the source IP in the RIB. It does not matter where the route is pointed.
>>
>> Strict mode won't work for us, because with our multi-homed transits and IX peers, we will almost certainly drop a legitimate packet because the best route is through another transit.
>>
>> Loose mode won't work for us, because all of our own prefixes are in our RIB, and thus the uRPF check on a transit would never block anything.
>>
>> Or am I missing something?
>>
>> Thanks,
>>
>> -Brian
>>
>> On 2020-10-13 17:22, Mel Beckman wrote:
>>
>> You can also use Unicast Reverse Path Forwarding. RPF is more efficient than ACLs, and has the added advantage of not requiring maintenance. In a nutshell, if your router has a route to a prefix in its local RIB, then incoming packets from a border interface having a matching source IP will be dropped.
>>
>> RPF has knobs and dials to make it work for various ISP environments. Implement it carefully (as is be standing next to the router involved :)
>>
>> Here's a Cisco brief on the topic:
>>
>>
>> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>>
>>
>>
>>
>>
>> I think all router vendors support this feature. Here's a similar article by Juniper:
>>
>> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>>
>>
>> -mel beckman
>>
>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org> wrote:
>>
>> We recently received an email notice from a group of security researchers who are looking at the feasibility of attacks using spoofed traffic. Their methodology, in broad strokes, was to send traffic to our DNS servers with a source IP that looked like it came from our network. Their attacks were successful, and they included a summary of what they found. So this message has started an internal conversation on what traffic we should be filtering and how.
>>
>> This security test was not about BCP 38 for ingress traffic from our customers, nor was it about BGP ingress filtering. This tested our ingress filtering from the rest of the Internet.
>>
>> It seems to me like we should be filtering traffic with spoofed IPs on our transit, IX, and peering links. I have done this filtering in the enterprise world extensively, and it's very helpful to keep out bad traffic. BCP 84 also discusses ingress filtering for SP's briefly and seems to advocate for it.
>>
>> We have about 15 IP blocks allocated to us. With a network as small as ours, I chose to go with a static ACL approach to start the conversation. I crafted a static ACL, blocking all ingress traffic sourced from any of our assigned IP ranges. I made sure to include:
>>
>> * Permit entries for P-t-P WAN subnets on peering links
>>
>> * Permit entries for IP assignments to customers running multi-homed BGP
>>
>> * The "permit ipv4 any any" at the end :)
>>
>> The questions I wanted to ask the SP community are:
>>
>> * What traffic filtering do you do on your transits, on IX ports, and your direct peering links?
>>
>> * How is it accomplished? Through static ACL or some flavor of uRPF?
>>
>> * If you use static ACLs, what is the administrative overhead like? What makes it easy or difficult to update?
>>
>> * How did you test your filters when they were implemented?
>>
>> Thanks a lot,
>>
>> -Brian
>>
>>

Aside from the BCPs currently being discussed for ingress filtering, I
would be very interested in seeing what this traffic looked like from the
perspective of your DNS servers' logs.

I assume you're talking about customer facing recursive/caching resolvers,
and not authoritative-only nameservers.

Considering that one can run an instance of an anycasted recursive
nameserver, under heavy load for a very large number of clients, on a $600
1U server these days... I wonder what exactly the threat model is.

Reverse amplification of DNS traffic returning to the spoofed IPs for DoS
purposes, such as to cause the nameserver to DoS a single /32 endpoint IP
being targeted, as in common online gaming disputes?

What volume of pps or Mbps would appear as spurious traffic as a result of
this attack?



On Tue, Oct 13, 2020 at 3:14 PM Brian Knight via NANOG <nanog@nanog.org>
wrote:

> We recently received an email notice from a group of security
> researchers who are looking at the feasibility of attacks using spoofed
> traffic. Their methodology, in broad strokes, was to send traffic to
> our DNS servers with a source IP that looked like it came from our
> network. Their attacks were successful, and they included a summary of
> what they found. So this message has started an internal conversation
> on what traffic we should be filtering and how.
>
> This security test was not about BCP 38 for ingress traffic from our
> customers, nor was it about BGP ingress filtering. This tested our
> ingress filtering from the rest of the Internet.
>
> It seems to me like we should be filtering traffic with spoofed IPs on
> our transit, IX, and peering links. I have done this filtering in the
> enterprise world extensively, and it's very helpful to keep out bad
> traffic. BCP 84 also discusses ingress filtering for SP's briefly and
> seems to advocate for it.
>
> We have about 15 IP blocks allocated to us. With a network as small as
> ours, I chose to go with a static ACL approach to start the
> conversation. I crafted a static ACL, blocking all ingress traffic
> sourced from any of our assigned IP ranges. I made sure to include:
>
> * Permit entries for P-t-P WAN subnets on peering links
> * Permit entries for IP assignments to customers running multi-homed BGP
> * The "permit ipv4 any any" at the end :)
>
> The questions I wanted to ask the SP community are:
>
> * What traffic filtering do you do on your transits, on IX ports, and
> your direct peering links?
> * How is it accomplished? Through static ACL or some flavor of uRPF?
> * If you use static ACLs, what is the administrative overhead like?
> What makes it easy or difficult to update?
> * How did you test your filters when they were implemented?
>
> Thanks a lot,
>
> -Brian
>

Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
> Considering that one can run an instance of an anycasted recursive
> nameserver, under heavy load for a very large number of clients, on a $600
> 1U server these days... I wonder what exactly the threat model is.

A customer forwarded one of these notices to us - looked like it's about
recursive DNS cache poisoning. It's been a while since I looked
closely, but I thought modern recursive DNS software was pretty
resistant to that, and anyway, the real answer to that is DNSSEC.

I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email
from some group I've never heard of (and haven't AFAIK engaged the
community about their "new" attack, scans, or notices)... seems more
like shameless self promotion.

--
Chris Adams <cma@cmadams.net>

If I had a dollar for every 'scary security alert' email received in a NOC
email inbox from a 'security researcher group' that is the results of a
port scan, or some small subset of trojan infected residential endpoint
computers attempting outbound connections on ($common_service_port), or
similar...



On Tue, Oct 13, 2020 at 7:50 PM Chris Adams <cma@cmadams.net> wrote:

> Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
> > Considering that one can run an instance of an anycasted recursive
> > nameserver, under heavy load for a very large number of clients, on a
> $600
> > 1U server these days... I wonder what exactly the threat model is.
>
> A customer forwarded one of these notices to us - looked like it's about
> recursive DNS cache poisoning. It's been a while since I looked
> closely, but I thought modern recursive DNS software was pretty
> resistant to that, and anyway, the real answer to that is DNSSEC.
>
> I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email
> from some group I've never heard of (and haven't AFAIK engaged the
> community about their "new" attack, scans, or notices)... seems more
> like shameless self promotion.
>
> --
> Chris Adams <cma@cmadams.net>
>

On 10/13/20 8:04 PM, Eric Kuhnke wrote:
> If I had a dollar for every 'scary security alert' email received in a
> NOC email inbox from a 'security researcher group' that is the results
> of a port scan, or some small subset of trojan infected residential
> endpoint computers attempting outbound connections on
> ($common_service_port), or similar...
>


I get stupid automated "abuse" notices all the time about being an evil
haxx0r, which is actually just having egress proxy enabled on GGC.

The most crazy email I've had so far was saying that I "breached Section
4 of the Terms and Conditions of the domain" and that my as (the AS the
GGC nodes are behind) is "to immediately cease and desist" followed by a
bunch of BS about how their IP addresses are restricted and no crawl
rights have been granted blah blah blah.

Brian Knight via NANOG wrote on 13/10/2020 23:49:
> Strict mode won't work for us, because with our multi-homed transits and
> IX peers, we will almost certainly drop a legitimate packet because the
> best route is through another transit.

there's no "almost" about it: strict mode is unfeasible for both transit
and IX ports.

Nick

On 10/13/20 9:40 PM, Nikolas Geyer wrote:
> Tl;dr - definitely don’t accept your own prefix from the site it originated from, or other sites that have internal connectivity. But also don’t assume that an AS has a full-mesh of internal connectivity behind it and shouldn’t accept its own prefixes for any reason.

While I can understand some reasons why people don't do it, I believe
the proper thing to do in this case is have multiple ASNs - one for each
island.

They obviously have distinct routing policy and thus qualify at least
under ARIN policy AFAIK. With AS4, we don't have any imminent shortage
of ASNs and don't need to be particularly stingy about allocating them
as long as a need is met.
--
Brandon Martin

Hi Marcos,

Thanks for your reply. But I am looking for guidance on traffic
filtering, not BGP prefix filtering.

I have looked at BCP 84, and it's a good overview of the methods
available to an ISP. My questions are more operational in nature and
are not covered by the document. Of the choices presented in BCP 84,
what do folks really use? If it's an ACL, what challenges have there
been with updates? etc.

-Brian


On 2020-10-13 18:52, Marcos Manoni wrote:
> Hi, Brian
>
> Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated
> by: RFC8704 Enhanced Feasible-Path uRPF).
>
> Ingress Access Lists require typically manual maintenance, but are
> the most bulletproof when done properly; typically, ingress access
> lists are best fit between the edge and the ISP when the
> configuration is not too dynamic if strict RPF is not an option,
> between ISPs if the number of used prefixes is low, or as an
> additional layer of protection
>
>
> Ingress filters Best Practices
> https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
> *Don’t accept BOGON ASNs
> *Don’t accept BOGON prefixes
> *Don’t accept your own prefix
> *Don’t accept default (unless you requested it)
> *Don’t accept prefixes that are too specific
> *Don’t accept if AS Path is too long
> *Create filters based on Internet Routing Registries
>
> And also BGP Best Current Practices by Philip Smith
> http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
>
> Regards.
>
> El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
> (<nanog@nanog.org>) escribió:
>>
>> Hi Mel,
>>
>> My understanding of uRPF is:
>>
>> * Strict mode will permit a packet only if there is a route for the
>> source IP in the RIB, and that route points to the interface where the
>> packet was received
>>
>> * Loose mode will permit a packet if there is a route for the source
>> IP in the RIB. It does not matter where the route is pointed.
>>
>> Strict mode won't work for us, because with our multi-homed transits
>> and IX peers, we will almost certainly drop a legitimate packet
>> because the best route is through another transit.
>>
>> Loose mode won't work for us, because all of our own prefixes are in
>> our RIB, and thus the uRPF check on a transit would never block
>> anything.
>>
>> Or am I missing something?
>>
>> Thanks,
>>
>> -Brian
>>
>> On 2020-10-13 17:22, Mel Beckman wrote:
>>
>> You can also use Unicast Reverse Path Forwarding. RPF is more
>> efficient than ACLs, and has the added advantage of not requiring
>> maintenance. In a nutshell, if your router has a route to a prefix in
>> its local RIB, then incoming packets from a border interface having a
>> matching source IP will be dropped.
>>
>> RPF has knobs and dials to make it work for various ISP environments.
>> Implement it carefully (as is be standing next to the router involved
>> :)
>>
>> Here's a Cisco brief on the topic:
>>
>>
>> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>>
>>
>>
>>
>>
>> I think all router vendors support this feature. Here's a similar
>> article by Juniper:
>>
>> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>>
>>
>> -mel beckman
>>
>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org>
>> wrote:
>>
>> We recently received an email notice from a group of security
>> researchers who are looking at the feasibility of attacks using
>> spoofed traffic. Their methodology, in broad strokes, was to send
>> traffic to our DNS servers with a source IP that looked like it came
>> from our network. Their attacks were successful, and they included a
>> summary of what they found. So this message has started an internal
>> conversation on what traffic we should be filtering and how.
>>
>> This security test was not about BCP 38 for ingress traffic from our
>> customers, nor was it about BGP ingress filtering. This tested our
>> ingress filtering from the rest of the Internet.
>>
>> It seems to me like we should be filtering traffic with spoofed IPs on
>> our transit, IX, and peering links. I have done this filtering in the
>> enterprise world extensively, and it's very helpful to keep out bad
>> traffic. BCP 84 also discusses ingress filtering for SP's briefly and
>> seems to advocate for it.
>>
>> We have about 15 IP blocks allocated to us. With a network as small
>> as ours, I chose to go with a static ACL approach to start the
>> conversation. I crafted a static ACL, blocking all ingress traffic
>> sourced from any of our assigned IP ranges. I made sure to include:
>>
>> * Permit entries for P-t-P WAN subnets on peering links
>>
>> * Permit entries for IP assignments to customers running multi-homed
>> BGP
>>
>> * The "permit ipv4 any any" at the end :)
>>
>> The questions I wanted to ask the SP community are:
>>
>> * What traffic filtering do you do on your transits, on IX ports, and
>> your direct peering links?
>>
>> * How is it accomplished? Through static ACL or some flavor of uRPF?
>>
>> * If you use static ACLs, what is the administrative overhead like?
>> What makes it easy or difficult to update?
>>
>> * How did you test your filters when they were implemented?
>>
>> Thanks a lot,
>>
>> -Brian
>>
>>

Hi Eric,

I shot a message over the folk who did the testing for more info about
their test. If I'm able to find anything useful in our logs from their
detail, I'll post it to the list.

The message referenced DNS cache poisoning and DDOS amplification, so it
seemed fairly general and more focused on whether ASes accepted spoofed
traffic. They also referenced the new NXNSAttack, which I did not know
about previously.

Thanks,

-Brian

On 2020-10-13 20:49, Eric Kuhnke wrote:

> Aside from the BCPs currently being discussed for ingress filtering, I would be very interested in seeing what this traffic looked like from the perspective of your DNS servers' logs.
>
> I assume you're talking about customer facing recursive/caching resolvers, and not authoritative-only nameservers.
>
> Considering that one can run an instance of an anycasted recursive nameserver, under heavy load for a very large number of clients, on a $600 1U server these days... I wonder what exactly the threat model is.
>
> Reverse amplification of DNS traffic returning to the spoofed IPs for DoS purposes, such as to cause the nameserver to DoS a single /32 endpoint IP being targeted, as in common online gaming disputes?
>
> What volume of pps or Mbps would appear as spurious traffic as a result of this attack?
>
> On Tue, Oct 13, 2020 at 3:14 PM Brian Knight via NANOG <nanog@nanog.org> wrote:
>
>> We recently received an email notice from a group of security
>> researchers who are looking at the feasibility of attacks using spoofed
>> traffic. Their methodology, in broad strokes, was to send traffic to
>> our DNS servers with a source IP that looked like it came from our
>> network. Their attacks were successful, and they included a summary of
>> what they found. So this message has started an internal conversation
>> on what traffic we should be filtering and how.
>>
>> This security test was not about BCP 38 for ingress traffic from our
>> customers, nor was it about BGP ingress filtering. This tested our
>> ingress filtering from the rest of the Internet.
>>
>> It seems to me like we should be filtering traffic with spoofed IPs on
>> our transit, IX, and peering links. I have done this filtering in the
>> enterprise world extensively, and it's very helpful to keep out bad
>> traffic. BCP 84 also discusses ingress filtering for SP's briefly and
>> seems to advocate for it.
>>
>> We have about 15 IP blocks allocated to us. With a network as small as
>> ours, I chose to go with a static ACL approach to start the
>> conversation. I crafted a static ACL, blocking all ingress traffic
>> sourced from any of our assigned IP ranges. I made sure to include:
>>
>> * Permit entries for P-t-P WAN subnets on peering links
>> * Permit entries for IP assignments to customers running multi-homed BGP
>> * The "permit ipv4 any any" at the end :)
>>
>> The questions I wanted to ask the SP community are:
>>
>> * What traffic filtering do you do on your transits, on IX ports, and
>> your direct peering links?
>> * How is it accomplished? Through static ACL or some flavor of uRPF?
>> * If you use static ACLs, what is the administrative overhead like?
>> What makes it easy or difficult to update?
>> * How did you test your filters when they were implemented?
>>
>> Thanks a lot,
>>
>> -Brian

> On Oct 13, 2020, at 8:49 PM, Chris Adams <cma@cmadams.net> wrote:
>
> Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
>> Considering that one can run an instance of an anycasted recursive
>> nameserver, under heavy load for a very large number of clients, on a $600
>> 1U server these days... I wonder what exactly the threat model is.
>
> A customer forwarded one of these notices to us - looked like it's about
> recursive DNS cache poisoning.

In part, yes. More generally, it's about allowing spoofed-source packets in your front door, appearing to be from your own network, and what a bad actor could do with them. The probes from the experiment were harmless. But if there were malicious intent, this access could facilitate cache poisoning, depending on your DNS server configuration.

> It's been a while since I looked
> closely, but I thought modern recursive DNS software was pretty
> resistant to that, and anyway, the real answer to that is DNSSEC.

It is. But DNSSEC requires support both on the side of the resolver (validation enabled) and the authoritative server (zone signed). Adoption is still far from universal. There are efforts to improve that, but it can't be your only hope, in its current state.

But, perhaps more importantly, cache poisoning is not the only concern. Other vulnerable DNS (for example) configurations might be exploited by a single packet being received and acted on as "trusted".

> I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email

Crafting wording in an alert email such that it should both be taken seriously and it doesn't sound too dramatic is hard. We have gotten many positive responses. But we've also gotten some *meh*. In the end we made a choice about whether individual reach-out was important and worth the effort, ahead of future publication and presentation. We decided that it was. Many operators have agreed with us. But I get that not everyone will feel the same about it.

> from some group I've never heard of (and haven't AFAIK engaged the
> community about their "new" attack, scans, or notices)

I suppose it depends on your definition of "engage the community". I think that's what we're doing right now. We're also no stranger to NANOG (though perhaps more of a lurker on the mailing list). But community is a much broader term. And anyway, there is some order to this whole thing, and broader announcements will come later.

Cheers,
Casey

Great for customer-facing interfaces, though.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Nick Hilliard" <nick@foobar.org>
To: "Brian Knight" <ml@knight-networks.com>
Cc: nanog@nanog.org
Sent: Wednesday, October 14, 2020 3:12:22 AM
Subject: Re: Ingress filtering on transits, peers, and IX ports

Brian Knight via NANOG wrote on 13/10/2020 23:49:
> Strict mode won't work for us, because with our multi-homed transits and
> IX peers, we will almost certainly drop a legitimate packet because the
> best route is through another transit.

there's no "almost" about it: strict mode is unfeasible for both transit
and IX ports.

Nick

On Tue, Oct 13, 2020 at 05:49:42PM -0500, Brian Knight via NANOG wrote:
> Hi Mel,
>
> My understanding of uRPF is:
>
> * Strict mode will permit a packet only if there is a route for the
> source IP in the RIB, and that route points to the interface where the
> packet was received
>
> * Loose mode will permit a packet if there is a route for the source IP
> in the RIB. It does not matter where the route is pointed.
>
> Strict mode won't work for us, because with our multi-homed transits and
> IX peers, we will almost certainly drop a legitimate packet because the
> best route is through another transit.
>
> Loose mode won't work for us, because all of our own prefixes are in our
> RIB, and thus the uRPF check on a transit would never block anything.

You'll be surprised at the garbage you would drop that you can't return.

- Jared

I too would like to know more about their methodology and actual
tangibles ideally in the form of PCAPs.


On 10/14/20 4:56 PM, Brian Knight via NANOG wrote:
> Hi Eric,
>
> I shot a message over the folk who did the testing for more info about
> their test.  If I'm able to find anything useful in our logs from their
> detail, I'll post it to the list.
>
> The message referenced DNS cache poisoning and DDOS amplification, so it
> seemed fairly general and more focused on whether ASes accepted spoofed
> traffic.  They also referenced the new NXNSAttack, which I did not know
> about previously.
>
> Thanks,
>
> -Brian
>

Hi Bryan,

> On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
>
> I too would like to know more about their methodology

We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.

Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.

> and actual tangibles ideally in the form of PCAPs.

What do you mean by "tangibles in the form of PCAPs"?

Casey

"What do you mean by "tangibles in the form of PCAPs””

I think Brian means packet captures, in PCAP format, so that we can observe example testing via Wireshark or some other protocol analyzer.

-mel


> On Oct 14, 2020, at 11:52 AM, Casey Deccio <casey@deccio.net> wrote:
>
> Hi Bryan,
>
>> On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
>>
>> I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.
>
>> and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey

I think he means packet captures from an example, voluntarily-tested
recursive nameserver subject to this attack.


On Wed, Oct 14, 2020 at 11:53 AM Casey Deccio <casey@deccio.net> wrote:

> Hi Bryan,
>
> > On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
> >
> > I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be
> available in a few weeks. Happy to post it here if folks are interested.
> Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet.
> Sorry.
>
> > and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey

So I have put together what I think is a reasonable and complete ACL.
From my time in the enterprise world, I know that a good ingress ACL
filters out traffic sourcing from:

* Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc
(well-documented in
https://team-cymru.com/community-services/bogon-reference/bogon-reference-http/)
* RIR-assigned blocks I am announcing to the rest of the world

However, I recognized a SP-specific case where we could receive
legitimate traffic sourcing from our own IP blocks: customers running
multi-homed BGP where we have assigned PA space to them. So I added
"permit" statements for traffic sourcing from these blocks.

Also, we have direct peering links that are numbered within our assigned
prefixes. So we can use the same ACL with these peer interfaces and
continue to have BGP work, I added "permit" statements for these
point-to-point subnets.

So the order of the statements is:

* Permit where source is direct peer PtP networks
* Permit where source is BGP customer PA prefix
* Deny where source is bogon
* Deny where source is our advertised prefixes
* Permit all other traffic

I considered BGP customer PI prefixes to be out of scope for ingress
filtering, since the customer is likely to be multi-homing. Should we
consider filtering them?

The Team Cymru Secure IOS Template
[https://www.cymru.com/Documents/secure-ios-template.html] also
references an ICMP fragment drop entry on the ingress ACL. I think
that's good for an enterprise network, but as an SP, I'm very hesitant
to include this. Is this included in anyone else's transit / peer / IX
ACL?

Is there anything else that I'm not thinking of?

Thanks,

-Brian


On 2020-10-14 09:25, Brian Knight via NANOG wrote:
> Hi Marcos,
>
> Thanks for your reply. But I am looking for guidance on traffic
> filtering, not BGP prefix filtering.
>
> I have looked at BCP 84, and it's a good overview of the methods
> available to an ISP. My questions are more operational in nature and
> are not covered by the document. Of the choices presented in BCP 84,
> what do folks really use? If it's an ACL, what challenges have there
> been with updates? etc.
>
> -Brian
>
>
> On 2020-10-13 18:52, Marcos Manoni wrote:
>> Hi, Brian
>>
>> Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks (Updated
>> by: RFC8704 Enhanced Feasible-Path uRPF).
>>
>> Ingress Access Lists require typically manual maintenance, but are
>> the most bulletproof when done properly; typically, ingress access
>> lists are best fit between the edge and the ISP when the
>> configuration is not too dynamic if strict RPF is not an option,
>> between ISPs if the number of used prefixes is low, or as an
>> additional layer of protection
>>
>>
>> Ingress filters Best Practices
>> https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
>> *Don’t accept BOGON ASNs
>> *Don’t accept BOGON prefixes
>> *Don’t accept your own prefix
>> *Don’t accept default (unless you requested it)
>> *Don’t accept prefixes that are too specific
>> *Don’t accept if AS Path is too long
>> *Create filters based on Internet Routing Registries
>>
>> And also BGP Best Current Practices by Philip Smith
>> http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
>>
>> Regards.
>>
>> El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
>> (<nanog@nanog.org>) escribió:
>>>
>>> Hi Mel,
>>>
>>> My understanding of uRPF is:
>>>
>>> * Strict mode will permit a packet only if there is a route for the
>>> source IP in the RIB, and that route points to the interface where
>>> the packet was received
>>>
>>> * Loose mode will permit a packet if there is a route for the source
>>> IP in the RIB. It does not matter where the route is pointed.
>>>
>>> Strict mode won't work for us, because with our multi-homed transits
>>> and IX peers, we will almost certainly drop a legitimate packet
>>> because the best route is through another transit.
>>>
>>> Loose mode won't work for us, because all of our own prefixes are in
>>> our RIB, and thus the uRPF check on a transit would never block
>>> anything.
>>>
>>> Or am I missing something?
>>>
>>> Thanks,
>>>
>>> -Brian
>>>
>>> On 2020-10-13 17:22, Mel Beckman wrote:
>>>
>>> You can also use Unicast Reverse Path Forwarding. RPF is more
>>> efficient than ACLs, and has the added advantage of not requiring
>>> maintenance. In a nutshell, if your router has a route to a prefix in
>>> its local RIB, then incoming packets from a border interface having a
>>> matching source IP will be dropped.
>>>
>>> RPF has knobs and dials to make it work for various ISP environments.
>>> Implement it carefully (as is be standing next to the router involved
>>> :)
>>>
>>> Here's a Cisco brief on the topic:
>>>
>>>
>>> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>>>
>>>
>>>
>>>
>>>
>>> I think all router vendors support this feature. Here's a similar
>>> article by Juniper:
>>>
>>> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>>>
>>>
>>> -mel beckman
>>>
>>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG <nanog@nanog.org>
>>> wrote:
>>>
>>> We recently received an email notice from a group of security
>>> researchers who are looking at the feasibility of attacks using
>>> spoofed traffic. Their methodology, in broad strokes, was to send
>>> traffic to our DNS servers with a source IP that looked like it came
>>> from our network. Their attacks were successful, and they included a
>>> summary of what they found. So this message has started an internal
>>> conversation on what traffic we should be filtering and how.
>>>
>>> This security test was not about BCP 38 for ingress traffic from our
>>> customers, nor was it about BGP ingress filtering. This tested our
>>> ingress filtering from the rest of the Internet.
>>>
>>> It seems to me like we should be filtering traffic with spoofed IPs
>>> on our transit, IX, and peering links. I have done this filtering in
>>> the enterprise world extensively, and it's very helpful to keep out
>>> bad traffic. BCP 84 also discusses ingress filtering for SP's
>>> briefly and seems to advocate for it.
>>>
>>> We have about 15 IP blocks allocated to us. With a network as small
>>> as ours, I chose to go with a static ACL approach to start the
>>> conversation. I crafted a static ACL, blocking all ingress traffic
>>> sourced from any of our assigned IP ranges. I made sure to include:
>>>
>>> * Permit entries for P-t-P WAN subnets on peering links
>>>
>>> * Permit entries for IP assignments to customers running multi-homed
>>> BGP
>>>
>>> * The "permit ipv4 any any" at the end :)
>>>
>>> The questions I wanted to ask the SP community are:
>>>
>>> * What traffic filtering do you do on your transits, on IX ports, and
>>> your direct peering links?
>>>
>>> * How is it accomplished? Through static ACL or some flavor of uRPF?
>>>
>>> * If you use static ACLs, what is the administrative overhead like?
>>> What makes it easy or difficult to update?
>>>
>>> * How did you test your filters when they were implemented?
>>>
>>> Thanks a lot,
>>>
>>> -Brian
>>>
>>>

> On 15 Oct 2020, at 04:09, Casey Deccio <casey@deccio.net> wrote:
>
>
>> On Oct 13, 2020, at 8:49 PM, Chris Adams <cma@cmadams.net> wrote:
>>
>> Once upon a time, Eric Kuhnke <eric.kuhnke@gmail.com> said:
>>> Considering that one can run an instance of an anycasted recursive
>>> nameserver, under heavy load for a very large number of clients, on a $600
>>> 1U server these days... I wonder what exactly the threat model is.
>>
>> A customer forwarded one of these notices to us - looked like it's about
>> recursive DNS cache poisoning.
>
> In part, yes. More generally, it's about allowing spoofed-source packets in your front door, appearing to be from your own network, and what a bad actor could do with them. The probes from the experiment were harmless. But if there were malicious intent, this access could facilitate cache poisoning, depending on your DNS server configuration.
>
>> It's been a while since I looked
>> closely, but I thought modern recursive DNS software was pretty
>> resistant to that, and anyway, the real answer to that is DNSSEC.
>
> It is. But DNSSEC requires support both on the side of the resolver (validation enabled) and the authoritative server (zone signed). Adoption is still far from universal. There are efforts to improve that, but it can't be your only hope, in its current state.
>
> But, perhaps more importantly, cache poisoning is not the only concern. Other vulnerable DNS (for example) configurations might be exploited by a single packet being received and acted on as "trusted”.

I know Casey knows this, but to the rest of you, this is why TSIG, SIG(0) and DNS COOKIE where invented. IP addresses, especially IP addresses on UDP packets, are not trustworthy. If you are not using one or more of these when sending a query you should be updating your DNS software. If you are a ISP that is purchasing CPEs you should be requiring that the CPE uses these by default.

If you are purchasing other equipment it also should be using it by default. Fixing security issues requires everyone to play their part. +10% (and growing) of the worlds authoritative DNS servers support DNS COOKIE. I don’t have measurements for recursive servers. For it to be of benefit the clients also need to be sending requests with an DNS COOKIE option present. Many recursive servers send this option by default as well. The option was design to allow independent upgrade of servers and clients to occur. The only coordination required is within a anycast server cluster where all the servers need to support the option and use a common shared secret and algorithm. Clients will workaround shared secret / algorithm misconfigurations.

Mark

>> I could be wrong, but getting a scary-sounding OMG SECURITY ALERT email
>
> Crafting wording in an alert email such that it should both be taken seriously and it doesn't sound too dramatic is hard. We have gotten many positive responses. But we've also gotten some *meh*. In the end we made a choice about whether individual reach-out was important and worth the effort, ahead of future publication and presentation. We decided that it was. Many operators have agreed with us. But I get that not everyone will feel the same about it.
>
>> from some group I've never heard of (and haven't AFAIK engaged the
>> community about their "new" attack, scans, or notices)
>
> I suppose it depends on your definition of "engage the community". I think that's what we're doing right now. We're also no stranger to NANOG (though perhaps more of a lurker on the mailing list). But community is a much broader term. And anyway, there is some order to this whole thing, and broader announcements will come later.
>
> Cheers,
> Casey

--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

All DNS resolvers discovered on our network belong to customers. Our own
resolvers, running unbound, were not discovered.

While filtering same AS on ingress could help those customers (but only one
was a open relay), filtering bogons is something the customer can also do.
Or the software can be fixed. Do we really expect the ISP to implement
firewalls instead of customers upgrading software?

I also note that apparently our own ISPs (transits) do not filter bogons
either.

The above is a principal question. I am going to filter bogons, it just is
not very high on my long list of stuff to do.

Regards

Baldur


ons. 14. okt. 2020 20.53 skrev Casey Deccio <casey@deccio.net>:

> Hi Bryan,
>
> > On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
> >
> > I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be
> available in a few weeks. Happy to post it here if folks are interested.
> Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet.
> Sorry.
>
> > and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey

Simple,

All stub autonomous systems should have a simple egress ACL allowing only PI of their customers and their own PAs -it’s a simple ACL at each AS-Exit points (towards transits/peers), that’s it.

-not sure why this isn’t the first sentence in every BCP and “security bulletin”…





adam



From: NANOG <nanog-bounces+adamv0025=netconsultings.com@nanog.org> On Behalf Of Baldur Norddahl
Sent: Thursday, October 15, 2020 8:38 AM
To: nanog@nanog.org
Subject: Re: Ingress filtering on transits, peers, and IX ports



All DNS resolvers discovered on our network belong to customers. Our own resolvers, running unbound, were not discovered.



While filtering same AS on ingress could help those customers (but only one was a open relay), filtering bogons is something the customer can also do. Or the software can be fixed. Do we really expect the ISP to implement firewalls instead of customers upgrading software?



I also note that apparently our own ISPs (transits) do not filter bogons either.



The above is a principal question. I am going to filter bogons, it just is not very high on my long list of stuff to do.



Regards



Baldur





ons. 14. okt. 2020 20.53 skrev Casey Deccio <casey@deccio.net <mailto:casey@deccio.net> >:

Hi Bryan,

> On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net <mailto:bryan@shout.net> > wrote:
>
> I too would like to know more about their methodology

We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.

Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.

> and actual tangibles ideally in the form of PCAPs.

What do you mean by "tangibles in the form of PCAPs"?

Casey

Hey,

> All stub autonomous systems should have a simple egress ACL allowing only PI of their customers and their own PAs -it’s a simple ACL at each AS-Exit points (towards transits/peers), that’s it.
>
> -not sure why this isn’t the first sentence in every BCP and “security bulletin”…

I will venture a guess.

1) it's very specific scenario to be stubby and have downstream PI
2) it won't address customers spoofing each other arbitrarily and
customer1 spoofing as customer2 on the internet, giving large chunk of
the utility of spoofing even with protection in place

How do you maintain that ACL? Why doesn't that same mechanism allow
ingress ACL on the customer port? Your proposal looks low utility for
work needed.


--
++ytti

This is about ingress ACL not egress.

tor. 15. okt. 2020 12.00 skrev <adamv0025@netconsultings.com>:

> Simple,
>
> All stub autonomous systems should have a simple egress ACL allowing only
> PI of their customers and their own PAs -it’s a simple ACL at each AS-Exit
> points (towards transits/peers), that’s it.
>
> -not sure why this isn’t the first sentence in every BCP and “security
> bulletin”…
>
>
>
>
>
> adam
>
>
>
> *From:* NANOG <nanog-bounces+adamv0025=netconsultings.com@nanog.org> *On
> Behalf Of *Baldur Norddahl
> *Sent:* Thursday, October 15, 2020 8:38 AM
> *To:* nanog@nanog.org
> *Subject:* Re: Ingress filtering on transits, peers, and IX ports
>
>
>
> All DNS resolvers discovered on our network belong to customers. Our own
> resolvers, running unbound, were not discovered.
>
>
>
> While filtering same AS on ingress could help those customers (but only
> one was a open relay), filtering bogons is something the customer can also
> do. Or the software can be fixed. Do we really expect the ISP to implement
> firewalls instead of customers upgrading software?
>
>
>
> I also note that apparently our own ISPs (transits) do not filter bogons
> either.
>
>
>
> The above is a principal question. I am going to filter bogons, it just is
> not very high on my long list of stuff to do.
>
>
>
> Regards
>
>
>
> Baldur
>
>
>
>
>
> ons. 14. okt. 2020 20.53 skrev Casey Deccio <casey@deccio.net>:
>
> Hi Bryan,
>
> > On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net> wrote:
> >
> > I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be
> available in a few weeks. Happy to post it here if folks are interested.
> Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet.
> Sorry.
>
> > and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey
>
>

Hi Brian,

"However, I recognized a SP-specific case where we could receive legitimate traffic sourcing from our own IP blocks: customers running multi-homed BGP where we have assigned PA space to them. So I added "permit" statements for traffic sourcing from these blocks."

If your customers are not using your DNS in your ip space, you could deny traffic sourcing from your ip block coming from these customers based on the UDP port DST 53. Then, accept all the rest. It's just one more line in your ACL. If they are using your DNS though, this won't work. You might want to add all the amplification ports like 123, 161, etc

Jean

-----Original Message-----
From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Brian Knight via NANOG
Sent: Wednesday, October 14, 2020 6:43 PM
To: nanog <nanog@nanog.org>
Subject: Re: Ingress filtering on transits, peers, and IX ports

So I have put together what I think is a reasonable and complete ACL.
From my time in the enterprise world, I know that a good ingress ACL filters out traffic sourcing from:

* Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc (well-documented in
https://team-cymru.com/community-services/bogon-reference/bogon-reference-http/)
* RIR-assigned blocks I am announcing to the rest of the world

However, I recognized a SP-specific case where we could receive legitimate traffic sourcing from our own IP blocks: customers running multi-homed BGP where we have assigned PA space to them. So I added "permit" statements for traffic sourcing from these blocks.

Also, we have direct peering links that are numbered within our assigned prefixes. So we can use the same ACL with these peer interfaces and continue to have BGP work, I added "permit" statements for these point-to-point subnets.

So the order of the statements is:

* Permit where source is direct peer PtP networks
* Permit where source is BGP customer PA prefix
* Deny where source is bogon
* Deny where source is our advertised prefixes
* Permit all other traffic

I considered BGP customer PI prefixes to be out of scope for ingress filtering, since the customer is likely to be multi-homing. Should we consider filtering them?

The Team Cymru Secure IOS Template
[https://www.cymru.com/Documents/secure-ios-template.html] also references an ICMP fragment drop entry on the ingress ACL. I think that's good for an enterprise network, but as an SP, I'm very hesitant to include this. Is this included in anyone else's transit / peer / IX ACL?

Is there anything else that I'm not thinking of?

Thanks,

-Brian


On 2020-10-14 09:25, Brian Knight via NANOG wrote:
> Hi Marcos,
>
> Thanks for your reply. But I am looking for guidance on traffic
> filtering, not BGP prefix filtering.
>
> I have looked at BCP 84, and it's a good overview of the methods
> available to an ISP. My questions are more operational in nature and
> are not covered by the document. Of the choices presented in BCP 84,
> what do folks really use? If it's an ACL, what challenges have there
> been with updates? etc.
>
> -Brian
>
>
> On 2020-10-13 18:52, Marcos Manoni wrote:
>> Hi, Brian
>>
>> Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks
>> (Updated
>> by: RFC8704 Enhanced Feasible-Path uRPF).
>>
>> Ingress Access Lists require typically manual maintenance, but are
>> the most bulletproof when done properly; typically, ingress access
>> lists are best fit between the edge and the ISP when the
>> configuration is not too dynamic if strict RPF is not an option,
>> between ISPs if the number of used prefixes is low, or as an
>> additional layer of protection
>>
>>
>> Ingress filters Best Practices
>> https://www.ripe.net/support/training/material/bgp-operations-and-sec
>> urity-training-course/BGP-Slides-Single.pdf
>> *Don’t accept BOGON ASNs
>> *Don’t accept BOGON prefixes
>> *Don’t accept your own prefix
>> *Don’t accept default (unless you requested it) *Don’t accept
>> prefixes that are too specific *Don’t accept if AS Path is too long
>> *Create filters based on Internet Routing Registries
>>
>> And also BGP Best Current Practices by Philip Smith
>> http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
>>
>> Regards.
>>
>> El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
>> (<nanog@nanog.org>) escribió:
>>>
>>> Hi Mel,
>>>
>>> My understanding of uRPF is:
>>>
>>> * Strict mode will permit a packet only if there is a route for the
>>> source IP in the RIB, and that route points to the interface where
>>> the packet was received
>>>
>>> * Loose mode will permit a packet if there is a route for the source
>>> IP in the RIB. It does not matter where the route is pointed.
>>>
>>> Strict mode won't work for us, because with our multi-homed transits
>>> and IX peers, we will almost certainly drop a legitimate packet
>>> because the best route is through another transit.
>>>
>>> Loose mode won't work for us, because all of our own prefixes are in
>>> our RIB, and thus the uRPF check on a transit would never block
>>> anything.
>>>
>>> Or am I missing something?
>>>
>>> Thanks,
>>>
>>> -Brian
>>>
>>> On 2020-10-13 17:22, Mel Beckman wrote:
>>>
>>> You can also use Unicast Reverse Path Forwarding. RPF is more
>>> efficient than ACLs, and has the added advantage of not requiring
>>> maintenance. In a nutshell, if your router has a route to a prefix
>>> in its local RIB, then incoming packets from a border interface
>>> having a matching source IP will be dropped.
>>>
>>> RPF has knobs and dials to make it work for various ISP environments.
>>> Implement it carefully (as is be standing next to the router
>>> involved
>>> :)
>>>
>>> Here's a Cisco brief on the topic:
>>>
>>>
>>> https://tools.cisco.com/security/center/resources/unicast_reverse_pa
>>> th_forwarding
>>>
>>>
>>>
>>>
>>>
>>> I think all router vendors support this feature. Here's a similar
>>> article by Juniper:
>>>
>>> https://www.juniper.net/documentation/en_US/junos/topics/task/config
>>> uration/interfaces-configuring-unicast-rpf.html
>>>
>>>
>>> -mel beckman
>>>
>>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG
>>> <nanog@nanog.org>
>>> wrote:
>>>
>>> We recently received an email notice from a group of security
>>> researchers who are looking at the feasibility of attacks using
>>> spoofed traffic. Their methodology, in broad strokes, was to send
>>> traffic to our DNS servers with a source IP that looked like it came
>>> from our network. Their attacks were successful, and they included
>>> a summary of what they found. So this message has started an
>>> internal conversation on what traffic we should be filtering and how.
>>>
>>> This security test was not about BCP 38 for ingress traffic from our
>>> customers, nor was it about BGP ingress filtering. This tested our
>>> ingress filtering from the rest of the Internet.
>>>
>>> It seems to me like we should be filtering traffic with spoofed IPs
>>> on our transit, IX, and peering links. I have done this filtering
>>> in the enterprise world extensively, and it's very helpful to keep
>>> out bad traffic. BCP 84 also discusses ingress filtering for SP's
>>> briefly and seems to advocate for it.
>>>
>>> We have about 15 IP blocks allocated to us. With a network as small
>>> as ours, I chose to go with a static ACL approach to start the
>>> conversation. I crafted a static ACL, blocking all ingress traffic
>>> sourced from any of our assigned IP ranges. I made sure to include:
>>>
>>> * Permit entries for P-t-P WAN subnets on peering links
>>>
>>> * Permit entries for IP assignments to customers running multi-homed
>>> BGP
>>>
>>> * The "permit ipv4 any any" at the end :)
>>>
>>> The questions I wanted to ask the SP community are:
>>>
>>> * What traffic filtering do you do on your transits, on IX ports,
>>> and your direct peering links?
>>>
>>> * How is it accomplished? Through static ACL or some flavor of uRPF?
>>>
>>> * If you use static ACLs, what is the administrative overhead like?
>>> What makes it easy or difficult to update?
>>>
>>> * How did you test your filters when they were implemented?
>>>
>>> Thanks a lot,
>>>
>>> -Brian
>>>
>>>

> From: Saku Ytti <saku@ytti.fi>
> Sent: Thursday, October 15, 2020 11:12 AM
>
> Hey,
>
Hey Saku,

> > All stub autonomous systems should have a simple egress ACL allowing
> only PI of their customers and their own PAs -it’s a simple ACL at each AS-Exit
> points (towards transits/peers), that’s it.
> >
> > -not sure why this isn’t the first sentence in every BCP and “security
> > bulletin”…
>
> I will venture a guess.
>
> 1) it's very specific scenario to be stubby and have downstream PI
Yes it is and in most cases it's about "I have these few /XYs from which I address my customers (eyeballs) so nothing outside of these should ever leave my AS" -as simple as that.

> 2) it won't address customers spoofing each other arbitrarily and
> customer1 spoofing as customer2 on the internet, giving large chunk of the
> utility of spoofing even with protection in place
>
Yes its not 100% effective as you pointed out,
However if every stub AS implemented this egress ACL on the few of their transit/peering links, there would be a lot less inter-AS garbage floating around.

> How do you maintain that ACL?
Well you're gonna update the ACL every time you acquire a new PA space (every now and then), or when another customer wants you route their PI (rare, simply cause you're a stub AS with mostly eyeballs).
It's one ACL you need to update on every box with AS-exit links -usually there are not that many in a stub-AS.

> Why doesn't that same mechanism allow
> ingress ACL on the customer port? Your proposal looks low utility for work
> needed.
>
Yes one should absolutely do that, but...
But considering to become a good netizen what is more work?
a) Testing and the enabling uRPF on every customer facing box or setting up precise ACLs on every customer facing port, and then maintaining all that?
b) Gathering all your PAs (potentially PIs) (hint: show bgp nei x.x.x.x advertised routes) crafting an ACL and apply it on several peering/transit links?
One of them is couple of weeks work and one is an afternoon job.

adam

On Thu, 15 Oct 2020 at 15:14, <adamv0025@netconsultings.com> wrote:


> Yes one should absolutely do that, but...
> But considering to become a good netizen what is more work?
> a) Testing and the enabling uRPF on every customer facing box or setting up precise ACLs on every customer facing port, and then maintaining all that?
> b) Gathering all your PAs (potentially PIs) (hint: show bgp nei x.x.x.x advertised routes) crafting an ACL and apply it on several peering/transit links?
> One of them is couple of weeks work and one is an afternoon job.

I am not fan of uRPF, expensive for what it does. But I don't view it
as an alternative here, I view it as either adding an ACE on all
egresses on egress direction or adding ACE on the ingress where
customer is on ingress direction.

To me these options seem equally complex but the latter one seems superior.

--
++ytti

We deploy urpf strict on all customer end-host and broadband circuits. In
this scenario urpf = ingress acl I don't have to think about.

We deploy urpf loose on all customer multihomed DIA circuits. I dont this
makes sense - ingress packet acl would be more sane.

Any flavour of urpf on upstream transit or peering would be challenging.
Ingress packet acl dropping source = own+customer prefix might make sense
depending on your AS topology.

You might argue that ingress packet acl would be operationally simpler on
customer and upstream, as you could cover all scenarios.

On Thu, Oct 15, 2020 at 10:05 AM Saku Ytti <saku@ytti.fi> wrote:

> On Thu, 15 Oct 2020 at 15:14, <adamv0025@netconsultings.com> wrote:
>
>
> > Yes one should absolutely do that, but...
> > But considering to become a good netizen what is more work?
> > a) Testing and the enabling uRPF on every customer facing box or setting
> up precise ACLs on every customer facing port, and then maintaining all
> that?
> > b) Gathering all your PAs (potentially PIs) (hint: show bgp nei x.x.x.x
> advertised routes) crafting an ACL and apply it on several peering/transit
> links?
> > One of them is couple of weeks work and one is an afternoon job.
>
> I am not fan of uRPF, expensive for what it does. But I don't view it
> as an alternative here, I view it as either adding an ACE on all
> egresses on egress direction or adding ACE on the ingress where
> customer is on ingress direction.
>
> To me these options seem equally complex but the latter one seems superior.
>
> --
> ++ytti
>


--
Tim:>

On Thu, 15 Oct 2020 at 17:22, Tim Durack <tdurack@gmail.com> wrote:


> We deploy urpf strict on all customer end-host and broadband circuits. In this scenario urpf = ingress acl I don't have to think about.

But you have to think about what prefixes a customer has. If BGP you
need to generate prefix-list, if static you need to generate a static
route. As you already have to know and manage this information, what
is the incremental cost to also emit an ACL?

--
++ytti

Saku Ytti wrote on 15/10/2020 15:29:
> But you have to think about what prefixes a customer has. If BGP you
> need to generate prefix-list, if static you need to generate a static
> route. As you already have to know and manage this information, what
> is the incremental cost to also emit an ACL?

the unfortunate reality is that many networks are run by CLI jockeys, so
the incremental cost of this can be high. There are no good
general-purpose networking sources of truth, which means that usually
provisioning databases need to be highly customised, which is only worth
it if the scale merits it.

Nick

> From: Saku Ytti <saku@ytti.fi>
> Sent: Thursday, October 15, 2020 3:30 PM
>
> On Thu, 15 Oct 2020 at 17:22, Tim Durack <tdurack@gmail.com> wrote:
>
>
> > We deploy urpf strict on all customer end-host and broadband circuits. In
> this scenario urpf = ingress acl I don't have to think about.
>
> But you have to think about what prefixes a customer has. If BGP you need
> to generate prefix-list, if static you need to generate a static route. As you
> already have to know and manage this information, what is the incremental
> cost to also emit an ACL?
>
Actually ideally there would be a feature/knob to automatically sync BGP (and static routes) with packet filters.

adam

On Thu, Oct 15, 2020 at 10:30 AM Saku Ytti <saku@ytti.fi> wrote:

> On Thu, 15 Oct 2020 at 17:22, Tim Durack <tdurack@gmail.com> wrote:
>
>
> > We deploy urpf strict on all customer end-host and broadband circuits.
> In this scenario urpf = ingress acl I don't have to think about.
>
> But you have to think about what prefixes a customer has. If BGP you
> need to generate prefix-list, if static you need to generate a static
> route. As you already have to know and manage this information, what
> is the incremental cost to also emit an ACL?
>
> --
> ++ytti
>

"You might argue that ingress packet acl would be operationally simpler on
customer and upstream, as you could cover all scenarios."

Although for a static customer urpf is hard to beat...

--
Tim:>

Once upon a time, adamv0025@netconsultings.com <adamv0025@netconsultings.com> said:
> Actually ideally there would be a feature/knob to automatically sync BGP (and static routes) with packet filters.

Junos has prefix-lists that can be referenced in both BGP policy and
firewall statements.
--
Chris Adams <cma@cmadams.net>

Speaking as an ISP:
    Most of the ISP networks I manage are multi-homed, and I don't
think uRPF provides the knobs to ensure legitimate traffic doesn't get
dropped in some cases, so we use static ACLs at the upstream edge on
ingress (and egress). These need updated any time new IP space is added
to the network (not very often).
        Ingress ACL: Discard if source or destination is a bogon, RFC
1912, RFC1918; discard any traffic sourced from our own IP space
(spoofed); discard any traffic that is not destined for our IP space
(bad/mis-routed).
        Egress ACL: Discard if source or destination is a bogon, RFC
1912, RFC1918; discard any traffic that is not sourced from our IP space
(spoofed).
        I believe in a policy of non-blocking and being net neutral,
but if any TCP/UDP ports, IP protocols, or IP options are blocked you
might add them to the above ACLs.

    On customer facing ports we use uRPF strict. Why? It's easy (one
line to implement, zero maintenance, and it works well on our Cisco
ASR1k/9k platforms)! Our customers are all single homed.

For a single homed enterprises, service providers, and end-users I'd
recommend uRPF strict. Why? Again, it's dead simple. I would love to see
this be the default on all home, prosumer, and branch office oriented
platform. Linux does this by default with the rp_filter kernel option. I
suspect that networking gear based on Linux probably leaves this at the
default setting (strict mode).

In practice I don't know that I've ever used uRPF loose mode. ACLs have
counters to verify they are working. I might have confirmed uRPF was
working as intended the first time I implemented it, but otherwise I've
used ACLs often enough to trust they are working as configured and trust
the same for uRPF.

--Blake


On 10/13/2020 5:13 PM, Brian Knight via NANOG wrote:
> We recently received an email notice from a group of security
> researchers who are looking at the feasibility of attacks using
> spoofed traffic.  Their methodology, in broad strokes, was to send
> traffic to our DNS servers with a source IP that looked like it came
> from our network.  Their attacks were successful, and they included a
> summary of what they found.  So this message has started an internal
> conversation on what traffic we should be filtering and how.
>
> This security test was not about BCP 38 for ingress traffic from our
> customers, nor was it about BGP ingress filtering.  This tested our
> ingress filtering from the rest of the Internet.
>
> It seems to me like we should be filtering traffic with spoofed IPs on
> our transit, IX, and peering links.  I have done this filtering in the
> enterprise world extensively, and it's very helpful to keep out bad
> traffic.  BCP 84 also discusses ingress filtering for SP's briefly and
> seems to advocate for it.
>
> We have about 15 IP blocks allocated to us.  With a network as small
> as ours, I chose to go with a static ACL approach to start the
> conversation.  I crafted a static ACL, blocking all ingress traffic
> sourced from any of our assigned IP ranges.  I made sure to include:
>
> * Permit entries for P-t-P WAN subnets on peering links
> * Permit entries for IP assignments to customers running multi-homed BGP
> * The "permit ipv4 any any" at the end :)
>
> The questions I wanted to ask the SP community are:
>
> * What traffic filtering do you do on your transits, on IX ports, and
> your direct peering links?
> * How is it accomplished?  Through static ACL or some flavor of uRPF?
> * If you use static ACLs, what is the administrative overhead like? 
> What makes it easy or difficult to update?
> * How did you test your filters when they were implemented?
>
> Thanks a lot,
>
> -Brian

> Chris Adams
> Sent: Thursday, October 15, 2020 3:59 PM
>
> Once upon a time, adamv0025@netconsultings.com
> <adamv0025@netconsultings.com> said:
> > Actually ideally there would be a feature/knob to automatically sync BGP
> (and static routes) with packet filters.
>
> Junos has prefix-lists that can be referenced in both BGP policy and
firewall
> statements.
>
What I mean was a firewall filter that would get updated with accordance to
prefixes received/advertised via BGP, even better if this was a default
behaviour.


adam

> On Oct 14, 2020, at 3:34 PM, Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
>
> I think he means packet captures from an example, voluntarily-tested recursive nameserver subject to this attack.


Thanks. We have updated all the report pages with a self-test tool specific to the network associated with the report. This should allow a network admin that received our report to check whether or not the condition still exists and to perform a packet capture from whatever vantage point they want in their network.

A more general tool (i.e., for anyone to use) will be made available in the future.

Cheers,
Casey

>
>
> On Wed, Oct 14, 2020 at 11:53 AM Casey Deccio <casey@deccio.net <mailto:casey@deccio.net>> wrote:
> Hi Bryan,
>
> > On Oct 14, 2020, at 12:43 PM, Bryan Holloway <bryan@shout.net <mailto:bryan@shout.net>> wrote:
> >
> > I too would like to know more about their methodology
>
> We've written up our methodology and results in a paper that will be available in a few weeks. Happy to post it here if folks are interested. Obviously, no networks are individually identified; it's all aggregate.
>
> Also, we're working on a self-test tool, but it's not quite ready yet. Sorry.
>
> > and actual tangibles ideally in the form of PCAPs.
>
> What do you mean by "tangibles in the form of PCAPs"?
>
> Casey

Thanks to the folks who responded to my messages on and off-list. A
couple of folks have asked me to summarize the responses that I
received.

* Static ACL is currently the best way to protect a multi-homed network.
Loose RPF may be used if bogon filtering is more important, but it does
not provide anti-spoofing security.

* Protect your infrastructure subnets with the ingress ACL [BCP 84 sec
3.2]. Loopbacks and point-to-point circuits can benefit from this. In
the draft ACL, for example, I permit ICMP and traceroute over UDP, and
block all else.

* Do an egress ACL also, to prevent clutter from reaching the rest of
the 'Net. Permit only your aggregate and customer prefixes going
outbound.

* As I worked through putting the ACLs together, I found that if one
implements an egress ACL, then customer prefixes must be enumerated
anyway. Once those are in an object group, it's easy to add an entry to
the ingress ACL permitting traffic destined to customer PI space and
aggregate space. Seems better than just permitting all traffic in.

Our ACLs, both v4 and v6, now look like the following:

Ingress

* Deny to and from bogon networks, where bogon is either source or dest
* Permit to and from WAN PtP subnets
* For IPv6, also permit link-local IPs (fe80::/10)
* Deny to and from multicast ranges 224.0.0.0/4 and ff00::/8
* Permit ICMP / traceroute over UDP to infrastructure
* Deny all other traffic to infrastructure
* Permit from customer PI / PA space
* Deny from originated aggregate space
* Permit all traffic to customer PI / PA space
* Permit all traffic to aggregate space
* Deny any any

Egress

* Deny to and from bogon networks
* Permit to and from WAN PtP subnets
* For IPv6, also permit link-local IPs
* Deny to and from multicast range
* Permit all traffic from customer PI / PA space
* Permit all traffic from aggregate space
* Deny any any

We have started implementing the ACLs by blocking the bogon traffic
only. The other deny rules are set up as permit rules for now with
logging turned on. I'll review matching traffic before I switch the
rules to deny.

Future work also includes automating the updates to the object groups
via IRR.

BTW, Team Cymru didn't have any guidance around IPv6 bogons, so I put
together the below object group based on the IANA IPv6 allocation list:
https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.xhtml.
Obviously this is only for space not yet allocated to RIRs.

object-group network ipv6 IPV6-BOGON
description Invalid IPV6 networks
::/3
4000::/3
6000::/3
8000::/3
a000::/3
c000::/3
e000::/4
f000::/5
f800::/6
fc00::/7
fe00::/9
fec0::/10
exit

Thanks,

-Brian



On 2020-10-14 17:43, Brian Knight wrote:
> So I have put together what I think is a reasonable and complete ACL.
> From my time in the enterprise world, I know that a good ingress ACL
> filters out traffic sourcing from:
>
> * Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc
> (well-documented in
> https://team-cymru.com/community-services/bogon-reference/bogon-reference-http/)
> * RIR-assigned blocks I am announcing to the rest of the world
>
> However, I recognized a SP-specific case where we could receive
> legitimate traffic sourcing from our own IP blocks: customers running
> multi-homed BGP where we have assigned PA space to them. So I added
> "permit" statements for traffic sourcing from these blocks.
>
> Also, we have direct peering links that are numbered within our
> assigned prefixes. So we can use the same ACL with these peer
> interfaces and continue to have BGP work, I added "permit" statements
> for these point-to-point subnets.
>
> So the order of the statements is:
>
> * Permit where source is direct peer PtP networks
> * Permit where source is BGP customer PA prefix
> * Deny where source is bogon
> * Deny where source is our advertised prefixes
> * Permit all other traffic
>
> I considered BGP customer PI prefixes to be out of scope for ingress
> filtering, since the customer is likely to be multi-homing. Should we
> consider filtering them?
>
> The Team Cymru Secure IOS Template
> [https://www.cymru.com/Documents/secure-ios-template.html] also
> references an ICMP fragment drop entry on the ingress ACL. I think
> that's good for an enterprise network, but as an SP, I'm very hesitant
> to include this. Is this included in anyone else's transit / peer /
> IX ACL?
>
> Is there anything else that I'm not thinking of?
>
> Thanks,
>
> -Brian
>
>
> On 2020-10-14 09:25, Brian Knight via NANOG wrote:
>> Hi Marcos,
>>
>> Thanks for your reply. But I am looking for guidance on traffic
>> filtering, not BGP prefix filtering.
>>
>> I have looked at BCP 84, and it's a good overview of the methods
>> available to an ISP. My questions are more operational in nature and
>> are not covered by the document. Of the choices presented in BCP 84,
>> what do folks really use? If it's an ACL, what challenges have there
>> been with updates? etc.
>>
>> -Brian
>>
>>
>> On 2020-10-13 18:52, Marcos Manoni wrote:
>>> Hi, Brian
>>>
>>> Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks
>>> (Updated
>>> by: RFC8704 Enhanced Feasible-Path uRPF).
>>>
>>> Ingress Access Lists require typically manual maintenance, but
>>> are
>>> the most bulletproof when done properly; typically, ingress
>>> access
>>> lists are best fit between the edge and the ISP when the
>>> configuration is not too dynamic if strict RPF is not an option,
>>> between ISPs if the number of used prefixes is low, or as an
>>> additional layer of protection
>>>
>>>
>>> Ingress filters Best Practices
>>> https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
>>> *Don’t accept BOGON ASNs
>>> *Don’t accept BOGON prefixes
>>> *Don’t accept your own prefix
>>> *Don’t accept default (unless you requested it)
>>> *Don’t accept prefixes that are too specific
>>> *Don’t accept if AS Path is too long
>>> *Create filters based on Internet Routing Registries
>>>
>>> And also BGP Best Current Practices by Philip Smith
>>> http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
>>>
>>> Regards.
>>>
>>> El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
>>> (<nanog@nanog.org>) escribió:
>>>>
>>>> Hi Mel,
>>>>
>>>> My understanding of uRPF is:
>>>>
>>>> * Strict mode will permit a packet only if there is a route for the
>>>> source IP in the RIB, and that route points to the interface where
>>>> the packet was received
>>>>
>>>> * Loose mode will permit a packet if there is a route for the source
>>>> IP in the RIB. It does not matter where the route is pointed.
>>>>
>>>> Strict mode won't work for us, because with our multi-homed transits
>>>> and IX peers, we will almost certainly drop a legitimate packet
>>>> because the best route is through another transit.
>>>>
>>>> Loose mode won't work for us, because all of our own prefixes are in
>>>> our RIB, and thus the uRPF check on a transit would never block
>>>> anything.
>>>>
>>>> Or am I missing something?
>>>>
>>>> Thanks,
>>>>
>>>> -Brian
>>>>
>>>> On 2020-10-13 17:22, Mel Beckman wrote:
>>>>
>>>> You can also use Unicast Reverse Path Forwarding. RPF is more
>>>> efficient than ACLs, and has the added advantage of not requiring
>>>> maintenance. In a nutshell, if your router has a route to a prefix
>>>> in its local RIB, then incoming packets from a border interface
>>>> having a matching source IP will be dropped.
>>>>
>>>> RPF has knobs and dials to make it work for various ISP
>>>> environments. Implement it carefully (as is be standing next to the
>>>> router involved :)
>>>>
>>>> Here's a Cisco brief on the topic:
>>>>
>>>>
>>>> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> I think all router vendors support this feature. Here's a similar
>>>> article by Juniper:
>>>>
>>>> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
>>>>
>>>>
>>>> -mel beckman
>>>>
>>>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG
>>>> <nanog@nanog.org> wrote:
>>>>
>>>> We recently received an email notice from a group of security
>>>> researchers who are looking at the feasibility of attacks using
>>>> spoofed traffic. Their methodology, in broad strokes, was to send
>>>> traffic to our DNS servers with a source IP that looked like it came
>>>> from our network. Their attacks were successful, and they included
>>>> a summary of what they found. So this message has started an
>>>> internal conversation on what traffic we should be filtering and
>>>> how.
>>>>
>>>> This security test was not about BCP 38 for ingress traffic from our
>>>> customers, nor was it about BGP ingress filtering. This tested our
>>>> ingress filtering from the rest of the Internet.
>>>>
>>>> It seems to me like we should be filtering traffic with spoofed IPs
>>>> on our transit, IX, and peering links. I have done this filtering
>>>> in the enterprise world extensively, and it's very helpful to keep
>>>> out bad traffic. BCP 84 also discusses ingress filtering for SP's
>>>> briefly and seems to advocate for it.
>>>>
>>>> We have about 15 IP blocks allocated to us. With a network as small
>>>> as ours, I chose to go with a static ACL approach to start the
>>>> conversation. I crafted a static ACL, blocking all ingress traffic
>>>> sourced from any of our assigned IP ranges. I made sure to include:
>>>>
>>>> * Permit entries for P-t-P WAN subnets on peering links
>>>>
>>>> * Permit entries for IP assignments to customers running multi-homed
>>>> BGP
>>>>
>>>> * The "permit ipv4 any any" at the end :)
>>>>
>>>> The questions I wanted to ask the SP community are:
>>>>
>>>> * What traffic filtering do you do on your transits, on IX ports,
>>>> and your direct peering links?
>>>>
>>>> * How is it accomplished? Through static ACL or some flavor of
>>>> uRPF?
>>>>
>>>> * If you use static ACLs, what is the administrative overhead like?
>>>> What makes it easy or difficult to update?
>>>>
>>>> * How did you test your filters when they were implemented?
>>>>
>>>> Thanks a lot,
>>>>
>>>> -Brian
>>>>
>>>>

term blocked-ports {
from {
protocol [ tcp udp ];
first-fragment;
destination-port
[. 0 sunrpc 135 netbios-ns netbios-dgm netbios-ssn 111 445 syslog 11211];
}
then {
sample;
discard;
}
}

and i block all external access to weak devices such as switches, pdus,
ipmi, ...

randy

I took a slightly different approach for my mental exercise, expressed in
IOS pigeon:


object-group ip address AS65001

192.0.2.0 255.255.255.0

end


object-group v6-network AS65001

2001:DB8::/32

end


object-group ip address TwentyFiveGigE1/0/1

192.0.2.0 255.255.255.254

end


object-group v6-network TwentyFiveGigE1/0/1

FE80::/10

2001:DB8::/127

end


ip access-list extended TwentyFiveGigE1/0/1_IPV4_IN

permit ip addrgroup TwentyFiveGigE1/0/1 any

deny ip addrgroup AS65001 any

permit ip any any

end


ipv6 access-list TwentyFiveGigE1/0/1_IPV6_IN

permit ipv6 object-group TwentyFiveGigE1/0/1 any

deny ipv6 object-group AS65001 any

permit ipv6 any any

end


interface TwentyFiveGigE1/0/1

ip access-group TwentyFiveGigE1/0/1_IPV4_IN in

ipv6 traffic-filter TwentyFiveGigE1/0/1_IPV6_IN in

end


I believe this is the minimum necessary to protect your AS from your
netblock(s) ingress. Note: there may be technical and business reasons why
you need to permit your netblock(s) ingress to your AS.


I believe this concept could be used on any EBGP Inter-AS link, including
peering addressed out of your own netblock.


I remain unconvinced, but shrewd is the one who sees the calamity...


Tim:>

On Mon, Oct 19, 2020 at 8:40 PM Brian Knight via NANOG <nanog@nanog.org>
wrote:

> Thanks to the folks who responded to my messages on and off-list. A
> couple of folks have asked me to summarize the responses that I
> received.
>
> * Static ACL is currently the best way to protect a multi-homed network.
> Loose RPF may be used if bogon filtering is more important, but it does
> not provide anti-spoofing security.
>
> * Protect your infrastructure subnets with the ingress ACL [BCP 84 sec
> 3.2]. Loopbacks and point-to-point circuits can benefit from this. In
> the draft ACL, for example, I permit ICMP and traceroute over UDP, and
> block all else.
>
> * Do an egress ACL also, to prevent clutter from reaching the rest of
> the 'Net. Permit only your aggregate and customer prefixes going
> outbound.
>
> * As I worked through putting the ACLs together, I found that if one
> implements an egress ACL, then customer prefixes must be enumerated
> anyway. Once those are in an object group, it's easy to add an entry to
> the ingress ACL permitting traffic destined to customer PI space and
> aggregate space. Seems better than just permitting all traffic in.
>
> Our ACLs, both v4 and v6, now look like the following:
>
> Ingress
>
> * Deny to and from bogon networks, where bogon is either source or dest
> * Permit to and from WAN PtP subnets
> * For IPv6, also permit link-local IPs (fe80::/10)
> * Deny to and from multicast ranges 224.0.0.0/4 and ff00::/8
> * Permit ICMP / traceroute over UDP to infrastructure
> * Deny all other traffic to infrastructure
> * Permit from customer PI / PA space
> * Deny from originated aggregate space
> * Permit all traffic to customer PI / PA space
> * Permit all traffic to aggregate space
> * Deny any any
>
> Egress
>
> * Deny to and from bogon networks
> * Permit to and from WAN PtP subnets
> * For IPv6, also permit link-local IPs
> * Deny to and from multicast range
> * Permit all traffic from customer PI / PA space
> * Permit all traffic from aggregate space
> * Deny any any
>
> We have started implementing the ACLs by blocking the bogon traffic
> only. The other deny rules are set up as permit rules for now with
> logging turned on. I'll review matching traffic before I switch the
> rules to deny.
>
> Future work also includes automating the updates to the object groups
> via IRR.
>
> BTW, Team Cymru didn't have any guidance around IPv6 bogons, so I put
> together the below object group based on the IANA IPv6 allocation list:
>
> https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.xhtml.
>
> Obviously this is only for space not yet allocated to RIRs.
>
> object-group network ipv6 IPV6-BOGON
> description Invalid IPV6 networks
> ::/3
> 4000::/3
> 6000::/3
> 8000::/3
> a000::/3
> c000::/3
> e000::/4
> f000::/5
> f800::/6
> fc00::/7
> fe00::/9
> fec0::/10
> exit
>
> Thanks,
>
> -Brian
>
>
>
> On 2020-10-14 17:43, Brian Knight wrote:
> > So I have put together what I think is a reasonable and complete ACL.
> > From my time in the enterprise world, I know that a good ingress ACL
> > filters out traffic sourcing from:
> >
> > * Bogon blocks, like 0.0.0.0/8, 127.0.0.0/8, RFC1918 space, etc
> > (well-documented in
> >
> https://team-cymru.com/community-services/bogon-reference/bogon-reference-http/
> )
> > * RIR-assigned blocks I am announcing to the rest of the world
> >
> > However, I recognized a SP-specific case where we could receive
> > legitimate traffic sourcing from our own IP blocks: customers running
> > multi-homed BGP where we have assigned PA space to them. So I added
> > "permit" statements for traffic sourcing from these blocks.
> >
> > Also, we have direct peering links that are numbered within our
> > assigned prefixes. So we can use the same ACL with these peer
> > interfaces and continue to have BGP work, I added "permit" statements
> > for these point-to-point subnets.
> >
> > So the order of the statements is:
> >
> > * Permit where source is direct peer PtP networks
> > * Permit where source is BGP customer PA prefix
> > * Deny where source is bogon
> > * Deny where source is our advertised prefixes
> > * Permit all other traffic
> >
> > I considered BGP customer PI prefixes to be out of scope for ingress
> > filtering, since the customer is likely to be multi-homing. Should we
> > consider filtering them?
> >
> > The Team Cymru Secure IOS Template
> > [https://www.cymru.com/Documents/secure-ios-template.html] also
> > references an ICMP fragment drop entry on the ingress ACL. I think
> > that's good for an enterprise network, but as an SP, I'm very hesitant
> > to include this. Is this included in anyone else's transit / peer /
> > IX ACL?
> >
> > Is there anything else that I'm not thinking of?
> >
> > Thanks,
> >
> > -Brian
> >
> >
> > On 2020-10-14 09:25, Brian Knight via NANOG wrote:
> >> Hi Marcos,
> >>
> >> Thanks for your reply. But I am looking for guidance on traffic
> >> filtering, not BGP prefix filtering.
> >>
> >> I have looked at BCP 84, and it's a good overview of the methods
> >> available to an ISP. My questions are more operational in nature and
> >> are not covered by the document. Of the choices presented in BCP 84,
> >> what do folks really use? If it's an ACL, what challenges have there
> >> been with updates? etc.
> >>
> >> -Brian
> >>
> >>
> >> On 2020-10-13 18:52, Marcos Manoni wrote:
> >>> Hi, Brian
> >>>
> >>> Check RFC3704/BCP84 Ingress Filtering for Multihomed Networks
> >>> (Updated
> >>> by: RFC8704 Enhanced Feasible-Path uRPF).
> >>>
> >>> Ingress Access Lists require typically manual maintenance, but
> >>> are
> >>> the most bulletproof when done properly; typically, ingress
> >>> access
> >>> lists are best fit between the edge and the ISP when the
> >>> configuration is not too dynamic if strict RPF is not an option,
> >>> between ISPs if the number of used prefixes is low, or as an
> >>> additional layer of protection
> >>>
> >>>
> >>> Ingress filters Best Practices
> >>>
> https://www.ripe.net/support/training/material/bgp-operations-and-security-training-course/BGP-Slides-Single.pdf
> >>> *Don’t accept BOGON ASNs
> >>> *Don’t accept BOGON prefixes
> >>> *Don’t accept your own prefix
> >>> *Don’t accept default (unless you requested it)
> >>> *Don’t accept prefixes that are too specific
> >>> *Don’t accept if AS Path is too long
> >>> *Create filters based on Internet Routing Registries
> >>>
> >>> And also BGP Best Current Practices by Philip Smith
> >>> http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf
> >>>
> >>> Regards.
> >>>
> >>> El mar., 13 oct. 2020 a las 19:52, Brian Knight via NANOG
> >>> (<nanog@nanog.org>) escribió:
> >>>>
> >>>> Hi Mel,
> >>>>
> >>>> My understanding of uRPF is:
> >>>>
> >>>> * Strict mode will permit a packet only if there is a route for the
> >>>> source IP in the RIB, and that route points to the interface where
> >>>> the packet was received
> >>>>
> >>>> * Loose mode will permit a packet if there is a route for the source
> >>>> IP in the RIB. It does not matter where the route is pointed.
> >>>>
> >>>> Strict mode won't work for us, because with our multi-homed transits
> >>>> and IX peers, we will almost certainly drop a legitimate packet
> >>>> because the best route is through another transit.
> >>>>
> >>>> Loose mode won't work for us, because all of our own prefixes are in
> >>>> our RIB, and thus the uRPF check on a transit would never block
> >>>> anything.
> >>>>
> >>>> Or am I missing something?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> -Brian
> >>>>
> >>>> On 2020-10-13 17:22, Mel Beckman wrote:
> >>>>
> >>>> You can also use Unicast Reverse Path Forwarding. RPF is more
> >>>> efficient than ACLs, and has the added advantage of not requiring
> >>>> maintenance. In a nutshell, if your router has a route to a prefix
> >>>> in its local RIB, then incoming packets from a border interface
> >>>> having a matching source IP will be dropped.
> >>>>
> >>>> RPF has knobs and dials to make it work for various ISP
> >>>> environments. Implement it carefully (as is be standing next to the
> >>>> router involved :)
> >>>>
> >>>> Here's a Cisco brief on the topic:
> >>>>
> >>>>
> >>>>
> https://tools.cisco.com/security/center/resources/unicast_reverse_path_forwarding
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> I think all router vendors support this feature. Here's a similar
> >>>> article by Juniper:
> >>>>
> >>>>
> https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html
> >>>>
> >>>>
> >>>> -mel beckman
> >>>>
> >>>> On Oct 13, 2020, at 3:15 PM, Brian Knight via NANOG
> >>>> <nanog@nanog.org> wrote:
> >>>>
> >>>> We recently received an email notice from a group of security
> >>>> researchers who are looking at the feasibility of attacks using
> >>>> spoofed traffic. Their methodology, in broad strokes, was to send
> >>>> traffic to our DNS servers with a source IP that looked like it came
> >>>> from our network. Their attacks were successful, and they included
> >>>> a summary of what they found. So this message has started an
> >>>> internal conversation on what traffic we should be filtering and
> >>>> how.
> >>>>
> >>>> This security test was not about BCP 38 for ingress traffic from our
> >>>> customers, nor was it about BGP ingress filtering. This tested our
> >>>> ingress filtering from the rest of the Internet.
> >>>>
> >>>> It seems to me like we should be filtering traffic with spoofed IPs
> >>>> on our transit, IX, and peering links. I have done this filtering
> >>>> in the enterprise world extensively, and it's very helpful to keep
> >>>> out bad traffic. BCP 84 also discusses ingress filtering for SP's
> >>>> briefly and seems to advocate for it.
> >>>>
> >>>> We have about 15 IP blocks allocated to us. With a network as small
> >>>> as ours, I chose to go with a static ACL approach to start the
> >>>> conversation. I crafted a static ACL, blocking all ingress traffic
> >>>> sourced from any of our assigned IP ranges. I made sure to include:
> >>>>
> >>>> * Permit entries for P-t-P WAN subnets on peering links
> >>>>
> >>>> * Permit entries for IP assignments to customers running multi-homed
> >>>> BGP
> >>>>
> >>>> * The "permit ipv4 any any" at the end :)
> >>>>
> >>>> The questions I wanted to ask the SP community are:
> >>>>
> >>>> * What traffic filtering do you do on your transits, on IX ports,
> >>>> and your direct peering links?
> >>>>
> >>>> * How is it accomplished? Through static ACL or some flavor of
> >>>> uRPF?
> >>>>
> >>>> * If you use static ACLs, what is the administrative overhead like?
> >>>> What makes it easy or difficult to update?
> >>>>
> >>>> * How did you test your filters when they were implemented?
> >>>>
> >>>> Thanks a lot,
> >>>>
> >>>> -Brian
> >>>>
> >>>>
>


--
Tim:>

Hi,

You are missing some IPv6 bogons prefixes. Check:
http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf (page 51-53)
http://bgpfilterguide.nlnog.net/guides/bogon_prefixes/#configuration-examples-ipv6

Regards.

> On 15 Oct 2020, at 05:43, Brian Knight via NANOG <nanog@nanog.org> wrote:
>
> I think that's good for an enterprise network, but as an SP, I'm very hesitant to include this. Is this included in anyone else's transit / peer / IX ACL?

This must not be done on peering links and on transit networks generally, as it breaks EDNS0, & everything that depends upon it, as well as some games, VoIP applications, etc.

For consumer eyeball access networks only, making use of flow telemetry to analyze non-initial fragments destined for *those networks only, and excepting both one’s own recursive DNS server farms and well-known/well-operated public DNS recursive farms*, one can determine the normal rate of non-initial fragments in that very specific context and utilize QoS to police it down, leaving plenty of headroom for upticks in legitimate traffic.

And with regards to disallowing one’s own address space except in special topological circumstances, it’s great to see that this initial topic has sparked renewed interest in both ingress and egress filtering. It’s highly laudable, and some good example are being posted and useful discussion taking place.

It should be noted, however, that filtering one’s own prefixes at one’s peering edge in the more general cases isn’t a new concept; this has been a BCP recommendation for more than 20 years. For example, it’s referenced on p.75 of this .pdf slide deck on the topic of infrastructure self-protection, which was last revised 11 years ago:

<https://app.box.com/s/osk4po8ietn1zrjjmn8b>

From the way this ’new’ set of findings has been promulgated, it sounds as if the authors of the paper didn’t necessarily understand that this is a longstanding recommendation. That doesn’t in any way detract from the value of their study, mind; and perhaps they were aware, and that information simply hasn’t been communicated in this context.

Also, a more proximate risk than DNS cache-poisoning or the specific, impractical, never-seen-in-the-wild DNS DDoS attack methodology cited, is for operators who aren’t filtering their own prefixes on ingres, and who’re also relying solely on iACLs to prevent off-net access to their recursive DNS server farms, thus allowing their abuse in DNS reflection/amplification attacks against their own infrastructure and access customers from outside their network.

Filtering one’s own prefixes on ingress whenever possible and to the degree of granularity possible, along with making use not only of iACLs but on-server configurations to disallow off-network abuse of recursive DNS servers, are highly recommended.

--------------------------------------------
Roland Dobbins <roland.dobbins@netscout.com>

Might filtering port 11211 like that not risk blocking random connections,
where the operating system picked that port as source, which then becomes
destination on the reply packets?

tir. 20. okt. 2020 07.19 skrev Randy Bush <randy@psg.com>:

> term blocked-ports {
> from {
> protocol [ tcp udp ];
> first-fragment;
> destination-port
> [. 0 sunrpc 135 netbios-ns netbios-dgm netbios-ssn 111 445
> syslog 11211];
> }
> then {
> sample;
> discard;
> }
> }
>
> and i block all external access to weak devices such as switches, pdus,
> ipmi, ...
>
> randy
>

Randy, thank you for the reminder to look also at what services (L4
ports) should be generally blocked.

As I was implementing a similar rule for logging purposes, I discovered
an oddity with $VENDOR_C_XR ACLs. I created the following:

object-group port TCPUDP-BLOCKED
eq 0
eq sunrpc
eq 445
range 137 139
exit

ipv4 access-list IPV4-INET-IN
10 remark BCP 84 for transits, IX, and peering
101 remark *** Block bogon networks as src or dest ***
110 deny ipv4 net-group IPV4-BOGON any
111 deny ipv4 any net-group IPV4-BOGON
201 remark *** Blocked protocols PERMIT FOR NOW ***
210 permit udp any port-group TCPUDP-BLOCKED any log
211 permit udp any any port-group TCPUDP-BLOCKED log
212 permit tcp any port-group TCPUDP-BLOCKED any log
213 permit tcp any any port-group TCPUDP-BLOCKED log
[snip]

ipv4 access-list IPV4-INET-OUT
10 remark BCP 84 for transits, IX, and peering
101 remark *** Block bogon networks as src or dest ***
110 deny ipv4 net-group IPV4-BOGON any
111 deny ipv4 any net-group IPV4-BOGON
201 remark *** Blocked protocols PERMIT FOR NOW ***
210 permit udp any port-group TCPUDP-BLOCKED any log
211 permit udp any any port-group TCPUDP-BLOCKED log
212 permit tcp any port-group TCPUDP-BLOCKED any log
213 permit tcp any any port-group TCPUDP-BLOCKED log
[snip]

After I did this, logs on our syslog server started growing like crazy.
It was full of entries like:

2020-10-21T01:47:17-05:00,info,RP/0/RSP1/CPU0:Oct 21 01:47:17.972 CDT:
ipv4_acl_mgr[305]: %ACL-IPV4_ACL-6-IPACCESSLOGP : access-list
IPV4-INET-OUT (210) permit udp on.net.ip.adr(0) -> off.net.ip.adr(0), 5
packets
2020-10-21T02:01:08-05:00,info,RP/0/RSP0/CPU0:Oct 21 02:01:08.490 CDT:
ipv4_acl_mgr[263]: %ACL-IPV4_ACL-6-IPACCESSLOGP : access-list
IPV4-INET-IN (210) permit udp off.net.ip.adr(0) -> on.net.ip.adr(0), 58
packets

After wondering why in the world my customers were sending so much data
on port 0, I found a few different sources saying that port 0 is
commonly used in place of valid information when dealing with fragments.
Turns out that $VENDOR_C_XR does this too.

It wasn't clear why fragments would be matching that rule until I found
the right vendor doc. The router will pass IP fragments with a "permit"
ACL line as long as that fragment's layer 3 info matches the layer 3
information in the ACL. The router logs the packet similar the above:
L4 protocol with source and dest port = 0. From the doc:

-----

For an access-list entry containing Layer 3 and Layer 4 information:
• The entry is applied to non-fragmented packets and initial fragments.
• If the entry matches and is a permit statement, the packet or
fragment is permitted.
• If the entry matches and is a deny statement, the packet or fragment
is denied.

The entry is also applied to non-initial fragments in the following
manner. Because non-initial fragments contain only Layer 3 information,
only the Layer 3 portion of an access-list entry can be applied. If the
Layer 3 portion of the access-list entry matches, and
• If the entry is a permit statement, the non-initial fragment is
permitted.
• If the entry is a deny statement, the next access-list entry is
processed.
The deny statements are handled differently for non-initial
fragments versus non-fragmented or initial fragments.

-----

Since my rule's L3 info was permit any source to any destination, any IP
fragment would match the rule, be passed, and be logged. The solution
was to add rules explicitly permitting fragments above the layer 4
rules:

ipv4 access-list IPV4-INET-IN
10 remark BCP 84 for transits, IX, and peering
101 remark *** Block bogon networks as src or dest ***
110 deny ipv4 net-group IPV4-BOGON any
111 deny ipv4 any net-group IPV4-BOGON
201 remark *** Blocked protocols PERMIT FOR NOW ***
203 permit ipv4 net-group IPV4-CUST any fragments
204 permit ipv4 net-group IPV4-BACKDOOR-HOSTS any fragments
205 permit ipv4 any net-group IPV4-BGP-AGG fragments
206 permit ipv4 any net-group IPV4-CUST fragments
210 permit udp any port-group TCPUDP-BLOCKED any log
211 permit udp any any port-group TCPUDP-BLOCKED log
212 permit tcp any port-group TCPUDP-BLOCKED any log
213 permit tcp any any port-group TCPUDP-BLOCKED log

Logs are a lot calmer now in terms of new lines per minute, and far more
relevant. When we switch those rules to deny statements, we can
eliminate the rules specifically permitting fragments.

Looks like $VENDOR_J makes things so much simpler for this task.

Thanks,


-Brian


On 2020-10-20 00:18, Randy Bush wrote:
> term blocked-ports {
> from {
> protocol [ tcp udp ];
> first-fragment;
> destination-port
> [. 0 sunrpc 135 netbios-ns netbios-dgm netbios-ssn 111 445 syslog
> 11211];
> }
> then {
> sample;
> discard;
> }
> }
>
> and i block all external access to weak devices such as switches, pdus,
> ipmi, ...
>
> randy

> Randy Bush
> Sent: Tuesday, October 20, 2020 6:19 AM
>
> term blocked-ports {
> from {
> protocol [ tcp udp ];
> first-fragment;
> destination-port
> [. 0 sunrpc 135 netbios-ns netbios-dgm netbios-ssn 111 445 syslog
> 11211];
> }
> then {
> sample;
> discard;
> }
> }
>
Actually what's the latest in the net neutrality talks? Shouldn't these be
just rate-limited rather than blocked? -transit traffic.
(assuming ICMP is the only thing that can talk to infrastructure ranges &
BGP to selected IPs with rest being dropped)

adam

As a final update to this thread, we started blocking spoofed and
invalid traffic as of early Thursday morning Nov 19th. So far, knock on
wood, no reports of issues from our customer base.

In addition, I've been able to verify with the security research team's
test tool that we are no longer responding to the spoofed DNS requests.

The ACL was implemented as follows:

Ingress

* Deny to and from bogon networks, where bogon is either source or dest
* Deny invalid TCP and UDP ports (currently only port 0) [log]
* Permit to and from transit / peer / IX connected subnets
* For IPv6, also permit link-local IPs (fe80::/10)
* Deny to and from multicast ranges 224.0.0.0/4 and ff00::/8
* Permit ICMP / traceroute over UDP to infrastructure
* Deny all other traffic to infrastructure [log]
* Permit from customer PI / PA space
* Deny from originated aggregate space [log]
* Permit all traffic to customer PI / PA space
* Permit all traffic to aggregate space
* Deny any any [log]

Egress

* Deny to and from bogon networks
* Deny invalid ports [log]
* Permit to and from transit / peer / IX connected subnets
* For IPv6, also permit link-local IPs
* Deny to and from multicast range
* Permit all traffic from any source to customer PI / PA space
* Permit all traffic from customer PI / PA space
* Permit all traffic from aggregate space
* Deny any any [log]

Below I've included the specific $VENDOR_C config I implemented for the
filtering, sans specifics on our IP blocks. I hope folks find this
useful as a guide to their own efforts, and constructive criticism is
always welcome.

Future work includes:

* Tightening the rules permitting access to/from the transit / peer / IX
connected subnets, while keeping the ACL general enough for use on all
Internet-facing interfaces
* Automation of updates to aggregate and customer IP blocks (looking at
using the irrpt project for this)

Once more, to those who provided valuable input, thank you very much
indeed!

-Brian


!-------------

! Static ACLs for Service Provider BCP 84 Compliance
! IOS XR config

! IPv4

object-group network ipv4 IPV4-BOGON
description Invalid IPV4 networks
0.0.0.0/8
10.0.0.0/8
100.64.0.0/10
127.0.0.0/8
169.254.0.0/16
172.16.0.0/12
192.0.0.0/24
192.0.2.0/24
192.168.0.0/16
198.18.0.0/15
198.51.100.0/24
203.0.113.0/24
240.0.0.0/4
exit

object-group network ipv4 IPV4-TRAN-WAN
description Transit WAN PtP subnets
[Point to point /30's go here]
exit

object-group network ipv4 IPV4-IX
description IX subnets
[IX /24 and /23 subnets here]
exit

object-group network ipv4 IPV4-PEER-WAN
description Direct peer WAN PtP subnets
[Direct peer WAN IPs go here]
exit

object-group network ipv4 IPV4-BGP-AGG
description ARIN IPV4 Aggregate Blocks
[Aggregated IP blocks go here]
exit

object-group network ipv4 IPV4-INFRA
description Infrastructure subnets to be protected
[.List of loopback blocks and backbone / core PtP /30's here]
exit

object-group network ipv4 IPV4-BACKDOOR-HOSTS
description Hosts observed to be sending valid traffic via Internet
[.One-off hosts, active TCP or UDP traffic was observed during data
collection]
exit

object-group network ipv4 IPV4-CUST
[full list of all customer IP blocks]
[Includes customer PI blocks, disaggregated PA from other providers,]
[and PA assigned from your aggregate space]
exit

object-group port TCPUDP-BLOCKED
eq 0
[additional ports to be generally blocked, list here]
exit

ipv4 access-list IPV4-INET-IN
10 remark BCP 84 for transits, IX, and peering
101 remark *** Block bogon networks as src or dest ***
110 deny ipv4 net-group IPV4-BOGON any
111 deny ipv4 any net-group IPV4-BOGON
201 remark *** Blocked protocols ***
210 deny udp any port-group TCPUDP-BLOCKED any log
211 deny udp any any port-group TCPUDP-BLOCKED log
212 deny tcp any port-group TCPUDP-BLOCKED any log
213 deny tcp any any port-group TCPUDP-BLOCKED log
301 remark *** Transit, IX, peer connected networks ***
310 permit ipv4 net-group IPV4-PEER-WAN any
311 permit ipv4 any net-group IPV4-PEER-WAN
312 permit ipv4 net-group IPV4-TRAN-WAN any
313 permit ipv4 any net-group IPV4-TRAN-WAN
314 permit ipv4 net-group IPV4-IX any
315 permit ipv4 any net-group IPV4-IX
401 remark *** Block multicast ***
410 deny ipv4 224.0.0.0/4 any
411 deny ipv4 any 224.0.0.0/4
501 remark *** Protect infrastructure subnets ***
510 deny icmp any net-group IPV4-INFRA fragments log
511 permit icmp any net-group IPV4-INFRA
512 permit udp any range 1024 65535 net-group IPV4-INFRA range 33435
33535
513 permit udp any range 33435 33535 net-group IPV4-INFRA range 1024
65535
515 deny ipv4 any net-group IPV4-INFRA
601 remark *** Customer Inet BGP Announced Prefixes ***
620 permit ipv4 net-group IPV4-CUST any
640 permit ipv4 net-group IPV4-BACKDOOR-HOSTS any
701 remark *** Block originated networks ***
710 deny ipv4 net-group IPV4-BGP-AGG any log
801 remark *** Permit traffic only to networks we announce ***
820 permit ipv4 any net-group IPV4-BGP-AGG
840 permit ipv4 any net-group IPV4-CUST
901 remark *** Deny all other traffic ***
910 deny ipv4 any any log
exit

ipv4 access-list IPV4-INET-OUT
10 remark BCP 84 for transits, IX, and peering
101 remark *** Block bogon networks as src or dest ***
110 deny ipv4 net-group IPV4-BOGON any
111 deny ipv4 any net-group IPV4-BOGON
201 remark *** Blocked protocols ***
210 deny udp any port-group TCPUDP-BLOCKED any log
211 deny udp any any port-group TCPUDP-BLOCKED log
212 deny tcp any port-group TCPUDP-BLOCKED any log
213 deny tcp any any port-group TCPUDP-BLOCKED log
301 remark *** Transit, IX, peer networks ***
310 permit ipv4 net-group IPV4-PEER-WAN any
311 permit ipv4 any net-group IPV4-PEER-WAN
312 permit ipv4 net-group IPV4-TRAN-WAN any
313 permit ipv4 any net-group IPV4-TRAN-WAN
314 permit ipv4 net-group IPV4-IX any
315 permit ipv4 any net-group IPV4-IX
401 remark *** Block multicast ***
410 deny ipv4 224.0.0.0/4 any
411 deny ipv4 any 224.0.0.0/4
601 remark *** Customer Inet BGP Announced Prefixes ***
620 permit ipv4 any net-group IPV4-CUST
640 permit ipv4 any net-group IPV4-BACKDOOR-HOSTS
801 remark *** Permit locally sourced traffic ***
820 permit ipv4 net-group IPV4-BGP-AGG any
840 permit ipv4 net-group IPV4-CUST any
901 remark *** Deny all other traffic ***
910 deny ipv4 any any log
exit

! IPv6

object-group network ipv6 IPV6-BOGON
description Invalid IPV6 networks
::/3
2001::/23
2001:2::/48
2001:10::/28
2001:db8::/32
2002::/16
3ffe::/16
4000::/3
6000::/3
8000::/3
a000::/3
c000::/3
e000::/4
f000::/5
f800::/6
fc00::/7
fe00::/9
fec0::/10
exit

object-group network ipv6 IPV6-TRAN-WAN
description Transit WAN PtP subnets
[Point to point /126's go here]
exit

object-group network ipv6 IPV6-IX
description IX subnets
[IX /64 subnets here]
exit

object-group network ipv6 IPV6-PEER-WAN
description Direct peer WAN PtP subnets
[Direct peer WAN /126's go here]
exit

object-group network ipv6 IPV6-BGP-AGG
description ARIN IPV6 Aggregate Blocks
[Aggregated IP blocks go here]
exit

object-group network ipv6 IPV6-INFRA
description Infrastructure subnets to be protected
[.List of loopback blocks and backbone / core PtP WAN IPs here]
exit

object-group network ipv6 IPV6-CUST
[full list of all customer IP blocks]
[Includes customer PI blocks, disaggregated PA from other providers,]
[and PA assigned from your aggregate space]
exit

ipv6 access-list IPV6-INET-IN
10 remark BCP 84 for transits, IX, and peering
101 remark *** Block bogon networks as src or dest ***
110 deny ipv6 net-group IPV6-BOGON any
111 deny ipv6 any net-group IPV6-BOGON
201 remark *** Blocked protocols ***
210 deny udp any port-group TCPUDP-BLOCKED any log
211 deny udp any any port-group TCPUDP-BLOCKED log
212 deny tcp any port-group TCPUDP-BLOCKED any log
213 deny tcp any any port-group TCPUDP-BLOCKED log
301 remark *** Transit, IX, peer connected networks ***
310 permit ipv6 fe80::/10 any
311 permit ipv6 net-group IPV6-PEER-WAN any
312 permit ipv6 any net-group IPV6-PEER-WAN
313 permit ipv6 net-group IPV6-TRAN-WAN any
314 permit ipv6 any net-group IPV6-TRAN-WAN
315 permit ipv6 net-group IPV6-IX any
316 permit ipv6 any net-group IPV6-IX
401 remark *** Block multicast ***
410 deny ipv6 ff00::/8 any
411 deny ipv6 any ff00::/8
501 remark *** Protect infrastructure subnets ***
510 deny icmp any net-group IPV6-INFRA fragments log
511 permit icmp any net-group IPV6-INFRA
512 permit udp any range 1024 65535 net-group IPV6-INFRA range 33435
33535
513 permit udp any range 33435 33535 net-group IPV6-INFRA range 1024
65535
515 deny ipv6 any net-group IPV6-INFRA
601 remark *** Customer Inet BGP Announced Prefixes ***
620 permit ipv6 net-group IPV6-CUST any
701 remark *** Block networks we originate ***
710 deny ipv6 net-group IPV6-BGP-AGG any log
801 remark *** Permit traffic only to networks we announce ***
820 permit ipv6 any net-group IPV6-BGP-AGG
840 permit ipv6 any net-group IPV6-CUST
901 remark *** Deny all other traffic ***
910 deny ipv6 any any log
exit

ipv6 access-list IPV6-INET-OUT
10 remark BCP 84 for transits, IX, and peering
101 remark *** Block bogon networks as src or dest ***
110 deny ipv6 net-group IPV6-BOGON any
111 deny ipv6 any net-group IPV6-BOGON
201 remark *** Blocked protocols ***
210 deny udp any port-group TCPUDP-BLOCKED any log
211 deny udp any any port-group TCPUDP-BLOCKED log
212 deny tcp any port-group TCPUDP-BLOCKED any log
213 deny tcp any any port-group TCPUDP-BLOCKED log
301 remark *** Transit, IX, peer networks ***
310 permit ipv6 fe80::/10 any
311 permit ipv6 net-group IPV6-PEER-WAN any
312 permit ipv6 any net-group IPV6-PEER-WAN
313 permit ipv6 net-group IPV6-TRAN-WAN any
314 permit ipv6 any net-group IPV6-TRAN-WAN
315 permit ipv6 net-group IPV6-IX any
316 permit ipv6 any net-group IPV6-IX
401 remark *** Block multicast ***
410 deny ipv6 ff00::/8 any
411 deny ipv6 any ff00::/8
601 remark *** Customer Inet BGP Announced Prefixes ***
620 permit ipv6 any net-group IPV6-CUST
801 remark *** Permit locally sourced traffic ***
820 permit ipv6 net-group IPV6-BGP-AGG any
840 permit ipv6 net-group IPV6-CUST any
901 remark *** Deny all other traffic ***
910 deny ipv6 any any log
exit


! On any transit, IX, or peer port
!
interface TenGigE0/0/0/0
ipv4 access-group IPV4-INET-IN ingress
ipv4 access-group IPV4-INET-OUT egress
ipv6 access-group IPV6-INET-IN ingress
ipv6 access-group IPV6-INET-OUT egress