Mailing List Archive

Lots of people are encountering this, yes.

You can try opening a case yourself, and hope it gets to someone with a
clue. If you don't have a support contract with them, your chances are
almost 0. If you do, your chances are slightly higher, but not by much.
most likely they will just tell you to 'contact the owner of the thing
you're trying to access and have them customize their WAF rules'.

AWS is doing some REALLY dumb things. For example, if your ASN announces a
single prefix that a 3rd party provider classifies as 'hosting provider' ,
AWS will flag EVERY prefix from that ASN as 'hosting provider', which are
all blocked in the default managed WAF rules. They also won't tell you in
any circumstance (even if you're a customer who is paying for support) who
that 3rd party provider IS.

Expect a lot of hassle to get this fixed, if you ever can.

On Thu, Mar 21, 2024 at 1:26?PM Jonathan Kalbfeld via NANOG <nanog@nanog.org>
wrote:

> Hi All,
>
> I just became aware that AWS has a list of hosting IP providers and that
> list is blocked by their WAF? (!?!?). None of my VM or colo customers
> can reach anything in AWS, such as Docker, Twilio, etc. I confirmed
> through source routing that when I access it using one of my peering
> partners as a source IP it is reachable, but using one of my net blocks, it
> is not reachable and times out. Checked all of my routing tables and those
> AWS blocks are definitely visible. Also confirmed from looking glass that
> my IP ranges are showing up.
>
> Has anyone else encountered that? If so, is there a way to get removed
> from that list? I have a very curated list of clients and I know all of
> them personally and none of them have been abusing AWS, so I was wondering
> if it was some kind of blanket ban?
>
> If you're internal to AWS, my ASN is 54380, IP ranges affected are
> 199.33.244.0/24, 199.79.202.0/24, 199.188.96.0/22, 45.59.144.0/22 and
> 206.197.110.0/24
>
> Feel free to reach out off-list.
>
> Thanks,
>
> Jonathan Kalbfeld
>
> Jonathan Kalbfeld
>
> office: +1 310 317 7933 <%28310%29%20317-7933>
> fax: +1 310 317 7901 <%28310%29%20317-7901>
> home: +1 310 317 7909 <%28310%29%20317-7909>
> mobile: +1 310 227 1662 <%28310%29%20227-1662>
>
> ThoughtWave Technologies, Inc.
> Studio City, CA 91604
> https://thoughtwave.com
>
> View our network at
> https://bgp.he.net/AS54380
>
> +1 844 42-LINUX
>
>

Yes.

our network is a mix of content and eyeballs and they listed the whole
thing.  This has prevented the local school district from using their
text to speech application (for their deaf students) as well as others. 
This has also affected the local library and residences.

It's a PITA.

Thanks Amazon.

Aaron


On 3/21/2024 12:16 PM, Jonathan Kalbfeld via NANOG wrote:
> Hi All,
>
> I just became aware that AWS has a list of hosting IP providers and
> that list is blocked by their WAF? (!?!?).  None of my VM or colo
> customers can reach anything in AWS, such as Docker, Twilio, etc.  I
> confirmed through source routing that when I access it using one of my
> peering partners as a source IP it is reachable, but using one of my
> net blocks, it is not reachable and times out.  Checked all of my
> routing tables and those AWS blocks are definitely visible.  Also
> confirmed from looking glass that my IP ranges are showing up.
>
> Has anyone else encountered that? If so, is there a way to get removed
> from that list? I have a very curated list of clients and I know all
> of them personally and none of them have been abusing AWS, so I was
> wondering if it was some kind of blanket ban?
>
> If you're internal to AWS, my ASN is 54380, IP ranges affected are
> 199.33.244.0/24, 199.79.202.0/24, 199.188.96.0/22, 45.59.144.0/22 and
> 206.197.110.0/24
>
> Feel free to reach out off-list.
>
> Thanks,
>
> Jonathan Kalbfeld
>
> Jonathan Kalbfeld
>
> office: +1 310 317 7933 <tel:%28310%29%20317-7933>
> fax: +1 310 317 7901 <tel:%28310%29%20317-7901>
> home: +1 310 317 7909 <tel:%28310%29%20317-7909>
> mobile: +1 310 227 1662 <tel:%28310%29%20227-1662>
>
> ThoughtWave Technologies, Inc.
> Studio City, CA 91604
> https://thoughtwave.com
>
> View our network at
> https://bgp.he.net/AS54380
>
> +1 844 42-LINUX
>

--
================================================================
Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
http://www.wholesaleinternet.com
aaron@wholesaleinternet.com
================================================================

Endured a similar problem last year. The company I was with migrated
prod to Global Accelerator, ALBs, and WAF with Shield Advanced on an
accelerated timeline. Several of our customer networks were blocked in
AWS's default blocklists.

The path to resolution was meeting with our AWS account executive and
their internal WAF team to expedite the removal of the addresses.

-G

On 2024-03-21 18:44, Aaron Wendel wrote:
> Yes.
>
> our network is a mix of content and eyeballs and they listed the whole
> thing.  This has prevented the local school district from using their
> text to speech application (for their deaf students) as well as
> others.  This has also affected the local library and residences.
>
> It's a PITA.
>
> Thanks Amazon.
>
> Aaron
>
>
> On 3/21/2024 12:16 PM, Jonathan Kalbfeld via NANOG wrote:
>> Hi All,
>>
>> I just became aware that AWS has a list of hosting IP providers and
>> that list is blocked by their WAF? (!?!?).  None of my VM or colo
>> customers can reach anything in AWS, such as Docker, Twilio, etc.  I
>> confirmed through source routing that when I access it using one of my
>> peering partners as a source IP it is reachable, but using one of my
>> net blocks, it is not reachable and times out.  Checked all of my
>> routing tables and those AWS blocks are definitely visible.  Also
>> confirmed from looking glass that my IP ranges are showing up.
>>
>> Has anyone else encountered that? If so, is there a way to get removed
>> from that list? I have a very curated list of clients and I know all
>> of them personally and none of them have been abusing AWS, so I was
>> wondering if it was some kind of blanket ban?
>>
>> If you're internal to AWS, my ASN is 54380, IP ranges affected are
>> 199.33.244.0/24, 199.79.202.0/24, 199.188.96.0/22, 45.59.144.0/22 and
>> 206.197.110.0/24
>>
>> Feel free to reach out off-list.
>>
>> Thanks,
>>
>> Jonathan Kalbfeld
>>
>> Jonathan Kalbfeld
>>
>> office: +1 310 317 7933 <tel:%28310%29%20317-7933>
>> fax: +1 310 317 7901 <tel:%28310%29%20317-7901>
>> home: +1 310 317 7909 <tel:%28310%29%20317-7909>
>> mobile: +1 310 227 1662 <tel:%28310%29%20227-1662>
>>
>> ThoughtWave Technologies, Inc.
>> Studio City, CA 91604
>> https://thoughtwave.com
>>
>> View our network at
>> https://bgp.he.net/AS54380
>>
>> +1 844 42-LINUX
>>

Hi.

Our experience from last month was quite similar to what others have
already described. Based on our tests all the allocations associated
with our ASN were added to "HostingProviderIPList". At first we tried
to reach the AWS by e-mail, but received no response. Once we opened a
ticket(we are both the AWS customer and a Direct Connect Delivery
partner) and explained that our address space is not solely used for
hosting services, then all our networks were whitelisted within few
days. We even had a call with AWS support team and they seemed to be
well aware about false positives in the "HostingProviderIPList". In
other words, we were clearly not a first ISP complaining.

Final update to our ticket from the AWS Support Engineering department
was following:

"I would like to inform you that as per the update I got from our
internal team, the IP Data which we use for our rules is provided by a
third party and changes frequently. Moving ahead I would like to
request you to kindly check your setup and see if you can separate the
network ranges reserved for any hosting type services in your ASN and
provide that to us so that we can check if we can allow the range
which is not used for hosting type services."

It's unfortunate that Amazon continues to offer this service despite
their underlying IP address data having a low level of granularity,
and they are well aware of the problems it causes for address owners.


AWS Web Application Firewall for testing purposes can be build for
example like this:

1) created a VPC("VPC only") with "IPv4 CIDR" 10.0.0.0/24
2) created a subnet 10.0.0.0/28 in one availability zone(for example
"eu-north-1a") and a subnet 10.0.0.16/28 in another availability
zone(for example "eu-north-1b")
3) launched an Ubuntu EC2 instance("t3.micro", free tier eligible).
Auto-assigned the public IP. Allowed SSH and HTTP traffic from
anywhere.
4) created an "Internet gateway" and attached it to VPC
5) added a default route to route table and pointed it to Internet gateway
6) installed(sudo apt update && sudo apt install apache2 -y) a web
server in Ubuntu VM
7) tested with "curl http://51.20.94.186" where 51.20.94.186 is the
public IP address automatically assigned to EC2 instance
8) created a "Target group" for "Instances" type target which is
associated with VPC created earlier. Protocol:Port configuration has
to be HTTP:80.
9) created an Internet-facing "Application Load balancer" and mapped
it with two subnets created previously in two different availability
zones. Load balancer has to forward to target group created in the
previous step.
10) created a WAF and associate it with Application Load Balancer
11) used an AWS managed rule group named "Anonymous IP list" in WAF.
This includes the notorious "HostingProviderIPList" with a default
rule action of "Block".
12) "curl http://test-213161342.eu-north-1.elb.amazonaws.com" returns
"HTTP/1.1 403" from the networks included in the
"HostingProviderIPList". DNS name is seen on the load balancer details
page.


P.S. Previous thread on AWS WAF was
https://mailman.nanog.org/pipermail/nanog/2024-February/224931.html


Martin