Mailing List Archive

A network that doesn't support IPv6, yet discriminates against CGNAT? That seems like a promising future.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Simon Lockhart" <simon@slimey.org>
To: nanog@nanog.org
Sent: Friday, September 16, 2016 8:12:46 AM
Subject: PlayStationNetwork blocking of CGNAT public addresses

All,

We operate an access network with several hundred thousand users. Increasingly
we're putting the users behind CGNAT in order to continue to give them an IPv4
service (we're all dual-stack, so they all get public IPv6 too). Due to the
demographic of our users, many of them are gamers.

We're hitting a problem with PlayStationNetwork 'randomly' blocking some of our
CGNAT outside addresses, because they claim to have received anomalous, or
'attack' traffic from that IP. This obviously causes problems for the other
legitimate users who end up behind the same public IPv4 address.

Despite numerous attempts to engage with PSN, they are unwilling to give us
any additional information which would allow us to identify the 'rogue' users
on our network, or to identify the 'unwanted' traffic so that we could either
block it, or use it to identify the rogue users ourselves.

Has anyone else come up against the problem, and/or have any suggestions on
how best to resolve it?

Many thanks in advance,

Simon

On 16 Sep 2016, at 20:12, Simon Lockhart wrote:

> Has anyone else come up against the problem, and/or have any
> suggestions on how best to resolve it?

I'm pretty sure that at least part of it has to do with DDoS-related
activity. The best bet is to try and identify and engage with the
relevant operational personnel with clue. Going the customer-service
route isn't fruitful, as you indicate.

Another aspect is ensuring that one has the ability to detect, classify,
traceback, and mitigate outbound badness southbound of the CGN.

This sort of thing has always been a problem with NAT; as CGN becomes
more prevalent on wireline broadband networks, it's only going to get
worse.

AFAIK, PSN doesn't support IPv6. That would be another topic of
discussion with the operational folks.

-----------------------------------
Roland Dobbins <rdobbins@arbor.net>

On Fri Sep 16, 2016 at 08:32:12PM +0700, Roland Dobbins wrote:
> Another aspect is ensuring that one has the ability to detect, classify,
> traceback, and mitigate outbound badness southbound of the CGN.

Unless PSN can tell us what traffic they consider bad, how can we detect and
classify it? We certainly have the ability to traceback and mitigate, once we
know what we're looking for.

My understanding of the issue is that there are infected PCs on our network,
which are being used as part of a distributed attack, but at the application
layer, rather than network layer - distributed password brute-force, or
similar. Unless we know what to look for, it's hard to detect and stop it.

Simon

Another aspect, for those users that need to go the PSN network but experience issues via the CGNAT, an opt-out solution (giving them public IPv4) may should mitigate the problem, that PSN network does not support IPv6.

After all what percentage of your total subscribers that uses PSN and are gamers 2-3% ? Which might be relatively small amount to give public IPv4.

Michalis

-----Original Message-----
From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Roland Dobbins
Sent: Friday, September 16, 2016 4:32 PM
To: nanog@nanog.org
Subject: Re: PlayStationNetwork blocking of CGNAT public addresses


On 16 Sep 2016, at 20:12, Simon Lockhart wrote:

> Has anyone else come up against the problem, and/or have any
> suggestions on how best to resolve it?

I'm pretty sure that at least part of it has to do with DDoS-related activity. The best bet is to try and identify and engage with the relevant operational personnel with clue. Going the customer-service route isn't fruitful, as you indicate.

Another aspect is ensuring that one has the ability to detect, classify, traceback, and mitigate outbound badness southbound of the CGN.

This sort of thing has always been a problem with NAT; as CGN becomes more prevalent on wireline broadband networks, it's only going to get worse.

AFAIK, PSN doesn't support IPv6. That would be another topic of discussion with the operational folks.

-----------------------------------
Roland Dobbins <rdobbins@arbor.net>

Hi,

as others have said, need to engage with one of their other units to get this sorted
out - as a network provider, their customers are relying on YOU to access their service, PSN should
care.

technically, you could start looking at netflows to the PSN and see if anyone is engaged in DDoS
via that route...and , if you offer IPv6 native service to end users, ask PSN when they are going to
be offer an IPv6 service to their users - so this CGNAT stuff can go ;-)

alan

On 16 Sep 2016, at 20:38, Simon Lockhart wrote:

> Unless we know what to look for, it's hard to detect and stop it.

It's not just application-layer stuff - they're subject to all sorts of
attacks. Screening out the obvious stuff would certainly help.

The main issue is a dearth of engagement of clueful folks in the global
operational community. Some gaming-oriented networks are
well-represented; others are not, sadly.

-----------------------------------
Roland Dobbins <rdobbins@arbor.net>

On Friday, September 16, 2016, Simon Lockhart <simon@slimey.org> wrote:

> All,
>
> We operate an access network with several hundred thousand users.
> Increasingly
> we're putting the users behind CGNAT in order to continue to give them an
> IPv4
> service (we're all dual-stack, so they all get public IPv6 too). Due to the
> demographic of our users, many of them are gamers.
>
> We're hitting a problem with PlayStationNetwork 'randomly' blocking some
> of our
> CGNAT outside addresses, because they claim to have received anomalous, or
> 'attack' traffic from that IP. This obviously causes problems for the other
> legitimate users who end up behind the same public IPv4 address.
>
> Despite numerous attempts to engage with PSN, they are unwilling to give us
> any additional information which would allow us to identify the 'rogue'
> users
> on our network, or to identify the 'unwanted' traffic so that we could
> either
> block it, or use it to identify the rogue users ourselves.
>
> Has anyone else come up against the problem, and/or have any suggestions on
> how best to resolve it?
>
> Many thanks in advance,
>
> Simon


Here is a picture of what you are experiencing

http://test-ipv6.com/faq_avoids_ipv6.html

Sometimes people need pictures to understand why IPv6 is important

So the pain has finally flowed down to other parts of the world. (APNIC ran
out of IP's a long time ago, so CGN has been in use here for a lot longer)
This issue is one I have been dealing with for the last four years. Only
with Sony, no other company has caused such a headache in regard to CGNAT. I
will not go into the long and painful saga of dealing with the constant
issue of Sony putting blocks on random pool addresses, refusing to supply
sufficient information to identify rouge users (timestamp, source IP,
destination IP and port) then telling our customers it is a problem at the
ISP end, but... Something happened about three months ago that Proves that
if the Sony technical people want to get off their asses they are perfectly
capable of supplying adequate information to identify a rogue user for the
ISP to deal with. One of the local Sony PSN helpline managers actually
managed to convince one of their technical people to supply a spreadsheet
that magically contained sufficient information to allow us to identify a
couple of users that did indeed have multiple infections. Great I thought,
now if we can just get them to automate/regularly sent this info we will
have a way forward. Alas, it appears it was a one off and we are back to the
start. I will quote below what the Sony Network guy said when explaining why
they can't send detailed information every time -


" From: SNEI-NOC-Abuse [mailto:SNEI-NOC-Abuse@am.sony.com]
Sent: Thursday, 11 August 2016 8:38 AM
To: ##me##
Cc: ##helpful Sony guy## Subject: RE: PSN / Flip Network blocks

Hello,

There is quite a bit of extra computing power required to produce the CSV
file with timestamps and destination IP addresses. We send out over 6000
emails per day which already takes a significant amount of resources and
time. We tend to get around 20-30 responses. Instead of wasting the
resources on all those emails we generate CSV files for those who respond.

We hope you understand.

Thank you for taking action on these."

So there you go, Sony can indeed solve this issue, but apparently a company
that makes computers has insufficient computing power and staff to do so. Oh
and after this, despite being asked many times they have never responded to
requests for the CSV or similar detailed info.




-----Original Message-----
From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Simon Lockhart
Sent: Saturday, 17 September 2016 1:13 AM
To: nanog@nanog.org
Subject: PlayStationNetwork blocking of CGNAT public addresses

All,

We operate an access network with several hundred thousand users.
Increasingly we're putting the users behind CGNAT in order to continue to
give them an IPv4 service (we're all dual-stack, so they all get public IPv6
too). Due to the demographic of our users, many of them are gamers.

We're hitting a problem with PlayStationNetwork 'randomly' blocking some of
our CGNAT outside addresses, because they claim to have received anomalous,
or 'attack' traffic from that IP. This obviously causes problems for the
other legitimate users who end up behind the same public IPv4 address.

Despite numerous attempts to engage with PSN, they are unwilling to give us
any additional information which would allow us to identify the 'rogue'
users on our network, or to identify the 'unwanted' traffic so that we could
either block it, or use it to identify the rogue users ourselves.

Has anyone else come up against the problem, and/or have any suggestions on
how best to resolve it?

Many thanks in advance,

Simon

Simon Lockhart wrote:

> Has anyone else come up against the problem, and/or have any suggestions on
> how best to resolve it?

The best solution is to have a common practice on a set of public
port numbers assigned to a host behind NAT.

For example, with a practice that, if a port in a range between N*8
and N*8+7 is assigned to a host, other ports in the range is not
assigned to other hosts, service providers can block packets
based on IP addresses and ranges, especially if correspondence between
hosts and ranges are rather stable.

But, it may be too late to make such practice common, I'm afraid.

Or, wait for a while until service providers receive enough amount
of feedback from innocent users. To accelerate it, you can make
correspondence between hosts and public addresses not so stable,
which makes almost all your IP addresses marked bad quickly,
which may make you loss some customer, unless other ISPs also do so.

Masataka Ohta

Hi Simon,

as other responders have said it is an inherent issue with NAT in general,
on workaround is to limit the ratio of actual users to an external IPv4
address, the other thing we have seen from our Abuse contact emails from
PSN, is that malicious activity towards the PSN is often accompanied by
other malicious activities such as SSH brute force outbound and spaming...

I would suggest that

1) limit the ratio of users to an external ipv4 address as much as possible
(which would reduce the impact of one compromised customer bringing down
play time for other clients behind the same nat

2)do some "canary in the mine" monitoring for obviously malicious traffic
(loads of SMTP traffic outbound) and lots of connection requests to SSH
servers ... if you see that traffic from behind your CGNAT device .. just
temporarily block the internal ip of the user until they clean up their
devices.

this is the pain with NAT you have to do extra work in order prevent
infected users interrupting internet connectivity for other innocent
users...
I think you can use simple firewall rules on your edge router to identify
multiple connections to SMTP and SSH in a short period of time..

If you do the minimum to detect that abuse then you cant be accused of
invading peoples privacy... (bear in mind obvious false positives)
(Monitoring systems etc) ...

Hope this helps,

On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart <simon@slimey.org> wrote:

> All,
>
> We operate an access network with several hundred thousand users.
> Increasingly
> we're putting the users behind CGNAT in order to continue to give them an
> IPv4
> service (we're all dual-stack, so they all get public IPv6 too). Due to the
> demographic of our users, many of them are gamers.
>
> We're hitting a problem with PlayStationNetwork 'randomly' blocking some
> of our
> CGNAT outside addresses, because they claim to have received anomalous, or
> 'attack' traffic from that IP. This obviously causes problems for the other
> legitimate users who end up behind the same public IPv4 address.
>
> Despite numerous attempts to engage with PSN, they are unwilling to give us
> any additional information which would allow us to identify the 'rogue'
> users
> on our network, or to identify the 'unwanted' traffic so that we could
> either
> block it, or use it to identify the rogue users ourselves.
>
> Has anyone else come up against the problem, and/or have any suggestions on
> how best to resolve it?
>
> Many thanks in advance,
>
> Simon
>
>


--
Kindest regards,
Tom Smyth

Mobile: +353 87 6193172
---------------------------------
PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL
This email contains information which may be confidential or privileged.
The information is intended solely for the use of the individual or entity
named above. If you are not the intended recipient, be aware that
any disclosure, copying, distribution or use of the contents of this
information is prohibited. If you have received this electronic
transmission in error, please notify me by telephone or by electronic mail
immediately. Any opinions expressed are those of the author, not the
company's .This email does not constitute either offer or acceptance of
any contractually binding agreement. Such offer or acceptance must be
communicated in
writing. You are requested to carry out your own virus check before opening
any attachment. Thomas Smyth accepts no liability for any loss or damage
which may be caused by malicious software or attachments.

On Sun, Sep 18, 2016 at 01:30:52PM +0100, Tom Smyth wrote:
> 2)do some "canary in the mine" monitoring for obviously malicious traffic
> (loads of SMTP traffic outbound) and lots of connection requests to SSH
> servers ... if you see that traffic from behind your CGNAT device .. just
> temporarily block the internal ip of the user until they clean up their
> devices.

Seconded. This is something I've recommended for years (decades, I suppose
by now). Simple measurements of what's "normal" for your operation in
terms of connection rates, types, etc., are easy to make. That in turn
enables measurements of what's abnormal and that in turn enables manual
or automatic actions. For example: if the average number of outbound
SSH connections established per hour per host across all hosts behind CGNAT
is 3.2, and you see a host making 1100/hour: that's a problem. It might be
someone who botched a Perl script; or it might be a botted host trying
to brute-force its way into something.

These kinds of measurements are relatively easy to make and don't require
invading user privacy. They won't catch everything, of course, but they're
not intended to. They may catch enough to solve the problem in front of
you at the moment *and*, if they do that, they may reduce the scope/scale
of the rest of the problems to make them more tractable via other techniques.

---rsk

This is, as many things are, a huge problem in communication.

Sony tells ISP 'Hey, you have customers abusing us. Fix it!'.
ISP says 'Oh crap, sorry, what's going on? We'll run it down.'
Sony says nothing.

Let's just stop here for a second. This is fundamentally no different then
the 'I have a problem, it's the network! complaints we've all dealt with
forever. You spend days/weeks/months working on it. Maybe you ultimately
find a goofy switchport, or maybe you discover that the server HDDs were
crapping the bed and the problem server was chugging because of that. But
you had to spend tons of time working on it because you couldn't get the
info you need because the reporter was CONVINCED they KNEW what it was.

Why should Simon have to spend hours of engineering time fishing through
traffic captures and logs when he doesn't even know what he's LOOKING for?
What does PSN consider 'abuse' here?

Does Simon have customers infected with botnets that are targeting PSN at
times? Or does PSN assume nobody will ever have more than a couple
Playstations in a house, so if they see more than N connections to PSN from
the same IP, it's malicious, since CGN is likely not something they
considered? ( If anyone wants to place beer wagers, I'm picking the later. )

I spend about 8 weeks this year going back and forth with a Very Large
Website Network who had blocked a /17 of IP space from accessing ANY of
their sites because of 'malicious traffic' from a specific /23. 5 of those
weeks, their responses consisted of 'it's malicious, you go find it, should
be obvious', 'you clearly don't know what you're doing, we're wasting our
time', etc. Week 5, I was able to extract that it was a specific web
crawler that they said was knocking their databases over. After a
conversation with their CIO the following week, they came back and admitted
that a junior system admin made some PHP changes on a bunch of servers that
he didn't think was in production,and when we crawled THOSE servers, Bad
Things Happened for them. We were doing nothing wrong ; they just refused
to look, and found it easier to blame us.

Simon's getting screwed because he's not being given any information to try
and solve the problem, and because his customers are likely blaming him
because he's their ISP.

Sony needs to stand up and work with him here.

On Sun, Sep 18, 2016 at 8:30 AM, Tom Smyth <tom.smyth@wirelessconnect.eu>
wrote:

> Hi Simon,
>
> as other responders have said it is an inherent issue with NAT in general,
> on workaround is to limit the ratio of actual users to an external IPv4
> address, the other thing we have seen from our Abuse contact emails from
> PSN, is that malicious activity towards the PSN is often accompanied by
> other malicious activities such as SSH brute force outbound and spaming...
>
> I would suggest that
>
> 1) limit the ratio of users to an external ipv4 address as much as possible
> (which would reduce the impact of one compromised customer bringing down
> play time for other clients behind the same nat
>
> 2)do some "canary in the mine" monitoring for obviously malicious traffic
> (loads of SMTP traffic outbound) and lots of connection requests to SSH
> servers ... if you see that traffic from behind your CGNAT device .. just
> temporarily block the internal ip of the user until they clean up their
> devices.
>
> this is the pain with NAT you have to do extra work in order prevent
> infected users interrupting internet connectivity for other innocent
> users...
> I think you can use simple firewall rules on your edge router to identify
> multiple connections to SMTP and SSH in a short period of time..
>
> If you do the minimum to detect that abuse then you cant be accused of
> invading peoples privacy... (bear in mind obvious false positives)
> (Monitoring systems etc) ...
>
> Hope this helps,
>
> On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart <simon@slimey.org> wrote:
>
> > All,
> >
> > We operate an access network with several hundred thousand users.
> > Increasingly
> > we're putting the users behind CGNAT in order to continue to give them an
> > IPv4
> > service (we're all dual-stack, so they all get public IPv6 too). Due to
> the
> > demographic of our users, many of them are gamers.
> >
> > We're hitting a problem with PlayStationNetwork 'randomly' blocking some
> > of our
> > CGNAT outside addresses, because they claim to have received anomalous,
> or
> > 'attack' traffic from that IP. This obviously causes problems for the
> other
> > legitimate users who end up behind the same public IPv4 address.
> >
> > Despite numerous attempts to engage with PSN, they are unwilling to give
> us
> > any additional information which would allow us to identify the 'rogue'
> > users
> > on our network, or to identify the 'unwanted' traffic so that we could
> > either
> > block it, or use it to identify the rogue users ourselves.
> >
> > Has anyone else come up against the problem, and/or have any suggestions
> on
> > how best to resolve it?
> >
> > Many thanks in advance,
> >
> > Simon
> >
> >
>
>
> --
> Kindest regards,
> Tom Smyth
>
> Mobile: +353 87 6193172
> ---------------------------------
> PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL
> This email contains information which may be confidential or privileged.
> The information is intended solely for the use of the individual or entity
> named above. If you are not the intended recipient, be aware that
> any disclosure, copying, distribution or use of the contents of this
> information is prohibited. If you have received this electronic
> transmission in error, please notify me by telephone or by electronic mail
> immediately. Any opinions expressed are those of the author, not the
> company's .This email does not constitute either offer or acceptance of
> any contractually binding agreement. Such offer or acceptance must be
> communicated in
> writing. You are requested to carry out your own virus check before opening
> any attachment. Thomas Smyth accepts no liability for any loss or damage
> which may be caused by malicious software or attachments.
>

People love to hate incumbent telcos because of their arrogance (and frankly it's deserved), but people forget that big content can be just as arrogant and just as deserving of hatred.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Tom Beecher" <beecher@beecher.cc>
To: "Tom Smyth" <tom.smyth@wirelessconnect.eu>
Cc: "NANOG" <nanog@nanog.org>
Sent: Sunday, September 18, 2016 8:15:08 AM
Subject: Re: PlayStationNetwork blocking of CGNAT public addresses

This is, as many things are, a huge problem in communication.

Sony tells ISP 'Hey, you have customers abusing us. Fix it!'.
ISP says 'Oh crap, sorry, what's going on? We'll run it down.'
Sony says nothing.

Let's just stop here for a second. This is fundamentally no different then
the 'I have a problem, it's the network! complaints we've all dealt with
forever. You spend days/weeks/months working on it. Maybe you ultimately
find a goofy switchport, or maybe you discover that the server HDDs were
crapping the bed and the problem server was chugging because of that. But
you had to spend tons of time working on it because you couldn't get the
info you need because the reporter was CONVINCED they KNEW what it was.

Why should Simon have to spend hours of engineering time fishing through
traffic captures and logs when he doesn't even know what he's LOOKING for?
What does PSN consider 'abuse' here?

Does Simon have customers infected with botnets that are targeting PSN at
times? Or does PSN assume nobody will ever have more than a couple
Playstations in a house, so if they see more than N connections to PSN from
the same IP, it's malicious, since CGN is likely not something they
considered? ( If anyone wants to place beer wagers, I'm picking the later. )

I spend about 8 weeks this year going back and forth with a Very Large
Website Network who had blocked a /17 of IP space from accessing ANY of
their sites because of 'malicious traffic' from a specific /23. 5 of those
weeks, their responses consisted of 'it's malicious, you go find it, should
be obvious', 'you clearly don't know what you're doing, we're wasting our
time', etc. Week 5, I was able to extract that it was a specific web
crawler that they said was knocking their databases over. After a
conversation with their CIO the following week, they came back and admitted
that a junior system admin made some PHP changes on a bunch of servers that
he didn't think was in production,and when we crawled THOSE servers, Bad
Things Happened for them. We were doing nothing wrong ; they just refused
to look, and found it easier to blame us.

Simon's getting screwed because he's not being given any information to try
and solve the problem, and because his customers are likely blaming him
because he's their ISP.

Sony needs to stand up and work with him here.

On Sun, Sep 18, 2016 at 8:30 AM, Tom Smyth <tom.smyth@wirelessconnect.eu>
wrote:

> Hi Simon,
>
> as other responders have said it is an inherent issue with NAT in general,
> on workaround is to limit the ratio of actual users to an external IPv4
> address, the other thing we have seen from our Abuse contact emails from
> PSN, is that malicious activity towards the PSN is often accompanied by
> other malicious activities such as SSH brute force outbound and spaming...
>
> I would suggest that
>
> 1) limit the ratio of users to an external ipv4 address as much as possible
> (which would reduce the impact of one compromised customer bringing down
> play time for other clients behind the same nat
>
> 2)do some "canary in the mine" monitoring for obviously malicious traffic
> (loads of SMTP traffic outbound) and lots of connection requests to SSH
> servers ... if you see that traffic from behind your CGNAT device .. just
> temporarily block the internal ip of the user until they clean up their
> devices.
>
> this is the pain with NAT you have to do extra work in order prevent
> infected users interrupting internet connectivity for other innocent
> users...
> I think you can use simple firewall rules on your edge router to identify
> multiple connections to SMTP and SSH in a short period of time..
>
> If you do the minimum to detect that abuse then you cant be accused of
> invading peoples privacy... (bear in mind obvious false positives)
> (Monitoring systems etc) ...
>
> Hope this helps,
>
> On Fri, Sep 16, 2016 at 2:12 PM, Simon Lockhart <simon@slimey.org> wrote:
>
> > All,
> >
> > We operate an access network with several hundred thousand users.
> > Increasingly
> > we're putting the users behind CGNAT in order to continue to give them an
> > IPv4
> > service (we're all dual-stack, so they all get public IPv6 too). Due to
> the
> > demographic of our users, many of them are gamers.
> >
> > We're hitting a problem with PlayStationNetwork 'randomly' blocking some
> > of our
> > CGNAT outside addresses, because they claim to have received anomalous,
> or
> > 'attack' traffic from that IP. This obviously causes problems for the
> other
> > legitimate users who end up behind the same public IPv4 address.
> >
> > Despite numerous attempts to engage with PSN, they are unwilling to give
> us
> > any additional information which would allow us to identify the 'rogue'
> > users
> > on our network, or to identify the 'unwanted' traffic so that we could
> > either
> > block it, or use it to identify the rogue users ourselves.
> >
> > Has anyone else come up against the problem, and/or have any suggestions
> on
> > how best to resolve it?
> >
> > Many thanks in advance,
> >
> > Simon
> >
> >
>
>
> --
> Kindest regards,
> Tom Smyth
>
> Mobile: +353 87 6193172
> ---------------------------------
> PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL
> This email contains information which may be confidential or privileged.
> The information is intended solely for the use of the individual or entity
> named above. If you are not the intended recipient, be aware that
> any disclosure, copying, distribution or use of the contents of this
> information is prohibited. If you have received this electronic
> transmission in error, please notify me by telephone or by electronic mail
> immediately. Any opinions expressed are those of the author, not the
> company's .This email does not constitute either offer or acceptance of
> any contractually binding agreement. Such offer or acceptance must be
> communicated in
> writing. You are requested to carry out your own virus check before opening
> any attachment. Thomas Smyth accepts no liability for any loss or damage
> which may be caused by malicious software or attachments.
>

* Rich Kulawiec:

> For example: if the average number of outbound SSH connections
> established per hour per host across all hosts behind CGNAT is 3.2,
> and you see a host making 1100/hour: that's a problem. It might be
> someone who botched a Perl script; or it might be a botted host
> trying to brute-force its way into something.

If you do this, you break Github.

(If I guess Simon's network correctly, then I've seen reports which
suggest that they might already be doing this.)

* Tom Beecher:

> Simon's getting screwed because he's not being given any information to try
> and solve the problem, and because his customers are likely blaming him
> because he's their ISP.

We don't know that for sure. Another potential issue is that the ISP
just cannot afford to notify its compromised customers, even if they
were able to detect them.

On Sun Sep 18, 2016 at 03:58:57PM +0200, Florian Weimer wrote:
> * Tom Beecher:
> > Simon's getting screwed because he's not being given any information to try
> > and solve the problem, and because his customers are likely blaming him
> > because he's their ISP.
>
> We don't know that for sure. Another potential issue is that the ISP
> just cannot afford to notify its compromised customers, even if they
> were able to detect them.

I'd like to think that we're pretty responsive to taking our users offline
when they're compromised and we're made aware of it - either through our own
tools, or through 3rd party notifications.

The process with Sony goes something like:

- User reports they can't reach PSN
- We report the Sony/PSN, they say "Yes, it's blocked because that IP attacked
us"
- We say "Okay, that's a CGNAT public IP, can you help us identify the which
inside user that is - (timestamp,ip,port) logs, or some way to identify the
bad traffic so we can look for it ourselves"
- Sony say no, either through silence, or explicitly.
- We have unhappy user(s), who blame us.

Simon

An email to a user notifying them they're likely compromised costs
basically nothing. An email to their entire subscriber base also costs
nothing. If you find me an ISP that can't afford to notify users, I'll show
you one that shouldn't be in business anyways.

There's this presumption of guilt here, that Sony is right, and Simon's
subscribers are doing something malicious, yet they won't provide any
evidence of that. Even if they didn't know what it was, come back with
'We're seeing weird bursts of [traffic characteristics] aimed at PSN during
these times. We're not quite sure what it is, but it's causing [problem
X].' It would still be a question of maliciousness or not, but it would be
something to work with. Providing nothing just perpetuates this finger
pointing game, and nothing gets solved.



On Sun, Sep 18, 2016 at 9:58 AM, Florian Weimer <fw@deneb.enyo.de> wrote:

> * Tom Beecher:
>
> > Simon's getting screwed because he's not being given any information to
> try
> > and solve the problem, and because his customers are likely blaming him
> > because he's their ISP.
>
> We don't know that for sure. Another potential issue is that the ISP
> just cannot afford to notify its compromised customers, even if they
> were able to detect them.
>

* Tom Beecher:

> An email to a user notifying them they're likely compromised costs
> basically nothing.

If this increases the probability that the customer contacts customer
support, in some markets, there is a risk that the account will never
turn profitable during the current contract period. (Granted, my
information may be woefully out of date, but my impression is that
price-based competition is still pretty much cut-throat over here.)

> If you find me an ISP that can't afford to notify users, I'll show
> you one that shouldn't be in business anyways.

I'm not blaming the ISP. (I may have done so in the past.) If we end
up in such a situation, it's hardly the fault of one single ISP.

> There's this presumption of guilt here, that Sony is right, and Simon's
> subscribers are doing something malicious, yet they won't provide any
> evidence of that. Even if they didn't know what it was, come back with
> 'We're seeing weird bursts of [traffic characteristics] aimed at PSN during
> these times. We're not quite sure what it is, but it's causing [problem
> X].' It would still be a question of maliciousness or not, but it would be
> something to work with. Providing nothing just perpetuates this finger
> pointing game, and nothing gets solved.

Yes, indeed. Resolving most networking problems needs cooperation,
because at the most basic level, the Internet is still about
connecting otherwise unrelated networks.

* Simon Lockhart:

> On Sun Sep 18, 2016 at 03:58:57PM +0200, Florian Weimer wrote:
>> * Tom Beecher:
>> > Simon's getting screwed because he's not being given any information to try
>> > and solve the problem, and because his customers are likely blaming him
>> > because he's their ISP.
>>
>> We don't know that for sure. Another potential issue is that the ISP
>> just cannot afford to notify its compromised customers, even if they
>> were able to detect them.
>
> I'd like to think that we're pretty responsive to taking our users offline
> when they're compromised and we're made aware of it - either through our own
> tools, or through 3rd party notifications.

Okay, then perhaps my guess of the ISP involved is wrong.

> The process with Sony goes something like:
>
> - User reports they can't reach PSN
> - We report the Sony/PSN, they say "Yes, it's blocked because that IP attacked
> us"
> - We say "Okay, that's a CGNAT public IP, can you help us identify the which
> inside user that is - (timestamp,ip,port) logs, or some way to identify the
> bad traffic so we can look for it ourselves"
> - Sony say no, either through silence, or explicitly.
> - We have unhappy user(s), who blame us.

Yes, that's not very constructive.

Out of curiosity, how common is end-to-end reporting of
source/destination port information (in addition to source IP
addresses and destination IP addresses)? Have the anti-abuse
mechanisms finalyl caught on with CGNAT, or is it possible that the
PSN operator themselves do not have such detailed data?

On Sun Sep 18, 2016 at 05:17:33PM +0200, Florian Weimer wrote:
> Okay, then perhaps my guess of the ISP involved is wrong.

It's not hard to find out who I work for :)

> Out of curiosity, how common is end-to-end reporting of
> source/destination port information (in addition to source IP
> addresses and destination IP addresses)? Have the anti-abuse
> mechanisms finalyl caught on with CGNAT, or is it possible that the
> PSN operator themselves do not have such detailed data?

99.99% of abuse reports we receive contain the information, but that's because
99.99% of abuse reports we receive are from the 'copyright police', and their
tools capture and include it in the reports.

Once you discard that 99.99%, and are left with the stuff that is worthy of
manual investigation, I'd say that almost all of it only contains timestamp and
source IP. Sometimes it'll also contain destination IP (so we can take a best
guess based on netflow data), and very occasionally it'll also contain source
port information.

I'd say the same also applies to requests for information that we receive from
law enforcement agencies. In most cases, they're working from weblogs, and I'd
be tempted to say that most webservers' 'out of the box' configuration does not
log source port, only source IP in the web access logs.

Simon

On 9/18/2016 08:19, Mike Hammett wrote:
> People love to hate incumbent telcos because of their arrogance (and
> frankly it's deserved), but people forget that big content can be
> just as arrogant and just as deserving of hatred.

I never did see the benefit or the approach. To anybody.


--
"Everybody is a genius. But if you judge a fish by
its ability to climb a tree, it will live its whole
life believing that it is stupid."

--Albert Einstein

From Larry's Cox account.

On 9/18/2016 16:26, Larry Sheldon wrote:
>
>
> On 9/18/2016 08:19, Mike Hammett wrote:
>> People love to hate incumbent telcos because of their arrogance (and
>> frankly it's deserved), but people forget that big content can be
>> just as arrogant and just as deserving of hatred.
>
> I never did see the benefit or the approach. To anybody.

> I never did see the benefit oF the approach. To anybody.

--
"Everybody is a genius. But if you judge a fish by
its ability to climb a tree, it will live its whole
life believing that it is stupid."

--Albert Einstein

From Larry's Cox account.

Interestingly, Sony (SNEI-NOC-Abuse <SNEI-NOC-Abuse@am. sony dot com) jut
replied to being forwarded back one of their notification blocks requesting
more detailed information with a csv file in under an hour!



>- Sony say no, either through silence, or explicitly.

On Mon, 19 Sep 2016 10:41:59 +1200, "Tony Wicks" said:
> Interestingly, Sony (SNEI-NOC-Abuse <SNEI-NOC-Abuse@am. sony dot com) jut
> replied to being forwarded back one of their notification blocks requesting
> more detailed information with a csv file in under an hour!

So I guess name-and-shame *does* work? :)

So I should try again to get them to tell me what an "Account Takeover
Attempt" is? They ignored my last request.

It's easy to explain DMCA or spam to an end-user, but it's difficult to
explain to some soccer mom that her kids are doing something to make Sony
mad, when I can't explain to them what Sony is mad about.

On Sun, Sep 18, 2016 at 5:58 PM, <Valdis.Kletnieks@vt.edu> wrote:

> On Mon, 19 Sep 2016 10:41:59 +1200, "Tony Wicks" said:
> > Interestingly, Sony (SNEI-NOC-Abuse <SNEI-NOC-Abuse@am. sony dot com)
> jut
> > replied to being forwarded back one of their notification blocks
> requesting
> > more detailed information with a csv file in under an hour!
>
> So I guess name-and-shame *does* work? :)
>

So the last one we successfully managed to isolate, our customer they had more than one PC with multiple infections. It’s not Playstation’s, but Windows machines that are infected with I assume some malware that is trying to log into PSN.



cheers



From: Jason Baugher [mailto:jason@thebaughers.com]
Sent: Monday, 19 September 2016 12:09 PM
To: Valdis.Kletnieks@vt.edu
Cc: Tony Wicks <tony@wicks.co.nz>; NANOG <nanog@nanog.org>
Subject: Re: PlayStationNetwork blocking of CGNAT public addresses



So I should try again to get them to tell me what an "Account Takeover Attempt" is? They ignored my last request.



It's easy to explain DMCA or spam to an end-user, but it's difficult to explain to some soccer mom that her kids are doing something to make Sony mad, when I can't explain to them what Sony is mad about.



On Sun, Sep 18, 2016 at 5:58 PM, <Valdis.Kletnieks@vt.edu <mailto:Valdis.Kletnieks@vt.edu> > wrote:

On Mon, 19 Sep 2016 10:41:59 +1200, "Tony Wicks" said:
> Interestingly, Sony (SNEI-NOC-Abuse <SNEI-NOC-Abuse@am. <mailto:SNEI-NOC-Abuse@am. %20 sony%20dot%20com)%20jut%0b> sony dot com) jut
> replied to being forwarded back one of their notification blocks requesting
> more detailed information with a csv file in under an hour!

So I guess name-and-shame *does* work? :)

On Sun, Sep 18, 2016 at 03:56:30PM +0200, Florian Weimer wrote:
> * Rich Kulawiec:
>
> > For example: if the average number of outbound SSH connections
> > established per hour per host across all hosts behind CGNAT is 3.2,
> > and you see a host making 1100/hour: that's a problem. It might be
> > someone who botched a Perl script; or it might be a botted host
> > trying to brute-force its way into something.
>
> If you do this, you break Github.

1. I didn't know that: *how* does this break Github?

2. This is just an *example* of how to use the technique. It's not
meant to be literal. The general approach of determining the statistical
characteristics of "normal" and then flagging things that are "way
outside normal" works -- but of course it requires sufficient knowledge
to account for things like Github usage and/or infrequent events and/or
usage spikes triggered by real-world events, etc. The more you do it,
and the longer you do it, the better you'll get at it. (But of course
the false positive rate will never be zero. That's why the question
of what to do when anomalies happen isn't easy: poke a human? throttle?
block? further analysis?)

---rsk

* Rich Kulawiec:

> On Sun, Sep 18, 2016 at 03:56:30PM +0200, Florian Weimer wrote:
>> * Rich Kulawiec:
>>
>> > For example: if the average number of outbound SSH connections
>> > established per hour per host across all hosts behind CGNAT is 3.2,
>> > and you see a host making 1100/hour: that's a problem. It might be
>> > someone who botched a Perl script; or it might be a botted host
>> > trying to brute-force its way into something.
>>
>> If you do this, you break Github.
>
> 1. I didn't know that: *how* does this break Github?

Github users create several orders of magnitude more SSH connections
than average users because the most convenient way to set up
read/write access is to use SSH. Depending on how you use Github, you
might update lots and lots of local repositories from Github at
certain times of the day.

> 2. This is just an *example* of how to use the technique. It's not
> meant to be literal. The general approach of determining the statistical
> characteristics of "normal" and then flagging things that are "way
> outside normal" works -- but of course it requires sufficient knowledge
> to account for things like Github usage and/or infrequent events and/or
> usage spikes triggered by real-world events, etc.

Sure, and people already do this, and are not very flexible about it.
Support staff isn't briefed, and claim they do such stochastic
behavior adjustment across all (server) products, which I find
difficult to believe.

I'm worried that this leads to a future where tunnelling everything
over HTTP(S) is no longer sufficient. You have to make it look like a
web server or browser, too. Everything else risks triggering
automated countermeasures.

That's the anti-thesis of good protocol design.

Something similar happened to a local FantasyConon I was helping set up, we
had only two PS4 machines there and accounts provided by Blizzard for
Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in
about 8h. There was no other traffic other then those two accounts playing
Overwatch so my guess is that they have some too aggressive checks. I've
managed to convince our ISP there to change the outside IP of the link so
we got them working the next day but it happened again in 8h.

--
*blap*

On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart <simon@slimey.org> wrote:

> All,
>
> We operate an access network with several hundred thousand users.
> Increasingly
> we're putting the users behind CGNAT in order to continue to give them an
> IPv4
> service (we're all dual-stack, so they all get public IPv6 too). Due to the
> demographic of our users, many of them are gamers.
>
> We're hitting a problem with PlayStationNetwork 'randomly' blocking some
> of our
> CGNAT outside addresses, because they claim to have received anomalous, or
> 'attack' traffic from that IP. This obviously causes problems for the other
> legitimate users who end up behind the same public IPv4 address.
>
> Despite numerous attempts to engage with PSN, they are unwilling to give us
> any additional information which would allow us to identify the 'rogue'
> users
> on our network, or to identify the 'unwanted' traffic so that we could
> either
> block it, or use it to identify the rogue users ourselves.
>
> Has anyone else come up against the problem, and/or have any suggestions on
> how best to resolve it?
>
> Many thanks in advance,
>
> Simon
>
>

PSN is one reason I am not a fan of CGNAT. All they see are tons of connections from the same IP. This results in them banning folks. Due to them being hacked so many times getting them to actually communicate is almost impossible. My .02 is just get the gamers a true public if at all possible.

Justin Wilson
j2sw@mtin.net

---
http://www.mtin.net Owner/CEO
xISP Solutions- Consulting – Data Centers - Bandwidth

http://www.midwest-ix.com COO/Chairman
Internet Exchange - Peering - Distributed Fabric

> On Sep 20, 2016, at 8:24 AM, Danijel Starman <theghost101@gmail.com> wrote:
>
> Something similar happened to a local FantasyConon I was helping set up, we
> had only two PS4 machines there and accounts provided by Blizzard for
> Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in
> about 8h. There was no other traffic other then those two accounts playing
> Overwatch so my guess is that they have some too aggressive checks. I've
> managed to convince our ISP there to change the outside IP of the link so
> we got them working the next day but it happened again in 8h.
>
> --
> *blap*
>
> On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart <simon@slimey.org> wrote:
>
>> All,
>>
>> We operate an access network with several hundred thousand users.
>> Increasingly
>> we're putting the users behind CGNAT in order to continue to give them an
>> IPv4
>> service (we're all dual-stack, so they all get public IPv6 too). Due to the
>> demographic of our users, many of them are gamers.
>>
>> We're hitting a problem with PlayStationNetwork 'randomly' blocking some
>> of our
>> CGNAT outside addresses, because they claim to have received anomalous, or
>> 'attack' traffic from that IP. This obviously causes problems for the other
>> legitimate users who end up behind the same public IPv4 address.
>>
>> Despite numerous attempts to engage with PSN, they are unwilling to give us
>> any additional information which would allow us to identify the 'rogue'
>> users
>> on our network, or to identify the 'unwanted' traffic so that we could
>> either
>> block it, or use it to identify the rogue users ourselves.
>>
>> Has anyone else come up against the problem, and/or have any suggestions on
>> how best to resolve it?
>>
>> Many thanks in advance,
>>
>> Simon
>>
>>
>

In message <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net>, Justin Wilson write
s:
> PSN is one reason I am not a fan of CGNAT. All they see are tons of
> connections from the same IP. This results in them banning folks. Due
> to them being hacked so many times getting them to actually communicate
> is almost impossible. My .02 is just get the gamers a true public if at
> all possible.
>
> Justin Wilson
> j2sw@mtin.net

What we need is business tech reporters to continually report on
these failures of content providers to deliver their services over
IPv6. 20 years lead time should be enough for any service.

Mark
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

Mark Andrews writes:
>
> In message <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net>, Justin Wilson wri
> te
> s:
> > PSN is one reason I am not a fan of CGNAT. All they see are tons of
> > connections from the same IP. This results in them banning folks. Due
> > to them being hacked so many times getting them to actually communicate
> > is almost impossible. My .02 is just get the gamers a true public if at
> > all possible.
> >
> > Justin Wilson
> > j2sw@mtin.net
>
> What we need is business tech reporters to continually report on
> these failures of content providers to deliver their services over
> IPv6. 20 years lead time should be enough for any service.

Additionally is the a role for the SEC in ensuring that companies
take IPv6 seriously? If I remember correctly they got involved
with Y2K. Just because there isn't a hard date it doesn't mean
that IPv6 is any less important than Y2K to your business's survival.

> Mark
> --
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

On Wed, 21 Sep 2016 11:29:49 +1000, Mark Andrews said:

> What we need is business tech reporters to continually report on
> these failures of content providers to deliver their services over
> IPv6. 20 years lead time should be enough for any service.

Interestingly enough, the Playstation 4 has at least rudimentary IPv6
support - it will DHCPv6 and answer pings. Threw me for a loop first
time I saw it, I couldn't figure out what unaccounted-for gear I had
that was grabbing an IPv6 address... :)

Hi

We have the opposite problem with PSN: Sometimes they will send abuse
reports with several of our IP addresses listed. The problem with that
is that we can not give data about one customer to another customer. By
listing multiple IP addresses we are prevented from forwarding the email
to the customer. Which means we may ignore it instead.

Regards,

Baldur

On 21 Sep 2016, at 15:37, Baldur Norddahl wrote:

> Which means we may ignore it instead.

. . . copy/paste or awk/sed or whatever isn't an option? If not, have
you requested a) separate notifications per source and/or b) a more
textual-manipulation-friendly format? Unless they're sending .gifs or
something, surely this might be possible, yes?

It seems within the realm of possibility this sort of response - or lack
thereof - could result in some gaming network operators becoming a bit
jaded. And perhaps some customers, too.

-----------------------------------
Roland Dobbins <rdobbins@arbor.net>

I have a hard time accepting that service providers should re-engineer
their networks because other companies cannot properly engineer their abuse
tooling.

On Tue, Sep 20, 2016 at 11:33 AM, Justin Wilson <lists@mtin.net> wrote:

> PSN is one reason I am not a fan of CGNAT. All they see are tons of
> connections from the same IP. This results in them banning folks. Due to
> them being hacked so many times getting them to actually communicate is
> almost impossible. My .02 is just get the gamers a true public if at all
> possible.
>
> Justin Wilson
> j2sw@mtin.net
>
> ---
> http://www.mtin.net Owner/CEO
> xISP Solutions- Consulting – Data Centers - Bandwidth
>
> http://www.midwest-ix.com COO/Chairman
> Internet Exchange - Peering - Distributed Fabric
>
> > On Sep 20, 2016, at 8:24 AM, Danijel Starman <theghost101@gmail.com>
> wrote:
> >
> > Something similar happened to a local FantasyConon I was helping set up,
> we
> > had only two PS4 machines there and accounts provided by Blizzard for
> > Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in
> > about 8h. There was no other traffic other then those two accounts
> playing
> > Overwatch so my guess is that they have some too aggressive checks. I've
> > managed to convince our ISP there to change the outside IP of the link so
> > we got them working the next day but it happened again in 8h.
> >
> > --
> > *blap*
> >
> > On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart <simon@slimey.org>
> wrote:
> >
> >> All,
> >>
> >> We operate an access network with several hundred thousand users.
> >> Increasingly
> >> we're putting the users behind CGNAT in order to continue to give them
> an
> >> IPv4
> >> service (we're all dual-stack, so they all get public IPv6 too). Due to
> the
> >> demographic of our users, many of them are gamers.
> >>
> >> We're hitting a problem with PlayStationNetwork 'randomly' blocking some
> >> of our
> >> CGNAT outside addresses, because they claim to have received anomalous,
> or
> >> 'attack' traffic from that IP. This obviously causes problems for the
> other
> >> legitimate users who end up behind the same public IPv4 address.
> >>
> >> Despite numerous attempts to engage with PSN, they are unwilling to
> give us
> >> any additional information which would allow us to identify the 'rogue'
> >> users
> >> on our network, or to identify the 'unwanted' traffic so that we could
> >> either
> >> block it, or use it to identify the rogue users ourselves.
> >>
> >> Has anyone else come up against the problem, and/or have any
> suggestions on
> >> how best to resolve it?
> >>
> >> Many thanks in advance,
> >>
> >> Simon
> >>
> >>
> >
>
>

Both gamers and content providers do not care. The gamers as they only care about the game itself and don't care about the technical mumbo jumbo. And the makers coz they only care about making money by producing content the gamers want. And you service providers are left with the headache of attempts to please both sides.
If this wasn't the case, then why after 20 years, ipv6 ain't rolled out.
Hence again I'd be voting for an ipv6 only day, but that will never happen.....
Kind regards,
Alexander Maassen
- Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer

-------- Oorspronkelijk bericht --------Van: Mark Andrews <marka@isc.org> Datum: 21-09-16 03:29 (GMT+01:00) Aan: Justin Wilson <lists@mtin.net> Cc: NANOG <nanog@nanog.org> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses

In message <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net>, Justin Wilson write
s:
> PSN is one reason I am not a fan of CGNAT. All they see are tons of
> connections from the same IP.  This results in them banning folks.  Due
> to them being hacked so many times getting them to actually communicate
> is almost impossible.  My .02 is just get the gamers a true public if at
> all possible.
>
> Justin Wilson
> j2sw@mtin.net

What we need is business tech reporters to continually report on
these failures of content providers to deliver their services over
IPv6.  20 years lead time should be enough for any service.

Mark
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: marka@isc.org

So you ignore/don't deal with the abuse coz it's shipped in a format you refuse to handle?
And you don't even bother telling the reporter you would like it in a per ip format? Or make attempts to make it work the way they report it (split out the ip's and modify the to be forwarded mail to only contain the ip's belonging to that customer)????

Kind regards,
Alexander Maassen
- Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer

-------- Oorspronkelijk bericht --------Van: Baldur Norddahl <baldur.norddahl@gmail.com> Datum: 21-09-16 10:37 (GMT+01:00) Aan: nanog@nanog.org Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses
Hi

We have the opposite problem with PSN: Sometimes they will send abuse
reports with several of our IP addresses listed. The problem with that
is that we can not give data about one customer to another customer. By
listing multiple IP addresses we are prevented from forwarding the email
to the customer. Which means we may ignore it instead.

Regards,

Baldur

As long as their is no international accepted standard as to how to report abuse and everyone cooking up his/her own methods.. I think you have either the choice of adapting and thus be able to deal with the abuse. Or be lazy and stubborn, ignore it, wait for the bad reputation to say hi to your company and face the effects it might cause.


Kind regards,
Alexander Maassen
- Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer

-------- Oorspronkelijk bericht --------Van: Tom Beecher <beecher@beecher.cc> Datum: 21-09-16 17:13 (GMT+01:00) Aan: Justin Wilson <lists@mtin.net> Cc: NANOG <nanog@nanog.org> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses
I have a hard time accepting that service providers should re-engineer
their networks because other companies cannot properly engineer their abuse
tooling.

On Tue, Sep 20, 2016 at 11:33 AM, Justin Wilson <lists@mtin.net> wrote:

> PSN is one reason I am not a fan of CGNAT. All they see are tons of
> connections from the same IP.  This results in them banning folks.  Due to
> them being hacked so many times getting them to actually communicate is
> almost impossible.  My .02 is just get the gamers a true public if at all
> possible.
>
> Justin Wilson
> j2sw@mtin.net
>
> ---
> http://www.mtin.net Owner/CEO
> xISP Solutions- Consulting – Data Centers - Bandwidth
>
> http://www.midwest-ix.com  COO/Chairman
> Internet Exchange - Peering - Distributed Fabric
>
> > On Sep 20, 2016, at 8:24 AM, Danijel Starman <theghost101@gmail.com>
> wrote:
> >
> > Something similar happened to a local FantasyConon I was helping set up,
> we
> > had only two PS4 machines there and accounts provided by Blizzard for
> > Overwatch. Outside IP of the LAN (as it was NATed) was banned by PSN in
> > about 8h. There was no other traffic other then those two accounts
> playing
> > Overwatch so my guess is that they have some too aggressive checks. I've
> > managed to convince our ISP there to change the outside IP of the link so
> > we got them working the next day but it happened again in 8h.
> >
> > --
> > *blap*
> >
> > On Fri, Sep 16, 2016 at 3:12 PM, Simon Lockhart <simon@slimey.org>
> wrote:
> >
> >> All,
> >>
> >> We operate an access network with several hundred thousand users.
> >> Increasingly
> >> we're putting the users behind CGNAT in order to continue to give them
> an
> >> IPv4
> >> service (we're all dual-stack, so they all get public IPv6 too). Due to
> the
> >> demographic of our users, many of them are gamers.
> >>
> >> We're hitting a problem with PlayStationNetwork 'randomly' blocking some
> >> of our
> >> CGNAT outside addresses, because they claim to have received anomalous,
> or
> >> 'attack' traffic from that IP. This obviously causes problems for the
> other
> >> legitimate users who end up behind the same public IPv4 address.
> >>
> >> Despite numerous attempts to engage with PSN, they are unwilling to
> give us
> >> any additional information which would allow us to identify the 'rogue'
> >> users
> >> on our network, or to identify the 'unwanted' traffic so that we could
> >> either
> >> block it, or use it to identify the rogue users ourselves.
> >>
> >> Has anyone else come up against the problem, and/or have any
> suggestions on
> >> how best to resolve it?
> >>
> >> Many thanks in advance,
> >>
> >> Simon
> >>
> >>
> >
>
>

If you told them they would have fewer NAT issues if they supported IPv6, they'd start to care. ;-) They know enough to hate NAT.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Alexander Maassen" <outsider@scarynet.org>
Cc: "NANOG" <nanog@nanog.org>
Sent: Thursday, September 22, 2016 3:35:01 AM
Subject: Re: PlayStationNetwork blocking of CGNAT public addresses

Both gamers and content providers do not care. The gamers as they only care about the game itself and don't care about the technical mumbo jumbo. And the makers coz they only care about making money by producing content the gamers want. And you service providers are left with the headache of attempts to please both sides.
If this wasn't the case, then why after 20 years, ipv6 ain't rolled out.
Hence again I'd be voting for an ipv6 only day, but that will never happen.....
Kind regards,
Alexander Maassen
- Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer

-------- Oorspronkelijk bericht --------Van: Mark Andrews <marka@isc.org> Datum: 21-09-16 03:29 (GMT+01:00) Aan: Justin Wilson <lists@mtin.net> Cc: NANOG <nanog@nanog.org> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses

In message <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net>, Justin Wilson write
s:
> PSN is one reason I am not a fan of CGNAT. All they see are tons of
> connections from the same IP. This results in them banning folks. Due
> to them being hacked so many times getting them to actually communicate
> is almost impossible. My .02 is just get the gamers a true public if at
> all possible.
>
> Justin Wilson
> j2sw@mtin.net

What we need is business tech reporters to continually report on
these failures of content providers to deliver their services over
IPv6. 20 years lead time should be enough for any service.

Mark
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

On 22 September 2016 at 10:42, Alexander Maassen <outsider@scarynet.org>
wrote:

> So you ignore/don't deal with the abuse coz it's shipped in a format you
> refuse to handle?
>
> And you don't even bother telling the reporter you would like it in a per
> ip format? Or make attempts to make it work the way they report it (split
> out the ip's and modify the to be forwarded mail to only contain the ip's
> belonging to that customer)????
>

You will have to remember that these are automated mails from the reporter.
If I write them back it goes into their bit bucket, because they do not
really care enough to bother replying. I am betting they are sending out
thousands mails each day and they can not handle manually replying to all
of that. In the same way we receive a large amount of automated mail so we
have to be able to handle it automatically. Send me something sane and I
will make a script that forwards it. Send me something unusable and I wont
- but I will not do manual handling of your automated mail.

All I am trying to do here is tell people that send abuse mails not to
combine multiple abuse complaints in one mail, because that makes it harder
for everybody and makes it more likely that your mail will be dropped as
too much work. Double so if your abuse mails is from an automated system,
because I will try to match your automated system with my own. However it
is much harder to make a system that can edit your complaint and duplicate
it to several recipients, than it is to run a simple filter that just
forwards the mail as is.

As to PSN they will usually send multiple mails if the abuse is ongoing. At
some point they will send a mail with just one IP and that one gets
forwarded. So we are dropping some of the mails, but the users eventually
get notified anyway. It is not ideal but it works.

Regards,

Baldur

Maybe its time then for a global accepted, unified way to send/report abuse? 
That should solve most of the issues and end points would be able to deal with it in a common way and only would need to think about how to integrate it in their crm's etc.
We are all using the same medium, but attempt to communicate issues using several methods. 
Perhaps iana can use those (m/b)illions they got from selling tld's and cook something up.



Kind regards,
Alexander Maassen
- Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer

-------- Oorspronkelijk bericht --------Van: Baldur Norddahl <baldur.norddahl@gmail.com> Datum: 22-09-16 14:10 (GMT+01:00) Aan: nanog@nanog.org Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses
On 22 September 2016 at 10:42, Alexander Maassen <outsider@scarynet.org>
wrote:

> So you ignore/don't deal with the abuse coz it's shipped in a format you
> refuse to handle?
>
> And you don't even bother telling the reporter you would like it in a per
> ip format? Or make attempts to make it work the way they report it (split
> out the ip's and modify the to be forwarded mail to only contain the ip's
> belonging to that customer)????
>

You will have to remember that these are automated mails from the reporter.
If I write them back it goes into their bit bucket, because they do not
really care enough to bother replying. I am betting they are sending out
thousands mails each day and they can not handle manually replying to all
of that. In the same way we receive a large amount of automated mail so we
have to be able to handle it automatically. Send me something sane and I
will make a script that forwards it. Send me something unusable and I wont
- but I will not do manual handling of your automated mail.

All I am trying to do here is tell people that send abuse mails not to
combine multiple abuse complaints in one mail, because that makes it harder
for everybody and makes it more likely that your mail will be dropped as
too much work. Double so if your abuse mails is from an automated system,
because I will try to match your automated system with my own. However it
is much harder to make a system that can edit your complaint and duplicate
it to several recipients, than it is to run a simple filter that just
forwards the mail as is.

As to PSN they will usually send multiple mails if the abuse is ongoing. At
some point they will send a mail with just one IP and that one gets
forwarded. So we are dropping some of the mails, but the users eventually
get notified anyway. It is not ideal but it works.

Regards,

Baldur

Ipv6 is there for 20+ years, cgnat is needed coz the net grows kinda exponentially due to stuff like IoT/mobiles/m2m, and isp's need to provide users with the ability to talk ipv4 simply because the other side refuses to deploy v6 abilities. Do the math if they really care.
Also the servers itself hosting the gameserver probably already are dual stacked. But the gamecode itself misses the support.
Then there is the issue of you as isp not being able or daring to show a fist and simply saying: screw you. Because you are risking to loose customers.
And as long as the company's earn plenty of money using outdated code, they won't change it, coz that would imply spending money that won't flow into fancy buildings, fast cars and all that other useless luxury.




Kind regards,
Alexander Maassen
- Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer

-------- Oorspronkelijk bericht --------Van: Mike Hammett <nanog@ics-il.net> Datum: 22-09-16 13:23 (GMT+01:00) Aan: Alexander Maassen <outsider@scarynet.org> Cc: NANOG <nanog@nanog.org> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses
If you told them they would have fewer NAT issues if they supported IPv6, they'd start to care.  ;-) They know enough to hate NAT.



-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

From: "Alexander Maassen" <outsider@scarynet.org>
Cc: "NANOG" <nanog@nanog.org>
Sent: Thursday, September 22, 2016 3:35:01 AM
Subject: Re: PlayStationNetwork blocking of CGNAT public addresses

Both gamers and content providers do not care. The gamers as they only care about the game itself and don't care about the technical mumbo jumbo. And the makers coz they only care about making money by producing content the gamers want. And you service providers are left with the headache of attempts to please both sides.
If this wasn't the case, then why after 20 years, ipv6 ain't rolled out.
Hence again I'd be voting for an ipv6 only day, but that will never happen.....
Kind regards,
Alexander Maassen
- Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL- Peplink Certified Engineer

-------- Oorspronkelijk bericht --------Van: Mark Andrews <marka@isc.org> Datum: 21-09-16  03:29  (GMT+01:00) Aan: Justin Wilson <lists@mtin.net> Cc: NANOG <nanog@nanog.org> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses

In message <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net>, Justin Wilson write
s:
> PSN is one reason I am not a fan of CGNAT. All they see are tons of
> connections from the same IP.  This results in them banning folks.  Due
> to them being hacked so many times getting them to actually communicate
> is almost impossible.  My .02 is just get the gamers a true public if at
> all possible.
>
> Justin Wilson
> j2sw@mtin.net

What we need is business tech reporters to continually report on
these failures of content providers to deliver their services over
IPv6.  20 years lead time should be enough for any service.

Mark
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: marka@isc.org

On Thursday, September 22, 2016, Alexander Maassen <outsider@scarynet.org>
wrote:

> Both gamers and content providers do not care. The gamers as they only
> care about the game itself and don't care about the technical mumbo jumbo.
> And the makers coz they only care about making money by producing content
> the gamers want. And you service providers are left with the headache of
> attempts to please both sides.


Very much agree


> If this wasn't the case, then why after 20 years, ipv6 ain't rolled out.
> Hence again I'd be voting for an ipv6 only day, but that will never
> happen.....


Disagree. IPv6 is meaningfully rolled out. Half or comcast and at&t subs
are observably on ipv6

http://www.worldipv6launch.org/measurements/

And every (i think) iphone 7 ships with ipv6 default on from t-mobile,
sprint, T , and VZ. Same can be said of samsung phones 2 years ago.

Now, if abc isp and xyz gaming company don't deploy ipv6, they have nobody
to blame but themselves. Many of us have moved on, but it is sad when
you all need help tweeking your cgn or need help finding an IPv4 broker.

I feel your pain. But don't say ipv6 is not deployed. It is deployed, and
it carries more traffic than ipv4

http://www.internetsociety.org/deploy360/blog/2016/08/facebook-akamai-pass-major-milestone-over-50-ipv6-from-us-mobile-networks/

CB

Kind regards,
> Alexander Maassen
> - Technical Maintenance Engineer Parkstad Support BV- Maintainer DroneBL-
> Peplink Certified Engineer
>
> -------- Oorspronkelijk bericht --------Van: Mark Andrews <marka@isc.org
> <javascript:;>> Datum: 21-09-16 03:29 (GMT+01:00) Aan: Justin Wilson <
> lists@mtin.net <javascript:;>> Cc: NANOG <nanog@nanog.org <javascript:;>>
> Onderwerp: Re: PlayStationNetwork blocking of CGNAT public addresses
>
> In message <09342130-874F-4FA4-B410-B7B66A75FA4D@mtin.net <javascript:;>>,
> Justin Wilson write
> s:
> > PSN is one reason I am not a fan of CGNAT. All they see are tons of
> > connections from the same IP. This results in them banning folks. Due
> > to them being hacked so many times getting them to actually communicate
> > is almost impossible. My .02 is just get the gamers a true public if at
> > all possible.
> >
> > Justin Wilson
> > j2sw@mtin.net <javascript:;>
>
> What we need is business tech reporters to continually report on
> these failures of content providers to deliver their services over
> IPv6. 20 years lead time should be enough for any service.
>
> Mark
> --
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
> <javascript:;>
>

On 9/22/2016 8:10 AM, Baldur Norddahl wrote:
> On 22 September 2016 at 10:42, Alexander Maassen <outsider@scarynet.org>
> wrote:
>
>> So you ignore/don't deal with the abuse coz it's shipped in a format you
>> refuse to handle?
>>
>> And you don't even bother telling the reporter you would like it in a per
>> ip format? Or make attempts to make it work the way they report it (split
>> out the ip's and modify the to be forwarded mail to only contain the ip's
>> belonging to that customer)????
>>
> You will have to remember that these are automated mails from the reporter.
> If I write them back it goes into their bit bucket, because they do not
> really care enough to bother replying. I am betting they are sending out
> thousands mails each day and they can not handle manually replying to all
> of that. In the same way we receive a large amount of automated mail so we
> have to be able to handle it automatically. Send me something sane and I
> will make a script that forwards it. Send me something unusable and I wont
> - but I will not do manual handling of your automated mail.
>
> All I am trying to do here is tell people that send abuse mails not to
> combine multiple abuse complaints in one mail, because that makes it harder
> for everybody and makes it more likely that your mail will be dropped as
> too much work. Double so if your abuse mails is from an automated system,
> because I will try to match your automated system with my own. However it
> is much harder to make a system that can edit your complaint and duplicate
> it to several recipients, than it is to run a simple filter that just
> forwards the mail as is.
>
> As to PSN they will usually send multiple mails if the abuse is ongoing. At
> some point they will send a mail with just one IP and that one gets
> forwarded. So we are dropping some of the mails, but the users eventually
> get notified anyway. It is not ideal but it works.
>
> Regards,
>
> Baldur

We've also started ignoring their abuse emails, for the same reason.
Their abuse emails at one point contained the line:

> P.S. If you would prefer an individual email for each IP address on
this list, please let us know.

But, they didn't respond after we contacted them requesting it (and that
line has since been removed).

Considering that there are likely to be many such emails - just how much time is it going to take your abuse desk staffer to just parse out those IPs from whatever log that they send you?

And how much time would processing say 50 individual emails take compared to 50 IPs in a single email?

--srs

> On 22-Sep-2016, at 6:58 PM, Brian Rak <brak@gameservers.com> wrote:
>
> We've also started ignoring their abuse emails, for the same reason. Their abuse emails at one point contained the line:
>
> > P.S. If you would prefer an individual email for each IP address on this list, please let us know.
>
> But, they didn't respond after we contacted them requesting it (and that line has since been removed).

Single IP per email: automated, zero time at all.

Multiple IPs per email: manual process, minutes per IP.


On 9/22/2016 9:34 AM, Suresh Ramasubramanian wrote:
> Considering that there are likely to be many such emails - just how
> much time is it going to take your abuse desk staffer to just parse
> out those IPs from whatever log that they send you?
>
> And how much time would processing say 50 individual emails take
> compared to 50 IPs in a single email?
>
> --srs
>
> On 22-Sep-2016, at 6:58 PM, Brian Rak <brak@gameservers.com
> <mailto:brak@gameservers.com>> wrote:
>
>> We've also started ignoring their abuse emails, for the same reason.
>> Their abuse emails at one point contained the line:
>>
>> > P.S. If you would prefer an individual email for each IP address on this list, please let us know.
>>
>> But, they didn't respond after we contacted them requesting it (and
>> that line has since been removed).

The format of the abuse complaint doesn't mean anything if it still doesn't
contain any relevant data to say what the abuse IS. (Or, even if it IS
abuse at all.)




On Thu, Sep 22, 2016 at 9:37 AM, Brian Rak <brak@gameservers.com> wrote:

> Single IP per email: automated, zero time at all.
>
> Multiple IPs per email: manual process, minutes per IP.
>
>
> On 9/22/2016 9:34 AM, Suresh Ramasubramanian wrote:
>
>> Considering that there are likely to be many such emails - just how much
>> time is it going to take your abuse desk staffer to just parse out those
>> IPs from whatever log that they send you?
>>
>> And how much time would processing say 50 individual emails take compared
>> to 50 IPs in a single email?
>>
>> --srs
>>
>> On 22-Sep-2016, at 6:58 PM, Brian Rak <brak@gameservers.com <mailto:
>> brak@gameservers.com>> wrote:
>>
>> We've also started ignoring their abuse emails, for the same reason.
>>> Their abuse emails at one point contained the line:
>>>
>>> > P.S. If you would prefer an individual email for each IP address on
>>> this list, please let us know.
>>>
>>> But, they didn't respond after we contacted them requesting it (and that
>>> line has since been removed).
>>>
>>
>

http://x-arf.org/ ?

--
Hugo Slabbert       | email, xmpp/jabber: hugo@slabnet.com
pgp key: B178313E   | also on Signal

On September 22, 2016 5:31:12 AM PDT, Alexander Maassen <outsider@scarynet.org> wrote:
>Maybe its time then for a global accepted, unified way to send/report
>abuse? 
>That should solve most of the issues and end points would be able to
>deal with it in a common way and only would need to think about how to
>integrate it in their crm's etc.
>We are all using the same medium, but attempt to communicate issues
>using several methods. 
>Perhaps iana can use those (m/b)illions they got from selling tld's and
>cook something up.
>
>
>
>Kind regards,
>Alexander Maassen
>- Technical Maintenance Engineer Parkstad Support BV- Maintainer
>DroneBL- Peplink Certified Engineer
>

On Thu, 22 Sep 2016 14:31:12 +0200, Alexander Maassen said:
> Maybe its time then for a global accepted, unified way to send/report abuse?

YOu mean ike these RFCs? (OK, so it's an XML schema. Just be glad
it isn't ASN.1 :)

5070 The Incident Object Description Exchange Format. R. Danyliw, J.
Meijer, Y. Demchenko. December 2007. (Format: TXT=171529 bytes)
(Updated by RFC6685) (Status: PROPOSED STANDARD) (DOI:
10.17487/RFC5070)

6684 Guidelines and Template for Defining Extensions to the Incident
Object Description Exchange Format (IODEF). B. Trammell. July 2012.
(Format: TXT=23550 bytes) (Status: INFORMATIONAL) (DOI:
10.17487/RFC6684)

6685 Expert Review for Incident Object Description Exchange Format
(IODEF) Extensions in IANA XML Registry. B. Trammell. July 2012.
(Format: TXT=4363 bytes) (Updates RFC5070) (Status: PROPOSED
STANDARD) (DOI: 10.17487/RFC6685)

7203 An Incident Object Description Exchange Format (IODEF) Extension for
Structured Cybersecurity Information. T. Takahashi, K. Landfield, Y.
Kadobayashi. April 2014. (Format: TXT=57694 bytes) (Status: PROPOSED
STANDARD) (DOI: 10.17487/RFC7203)

7495 Enumeration Reference Format for the Incident Object Description
Exchange Format (IODEF). A. Montville, D. Black. March 2015.
(Format: TXT=19891 bytes) (Status: PROPOSED STANDARD) (DOI:
10.17487/RFC7495)

Well yes – if you have the automation, that is great.



Of course the format of whatever log they send you matters too.



I’ve had abuse complaints in a past life where the abuse report was a screenshot from a checkpoint firewall with “Dear team, for your attention” in bright red in a large font.



Personally I don’t trash abuse reports that are valid.



--srs



From: Tom Beecher <beecher@beecher.cc>
Date: Thursday, 22 September 2016 at 7:35 PM
To: Brian Rak <brak@gameservers.com>
Cc: Suresh Ramasubramanian <ops.lists@gmail.com>, "nanog@nanog.org" <nanog@nanog.org>
Subject: Re: PlayStationNetwork blocking of CGNAT public addresses



The format of the abuse complaint doesn't mean anything if it still doesn't contain any relevant data to say what the abuse IS. (Or, even if it IS abuse at all.)









On Thu, Sep 22, 2016 at 9:37 AM, Brian Rak <brak@gameservers.com> wrote:

Single IP per email: automated, zero time at all.

Multiple IPs per email: manual process, minutes per IP.


On 9/22/2016 9:34 AM, Suresh Ramasubramanian wrote:

Considering that there are likely to be many such emails - just how much time is it going to take your abuse desk staffer to just parse out those IPs from whatever log that they send you?

And how much time would processing say 50 individual emails take compared to 50 IPs in a single email?

--srs

On 22-Sep-2016, at 6:58 PM, Brian Rak <brak@gameservers.com <mailto:brak@gameservers.com>> wrote:

We've also started ignoring their abuse emails, for the same reason. Their abuse emails at one point contained the line:

> P.S. If you would prefer an individual email for each IP address on this list, please let us know.

But, they didn't respond after we contacted them requesting it (and that line has since been removed).

On Mon, Sep 19, 2016 at 09:55:56PM +0200, Florian Weimer wrote:
> Github users create several orders of magnitude more SSH connections
> [snip]

Ah. I didn't know that. Thanks!

> Sure, and people already do this, and are not very flexible about it.
> Support staff isn't briefed, and claim they do such stochastic
> behavior adjustment across all (server) products, which I find
> difficult to believe.

You're right: those are serious drawbacks. If folks are going to do
this, then they need to do it right, which means making sure everyone
is in the loop and making sure that support staff are clueful/diligent
enough to investigate -- or at least hand off to someone who'll investigate.
This stuff works but only if you're adaptive/flexible and willing to
learn and adjust on an ongoing basis.

> I'm worried that this leads to a future where tunnelling everything
> over HTTP(S) is no longer sufficient. You have to make it look like a
> web server or browser, too. Everything else risks triggering
> automated countermeasures.

And as someone who constantly beats the "Internet != web" drum,
I second this. Marginalizing other protocols doesn't serve us well
in short term (it breaks things) or the long term (it stifles innovation).

---rsk

On Thu, Sep 22, 2016 at 02:31:12PM +0200, Alexander Maassen wrote:
> Maybe its time then for a global accepted, unified way to send/report abuse???

There are -- see Valdis's followup.

But there's still no viable substitute for a working abuse@ address
with clueful eyeballs on the other side of it. Every responsible and
professional operation on this planet has that. The really good
ones learn from what shows up there and pro-actively deal with abuse
issues before anyone else is bothered by them, which not only makes
them better netizens but reduces the volume of incoming complaints.

---rsk