Mailing List Archive

I believe all devices will translate a privileged ports, but it won't translate to the same number on the other side. It will translate to an unprivileged port. Is it what you meant or really there are some devices that will not translate at all a privileged port?

What are you trying to achieve?

Jean

-----Original Message-----
From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Fernando Gont
Sent: June 4, 2021 3:00 AM
To: nanog@nanog.org
Subject: NAT devices not translating privileged ports

Folks,

While discussing port randomization (in the context of https://www.ietf.org/archive/id/draft-ietf-ntp-port-randomization-06.txt
), it has been raised to us that some NAT devices do not translate the source port if the source port is a privileged port (<1024).

Any clues/examples of this type of NATs?

Thanks!

Regards,
--
Fernando Gont
Director of Information Security
EdgeUno, Inc.
PGP Fingerprint: DFBD 63E3 B248 AE79 C598 AF23 EBAE DA03 0644 1531

Current gen Cisco ASA firewalls have logic so that if the connection
from a private host originated from a privileged source port, the NAT
translation to public IP also uses an unprivileged source port (not
necessarily the same source port though).

I found out that this behavior can cause issues when you have devices on
your network that implement older DNS libraries or configs using UDP 53
as a source and destination port for their DNS lookups. Occasionally the
source port gets translated to one that ISC BIND servers have in a
blocklist (chargen, echo, time, and a few others) and the query is
ignored. As I recall, this behavior is hard coded so patching and
recompiling BIND is required to work around it.

I forget what the older ASA behavior was. It may have been to leave the
source port unchanged through the NAT process (I think this is what you
mean by "not translated"). In that case the client doesn't implement
source port randomization and the NAT doesn't "upgrade" the connection
to a random source port so I don't really see it as an issue. Ideally
the client would implement source port randomization itself so it would
be using source ports within its ephemeral port range for outgoing
connections.

--Blake


On 6/4/2021 7:36 AM, Jean St-Laurent via NANOG wrote:
> I believe all devices will translate a privileged ports, but it won't translate to the same number on the other side. It will translate to an unprivileged port. Is it what you meant or really there are some devices that will not translate at all a privileged port?
>
> What are you trying to achieve?
>
> Jean
>
> -----Original Message-----
> From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Fernando Gont
> Sent: June 4, 2021 3:00 AM
> To: nanog@nanog.org
> Subject: NAT devices not translating privileged ports
>
> Folks,
>
> While discussing port randomization (in the context of https://www.ietf.org/archive/id/draft-ietf-ntp-port-randomization-06.txt
> ), it has been raised to us that some NAT devices do not translate the source port if the source port is a privileged port (<1024).
>
> Any clues/examples of this type of NATs?
>
> Thanks!
>
> Regards,
> --
> Fernando Gont
> Director of Information Security
> EdgeUno, Inc.
> PGP Fingerprint: DFBD 63E3 B248 AE79 C598 AF23 EBAE DA03 0644 1531
>
>
>
>
>

For Linux iptables SNAT (used with --to-source), the default is to change
the packet as little as possible.

https://linux.die.net/man/8/iptables
"If no port range is specified, then source ports below 512 will be mapped
to other ports below 512: those between 512 and 1023 inclusive will be
mapped to ports below 1024, and other ports will be mapped to 1024 or
above.
Where possible, no port alteration will occur."

So, if there are no "collisions", the same src port will be used. If there
are "collisions" (multiple flows with the same src port and dst IP/port),
then another src port within its "range" will be used.

But it can be configured, for example, to use ports 1024-65535, in which
case flows with src port < 1024 will endup using ports > 1024 after they
are NATed.

https://datatracker.ietf.org/doc/html/rfc6335#section-6 is also a good
reference.

Alvaro

On Fri, Jun 4, 2021 at 10:14 AM Blake Hudson <blake@ispn.net> wrote:

> Current gen Cisco ASA firewalls have logic so that if the connection
> from a private host originated from a privileged source port, the NAT
> translation to public IP also uses an unprivileged source port (not
> necessarily the same source port though).
>
> I found out that this behavior can cause issues when you have devices on
> your network that implement older DNS libraries or configs using UDP 53
> as a source and destination port for their DNS lookups. Occasionally the
> source port gets translated to one that ISC BIND servers have in a
> blocklist (chargen, echo, time, and a few others) and the query is
> ignored. As I recall, this behavior is hard coded so patching and
> recompiling BIND is required to work around it.
>
> I forget what the older ASA behavior was. It may have been to leave the
> source port unchanged through the NAT process (I think this is what you
> mean by "not translated"). In that case the client doesn't implement
> source port randomization and the NAT doesn't "upgrade" the connection
> to a random source port so I don't really see it as an issue. Ideally
> the client would implement source port randomization itself so it would
> be using source ports within its ephemeral port range for outgoing
> connections.
>
> --Blake
>
>
> On 6/4/2021 7:36 AM, Jean St-Laurent via NANOG wrote:
> > I believe all devices will translate a privileged ports, but it won't
> translate to the same number on the other side. It will translate to an
> unprivileged port. Is it what you meant or really there are some devices
> that will not translate at all a privileged port?
> >
> > What are you trying to achieve?
> >
> > Jean
> >
> > -----Original Message-----
> > From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of
> Fernando Gont
> > Sent: June 4, 2021 3:00 AM
> > To: nanog@nanog.org
> > Subject: NAT devices not translating privileged ports
> >
> > Folks,
> >
> > While discussing port randomization (in the context of
> https://www.ietf.org/archive/id/draft-ietf-ntp-port-randomization-06.txt
> > ), it has been raised to us that some NAT devices do not translate the
> source port if the source port is a privileged port (<1024).
> >
> > Any clues/examples of this type of NATs?
> >
> > Thanks!
> >
> > Regards,
> > --
> > Fernando Gont
> > Director of Information Security
> > EdgeUno, Inc.
> > PGP Fingerprint: DFBD 63E3 B248 AE79 C598 AF23 EBAE DA03 0644 1531
> >
> >
> >
> >
> >
>
>

Hi, Blake,

Thanks a lot for your comments! In-line....


On Fri, 2021-06-04 at 11:13 -0500, Blake Hudson wrote:
> Current gen Cisco ASA firewalls have logic so that if the connection
> from a private host originated from a privileged source port, the
> NAT
> translation to public IP also uses an unprivileged source port (not
> necessarily the same source port though).

Did you actaully mean "...also uses a *privileged port*"?



>
> I found out that this behavior can cause issues when you have devices
> on
> your network that implement older DNS libraries or configs using UDP
> 53
> as a source and destination port for their DNS lookups. Occasionally
> the
> source port gets translated to one that ISC BIND servers have in a
> blocklist (chargen, echo, time, and a few others) and the query is
> ignored. As I recall, this behavior is hard coded so patching and
> recompiling BIND is required to work around it.
>
> I forget what the older ASA behavior was. It may have been to leave
> the
> source port unchanged through the NAT process (I think this is what
> you
> mean by "not translated"). In that case the client doesn't implement
> source port randomization and the NAT doesn't "upgrade" the
> connection
> to a random source port so I don't really see it as an issue.

The issue would be that if the port is not translated, and multiple
systems in the internal real of the NAT try to use the same privileged
port (say, 123) simultaneously, things wouldn't work.



Thanks,
--
Fernando Gont
Director of Information Security
EdgeUno, Inc.
PGP Fingerprint: DFBD 63E3 B248 AE79 C598 AF23 EBAE DA03 0644 1531

Hi, Jean,

On Fri, 2021-06-04 at 08:36 -0400, Jean St-Laurent wrote:
> I believe all devices will translate a privileged ports, but it won't
> translate to the same number on the other side. It will translate to
> an unprivileged port. Is it what you meant or really there are some
> devices that will not translate at all a privileged port?

What has been reported to us is that some boxes do not translate the
src port if it's a privileged port.

IN such scenarios, NTP implementations that always use src port=123,
dst port=123 might be in trouble if there are multiple NTP clients
behind the same NAT device....

Thanks!

Regards,
--
Fernando Gont
Director of Information Security
EdgeUno, Inc.
PGP Fingerprint: DFBD 63E3 B248 AE79 C598 AF23 EBAE DA03 0644 1531

Fernando Gont via NANOG <nanog@nanog.org> writes:

> What has been reported to us is that some boxes do not translate the
> src port if it's a privileged port.
>
> IN such scenarios, NTP implementations that always use src port=123,
> dst port=123 might be in trouble if there are multiple NTP clients
> behind the same NAT device....

This problem used to be very common for 500/udp. Ref
https://datatracker.ietf.org/doc/html/rfc3715#section-2.3


Bjørn

Hi, Bjørn,

On Thu, 2021-06-10 at 12:10 +0200, Bjørn Mork wrote:
> Fernando Gont via NANOG <nanog@nanog.org> writes:
>
> > What has been reported to us is that some boxes do not translate
> > the
> > src port if it's a privileged port.
> >
> > IN such scenarios, NTP implementations that always use src
> > port=123,
> > dst port=123 might be in trouble if there are multiple NTP clients
> > behind the same NAT device....
>
> This problem used to be very common for 500/udp. Ref
> https://datatracker.ietf.org/doc/html/rfc3715#section-2.3

THanks a lot for the link! -- this is indeed a good read. I'm curious
if there exists something similar for UDP/123?


FWIW, we have this IETF I-D on NTP port randomization:
https://datatracker.ietf.org/doc/html/draft-ietf-ntp-port-randomization-06
, which has this section on the same kind of behavior, but for the NTP
port:

---- cut here ----
3.4. Effect on NAT devices

Some NAT devices will not translate the source port of a packet when
a privileged port number is employed. In networks where such NAT
devices are employed, use of the NTP well-known port for the client
port will essentially limit the number of hosts that may successfully
employ NTP client implementations.

In the case of NAT devices that will translate the source port even
when a privileged port is employed, packets reaching the external
realm of the NAT will not employ the NTP well-known port as the local
port, since the local port will normally be translated by the NAT
device possibly, but not necessarily, with a random port.
---- cut here ----

So I'm trying to find some reference that documents such behavior for
the NTP case....

Thanks!

Regards,
--
Fernando Gont
Director of Information Security
EdgeUno, Inc.
PGP Fingerprint: DFBD 63E3 B248 AE79 C598 AF23 EBAE DA03 0644 1531

Hi Fernando,

NTP sounds simple but it could be very complex when you dig deep down and/or get lost in details.
Here are 2 things to consider:

1. NTP clients can query NTP servers by using SRC UDP ports > 1024.
2. NTP servers cannot query/sync/communicate to another NTP server when using SRC UDP port > 1024.

In short, server to server wants SRC and DST UDP 123. The query and the response will be fully 123 when server to server.

In your case, it sounds like you want to achieve NTP server to NTP server, but you mention NTP clients behind NAT devices.

Because multiple clients behind the same NAT devices should work. Multiple NTP servers behind the same NAT and wanting to use other NTP server *should* not work.

Can you give us more details on what kind of communication you need here? From what I understand client to server should work just fine with any NAT devices.

Maybe you meant multiple NTP servers behind the same NAT to external NTP server?

Thanks
Jean

Hi, Jean,

On Thu, 2021-06-10 at 06:54 -0400, Jean St-Laurent via NANOG wrote:
> Hi Fernando,
>
> NTP sounds simple but it could be very complex when you dig deep down
> and/or get lost in details.
> Here are 2 things to consider:
>
> 1. NTP clients can query NTP servers by using SRC UDP ports > 1024.

This is indeed the case we're addressing. The NTP spec mandates srt
port=123, even for client-to-server cases.



> In your case, it sounds like you want to achieve NTP server to NTP
> server, but you mention NTP clients behind NAT devices.

Nope. We simply recommend to randomize the source port for client-to-
server cases.

So in the quoted section we make the case that requiring src port=123
clients doesnt really make sense:
1) if the NAT translates the port, the server won-t see src 123 anyway
2) if the NAT doesn't translate the port, you won't be able to ahve
multiple NTP clients behind the same firewall.



> Can you give us more details on what kind of communication you need
> here? From what I understand client to server should work just fine
> with any NAT devices.
>
> Maybe you meant multiple NTP servers behind the same NAT to external
> NTP servers

Please let me know if what I wrote above clarifies our intent.

Thanks!

Regards,
--
Fernando Gont
Director of Information Security
EdgeUno, Inc.
PGP Fingerprint: DFBD 63E3 B248 AE79 C598 AF23 EBAE DA03 0644 1531

Let's start with this example. When I click sync my clock in windows, this happened.

On the inside or Private side
08:15:07.434344 IP 192.168.254.205.123 > 13.86.101.172.123: NTPv3, Client, length 48
08:15:07.473681 IP 13.86.101.172.123 > 192.168.254.205.123: NTPv3, Server, length 48

You are indeed right that the client must use UDP port 123. Is the RFC saying must or should on the client SRC port? I'm not sure.

But, on the Public, this happened.
08:15:07.434381 IP 192.2XX.XXX.58291 > 13.86.101.172.123: NTPv3, Client, length 48
08:15:07.473656 IP 13.86.101.172.123 > 192.2XX.XXX.58291: NTPv3, Server, length 48

// Public ip obfuscated. I know, it indeed starts with 192.2. It's EBOX in Canada.

What we see on the public side, is that a network device did a NAT translation of the SRC UDP port to 58921. My clock synced perfectly.

So your goal is to find the devices that don't follow this behaviour, right?

Jean


-----Original Message-----
From: Fernando Gont <fernando.gont@edgeuno.com>
Sent: June 10, 2021 7:09 AM
To: jean@ddostest.me; nanog@nanog.org
Subject: Re: NAT devices not translating privileged ports

Hi, Jean,

On Thu, 2021-06-10 at 06:54 -0400, Jean St-Laurent via NANOG wrote:
> Hi Fernando,
>
> NTP sounds simple but it could be very complex when you dig deep down
> and/or get lost in details.
> Here are 2 things to consider:
>
> 1. NTP clients can query NTP servers by using SRC UDP ports > 1024.

This is indeed the case we're addressing. The NTP spec mandates srt port=123, even for client-to-server cases.

On 6/10/2021 4:04 AM, Fernando Gont wrote:
> Hi, Blake,
>
> Thanks a lot for your comments! In-line....
>
>
> On Fri, 2021-06-04 at 11:13 -0500, Blake Hudson wrote:
>> Current gen Cisco ASA firewalls have logic so that if the connection
>> from a private host originated from a privileged source port, the
>> NAT
>> translation to public IP also uses an unprivileged source port (not
>> necessarily the same source port though).
> Did you actaully mean "...also uses a *privileged port*"?
>
Yes I did. Thanks.

>> I found out that this behavior can cause issues when you have devices
>> on
>> your network that implement older DNS libraries or configs using UDP
>> 53
>> as a source and destination port for their DNS lookups. Occasionally
>> the
>> source port gets translated to one that ISC BIND servers have in a
>> blocklist (chargen, echo, time, and a few others) and the query is
>> ignored. As I recall, this behavior is hard coded so patching and
>> recompiling BIND is required to work around it.
>>
>> I forget what the older ASA behavior was. It may have been to leave
>> the
>> source port unchanged through the NAT process (I think this is what
>> you
>> mean by "not translated"). In that case the client doesn't implement
>> source port randomization and the NAT doesn't "upgrade" the
>> connection
>> to a random source port so I don't really see it as an issue.
> The issue would be that if the port is not translated, and multiple
> systems in the internal real of the NAT try to use the same privileged
> port (say, 123) simultaneously, things wouldn't work.
>
>

Not quite. If multiple devices behind a NAT use SRC=123 & DST = 123 for
connections, their connections will still work most of the time. First,
if the connections are to different destinations there would be no NAT
conflict. Second, if the connections occur at different times, there
would be no NAT conflict. Third, if there was a NAT conflict (meaning
connections using the same SRC port, DST port, and DST host at the same
time) the NAT device would see this and would either adjust the
translation to use a different SRC port (in which case the connection
succeeds) or may drop/reject the connection (in which case the client
would eventually retry).

So saying "things wouldn't work" or "you won't be able to have multiple
NTP clients behind the same firewall" is not true. Saying that "the
number of simultaneous connections is limited" when behind a NAT would
be more accurate. _But this statement is true of any one-to-many NAT_.
Most NTP clients utilize multiple NTP servers, often from a pool of
available servers, and initiate connections rather infrequently so I do
not expect this to be a problem in practice unless there are thousands
of NTP clients behind a single NAT accessing a common NTP server (and
that NAT does not do a good job of dealing with collisions).

I do, however, agree that clients should probably use ephemeral ports
when making any outbound connections as this provides more entropy for
NAT as well as for connection security. This extends to NTP.

Hi, Jean,

On Thu, 2021-06-10 at 08:23 -0400, Jean St-Laurent wrote:
> Let's start with this example. When I click sync my clock in windows,
> this happened.
>
> On the inside or Private side
> 08:15:07.434344 IP 192.168.254.205.123 > 13.86.101.172.123: NTPv3,
> Client, length 48
> 08:15:07.473681 IP 13.86.101.172.123 > 192.168.254.205.123: NTPv3,
> Server, length 48
>
> You are indeed right that the client must use UDP port 123. Is the
> RFC saying must or should on the client SRC port? I'm not sure.

Section 9.1 ("Peer Process Variables") of [RFC5905] SAYS:

dstport: UDP port number of the client, ordinarily the NTP port
number PORT (123) assigned by the IANA. This becomes the source
port number in packets sent from this association.


That said, as noted in
https://datatracker.ietf.org/doc/html/draft-ietf-ntp-port-randomization-06#section-5
, other client implementations don't bind the local port to 123, and
hence assign an ephemeral port.



> But, on the Public, this happened.
> 08:15:07.434381 IP 192.2XX.XXX.58291 > 13.86.101.172.123: NTPv3,
> Client, length 48
> 08:15:07.473656 IP 13.86.101.172.123 > 192.2XX.XXX.58291: NTPv3,
> Server, length 48
>
> // Public ip obfuscated. I know, it indeed starts with 192.2. It's
> EBOX in Canada.
>
> What we see on the public side, is that a network device did a NAT
> translation of the SRC UDP port to 58921. My clock synced perfectly.
>
> So your goal is to find the devices that don't follow this behaviour,
> right?

> No. The goal of our I-D is that NTP clients randomize their source
> port -- there's no need for clients to use port 123, and using that
> port on the client side has negative security implications.

Thanks,
--
Fernando Gont
Director of Information Security
EdgeUno, Inc.
PGP Fingerprint: DFBD 63E3 B248 AE79 C598 AF23 EBAE DA03 0644 1531