Mailing List Archive: Apple Catalina Appears to Introduce Massive Jitter

Apple Catalina Appears to Introduce Massive Jitter

mark.tinka at seacom

Oct 29, 2020, 5:07 AM

Post #1 of 56 (1594 views)

Hi all.

I've been on High Sierra for several years now due to a limitation with
an app that couldn't deal with Apple's latest rounds of system
permissions since Mojave. Eventually, I gave up on waiting for them to
fix it and upgraded my older Butterfly keyboard laptop to Catalina 4
weeks ago.

At the same time, I picked up the new Magic keyboard laptop 2 weeks ago
which came with Catalina.

Over the past week, I've been troubleshooting a massive jitter issue on
Catalina, just between itself and my home router. For control, I have a
Windows PC (tower-top) using a wireless adapter to connect to my home
network. That has no jitter at all.

I have noticed as much as 300ms+ jitter on Catalina.

I then asked a few friends around the world to run tests for me on their
own Catalina installations to their local router over wi-fi, and the
results are the same. Jitter so high that what should be a 1ms - 5ms
latency can (for a short period) jump to 200ms+, 300ms+, 400ms+.

On the off-chance that it is an issue with the new wireless chips on the
later MacBook models, one of my friends tested the same on a 2013
MacBook Pro running a beta version of Big Sur. Same story!

Another friend in South East Asia, testing on a 2018 13-inch MacBook Pro
running Catalina, also had the same issue.

A Google search suggests that this is some known issue since Mojave, to
do with Location Services, and some other apps, in a non-deterministic way:

https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router

For me, even after disabling all or some Location Services features, the
problem remains.

Is anyone else seeing this on their Catalina Mac's while on wi-fi? If
so, does anyone know what's going on here?

Ideally, this wouldn't matter if it was just a cosmetic issue - but I do
actually see physical impact to performance of network access to/from
the laptop, which has all the hallmarks of high jitter and/or packet loss.

An app like Zoom, which can display network performance data for a
session in real-time, does indicate nominal packet loss for audio and
video on this device, while other devices on the same WLAN are happy.

Thoughts?

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mattlists at rivervalleyinternet

Oct 29, 2020, 5:27 AM

Post #2 of 56 (1594 views)

Is it actually jitter or is it potentially the wireless network card going into sleep mode? I have seen that type of behavior on Apple products when the cards go into low power mode although I can’t say I have noticed that on my laptop.

> On Oct 29, 2020, at 8:11 AM, Mark Tinka <mark.tinka@seacom.com> wrote:
>
> ? Hi all.
>
> I've been on High Sierra for several years now due to a limitation with an app that couldn't deal with Apple's latest rounds of system permissions since Mojave. Eventually, I gave up on waiting for them to fix it and upgraded my older Butterfly keyboard laptop to Catalina 4 weeks ago.
>
> At the same time, I picked up the new Magic keyboard laptop 2 weeks ago which came with Catalina.
>
> Over the past week, I've been troubleshooting a massive jitter issue on Catalina, just between itself and my home router. For control, I have a Windows PC (tower-top) using a wireless adapter to connect to my home network. That has no jitter at all.
>
> I have noticed as much as 300ms+ jitter on Catalina.
>
> I then asked a few friends around the world to run tests for me on their own Catalina installations to their local router over wi-fi, and the results are the same. Jitter so high that what should be a 1ms - 5ms latency can (for a short period) jump to 200ms+, 300ms+, 400ms+.
>
> On the off-chance that it is an issue with the new wireless chips on the later MacBook models, one of my friends tested the same on a 2013 MacBook Pro running a beta version of Big Sur. Same story!
>
> Another friend in South East Asia, testing on a 2018 13-inch MacBook Pro running Catalina, also had the same issue.
>
> A Google search suggests that this is some known issue since Mojave, to do with Location Services, and some other apps, in a non-deterministic way:
>
> https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router
>
> For me, even after disabling all or some Location Services features, the problem remains.
>
> Is anyone else seeing this on their Catalina Mac's while on wi-fi? If so, does anyone know what's going on here?
>
> Ideally, this wouldn't matter if it was just a cosmetic issue - but I do actually see physical impact to performance of network access to/from the laptop, which has all the hallmarks of high jitter and/or packet loss.
>
> An app like Zoom, which can display network performance data for a session in real-time, does indicate nominal packet loss for audio and video on this device, while other devices on the same WLAN are happy.
>
> Thoughts?
>
> Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mark.tinka at seacom

Oct 29, 2020, 5:28 AM

Post #3 of 56 (1594 views)

Ah yes, an example of what I am seeing:

Marks-MacBook-Pro.local (172.16.0.239) 2020-10-29T14:28:27+0200
Keys: Help   Display mode   Restart statistics   Order of fields   quit
Packets               Pings
Host Loss%   Snt   Last   Avg Best Wrst StDev
1. 172.16.0.254 0.8%   126    3.9 34.7   2.5 232.1 54.9

Mark.

On 10/29/20 14:07, Mark Tinka wrote:
> Hi all.
>
> I've been on High Sierra for several years now due to a limitation
> with an app that couldn't deal with Apple's latest rounds of system
> permissions since Mojave. Eventually, I gave up on waiting for them to
> fix it and upgraded my older Butterfly keyboard laptop to Catalina 4
> weeks ago.
>
> At the same time, I picked up the new Magic keyboard laptop 2 weeks
> ago which came with Catalina.
>
> Over the past week, I've been troubleshooting a massive jitter issue
> on Catalina, just between itself and my home router. For control, I
> have a Windows PC (tower-top) using a wireless adapter to connect to
> my home network. That has no jitter at all.
>
> I have noticed as much as 300ms+ jitter on Catalina.
>
> I then asked a few friends around the world to run tests for me on
> their own Catalina installations to their local router over wi-fi, and
> the results are the same. Jitter so high that what should be a 1ms -
> 5ms latency can (for a short period) jump to 200ms+, 300ms+, 400ms+.
>
> On the off-chance that it is an issue with the new wireless chips on
> the later MacBook models, one of my friends tested the same on a 2013
> MacBook Pro running a beta version of Big Sur. Same story!
>
> Another friend in South East Asia, testing on a 2018 13-inch MacBook
> Pro running Catalina, also had the same issue.
>
> A Google search suggests that this is some known issue since Mojave,
> to do with Location Services, and some other apps, in a
> non-deterministic way:
>
> https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router
>
> For me, even after disabling all or some Location Services features,
> the problem remains.
>
> Is anyone else seeing this on their Catalina Mac's while on wi-fi? If
> so, does anyone know what's going on here?
>
> Ideally, this wouldn't matter if it was just a cosmetic issue - but I
> do actually see physical impact to performance of network access
> to/from the laptop, which has all the hallmarks of high jitter and/or
> packet loss.
>
> An app like Zoom, which can display network performance data for a
> session in real-time, does indicate nominal packet loss for audio and
> video on this device, while other devices on the same WLAN are happy.
>
> Thoughts?
>
> Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mark.tinka at seacom

Oct 29, 2020, 5:31 AM

Post #4 of 56 (1594 views)

On 10/29/20 14:27, Matt Hoppes wrote:

> Is it actually jitter or is it potentially the wireless network card
> going into sleep mode? I have seen that type of behavior on Apple
> products when the cards go into low power mode although I can’t say I
> have noticed that on my laptop.

Not, not sleep mode, because I am actively using the device to move data
to/from the Internet.

I was actually struggling to upload some files to Youtube last weekend,
and had to use another computer to do it as I couldn't figure out what
was going on. That it is how bad the jitter is.

It seems to be a much bigger problem for the upload direction than the
download, but it, inevitably creates a symmetrical performance problem.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

nanog at ics-il

Oct 29, 2020, 5:37 AM

Post #5 of 56 (1594 views)

Have you ruled out local wireless issues, such as a literal side-by-side test?

Does the problem still exist when wired?

-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Mark Tinka" <mark.tinka@seacom.com>
To: "Matt Hoppes" <mattlists@rivervalleyinternet.net>
Cc: "NANOG" <nanog@nanog.org>
Sent: Thursday, October 29, 2020 7:31:59 AM
Subject: Re: Apple Catalina Appears to Introduce Massive Jitter

On 10/29/20 14:27, Matt Hoppes wrote:

> Is it actually jitter or is it potentially the wireless network card
> going into sleep mode? I have seen that type of behavior on Apple
> products when the cards go into low power mode although I can’t say I
> have noticed that on my laptop.

Not, not sleep mode, because I am actively using the device to move data
to/from the Internet.

I was actually struggling to upload some files to Youtube last weekend,
and had to use another computer to do it as I couldn't figure out what
was going on. That it is how bad the jitter is.

It seems to be a much bigger problem for the upload direction than the
download, but it, inevitably creates a symmetrical performance problem.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

Oct 29, 2020, 5:40 AM

Post #6 of 56 (1594 views)

On Thu, Oct 29, 2020 at 02:31:59PM +0200, Mark Tinka wrote:
>
>
> On 10/29/20 14:27, Matt Hoppes wrote:
>
> > Is it actually jitter or is it potentially the wireless network card
> > going into sleep mode? I have seen that type of behavior on Apple
> > products when the cards go into low power mode although I can’t say I
> > have noticed that on my laptop.
>
> Not, not sleep mode, because I am actively using the device to move data
> to/from the Internet.
>
> I was actually struggling to upload some files to Youtube last weekend, and
> had to use another computer to do it as I couldn't figure out what was going
> on. That it is how bad the jitter is.
>
> It seems to be a much bigger problem for the upload direction than the
> download, but it, inevitably creates a symmetrical performance problem.

I know there was a recent fix Apple did for devices talking to UBNT APs
for their handsets, perhaps there's a similar fix needed on your side?

I have all UBNT at home for wireless and periodically have some random
issues which I can't explain, but for the most part have things tuned to ensure
there's little to no interference.

Do you see the same when hardwired? I keep many of my devices hardwired
to avoid odd jitter issues. I also saw some older versions of the Pulse Secure
VPN add the behavior you describe, including the more uptime the slower it would
get.

- Jared

--
Jared Mauch | pgp key available via finger from jared@puck.nether.net
clue++; | http://puck.nether.net/~jared/ My statements are only mine.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mark.tinka at seacom

Oct 29, 2020, 5:48 AM

Post #7 of 56 (1594 views)

On 10/29/20 14:37, Mike Hammett wrote:

> Have you ruled out local wireless issues, such as a literal
> side-by-side test?

Yes - all other wi-fi devices don't exhibit this issue, including my
wireless-connected PC. Only this MacBook running Catalina.

The problem exists at all wi-fi AP's around the house, all of which are
easily 15m apart to cover the whole house.

> Does the problem still exist when wired?

No problem on wire. As you can see below, my worst record latency value
is 11.8ms only:

Marks-MacBook-Pro.local (172.16.0.239) 2020-10-29T14:48:12+0200
Keys: Help   Display mode   Restart statistics   Order of fields quit
Packets               Pings
Host Loss%   Snt   Last   Avg Best Wrst StDev
1. 172.16.0.254 0.0%    50    1.0   2.6   0.7 11.8   2.1

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mark.tinka at seacom

Oct 29, 2020, 5:55 AM

Post #8 of 56 (1594 views)

On 10/29/20 14:40, Jared Mauch wrote:

> I know there was a recent fix Apple did for devices talking to UBNT APs
> for their handsets, perhaps there's a similar fix needed on your side?
>
> I have all UBNT at home for wireless and periodically have some random
> issues which I can't explain, but for the most part have things tuned to ensure
> there's little to no interference.

I am running TP-Link AP's. Two of them are Google OnHub, which was built
by TP-Link, and the 3rd one is the TP-Link Archer C6.

So they all support 802.11ac, which is where my device spends most of
its time (5GHz).

No interference from my neighbors (separated by thick walls), and I am
running separate channels for both frequencies per device.

Also, no other wireless device is suffering like this.

> Do you see the same when hardwired? I keep many of my devices hardwired
> to avoid odd jitter issues.

No issues on the wire at all. Quite perfect.

Like you, I hard-wire all fixed devices (TV's, a/v receivers, satellite
decoders, gaming consoles, energy meters, e.t.c.). The only devices on
wireless are mobile devices, tablets, laptops and the Windows PC which
is hooked up via wi-fi as well.

> I also saw some older versions of the Pulse Secure
> VPN add the behavior you describe, including the more uptime the slower it would
> get.

I use Viscosity as an SSL/VPN client. The issue is the same whether it
is enabled or offline.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

Oct 29, 2020, 6:04 AM

Post #9 of 56 (1591 views)

Might be worth disabling each AP to see if there's one out there having an issue playing nice with the MacBook. Also try different combinations of two APs working together. It's possible the MacBook is flip flopping because the power levels are fighting each other.

Does the Mac have this issue at your local coffee shop or another establishment with Wi-Fi? You can try to rule out the AirPort card in the Mac itself.

Sent from ProtonMail mobile

-------- Original Message --------
On Oct 29, 2020, 7:55 AM, Mark Tinka wrote:

> On 10/29/20 14:40, Jared Mauch wrote:
>
>> I know there was a recent fix Apple did for devices talking to UBNT APs
>> for their handsets, perhaps there's a similar fix needed on your side?
>>
>> I have all UBNT at home for wireless and periodically have some random
>> issues which I can't explain, but for the most part have things tuned to ensure
>> there's little to no interference.
>
> I am running TP-Link AP's. Two of them are Google OnHub, which was built
> by TP-Link, and the 3rd one is the TP-Link Archer C6.
>
> So they all support 802.11ac, which is where my device spends most of
> its time (5GHz).
>
> No interference from my neighbors (separated by thick walls), and I am
> running separate channels for both frequencies per device.
>
> Also, no other wireless device is suffering like this.
>
>> Do you see the same when hardwired? I keep many of my devices hardwired
>> to avoid odd jitter issues.
>
> No issues on the wire at all. Quite perfect.
>
> Like you, I hard-wire all fixed devices (TV's, a/v receivers, satellite
> decoders, gaming consoles, energy meters, e.t.c.). The only devices on
> wireless are mobile devices, tablets, laptops and the Windows PC which
> is hooked up via wi-fi as well.
>
>> I also saw some older versions of the Pulse Secure
>> VPN add the behavior you describe, including the more uptime the slower it would
>> get.
>
> I use Viscosity as an SSL/VPN client. The issue is the same whether it
> is enabled or offline.
>
> Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mark.tinka at seacom

Oct 29, 2020, 6:23 AM

Post #10 of 56 (1594 views)

On 10/29/20 15:04, Cory Sell wrote:

> Might be worth disabling each AP to see if there's one out there
> having an issue playing nice with the MacBook. Also try different
> combinations of two APs working together. It's possible the MacBook is
> flip flopping because the power levels are fighting each other.

Tested all that, as well as dropping Tx power levels on each of the AP's
to Low so that there isn't any power coming from any other AP (despite
being quite far, already).

And to confirm, when the laptop locks into an AP, it doesn't try to join
another one. When in range, power is very good (between -37dB and
-52dB). When I walk away, that AP becomes too far (as bad as -80dB), but
the next one close by is far better (same good values as before) and
laptop connects and sticks to that.

Again, only impacts Catalina. No other Apple device, or the Windows PC
that is on the same WLAN.

> Does the Mac have this issue at your local coffee shop or another
> establishment with Wi-Fi? You can try to rule out the AirPort card in
> the Mac itself.

Never tried, I generally work from home. If I'm out, it's faster to
tether to my 4G service rather than any public wi-fi.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

Oct 29, 2020, 7:08 AM

Post #11 of 56 (1594 views)

I believe I have seen the same thing with a Mid 2015 11,4 running catalina. Not diagnosing further because I could not find a reason for it fast enough and not sure if it really had an impact at the moment…. but could you try the following

sudo sysctl net.link.generic.system.hwcksum_tx=0
sudo sysctl net.link.generic.system.hwcksum_rx=0
sudo ifconfig en0 -rxcsum

in reverse … to restore the settings

sudo sysctl net.link.generic.system.hwcksum_tx=1
sudo sysctl net.link.generic.system.hwcksum_rx=1
sudo ifconfig en0 rxcsum

If you have some specific tests to run I would be willing to run them here on Big Sur with the same laptop but I have nothing now that runs Catalina

Wireshark used to in Catalina rack up cksum errors a lot while these were all at their defaults.

> On Oct 29, 2020, at 08:23, Mark Tinka <mark.tinka@seacom.com> wrote:
>
>
>
> On 10/29/20 15:04, Cory Sell wrote:
>
>> Might be worth disabling each AP to see if there's one out there having an issue playing nice with the MacBook. Also try different combinations of two APs working together. It's possible the MacBook is flip flopping because the power levels are fighting each other.
>
> Tested all that, as well as dropping Tx power levels on each of the AP's to Low so that there isn't any power coming from any other AP (despite being quite far, already).
>
> And to confirm, when the laptop locks into an AP, it doesn't try to join another one. When in range, power is very good (between -37dB and -52dB). When I walk away, that AP becomes too far (as bad as -80dB), but the next one close by is far better (same good values as before) and laptop connects and sticks to that.
>
> Again, only impacts Catalina. No other Apple device, or the Windows PC that is on the same WLAN.
>
>
>> Does the Mac have this issue at your local coffee shop or another establishment with Wi-Fi? You can try to rule out the AirPort card in the Mac itself.
>
> Never tried, I generally work from home. If I'm out, it's faster to tether to my 4G service rather than any public wi-fi.
>
> Mark.

--

J. Hellenthal

The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

Oct 29, 2020, 7:43 AM

Post #12 of 56 (1594 views)

Should also state here that net.inet.icmp.icmplim=0 and the command I have been testing from is: (ping -c 5000 -i 0.1 router)

--- router ping statistics ---
5000 packets transmitted, 5000 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.118/4.060/172.031/6.841 ms

> On Oct 29, 2020, at 09:08, J. Hellenthal <jhellenthal@dataix.net> wrote:
>
> I believe I have seen the same thing with a Mid 2015 11,4 running catalina. Not diagnosing further because I could not find a reason for it fast enough and not sure if it really had an impact at the moment…. but could you try the following
>
>
> sudo sysctl net.link.generic.system.hwcksum_tx=0
> sudo sysctl net.link.generic.system.hwcksum_rx=0
> sudo ifconfig en0 -rxcsum
>
>
> in reverse … to restore the settings
>
> sudo sysctl net.link.generic.system.hwcksum_tx=1
> sudo sysctl net.link.generic.system.hwcksum_rx=1
> sudo ifconfig en0 rxcsum
>
>
> If you have some specific tests to run I would be willing to run them here on Big Sur with the same laptop but I have nothing now that runs Catalina
>
>
> Wireshark used to in Catalina rack up cksum errors a lot while these were all at their defaults.
>
>
>
>> On Oct 29, 2020, at 08:23, Mark Tinka <mark.tinka@seacom.com> wrote:
>>
>>
>>
>> On 10/29/20 15:04, Cory Sell wrote:
>>
>>> Might be worth disabling each AP to see if there's one out there having an issue playing nice with the MacBook. Also try different combinations of two APs working together. It's possible the MacBook is flip flopping because the power levels are fighting each other.
>>
>> Tested all that, as well as dropping Tx power levels on each of the AP's to Low so that there isn't any power coming from any other AP (despite being quite far, already).
>>
>> And to confirm, when the laptop locks into an AP, it doesn't try to join another one. When in range, power is very good (between -37dB and -52dB). When I walk away, that AP becomes too far (as bad as -80dB), but the next one close by is far better (same good values as before) and laptop connects and sticks to that.
>>
>> Again, only impacts Catalina. No other Apple device, or the Windows PC that is on the same WLAN.
>>
>>
>>> Does the Mac have this issue at your local coffee shop or another establishment with Wi-Fi? You can try to rule out the AirPort card in the Mac itself.
>>
>> Never tried, I generally work from home. If I'm out, it's faster to tether to my 4G service rather than any public wi-fi.
>>
>> Mark.
>
>
> --
>
> J. Hellenthal
>
> The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume.
>
>
>
>
>
>

--

J. Hellenthal

The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

niels=nanog at bakker

Oct 29, 2020, 7:44 AM

Post #13 of 56 (1594 views)

* nanog@nanog.org (J. Hellenthal via NANOG) [Thu 29 Oct 2020, 15:10 CET]:
[disabling checksum offload]
>Wireshark used to in Catalina rack up cksum errors a lot while these were all at their defaults.

This is completely expected behaviour for outgoing packets.

-- Niels.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mark.tinka at seacom

Oct 29, 2020, 8:50 AM

Post #14 of 56 (1594 views)

On 10/29/20 16:08, J. Hellenthal wrote:

> I believe I have seen the same thing with a Mid 2015 11,4 running catalina. Not diagnosing further because I could not find a reason for it fast enough and not sure if it really had an impact at the moment…. but could you try the following
>
>
> sudo sysctl net.link.generic.system.hwcksum_tx=0
> sudo sysctl net.link.generic.system.hwcksum_rx=0
> sudo ifconfig en0 -rxcsum

Thanks, I'll have a sniff.

> If you have some specific tests to run I would be willing to run them here on Big Sur with the same laptop but I have nothing now that runs Catalina

One of my mates found the same issue on Big Sur (beta) on a 2013 MacBook
Pro.

Just a simple mtr test to your local home router's IP address should,
over wi-fi, should show you the jitter.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

Oct 29, 2020, 9:05 AM

Post #15 of 56 (1594 views)

On the latest Catalina 10.15.7 from a MacBook Air (early 2014) via WiFi
to Google Wifi Mesh router (only a single unit network):

Over 2.4Ghz through 3 interior walls:
--- 192.168.86.1 ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.440/15.078/213.538/30.541 ms

Over 5Ghz through 1 door:
--- 192.168.86.1 ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.557/24.614/137.233/39.015 ms

This is a real world test with several iPhones, 2.4GHz WiFi only
cameras, and an iPad being used for zoom (remote school) while running
these tests. If you were not aware, on a Mac you can hold the Option key
while clicking on the WiFi icon to see more options as well as more
information about your current WiFi connection, including the realtime
Tx rate, channel, and when your Mac is searching for new networks (mine
does this every few seconds, it seems).

From an experience perspective, all applications seem to work fine for
us. Including uploading to YouTube (a regular event as my spouse is a
teacher) and Zoom/Teams/FaceTime/your teleconference app of choice (used
by all of us).

--B

On 10/29/2020 7:28 AM, Mark Tinka wrote:
> Ah yes, an example of what I am seeing:
>
>
> Marks-MacBook-Pro.local (172.16.0.239) 2020-10-29T14:28:27+0200
> Keys: Help   Display mode   Restart statistics   Order of fields   quit
> Packets               Pings
> Host Loss%   Snt   Last   Avg Best Wrst StDev
> 1. 172.16.0.254 0.8%   126    3.9 34.7   2.5 232.1 54.9
>
> Mark.
>
> On 10/29/20 14:07, Mark Tinka wrote:
>> Hi all.
>>
>> I've been on High Sierra for several years now due to a limitation
>> with an app that couldn't deal with Apple's latest rounds of system
>> permissions since Mojave. Eventually, I gave up on waiting for them
>> to fix it and upgraded my older Butterfly keyboard laptop to Catalina
>> 4 weeks ago.
>>
>> At the same time, I picked up the new Magic keyboard laptop 2 weeks
>> ago which came with Catalina.
>>
>> Over the past week, I've been troubleshooting a massive jitter issue
>> on Catalina, just between itself and my home router. For control, I
>> have a Windows PC (tower-top) using a wireless adapter to connect to
>> my home network. That has no jitter at all.
>>
>> I have noticed as much as 300ms+ jitter on Catalina.
>>
>> I then asked a few friends around the world to run tests for me on
>> their own Catalina installations to their local router over wi-fi,
>> and the results are the same. Jitter so high that what should be a
>> 1ms - 5ms latency can (for a short period) jump to 200ms+, 300ms+,
>> 400ms+.
>>
>> On the off-chance that it is an issue with the new wireless chips on
>> the later MacBook models, one of my friends tested the same on a 2013
>> MacBook Pro running a beta version of Big Sur. Same story!
>>
>> Another friend in South East Asia, testing on a 2018 13-inch MacBook
>> Pro running Catalina, also had the same issue.
>>
>> A Google search suggests that this is some known issue since Mojave,
>> to do with Location Services, and some other apps, in a
>> non-deterministic way:
>>
>> https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router
>>
>> For me, even after disabling all or some Location Services features,
>> the problem remains.
>>
>> Is anyone else seeing this on their Catalina Mac's while on wi-fi? If
>> so, does anyone know what's going on here?
>>
>> Ideally, this wouldn't matter if it was just a cosmetic issue - but I
>> do actually see physical impact to performance of network access
>> to/from the laptop, which has all the hallmarks of high jitter and/or
>> packet loss.
>>
>> An app like Zoom, which can display network performance data for a
>> session in real-time, does indicate nominal packet loss for audio and
>> video on this device, while other devices on the same WLAN are happy.
>>
>> Thoughts?
>>
>> Mark.
>

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mark.tinka at seacom

Oct 29, 2020, 9:39 AM

Post #16 of 56 (1594 views)

On 10/29/20 18:05, Blake Hudson wrote:

> On the latest Catalina 10.15.7 from a MacBook Air (early 2014) via
> WiFi to Google Wifi Mesh router (only a single unit network):
>
> Over 2.4Ghz through 3 interior walls:
> --- 192.168.86.1 ping statistics ---
> 100 packets transmitted, 100 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 1.440/15.078/213.538/30.541 ms
>
> Over 5Ghz through 1 door:
> --- 192.168.86.1 ping statistics ---
> 100 packets transmitted, 100 packets received, 0.0% packet loss
> round-trip min/avg/max/stddev = 1.557/24.614/137.233/39.015 ms

This is what I'm talking about. Why would you have such massive jitter
on wi-fi to your local router?

You're hitting 213ms on 2.4GHz and 137ms on 5GHz. This doesn't seem odd
to you?

Never had this issue pre-Catalina (I understand it may have been
introduced in Mojave, but I came from High Sierra).

> This is a real world test with several iPhones, 2.4GHz WiFi only
> cameras, and an iPad being used for zoom (remote school) while running
> these tests.

Same for me. There are tons of devices on my WLAN, and only the one
running Catalina behaves this way.

No drama for the rest.

> If you were not aware, on a Mac you can hold the Option key while
> clicking on the WiFi icon to see more options as well as more
> information about your current WiFi connection, including the realtime
> Tx rate, channel, and when your Mac is searching for new networks
> (mine does this every few seconds, it seems).

Yes, I am aware about the on-board macOS wi-fi diagnostics, and the good
ol' trusted "airport -s". But those aren't enough.

I ended up spending money on NetSpot to see what's going on. Nothing
there either. All I can tell is that Catalina is the problem; how
exactly, is not yet very clear.

> From an experience perspective, all applications seem to work fine for
> us. Including uploading to YouTube (a regular event as my spouse is a
> teacher) and Zoom/Teams/FaceTime/your teleconference app of choice
> (used by all of us).

So even though apps like Zoom report mild packet loss while on wi-fi
(0.1% - 0.10%), it's not a train smash.

In general, everything works fine. But it doesn't feel 100%.

For the Youtube upload, I've found it to be temperamental, but I'm
making it work for the time being.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

colinj at gt86car

Oct 29, 2020, 10:12 AM

Post #17 of 56 (1594 views)

Hey Mark,
Good shout with debug, same issue seen on MacBook Air with Catalina 10.15.6 beta, pings upto 150ms seen
iMac with Sierra zero jitter and usually sub 1m pings
Now need to find out why, I never noticed as wife using the MacBook Air :(
I cant yet update to big sur since need lots of sad space, need to cutdown on university docs me thinks

Col

> On 29 Oct 2020, at 12:07, Mark Tinka <mark.tinka@seacom.com> wrote:
>
> Hi all.
>
> I've been on High Sierra for several years now due to a limitation with an app that couldn't deal with Apple's latest rounds of system permissions since Mojave. Eventually, I gave up on waiting for them to fix it and upgraded my older Butterfly keyboard laptop to Catalina 4 weeks ago.
>
> At the same time, I picked up the new Magic keyboard laptop 2 weeks ago which came with Catalina.
>
> Over the past week, I've been troubleshooting a massive jitter issue on Catalina, just between itself and my home router. For control, I have a Windows PC (tower-top) using a wireless adapter to connect to my home network. That has no jitter at all.
>
> I have noticed as much as 300ms+ jitter on Catalina.
>
> I then asked a few friends around the world to run tests for me on their own Catalina installations to their local router over wi-fi, and the results are the same. Jitter so high that what should be a 1ms - 5ms latency can (for a short period) jump to 200ms+, 300ms+, 400ms+.
>
> On the off-chance that it is an issue with the new wireless chips on the later MacBook models, one of my friends tested the same on a 2013 MacBook Pro running a beta version of Big Sur. Same story!
>
> Another friend in South East Asia, testing on a 2018 13-inch MacBook Pro running Catalina, also had the same issue.
>
> A Google search suggests that this is some known issue since Mojave, to do with Location Services, and some other apps, in a non-deterministic way:
>
> https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router <https://apple.stackexchange.com/questions/263638/macbook-pro-experiencing-ping-spikes-to-local-router>
>
> For me, even after disabling all or some Location Services features, the problem remains.
>
> Is anyone else seeing this on their Catalina Mac's while on wi-fi? If so, does anyone know what's going on here?
>
> Ideally, this wouldn't matter if it was just a cosmetic issue - but I do actually see physical impact to performance of network access to/from the laptop, which has all the hallmarks of high jitter and/or packet loss.
>
> An app like Zoom, which can display network performance data for a session in real-time, does indicate nominal packet loss for audio and video on this device, while other devices on the same WLAN are happy.
>
> Thoughts?
>
> Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mark.tinka at seacom

Oct 29, 2020, 10:17 AM

Post #18 of 56 (1594 views)

On 10/29/20 19:12, colin johnston wrote:

> Hey Mark,
> Good shout with debug, same issue seen on MacBook Air with Catalina
> 10.15.6 beta, pings upto 150ms seen
> iMac with Sierra zero jitter and usually sub 1m pings
> Now need to find out why, I never noticed as wife using the MacBook Air :(
> I cant yet update to big sur since need lots of sad space, need to
> cutdown on university docs me thinks

Thanks, Col.

So confirmed that Big Sur has the same problem, both on newer and older
Mac's.

Issue seems to have surfaced somewhere around Mojave.

Most folk from various fora suggested Location Services were to blame. I
turned all of mine off, no joy.

A friend of mine in Malaysia noticed an improvement when he disabled
Handoff between the Mac and all his other iCloud-enabled devices. But
that didn't work for me.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

colinj at gt86car

Oct 29, 2020, 10:24 AM

Post #19 of 56 (1594 views)

This does seem to be solved with the checksum disable below, or at least pings down to sub 10ms on Mac book air with Catalina beta 10.15.6, why aim performs far better I don’t know. I tried to introduce load after cksum disable and it did not see ping spikes as before

How do we now explain to Apple to fix ?

Col

> On 29 Oct 2020, at 14:08, J. Hellenthal via NANOG <nanog@nanog.org> wrote:
>
> I believe I have seen the same thing with a Mid 2015 11,4 running catalina. Not diagnosing further because I could not find a reason for it fast enough and not sure if it really had an impact at the moment…. but could you try the following
>
>
> sudo sysctl net.link.generic.system.hwcksum_tx=0
> sudo sysctl net.link.generic.system.hwcksum_rx=0
> sudo ifconfig en0 -rxcsum
>
>
> in reverse … to restore the settings
>
> sudo sysctl net.link.generic.system.hwcksum_tx=1
> sudo sysctl net.link.generic.system.hwcksum_rx=1
> sudo ifconfig en0 rxcsum
>
>
> If you have some specific tests to run I would be willing to run them here on Big Sur with the same laptop but I have nothing now that runs Catalina
>
>
> Wireshark used to in Catalina rack up cksum errors a lot while these were all at their defaults.
>
>
>
>> On Oct 29, 2020, at 08:23, Mark Tinka <mark.tinka@seacom.com> wrote:
>>
>>
>>
>> On 10/29/20 15:04, Cory Sell wrote:
>>
>>> Might be worth disabling each AP to see if there's one out there having an issue playing nice with the MacBook. Also try different combinations of two APs working together. It's possible the MacBook is flip flopping because the power levels are fighting each other.
>>
>> Tested all that, as well as dropping Tx power levels on each of the AP's to Low so that there isn't any power coming from any other AP (despite being quite far, already).
>>
>> And to confirm, when the laptop locks into an AP, it doesn't try to join another one. When in range, power is very good (between -37dB and -52dB). When I walk away, that AP becomes too far (as bad as -80dB), but the next one close by is far better (same good values as before) and laptop connects and sticks to that.
>>
>> Again, only impacts Catalina. No other Apple device, or the Windows PC that is on the same WLAN.
>>
>>
>>> Does the Mac have this issue at your local coffee shop or another establishment with Wi-Fi? You can try to rule out the AirPort card in the Mac itself.
>>
>> Never tried, I generally work from home. If I'm out, it's faster to tether to my 4G service rather than any public wi-fi.
>>
>> Mark.
>
>
> --
>
> J. Hellenthal
>
> The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume.
>
>
>
>
>
>

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

Oct 29, 2020, 10:27 AM

Post #20 of 56 (1594 views)

> Most folk from various fora suggested Location Services were to
> blame. I turned all of mine off, no joy.

you only *think* you turned off location services. as they are a vital
component of providing a good user experience ...

:(

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mark.tinka at seacom

Oct 29, 2020, 10:35 AM

Post #21 of 56 (1594 views)

On 10/29/20 19:27, Randy Bush wrote:

> you only *think* you turned off location services. as they are a vital
> component of providing a good user experience ...
>
> :(

That was an honest, lingering thought.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

colinj at gt86car

Oct 29, 2020, 10:38 AM

Post #22 of 56 (1594 views)

Be careful using Apple wireless diagnostic package, uses a lot of /var/tmp space on a small Macbook air 128ssd

Col

> On 29 Oct 2020, at 17:24, colin johnston <colinj@gt86car.org.uk> wrote:
>
> This does seem to be solved with the checksum disable below, or at least pings down to sub 10ms on Mac book air with Catalina beta 10.15.6, why aim performs far better I don’t know. I tried to introduce load after cksum disable and it did not see ping spikes as before
>
> How do we now explain to Apple to fix ?
>
> Col
>
>
>> On 29 Oct 2020, at 14:08, J. Hellenthal via NANOG <nanog@nanog.org> wrote:
>>
>> I believe I have seen the same thing with a Mid 2015 11,4 running catalina. Not diagnosing further because I could not find a reason for it fast enough and not sure if it really had an impact at the moment…. but could you try the following
>>
>>
>> sudo sysctl net.link.generic.system.hwcksum_tx=0
>> sudo sysctl net.link.generic.system.hwcksum_rx=0
>> sudo ifconfig en0 -rxcsum
>>
>>
>> in reverse … to restore the settings
>>
>> sudo sysctl net.link.generic.system.hwcksum_tx=1
>> sudo sysctl net.link.generic.system.hwcksum_rx=1
>> sudo ifconfig en0 rxcsum
>>
>>
>> If you have some specific tests to run I would be willing to run them here on Big Sur with the same laptop but I have nothing now that runs Catalina
>>
>>
>> Wireshark used to in Catalina rack up cksum errors a lot while these were all at their defaults.
>>
>>
>>
>>> On Oct 29, 2020, at 08:23, Mark Tinka <mark.tinka@seacom.com> wrote:
>>>
>>>
>>>
>>> On 10/29/20 15:04, Cory Sell wrote:
>>>
>>>> Might be worth disabling each AP to see if there's one out there having an issue playing nice with the MacBook. Also try different combinations of two APs working together. It's possible the MacBook is flip flopping because the power levels are fighting each other.
>>>
>>> Tested all that, as well as dropping Tx power levels on each of the AP's to Low so that there isn't any power coming from any other AP (despite being quite far, already).
>>>
>>> And to confirm, when the laptop locks into an AP, it doesn't try to join another one. When in range, power is very good (between -37dB and -52dB). When I walk away, that AP becomes too far (as bad as -80dB), but the next one close by is far better (same good values as before) and laptop connects and sticks to that.
>>>
>>> Again, only impacts Catalina. No other Apple device, or the Windows PC that is on the same WLAN.
>>>
>>>
>>>> Does the Mac have this issue at your local coffee shop or another establishment with Wi-Fi? You can try to rule out the AirPort card in the Mac itself.
>>>
>>> Never tried, I generally work from home. If I'm out, it's faster to tether to my 4G service rather than any public wi-fi.
>>>
>>> Mark.
>>
>>
>> --
>>
>> J. Hellenthal
>>
>> The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume.
>>
>>
>>
>>
>>
>>
>

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mark.tinka at seacom

Oct 29, 2020, 10:40 AM

Post #23 of 56 (1594 views)

On 10/29/20 19:24, colin johnston wrote:
> This does seem to be solved with the checksum disable below, or at least pings down to sub 10ms on Mac book air with Catalina beta 10.15.6, why aim performs far better I don’t know. I tried to introduce load after cksum disable and it did not see ping spikes as before

Didn't work for me, sadly. I still had times ending up in the 235ms
range after making the changes.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

mark.tinka at seacom

Oct 29, 2020, 10:42 AM

Post #24 of 56 (1594 views)

On 10/29/20 19:38, colin johnston wrote:

> Be careful using Apple wireless diagnostic package, uses a lot of /var/tmp space on a small Macbook air 128ssd

Mine cost me 300MB.

Mark.

Re: Apple Catalina Appears to Introduce Massive Jitter [ In reply to ]

davec at curado

Oct 29, 2020, 11:14 AM

Post #25 of 56 (1591 views)

I was curious, so poked at this... my results from a macbook pro 2019
running Catalina 10.15.3

sudo /usr/local/sbin/mtr -r 10.200.200.200

Start: 2020-10-29T14:09:08-0400
HOST: bos-mp36c Loss% Snt Last Avg Best Wrst StDev
1.|-- 10.200.200.200 0.0% 10 11.9 63.7 9.0 340.2 104.1

sudo sysctl net.link.generic.system.hwcksum_tx=0
net.link.generic.system.hwcksum_tx: 1 -> 0
sudo sysctl net.link.generic.system.hwcksum_rx=0
net.link.generic.system.hwcksum_rx: 1 -> 0
sudo ifconfig en0 -rxcsum

sudo /usr/local/sbin/mtr -r 10.200.200.200

Start: 2020-10-29T14:09:43-0400
HOST: bos-mp36c Loss% Snt Last Avg Best Wrst StDev
1.|-- 10.200.200.200 0.0% 10 19.8 13.6 9.8 20.9 4.5

So, seems to make things better. =-)

On Thu, Oct 29, 2020 at 11:50 AM Mark Tinka <mark.tinka@seacom.com> wrote:

>
>
> On 10/29/20 16:08, J. Hellenthal wrote:
>
> > I believe I have seen the same thing with a Mid 2015 11,4 running
> catalina. Not diagnosing further because I could not find a reason for it
> fast enough and not sure if it really had an impact at the moment…. but
> could you try the following
> >
> >
> > sudo sysctl net.link.generic.system.hwcksum_tx=0
> > sudo sysctl net.link.generic.system.hwcksum_rx=0
> > sudo ifconfig en0 -rxcsum
>
> Thanks, I'll have a sniff.
>
> > If you have some specific tests to run I would be willing to run them
> here on Big Sur with the same laptop but I have nothing now that runs
> Catalina
>
> One of my mates found the same issue on Big Sur (beta) on a 2013 MacBook
> Pro.
>
> Just a simple mtr test to your local home router's IP address should,
> over wi-fi, should show you the jitter.
>
> Mark.
>