Mailing List Archive

[PATCH 0/5] ZSTD compression support for OpenSSH
I added ZSTD support to OpenSSH roughly over a year and I've been
playing with it ever since.
The nice part is that ZSTD achieves reasonable compression (like zlib)
but consumes little CPU so it is unlikely that compression becomes the
bottle neck of a transfer. The compression overhead (CPU) is negligible
even when uncompressed data is tunneled over the SSH connection (SOCKS
proxy, port forward).

Sebastian


_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: [PATCH 0/5] ZSTD compression support for OpenSSH [ In reply to ]
On Sun, 6 Sep 2020 at 01:52, Sebastian Andrzej Siewior
<openssh@ml.breakpoint.cc> wrote:
[...]

Patches 1-3 seem fine to apply as is. (I also have a working
directory with an m4 subdirectory as I was investigating using libtool
to build the sk-dummy shared library, and that needs extra macros).
@djm: any objections?

The zstd part would be a larger discussion because we would need to
either carry it as a Portable patch or have zstd added to OpenBSD
base, and I don't know if that would be accepted. Do you have any
performance numbers for zstd in this application?

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: [PATCH 0/5] ZSTD compression support for OpenSSH [ In reply to ]
On Mon, 7 Sep 2020, Darren Tucker wrote:

> On Sun, 6 Sep 2020 at 01:52, Sebastian Andrzej Siewior
> <openssh@ml.breakpoint.cc> wrote:
> [...]
>
> Patches 1-3 seem fine to apply as is. (I also have a working
> directory with an m4 subdirectory as I was investigating using libtool
> to build the sk-dummy shared library, and that needs extra macros).
> @djm: any objections?

Why do you need it? I'm not sure I want to invite that particular
vampire into the house without a good reason.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: [PATCH 0/5] ZSTD compression support for OpenSSH [ In reply to ]
On Mon, 7 Sep 2020 at 11:39, Damien Miller <djm@mindrot.org> wrote:
> On Mon, 7 Sep 2020, Darren Tucker wrote:
[...]
> > Patches 1-3 seem fine to apply as is. (I also have a working
> > directory with an m4 subdirectory as I was investigating using libtool
> > to build the sk-dummy shared library, and that needs extra macros).
> > @djm: any objections?
>
> Why do you need it? I'm not sure I want to invite that particular
> vampire into the house without a good reason.

The soliciting for objections were only for the patches 1-3 in this
thread, not libtool.

I don't have libtool working yet, but I'm trying it so it can build
shared libraries on platforms like AIX without having to learn how to
do it myself. Right now none of the tests that rely on sk-dummy work
there (at least, not on the versions I have access to).

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: [PATCH 0/5] ZSTD compression support for OpenSSH [ In reply to ]
On 2020-09-07 11:21:13 [+1000], Darren Tucker wrote:
> The zstd part would be a larger discussion because we would need to
> either carry it as a Portable patch or have zstd added to OpenBSD
> base, and I don't know if that would be accepted. Do you have any
> performance numbers for zstd in this application?

A key stroke is here 10 bytes of raw data which zstd compresses usually
into 10 bytes while zlib manages to squeeze it into 5 bytes. This leads
to better compression ratio for zlib in ssh's accounting (visible in
verbose mode after connection terminates). The data length, that will be
transferred over the wire, is the same for 5 and 10 bytes data after the
crypto part (with padding and so on).

Regarding statistics, do you have anything specific in mind?
A ~100MiB file copied with scp over a 10MBit link (the percentage number
after CPU indicates the CPU load as observed by top):
(scp file server:)
zstd
| CPU, ssh 4-6%, sshd 7-16%
| file 100% 107MB 11.5MB/s 00:09
| Transferred: sent 11144324, received 30760 bytes, in 9.5 seconds
| Bytes per second: sent 1176963.9, received 3248.6
| debug1: compress outgoing: raw data 112653640, compressed 11056148, factor 0.10
| debug1: compress incoming: raw data 9281, compressed 12113, factor 1.31

zlib
| CPU, ssh 30%, sshd 7-16%
| file 100% 107MB 14.0MB/s 00:07
| Transferred: sent 9037812, received 21984 bytes, in 7.8 seconds
| Bytes per second: sent 1159233.8, received 2819.8
| debug1: compress outgoing: raw data 112653676, compressed 8949791, factor 0.08
| debug1: compress incoming: raw data 8885, compressed 4382, factor 0.49

A 530MiB file over a ~200MBit link:
(scp server:file file)

zlib
| CPU, sshd 100%, ssh 30%
| t 100% 537MB 59.3MB/s 00:09
| Transferred: sent 100148, received 45178992 bytes, in 9.2 seconds
| Bytes per second: sent 10924.6, received 4928350.1
| debug1: compress outgoing: raw data 44367, compressed 21128, factor 0.48
| debug1: compress incoming: raw data 563267178, compressed 44746500, factor 0.08

zstd
| CPU, sshd 30-40%, ssh 26-28%
| t 100% 537MB 82.6MB/s 00:06
| Transferred: sent 144496, received 55714260 bytes, in 6.6 seconds
| Bytes per second: sent 21789.3, received 8401454.6
| debug1: compress outgoing: raw data 46014, compressed 61226, factor 1.33
| debug1: compress incoming: raw data 563267187, compressed 55281740, factor 0.10

incompressible data
zlib
| CPU, sshd 70%, ssh 14%
| u 100% 300MB 22.5MB/s 00:13
| Transferred: sent 57068, received 315112228 bytes, in 13.5 seconds
| Bytes per second: sent 4236.6, received 23393315.6
| debug1: compress outgoing: raw data 24981, compressed 11877, factor 0.48
| debug1: compress incoming: raw data 314745446, compressed 314860750, factor 1.00

zstd
| CPU, sshd 18%, ssh 12%
| u 100% 300MB 21.6MB/s 00:13
| Transferred: sent 79060, received 315111884 bytes, in 14.0 seconds
| Bytes per second: sent 5644.4, received 22496976.5
| debug1: compress outgoing: raw data 24981, compressed 33182, factor 1.33
| debug1: compress incoming: raw data 314745500, compressed 314802781, factor 1.00

Sebastian
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: [PATCH 0/5] ZSTD compression support for OpenSSH [ In reply to ]
Sebastian Andrzej Siewior wrote in
<20200908073415.ymmao6fcemn5hqns@flow>:
|On 2020-09-07 11:21:13 [+1000], Darren Tucker wrote:
|> The zstd part would be a larger discussion because we would need to
|> either carry it as a Portable patch or have zstd added to OpenBSD
|> base, and I don't know if that would be accepted. Do you have any
|> performance numbers for zstd in this application?
|
|A key stroke is here 10 bytes of raw data which zstd compresses usually
|into 10 bytes while zlib manages to squeeze it into 5 bytes. This leads
|to better compression ratio for zlib in ssh's accounting (visible in
|verbose mode after connection terminates). The data length, that will be
|transferred over the wire, is the same for 5 and 10 bytes data after the
|crypto part (with padding and so on).

Then something must be wrong on the configuration side for sure.
I personally have switched all possible private use cases to zstd
years ago, in can be configured to be tremendously fast, faster
even than lz4, no?, or compress almost as good as xz, it depends
on the data, xz can surely outperform it in size.
On the decompression side zstd is just tremendously fast.
And all in one algorithm.

--steffen
|
|Der Kragenbaer, The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: [PATCH 0/5] ZSTD compression support for OpenSSH [ In reply to ]
On 2020-09-08 15:06:31 [+0200], Steffen Nurpmeso wrote:
> Sebastian Andrzej Siewior wrote in
> <20200908073415.ymmao6fcemn5hqns@flow>:
> |A key stroke is here 10 bytes of raw data which zstd compresses usually
> |into 10 bytes while zlib manages to squeeze it into 5 bytes. This leads
> |to better compression ratio for zlib in ssh's accounting (visible in
> |verbose mode after connection terminates). The data length, that will be
> |transferred over the wire, is the same for 5 and 10 bytes data after the
> |crypto part (with padding and so on).
>
> Then something must be wrong on the configuration side for sure.
> I personally have switched all possible private use cases to zstd
> years ago, in can be configured to be tremendously fast, faster
> even than lz4, no?, or compress almost as good as xz, it depends
> on the data, xz can surely outperform it in size.
> On the decompression side zstd is just tremendously fast.
> And all in one algorithm.

I don't follow. A key stroke is one ssh packet which has 10 bytes.
This packet must be compressed in a way that it can be fully
decompressed once received on the remote side (ZSTD_e_flush in zstd
terms which finishes the current block). Due to the way framing works in
zstd we end up with a 10 bytes block. Higher compression levels can
achieve 9 bytes but as I stated it does not make sense due to padding of
the compressed data. The zlib framing is smaller here but then zstd
knows exactly how many bytes it will decompress and when the block/frame
ends.

Do note that this is not simple file compression where you end up with
one frame and blocks created on demand.

> --steffen

Sebastian
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: [PATCH 0/5] ZSTD compression support for OpenSSH [ In reply to ]
Sebastian Andrzej Siewior wrote in
<20200908195245.gs5ve6fsvbyseuhx@flow>:
|On 2020-09-08 15:06:31 [+0200], Steffen Nurpmeso wrote:
|> Sebastian Andrzej Siewior wrote in
|> <20200908073415.ymmao6fcemn5hqns@flow>:
|>|A key stroke is here 10 bytes of raw data which zstd compresses usually
|>|into 10 bytes while zlib manages to squeeze it into 5 bytes. This leads
|>|to better compression ratio for zlib in ssh's accounting (visible in
|>|verbose mode after connection terminates). The data length, that will be
|>|transferred over the wire, is the same for 5 and 10 bytes data after the
|>|crypto part (with padding and so on).
|>
|> Then something must be wrong on the configuration side for sure.
|> I personally have switched all possible private use cases to zstd
|> years ago, in can be configured to be tremendously fast, faster
|> even than lz4, no?, or compress almost as good as xz, it depends
|> on the data, xz can surely outperform it in size.
|> On the decompression side zstd is just tremendously fast.
|> And all in one algorithm.
|
|I don't follow. A key stroke is one ssh packet which has 10 bytes.
|This packet must be compressed in a way that it can be fully
|decompressed once received on the remote side (ZSTD_e_flush in zstd
|terms which finishes the current block). Due to the way framing works in
|zstd we end up with a 10 bytes block. Higher compression levels can
|achieve 9 bytes but as I stated it does not make sense due to padding of
|the compressed data. The zlib framing is smaller here but then zstd
|knows exactly how many bytes it will decompress and when the block/frame
|ends.
|
|Do note that this is not simple file compression where you end up with
|one frame and blocks created on demand.

I see. Thanks for the explanation.

--steffen
|
|Der Kragenbaer, The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: [PATCH 0/5] ZSTD compression support for OpenSSH [ In reply to ]
On 2020-09-08 09:34:15 [+0200], Sebastian Andrzej Siewior wrote:
> Regarding statistics, do you have anything specific in mind?

Another data point: I have a VM image of 8GiB. To copy it via `scp' over
a gbit link (Ryzen7 on the sending/compressing side):

uncompressed:
|vm.img 100% 8192MB 111.8MB/s 01:13
|Transferred: sent 8600429856, received 1510848 bytes, in 73.4 seconds
|Bytes per second: sent 117222196.9, received 20592.6

zstd:
|vm.img 100% 8192MB 149.7MB/s 00:54
|Transferred: sent 943612408, received 5181584 bytes, in 54.8 seconds
|Bytes per second: sent 17220903.6, received 94563.8
|debug1: compress outgoing: raw data 8594657653, compressed 937958643, factor 0.11
|debug1: compress incoming: raw data 1665529, compressed 2220303, factor 1.33

zlib:
|vm.img 100% 8192MB 46.0MB/s 02:57
|Transferred: sent 1438436560, received 1422144 bytes, in 178.0 seconds
|Bytes per second: sent 8080210.1, received 7988.7
|debug1: compress outgoing: raw data 1574650157, compressed 362009415, factor 0.23
|debug1: compress incoming: raw data 119367, compressed 56535, factor 0.47

In the uncompressed case the link was the limiting speed. With zstd's
compression the remote side was the limiting factor (mostlikely the
disk). With zlib's compression, the CPU was the limiting factor.

After compressing it with `xz' the file size was 848MiB. Sending tha
file resulted in:

uncompressed:
| vm.img.xz 100% 848MB 112.1MB/s 00:07
| Transferred: sent 890169768, received 138532 bytes, in 7.6 seconds
| Bytes per second: sent 116547054.4, received 18137.5

zstd:
| vm.img.xz 100% 848MB 112.1MB/s 00:07
| Transferred: sent 890388312, received 192832 bytes, in 7.6 seconds
| Bytes per second: sent 116563363.4, received 25244.2
| debug1: compress outgoing: raw data 889869293, compressed 889950452, factor 1.00
| debug1: compress incoming: raw data 62005, compressed 82269, factor 1.33

zlib:
| vm.img.xz 100% 848MB 35.2MB/s 00:24
| Transferred: sent 890667956, received 157768 bytes, in 24.2 seconds
| Bytes per second: sent 36815106.4, received 6521.2
| debug1: compress outgoing: raw data 890112824, compressed 889965916, factor 1.00
| debug1: compress incoming: raw data 70699, compressed 33689, factor 0.48

There is no visible overhead with zstd compression compared to
uncompressed.

Sebastian
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev