I am a huge proponent of zstd after I learned about it on HN.
I've recently had to compress a 12 Gb csv file. zstd took ~3 sec for compression and ~11 sec for decompression and got the file to ~1.1 Gb. Xz took ~3.5 min for compression(!) and the resulting file was ~740 Mb(I didn't measure the decompression time). I just realized that in most cases it's more efficient to use zstd, especially for file transfer.
The majority of OS that people use either have it installed or can be trivially downloaded to, so I see no point in using gzip nowadays, unless it is mandated by API
Hey, I didn't confuse them but I guess I should have been more specific. I've addressed 2 main points in my text. 1) In my opinion, for the vast majority of cases you don't need to be worried about comparability, as you can easily install better alternatives to gzip for platforms a lot of people use. 2) I think zstd can become a primary replacement for gzip due to its high speed of compression and good compression ratio, even when compared to such great algorithms like xz/lzma. Sacrificing some compression ratio for (de)compression speed is worth it, for me at least
zstd is faster and provides better compression than gzip at every point on the curve. There is no reason to use gzip these days other than backwards compatibility.
I've reached the same conclusion, and I've been using zstd unless I want extremely fast (de)compression with lz4.
And even when I still have to use gzip (the format), the executables of libdeflate are fully compatible and noticeably faster than gzip/gunzip (the tool).
Do zless, zcat and zgrep support zstd everywhere? And I mean everywhere? VMs? Alpine? FreeBSD? OSX? Openwrt?
Nothing is shittier than sshing into a box that doesn’t understand half of your command line tricks. Or the clever shell script you just tested six ways to Sunday. It’s like fighting with your hands tied behind your back.
bash4 was released Feb 2009. 16 years ago.
The very latest version of osx ships with bash3, from 20 years ago.
It's a fair question, and "currently supported" is essentially meaningless.
(Yes we all know about macports and the other more popular but less correct systems, and yes most bash scripts should be written to specifically avoid relying on any special features exactly to avoid being broken so easily. These are both true and both beside the point. Fact is still, this is not only a "currently supported" system by some technicality like the final months of a 5 year lts release or something, this is the up to this very minute version of a massive install base, with a 20 year old version of something as basic and all-touching as the very shell itself.
I know about this so intimately because I actually have scripts that don't work because they were intentional stunts to see just what could be done without forking any external exes or even child shells, but allow using every possible feature and trick in bash itself.)
It doesn't matter why. All that matters is that "current" is not a valid word. Old things exist in "current" systems. And current systems may also be old systems.
Ok. I think engineers are well capable of evaluating what systems they need to support and if zstd is a usable option for them. In many situations, the answer will be "yes."
You can download a supported zstd release through homebrew for every version of macOS that is supported by Apple (as those are the releases homebrew supports), and it compiles from source for older ones.
I don't even know why we're talking about bash here. But for the record, this is a unique circumstance that affects only software which made the transition from GPLv2 -> GPLv3. Changing your license can cause you to fork away from some users. News at 11.
You can get bash4 or 5 too, as I already said. I can't help you with your depth of experience. I didn't even say zstd was unfit to depend on, just that it was a fair question.
Once again, I’m having to remind a person that coding is a team sport and personal choices are bullshit.
Every deviation you make from stock is something else you have to put into. The onboarding docs and convince all your coworkers to install. They won’t. Or they think they will but will miss a step or machine. Or for language specific tools, they’ll use a multiversion tool like asdf or nvm or ram and have to reinstall again every upgrade.
It’s not up to me. It’s not up to you. It’s up to us. And if you can’t see that, maybe you shouldn’t be working with other developers. Or maybe you haven’t been and you need to.
Java 8 will be supported until 2032 if you pick the right distro, and that distro comes with zstd in the repos (if not pre-installed).
And if your system is so old that you don't have access to eight-year-old software, I don't see why the rest of the world needs to accommodate you. Sticking with weird and old software is fine, but that comes with the risk of having to deal with everyone else moving on.
zstd is a great setup, for sure, but the build system they use is patently awful. Can someone make an autoconf or hell, even CMake build system for them pleasee???
You could easily do something similar in GNU make or whatever without the dependencies on the FBSD build system. It's basically just the classic "cc -c a bunch of C files, then link them."
Yeah, not sure what the original commenter is complaining about. I did "git clone https://github.com/facebook/zstd", "cd zstd", "make" and have a working binary. Doesn't get easier than that.
Not everyone is using a standard tool chain or anything like that and oftentimes when I find projects with atypical build systems that give me trouble I tend to just move on to something else
> Just install the binary package from whatever distro you use? Why do you need to build it?
Cuz I'm actually a packaging maintainer for a couple different old operating systems. Regardless I thank you for being thankful even though I feel like you were being backhanded and demeaning on a number of levels.
Last I touched it they only had glitchy make files and I don't like cmake particularly, I was just like "bespoke make files are fine if they're posix compliant, but these aren't"
Same here. Storage is not that expensive so I do NOT care to squize every byte out of archive. Also, im more into retro, so portability and memory usage is more importand for me :)
Regarding your first remark: high ratio data compression has its time and place, and I personally understand that to many people it is not very desirable. In a lot of scenarios something as plain and simple as LZ4 generally suffices.
On the other hand, there is an unofficial (= unsupported) port of bzip3 to older (386+) machines that run MS-DOS6.22. I have prepared it for a retrocomputing meeting in Ontario that I attended a while back. Let me know what you think :).
After reading this, I see lots of zstd fans around here, and with good reason. That being said, I think our shop will stick with gzip until zstd arrives in the Python Standard Library.
