FreeBSD, big-endian, and many new architectures

This release finally includes FreeBSD in the list of supported operating systems. That should make it much easier to port DwarFS to other *BSDs as well. Big-endian platforms are also supported, and the file system images (which use little-endian) are fully portable between architectures. Binary releases are now available for a wide range of architectures: aarch64, arm, i386, loongarch64, ppc64, ppc64le, riscv64, s390x, and x86_64.

Metadata manipulation

Before this release, file system metadata was immutable once a DwarFS image had been created, and the only way to manipulate metadata was to build a new image from scratch. This release adds two options to mkdwarfs:

• --rebuild-metadata allows changes/upgrades to the metadata block. This makes it easy to change how the metadata is packed, or even perform manipulations such as --chmod after the fact.
• --change-block-size allows you to change the physical block size of the file system image.

Bug fixes

• The linker configuration for the release binaries was broken. The symptom was that, very occasionally, tests would fail in CI with std::terminate being called after the exception handling code failed to unwind the stack. The root cause was that, in clang builds, code from libunwind and libstdc++ was arbitrarily mixed, which—depending on the order in which individual threads were scheduled in the unit tests—could lead to stack unwinding working flawlessly or not at all. This was one of the hardest bugs to track down this year; fortunately, the fix was quite simple. It’s possible this issue is present in previously released binaries, although there have been no reports.

• Made section index discovery more robust. Fixes #264.

• A recent kernel change (LKML thread, 2025-05-05) caused tools_test to fail on Linux 6.14 and later. This has been fixed by accepting both EPERM and ENOSYS as valid error codes for link() calls.

Features

• FreeBSD support. Everything that works on Linux should also work on FreeBSD. No static binaries are provided for FreeBSD, but the build should work out of the box once all dependencies are installed.

• Big-endian architectures. This is still experimental, even though all unit tests pass under QEMU and the benchmark suite runs on real hardware. This currently requires forked versions of folly and fsst. The changes are small, and the pull requests will hopefully be merged upstream soon. (facebook/folly#2484, cwida/fsst#36)

• Experimental 32-bit support. While DwarFS should largely “just work” on 32-bit with small images (a few hundred megabytes), limited address space is a problem due to DwarFS’s extensive use of memory-mapped files. There will be changes to limit the use of mmap in the future (primarily due to other issues), which should improve 32-bit compatibility as a side effect. Fixes #268.

• Wider CPU architecture coverage. Static binary releases (including universal binaries) are now available for x86_64, aarch64, i386, arm, ppc64, ppc64le, riscv64, s390x, and loongarch64. Building the new release binaries uncovered a few bugs in clang (llvm/llvm-project#150913), binutils (https://sourceware.org/bugzilla/show_bug.cgi?id=33223), mold (rui314/mold#1490, rui314/mold#1496, rui314/mold#1497, rui314/mold#1498), and UPX (upx/upx#925), not all of which have been fixed. As a result, the binaries use slightly different toolchains and configurations depending on the architecture. Fixes #266, #268.

• Custom self-extracting stub for universal binaries. It aims for simplicity and portability and should work on most Linux systems. It is used if UPX support for an architecture is unavailable, or if the binaries extract much faster than the UPX-compressed version. The stub also supports --extract-wrapped-binary <file> to extract the embedded binary.

• Category metadata stored by default. The category metadata for categorized blocks is now stored in the metadata block by default. This allows recompressing blocks with a metadata-dependent algorithm (e.g., FLAC) even if they were previously compressed with a metadata-independent algorithm. You can disable this with --no-category-metadata. See the mkdwarfs man page for details.

• Options for smaller metadata. The --no-category-names and --no-category-metadata options can be used to reduce metadata size. However, this makes it impossible to use metadata-dependent compression algorithms (e.g., FLAC) or to select category-specific compression when recompressing the image.

• Metadata rebuilding in mkdwarfs. In addition to recompressing, it is now possible to change metadata packing and apply operations such as --set-owner, --set-group, --set-time, --time-resolution, --chmod, or --no-create-timestamp. Note that these operations are potentially lossy and may be irreversible. By default, the history of metadata rebuilds is tracked in the metadata itself; you can disable this with --no-metadata-version-history.

• Change block size on existing images. You can now change the block size of an existing image using --change-block-size. This implies --rebuild-metadata and --recompress=all and can be useful for tuning performance without recreating the image from scratch.

• Runtime memory display in mkdwarfs. mkdwarfs now shows its current memory usage while running. Note that -L/--memory-limit still only limits the memory used for the block queue, not overall memory usage. Fixing this is on the roadmap; there’s no need to file an issue.

• dwarfsextract format controls. New options --format-options and --format-filters control the output format. There is also --format=auto to automatically “guess” the format and filters based on the output file name. (Thanks to @oxalica for the pull request.)

• dwarfsck detail level. New frozen_details detail level shows frozen_analysis content ordered by memory location instead of memory usage, and also shows the address range of each section.

• Lean LRU cache. Replaced folly’s EvictingCacheMap with a simple in-repo LRU cache implementation. This reduces external dependencies and binary size without sacrificing performance.

• Windows extended attributes. The pxattr utility now supports all extended attribute operations on Windows, including setxattr() and removexattr(). Error handling and reporting for extended attributes on Windows has also been improved.

Docs

• Updated dwarfs-format.md with more information on section types, compression metadata, and the Frozen2 binary metadata layout. (Thanks to @oxalica for asking the right questions, reporting bugs, and ultimately releasing a Rust library to read/write DwarFS images.)

• Added a notable users section to the README. (Thanks to Vitaly Zdanevich for the PRs.)

• Updated mkdwarfs docs with more information on worker threads and the requirements for bit-identical images.

• Major README overhaul: added a Quick Start section, added more links, and fixed typos and wording.

Full Changelog: v0.12.4...v0.13.0

eb4a21fe560721a17059eb26b14abb894420008cc0dc990b829bdfda08e97af8  dwarfs-0.13.0-Linux-aarch64.tar.xz
3781cf4e5dde77f4e7da9900dca4250d300b57bbf5ba6640ba7f61e2efb5782f  dwarfs-0.13.0-Linux-arm.tar.xz
d7157d7a2faedea61829835060edfb1614d75b8559252f237aae38a97a684e9c  dwarfs-0.13.0-Linux-i386.tar.xz
786351112f1659d041e40ef70a4376c55f68bb783fc710ff71a1cb612b6786f5  dwarfs-0.13.0-Linux-loongarch64.tar.xz
465c0a2c14f13612a0413ba4e79f853cefdcc25a82d8ea7e74e3a349ac887983  dwarfs-0.13.0-Linux-ppc64le.tar.xz
cb44187074793b4aca9e252fb0bee269725180448e7a47abbf17e38b0b32e6ac  dwarfs-0.13.0-Linux-ppc64.tar.xz
0d8eac724b5c00f22592c68c2f61a1ae7be92aef0b236ace208e73e759e6965f  dwarfs-0.13.0-Linux-riscv64.tar.xz
52dbd8e44deede8e237d4b8c2fe1388b1478c14b1f76b63b4045a7103110acfb  dwarfs-0.13.0-Linux-s390x.tar.xz
81d2c5178b51367207df819a3a754fc3141d0f74d4ac80dbf20b1f0bd9d1be44  dwarfs-0.13.0-Linux-x86_64.tar.xz
d0654fcc1219bfd11c96f737011d141c3ae5929620cd22928e49f25c37a15dc9  dwarfs-0.13.0.tar.xz
fbbdf50657caf6be3c6864768bd2f0c2f6ea955b66e07875408e0a78bed2f9b9  dwarfs-0.13.0-Windows-AMD64.7z
78c52f9ca120e11d4a4620c614d532425ab694c0f8b6c25f04b475f10b3e0b2e  dwarfs-fuse-extract-0.13.0-Linux-aarch64
451744a2be3312fcd2968aa7821d61d97ece24177cebdb4cb8cf463409f9de7d  dwarfs-fuse-extract-0.13.0-Linux-aarch64.upx
0079697de87e14ea5bbf3a0dd0e95488d8343bd423f2ca3643462520a6c95b89  dwarfs-fuse-extract-0.13.0-Linux-arm.upx
7d3fc8474c1372a92b4301c28124fac16f858d81686ee4cafe21e8b6a1cebc14  dwarfs-fuse-extract-0.13.0-Linux-i386.upx
da9558153264fd14ba49f02d9fb26a8ef5c090a3e99a643fbeb9e88b68c3fd93  dwarfs-fuse-extract-0.13.0-Linux-loongarch64
b42ad7d4a229d22ecf219fd1759f1f1ebb0ccd28b4be16e49cf307f05b154f63  dwarfs-fuse-extract-0.13.0-Linux-ppc64
a81001d3a195eab8cdfd2b29768718ff5eae64a69174d33c5114aa04b87bdb6a  dwarfs-fuse-extract-0.13.0-Linux-ppc64le
9c9f4dcee5c78466cda8d3f3b5dccaa57be0270774e373198ab5cdc6f46a8c5d  dwarfs-fuse-extract-0.13.0-Linux-riscv64
aeae39035e52632a013f318909b38e0525dc04ba4e52179ab678807b6a3755ea  dwarfs-fuse-extract-0.13.0-Linux-s390x
04c9587395dbbdfdb6562f945daf2aa83e5ff80691e1dfeb7854dfc2e0e6993e  dwarfs-fuse-extract-0.13.0-Linux-x86_64
4a31bcbf9145fa422d3cfaf3f1f58b40cd095ba9a4bee508123f97e208684f52  dwarfs-fuse-extract-0.13.0-Linux-x86_64.upx
74b730031dfe4d4aa0caaa411ea806d02ba44f32dc2618e52739ce7b52faeb4c  dwarfs-universal-0.13.0-Linux-aarch64
3ce4f428869f96c025487ffe0f6625e60bffe3768be19c21062faeaab30c8901  dwarfs-universal-0.13.0-Linux-aarch64.upx
6c911b5fbc891d9ae29f5ac20dca6748dc2ed9fc7cafb39f44163439d563431e  dwarfs-universal-0.13.0-Linux-arm.upx
da7e154272bc9efb47acbf57c5f065e0c45456fc9aa5c19e6a14395516dbaf78  dwarfs-universal-0.13.0-Linux-i386.upx
77d26cd3ada34f0ab846850b12c319c32c87ddb06940cc371014dd6d6f8c20e0  dwarfs-universal-0.13.0-Linux-loongarch64
c64d49d14043bebf72d374dbf332397acf242eb72caabd77c2d22c2529a22f03  dwarfs-universal-0.13.0-Linux-ppc64
83aa8161f9f305246e89830c15f382a3d7e5f88a7dd2dc256f2c4c5ddf2177d2  dwarfs-universal-0.13.0-Linux-ppc64le
ab51c1aa3f55b148b7360935df68d342c91c77859a0f62bb9facb6002fa15d6b  dwarfs-universal-0.13.0-Linux-riscv64
97037f5df06e12f705c54ad77da4b0defd835779b9378cb0196c6e0f97e4fe2f  dwarfs-universal-0.13.0-Linux-s390x
6e003ec681abc5542af54eb7dda3efb7c3742fc319e8973f53392b1e63de8ac6  dwarfs-universal-0.13.0-Linux-x86_64
df74d98efdce7e3deaef711fd65868a2e5afc9f0a653f326ea7b08c28e99709d  dwarfs-universal-0.13.0-Linux-x86_64.upx
17465a7a9df4bccc3bda86eeca66a85806f1b19de0a21d45d3486fa2bed3a4fa  dwarfs-universal-0.13.0-Windows-AMD64.exe