Fix libjxl#4410 #16460

Job	Run time
Ubuntu Build scalar	11m 33s
Ubuntu Build msan	1h 7m 34s
Ubuntu Build release-nojpeg	8m 14s
Ubuntu Build release-lcms2	8m 24s
Ubuntu Build debug	16m 29s
Ubuntu Build tsan	52m 12s
Ubuntu Build release:arm64	15m 24s
Ubuntu Build coverage	56s
Ubuntu Build lowprecision	21m 37s
Ubuntu Build release-system-lcms2	8m 18s
Ubuntu Build release	14m 12s
Ubuntu Build release:gcc-old-old	12m 50s
Ubuntu Build release:gcc-old	9m 23s
Ubuntu Build asan	32m 2s
Ubuntu Build release:clang-old	8m 23s
Ubuntu Build release:gcc	10m 43s
Ubuntu Build release:clang-old-old	9m 31s
MacOS Build release:osx	11m 18s
Ubuntu Build release:minimal	8m 16s
MacOS Build release:osx-15	10m 26s
Ubuntu Build release:arm64-lowprecision	14m 49s
	5h 52m 34s

Provide feedback