Should you also add aapt2 to the reverse dependency revision bumps?

When I build and then install the abseil-cpp package,

aapt2 does print:

CANNOT LINK EXECUTABLE "aapt2": cannot locate symbol "_ZN4absl12lts_2024011612log_internal15LogMessageFatalC1EPKciNSt6__ndk117basic_string_viewIcNS5_11char_traitsIcEEEE" referenced by "/data/data/com.termux/files/usr/bin/aapt2"...

Similarly, what about opencv and pika?

Maybe you mean to keep the number of packages built in a single PR limited, and then create a second PR afterward to rebuild the rest of the packages, if so then never mind, that would make sense.

python-grpcio doesn't seem to want to build right now.

It is weird, it is built fine locally. I'll try to build in local docker environment.

Seems like everything is fine if I rebuild only abseil-cpp and python-grpcio everything is fine, but if I rebuild everything the error from the CI logs appears. Most likely some other package pollutes environment.

Throw python-grpcio into a separate PR and merge it immediately after this one then.
Sometimes batches of packages just do not wanna be compiled together.

1. Most likely it will break in build-all.sh.
2. There is possibility for breakage by one of recently rebuilt packages.

Anyway I want to investigate this for a while.

Yeah I'm already planning another round of build testing,

• Some packages cannot be built 3: Revenge of the Xiph #23492

and with input from @robertkirkman we're gonna be expanding the scope of that testing to cover more build edgecases.

distutils.errors.CompileError: command '/home/builder/.termux-build/_cache/android-r27c-api-24-v1/bin/aarch64-linux-android-clang' failed with exit code 1

The error is internally:

third_party/upb/upb/wire/internal/decoder.h:62:10: error: call to undeclared function 'utf8_range_IsValid'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]

It also affected someone's MacOS here in a slightly different situation.

• macOS/iOS error: call to undeclared function 'utf8_range_IsValid' protocolbuffers/protobuf#19565

The minimum reproducible number and order of packages to build to reproduce the problem is:

scripts/run-docker.sh ./build-package.sh -I -f libprotobuf python-grpcio

It is some kind of a conflict between libprotobuf and python-grpcio, where libprotobuf is the polluter and python-grpcio is the pollutee.

grpc's code contains this directory, which is based on copying some code out of protobuf's code and committing it into their repository.

https://github.com/grpc/grpc/tree/master/third_party/upb/upb

It is currently copied from protobuf 29, but libprotobuf in termux-packages is at 25.1 currently, which might have something to do with it.

It seems like python-grpcio is entering an invalid build configuration if it detects libprotobuf in $TERMUX_PREFIX.

Coincidentally, it must have something to do with headers in $TERMUX_PREFIX/include, because when I test, applying

• scripts(toolchain): several changes related to $TERMUX_PREFIX/include #23513

causes the error to go away.

Oh, I was close to this. I had this as a theory and wanted to test this practically. I'll rebase the PR.

and with input from @robertkirkman we're gonna be expanding the scope of that testing to cover more build edgecases.

I will try to make a script that I might be able to run locally to generate the entire spreadsheet automatically, I'm not sure how long that will take though.

CI building fails because of lack of disk space but local building is fine. Probably I will split it in two parts.

@TomJo2000 How do you think, can we implement cleaning up $TERMUX_TOPDIR/*/{src,cache,tmp} to obtain some disk space in the case if there is no space available to avoid fails like this in the future?

CI building fails because of lack of disk space but local building is fine. Probably I will split it in two parts.

@TomJo2000 How do you think, can we implement cleaning up $TERMUX_TOPDIR/*/{src,cache,tmp} to obtain some disk space in the case if there is no space available to avoid fails like this in the future?

That should be a safe bet after any build.
So sounds good to me.
Might wanna CC: @robertkirkman on that since he's been working on prefix pollution for a while.

That should be a safe bet after any build.
So sounds good to me.

I do not think so. Termux supports continuing building (-c option of ./build-package.sh) so cleaning up right after building package is not an option. But we can make a new step which will check if we are out of disk space, let's say less than 10% and cleanup already built packages (in $TERMUX_TOPDIR, which is ~/.termux-build).

Yes, I think that it is ok to delete ~/.termux-build, after a package has been completely built but before starting the build of other packages, and that does not derail the build process of subsequent packages, but it is correct that it should be a workflow step and should only be in workflows that do not use -c, because when building packages locally, I manually enter the container and read the contents of ~/.termux-build a lot.

I think that it might be possible to get a lot of extra free space by doing that, because that is where the extracted source code of packages and the intermediate build artifacts are, and for some packages, that is what takes up the largest amount of space.

I am not sure whether it is easy to insert a step like this in between the build of every package in the CI workflow, though, without making some modification to build-package.sh. It would be great if there is a way to do it without modifying build-package.sh, but I am not sure of that right now.

Probably it will look like

termux_step_cleanup() {
	[[ -d "$TERMUX_TOPDIR" ]] || return 0

	# Extract the integer usage percentage of the disk containing $TERMUX_TOPDIR
	local USAGE=$(df --output=pcent "$TERMUX_TOPDIR" | tail -1 | tr -dc '0-9')

	# No need to cleanup if there is enough disk space
	[ "$USAGE" -lt 90 ] || return 0

	TERMUX_PACKAGES_DIRECTORIES="$(jq --raw-output 'del(.pkg_format) | keys | .[]' "${TERMUX_SCRIPTDIR}"/repo.json)"

	# Build package name regex to be used with `find`, avoiding loops.
	local PKG_REGEX="$(find ${TERMUX_PACKAGES_DIRECTORIES} -mindepth 1 -maxdepth 1 -type d -printf '%f|')^$"

	# Exclude current package from the list.
	PKG_REGEX="${PKG_REGEX//$TERMUX_PKG_NAME|/}"

	echo "INFO: cleaning up some disk space for building \"${TERMUX_PKG_NAME}\"."

	find "$TERMUX_TOPDIR" -mindepth 1 -maxdepth 1 -type d -regextype posix-extended -regex ".*/($PKG_REGEX)$" -exec rm -rf {} +
}

Yes, I think that it is ok to delete ~/.termux-build

In the case of deleting the whole $TERMUX_TOPDIR we will lose cached toolchain and cached packages. That is the reason the script I posted in the last message removes only built packages. We should delete only built packages because these directories will never be used after package was built and it's .deb or .apk file was cached.

The cached packages in particular are very important to keep since it makes it a lot quicker to rebuild a dependency and its dependents in a batch.

Yeah, that is the point of my cleanup step.

Ok yes it is good then that your method does not delete ~/.termux-build/_cache or ~/.termux-build/_cache-aarch64, etc, since it saves a lot of time to keep them when possible. I was just thinking that, since this is for running when there is very little storage space, it would save a small amount because not all packages will reuse every cached file from the ~/.termux-build/_cache or ~/.termux-build/_cache-aarch64 folders, they just redownload only the ones that they require. However, it is true that they aren't what takes up the majority of the space, so avoiding deleting them is probably the best choice.

	# Extract the integer usage percentage of the disk containing $TERMUX_TOPDIR
	local USAGE=$(df --output=pcent "$TERMUX_TOPDIR" | tail -1 | tr -dc '0-9')

Personally, my opinion is that a fixed value, and not a percentage, for example "less than 4 Gigabytes", should be used for the condition, because, if this code happens to run on local builds as well as CI, then in the event that the data-root setting of someone's Docker Daemon (default /var/lib/docker) is set to, for example, a 4 Terabye drive, and that drive is showing as 95% filled in df, then that means that the drive still has approximately 200 Gigabytes of storage available.

If there is a way that this can be made to only run in CI, though, or if you believe that using the percentage is overall better than a fixed value for a different reason, then maybe the percentage value is best.

However, it is true that they aren't what takes up the majority of the space, so avoiding deleting them is probably the best choice.

Not true for big packages like clvk (2.2GB), gdal (227MB), libarrow-cpp (264MB), python-grpcio (2.2GB), python-onnxruntime (2.6GB), wasm-component-ld (225MB), etc. It is only a few examples of what I currently see on my disk, the value in parentheses is a size of directory in ~/.termux-build. Even in the case if it is not what takes up the majority of the space it is the only thing we can safely remove without side effects.
In the case if you think it is still not valuable I should remind you about batch updating packages with their reverse dependencies, like abseil-cpp or protobuf. These batches require a lot of disk space since they rebuild a lot of packages.

Personally, my opinion is that a fixed value, and not a percentage, for example "less than 4 Gigabytes", should be used for the condition, because, if this code happens to run on local builds as well as CI, then in the event that the data-root setting of someone's Docker Daemon (default /var/lib/docker) is set to, for example, a 4 Terabye drive, and that drive is showing as 95% filled in df, then that means that the drive still has approximately 200 Gigabytes of storage available.

True. I will replace disk usage command checker with something like

# Extracts available disk space in GB
AVAILABLE=`df -B $((1024**3)) --output=avail . | tail -1`

# ...

# No need to cleanup if there is enough disk space
[ "$AVAILABLE" -lt 5 ] || return 0

Not true for big packages like clvk (2.2GB), gdal (227MB), libarrow-cpp (264MB), python-grpcio (2.2GB), python-onnxruntime (2.6GB), wasm-component-ld (225MB), etc. It is only a few examples of what I currently see on my disk, the value in parentheses is a size of directory in ~/.termux-build. Even in the case if it is not what takes up the majority of the space it is the only thing we can safely remove without side effects.
In the case if you think it is still not valuable I should remind you about batch updating packages with their reverse dependencies, like abseil-cpp or protobuf. These batches require a lot of disk space since they rebuild a lot of packages.

I am sorry I think that you misunderstood me, I was not clear in explaining what I meant. Stated clearly, I was only trying to say:

• I believe that the clvk, etc folders take up the majority of the space, and should be deleted
• I believe that the _cache, etc folders do not take up the majority of the space, and should not be deleted

Oh, now I get it. Sorry for misunderstanding.

I will open new PR with implementing new cleanup step and updating abseil-cpp along with its reverse dependencies after taking a nap and some testing.
Currently I am closing this PR.