这是indexloc提供的服务，不要输入任何密码
Skip to content

Fixes and improvements to scripts for working with cyclic dependencies #20513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Fixes and improvements to scripts for working with cyclic dependencies #20513

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8038b40
Adding the ability to work with cyclic dependencies
Maxython Jul 26, 2024
3b9b04c
termux_step_setup_variables.sh: adding new variables
Maxython Jul 26, 2024
da8a4c2
Implementation of virtual packages
Maxython Sep 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions build-all.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,7 @@ test -f "$HOME"/.termuxrc && . "$HOME"/.termuxrc
: ${TERMUX_ARCH:="aarch64"}
: ${TERMUX_FORMAT:="debian"}
: ${TERMUX_DEBUG_BUILD:=""}
: ${TERMUX_INSTALL_DEPS:="-s"}
# Set TERMUX_INSTALL_DEPS to -s unless set to -i
: ${TERMUX_INSTALL_DEPS:=""}

_show_usage() {
echo "Usage: ./build-all.sh [-a ARCH] [-d] [-i] [-o DIR] [-f FORMAT]"
Expand Down Expand Up @@ -99,6 +98,16 @@ while read -r PKG PKG_DIR; do

# Update build status
echo "$PKG" >> "$BUILDSTATUS_FILE"

# Check which packages were also compiled
if [ -z "$TERMUX_INSTALL_DEPS" ]; then
for build_pkg in ~/.termux-build/*; do
pkgname="${build_pkg##*/}"
(grep -q '^_' <<< "${pkgname}" || grep -q "^${pkgname}\$" "$BUILDSTATUS_FILE") && continue
echo "The \"${pkgname}\" package was also compiled"
echo "${pkgname}" >> "$BUILDSTATUS_FILE"
Copy link
Member

@Grimler91 Grimler91 Aug 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a good idea. If build environment is not clean then the previously built packages will be added to buildstatus by this. That is most likely not what the builder wants.

I.e. if I run:

./build-all.sh -o ./special-build-dir/
# .... ctrl-c after X packages
./build-package.sh some-other-package
./build-all.sh -o ./special-build-dir/

Then some-other-package foo will be added to buildstatus file, but it will not be in special-build-dir/. I think it is reasonable for build-all.sh to build all packages, and not add extra packages it finds to the buildstatus file.

Copy link
Member

@TomJo2000 TomJo2000 Aug 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think preserving the build status file would be very good to keep around as a commandline option.

I've been using that extensively as part of the #21130 build testing.

  • Run ./build-all.sh
  • Some package errors
  • Read the ${pkgname}.log, add it manually to the end of buildstatus.txt to skip it
  • Repeat

Copy link
Member

@TomJo2000 TomJo2000 Aug 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a good idea. If build environment is not clean then the previously built packages will be added to buildstatus by this. That is most likely not what the builder wants.

I.e. if I run:

./build-all.sh -o ./special-build-dir/
# .... ctrl-c after X packages
./build-package.sh some-other-package
./build-all.sh -o ./special-build-dir/

Then some-other-package foo will be added to buildstatus file, but it will not be in special-build-dir/.

This example I absolutely agree with.
I've had to do a couple spot checks for individual packages out of order.
Running something like ./build-package.sh coreutils should not add to the buildstatus.txt file.

Copy link
Member Author

@Maxython Maxython Sep 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check allows you to check if other packages have been compiled while other packages are being compiled. This mostly happens with package dependencies that have a circular dependency with the package. Thanks to this check, build-all.sh avoids recompiling the package.

I don't think this is a good idea. If build environment is not clean then the previously built packages will be added to buildstatus by this. That is most likely not what the builder wants.

Thanks for noting. I think that in the script build-all.sh I need to add the process of cleaning the environment (i.e. running the script clean.sh) before compiling all the packages. I'll think about how to solve this.

done
fi
done<"${BUILDORDER_FILE}"

# Update build status
Expand Down
130 changes: 79 additions & 51 deletions build-package.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -486,6 +486,44 @@ termux_run_base_and_multilib_build_step() {
fi
}

# Sets up TERMUX_PKG_BUILDER_DIR and TERMUX_PKG_BUILDER_SCRIPT
termux_step_get_build_script_directory() {
TERMUX_PKG_NAME="${1}"
if [[ ${TERMUX_PKG_NAME} == *"/"* ]]; then
# Path to directory which may be outside this repo:
if [ ! -d "${TERMUX_PKG_NAME}" ]; then termux_error_exit "'${TERMUX_PKG_NAME}' seems to be a path but is not a directory"; fi
export TERMUX_PKG_BUILDER_DIR=$(realpath "${TERMUX_PKG_NAME}")
TERMUX_PKG_NAME=$(basename "${TERMUX_PKG_NAME}")
else
# Package name:
for package_directory in $TERMUX_PACKAGES_DIRECTORIES; do
if [ -d "${TERMUX_SCRIPTDIR}/${package_directory}/${TERMUX_PKG_NAME}" ]; then
export TERMUX_PKG_BUILDER_DIR=${TERMUX_SCRIPTDIR}/$package_directory/$TERMUX_PKG_NAME
break
elif [ -n "${TERMUX_IS_DISABLED=""}" ] && [ -d "${TERMUX_SCRIPTDIR}/disabled-packages/${TERMUX_PKG_NAME}" ]; then
export TERMUX_PKG_BUILDER_DIR=$TERMUX_SCRIPTDIR/disabled-packages/$TERMUX_PKG_NAME
break
fi
done
if [ -z "${TERMUX_PKG_BUILDER_DIR}" ]; then
termux_error_exit "No package $TERMUX_PKG_NAME found in any of the enabled repositories. Are you trying to set up a custom repository?"
fi
fi
TERMUX_PKG_BUILDER_SCRIPT=$TERMUX_PKG_BUILDER_DIR/build.sh
if test ! -f "$TERMUX_PKG_BUILDER_SCRIPT"; then
termux_error_exit "No build.sh script at package dir $TERMUX_PKG_BUILDER_DIR!"
fi
}

# Removes included sources in a virtual package
# Should only be run on error or when the script exits.
termux_remove_include_virtual_files() {
[ -z "${TERMUX_VIRTUAL_PKG-}" ] || [ "$TERMUX_VIRTUAL_PKG" = "false" ] || [ "$TERMUX_VIRTUAL_PKG_INCLUDE" = "false" ] && return
for file in $TERMUX_VIRTUAL_PKG_INCLUDE; do
rm -fr "${TERMUX_VIRTUAL_PKG_BUILDER_DIR}/${file}"
done
}

# Special hook to prevent use of "sudo" inside package build scripts.
# build-package.sh shouldn't perform any privileged operations.
sudo() {
Expand Down Expand Up @@ -516,6 +554,7 @@ _show_usage() {
echo " flags are not passed."
echo " -w Install dependencies without version binding."
echo " -s Skip dependency check."
echo " -b Fix circular dependencies using virtual packages only."
echo " -o Specify directory where to put built packages. Default: output/."
echo " --format Specify package output format (debian, pacman)."
echo " --library Specify library of package (bionic, glibc)."
Expand Down Expand Up @@ -587,6 +626,7 @@ while (($# >= 1)); do
-r) export TERMUX_PKGS__BUILD__RM_ALL_PKG_BUILD_DEPENDENT_DIRS=true;;
-w) export TERMUX_WITHOUT_DEPVERSION_BINDING=true;;
-s) export TERMUX_SKIP_DEPCHECK=true;;
-b) export TERMUX_FIX_CYCLIC_DEPS_WITH_VIRTUAL_PKGS=true;;
-o)
if [ $# -ge 2 ]; then
shift 1
Expand Down Expand Up @@ -632,7 +672,7 @@ if [ -n "${TERMUX_PACKAGE_LIBRARY-}" ]; then
esac
fi

if [ "${TERMUX_INSTALL_DEPS-false}" = "true" ] || [ "${TERMUX_PACKAGE_LIBRARY-bionic}" = "glibc" ]; then
if [ "$TERMUX_REPO_APP__PACKAGE_NAME" = "$TERMUX_APP_PACKAGE" ]; then
# Setup PGP keys for verifying integrity of dependencies.
# Keys are obtained from our keyring package.
gpg --list-keys 2C7F29AE97891F6419A9E2CDB0076E490B71616B > /dev/null 2>&1 || {
Expand Down Expand Up @@ -683,40 +723,24 @@ for ((i=0; i<${#PACKAGE_LIST[@]}; i++)); do
fi

# Check the package to build:
TERMUX_PKG_NAME=$(basename "${PACKAGE_LIST[i]}")
export TERMUX_PKG_BUILDER_DIR=
if [[ ${PACKAGE_LIST[i]} == *"/"* ]]; then
# Path to directory which may be outside this repo:
if [ ! -d "${PACKAGE_LIST[i]}" ]; then termux_error_exit "'${PACKAGE_LIST[i]}' seems to be a path but is not a directory"; fi
export TERMUX_PKG_BUILDER_DIR=$(realpath "${PACKAGE_LIST[i]}")
else
# Package name:
for package_directory in $TERMUX_PACKAGES_DIRECTORIES; do
if [ -d "${TERMUX_SCRIPTDIR}/${package_directory}/${TERMUX_PKG_NAME}" ]; then
export TERMUX_PKG_BUILDER_DIR=${TERMUX_SCRIPTDIR}/$package_directory/$TERMUX_PKG_NAME
break
elif [ -n "${TERMUX_IS_DISABLED=""}" ] && [ -d "${TERMUX_SCRIPTDIR}/disabled-packages/${TERMUX_PKG_NAME}" ]; then
export TERMUX_PKG_BUILDER_DIR=$TERMUX_SCRIPTDIR/disabled-packages/$TERMUX_PKG_NAME
break
fi
done
if [ -z "${TERMUX_PKG_BUILDER_DIR}" ]; then
termux_error_exit "No package $TERMUX_PKG_NAME found in any of the enabled repositories. Are you trying to set up a custom repository?"
fi
fi
TERMUX_PKG_BUILDER_SCRIPT=$TERMUX_PKG_BUILDER_DIR/build.sh
if test ! -f "$TERMUX_PKG_BUILDER_SCRIPT"; then
termux_error_exit "No build.sh script at package dir $TERMUX_PKG_BUILDER_DIR!"
fi
termux_step_get_build_script_directory "${PACKAGE_LIST[i]}"

# Add trap to remove included sources in virtual package
trap 'termux_remove_include_virtual_files' ERR EXIT

termux_step_setup_variables
termux_step_handle_buildarch

termux_step_cleanup_packages
termux_step_start_build

if ! termux_check_package_in_building_packages_list "${TERMUX_PKG_BUILDER_DIR#${TERMUX_SCRIPTDIR}/}"; then
echo "${TERMUX_PKG_BUILDER_DIR#${TERMUX_SCRIPTDIR}/}" >> $TERMUX_BUILD_PACKAGE_CALL_BUILDING_PACKAGES_LIST_FILE_PATH
TERMUX_PKG_PATH="${TERMUX_PKG_BUILDER_DIR#${TERMUX_SCRIPTDIR}/}"
if ! termux_check_package_in_building_packages_list "$TERMUX_PKG_PATH"; then
echo "$TERMUX_PKG_PATH" >> $TERMUX_BUILD_PACKAGE_CALL_BUILDING_PACKAGES_LIST_FILE_PATH
else
cat -n $TERMUX_BUILD_PACKAGE_CALL_BUILDING_PACKAGES_LIST_FILE_PATH
termux_error_exit "A cyclic dependency was detected that was not resolved during the assembly of packages. Abort."
fi

if [ "$TERMUX_CONTINUE_BUILD" == "false" ]; then
Expand All @@ -727,7 +751,9 @@ for ((i=0; i<${#PACKAGE_LIST[@]}; i++)); do
termux_step_override_config_scripts
fi

termux_step_create_timestamp_file
if [ "$TERMUX_VIRTUAL_PKG" == "false" ]; then
termux_step_create_timestamp_file
fi

if [ "$TERMUX_CONTINUE_BUILD" == "false" ]; then
cd "$TERMUX_PKG_CACHEDIR"
Expand Down Expand Up @@ -762,31 +788,33 @@ for ((i=0; i<${#PACKAGE_LIST[@]}; i++)); do
termux_step_install_pacman_hooks
termux_step_install_service_scripts
termux_step_install_license
cd "$TERMUX_PKG_MASSAGEDIR"
termux_step_copy_into_massagedir
cd "$TERMUX_PKG_MASSAGEDIR/$TERMUX_PREFIX_CLASSICAL"
termux_step_pre_massage
cd "$TERMUX_PKG_MASSAGEDIR/$TERMUX_PREFIX_CLASSICAL"
termux_step_massage
cd "$TERMUX_PKG_MASSAGEDIR/$TERMUX_PREFIX_CLASSICAL"
termux_step_post_massage
# At the final stage (when the package is archiving) it is better to use commands from the system
if [ "$TERMUX_ON_DEVICE_BUILD" = "false" ]; then
export PATH="/usr/bin:$PATH"
fi
cd "$TERMUX_PKG_MASSAGEDIR"
if [ "$TERMUX_PACKAGE_FORMAT" = "debian" ]; then
termux_step_create_debian_package
elif [ "$TERMUX_PACKAGE_FORMAT" = "pacman" ]; then
termux_step_create_pacman_package
else
termux_error_exit "Unknown packaging format '$TERMUX_PACKAGE_FORMAT'."
if [ "$TERMUX_VIRTUAL_PKG" == "false" ]; then
cd "$TERMUX_PKG_MASSAGEDIR"
termux_step_copy_into_massagedir
cd "$TERMUX_PKG_MASSAGEDIR/$TERMUX_PREFIX_CLASSICAL"
termux_step_pre_massage
cd "$TERMUX_PKG_MASSAGEDIR/$TERMUX_PREFIX_CLASSICAL"
termux_step_massage
cd "$TERMUX_PKG_MASSAGEDIR/$TERMUX_PREFIX_CLASSICAL"
termux_step_post_massage
# At the final stage (when the package is archiving) it is better to use commands from the system
if [ "$TERMUX_ON_DEVICE_BUILD" = "false" ]; then
export PATH="/usr/bin:$PATH"
fi
cd "$TERMUX_PKG_MASSAGEDIR"
if [ "$TERMUX_PACKAGE_FORMAT" = "debian" ]; then
termux_step_create_debian_package
elif [ "$TERMUX_PACKAGE_FORMAT" = "pacman" ]; then
termux_step_create_pacman_package
else
termux_error_exit "Unknown packaging format '$TERMUX_PACKAGE_FORMAT'."
fi
# Saving a list of compiled packages for further work with it
termux_add_package_to_built_packages_list "$TERMUX_PKG_NAME"
fi
# Saving a list of compiled packages for further work with it
if termux_check_package_in_building_packages_list "${TERMUX_PKG_BUILDER_DIR#${TERMUX_SCRIPTDIR}/}"; then
sed -i "\|^${TERMUX_PKG_BUILDER_DIR#${TERMUX_SCRIPTDIR}/}$|d" "$TERMUX_BUILD_PACKAGE_CALL_BUILDING_PACKAGES_LIST_FILE_PATH"
if termux_check_package_in_building_packages_list "$TERMUX_PKG_PATH"; then
sed -i "\|^${TERMUX_PKG_PATH}$|d" "$TERMUX_BUILD_PACKAGE_CALL_BUILDING_PACKAGES_LIST_FILE_PATH"
fi
termux_add_package_to_built_packages_list "$TERMUX_PKG_NAME"
termux_step_finish_build
) 5< "$TERMUX_BUILD_LOCK_FILE"
done
Expand Down
66 changes: 8 additions & 58 deletions packages/glib/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,15 @@ TERMUX_PKG_DESCRIPTION="Library providing core building blocks for libraries and
TERMUX_PKG_LICENSE="LGPL-2.1"
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION="2.86.2"
TERMUX_PKG_REVISION=1
TERMUX_PKG_SRCURL=https://download.gnome.org/sources/glib/${TERMUX_PKG_VERSION%.*}/glib-${TERMUX_PKG_VERSION}.tar.xz
TERMUX_PKG_SHA256=8a724e970855357ea8101e27727202392a0ffd5410a98336aed54ec59113e611
TERMUX_PKG_AUTO_UPDATE=true
TERMUX_PKG_DEPENDS="libandroid-support, libffi, libiconv, pcre2, resolv-conf, zlib, python"
TERMUX_PKG_BUILD_DEPENDS="gobject-introspection"
TERMUX_PKG_BREAKS="glib-dev, glib-bin"
TERMUX_PKG_REPLACES="glib-dev, glib-bin"
TERMUX_PKG_ACCEPT_PKG_IN_DEP=true
TERMUX_PKG_VERSIONED_GIR=false
TERMUX_PKG_DISABLE_GIR=false
TERMUX_PKG_EXTRA_CONFIGURE_ARGS="
Expand Down Expand Up @@ -70,64 +73,6 @@ termux_step_pre_configure() {
rm -rf $TERMUX_HOSTBUILD_MARKER
# glib checks for __BIONIC__ instead of __ANDROID__:
CFLAGS+=" -D__BIONIC__=1"
_PREFIX="$TERMUX_PKG_TMPDIR/prefix"
local _WRAPPER_BIN="${TERMUX_PKG_BUILDDIR}/_wrapper/bin"
rm -rf "$_PREFIX" "$_WRAPPER_BIN"
mkdir -p "$_PREFIX" "$_WRAPPER_BIN"

sed '/^export PKG_CONFIG_LIBDIR=/s|$|:'${_PREFIX}'/lib/pkgconfig|' \
"${TERMUX_STANDALONE_TOOLCHAIN}/bin/pkg-config" \
> "${_WRAPPER_BIN}/pkg-config"
chmod +x "${_WRAPPER_BIN}/pkg-config"
export PKG_CONFIG="${_WRAPPER_BIN}/pkg-config"
export PATH="${_WRAPPER_BIN}:${PATH}"

# Magic happens here.
# I borrowed nested building method from https://github.com/termux/termux-packages/blob/1244c75380beefc7f7da9744d55aa88df1640acb/x11-packages/qbittorrent/build.sh#L21-L28
# and modified termux_step_configure_meson in runtime to make it use another prefix
# Also I used advice from here https://github.com/termux/termux-packages/issues/20447#issuecomment-2156066062

# Running a subshell to not mess with variables
(
# Building `glib` with `-Dintrospection=disabled` and installing it to temporary directory
TERMUX_PKG_BUILDDIR="$TERMUX_PKG_TMPDIR/glib-build"
mkdir -p "$TERMUX_PKG_BUILDDIR"
TERMUX_PKG_EXTRA_CONFIGURE_ARGS="${TERMUX_PKG_EXTRA_CONFIGURE_ARGS/"-Dintrospection=enabled"/"-Dintrospection=disabled"}"
termux_setup_gir

cd "$TERMUX_PKG_BUILDDIR"
TERMUX_PREFIX="$_PREFIX" termux_step_configure
cd "$TERMUX_PKG_BUILDDIR"
termux_step_make
cd "$TERMUX_PKG_BUILDDIR"
termux_step_make_install
)

# Running a subshell to not mess with variables
(
# Building `gobject-introspection` and installing it to temporary directory
TERMUX_PKG_BUILDER_DIR="$TERMUX_SCRIPTDIR/packages/gobject-introspection"
TERMUX_PKG_BUILDDIR="$TERMUX_PKG_TMPDIR/gobject-introspection-build"
TERMUX_PKG_SRCDIR="$TERMUX_PKG_TMPDIR/gobject-introspection-src"
LDFLAGS+=" -L${_PREFIX}/lib"
mkdir -p "$TERMUX_PKG_BUILDDIR" "$TERMUX_PKG_SRCDIR"
# Sourcing another build script for nested build
. "$TERMUX_PKG_BUILDER_DIR/build.sh"
cd "$TERMUX_PKG_CACHEDIR"

termux_step_get_source
termux_step_get_dependencies_python
termux_step_patch_package

termux_step_pre_configure

cd "$TERMUX_PKG_BUILDDIR"
TERMUX_PREFIX="$_PREFIX" termux_step_configure
cd "$TERMUX_PKG_BUILDDIR"
termux_step_make
cd "$TERMUX_PKG_BUILDDIR"
termux_step_make_install
)

# Place the GIR files inside the root of the GIR directory (gir/.) of the package
termux_setup_gir
Expand All @@ -136,6 +81,11 @@ termux_step_pre_configure() {
export TERMUX_MESON_ENABLE_SOVERSION=1
}

termux_step_make_install() {
rm -f "${TERMUX_PREFIX}"/lib/libg{lib,object,thread,module,io,irepository}-2.0.so
ninja -j $TERMUX_PKG_MAKE_PROCESSES install
}

termux_step_post_make_install() {
local pc_files=$(ls "${TERMUX_PREFIX}/opt/glib/cross/lib/x86_64-linux-gnu/pkgconfig")
for pc in ${pc_files}; do
Expand Down
17 changes: 11 additions & 6 deletions scripts/build/termux_step_finish_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,18 @@
termux_step_finish_build() {
echo "termux - build of '$TERMUX_PKG_NAME' done"
echo "termux - build of $(test "${TERMUX_VIRTUAL_PKG}" = "true" && echo "virtual ")'$TERMUX_PKG_NAME' done"
test -t 1 && printf "\033]0;%s - DONE\007" "$TERMUX_PKG_NAME"

mkdir -p "$TERMUX_BUILT_PACKAGES_DIRECTORY"
echo "$TERMUX_PKG_FULLVERSION" > "$TERMUX_BUILT_PACKAGES_DIRECTORY/$TERMUX_PKG_NAME"
if [ "$TERMUX_VIRTUAL_PKG" = "false" ]; then
echo "$TERMUX_PKG_FULLVERSION" > "$TERMUX_BUILT_PACKAGES_DIRECTORY/$TERMUX_PKG_NAME"

for subpackage in "$TERMUX_PKG_BUILDER_DIR"/*.subpackage.sh; do
local subpkg_name="$(basename $subpackage | sed 's@\.subpackage\.sh@@g')"
echo "$TERMUX_PKG_FULLVERSION" > "$TERMUX_BUILT_PACKAGES_DIRECTORY/${subpkg_name}"
done
else
echo "" > "$TERMUX_BUILT_PACKAGES_DIRECTORY/$TERMUX_PKG_NAME-virtual"
fi

for subpackage in "$TERMUX_PKG_BUILDER_DIR"/*.subpackage.sh; do
local subpkg_name="$(basename $subpackage | sed 's@\.subpackage\.sh@@g')"
echo "$TERMUX_PKG_FULLVERSION" > "$TERMUX_BUILT_PACKAGES_DIRECTORY/${subpkg_name}"
done
exit 0
}
Loading