-
-
Notifications
You must be signed in to change notification settings - Fork 3.5k
Auto updates: add support for source arrays. #23678
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
@@ -54,27 +54,52 @@ for package in "${@}"; do | |||||||||||||||||||||||||||||||||||||||||||
continue | ||||||||||||||||||||||||||||||||||||||||||||
fi | ||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||
# Command substitution create a subshell, we want to do this | ||||||||||||||||||||||||||||||||||||||||||||
# to not effect the main scope's variables by sourcing the build script | ||||||||||||||||||||||||||||||||||||||||||||
new_checksum=($( | ||||||||||||||||||||||||||||||||||||||||||||
source "${buildsh_path}" 2>/dev/null | ||||||||||||||||||||||||||||||||||||||||||||
for url in "${TERMUX_PKG_SRCURL[@]}"; do | ||||||||||||||||||||||||||||||||||||||||||||
echo "Downloading ${url}" >&2 | ||||||||||||||||||||||||||||||||||||||||||||
dl_tmpdir=$(mktemp -d "${TMPDIR-/tmp}/termux.src.dl.XXXXXXXX") | ||||||||||||||||||||||||||||||||||||||||||||
( | ||||||||||||||||||||||||||||||||||||||||||||
function get_sha256() { | ||||||||||||||||||||||||||||||||||||||||||||
local dl_tmpdir | ||||||||||||||||||||||||||||||||||||||||||||
for url in "$@"; do | ||||||||||||||||||||||||||||||||||||||||||||
echo "Downloading ${url}" >&2 | ||||||||||||||||||||||||||||||||||||||||||||
dl_tmpdir="$(mktemp -d "${TMPDIR-/tmp}/termux.src.dl.XXXXXXXX")" | ||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||
if [[ -d "${dl_tmpdir}" ]]; then | ||||||||||||||||||||||||||||||||||||||||||||
if ! curl --fail --location --retry 3 --output "${dl_tmpdir}/source-bundle" "$url"; then | ||||||||||||||||||||||||||||||||||||||||||||
rm -rf "${dl_tmpdir}" | ||||||||||||||||||||||||||||||||||||||||||||
fi | ||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||
if [[ -f "${dl_tmpdir}/source-bundle" ]]; then | ||||||||||||||||||||||||||||||||||||||||||||
sha256sum "${dl_tmpdir}/source-bundle" | awk '{ print $1 }' | ||||||||||||||||||||||||||||||||||||||||||||
fi | ||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||
if [[ -d "${dl_tmpdir}" ]]; then | ||||||||||||||||||||||||||||||||||||||||||||
if ! curl --fail --location --retry 3 --output "${dl_tmpdir}/source-bundle" "$url"; then | ||||||||||||||||||||||||||||||||||||||||||||
rm -rf "${dl_tmpdir}" | ||||||||||||||||||||||||||||||||||||||||||||
fi | ||||||||||||||||||||||||||||||||||||||||||||
Comment on lines
+62
to
74
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||||||||||
done | ||||||||||||||||||||||||||||||||||||||||||||
} | ||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||
if [[ -f "${dl_tmpdir}/source-bundle" ]]; then | ||||||||||||||||||||||||||||||||||||||||||||
sha256sum "${dl_tmpdir}/source-bundle" | awk '{ print $1 }' | ||||||||||||||||||||||||||||||||||||||||||||
fi | ||||||||||||||||||||||||||||||||||||||||||||
# shellcheck source=/dev/null | ||||||||||||||||||||||||||||||||||||||||||||
source "${buildsh_path}" 2>/dev/null | ||||||||||||||||||||||||||||||||||||||||||||
source "$REPO_ROOT/scripts/updates/utils/termux_pkg_patch_value.sh" | ||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||
rm -rf "${dl_tmpdir}" | ||||||||||||||||||||||||||||||||||||||||||||
fi | ||||||||||||||||||||||||||||||||||||||||||||
done | ||||||||||||||||||||||||||||||||||||||||||||
)) | ||||||||||||||||||||||||||||||||||||||||||||
old_checksum='' | ||||||||||||||||||||||||||||||||||||||||||||
new_checksum='' | ||||||||||||||||||||||||||||||||||||||||||||
case "${TERMUX_PKG_SRCURL@a}" in | ||||||||||||||||||||||||||||||||||||||||||||
'a') { | ||||||||||||||||||||||||||||||||||||||||||||
for (( i = 0; i < ${#TERMUX_PKG_SRCURL[*]}; i++ )); do | ||||||||||||||||||||||||||||||||||||||||||||
old_checksum="${TERMUX_PKG_SHA256[$i]}" | ||||||||||||||||||||||||||||||||||||||||||||
new_checksum="$(get_sha256 "${TERMUX_PKG_SRCURL[$i]}")" | ||||||||||||||||||||||||||||||||||||||||||||
termux_pkg_patch_value 'TERMUX_PKG_SHA256' \ | ||||||||||||||||||||||||||||||||||||||||||||
"$old_checksum" "$new_checksum" "$buildsh_path" | ||||||||||||||||||||||||||||||||||||||||||||
done | ||||||||||||||||||||||||||||||||||||||||||||
} > "${TMPDIR-/tmp}/${package}.auto.patch" | ||||||||||||||||||||||||||||||||||||||||||||
;; | ||||||||||||||||||||||||||||||||||||||||||||
'') { | ||||||||||||||||||||||||||||||||||||||||||||
old_checksum="${TERMUX_PKG_SHA256}" | ||||||||||||||||||||||||||||||||||||||||||||
new_checksum="$(get_sha256 "${TERMUX_PKG_SRCURL}")" | ||||||||||||||||||||||||||||||||||||||||||||
termux_pkg_patch_value 'TERMUX_PKG_SHA256' \ | ||||||||||||||||||||||||||||||||||||||||||||
"$old_checksum" "$new_checksum" "$buildsh_path" | ||||||||||||||||||||||||||||||||||||||||||||
} > "${TMPDIR-/tmp}/${package}.auto.patch" | ||||||||||||||||||||||||||||||||||||||||||||
;; | ||||||||||||||||||||||||||||||||||||||||||||
esac | ||||||||||||||||||||||||||||||||||||||||||||
) | ||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||
if (( ${#new_checksum[@]} )); then | ||||||||||||||||||||||||||||||||||||||||||||
# Replace old SHA-256. | ||||||||||||||||||||||||||||||||||||||||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
#!/bin/false | ||
# This script should be sourced, not run standalone. | ||
|
||
# shellcheck shell=bash | ||
# $1 - What variable are we patching? | ||
# $2 - Old value to replace | ||
# $3 - New value to substitute | ||
# $4 - File to patch | ||
termux_pkg_patch_value() { | ||
local target_value="$1" | ||
local old_value="$2" | ||
local new_value="$3" | ||
local file_path="$4" | ||
|
||
local regex | ||
case "$target_value" in | ||
'TERMUX_PKG_VERSION') regex='(?m)^\s*TERMUX_PKG_VERSION[^=]*=(?:[^\(\n]*$|\((?:[^\)]*\))+?(?=\s*#|$))';; | ||
'TERMUX_PKG_SHA256') regex='(?m)^\s*TERMUX_PKG_SHA256[^=]*=(?:[^\(\n]*$|\((?:[^\)]*\))+?(?=\s*#|$))';; | ||
*) return 1;; | ||
esac | ||
|
||
local match | ||
match="$(grep -bzoP "${regex}" "${file_path}")" | ||
git -P diff --ignore-space-at-eol --text --no-index \ | ||
<(tail -c+$(( ${match%:*} + 1 )) "${file_path}" | head -c${#match#*:}) \ | ||
<(sed -e "s/${old_value}/${new_value}/;s/\x0//" <<< "${match#*:}") \ | ||
| sed -e "s@proc/self/fd/[0-9]*@${file_path}@g" | ||
} |
Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
@@ -84,8 +84,8 @@ termux_pkg_upgrade_version() { | |||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||
echo "INFO: package being updated to ${LATEST_VERSION}." | ||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||
sed \ | ||||||||||||||||||||||||||||||||||
-e "s/^\(TERMUX_PKG_VERSION=\)\(.*\)\$/\1\"${EPOCH}${LATEST_VERSION}\"/g" \ | ||||||||||||||||||||||||||||||||||
sed -r \ | ||||||||||||||||||||||||||||||||||
-e "s/^(TERMUX_PKG_VERSION=\(?)(.*)\$/\1\"${EPOCH}${LATEST_VERSION}\"/g" \ | ||||||||||||||||||||||||||||||||||
twaik marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||||||||||||||||||||||||||
-e "/TERMUX_PKG_REVISION=/d" \ | ||||||||||||||||||||||||||||||||||
-i "${TERMUX_PKG_BUILDER_DIR}/build.sh" | ||||||||||||||||||||||||||||||||||
Comment on lines
+87
to
90
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Probably I found it out, but it is a literal nightmare.
Suggested change
Or it you think perl is better perl -0777 -i -pe '
s/^TERMUX_PKG_REVISION=.*\n//mg;
s{(TERMUX_PKG_VERSION=\(?\s*)[^)\s]+}{
qq{$1"'"${EPOCH}${LATEST_VERSION}"'"}
}em
' "${TERMUX_PKG_BUILDER_DIR}/build.sh"
|
sed -r \ | |
-e "s/^(TERMUX_PKG_VERSION=\(?)(.*)\$/\1\"${EPOCH}${LATEST_VERSION}\"/g" \ | |
-e "/TERMUX_PKG_REVISION=/d" \ | |
-i "${TERMUX_PKG_BUILDER_DIR}/build.sh" | |
sed -z -r \ | |
-e "s/(TERMUX_PKG_VERSION=\(?[[:space:]]*)[^[:space:]\)]+/\1\"${EPOCH}${LATEST_VERSION}\"/" \ | |
-e "s/(^|\n)TERMUX_PKG_REVISION=[^\n]*(\n|$)/\1/" \ | |
-i "${TERMUX_PKG_BUILDER_DIR}/build.sh" |
Or it you think perl is better
perl -0777 -i -pe '
s/^TERMUX_PKG_REVISION=.*\n//mg;
s{(TERMUX_PKG_VERSION=\(?\s*)[^)\s]+}{
qq{$1"'"${EPOCH}${LATEST_VERSION}"'"}
}em
' "${TERMUX_PKG_BUILDER_DIR}/build.sh"
tests.sh
Not my best work but it did the job.
#!/bin/bash
temp_dir=$(mktemp -d) || { echo failed to create temp directory; exit 1; }
cd "${temp_dir}"
LATEST_VERSION=6.18
EPOCH01=
cat << EOF > build.sh.01
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(6.13
2017)
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.01
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=("6.18"
2017)
TERMUX_PKG_SHA256=(
EOF
EPOCH02=
cat << EOF > build.sh.02
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(6.13 2017)
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.02
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=("6.18" 2017)
TERMUX_PKG_SHA256=(
EOF
EPOCH03=
cat << EOF > build.sh.03
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
6.13
2017
)
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.03
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
"6.18"
2017
)
TERMUX_PKG_SHA256=(
EOF
EPOCH04=
cat << EOF > build.sh.04
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
6.13
2017
)
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.04
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
"6.18"
2017
)
TERMUX_PKG_SHA256=(
EOF
EPOCH05=
cat << EOF > build.sh.05
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
6.13 2017
)
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.05
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
"6.18" 2017
)
TERMUX_PKG_SHA256=(
EOF
EPOCH06=
cat << EOF > build.sh.06
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
6.13 2017)
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.06
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
"6.18" 2017)
TERMUX_PKG_SHA256=(
EOF
EPOCH07=
cat << EOF > build.sh.07
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
"6.13" 2017)
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.07
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
"6.18" 2017)
TERMUX_PKG_SHA256=(
EOF
EPOCH08=3:
cat << EOF > build.sh.08
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
3:6.13 2017)
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.08
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
"3:6.18" 2017)
TERMUX_PKG_SHA256=(
EOF
EPOCH09=3:
cat << EOF > build.sh.09
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
3:6.13 2017)
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.09
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
"3:6.18" 2017)
TERMUX_PKG_SHA256=(
EOF
EPOCH10=3:
cat << EOF > build.sh.10
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
3:6.13 2017
)
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.10
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
"3:6.18" 2017
)
TERMUX_PKG_SHA256=(
EOF
EPOCH11=3:
cat << EOF > build.sh.11
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
"3:6.13" 2017
)
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.11
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=(
"3:6.18" 2017
)
TERMUX_PKG_SHA256=(
EOF
EPOCH12=3:
cat << EOF > build.sh.12
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION="3:6.13"
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.12
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION="3:6.18"
TERMUX_PKG_SHA256=(
EOF
EPOCH13=3:
cat << EOF > build.sh.13
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION=3:6.13
TERMUX_PKG_REVISION=1
TERMUX_PKG_SHA256=(
EOF
cat << EOF > expected.build.sh.13
TERMUX_PKG_MAINTAINER="@termux"
TERMUX_PKG_VERSION="3:6.18"
TERMUX_PKG_SHA256=(
EOF
for i in build.sh.*; do
var="EPOCH${i##*.}"
EPOCH="${!var}"
false && \
perl -0777 -i -pe '
s/^TERMUX_PKG_REVISION=.*\n//mg;
s{(TERMUX_PKG_VERSION=\(?\s*)[^)\s]+}{
qq{$1"'"${EPOCH}${LATEST_VERSION}"'"}
}em
' "$i"
sed -z -r -i \
-e "s/(TERMUX_PKG_VERSION=\(?[[:space:]]*)[^[:space:]\)]+/\1\"${EPOCH}${LATEST_VERSION}\"/" \
-e "s/(^|\n)TERMUX_PKG_REVISION=[^\n]*(\n|$)/\1/" \
"$i"
if diff -Naur "$i" "expected.$i"; then
echo "test ${i##.*} passed"
else
echo "test ${i##.*} failed"
fi
done
cd - &>-
rm -rf "${temp_dir}"
As Apollo pointed out in the other PR.
I'd like to avoid adding perl
as an additional dependency to our build scripts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this case you can test the sed
solution. Or I can try it when I get home.
Also as Robert mentioned perl
is a dependency of at least autoconf autogen automake build-essential gobject-introspection gobject-introspection-bin git
which are extensively used in our buildscripts so I hardly believe it will not be present on build host.
#24518 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sooooo
I found all packages like manpages with
$ grep -nr TERMUX_PKG_VERSION= --include=build.sh . | grep -v -E -e '="' -e '=[0-9]+' -e '=\$'
./packages/maxima/build.sh:5:TERMUX_PKG_VERSION=()
./packages/ncurses/build.sh:15:TERMUX_PKG_VERSION=(6.5.20240831
./packages/glulxe/build.sh:5:TERMUX_PKG_VERSION=(0.6.1
./packages/libandroid-support/build.sh:4:TERMUX_PKG_VERSION=(29
./packages/softether-vpn/build.sh:5:TERMUX_PKG_VERSION=(5.02.5180)
./packages/unar/build.sh:5:TERMUX_PKG_VERSION=()
./packages/perl/build.sh:12:TERMUX_PKG_VERSION=(5.38.2
./packages/hilbish/build.sh:5:TERMUX_PKG_VERSION=(
./packages/manpages/build.sh:6:TERMUX_PKG_VERSION=(6.13
./packages/ldc/build.sh:5:TERMUX_PKG_VERSION=(1.30.0)
./sample/build.sh:9:TERMUX_PKG_VERSION=
./disabled-packages/mingw-w64-crt/build.sh:6:TERMUX_PKG_VERSION=(10.0.0)
./disabled-packages/mingw-w64-gcc-libs/build.sh:5:TERMUX_PKG_VERSION=(12.2.0)
./disabled-packages/libmusicbrainz/build.sh:5:TERMUX_PKG_VERSION=()
./x11-packages/tigervnc/build.sh:5:TERMUX_PKG_VERSION=(1.15.0
./root-packages/zmap/build.sh:5:TERMUX_PKG_VERSION=(1:4.3.2
So I checked ncurses
, glulxe
, libandroid-support
, softether-vpn
, perl
, hilbish
, ldc
, tigervnc
and zmap
and it works there as well so I assume it works fine. And version replacing on regular packages was not broken.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually 13 lines of code since it removes revision separately.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the headache I guess...
I'll look into it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure why by update-checksums
replaces all checksums in the checksums array so it is not the same case as in here. But there are a few things that can be refactored.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Twaik.
Can you get me done with one headache before adding another?
I'm looking at it. I'll hear more suggestions when I'm done with this one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Heh.