Implement multi threaded file processing #27

XniceCraft · 2023-12-07T13:51:15Z

No description provided.

Grimler91 · 2023-12-07T15:29:05Z

Very helpful, thanks for working on this!

I don't see a reason to convert the files from tabs to space indentation. I guess it is possible to change the clang-format config to match current style, and re-format with that instead?

If we want to reformat the files with a tool like clang-format, then that should preferably be done in a separate PR, to make review easier.

XniceCraft · 2023-12-07T22:58:06Z

My bad, sorry. I shouldn't format the code. I should done it at other PR.

The code style for arghandling.c is GNU, but elf-cleaner.cpp seems using different style. The closest one i could find is Mozilla with TabSize set to 2. Or i undo the formatting to make it easier?

XniceCraft · 2023-12-09T01:21:23Z

@Grimler91 I undo the formatting and keep the code

Grimler91 · 2023-12-09T10:06:45Z

@XniceCraft great, thanks, will do some tests!

I kept formatting in arghandling.c to make it as similar as possible to the source, but having different styles in different files is indeed confusing. Feel free to open another PR to convert them to the same style after this is merged (preferably GNU style).

Grimler91 · 2023-12-09T15:35:05Z

There seem to be some kind of memory issue here, if I run something like:

autoreconf -fi
mkdir -p build && cd build
../configure
make
mkdir -p tmp 
for i in $(seq 1 100); do 
    cp ../tests/curl-7.83.1-aarch64-original tmp/curl-7.83.1-aarch64-original.$i; 
done 
./termux-elf-cleaner --api-level 24 --jobs 4 tmp/curl-7.83.1-aarch64-original*

then I get:

termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.1'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.10'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.100'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.11'
open("tmp/curl-7.83.1-aarch64-original.130"): No such file or directory
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.12'
open("tmp/curl-7.83.1-aarch64-original.150"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.170"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.180"): No such file or directory
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.14'
open("tmp/curl-7.83.1-aarch64-original.190"): No such file or directory
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.16'
open("tmp/curl-7.83.1-aarch64-original.290"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.200"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.210"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.220"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.230"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.240"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.250"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.260"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.270"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.280"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.290"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.390"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.300"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.310"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.320"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.330"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.340"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.350"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.360"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.370"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.380"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.390"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.490"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.400"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.410"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.420"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.430"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.440"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.450"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.460"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.470"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.480"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.490"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.590"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.500"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.510"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.520"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.530"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.540"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.550"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.560"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.570"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.580"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.590"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.690"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.600"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.610"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.620"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.630"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.640"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.650"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.660"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.670"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.680"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.690"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.790"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.700"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.710"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.720"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.730"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.740"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.750"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.760"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.770"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.780"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.790"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.890"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.800"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.810"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.820"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.830"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.840"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.850"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.860"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.870"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.880"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.890"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.990"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.900"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.910"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.920"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.930"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.940"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.950"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.960"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.970"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.980"): No such file or directory
open("tmp/curl-7.83.1-aarch64-original.990"): No such file or directory

running termux-elf-cleaner in the same loop under valgrind -s I get:

==49598== Memcheck, a memory error detector
==49598== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==49598== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==49598== Command: ./termux-elf-cleaner --api-level 24 --jobs 4 tmp/curl-7.83.1-aarch64-original.1 tmp/curl-7.83.1-aarch64-original.10 tmp/curl-7.83.1-aarch64-original.100 tmp/curl-7.83.1-aarch64-original.11 tmp/curl-7.83.1-aarch64-original.12 tmp/curl-7.83.1-aarch64-original.13 tmp/curl-7.83.1-aarch64-original.14 tmp/curl-7.83.1-aarch64-original.15 tmp/curl-7.83.1-aarch64-original.16 tmp/curl-7.83.1-aarch64-original.17 tmp/curl-7.83.1-aarch64-original.18 tmp/curl-7.83.1-aarch64-original.19 tmp/curl-7.83.1-aarch64-original.2 tmp/curl-7.83.1-aarch64-original.20 tmp/curl-7.83.1-aarch64-original.21 tmp/curl-7.83.1-aarch64-original.22 tmp/curl-7.83.1-aarch64-original.23 tmp/curl-7.83.1-aarch64-original.24 tmp/curl-7.83.1-aarch64-original.25 tmp/curl-7.83.1-aarch64-original.26 tmp/curl-7.83.1-aarch64-original.27 tmp/curl-7.83.1-aarch64-original.28 tmp/curl-7.83.1-aarch64-original.29 tmp/curl-7.83.1-aarch64-original.3 tmp/curl-7.83.1-aarch64-original.30 tmp/curl-7.83.1-aarch64-original.31 tmp/curl-7.83.1-aarch64-original.32 tmp/curl-7.83.1-aarch64-original.33 tmp/curl-7.83.1-aarch64-original.34 tmp/curl-7.83.1-aarch64-original.35 tmp/curl-7.83.1-aarch64-original.36 tmp/curl-7.83.1-aarch64-original.37 tmp/curl-7.83.1-aarch64-original.38 tmp/curl-7.83.1-aarch64-original.39 tmp/curl-7.83.1-aarch64-original.4 tmp/curl-7.83.1-aarch64-original.40 tmp/curl-7.83.1-aarch64-original.41 tmp/curl-7.83.1-aarch64-original.42 tmp/curl-7.83.1-aarch64-original.43 tmp/curl-7.83.1-aarch64-original.44 tmp/curl-7.83.1-aarch64-original.45 tmp/curl-7.83.1-aarch64-original.46 tmp/curl-7.83.1-aarch64-original.47 tmp/curl-7.83.1-aarch64-original.48 tmp/curl-7.83.1-aarch64-original.49 tmp/curl-7.83.1-aarch64-original.5 tmp/curl-7.83.1-aarch64-original.50 tmp/curl-7.83.1-aarch64-original.51 tmp/curl-7.83.1-aarch64-original.52 tmp/curl-7.83.1-aarch64-original.53 tmp/curl-7.83.1-aarch64-original.54 tmp/curl-7.83.1-aarch64-original.55 tmp/curl-7.83.1-aarch64-original.56 tmp/curl-7.83.1-aarch64-original.57 tmp/curl-7.83.1-aarch64-original.58 tmp/curl-7.83.1-aarch64-original.59 tmp/curl-7.83.1-aarch64-original.6 tmp/curl-7.83.1-aarch64-original.60 tmp/curl-7.83.1-aarch64-original.61 tmp/curl-7.83.1-aarch64-original.62 tmp/curl-7.83.1-aarch64-original.63 tmp/curl-7.83.1-aarch64-original.64 tmp/curl-7.83.1-aarch64-original.65 tmp/curl-7.83.1-aarch64-original.66 tmp/curl-7.83.1-aarch64-original.67 tmp/curl-7.83.1-aarch64-original.68 tmp/curl-7.83.1-aarch64-original.69 tmp/curl-7.83.1-aarch64-original.7 tmp/curl-7.83.1-aarch64-original.70 tmp/curl-7.83.1-aarch64-original.71 tmp/curl-7.83.1-aarch64-original.72 tmp/curl-7.83.1-aarch64-original.73 tmp/curl-7.83.1-aarch64-original.74 tmp/curl-7.83.1-aarch64-original.75 tmp/curl-7.83.1-aarch64-original.76 tmp/curl-7.83.1-aarch64-original.77 tmp/curl-7.83.1-aarch64-original.78 tmp/curl-7.83.1-aarch64-original.79 tmp/curl-7.83.1-aarch64-original.8 tmp/curl-7.83.1-aarch64-original.80 tmp/curl-7.83.1-aarch64-original.81 tmp/curl-7.83.1-aarch64-original.82 tmp/curl-7.83.1-aarch64-original.83 tmp/curl-7.83.1-aarch64-original.84 tmp/curl-7.83.1-aarch64-original.85 tmp/curl-7.83.1-aarch64-original.86 tmp/curl-7.83.1-aarch64-original.87 tmp/curl-7.83.1-aarch64-original.88 tmp/curl-7.83.1-aarch64-original.89 tmp/curl-7.83.1-aarch64-original.9 tmp/curl-7.83.1-aarch64-original.90 tmp/curl-7.83.1-aarch64-original.91 tmp/curl-7.83.1-aarch64-original.92 tmp/curl-7.83.1-aarch64-original.93 tmp/curl-7.83.1-aarch64-original.94 tmp/curl-7.83.1-aarch64-original.95 tmp/curl-7.83.1-aarch64-original.96 tmp/curl-7.83.1-aarch64-original.97 tmp/curl-7.83.1-aarch64-original.98 tmp/curl-7.83.1-aarch64-original.99
==49598== 
==49598== Thread 4:
==49598== Syscall param openat(filename) points to unaddressable byte(s)
==49598==    at 0x4D31C70: open (open64.c:41)
==49598==    by 0x10AD5A: parse_file(char const*) (elf-cleaner.cpp:218)
==49598==    by 0x10B007: parse_file_handler(std::deque<char const*, std::allocator<char const*> >*) (elf-cleaner.cpp:325)
==49598==    by 0x4983942: execute_native_thread_routine (thread.cc:104)
==49598==    by 0x4CBB9EA: start_thread (pthread_create.c:444)
==49598==    by 0x4D3F653: clone (clone.S:100)
==49598==  Address 0x4e28d02 is 0 bytes after a block of size 34 alloc'd
==49598==    at 0x4841828: malloc (vg_replace_malloc.c:442)
==49598==    by 0x10B048: parse_file_handler(std::deque<char const*, std::allocator<char const*> >*) (elf-cleaner.cpp:317)
==49598==    by 0x4983942: execute_native_thread_routine (thread.cc:104)
==49598==    by 0x4CBB9EA: start_thread (pthread_create.c:444)
==49598==    by 0x4D3F653: clone (clone.S:100)
==49598== 
==49598== Thread 3:
==49598== Invalid read of size 1
==49598==    at 0x484A234: strlen (vg_replace_strmem.c:502)
==49598==    by 0x4C8F6E8: __printf_buffer (vfprintf-process-arg.c:435)
==49598==    by 0x4C8FCC9: __vfprintf_internal (vfprintf-internal.c:1538)
==49598==    by 0x4C852FE: printf (printf.c:33)
==49598==    by 0x10BC3D: bool process_elf<unsigned long, Elf64_Ehdr, Elf64_Shdr, elf64_phdr, Elf64_Dyn>(unsigned char*, unsigned long, char const*) (elf-cleaner.cpp:191)
==49598==    by 0x10AE00: parse_file(char const*) (elf-cleaner.cpp:286)
==49598==    by 0x10B007: parse_file_handler(std::deque<char const*, std::allocator<char const*> >*) (elf-cleaner.cpp:325)
==49598==    by 0x4983942: execute_native_thread_routine (thread.cc:104)
==49598==    by 0x4CBB9EA: start_thread (pthread_create.c:444)
==49598==    by 0x4D3F653: clone (clone.S:100)
==49598==  Address 0x4e28d73 is 0 bytes after a block of size 35 alloc'd
==49598==    at 0x4841828: malloc (vg_replace_malloc.c:442)
==49598==    by 0x10B048: parse_file_handler(std::deque<char const*, std::allocator<char const*> >*) (elf-cleaner.cpp:317)
==49598==    by 0x4983942: execute_native_thread_routine (thread.cc:104)
==49598==    by 0x4CBB9EA: start_thread (pthread_create.c:444)
==49598==    by 0x4D3F653: clone (clone.S:100)
==49598== 
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.10'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.1'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.100'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.11'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.12'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.13'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.14'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.15'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.16'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.17'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.19'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.18'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.2'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.20'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.21'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.22'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.23'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.24'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.25'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.27'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.26'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.28'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.29'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.3'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.30'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.31'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.32'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.33'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.34'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.35'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.36'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.38'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.37'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.39'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.41'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.4'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.40'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.42'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.43'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.44'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.45'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.46'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.47'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.48'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.49'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.5'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.50'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.51'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.52'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.53'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.54'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.55'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.56'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.57'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.58'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.59'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.6'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.60'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.61'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.62'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.63'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.64'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.65'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.66'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.67'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.68'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.69'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.7'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.70'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.71'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.72'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.73'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.74'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.75'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.76'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.77'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.78'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.79'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.8'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.80'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.81'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.82'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.83'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.84'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.85'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.86'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.87'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.88'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.89'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.9'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.90'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.91'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.92'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.94'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.93'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.95'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.96'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.97'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.98'
termux-elf-cleaner: Replacing unsupported DF_1_* flags 134217737 with 9 in 'tmp/curl-7.83.1-aarch64-original.99'
==49598== 
==49598== HEAP SUMMARY:
==49598==     in use at exit: 0 bytes in 0 blocks
==49598==   total heap usage: 114 allocs, 114 frees, 80,612 bytes allocated
==49598== 
==49598== All heap blocks were freed -- no leaks are possible
==49598== 
==49598== ERROR SUMMARY: 200 errors from 2 contexts (suppressed: 0 from 0)
==49598== 
==49598== 100 errors in context 1 of 2:
==49598== Invalid read of size 1
==49598==    at 0x484A234: strlen (vg_replace_strmem.c:502)
==49598==    by 0x4C8F6E8: __printf_buffer (vfprintf-process-arg.c:435)
==49598==    by 0x4C8FCC9: __vfprintf_internal (vfprintf-internal.c:1538)
==49598==    by 0x4C852FE: printf (printf.c:33)
==49598==    by 0x10BC3D: bool process_elf<unsigned long, Elf64_Ehdr, Elf64_Shdr, elf64_phdr, Elf64_Dyn>(unsigned char*, unsigned long, char const*) (elf-cleaner.cpp:191)
==49598==    by 0x10AE00: parse_file(char const*) (elf-cleaner.cpp:286)
==49598==    by 0x10B007: parse_file_handler(std::deque<char const*, std::allocator<char const*> >*) (elf-cleaner.cpp:325)
==49598==    by 0x4983942: execute_native_thread_routine (thread.cc:104)
==49598==    by 0x4CBB9EA: start_thread (pthread_create.c:444)
==49598==    by 0x4D3F653: clone (clone.S:100)
==49598==  Address 0x4e28d73 is 0 bytes after a block of size 35 alloc'd
==49598==    at 0x4841828: malloc (vg_replace_malloc.c:442)
==49598==    by 0x10B048: parse_file_handler(std::deque<char const*, std::allocator<char const*> >*) (elf-cleaner.cpp:317)
==49598==    by 0x4983942: execute_native_thread_routine (thread.cc:104)
==49598==    by 0x4CBB9EA: start_thread (pthread_create.c:444)
==49598==    by 0x4D3F653: clone (clone.S:100)
==49598== 
==49598== 
==49598== 100 errors in context 2 of 2:
==49598== Thread 4:
==49598== Syscall param openat(filename) points to unaddressable byte(s)
==49598==    at 0x4D31C70: open (open64.c:41)
==49598==    by 0x10AD5A: parse_file(char const*) (elf-cleaner.cpp:218)
==49598==    by 0x10B007: parse_file_handler(std::deque<char const*, std::allocator<char const*> >*) (elf-cleaner.cpp:325)
==49598==    by 0x4983942: execute_native_thread_routine (thread.cc:104)
==49598==    by 0x4CBB9EA: start_thread (pthread_create.c:444)
==49598==    by 0x4D3F653: clone (clone.S:100)
==49598==  Address 0x4e28d02 is 0 bytes after a block of size 34 alloc'd
==49598==    at 0x4841828: malloc (vg_replace_malloc.c:442)
==49598==    by 0x10B048: parse_file_handler(std::deque<char const*, std::allocator<char const*> >*) (elf-cleaner.cpp:317)
==49598==    by 0x4983942: execute_native_thread_routine (thread.cc:104)
==49598==    by 0x4CBB9EA: start_thread (pthread_create.c:444)
==49598==    by 0x4D3F653: clone (clone.S:100)
==49598== 
==49598== ERROR SUMMARY: 200 errors from 2 contexts (suppressed: 0 from 0)

XniceCraft · 2023-12-09T18:29:55Z

I found a solution. Instead of popping the item from deque (which is we need to copy the item to temporary store it), we create a variable that store the index that the thread need to access

Instead of copying the front item then temporary store it, we create a variable that store where the thread need to access (according to std::deque docs, accessing by index require constant time O(1))"

Needed on some systems to get pthread functions.

Grimler91 · 2023-12-15T20:04:20Z

Thanks a lot, works in my tests now, roughly 10 times faster than before :)

XniceCraft mentioned this pull request Dec 7, 2023

Run termux-elf-cleaner on multiple threads when there's many input files #19

Closed

XniceCraft force-pushed the multithreading branch from 853a6da to 52f72be Compare December 9, 2023 01:19

XniceCraft force-pushed the multithreading branch from d9a61a2 to f63b9b3 Compare December 9, 2023 18:47

Grimler91 force-pushed the multithreading branch from 19a8c40 to 97ac438 Compare December 15, 2023 20:01

XniceCraft and others added 4 commits December 15, 2023 21:02

Implement multithreading

4384137

Use index instead of copying the item

04755b3

Instead of copying the front item then temporary store it, we create a variable that store where the thread need to access (according to std::deque docs, accessing by index require constant time O(1))"

Pass pos variable by reference

5e679bc

Makefile: compile with -pthread

2d93754

Needed on some systems to get pthread functions.

Grimler91 force-pushed the multithreading branch from 97ac438 to 2d93754 Compare December 15, 2023 20:03

Grimler91 merged commit 2d93754 into termux:master Dec 15, 2023

XniceCraft deleted the multithreading branch December 16, 2023 10:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement multi threaded file processing #27

Implement multi threaded file processing #27

Uh oh!

XniceCraft commented Dec 7, 2023

Uh oh!

Grimler91 commented Dec 7, 2023 •

edited

Loading

Uh oh!

XniceCraft commented Dec 7, 2023

Uh oh!

XniceCraft commented Dec 9, 2023 •

edited

Loading

Uh oh!

Grimler91 commented Dec 9, 2023

Uh oh!

Grimler91 commented Dec 9, 2023

Uh oh!

XniceCraft commented Dec 9, 2023

Uh oh!

Grimler91 commented Dec 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Implement multi threaded file processing #27

Implement multi threaded file processing #27

Uh oh!

Conversation

XniceCraft commented Dec 7, 2023

Uh oh!

Grimler91 commented Dec 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XniceCraft commented Dec 7, 2023

Uh oh!

XniceCraft commented Dec 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Grimler91 commented Dec 9, 2023

Uh oh!

Grimler91 commented Dec 9, 2023

Uh oh!

XniceCraft commented Dec 9, 2023

Uh oh!

Grimler91 commented Dec 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Grimler91 commented Dec 7, 2023 •

edited

Loading

XniceCraft commented Dec 9, 2023 •

edited

Loading