Run termux-elf-cleaner on multiple threads when there's many input files #19
Conversation
When we want to loop over many files we can then save quite some time by running on parallel inside termux-elf-cleaner, instead of outside in the shell. Basically, changing from: parallel termux-elf-cleaner foo bar baz foo bar baz ... and instead handle the parallelisation inside termux-elf-cleaner so we can run just: termux-elf-cleaner foo bar baz foo bar baz ... decreases the amount of overhead and saves time compared to running termux-elf-cleaner in a bash for loop, or with the help of tools like parallel.
Check so no errors are raised.
| for (int i = 0; i<num_threads; i++) { | ||
| if (data[i].argc == 0) | ||
| break; | ||
| pthread_create(&thread[i], NULL, &file_parse_loop, &data[i]); |
There was a problem hiding this comment.
I guess this is too ugly. We probably should be relying on C++11's std::thread implementation instead of creating the thread ourselves using the lower level pthread library. Also instead of going this way and dividing all the input files equally among the available CPUs, it would be much better if we:
- First spawn a number of threads (either as specified by the user, or upon number of CPUs).
- Have a sort of atomic object which the called thread will call
std::atomic::notify_all()just before completing - The main thread will be waiting for threads to finish using
std::atomic::wait, and spawn a new thread as soon as one finishes.
This should also increase the performance improvements in your benchmarks as not all CPUs might perform in the same fashion I guess. And also will definitely improve in conditions where the user input contains a lot of small binaries with a large binary.
I don't know whether that's the optimal approach, but I've used this in one of my personal projects'test runner, and it works pretty well. Although I haven't battle tested it. In case anyone else has a better approach feel free to comment
There was a problem hiding this comment.
I made the multithreaded implementation with std::thread.
Here's the repo: https://github.com/XniceCraft/termux-elf-cleaner/tree/multithreading
There was a problem hiding this comment.
@XniceCraft that is great! Would you like to open a pull request to replace this one?
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
|
Closing in favour of #27 |
Here's a comparison of old termux-elf-cleaner (
1.10), new one without these changes (2.1.1), and new one with these changes (threads) on 100, 1000 and 10000 files:which shows that processing time is decreased to roughly 1/3 with the help of running on multiple threads (at least on a 8 core machine)