The benchmarking should be combined with regression testing of performance, which should run in CI. Since such tests take a lot of compute to be meaningful, they should be triggered manually and have dedicated runners. I will try to set up some infrastructure at appliedAI for this.

The results of the benchmarking should be included in the documentation pages, and running the benchmarking should be necessary for release.

This issue will require quite some work, so it will probably be one of the last things to be done before the stable release.