Questions about Perforator #7

zamazan4ik · 2025-01-30T19:28:06Z

zamazan4ik
Jan 30, 2025

Hi!

Firstly, thank you so much for open-sourcing Perforator! Finally, we have an open-source solution for performing Continuous sPGO for different languages.

I have a couple of questions about Perforator. Most of them are about sPGO-related functionality but some of them are about the whole system. I didn't create a dozen of various issues because I don't want to burden maintainers with that ;) If for any question the answer is "no", it would be nice if you can describe a bit further like "why we don't do that" or "no, we don't do it yet but we plan to support it in the future".

sPGO-related questions:

Do you plan to support sPGO for other compilers like GCC (or any other)? As far as I understand it isn't implemented since internally Yandex doesn't use GCC. If there are not such plans, will you accept contribution with adding GCC sPGO? Maybe some converter from Clang-compatible profile format into GCC-compatible format would be a good idea (I don't know is it possible or not) or support generating GCC-compatible profile directly. Maybe some AutoFDO code could be reused in this case.
Right now, only x86-64 Intel CPUs are supported for PGO - it's a limitation from LBR support. Do you plan to support other CPU vendors and hardware architectures. E.g. AMD supports a similar technology - BRS, since Zen3. Other hardware architectures (like aarch64, e2k, PowerPC) also have similar technologies. I am interested about your plans/thoughts for supporting them in the future or accepting contribution from community. Personally, I am mostly interested in AMD Zen CPUs and ARM things. I see your plans about AArch64 support here but I guess it's not the same as AArch64 support for sPGO use-cases - just want to clarify it with you.
It would be nice to get support for other operating systems like *BSD (still used for server fleets), Windows (for running Windows-only apps) and macOS. Windows and macOS are especially interesting cases since we can try to use Perforator agents to perform sPGO optimization for client applications. I completely understand that it could be impossible since the agent heavily relies on eBPF infrastructure. But it could an interesting use case for the community.
Are there any minimal required Clang (or LLVM version) for using sPGO with Perforator? I am asking not only for Clang since we have a bunch of LLVM-based compilers in the ecosystem like Rustc/LDC/Flang.
In the toolchain requirements, I see lld requirement. Is it a strict requirement or you just didn't test sPGO with Perforator for other linkers like gold or mold? I am interested in all linkers but especially in Mold.
Is it possible to use Perforator-generated PGO profile for using Post-Link Optimizers (PLO) like BOLT, Propeller or Intel TLO? If yes, it would be nice to see instructions in the documentation. I think the most interesting for the community will be LLVM BOLT since it's the most widely used PLO tool nowadays. If you already have experience with applying PLO tools with Perforator with your fleet - it would be nice to hear your story!
Did you perform tests for -fsample-profile-use-profi Clang flag for your sPGO use cases? If yes, does it make a difference with Perforator-generated sPGO profiles?
What is the recommended sampling frequency for doing sPGO from your experience? If you performed sPGO tests with different frequencies, it would be nice to see the dependency on sPGO efficiency on the sampling frequency at least in your use cases.
Regarding language support. It would be nice to see Fortran support for HPC use cases. Support for C# also would be nice to see since nowadays many backend services are written in C#. I am not an expert in C# so I'll leave here some links about PGO in C#: one, two.
Am I right that Perforator-generated sPGO profiles are compatible with llvm-profdata? If yes, is there any requirement regarding minimally required llvm-profdata version?

General questions:

Are there recommendations regarding Perforator sizing? At least something from your experience. E.g. "We have a fleet of N servers. On each, we collect profiles with M frequency. For this setup, we have ClickHouse cluster of K size, PostgreSQL cluster of L size, S3 storage for V Tib, etc.". It would be helpful to estimate CAPEX for such a system in practice. If you have examples for different environments like DEV, PROD - it would be nice to get multiple data points.
What is the recommended monitoring and alerting approach for the system (or each part of it)? In Helm charts I see some Prometheus mentions but I cannot find easily the provided metrics. Are there specific metrics that are the most important to check like /health endpoints? The same goes about alerting. If you already have some useful in practice set of alerting rules - it would be nice to see them in the documentation at least.
Does Perforator support zero-downtime upgrade? Zero-downtime downgrade (if something goes wrong during the upgrade)?
I see that there is some kind of UI in Perforator. What is the authentication/authorization approach is used in it? Is there an integration with Active Directory (AD) or any other similar alternative? If not, what is the recommended way to secure the app?
Is there any built-in audit functionality in Perforator. E.g. for tracking performed queries by its users and sending them into a company-centralized SIEM system?

Thank you for your answers.

P.S. Search over the documentation works strangely, if works at all, for the English version of the documentation. E.g. it can find nothing for the "PGO" search query, but you definitely have this word in the docs.

Answered by BigRedEye

Feb 3, 2025

I have tried to answer all of your questions. They should really be part of documentation😃

sPGO-related questions:

Do you plan to support sPGO for other compilers like GCC (or any other)? As far as I understand it isn't implemented since internally Yandex doesn't use GCC. If there are not such plans, will you accept contribution with adding GCC sPGO? Maybe some converter from Clang-compatible profile format into GCC-compatible format would be a good idea (I don't know is it possible or not) or support generating GCC-compatible profile directly. Maybe some AutoFDO code could be reused in this case.

Yeah, we do not use GCC extensively, so we have not tested this. It looks like that is j…

View full answer

BigRedEye · 2025-01-31T08:32:57Z

BigRedEye
Jan 31, 2025
Maintainer

Hi! Great to see you here:) These are excellent, deep questions. We'll need some time to prepare a detailed answer, and we'll get back to you soon.

0 replies

aaupov · 2025-02-02T08:54:42Z

aaupov
Feb 2, 2025

Regarding producing data for BOLT: if you collect LBR stacks and store them, it's trivial to produce the profile suitable for BOLT. Please take a look at pre-aggregated profile format described here: https://github.com/llvm/llvm-project/blob/fea18afeed39fe4435d67eee1834f0f34b23013d/bolt/include/bolt/Profile/DataAggregator.h#L372-L415
It's recommended to pass pre-aggregated profile to perf2bolt and convert it to either symbolized fdata or rich YAML format for the actual use in BOLT:

perf2bolt <sampled_binary> -p <pre-agg profile> --pa -o <fdata> -w <yaml> # convert pre-aggregated to fdata and yaml profiles
<store profiles>
llvm-bolt <pre-bolt binary> -data <fdata or yaml> ... # optimize the same or a different binary, need -infer-stale-profile and YAML profile for the latter case

0 replies

BigRedEye · 2025-02-03T13:43:14Z

BigRedEye
Feb 3, 2025
Maintainer

I have tried to answer all of your questions. They should really be part of documentation😃

sPGO-related questions:

Do you plan to support sPGO for other compilers like GCC (or any other)? As far as I understand it isn't implemented since internally Yandex doesn't use GCC. If there are not such plans, will you accept contribution with adding GCC sPGO? Maybe some converter from Clang-compatible profile format into GCC-compatible format would be a good idea (I don't know is it possible or not) or support generating GCC-compatible profile directly. Maybe some AutoFDO code could be reused in this case.

Yeah, we do not use GCC extensively, so we have not tested this. It looks like that is just a matter of yet another profile converting routine, and we will be happy to accept potential contributions.

Right now, only x86-64 Intel CPUs are supported for PGO - it's a limitation from LBR support. Do you plan to support other CPU vendors and hardware architectures. E.g. AMD supports a similar technology - BRS, since Zen3. Other hardware architectures (like aarch64, e2k, PowerPC) also have similar technologies. I am interested about your plans/thoughts for supporting them in the future or accepting contribution from community. Personally, I am mostly interested in AMD Zen CPUs and ARM things. I see your plans about AArch64 support here but I guess it's not the same as AArch64 support for sPGO use-cases - just want to clarify it with you.

Currently we support LBR on specific CPU model if and only if Linux kernel supports it, the documentation is misleading there. There is no CPU vendor-specific code inside Perforator. LBR should work on modern AMD models on Linux 6.1+ after torvalds/linux@703fb76 and ARM64 after https://lwn.net/Articles/951316/.

It would be nice to get support for other operating systems like *BSD (still used for server fleets), Windows (for running Windows-only apps) and macOS. Windows and macOS are especially interesting cases since we can try to use Perforator agents to perform sPGO optimization for client applications. I completely understand that it could be impossible since the agent heavily relies on eBPF infrastructure. But it could an interesting use case for the community.

I think that this is not in our roadmap for now. While differences between executable/profile formats can be solved, our agent heavily relies on eBPF, making it not portable. Maybe we can provide agents for different operating systems, but I'm afraid that without the need and help from the community this will not happen soon.

Are there any minimal required Clang (or LLVM version) for using sPGO with Perforator? I am asking not only for Clang since we have a bunch of LLVM-based compilers in the ecosystem like Rustc/LDC/Flang.

We have successfully used sPGO using LLVM 16 and LLVM 18. Previous versions will likely work, but we don't have much evidence.

In the toolchain requirements, I see lld requirement. Is it a strict requirement or you just didn't test sPGO with Perforator for other linkers like gold or mold? I am interested in all linkers but especially in Mold.

Again, you are right: we didn't test other linkers. They will probably work, but this requires some validation.

Is it possible to use Perforator-generated PGO profile for using Post-Link Optimizers (PLO) like BOLT, Propeller or Intel TLO? If yes, it would be nice to see instructions in the documentation. I think the most interesting for the community will be LLVM BOLT since it's the most widely used PLO tool nowadays. If you already have experience with applying PLO tools with Perforator with your fleet - it would be nice to hear your story!

Yes. We are testing BOLT now and are already running some BOLTed binaries. However, things didn't go as smoothly with BOLT as with AutoFDO, so we have not mentioned it yet. We are investigating issues we discovered and hopefully soon will provide documentation on complete BOLT integration.

Did you perform tests for -fsample-profile-use-profi Clang flag for your sPGO use cases? If yes, does it make a difference with Perforator-generated sPGO profiles?

No, for now we use only -fprofile-sample-use with -fdebug-info-for-profiling. Will check -fsample-profile-use-profi, thanks!

What is the recommended sampling frequency for doing sPGO from your experience? If you performed sPGO tests with different frequencies, it would be nice to see the dependency on sPGO efficiency on the sampling frequency at least in your use cases.

I don't have data available now, but we have seen a few percent improvement in performance when aggregating a lot of sPGO profiles (thousands of atomic profiles sampling 99Hz per CPU core for one minute).

Regarding language support. It would be nice to see Fortran support for HPC use cases. Support for C# also would be nice to see since nowadays many backend services are written in C#. I am not an expert in C# so I'll leave here some links about PGO in C#: one, two.

I suspect that Fortran should already work. I've compiled a simple executable via gfortran and it seems to contain correct .eh_frame section. C# support should be discussed, we do not have strong opinion here yet.

Am I right that Perforator-generated sPGO profiles are compatible with llvm-profdata? If yes, is there any requirement regarding minimally required llvm-profdata version?

Yes, the sPGO profiles are encoded in the LLVM's extbinary format and are generated via AutoFDO's create_llvm_prof. We didn't check exact llvm-profdata versions though.

Maybe @itrofimow can clarify some of my answers regarding sPGO and the LLVM toolchain we use.

General questions:

Are there recommendations regarding Perforator sizing? At least something from your experience. E.g. "We have a fleet of N servers. On each, we collect profiles with M frequency. For this setup, we have ClickHouse cluster of K size, PostgreSQL cluster of L size, S3 storage for V Tib, etc.". It would be helpful to estimate CAPEX for such a system in practice. If you have examples for different environments like DEV, PROD - it would be nice to get multiple data points.

I'm unable to publish exact numbers publicly, but:

We are running Perforator on tens to hundreds of thousands of nodes.
On each node we collect samples at a frequency of 99Hz per single CPU core.
Our ClickHouse cluster is not sharded yet, with replicas of 32CPU / 128GiB RAM / 6TiB SSD. However, the cluster is heavily underutilized now. I think that 8CPU / 32GiB RAM / 500GiB SSD will be enough for a long time.
PostgreSQL cluster is small, there is no need to scale it. The largest table is per-build-id info, a few million rows.
S3 usage is hundreds of terabytes. But this value depends on TTL settings, profile sampling rate, number of unique binaries, usage of advanced profile features like A/B tests, the complexity of profiled applications and so on.

What is the recommended monitoring and alerting approach for the system (or each part of it)? In Helm charts I see some Prometheus mentions but I cannot find easily the provided metrics. Are there specific metrics that are the most important to check like /health endpoints? The same goes about alerting. If you already have some useful in practice set of alerting rules - it would be nice to see them in the documentation at least.

Each component provides Prometheus metrics and structured logging, and user-facing components export OpenTelemetry traces. We are going to publish extensive documentation on operating Perforator cluster, including monitoring, logging, tracing and common debugging scenarios. We also think about providing Grafana panels based on our experience inside Yandex.

Does Perforator support zero-downtime upgrade? Zero-downtime downgrade (if something goes wrong during the upgrade)?

Our k8s integration is not as mature as the core of the system, because of differences between internal and open-source environments. Internally we deploy each component of the system without downtime. There can be some problems with Helm charts, but we will fix them.

What is the authentication/authorization approach is used in it?

Is there any built-in audit functionality in Perforator.

Not yet for both questions for now. Internally we use custom authentication provider, so it is not available in the open-source version. We have decided to wait for the community input and integrate with the most-requested authentication systems. I think it will be great if you can share some details in #8 (e.g. regarding LDAP vs SAML vs OIDC choice).

Yes, search over the documentation does not work. Will fix it.

1 reply

zamazan4ik Feb 3, 2025
Author

Thank you a lot for the answers!

itrofimow · 2025-02-06T14:27:21Z

itrofimow
Feb 6, 2025
Maintainer

Hi @zamazan4ik! I've read through a lot of your posts when putting sPGO together in Perforator, thanks for sharing such a valuable information, much appreciated.

To add to what @BigRedEye said:

Do you plan to support sPGO for other compilers like GCC (or any other)?

Perforator doesn't really contain much toolchain-specific code w.r.t. sPGO, all Perforator does is aggregating branch records and passing this aggregation to AutoFDO to do its magic. AFAIK AutoFDO supports generating gcc-specific sPGO profiles via create_gcov tool, but we haven't tested it.
The bottom line is that Perforator might support whatever AutoFDO (or other tools, if any) supports, but that would likely require contributions from the community.

Right now, only x86-64 Intel CPUs are supported for PGO - it's a limitation from LBR support. Do you plan to support other CPU vendors and hardware architectures. E.g. AMD supports a similar technology - BRS, since Zen3.

As Sergey said, we rely on the Kernel here, specifically on the bpf_read_branch_records function. As long as it is implemented and produces meaningful results, sPGO-profiles generation might be possible. IIRC it should work for Zen3 BRS since 5.17 Kernel, but don't quote me on that. Personally I wouldn't call the documentation misleading here, since we've tried to document things we for sure know work, but it definitely is incomplete.

In the toolchain requirements, I see lld requirement. Is it a strict requirement or you just didn't test sPGO with Perforator for other linkers like gold or mold? I am interested in all linkers but especially in Mold.

We didn't test other linkers, but the answer is basically the same as for GCC: as long as AutoFDO supports generating profiles in a format the linker of interest understands -- it might work.

Is it possible to use Perforator-generated PGO profile for using Post-Link Optimizers (PLO) like BOLT, Propeller or Intel TLO?

Generating profiles for Propeller should be doable, since AutoFDO does support it, however using Propeller itself required llvm19+, which is not yet feasible for us to test, unfortunately.
I can't say much about Intel TLO, but we have already implemented generating rich YAML profiles for BOLT, and we have also managed to achieve measurable improvements with BOLT atop of LTO + sPGO, but only when BOLT-ing in a two-phase manner, which is not the most interesting use-case for us.
Me and @aaupov had a little discussion about BOLT-ing in a continuous manner llvm/llvm-project#111920, but the experiments I've conducted so far are yet to produce any improvements for our workloads. There are a lot of moving parts here (LTO, PGO, BOLT and variety of its flags) and I very well might be missing something, but for now we can't say that BOLT works for us the way we want it to, so we didn't document it.
Also, Perforator fails to correctly symbolize BOLT-ed binaries due to such binaries having multiple executable sections, which breaks some assumptions we rely on, so I'm not even sure we run BOLT-ed binaries anywhere. @BigRedEye please correct me if my information is outdated.

What is the recommended sampling frequency for doing sPGO from your experience?

I don't think the frequency matters here, but rather the amount of branches which goes into resulting sPGO-profile. Why do you think the frequency could be important?

1 reply

zamazan4ik Feb 11, 2025
Author

Hi!

Thanks a lot for the answers! Sorry for the so late reply from my side - I somehow missed the notification. My bad.

AFAIK AutoFDO supports generating gcc-specific sPGO profiles via create_gcov tool, but we haven't tested it.

Yep, AutoFDO supports it. The only possible caveat is that AutoFDO devs plan to drop GCC support soon. I hope GCC devs will mitigate it somehow.

I don't think the frequency matters here, but rather the amount of branches which goes into resulting sPGO-profile. Why do you think the frequency could be important?

My assumption is if the frequency is low, we can overlook execution of very small functions between two samples. If we miss them, they won't be optimized. That's why in my mind, increasing the frequency can improve the sPGO quality overall.

Questions about Perforator #7

Uh oh!

zamazan4ik Jan 30, 2025

sPGO-related questions:

Replies: 4 comments · 2 replies

Uh oh!

Uh oh!

BigRedEye Jan 31, 2025 Maintainer

Uh oh!

Uh oh!

aaupov Feb 2, 2025

Uh oh!

Uh oh!

BigRedEye Feb 3, 2025 Maintainer

sPGO-related questions:

General questions:

Uh oh!

zamazan4ik Feb 3, 2025 Author

Uh oh!

itrofimow Feb 6, 2025 Maintainer

Uh oh!

zamazan4ik Feb 11, 2025 Author

zamazan4ik
Jan 30, 2025

Replies: 4 comments 2 replies

BigRedEye
Jan 31, 2025
Maintainer

aaupov
Feb 2, 2025

BigRedEye
Feb 3, 2025
Maintainer

zamazan4ik Feb 3, 2025
Author

itrofimow
Feb 6, 2025
Maintainer

zamazan4ik Feb 11, 2025
Author