What is preventing an IdP from inserting IDs as path parameters into the endpoints it specifies as a response to the config endpoint and using these IDs to match the accounts and client_metadata requests?
Suppose the response to the config request is as follows:

{
  "accounts_endpoint": "/random_id/accounts",
  "client_metadata_endpoint": "/random_id/metadata",
  "id_assertion_endpoint": "/assertion",
  "login_URL":"/login"
}

This leads to a credentialed request to "/random_id/accounts" and a request with the RP identifier to "/random_id/metadata" without any user interaction in Chromium as long as the IdP returns exactly one account.
An IdP can then use the "random_id" that is consistent between the two requests to combine them and track the user without permission.

I hope I'm wrong and have missed something obvious as FedCM relies heavily on separating these requests and I can't think of an obvious way to fix this.