There's some code duplication in these loops for setting subnet-based metrics. To improve maintainability, you could extract this logic into a helper function. This function could take the map of peers, the metrics gauge, the subnet count, and an optional subnet label formatter as arguments.

For example:

function setSubnetMetrics(
  peersBySubnetByBoundary: Map<ForkBoundaryLabel, number[]>,
  metricsGauge: {set: (labels: {boundary: ForkBoundaryLabel; subnet: string | number}, value: number) => void},
  subnetCount: number,
  subnetLabelFn: (subnet: number) => string | number = (s) => s
) {
  for (const [boundary, peersBySubnet] of peersBySubnetByBoundary.map) {
    for (let subnet = 0; subnet < subnetCount; subnet++) {
      metricsGauge.set(
        {boundary, subnet: subnetLabelFn(subnet)},
        peersBySubnet[subnet] ?? 0
      );
    }
  }
}

Then you could replace the loops with calls to this helper:

setSubnetMetrics(
  peersByBeaconAttSubnetByBoundary,
  metricsGossip.peersByBeaconAttestationSubnet,
  ATTESTATION_SUBNET_COUNT,
  attSubnetLabel
);

setSubnetMetrics(
  peersByBeaconSyncSubnetByBoundary,
  metricsGossip.peersBySyncCommitteeSubnet,
  SYNC_COMMITTEE_SUBNET_COUNT
);

setSubnetMetrics(
  peersByDataColumnSubnetByBoundary,
  metricsGossip.peersByDataColumnSubnet,
  NUMBER_OF_COLUMNS
);

this metric can be pretty expensive in testnets if BPOs are close to each other, eg. if we have fulu + 5 BPOs this results in 768 metric entries, not a blocker but just need to keep that in mind

to add more on that, if someone come up with a config like BPO at epoch n, n+1, n+2, n+3, n+4 then I think we have bigger issues to deal with than the metric

sample config from fusaka-devnet-5

BLOB_SCHEDULE:
  - EPOCH: 512
    MAX_BLOBS_PER_BLOCK: 15
  - EPOCH: 768
    MAX_BLOBS_PER_BLOCK: 21
  - EPOCH: 1024
    MAX_BLOBS_PER_BLOCK: 33
  - EPOCH: 1280
    MAX_BLOBS_PER_BLOCK: 48
  - EPOCH: 1536
    MAX_BLOBS_PER_BLOCK: 72

we only subscribe to +- 2 epochs at fork boundary FORK_EPOCH_LOOKAHEAD = 2 so most of the time we only get metric for 1 fork boundary

but the node would still serve them via /metrics until next restart and based on prometheus retention it would be still stored there

I wonder if we even need to fork boundary label or can just assess the metrics based on time, you would probably see a drop at each fork boundary and then it should go up again

I think we should only do that if there is a clear performance issue
seeing different peer counts at different fork boundary give us confidence on the implementation of lodestar nodes, and help us investigate issues just in case
we did not even get through any fork boundaries on mainnet, can revisit later if needed

I tend to agree with @nflaig that we will just see the boundary in the metric dislocation. the forks will never overlap so we will never have two lines at the same time needing to be differentiated by different labels. they will always be mutually exclusive and not sure its worth all the extra buckets/data-points for the label that is not "technically differentiating anything" because there would never be boundary overlap. There will just be a metric dislocation. I guess I also see what @twoeths is saying, that if the metric stays the same across a boundary and there is no color change on the dashboard we will not see the change (if there should be one) but not sure that is compelling enough to offset the hundreds of "0" data points we will collect to support the second label

I suppose I'm fine with either but just wanted to bring that up. What do you think @wemeetagain ?

is there a cheaper way to get the information we're looking for? Imo the primary piece of information / questions we're answering are:

• how many subnets have 0 mesh peers?
• what is the distribution of mesh peers for the subnets?

I don't think we really need to know that eg subnet 13 has 3 mesh peers. Its not actionable.

sorry I did not notice we have a concern before merging the PR
imo the performance concern is common for attnets/syncnets as well, so we should resolve all if someone can prove we have a performance issue with it

I don't think we really need to know that eg subnet 13 has 3 mesh peers. Its not actionable.

if we miss data column of subnet 13 of a block, block is not imported and it's not easy to find which missing DataColumnSidecars just looking into the gossip log. Then this metric should help

also we already have discovery metrics, for example lodestar_discovery_custody_groups_to_connect
these metrics are addition to that metrics when we need investigation

created this issue to track #8454