Support HA for shards #100
Description
Hello
Currently i'm trying to setup shards with high availability ie running the multiple replicas of each shard. I'm not sure whether this is already possible. Could you please let me know whether this can be achieved or not?
I did try to deploy 2 statefulsets with same config/coordinator, but the targets distribution was not the same in both sts pods.
If it's not supported, would love to help with the implementation (with some guidance to start with).
Example:
Let's assume for 2 sts with 2 shards -
I would expect each shard/pods (prom-replica-0-0 & prom-replica-0-1) to scrape the same targets & so on for other shards.
sts: prom-replica-0
pod: prom-replica-0-$(SHARD)
----config----
global:
external_labels:
prometheus: prom-kvass-$(SHARD)
prometheus_replica: $(POD_NAME)
scrape_configs:
- job_name: metrics
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 20s
metrics_path: /metrics
scheme: http
authorization:
type: Bearer
credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
proxy_url: http://127.0.0.1:8008
follow_redirects: true
relabel_configs:
- separator: ;
regex: __invalid_label_(.+)
replacement: $1
action: labelmap
static_configs:
- targets:
- "TARGET_0"
labels:
__address__: "TARGET_0"
__metrics_path__: /metrics
__param__hash: "TARGET_0_HASH"
__param__jobName: metrics
__param__scheme: http
__scheme__: http
....sts: prom-replica-1
pod: prom-replica-1-$(SHARD)
----config----
global:
external_labels:
prometheus: prom-kvass-$(SHARD)
prometheus_replica: $(POD_NAME)
scrape_configs:
- job_name: metrics
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 20s
metrics_path: /metrics
scheme: http
authorization:
type: Bearer
credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
proxy_url: http://127.0.0.1:8008
follow_redirects: true
relabel_configs:
- separator: ;
regex: __invalid_label_(.+)
replacement: $1
action: labelmap
static_configs:
- targets:
- "TARGET_0"
labels:
__address__: "TARGET_0"
__metrics_path__: /metrics
__param__hash: "TARGET_0_HASH"
__param__jobName: metrics
__param__scheme: http
__scheme__: httpThanks,
Jet