Closed
Description
What happened?
I have followed the configurations from the documentation: cri-o/tutorials/metrics.md.
Everything seems fine and I can see that the metrics on localhost is exposed on the default port:
$ sudo netstat -an | grep 9090
tcp 0 0 127.0.0.1:9090 0.0.0.0:* LISTENThe metrics exporter can not connect with any node on default metric port because the node is only listening on localhost.
Sample of logs:
level=info msg="Getting cluster configuration"
level=info msg="Creating Kubernetes client"
level=info msg="Retrieving nodes"
level=info msg="Registering handler /linuxvmmaster01 (for 10.240.0.13)"
level=info msg="Registering handler /linuxvmmaster02 (for 10.240.0.14)"
level=info msg="Registering handler /linuxvmmaster03 (for 10.240.0.15)"
level=info msg="Registering handler /linuxvmworker01 (for 10.240.0.16)"
level=info msg="Registering handler /linuxvmworker02 (for 10.240.0.17)"
level=info msg="Registering handler /linuxvmworker03 (for 10.240.0.18)"
level=info msg="Wrote scrape configs to configMap cri-o-metrics-exporter"
level=info msg="Serving HTTP on :8080"
level=error msg="Unable to retrieve metrics from http://10.240.0.15:9090/metrics: Get \"http://10.240.0.15:9090/metrics\": dial tcp 10.240.0.15:9090: connect: connection refused"
level=error msg="Unable to retrieve metrics from http://10.240.0.16:9090/metrics: Get \"http://10.240.0.16:9090/metrics\": dial tcp 10.240.0.16:9090: connect: connection refused"
level=error msg="Unable to retrieve metrics from http://10.240.0.18:9090/metrics: Get \"http://10.240.0.18:9090/metrics\": dial tcp 10.240.0.18:9090: connect: connection refused"
level=error msg="Unable to retrieve metrics from http://10.240.0.13:9090/metrics: Get \"http://10.240.0.13:9090/metrics\": dial tcp 10.240.0.13:9090: connect: connection refused"
level=error msg="Unable to retrieve metrics from http://10.240.0.14:9090/metrics: Get \"http://10.240.0.14:9090/metrics\": dial tcp 10.240.0.14:9090: connect: connection refused"
level=error msg="Unable to retrieve metrics from http://10.240.0.17:9090/metrics: Get \"http://10.240.0.17:9090/metrics\": dial tcp 10.240.0.17:9090: connect: connection refused"What did you expect to happen?
I would expect CRI-O socket to be exposing port 9090 to all IPs. Sample:
$ sudo netstat -an | grep LISTEN
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTENHow can we reproduce it (as minimally and precisely as possible)?
Deploy CRI-O metrics server and check logs of the cri-o metrics exporter pod:
$ kubectl logs cri-o-metrics-exporter-56855d56c8-fvptq -n cri-o-metrics-exporterAnything else we need to know?
No response
CRI-O and Kubernetes version
$ crio --version
crio version 1.31.5
GitCommit: 5184b74964ef620d64169a41fd27afdba6fe130e
GitCommitDate: 2025-02-03T17:22:57Z
GitTreeState: dirty
BuildDate: 1970-01-01T00:00:00Z
GoVersion: go1.22.5
Compiler: gc
Platform: linux/amd64
Linkmode: static
BuildTags:
static
netgo
osusergo
exclude_graphdriver_btrfs
seccomp
apparmor
selinux
exclude_graphdriver_devicemapper
LDFlags: unknown
SeccompEnabled: true
AppArmorEnabled: false$ sudo kubectl version --output=json --kubeconfig /etc/kubernetes/admin.conf
{
"clientVersion": {
"major": "1",
"minor": "31",
"gitVersion": "v1.31.10",
"gitCommit": "61183587c03f420214aac57f81dc0ecb43e1b0d6",
"gitTreeState": "clean",
"buildDate": "2025-06-17T18:40:26Z",
"goVersion": "go1.23.10",
"compiler": "gc",
"platform": "linux/amd64"
},
"kustomizeVersion": "v5.4.2",
"serverVersion": {
"major": "1",
"minor": "31",
"gitVersion": "v1.31.10",
"gitCommit": "61183587c03f420214aac57f81dc0ecb43e1b0d6",
"gitTreeState": "clean",
"buildDate": "2025-06-17T18:33:35Z",
"goVersion": "go1.23.10",
"compiler": "gc",
"platform": "linux/amd64"
}
}OS version
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
Linux linuxVmmaster01 6.8.0-1030-azure #35~22.04.1-Ubuntu SMP Mon May 26 18:08:30 UTC 2025 x86_64 x86_64 x86_64 GNU/LinuxAdditional environment details (AWS, VirtualBox, physical, etc.)
Azure Virtual Machines (not AKS)