crio panics and writes strange container started at time #9419
Description
What happened?
There was a node with several pods that appeared to be unready, despite looking completely healthy. Upon further investigation and digging into kubelet logs, we noticed that kubelet was not running probes against these pods. Manual run of the probe resulted in success. Restarting kubelet did not change anything, and other very similar pods had probes running on them just fine.
Further investigation with crictl inspect showed "startedAt": "1754-08-30T22:43:41.128654848Z", for one container, and a similar date in 8/30/1754 17:47:39 for the other. Unaffected containers showed normal startedAt dates: 2025-08-21T21:24:16Z
crio logs showed Aug 21 21:23:49 ip-172-27-15-58 crio[12172]: fatal error: concurrent map iteration and map write
What did you expect to happen?
I expected a normal looking startedAt and no concurrent map iteration/write panics
How can we reproduce it (as minimally and precisely as possible)?
Hard to repro, we have only encountered this a few times in the wild
Anything else we need to know?
No response
CRI-O and Kubernetes version
$ crio --version
crio version 1.31.2
GitCommit: 920af25f2b6fd4f52bced0e73f696e48464da543
GitCommitDate: 2024-11-01T10:41:59Z
GitTreeState: dirty
BuildDate: 1970-01-01T00:00:00Z
GoVersion: go1.22.5
Compiler: gc
Platform: linux/arm64
Linkmode: static
BuildTags:
static
netgo
osusergo
exclude_graphdriver_btrfs
seccomp
apparmor
selinux
exclude_graphdriver_devicemapper
LDFlags: unknown
SeccompEnabled: true
AppArmorEnabled: false
$ kubectl version --output=json
{
"clientVersion": {
"major": "1",
"minor": "31",
"gitVersion": "v1.31.7",
"gitCommit": "da53587841b4960dc3bd2af1ec6101b57c79aff4",
"gitTreeState": "clean",
"buildDate": "2025-03-11T20:03:56Z",
"goVersion": "go1.23.6",
"compiler": "gc",
"platform": "darwin/arm64"
},
"kustomizeVersion": "v5.4.2",
"serverVersion": {
"major": "1",
"minor": "31",
"gitVersion": "v1.31.7",
"gitCommit": "da53587841b4960dc3bd2af1ec6101b57c79aff4",
"gitTreeState": "archive",
"buildDate": "2025-03-26T14:06:57Z",
"goVersion": "go1.23.6",
"compiler": "gc",
"platform": "linux/amd64"
}
}
OS version
# On Linux:
$ cat /etc/os-release
NAME="AlmaLinux"
VERSION="9.4 (Seafoam Ocelot)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.4 (Seafoam Ocelot)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"
ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.4"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
SUPPORT_END=2032-06-01
$ uname -a
Linux ip-172-27-29-56 6.1.127-hs95.el9.aarch64 #1 SMP Wed Feb 19 00:18:56 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux