ctr name reserved by Exited container

### What happened?

1. kube-apiserver pods show one kube-apiserver-check-endpoints container of static pod kube-apiserver is not ready, as follows:
```
kube-apiserver-master0         5/5     Running             6 (120m ago)    44m
kube-apiserver-master1         4/5     RunContainerError   5 (120m ago)    3h32m
kube-apiserver-master2         4/5     RunContainerError   11 (120m ago)   3h38m
```

2. but as we use crictl show the kube-apiserver container status, they are were running (except one kube-apiserver-check-endpoints is Exited), as follows:
```
[root@master1 ~]# crictl ps -a | grep kube-apiserver
9adf6c538f12e       e96948b107f8085d42d1e4ef33e665558ad5a44deaedf42d1fb1763b5493eabe   2 hours ago         Exited              kube-apiserver-check-endpoints                1                   3d4536c7f85bd
95829bb6fdf37       e96948b107f8085d42d1e4ef33e665558ad5a44deaedf42d1fb1763b5493eabe   2 hours ago         Running             kube-apiserver-insecure-readyz                1                   3d4536c7f85bd
1b142b197e984       e96948b107f8085d42d1e4ef33e665558ad5a44deaedf42d1fb1763b5493eabe   2 hours ago         Running             kube-apiserver-cert-regeneration-controller   1                   3d4536c7f85bd
af47cd69b47a8       e96948b107f8085d42d1e4ef33e665558ad5a44deaedf42d1fb1763b5493eabe   2 hours ago         Running             kube-apiserver-cert-syncer                    1                   3d4536c7f85bd
f19262e77cb76       0c6d19deda5e02378df6cf392b4ceafece7dc2ddeb0b4b2a27f22a594dc0bde0   2 hours ago         Running             kube-apiserver                                1                   3d4536c7f85bd
27e4aa876dec4       e96948b107f8085d42d1e4ef33e665558ad5a44deaedf42d1fb1763b5493eabe   2 hours ago         Running             kube-apiserver-check-endpoints                2                   3d4536c7f85bd    
```

3. then we use crictl to remove the Exited kube-apiserver-check-endpoints in master0, the kube-apiserver-master0 pod shows ready now

4. and then we turn the master2 crio log into debug, I found the crio return the wrong ContainerStatusResponse as follows:
```
Dec 30 15:44:09 master1 crio[1078]: time="2024-12-30 15:44:09.792796475+08:00" level=info msg="Creating container: kube-apiserver/kube-apiserver-master1/kube-apiserver-check-endpoints" file="server/container_create.go:294" id=787f4ba8-ba84-4a03-aead-5dc32f012fe0 name=/runtime.v1.RuntimeService/CreateContainer
Dec 30 15:44:09 master1 crio[1078]: time="2024-12-30 15:44:09.792877974+08:00" level=warning msg="error reserving ctr name k8s_kube-apiserver-check-endpoints_kube-apiserver-master1_kube-apiserver_e144a447563d485f128fa54e17f09f2f_2 for id a1ec898c6158b1d6fe0cf9b9a7522dd529ff75e2197b3ad1494388e6e2c548d7: name is reserved" file="lib/container_server.go:499"

Dec 30 16:47:56 master1 crio[1078]: time="2024-12-30 16:47:56.565896059+08:00" level=debug msg="Response: &ContainerStatusResponse{Status:&ContainerStatus{Id:27e4aa876dec414c40569bd53782a799815038f389fedd89fcfd9353597a1734,Metadata:&ContainerMetadata{Name:kube-apiserver-check-endpoints,Attempt:2,},State:CONTAINER_RUNNING,CreatedAt:1735537219167849458,StartedAt:1735537219259293383,FinishedAt:0,ExitCode:0,Image:&ImageSpec{Image:image.cestc.cn/cke/cke@sha256:0d37cd2fbf478b7bff972a95c68159f8886e9439f47ed073b0a5ad7c35dc9586,Annotations:map[string]string{},},ImageRef:image.cestc.cn/cke/cke@sha256:0d37cd2fbf478b7bff972a95c68159f8886e9439f47ed073b0a5ad7c35dc9586,Reason:,Message:,Labels:map[string]string{io.kubernetes.container.name: kube-apiserver-check-endpoints,io.kubernetes.pod.name: kube-apiserver-master1,io.kubernetes.pod.namespace: kube-apiserver,io.kubernetes.pod.uid: e144a447563d485f128fa54e17f09f2f,},Annotations:map[string]string{io.kubernetes.container.hash: dde67ace,io.kubernetes.container.ports: [{\"name\":\"check-endpoints\",\"hostPort\":17697,\"containerPort\":17697,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: FallbackToLogsOnError,io.kubernetes.pod.terminationGracePeriod: 135,},Mounts:[]*Mount{&Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/kubelet/pods/e144a447563d485f128fa54e17f09f2f/etc-hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,},&Mount{ContainerPath:/dev/termination-log,HostPath:/var/lib/kubelet/pods/e144a447563d485f128fa54e17f09f2f/containers/kube-apiserver-check-endpoints/6e2ab296,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,},&Mount{ContainerPath:/etc/kubernetes/static-pod-resources,HostPath:/etc/kubernetes/static-pod-resources/kube-apiserver-pod-9,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,},&Mount{ContainerPath:/etc/kubernetes/static-pod-certs,HostPath:/etc/kubernetes/static-pod-resources/kube-apiserver-certs,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,},},LogPath:/var/log/pods/kube-apiserver_kube-apiserver-master1_e144a447563d485f128fa54e17f09f2f/kube-apiserver-check-endpoints/2.log,Resources:nil,},Info:map[string]string{},}" file="go-grpc-middleware/chain.go:25" id=29325288-9811-43d2-983d-04c2c64b5557 name=/runtime.v1.RuntimeService/ContainerStatus
Dec 30 16:47:56 master1 crio[1078]: time="2024-12-30 16:47:56.793600968+08:00" level=debug msg="Request: &CreateContainerRequest{PodSandboxId:3d4536c7f85bda0785bbdd6b146f1f2f624d578c4ae9aaad29efad16313e7580,Config:&ContainerConfig{Metadata:&ContainerMetadata{Name:kube-apiserver-check-endpoints,Attempt:2,},Image:&ImageSpec{Image:e96948b107f8085d42d1e4ef33e665558ad5a44deaedf42d1fb1763b5493eabe,Annotations:map[string]string{},},Command:[cluster-kube-apiserver-operator check-endpoints],Args:[--kubeconfig /etc/kubernetes/static-pod-certs/configmaps/check-endpoints-kubeconfig/kubeconfig --listen 10.253.41.138:17697 --namespace kube-apiserver --v 2],WorkingDir:,Envs:[]*KeyValue{&KeyValue{Key:POD_NAME,Value:kube-apiserver-master1,},&KeyValue{Key:POD_IP,Value:10.253.41.138,},&KeyValue{Key:POD_NAMESPACE,Value:kube-apiserver,},&KeyValue{Key:KUBERNETES_SERVICE_PORT,Value:443,},&KeyValue{Key:KUBERNETES_SERVICE_PORT_HTTPS,Value:443,},&KeyValue{Key:KUBERNETES_PORT,Value:tcp://21.101.0.1:443,},&KeyValue{Key:KUBERNETES_PORT_443_TCP_PORT,Value:443,},&KeyValue{Key:KUBERNETES_PORT_443_TCP_ADDR,Value:21.101.0.1,},&KeyValue{Key:APISERVER_SERVICE_PORT_HTTPS,Value:443,},&KeyValue{Key:APISERVER_PORT_443_TCP_ADDR,Value:21.101.11.102,},&KeyValue{Key:KUBERNETES_PORT_443_TCP,Value:tcp://21.101.0.1:443,},&KeyValue{Key:KUBERNETES_SERVICE_HOST,Value:21.101.0.1,},&KeyValue{Key:APISERVER_SERVICE_PORT,Value:443,},&KeyValue{Key:APISERVER_PORT,Value:tcp://21.101.11.102:443,},&KeyValue{Key:APISERVER_PORT_443_TCP,Value:tcp://21.101.11.102:443,},&KeyValue{Key:APISERVER_PORT_443_TCP_PROTO,Value:tcp,},&KeyValue{Key:APISERVER_PORT_443_TCP_PORT,Value:443,},&KeyValue{Key:KUBERNETES_PORT_443_TCP_PROTO,Value:tcp,},&KeyValue{Key:APISERVER_SERVICE_HOST,Value:21.101.11.102,},},Mounts:[]*Mount{&Mount{ContainerPath:/etc/kubernetes/static-pod-resources,HostPath:/etc/kubernetes/static-pod-resources/kube-apiserver-pod-9,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,},&Mount{ContainerPath:/etc/kubernetes/static-pod-certs,HostPath:/etc/kubernetes/static-pod-resources/kube-apiserver-certs,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,},&Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/kubelet/pods/e144a447563d485f128fa54e17f09f2f/etc-hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,},&Mount{ContainerPath:/dev/termination-log,HostPath:/var/lib/kubelet/pods/e144a447563d485f128fa54e17f09f2f/containers/kube-apiserver-check-endpoints/24a3a463,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,},},Devices:[]*Device{},Labels:map[string]string{io.kubernetes.container.name: kube-apiserver-check-endpoints,io.kubernetes.pod.name: kube-apiserver-master1,io.kubernetes.pod.namespace: kube-apiserver,io.kubernetes.pod.uid: e144a447563d485f128fa54e17f09f2f,},Annotations:map[string]string{io.kubernetes.container.hash: dde67ace,io.kubernetes.container.ports: [{\"name\":\"check-endpoints\",\"hostPort\":17697,\"containerPort\":17697,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: FallbackToLogsOnError,io.kubernetes.pod.terminationGracePeriod: 135,},LogPath:kube-apiserver-check-endpoints/2.log,Stdin:false,StdinOnce:false,Tty:false,Linux:&LinuxContainerConfig{Resources:&LinuxContainerResources{CpuPeriod:100000,CpuQuota:0,CpuShares:10,MemoryLimitInBytes:0,OomScoreAdj:-997,CpusetCpus:,CpusetMems:,HugepageLimits:[]*HugepageLimit{&HugepageLimit{PageSize:2MB,Limit:0,},&HugepageLimit{PageSize:1GB,Limit:0,},},Unified:map[string]string{},MemorySwapLimitInBytes:0,},SecurityContext:&LinuxContainerSecurityContext{Capabilities:nil,Privileged:false,NamespaceOptions:&NamespaceOption{Network:NODE,Pid:CONTAINER,Ipc:POD,TargetId:,UsernsOptions:nil,},SelinuxOptions:nil,RunAsUser:&Int64Value{Value:0,},RunAsUsername:,ReadonlyRootfs:false,SupplementalGroups:[],ApparmorProfile:,SeccompProfilePath:,NoNewPrivs:false,RunAsGroup:nil,MaskedPaths:[/proc/acpi /proc/kcore /proc/keys /proc/latency_stats /proc/timer_list /proc/timer_stats /proc/sched_debug /proc/scsi /sys/firmware],ReadonlyPaths:[/proc/asound /proc/bus /proc/fs /proc/irq /proc/sys /proc/sysrq-trigger],Seccomp:&SecurityProfile{ProfileType:Unconfined,LocalhostRef:,},Apparmor:nil,},},Windows:nil,},SandboxConfig:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:kube-apiserver-master1,Uid:e144a447563d485f128fa54e17f09f2f,Namespace:kube-apiserver,Attempt:0,},Hostname:,LogDirectory:/var/log/pods/ccos-kube-apiserver_kube-apiserver-master1_e144a447563d485f128fa54e17f09f2f,DnsConfig:&DNSConfig{Servers:[10.253.41.138 10.253.41.140 223.5.5.5],Searches:[],Options:[],},PortMappings:[]*PortMapping{&PortMapping{Protocol:TCP,ContainerPort:6443,HostPort:6443,HostIp:,},&PortMapping{Protocol:TCP,ContainerPort:6080,HostPort:6080,HostIp:,},&PortMapping{Protocol:TCP,ContainerPort:17697,HostPort:17697,HostIp:,},},Labels:map[string]string{apiserver: true,app: kube-apiserver,io.kubernetes.pod.name: kube-apiserver-master1,io.kubernetes.pod.namespace: kube-apiserver,io.kubernetes.pod.uid: e144a447563d485f128fa54e17f09f2f,revision: 9,},Annotations:map[string]string{kubectl.kubernetes.io/default-container: kube-apiserver,kubernetes.io/config.hash: e144a447563d485f128fa54e17f09f2f,kubernetes.io/config.seen: 2024-12-30T13:42:11.083863444+08:00,kubernetes.io/config.source: file,target.workload.io/management: {\"effect\": \"PreferredDuringScheduling\"},},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode144a447563d485f128fa54e17f09f2f.slice,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:NODE,Pid:CONTAINER,Ipc:POD,TargetId:,UsernsOptions:nil,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,SeccompProfilePath:runtime/default,RunAsGroup:nil,Seccomp:&SecurityProfile{ProfileType:RuntimeDefault,LocalhostRef:,},Apparmor:nil,},Sysctls:map[string]string{},Overhead:&LinuxContainerResources{CpuPeriod:0,CpuQuota:0,CpuShares:0,MemoryLimitInBytes:0,OomScoreAdj:0,CpusetCpus:,CpusetMems:,HugepageLimits:[]*HugepageLimit{},Unified:map[string]string{},MemorySwapLimitInBytes:0,},Resources:&LinuxContainerResources{CpuPeriod:100000,CpuQuota:0,CpuShares:163,MemoryLimitInBytes:0,OomScoreAdj:0,CpusetCpus:,CpusetMems:,HugepageLimits:[]*HugepageLimit{},Unified:map[string]string{},MemorySwapLimitInBytes:0,},},Windows:nil,},}" file="go-grpc-middleware/chain.go:25" id=fd7b68a9-4e15-4b95-887a-5a450adf6605 name=/runtime.v1.RuntimeService/CreateContainer
Dec 30 16:47:56 master1 crio[1078]: time="2024-12-30 16:47:56.793762531+08:00" level=info msg="Creating container: kube-apiserver/kube-apiserver-master1/kube-apiserver-check-endpoints" file="server/container_create.go:294" id=fd7b68a9-4e15-4b95-887a-5a450adf6605 name=/runtime.v1.RuntimeService/CreateContainer
Dec 30 16:47:56 master1 crio[1078]: time="2024-12-30 16:47:56.793857258+08:00" level=warning msg="error reserving ctr name k8s_kube-apiserver-check-endpoints_kube-apiserver-master1_kube-apiserver_e144a447563d485f128fa54e17f09f2f_2 for id 8789cbd52c086fdae5b07b8937881dbfaf7e22df62beab2446db41f9693e7c9e: name is reserved" file="lib/container_server.go:499"

,&Container{Id:9adf6c538f12e7ba6292f208f2a366a21004514a4b81554f5878de597442a539,PodSandboxId:3d4536c7f85bda0785bbdd6b146f1f2f624d578c4ae9aaad29efad16313e7580,Metadata:&ContainerMetadata{Name:kube-apiserver-check-endpoints,Attempt:1,}

&Container{Id:27e4aa876dec414c40569bd53782a799815038f389fedd89fcfd9353597a1734,PodSandboxId:3d4536c7f85bda0785bbdd6b146f1f2f624d578c4ae9aaad29efad16313e7580,Metadata:&ContainerMetadata{Name:kube-apiserver-check-endpoints,Attempt:2,},

&Container{Id:9adf6c538f12e7ba6292f208f2a366a21004514a4b81554f5878de597442a539,PodSandboxId:3d4536c7f85bda0785bbdd6b146f1f2f624d578c4ae9aaad29efad16313e7580,Metadata:&ContainerMetadata{Name:kube-apiserver-check-endpoints,Attempt:1,}

&Container{Id:27e4aa876dec414c40569bd53782a799815038f389fedd89fcfd9353597a1734,PodSandboxId:3d4536c7f85bda0785bbdd6b146f1f2f624d578c4ae9aaad29efad16313e7580,Metadata:&ContainerMetadata{Name:kube-apiserver-check-endpoints,Attempt:2,}
```

5. as can be seen from the crio logs, the Running kube-apiserver-check-endpoints id is 27e4aa876dec4, 
6. and the Exited kube-apiserver-check-endpoints   is 9adf6c538f12e, 
7. but the ContainerStatusResponse return shows the 27e4aa876de has already ExitCode:0, and shows the name is reserved by k8s_kube-apiserver-check-endpoints_kube-apiserver-master1_kube-apiserver_e144a447563d485f128fa54e17f09f2f_2

<img width="1104" alt="image" src="https://github.com/user-attachments/assets/39199b8a-c9e9-46ae-bce0-10ae15595ce8" />


### What did you expect to happen?

ctr name will not be occupied by Exited container

### How can we reproduce it (as minimally and precisely as possible)?

rarely, but related to https://github.com/cri-o/cri-o/issues/8629

### Anything else we need to know?

_No response_

### CRI-O and Kubernetes version

<details>

```console
$ crio --version
# paste output here
1.25.4
```

```console
$ kubectl version --output=json
# paste output here
1.25.8
```

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
5.15.67-2.cl9
```

</details>


### Additional environment details (AWS, VirtualBox, physical, etc.)

<details>
physical
</details>


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ctr name reserved by Exited container #8884

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

CRI-O and Kubernetes version

OS version

Additional environment details (AWS, VirtualBox, physical, etc.)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ctr name reserved by Exited container #8884

Description

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

CRI-O and Kubernetes version

OS version

Additional environment details (AWS, VirtualBox, physical, etc.)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions