Open
Description
What happened?
As the Crio create container failed by some reason, and then try to delete the failed container, but it just try to "Retried cleanup function" not more than 20 times, and then the container will stay in "name reserved" failed logs
And finally restart the cri-o will release the "reserved name" and create container successfully
What did you expect to happen?
The Cri-o will try to recover the container , or show the extract message of creating container failed
How can we reproduce it (as minimally and precisely as possible)?
The Cri-o logs show as follows:
Apr 01 11:54:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:54:11.555223670+08:00" level=info msg="Checking image status: image.ccos.io/ceake/ceake@sha256:85e4f51eadda175c8ffc35f1b8138c7a934beed067cfae7c5a9eabd04deba193" id=c2cccccf-f693-47f5-87ae-f3805024f2fb name=/runtime.v1.ImageService/ImageStatus
Apr 01 11:54:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:54:11.555585869+08:00" level=info msg="Image status: &ImageStatusResponse{Image:&Image{Id:d6554f6a8b339e63d8dd25544efeeb03b4ee3eb16b0b2fe75e148296f8ff3e5c,RepoTags:[],RepoDigests:[image.ccos.io/ceake/ceake@sha256:63861697974fe2f05e13877f793513f4713930efc5efaa4c6760291d441f2b28 image.ccos.io/ceake/ceake@sha256:85e4f51eadda175c8ffc35f1b8138c7a934beed067cfae7c5a9eabd04deba193],Size_:697555508,Uid:nil,Username:,Spec:&ImageSpec{Image:,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},Pinned:false,},Info:map[string]string{},}" id=c2cccccf-f693-47f5-87ae-f3805024f2fb name=/runtime.v1.ImageService/ImageStatus
Apr 01 11:54:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:54:11.558079292+08:00" level=info msg="Checking image status: image.ccos.io/ceake/ceake@sha256:85e4f51eadda175c8ffc35f1b8138c7a934beed067cfae7c5a9eabd04deba193" id=bd22ff44-c6f2-4cc9-8582-13bd75745eb5 name=/runtime.v1.ImageService/ImageStatus
Apr 01 11:54:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:54:11.559219292+08:00" level=info msg="Image status: &ImageStatusResponse{Image:&Image{Id:d6554f6a8b339e63d8dd25544efeeb03b4ee3eb16b0b2fe75e148296f8ff3e5c,RepoTags:[],RepoDigests:[image.ccos.io/ceake/ceake@sha256:63861697974fe2f05e13877f793513f4713930efc5efaa4c6760291d441f2b28 image.ccos.io/ceake/ceake@sha256:85e4f51eadda175c8ffc35f1b8138c7a934beed067cfae7c5a9eabd04deba193],Size_:697555508,Uid:nil,Username:,Spec:&ImageSpec{Image:,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},Pinned:false,},Info:map[string]string{},}" id=bd22ff44-c6f2-4cc9-8582-13bd75745eb5 name=/runtime.v1.ImageService/ImageStatus
Apr 01 11:54:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:54:11.564019393+08:00" level=info msg="Creating container: ccos-kube-scheduler/ccos-kube-scheduler-5dx2ipsl8ejbh17t/kube-scheduler" id=1c248f37-caf9-41f6-b501-54ec65e449c1 name=/runtime.v1.RuntimeService/CreateContainer
Apr 01 11:54:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:54:11.564389360+08:00" level=warning msg="error reserving ctr name k8s_kube-scheduler_ccos-kube-scheduler-5dx2ipsl8ejbh17t_ccos-kube-scheduler_b50f9084bb9cf48f965f71165cbaab72_0 for id 3ce74949b544bed24651dfa10c0e3445b171f4f88e38c11af32f3bec7b99a9cd: name is reserved"
Apr 01 11:54:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:54:11.564463839+08:00" level=info msg="Creation of container k8s_kube-scheduler_ccos-kube-scheduler-5dx2ipsl8ejbh17t_ccos-kube-scheduler_b50f9084bb9cf48f965f71165cbaab72_0 not yet finished. Currently at stage container runtime creation. Waiting up to 5m59.999082468s for it to finish" id=1c248f37-caf9-41f6-b501-54ec65e449c1 name=/runtime.v1.RuntimeService/CreateContainer
Apr 01 11:54:57 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:54:57.609969252+08:00" level=info msg="createCtr: deleting container 367e907c177f1d87092b2133e142b120f07c193d89ca03e8b87720752cd5646f from storage" id=b418d66d-ece8-43d2-a13c-18438fb987ac name=/runtime.v1.RuntimeService/CreateContainer
Apr 01 11:54:57 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:54:57.611117060+08:00" level=error msg="Failed to cleanup (probably retrying): failed to cleanup container storage: removing mount point \"/data/ceake/imagefs/overlay/2dcf93041c850b570ed42d3a37a2cd67b6280711b36af81d2cddc6443b79bf90/merged\": directory not empty" id=b418d66d-ece8-43d2-a13c-18438fb987ac name=/runtime.v1.RuntimeService/CreateContainer
Apr 01 11:54:57 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:54:57.611177651+08:00" level=error msg="Retried cleanup function \"createCtr: deleting container 367e907c177f1d87092b2133e142b120f07c193d89ca03e8b87720752cd5646f from storage\" too often, giving up" id=b418d66d-ece8-43d2-a13c-18438fb987ac name=/runtime.v1.RuntimeService/CreateContainer
Apr 01 11:54:57 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:54:57.611190825+08:00" level=error msg="Unable to cleanup: wait on retry: timed out waiting for the condition" id=b418d66d-ece8-43d2-a13c-18438fb987ac name=/runtime.v1.RuntimeService/CreateContainer
Apr 01 11:56:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:56:11.908311356+08:00" level=info msg="Checking image status: image.ccos.io/ceake/ceake@sha256:85e4f51eadda175c8ffc35f1b8138c7a934beed067cfae7c5a9eabd04deba193" id=dbb7eed8-c97a-4682-856e-2d0be9619507 name=/runtime.v1.ImageService/ImageStatus
Apr 01 11:56:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:56:11.908552871+08:00" level=info msg="Image status: &ImageStatusResponse{Image:&Image{Id:d6554f6a8b339e63d8dd25544efeeb03b4ee3eb16b0b2fe75e148296f8ff3e5c,RepoTags:[],RepoDigests:[image.ccos.io/ceake/ceake@sha256:63861697974fe2f05e13877f793513f4713930efc5efaa4c6760291d441f2b28 image.ccos.io/ceake/ceake@sha256:85e4f51eadda175c8ffc35f1b8138c7a934beed067cfae7c5a9eabd04deba193],Size_:697555508,Uid:nil,Username:,Spec:&ImageSpec{Image:,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},Pinned:false,},Info:map[string]string{},}" id=dbb7eed8-c97a-4682-856e-2d0be9619507 name=/runtime.v1.ImageService/ImageStatus
Apr 01 11:56:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:56:11.910235749+08:00" level=info msg="Checking image status: image.ccos.io/ceake/ceake@sha256:85e4f51eadda175c8ffc35f1b8138c7a934beed067cfae7c5a9eabd04deba193" id=36238312-f8cd-494f-950a-962ff22ec40c name=/runtime.v1.ImageService/ImageStatus
Apr 01 11:56:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:56:11.910518486+08:00" level=info msg="Image status: &ImageStatusResponse{Image:&Image{Id:d6554f6a8b339e63d8dd25544efeeb03b4ee3eb16b0b2fe75e148296f8ff3e5c,RepoTags:[],RepoDigests:[image.ccos.io/ceake/ceake@sha256:63861697974fe2f05e13877f793513f4713930efc5efaa4c6760291d441f2b28 image.ccos.io/ceake/ceake@sha256:85e4f51eadda175c8ffc35f1b8138c7a934beed067cfae7c5a9eabd04deba193],Size_:697555508,Uid:nil,Username:,Spec:&ImageSpec{Image:,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},Pinned:false,},Info:map[string]string{},}" id=36238312-f8cd-494f-950a-962ff22ec40c name=/runtime.v1.ImageService/ImageStatus
Apr 01 11:56:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:56:11.913002638+08:00" level=info msg="Creating container: ccos-kube-scheduler/ccos-kube-scheduler-5dx2ipsl8ejbh17t/kube-scheduler" id=8aea658a-a2fc-44a5-a56f-3703532c28e3 name=/runtime.v1.RuntimeService/CreateContainer
Apr 01 11:56:11 5dx2ipsl8ejbh17t crio[33667]: time="2025-04-01 11:56:11.913123942+08:00" level=warning msg="error reserving ctr name k8s_kube-scheduler_ccos-kube-scheduler-5dx2ipsl8ejbh17t_ccos-kube-scheduler_b50f9084bb9cf48f965f71165cbaab72_0 for id 6ee5679adcb8998d79b38729370d18a91058a156051f4dc159446a02487078dd: name is reserved"
But After a long time ,we tried the umount the ponit, it has no longer mouted, but the cri- still hold the name, but no released
Anything else we need to know?
No response
CRI-O and Kubernetes version
$ crio --version
# paste output here
crio version 1.29.13
Version: 1.29.13
runc 1.12$ kubectl version --output=json
# paste output here
Server Version: v1.29.6OS version
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here