What happened?

When creating a container (usually etcd), the systemd cgroup slice intermittently does not get created. This causes the container create step to fail.

Mar 25 12:06:00 APP systemd[1]: Started crio-conmon-6353dddb7ceab1a0a48236ab99cdcabcc48a3fde2b7d9e8b7e4d8b2f42775444.scope.
Mar 25 12:06:00 APP systemd[1]: tmp-crun.BJkEFd.mount: Deactivated successfully.
Mar 25 12:06:00 APP systemd[1]: kubepods-burstable-pod1864dd0546310be9937c37cf88024cb8.slice: Failed to open /run/systemd/transient/kubepods-burstable-pod1864dd0546310be9937c37cf88024cb8.slice: No such file or directory
Mar 25 12:06:00 APP systemd[1]: logrotate.service: Deactivated successfully.
Mar 25 12:06:00 APP systemd[1]: Finished Rotate log files.
Mar 25 12:06:00 APP [40982]: conmon 6353dddb7ceab1a0a482 : runtime stderr: sd-bus call: Unit kubepods-burstable-pod1864dd0546310be9937c37cf88024cb8.slice not found.: I/O error
Mar 25 12:06:00 APP [40982]: conmon 6353dddb7ceab1a0a482 : Failed to create container: exit status 1
Mar 25 12:06:00 APP crio[39250]: time="2025-03-25 17:06:00.579293004Z" level=error msg="Container creation error: sd-bus call: Unit kubepods-burstable-pod1864dd0546310be9937c37cf88024cb8.slice not found.: I/O error\n" id=035832fc-ced3-4ad5-93b4-0cd6f67c68c5 name=/runtime.v1.RuntimeService/CreateContainer
Mar 25 12:06:00 APP crio[39250]: time="2025-03-25 17:06:00.586592578Z" level=info msg="createCtr: deleting container ID 6353dddb7ceab1a0a48236ab99cdcabcc48a3fde2b7d9e8b7e4d8b2f42775444 from idIndex" id=035832fc-ced3-4ad5-93b4-0cd6f67c68c5 name=/runtime.v1.RuntimeService/CreateContainer
Mar 25 12:06:00 APP crio[39250]: time="2025-03-25 17:06:00.586753432Z" level=info msg="createCtr: removing container 6353dddb7ceab1a0a48236ab99cdcabcc48a3fde2b7d9e8b7e4d8b2f42775444" id=035832fc-ced3-4ad5-93b4-0cd6f67c68c5 name=/runtime.v1.RuntimeService/CreateContainer
Mar 25 12:06:00 APP crio[39250]: time="2025-03-25 17:06:00.586839815Z" level=info msg="createCtr: deleting container 6353dddb7ceab1a0a48236ab99cdcabcc48a3fde2b7d9e8b7e4d8b2f42775444 from storage" id=035832fc-ced3-4ad5-93b4-0cd6f67c68c5 name=/runtime.v1.RuntimeService/CreateContainer
Mar 25 12:06:00 APP crio[39250]: time="2025-03-25 17:06:00.588930859Z" level=info msg="createCtr: releasing container name k8s_etcd_etcd-zcpa2_kube-system_1864dd0546310be9937c37cf88024cb8_2" id=035832fc-ced3-4ad5-93b4-0cd6f67c68c5 name=/runtime.v1.RuntimeService/CreateContainer
Mar 25 12:06:00 APP kubelet[40202]: E0325 17:06:00.589468 40202 log.go:32] "CreateContainer in sandbox from runtime service failed" err=<
Mar 25 12:06:00 APP kubelet[40202]: rpc error: code = Unknown desc = container create failed: sd-bus call: Unit kubepods-burstable-pod1864dd0546310be9937c37cf88024cb8.slice not found.: I/O error
Mar 25 12:06:00 APP kubelet[40202]: > podSandboxID="17a47d1bf1ca476476b415b6bfd3d0b60a1ddf723de3b481410a1438f70eef17"
Mar 25 12:06:00 APP kubelet[40202]: E0325 17:06:00.590638 40202 kuberuntime_manager.go:1341] "Unhandled Error" err=<
Mar 25 12:06:00 APP kubelet[40202]: container &Container{Name:etcd,Image:registry.k8s.io/etcd:3.5.16-0,Command:[etcd --advertise-client-urls=https://ip2:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 --client-cert-auth=true --data-dir=/media/data/etcd --experimental-initial-corrupt-check=true --experimental-watch-progress-notify-interval=5s --initial-advertise-peer-urls=https://ip2:2380 --initial-cluster=zcpa2=https://ip2:2380,zcpa1=https://ip1:2380 --initial-cluster-state=existing --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://ip2:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://ip2:2380 --name=zcpa2 --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --tls-min-version=TLS1.2 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{100 -3} {} 100m DecimalSI},memory: {{104857600 0} {} 100Mi BinarySI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:etcd-data,ReadOnly:false,MountPath:/media/data/etcd,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},VolumeMount{Name:etcd-certs,ReadOnly:false,MountPath:/etc/kubernetes/pki/etcd,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/livez,Port:{0 2381 },Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:10,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,TerminationGracePeriodSeconds:nil,},ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/readyz,Port:{0 2381 },Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:15,PeriodSeconds:1,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/readyz,Port:{0 2381 },Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:10,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:24,TerminationGracePeriodSeconds:nil,},ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod etcd-zcpa2_kube-system(1864dd0546310be9937c37cf88024cb8): CreateContainerError: container create failed: sd-bus call: Unit kubepods-burstable-pod1864dd0546310be9937c37cf88024cb8.slice not found.: I/O error
Mar 25 12:06:00 APP kubelet[40202]: >
Mar 25 12:06:00 APP kubelet[40202]: E0325 17:06:00.592592 40202 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to "StartContainer" for "etcd" with CreateContainerError: "container create failed: sd-bus call: Unit kubepods-burstable-pod1864dd0546310be9937c37cf88024cb8.slice not found.: I/O error\n"" pod="kube-system/etcd-zcpa2" podUID="1864dd0546310be9937c37cf88024cb8"
Mar 25 12:06:00 APP systemd[1]: media-data-containers-storage-overlay-53cd886f518f2c31f913f416e84f015a3a5310fcf426ba1fc8f715914f71f8cd-merged.mount: Deactivated successfully.
Mar 25 12:06:00 APP systemd[1]: crio-conmon-6353dddb7ceab1a0a48236ab99cdcabcc48a3fde2b7d9e8b7e4d8b2f42775444.scope: Deactivated successfully.
Mar 25 12:06:15 APP crio[39250]: time="2025-03-25 17:06:15.364085156Z" level=info msg="Checking image status: registry.k8s.io/etcd:3.5.16-0" id=08242feb-098e-4d71-b7cb-aeda0cd22139 name=/runtime.v1.ImageService/ImageStatus
Mar 25 12:06:15 APP crio[39250]: time="2025-03-25 17:06:15.366636777Z" level=info msg="Image status: &ImageStatusResponse{Image:&Image{Id:75922420560b93ad374771e472186633179f447b1e378321140183f892e01c02,RepoTags:[registry.k8s.io/etcd:3.5.16-0],RepoDigests:[registry.k8s.io/etcd@sha256:1e61daa0ad3b0172ccb106441068d64c18628d4f7ed8ee61302d6480ce353ff6],Size_:152984275,Uid:&Int64Value{Value:0,},Username:,Spec:&ImageSpec{Image:,Annotations:map[string]string{org.opencontainers.image.base.digest: sha256:a40ada13b43396ccb7ee1799b06a3e46404ddd84a26001f9991e7f121d1e3dc8,org.opencontainers.image.base.name: icr.io/sys-zos-zcontainers-docker/registry.k8s.io/etcd:3.5.16-0,},UserSpecifiedImage:,RuntimeHandler:,},Pinned:false,},Info:map[string]string{},}" id=08242feb-098e-4d71-b7cb-aeda0cd22139 name=/runtime.v1.ImageService/ImageStatus
Mar 25 12:06:15 APP crio[39250]: time="2025-03-25 17:06:15.372491655Z" level=info msg="Checking image status: registry.k8s.io/etcd:3.5.16-0" id=d89f4e52-be16-4e93-b542-8f0ef67295d9 name=/runtime.v1.ImageService/ImageStatus
Mar 25 12:06:15 APP crio[39250]: time="2025-03-25 17:06:15.375138175Z" level=info msg="Image status: &ImageStatusResponse{Image:&Image{Id:75922420560b93ad374771e472186633179f447b1e378321140183f892e01c02,RepoTags:[registry.k8s.io/etcd:3.5.16-0],RepoDigests:[registry.k8s.io/etcd@sha256:1e61daa0ad3b0172ccb106441068d64c18628d4f7ed8ee61302d6480ce353ff6],Size_:152984275,Uid:&Int64Value{Value:0,},Username:,Spec:&ImageSpec{Image:,Annotations:map[string]string{org.opencontainers.image.base.digest: sha256:a40ada13b43396ccb7ee1799b06a3e46404ddd84a26001f9991e7f121d1e3dc8,org.opencontainers.image.base.name: icr.io/sys-zos-zcontainers-docker/registry.k8s.io/etcd:3.5.16-0,},UserSpecifiedImage:,RuntimeHandler:,},Pinned:false,},Info:map[string]string{},}" id=d89f4e52-be16-4e93-b542-8f0ef67295d9 name=/runtime.v1.ImageService/ImageStatus
Mar 25 12:06:15 APP crio[39250]: time="2025-03-25 17:06:15.392366518Z" level=info msg="Creating container: kube-system/etcd-zcpa2/etcd" id=35660e2e-a18c-4046-92bb-979d0e85e4d2 name=/runtime.v1.RuntimeService/CreateContainer
Mar 25 12:06:15 APP crio[39250]: time="2025-03-25 17:06:15.409139654Z" level=info msg="Allowed annotations are specified for workload [io.containers.trace-syscall]"
Mar 25 12:06:15 APP crio[39250]: time="2025-03-25 17:06:15.474368726Z" level=info msg="Allowed annotations are specified for workload [io.containers.trace-syscall]"
Mar 25 12:06:15 APP crio[39250]: time="2025-03-25 17:06:15.475639383Z" level=info msg="Allowed annotations are specified for workload [io.containers.trace-syscall]"
Mar 25 12:06:15 APP systemd[1]: Started crio-conmon-bd0b464d68b66d57cf98856fe95355314c64f3ab452cd28dcd972602f3b6d9a6.scope.
Mar 25 12:06:15 APP systemd[1]: tmp-crun.oMBpAJ.mount: Deactivated successfully.
Mar 25 12:06:15 APP systemd[1]: kubepods-burstable-pod1864dd0546310be9937c37cf88024cb8.slice: Failed to open /run/systemd/transient/kubepods-burstable-pod1864dd0546310be9937c37cf88024cb8.slice: No such file or directory
Mar 25 12:06:15 APP [41107]: conmon bd0b464d68b66d57cf98 : runtime stderr: sd-bus call: Unit kubepods-burstable-pod1864dd0546310be9937c37cf88024cb8.slice not found.: I/O error
Mar 25 12:06:15 APP [41107]: conmon bd0b464d68b66d57cf98 : Failed to create container: exit status 1
Mar 25 12:06:15 APP crio[39250]: time="2025-03-25 17:06:15.539753406Z" level=error msg="Container creation error: sd-bus call: Unit kubepods-burstable-pod1864dd0546310be9937c37cf88024cb8.slice not found.: I/O error\n" id=35660e2e-a18c-4046-92bb-979d0e85e4d2 name=/runtime.v1.RuntimeService/CreateContainer
Mar 25 12:06:15 APP systemd[1]: crio-conmon-bd0b464d68b66d57cf98856fe95355314c64f3ab452cd28dcd972602f3b6d9a6.scope: Deactivated successfully.

What did you expect to happen?

The slice and container to be successfully created.

How can we reproduce it (as minimally and precisely as possible)?

Several iterations of creating and destroying a container (especially etcd)

Anything else we need to know?

No response

CRI-O and Kubernetes version

$ crio --version
Version:        1.30.5
GitCommit:      df27b8f8eb49a13c522aca56ee4ec27bc7482fad
GitCommitDate:  2024-09-02T07:15:35Z
GitTreeState:   clean
BuildDate:      1970-01-01T00:00:00Z
GoVersion:      go1.22.0
Compiler:       gc
Platform:       linux/s390x
Linkmode:       static
BuildTags:
  static
  netgo
  osusergo
  exclude_graphdriver_btrfs
  exclude_graphdriver_devicemapper
  seccomp
  apparmor
  selinux
LDFlags:          unknown
SeccompEnabled:   true
AppArmorEnabled:  false

$ kubectl version --output=json
{
  "clientVersion": {
    "major": "1",
    "minor": "32",
    "gitVersion": "v1.32.0",
    "gitCommit": "70d3cc986aa8221cd1dfb1121852688902d3bf53",
    "gitTreeState": "clean",
    "buildDate": "2024-12-11T18:05:36Z",
    "goVersion": "go1.23.3",
    "compiler": "gc",
    "platform": "linux/s390x"
  },
  "kustomizeVersion": "v5.5.0"
}

OS version

# On Linux:
$ cat /etc/os-release
RHEL9.4
$ uname -a
Linux host 5.14.0-427.37.1.el9.s390x #1 SMP Thu Oct 17 04:26:45 CDT 2024 s390x s390x s390x GNU/Linux

Additional environment details (AWS, VirtualBox, physical, etc.)