+
Skip to content

Segmentation Fault when deploying many pods at a time #9181

Closed
@bitoku

Description

@bitoku

What happened?

When I tried creating a deployment in local cluster, which has 50 pods, CRI-O exited with segmentation fault error, and there's no stack trace.

zsh: segmentation fault  sudo bin/crio

I tried to debug with dlv, but it didn't show any information other than its exit status.

❯ sudo dlv exec bin/crio
Type 'help' for list of commands.
(dlv) break runtime.sigpanic
Breakpoint 1 set at 0x10da00e for runtime.sigpanic() runtime/signal_unix.go:906
(dlv) break runtime.crash
Breakpoint 2 set at 0x10b7f44 for runtime.crash() runtime/signal_unix.go:1055
(dlv) c

Some logs...

Process 128359 has exited with status -11
(dlv)

What did you expect to happen?

No segmentation fault happens and CRI-O keeps running.

How can we reproduce it (as minimally and precisely as possible)?

Run CRI-O
Launch local kubernetes cluster.

sudo rm -rf /tmp/*
sudo rm -rf /var/run/kubernetes

export CGROUP_DRIVER=systemd
export CONTAINER_RUNTIME=remote
export CONTAINER_RUNTIME_ENDPOINT=unix:///var/run/crio/crio.sock
export FEATURE_GATES=""

cd ~/kubernetes/kubernetes
./hack/local-up-cluster.sh

Create the deployment

cat << EOF | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  labels:
    app: test
spec:
  replicas: 50
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - name: test-container
        image: quay.io/libpod/busybox
        command: ["sleep", "infinity"]
EOF

Anything else we need to know?

It always happens so far.

CRI-O and Kubernetes version

$ crio --version

crio version 1.33.0
   GitCommit:      2fe75a93f6526cf5c649476692cdecfc982e13e8
   GitCommitDate:  2025-04-30T07:36:51Z
   GitTreeState:   dirty
   BuildDate:      2025-05-02T14:09:48Z
   GoVersion:      go1.24.2
   Compiler:       gc
   Platform:       linux/amd64
   Linkmode:       dynamic
   BuildTags:
     containers_image_ostree_stub
     apparmor
     seccomp
     selinux
   LDFlags:          unknown
   SeccompEnabled:   true
   AppArmorEnabled:  false
$ cd kubernetes/kubernetes
$ git log --oneline
cf6cc0f5 (grafted, HEAD -> master, upstream/master, upstream/HEAD) Merge pull request #130381 from royalsflush/probe-total-beta

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 25.04"
NAME="Ubuntu"
VERSION_ID="25.04"
VERSION="25.04 (Plucky Puffin)"
VERSION_CODENAME=plucky
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=plucky
LOGO=ubuntu-logo
$ uname -a
Linux atokubi-ubuntu-os-cloud-dev 6.14.0-1006-gcp #6-Ubuntu SMP Tue Apr  8 15:59:29 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Additional environment details (AWS, VirtualBox, physical, etc.)

GCP

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      点击 这是indexloc提供的php浏览器服务,不要输入任何密码和下载