CRI-O fails to find plugin that got created after CRI-O got started #9439
Description
What happened?
I use K3s with CRI-O. To do that, I first install CRI-O with configuration that points crio.network's plugin_dirs to /var/lib/rancher/k3s/data/cni which gets populated later by K3s to symlinks like /var/lib/rancher/k3s/data/cni/bridge -> /var/lib/rancher/k3s/data/9ba85800b7128afafe1105efe9e7a1dac1fbb1c762c61fb2a99971e45e157779/bin/cni.
This approach works on a fresh Ubuntu 24.04 VM, but on a GitHub Actions's Ubuntu 24.04 worker, I see
level=info msg="CNI monitoring event CREATE \"/var/lib/rancher/k3s/data/cni/bridge\""
level=warning msg="Error validating CNI config file /etc/cni/net.d/10-crio-bridge.conflist: [failed to find plugin \"bridge\" in path [/var/lib/rancher/k3s/data/cni]]"
even if both the /var/lib/rancher/k3s/data/cni/bridge symlink and the target exist.
I have to restart crio.service for CRI-O to properly find the plugin. But doing that restart tends to cause k3s-io/k3s#12846.
What did you expect to happen?
I expected CRI-O to find the CNI plugin, especially since it clearly got the CNI monitoring event CREATE event, and since on a vanilla Ubuntu 24.04 VM that approach works.
How can we reproduce it (as minimally and precisely as possible)?
Add .github/workflows/upterm.yaml in some GitHub repo with content
name: CI
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Setup upterm session
uses: owenthereal/action-upterm@v1
Push commit with this workflow definition to GitHub.
Use ssh to log in to the GitHub Actions runner using the information displayed by the workflow job.
Once logged in to the runner, run
CRIO_VERSION=v1.32
curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key \
| gpg --dearmor | sudo tee /etc/apt/keyrings/cri-o-apt-keyring.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/ /" \
| sudo tee /etc/apt/sources.list.d/cri-o.list
sudo apt update
sudo apt install -y cri-o
sudo rm -f /etc/cni/net.d/87-podman-bridge.conflist
sudo cp /etc/cni/net.d/10-crio-bridge.conflist.disabled /etc/cni/net.d/10-crio-bridge.conflist
( echo '[crio.network]' ; echo 'plugin_dirs = [ "/var/lib/rancher/k3s/data/cni" ]' ) | sudo tee /etc/crio/crio.conf.d/20-cni.conf
sudo systemctl start crio.service
to install and configure CRI-O.
Check with
sudo journalctl -l | grep -i cni
that our bridge configuration was found by CRI-O and that the plugin was not found because it does not exist yet:
level=warning msg="Error validating CNI config file /etc/cni/net.d/10-crio-bridge.conflist: [failed to find plugin \"bridge\" in path [/var/lib/rancher/k3s/data/cni]]"
Then install K3s:
export INSTALL_K3S_EXEC='--container-runtime-endpoint /var/run/crio/crio.sock'
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig ~/.kube/config --write-kubeconfig-group $( id -g ) --write-kubeconfig-mode 640 --disable=traefik --disable=metrics-server
Check that the plugin exists
$ ls -la /var/lib/rancher/k3s/data/cni/bridge
lrwxrwxrwx 1 root root 98 Aug 29 18:33 /var/lib/rancher/k3s/data/cni/bridge -> /var/lib/rancher/k3s/data/9ba85800b7128afafe1105efe9e7a1dac1fbb1c762c61fb2a99971e45e157779/bin/cni
Run
sudo journalctl -l | grep -i cni
again. See
level=info msg="CNI monitoring event CREATE \"/var/lib/rancher/k3s/data/cni/bridge\""
level=warning msg="Error validating CNI config file /etc/cni/net.d/10-crio-bridge.conflist: [failed to find plugin \"bridge\" in path [/var/lib/rancher/k3s/data/cni]]"
Anything else we need to know?
No response
CRI-O and Kubernetes version
$ crio --version
crio version 1.32.1
GitCommit: 217bc2fe5c7b94e217d13c1c3809922c76f6107d
GitCommitDate: 2025-02-03T21:05:42Z
GitTreeState: dirty
BuildDate: 1970-01-01T00:00:00Z
GoVersion: go1.23.3
Compiler: gc
Platform: linux/amd64
Linkmode: static
BuildTags:
static
netgo
osusergo
exclude_graphdriver_btrfs
seccomp
apparmor
selinux
exclude_graphdriver_devicemapper
LDFlags: unknown
SeccompEnabled: true
AppArmorEnabled: false$ kubectl version --output=json
{
"clientVersion": {
"major": "1",
"minor": "33",
"gitVersion": "v1.33.4",
"gitCommit": "74cdb4273add43f53ddcad2de8ea9fd93c810dc4",
"gitTreeState": "clean",
"buildDate": "2025-08-14T18:51:53Z",
"goVersion": "go1.24.6 X:systemcrypto",
"compiler": "gc",
"platform": "linux/amd64"
},
"kustomizeVersion": "v5.6.0",
"serverVersion": {
"major": "1",
"minor": "33",
"emulationMajor": "1",
"emulationMinor": "33",
"minCompatibilityMajor": "1",
"minCompatibilityMinor": "32",
"gitVersion": "v1.33.4+k3s1",
"gitCommit": "148243c49519922720fe1b340008dbce8fb02516",
"gitTreeState": "clean",
"buildDate": "2025-08-25T16:59:07Z",
"goVersion": "go1.24.5",
"compiler": "gc",
"platform": "linux/amd64"
}
}OS version
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
$ uname -a
Linux pkrvmccyg1gnepe 6.11.0-1018-azure #18~24.04.1-Ubuntu SMP Sat Jun 28 04:46:03 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux