Initialization of k3s scheduler fails #7573
Description
Description of problem
Running the following:
dokku scheduler-k3s:initialize --server-ip AAA.BBB.CCC.DD
results in this error:
Deployment is not ready: keda/keda-add-ons-http-interceptor. 1 out of 3 expected pods are ready
! Unable to install helm charts: Error installing chart keda-add-ons-http: Error deploying: client rate limiter Wait returned an error: context deadline exceeded
Steps to reproduce
ufw disable
dokku scheduler-k3s:initialize --server-ip AAA.BBB.CCC.DD
dokku report $APP_NAME
before any app is created
Additional information
dokku version 0.35.15
-----> uname: Linux dokkuvm 6.1.0-31-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07) x86_64 GNU/Linux
-----> memory:
total used free shared buff/cache available
Mem: 1966 1675 87 4 387 290
Swap: 974 475 499
-----> disk utilization:
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 19G 8.7G 9.0G 50% /
-----> disk inode utilization:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda1 1.2M 143K 1.1M 12% /
-----> docker version:
Client: Docker Engine - Community
Version: 28.0.1
API version: 1.48
Go version: go1.23.6
Git commit: 068a01e
Built: Wed Feb 26 10:41:16 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 28.0.1
API version: 1.48 (minimum version 1.24)
Go version: go1.23.6
Git commit: bbd0a17
Built: Wed Feb 26 10:41:16 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.25
GitCommit: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc:
Version: 1.2.4
GitCommit: v1.2.4-0-g6c52b3f
docker-init:
Version: 0.19.0
GitCommit: de40ad0
-----> docker daemon info:
Client: Docker Engine - Community
Version: 28.0.1
Context: default
Debug Mode: true
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.21.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.33.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 28.0.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc version: v1.2.4-0-g6c52b3f
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.0-31-amd64
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.921GiB
Name: dokkuvm
ID: bcd2ee76-c01d-42ab-9f8e-8d83c5b0abb7
Docker Root Dir: /var/lib/docker
Debug Mode: false
File Descriptors: 24
Goroutines: 43
System Time: 2025-03-08T12:44:22.294184132Z
EventsListeners: 1
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
-----> herokuish version:
herokuish: v0.10.3
buildpacks:
heroku-buildpack-multi v1.2.0
heroku-buildpack-ruby v285
heroku-buildpack-nodejs v273
heroku-buildpack-clojure v91
heroku-buildpack-python v267
heroku-buildpack-java v74
heroku-buildpack-gradle v40
heroku-buildpack-scala v98
heroku-buildpack-php v259
heroku-buildpack-go v200
heroku-buildpack-nginx v26
buildpack-null v3
-----> dokku version: dokku version 0.35.15
-----> dokku-event-listener version: v0.17.1
-----> dokku-update version: dokku-update v0.9.6
-----> docker-container-healthchecker version: v0.11.4
-----> docker-image-labeler version: v0.8.1
-----> git version: git version 2.39.5
-----> lambda-builder version: v0.8.0
-----> netrc version: v0.10.1
! pack binary is not available
-----> plugn version: plugn: v0.16.0
-----> sigil version: v0.11.1
-----> sshcommand version: sshcommand v0.19.0
-----> dokku plugins:
00_dokku-standard 0.35.15 enabled dokku core standard plugin
20_events 0.35.15 enabled dokku core events logging plugin
app-json 0.35.15 enabled dokku core app-json plugin
apps 0.35.15 enabled dokku core apps plugin
builder 0.35.15 enabled dokku core builder plugin
builder-dockerfile 0.35.15 enabled dokku core builder-dockerfile plugin
builder-herokuish 0.35.15 enabled dokku core builder-herokuish plugin
builder-lambda 0.35.15 enabled dokku core builder-lambda plugin
builder-nixpacks 0.35.15 enabled dokku core builder-nixpacks plugin
builder-null 0.35.15 enabled dokku core builder-null plugin
builder-pack 0.35.15 enabled dokku core builder-pack plugin
buildpacks 0.35.15 enabled dokku core buildpacks plugin
caddy-vhosts 0.35.15 enabled dokku core caddy-vhosts plugin
certs 0.35.15 enabled dokku core certificate management plugin
checks 0.35.15 enabled dokku core checks plugin
common 0.35.15 enabled dokku core common plugin
config 0.35.15 enabled dokku core config plugin
cron 0.35.15 enabled dokku core cron plugin
docker-options 0.35.15 enabled dokku core docker-options plugin
domains 0.35.15 enabled dokku core domains plugin
enter 0.35.15 enabled dokku core enter plugin
git 0.35.15 enabled dokku core git plugin
haproxy-vhosts 0.35.15 enabled dokku core haproxy-vhosts plugin
logs 0.35.15 enabled dokku core logs plugin
network 0.35.15 enabled dokku core network plugin
nginx-vhosts 0.35.15 enabled dokku core nginx-vhosts plugin
openresty-vhosts 0.35.15 enabled dokku core openresty-vhosts plugin
plugin 0.35.15 enabled dokku core plugin plugin
ports 0.35.15 enabled dokku core ports plugin
proxy 0.35.15 enabled dokku core proxy plugin
ps 0.35.15 enabled dokku core ps plugin
registry 0.35.15 enabled dokku core registry plugin
repo 0.35.15 enabled dokku core repo plugin
resource 0.35.15 enabled dokku core resource plugin
run 0.35.15 enabled dokku core run plugin
scheduler 0.35.15 enabled dokku core scheduler plugin
scheduler-docker-local 0.35.15 enabled dokku core scheduler-docker-local plugin
scheduler-k3s 0.35.15 enabled dokku core scheduler-k3s plugin
scheduler-null 0.35.15 enabled dokku core scheduler-null plugin
shell 0.35.15 enabled dokku core shell plugin
ssh-keys 0.35.15 enabled dokku core ssh-keys plugin
storage 0.35.15 enabled dokku core storage plugin
trace 0.35.15 enabled dokku core trace plugin
traefik-vhosts 0.35.15 enabled dokku core traefik-vhosts plugin
Output of failing commands after running: dokku trace:off
[...]
Deployment is not ready: keda/keda-add-ons-http-interceptor. 1 out of 3 expected pods are ready
! Unable to install helm charts: Error installing chart keda-add-ons-http: Error deploying: client rate limiter Wait returned an error: context deadline exceededOutput of failing commands after running: dokku trace:on
same as above