[Traefik] Rootless Docker with slirp4netns doesn't show user IP #45337

aureliensoria · 2023-03-02T17:36:24Z

aureliensoria
Mar 2, 2023

Description

When using rootless docker with slip4netns an internal IP is shown instead of the real one.
Example:

Hostname: 44220974d172
IP: 127.0.0.1
IP: 172.20.0.18
RemoteAddr: 172.20.0.7:33402
GET / HTTP/1.1
Host: XXX
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US;q=0.7,en;q=0.3
Dnt: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Te: trailers
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 172.20.0.1
X-Forwarded-Host: XXX
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 7770b7ffda5a
X-Real-Ip: 172.20.0.1

Reproduce

Add
`[Service]
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=slirp4netns"

in ~/.config/systemd/user/docker.service.d/override.conf `
Reload the docker daemon and restart it.

Run two containers : traefik and traefik/whoami instance.

Expected behavior

X-Real-Ip: Should contain the real user IP instead of the IP of the docker network gateway.

docker version

Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:46:54 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:46:54 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.0
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       slirp4netns
  StateDir:         /tmp/rootlesskit3603596150
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 29
  Running: 29
  Paused: 0
  Stopped: 0
 Images: 29
 Server Version: 23.0.1
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.1.0-5-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 15.63GiB
 Name: XXX
 ID: X2LR:NAD7:AJMU:YOIV:TZY3:EPGN:VRCK:ZU3A:VVGV:M23Y:PFX2:B5PH
 Docker Root Dir: /home/virt/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional Info

No response

Answered by aureliensoria

Apr 26, 2025

Hello there !
The solution I found to this issue is to set traefik ports mode to "host" in my docker compose:

  traefik:
    image: traefik:latest
    container_name: traefik
    restart: always
    ports:
      - target: 80
        published: 80
        mode: host
      - target: 443
        published: 443
        mode: host
      - target: 8448
        published: 8448
        mode: host
    volumes:
      - "/data/cloud/traefik/:/etc/traefik/"
      - "/run/user/1001/docker.sock:/var/run/docker.sock:ro"
    labels:
      - "traefik.http.routers.traefik.rule=Host(`traefik.DOMAIN_NAME`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.traefik.service=api@…

View full answer

aureliensoria · 2023-03-02T17:37:23Z

aureliensoria
Mar 2, 2023
Author

In addition here is the slirp4netns launch options

ps aux | grep slirp
virt     2675625  0.0  0.1 1311380 20948 ?       Ssl  18:25   0:00 rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=slirp4netns --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
virt     2675636  0.0  0.1 1163324 19712 ?       Sl   18:25   0:00 /proc/self/exe --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=slirp4netns --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
virt     2675654  0.3  0.0  14216  4516 ?        S    18:25   0:02 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --api-socket /tmp/rootlesskit3603596150/.s4nn.sock --enable-sandbox --enable-seccomp 2675636 tap0

Thank you !

0 replies

danishprakash · 2023-03-06T12:26:57Z

danishprakash
Mar 6, 2023

Mind If I take a look at this?

0 replies

AkihiroSuda · 2023-03-09T14:04:41Z

AkihiroSuda
Mar 9, 2023
Maintainer

Run two containers : traefik and traefik/whoami instance.

Please provide a full reproducer.
Also wondering if this is reproducible with a simple docker run -p 8080:80 nginx. (Not reproducible for me)

Mind If I take a look at this?

You don't need to ask for a permission 🙂

0 replies

aureliensoria · 2023-03-09T14:13:34Z

aureliensoria
Mar 9, 2023
Author

When using docker run -p 8080:80 nginx, I can't reproduce my issue. I will provide you my full docker-compose ASAP.

0 replies

aureliensoria · 2023-03-09T14:23:07Z

aureliensoria
Mar 9, 2023
Author

Here is a dump of my traefik configuration. It was working well when using standard docker instead of rootless docker.

version: '3.4'

services:

  # External dependencies
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: always
    ports:
      - "SERVER_IP:80:80"
      - "SERVER_IP:443:443"
      - "SERVER_IP:9080:8080"
    volumes:
      - "/data/cloud/traefik/:/etc/traefik/"
      - "/run/user/1001/docker.sock:/var/run/docker.sock:ro"
    labels:
      - "traefik.http.routers.traefik.rule=Host(`traefik.DOMAIN_NAME`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.tls=true"
      - "traefik.http.routers.traefik.tls.certresolver=myresolver"
      - "traefik.http.routers.traefik.middlewares=sso@docker,https_headers@file"
    networks:
      - default
      - traefik

  whoami:
    image: traefik/whoami
    restart: always
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.who.rule=Host(`who.DOMAIN_NAME`) || Host(`who.DOMAIN_NAME2`)"
      - "traefik.http.routers.who.service=svc_who"
      - "traefik.http.routers.who.tls=true"
      - "traefik.http.routers.who.tls.certresolver=myresolver"
      - "traefik.http.routers.who.middlewares=admin@file,https_headers@file"
      - "traefik.http.services.svc_who.loadbalancer.server.port=80"
    networks:
      - default
      - traefik

Traefik configuration

[global]
  sendAnonymousUsage = false
  checkNewVersion = true

[log]
  level = "ERROR"
  filePath = "/etc/traefik/traefik.log"

[accessLog]
  filePath = "/etc/traefik/access.log"
    [accessLog.fields.headers]
      defaultMode = "keep"
      [accessLog.fields.headers.names]
        "ClientAddr" = "keep"

[entryPoints]
  [entryPoints.web]
      address = ":80"
      [entryPoints.web.http.redirections]
        [entryPoints.web.http.redirections.entryPoint]
          to = "websecure"

  [entryPoints.websecure]
      address = ":443"

[serversTransport]
  insecureSkipVerify = true

[providers]
  docker = true
  [providers.file]
    directory = "/etc/traefik/"
    watch = true

[api]
  dashboard = true

[tls.options]
  [tls.options.default]
    minVersion = "VersionTLS12"
  [tls.options.mintls13]
    minVersion = "VersionTLS13"

[certificatesResolvers.myresolver.acme]
  email = "admin@DOMAIN_NAME"
  storage = "/etc/traefik/acme.json"
  [certificatesResolvers.myresolver.acme.httpChallenge]
    # used during the challenge
    entryPoint = "web"

0 replies

sam-thibault · 2023-04-14T09:59:25Z

sam-thibault
Apr 14, 2023

Given that docker run works as expected, this seems to be a traefik specific issue. I will move to a discussion.

0 replies

Georgekonsec · 2023-06-06T14:17:51Z

Georgekonsec
Jun 6, 2023

Actually this does not seem to be a traefik specific issue. The problem exists if you run haproxy, httpd or nginx applications and it is related how you run the docker container.

It works if you run the following docker container:

docker run --rm -d --name without_ip -p 7000:80 httpd
docker logs -f without_ip

# Then curl the server to that port
curl <hostname>:7000

but it returns docker internal IP, if you run the same docker container with a Server IP:

docker run --rm -d --name with_ip -p <server_ip>:7001:80 httpd
docker logs -f with_ip

# Then curl the server to that port
curl <hostname>:7001

We would like to be able to expose an application to a specific Server ip and not to all of them (0.0.0.0).

After checking slirp4netns there is an option (--outbound-addr) to define preferred outbound ipv4/ipv6 addresses or interface names.
Could we provide an environment variable for dockerd-rootless.sh script like DOCKERD_ROOTLESS_ROOTLESSKIT_OUTBOUND_ADDR so to be able to define an ip address/network interface ?

Could we use/check pasta network driver because it handles differently the network interfaces and this probably fixes the exact problem with the Docker Internal IP instead of the client IP.

0 replies

Xyz00777 · 2025-04-25T10:48:48Z

Xyz00777
Apr 25, 2025

having the same problem at the moment with an home assistant container which gets started with an docker compose.
did someone have a solution for this problem beside starting the container outside and docker compose?

4 replies

aureliensoria Apr 26, 2025
Author

Hello there !
The solution I found to this issue is to set traefik ports mode to "host" in my docker compose:

  traefik:
    image: traefik:latest
    container_name: traefik
    restart: always
    ports:
      - target: 80
        published: 80
        mode: host
      - target: 443
        published: 443
        mode: host
      - target: 8448
        published: 8448
        mode: host
    volumes:
      - "/data/cloud/traefik/:/etc/traefik/"
      - "/run/user/1001/docker.sock:/var/run/docker.sock:ro"
    labels:
      - "traefik.http.routers.traefik.rule=Host(`traefik.DOMAIN_NAME`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.entrypoints=web,websecure"
      - "traefik.http.routers.traefik.tls=true"
      - "traefik.http.routers.traefik.tls.certresolver=myresolver"
      - "traefik.http.routers.traefik.middlewares=sso@docker,https_headers@file"
    networks:
      default:
      traefik:

You might want try the same kind of configuration to your home-assistant container if it is exposed directly or to your reverse proxy like in my case if you have one.

Answer selected by aureliensoria

adoyle-h Apr 26, 2025

I tried set traefik ports mode to "host". But the traefik access logs still show the container ip, and X-Real-Ip header is 172.20.0.1. @aureliensoria

aureliensoria Apr 26, 2025
Author

If you start a traefik/whoami container, what IP do you see in the X-Forwarded-For and X-Real-Ip fields ?

Here is my whoami configuration for example

  whoami:
    image: traefik/whoami
    restart: always
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.who.rule=Host(`who.DOMAIN_NAME`) || Host(`who.DOMAIN_NAME2`)"
      - "traefik.http.routers.who.service=svc_who"
      - "traefik.http.routers.who.tls=true"
      - "traefik.http.routers.who.tls.certresolver=myresolver"
      - "traefik.http.routers.who.middlewares=admin@file,https_headers@file"
      - "traefik.http.services.svc_who.loadbalancer.server.port=80"
    networks:
      - default
      - traefik

adoyle-h May 3, 2025

Both X-Forwarded-For and X-Real-Ip fields are 172.20.0.1 when I visit the whoami service.

eriksjolund · 2025-04-26T12:58:44Z

eriksjolund
Apr 26, 2025

Side note

Here is another solution to the problem: using socket activation of containers because
nginx, traefik and caddy support socket activation. The socket is created by systemd on the host network.
The real source IP address is therefore seen by the HTTP reverse proxy container. Slirp4netns (or Pasta) do not handle this
socket at all so there will not be any IP address translation.

nerdctl and moby have feature requests

Ther are feature requests for adding support for passing file descriptors to the container:

If this would be implemented then it should be possible to use socket activation to get the real source IP address.
But docker-compose does not support socket activation of containers so you would need to create systemd service units that specify

ExecStart=nerdctl run ...

podman already supports socket activation

Alternatively you could use podman with quadlet file. That already works today. See socket activation of containers.
You could convert the compose file to quadlet files with podlet. I wrote some some tips in HTTP reverse proxy about how to connect from a container on the custom network to the proxy (with the help of NetworkAlias=whoami.example.com or AddHost=whoami.example.com:host-gateway)

0 replies

[Traefik] Rootless Docker with slirp4netns doesn't show user IP #45337

Uh oh!

Uh oh!

Description

Reproduce

Expected behavior

docker version

docker info

Additional Info

Replies: 9 comments · 4 replies

Uh oh!

Uh oh!

aureliensoria Mar 2, 2023 Author

Uh oh!

Uh oh!

AkihiroSuda Mar 9, 2023 Maintainer

Uh oh!

aureliensoria Mar 9, 2023 Author

Uh oh!

Uh oh!

aureliensoria Mar 9, 2023 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aureliensoria Apr 26, 2025 Author

Uh oh!

Uh oh!

Uh oh!

aureliensoria Apr 26, 2025 Author

Uh oh!

Uh oh!

nerdctl and moby have feature requests

podman already supports socket activation

Replies: 9 comments 4 replies

aureliensoria
Mar 2, 2023
Author

AkihiroSuda
Mar 9, 2023
Maintainer

aureliensoria
Mar 9, 2023
Author

aureliensoria
Mar 9, 2023
Author

aureliensoria Apr 26, 2025
Author

aureliensoria Apr 26, 2025
Author