Multiple issues with parallel processing in latest Docker Desktop for Windows #14889
Description
Description
We have a compose project with about 100 services defined. Most of them are .NET based services that need to be compiled. When we run docker compose for the full project, bake takes over, converts the compose project and triggers building of all services. But none of the available methods to limit parallel building of the images work anymore and all images are built at the same time. When I look at the Docker Desktop UI in the Builds section I can see all 100 build tasks active at the same time. I use Docker Desktop on my moderately equipped notebook (6 cores, 32 GB memory) which is just not capable of handling over 60 parallel .NET builds. Most of the times the build fails noting that a job couldn't be found.
I've tried every option I could find to limit parallel processing but nothing worked:
- environment variable COMPOSE_PARALLEL_LIMIT was working fine before bake was used as the builder
- environment variable BUILDX_BUILD_MAX_PARALLELISM doesn't work (found somewhere on the Internet)
- running docker compose --parallel=4 does not limit the number of parallel builds
- tried a custom buildx builder with a custom configuration file but that configuration file is not applied
- created a configuration file according to https://docs.docker.com/build/buildkit/toml-configuration/
- created the builder
docker buildx create --name my-lovely-builder --driver docker-container --config .\builder.toml - started the builder
- within the builder container the configuration file looks like this (no longer matches the documentation)
- in the Docker Desktop Build UI the builder is shown like this
- this already indicates that the builder configuration is ignored (too many supported platforms)
- activated the builder via the environment variable BUILDX_BUILDER and restarted Docker Desktop and my PowerShell session
- ran docker compose build for the whole project, but all all images are still built in parallel, although the custom builder is used accoding to the Active Builds list in the Docker Desktop UI
- adjusted the configuration file in the builder container to match my original file
- restart everything again
- ran docker compose build for the whole project, but all all images are still build in parallel on the custom builder
- created a configuration file according to https://docs.docker.com/build/buildkit/toml-configuration/
How am I supposed to limit the parallelisation in Docker Desktop for Windows with the bake backend? Or ist there a ways to disable bake but keeping buildkit?
Triggering 100 builds in parallel on a system with 6 available cores is a strange default behavior. Aside from the CPU load there might also be networking issues pulling a huge amount of Docker images, Nuget or NPM packages at the same time.
Reproduce
- create docker compose project with lots of services
- run docker compose build
- watch active builds list in Docker Desktop UI
Expected behavior
- at least one parallelisation setting to take effect
- parallelisation is limitted based on the available cores (e.g. two times the number of cores, e.g. 12 builds a a time in my case)
docker version
Client:
Version: 28.3.0
API version: 1.51
Go version: go1.24.4
Git commit: 38b7060
Built: Tue Jun 24 15:44:57 2025
OS/Arch: windows/amd64
Context: desktop-linux
Server: Docker Desktop 4.43.1 (198352)
Engine:
Version: 28.3.0
API version: 1.51 (minimum version 1.24)
Go version: go1.24.4
Git commit: 265f709
Built: Tue Jun 24 15:44:25 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.27
GitCommit: 05044ec0a9a75232cad458027ca83437aae3f4da
runc:
Version: 1.2.5
GitCommit: v1.2.5-0-g59923ef
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client:
Version: 28.3.0
Context: desktop-linux
Debug Mode: false
Plugins:
ai: Docker AI Agent - Ask Gordon (Docker Inc.)
Version: v1.6.0
Path: C:\Program Files\Docker\cli-plugins\docker-ai.exe
buildx: Docker Buildx (Docker Inc.)
Version: v0.25.0-desktop.1
Path: C:\Program Files\Docker\cli-plugins\docker-buildx.exe
cloud: Docker Cloud (Docker Inc.)
Version: v0.4.2
Path: C:\Program Files\Docker\cli-plugins\docker-cloud.exe
compose: Docker Compose (Docker Inc.)
Version: v2.38.1-desktop.1
Path: C:\Program Files\Docker\cli-plugins\docker-compose.exe
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.41
Path: C:\Program Files\Docker\cli-plugins\docker-debug.exe
desktop: Docker Desktop commands (Docker Inc.)
Version: v0.1.11
Path: C:\Program Files\Docker\cli-plugins\docker-desktop.exe
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.29
Path: C:\Program Files\Docker\cli-plugins\docker-extension.exe
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.4.0
Path: C:\Program Files\Docker\cli-plugins\docker-init.exe
mcp: Docker MCP Plugin (Docker Inc.)
Version: v0.9.3
Path: C:\Program Files\Docker\cli-plugins\docker-mcp.exe
model: Docker Model Runner (EXPERIMENTAL) (Docker Inc.)
Version: v0.1.32
Path: C:\Program Files\Docker\cli-plugins\docker-model.exe
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: C:\Program Files\Docker\cli-plugins\docker-sbom.exe
scout: Docker Scout (Docker Inc.)
Version: v1.18.1
Path: C:\Program Files\Docker\cli-plugins\docker-scout.exe
Server:
Containers: 5
Running: 2
Paused: 0
Stopped: 3
Images: 975
Server Version: 28.3.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Discovered Devices:
cdi: docker.com/gpu=webgpu
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
runc version: v1.2.5-0-g59923ef
init version: de40ad0
Security Options:
seccomp
Profile: builtin
Kernel Version: 5.15.153.1-microsoft-standard-WSL2
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 23.47GiB
Name: docker-desktop
ID: 276c57ef-9c88-4be6-8ecb-4952371caab6
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=npipe://\\.\pipe\docker_cli
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
::1/128
127.0.0.0/8
Live Restore Enabled: false
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: DOCKER_INSECURE_NO_IPTABLES_RAW is setDiagnostics ID
B4C7AFE0-817D-4018-A906-F6ECE15BF7C2/20250723084913
Additional Info
No response