[Bug]: Watchtower got lost during scheduled start after socket-proxy container was updated

[bondskin]

Bug Report Checklist

• I checked existing issues and found no duplicates.
• I have reviewed Watchtower's documentation and confirmed compliance with documented configuration specifications.
• I have restarted Docker, re-pulled the latest image, and confirmed that the issue persists.

Bug Description

I updated yesterday (13-02-2026) manually the socket proxy container which is used by Watchtower, this went without issues.
Tonight, Watchtower got heavily stuck and I got tons of debug logs but I have no clue what happened.
I saved the log file, restarted the Watchtower container and everything is working fine.

Is it recommended to start Watchtower after socket-proxy update?

See some logs below, I have the whole file available, if needed.

Steps to Reproduce

unknown

Expected Behavior

smooth sailing

Watchtower Version

latest

Docker Version

docker version
Client: Docker Engine - Community
 Version:           29.2.1
 API version:       1.53
 Go version:        go1.25.6
 Git commit:        a5c7197
 Built:             Mon Feb  2 17:17:31 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.2.1
  API version:      1.53 (minimum version 1.44)
  Go version:       go1.25.6
  Git commit:       6bc6209
  Built:            Mon Feb  2 17:17:31 2026
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v2.2.1
  GitCommit:        dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker Info (optional)

docker info
Client: Docker Engine - Community
 Version:    29.2.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.31.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.0.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  model: Docker Model Runner (Docker Inc.)
    Version:  v1.0.12
    Path:     /usr/libexec/docker/cli-plugins/docker-model

Server:
 Containers: 73
  Running: 69
  Paused: 0
  Stopped: 4
 Images: 88
 Server Version: 29.2.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.69+deb13-amd64
 Operating System: Debian GNU/Linux 13 (trixie)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.37GiB
 Name: mcp
 ID: fa9d03ad-1d05-468e-b253-09bfbc988011
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: true
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

[DEPRECATION NOTICE]: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
In future versions this will be a hard failure preventing the daemon from starting! Learn more at: https://docs.docker.com/go/api-security/

Docker Compose Configuration

---
services:
  watchtower:
    image: nickfedor/watchtower:latest
    container_name: watchtower
    hostname: mcp
    command: --stop-timeout 300s --debug 
    networks: 
       - watchtower_net
    depends_on:
      - socket-proxy-write
    volumes:
      - /etc/localtime:/etc/localtime:ro

    environment:
      - TZ=Europe/Berlin
      - DOCKER_HOST=tcp://socket-proxy-write:2375 
      - WATCHTOWER_SCHEDULE=0 5 3 * * *
      - WATCHTOWER_CLEANUP=true
      #- WATCHTOWER_POLL_INTERVAL=86400 
      - WATCHTOWER_NOTIFICATIONS=email
      - WATCHTOWER_INCLUDE_STOPPED=true
      - WATCHTOWER_INCLUDE_RESTARTING=true
      - WATCHTOWER_UPDATE_ON_START=true
      - WATCHTOWER_ROLLING_RESTART=false
    restart: always

    mem_limit: 64m
    cpus: "0.1"

  socket-proxy-write:
    image: lscr.io/linuxserver/socket-proxy:latest
    container_name: socket-proxy-write
    privileged: true
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    mem_limit: 64m
    cpus: "0.1"
    environment:
      - POST=1
      - ALLOW_RESTART=1
      - CONTAINERS=1
      - IMAGES=1
      - LOG_LEVEL=err
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    networks:
      - watchtower_net 

networks:
  watchtower_net:
    internal: false  
    name: watchtower_fixed_net

Docker CLI Command

Debug Logs

2026-02-14T02:05:00.830035020Z time="2026-02-14T03:05:00+01:00" level=debug msg="Inspecting container" container_id=d941608e28cbdfaf4763bc6560c860e9ea0d84260e19dff1ab73a94a3049ae46
2026-02-14T02:05:00.833784589Z time="2026-02-14T03:05:00+01:00" level=debug msg="Retrieved container and image info" container_id=d941608e28cbdfaf4763bc6560c860e9ea0d84260e19dff1ab73a94a3049ae46 image="sha256:79a17d79ff7bd818528f82040db1665d8f2309109bcd654451143dc30ee5b23d"
2026-02-14T02:05:00.833807812Z time="2026-02-14T03:05:00+01:00" level=debug msg="Created new container instance" container=dockpeek id=d941608e28cb image="sha256:79a17d79ff7bd818528f82040db1665d8f2309109bcd654451143dc30ee5b23d"
2026-02-14T02:05:00.833811832Z time="2026-02-14T03:05:00+01:00" level=debug msg="No filter applied" container=dockpeek

2026-02-14T02:05:00.852178148Z time="2026-02-14T03:05:00+01:00" level=debug msg="Label not found" container=traefik label=com.centurylinklabs.watchtower.depends-on
2026-02-14T02:05:00.852180244Z time="2026-02-14T03:05:00+01:00" level=debug msg="Label not found" container=speedtest-tracker label=com.centurylinklabs.watchtower.depends-on
2026-02-14T02:05:00.852182372Z time="2026-02-14T03:05:00+01:00" level=debug msg="Label not found" container=crowdsec label=com.centurylinklabs.watchtower.depends-on
2026-02-14T02:05:00.852186787Z time="2026-02-14T03:05:00+01:00" level=debug msg="Label not found" container=heimdall label=com.centurylinklabs.watchtower.depends-on

2026-02-14T02:46:49.788559186Z time="2026-02-14T03:46:49+01:00" level=info msg="Started linked container" container=bazarr id=50f8f48d3802 new_id=f2f783c9bc38
2026-02-14T02:46:49.788564790Z time="2026-02-14T03:46:49+01:00" level=debug msg="Started new container" container=bazarr image="lscr.io/linuxserver/bazarr:latest" new_id=f2f783c9bc38
2026-02-14T02:46:49.788568869Z time="2026-02-14T03:46:49+01:00" level=debug msg="Marked container as restarted" container_id=50f8f48d3802 name=bazarr
2026-02-14T02:46:49.788572371Z time="2026-02-14T03:46:49+01:00" level=debug msg="Restarted container" container=bazarr image="lscr.io/linuxserver/bazarr:latest"
2026-02-14T02:46:49.788575718Z time="2026-02-14T03:46:49+01:00" level=debug msg="Generating report" count=72
2026-02-14T02:46:49.788707292Z time="2026-02-14T03:46:49+01:00" level=debug msg="Categorizing container status" container=beszel-agent images_equal=true new_image=1c7fd94e7d9e old_image=1c7fd94e7d9e state_before_image_check=2
2026-02-14T02:46:49.788718615Z time="2026-02-14T03:46:49+01:00" level=debug msg="Setting container state to fresh due to equal images" container=beszel-agent new_state=5 old_state=2
2026-02-14T02:46:49.788722585Z time="2026-02-14T03:46:49+01:00" level=debug msg="Categorizing container status" container=kuma images_equal=true new_image=df8c3b1aceb7 old_image=df8c3b1aceb7 state_before_image_check=2
2026-02-14T02:46:49.788736207Z time="2026-02-14T03:46:49+01:00" level=debug msg="Setting container state to fresh due to equal images" container=kuma new_state=5 old_state=2
2026-02-14T02:46:49.788917295Z time="2026-02-14T03:46:49+01:00" level=debug msg="Categorizing container status" container=jellystat images_equal=true new_image=4f1399257ec0 old_image=4f1399257ec0 state_before_image_check=2

2026-02-14T02:46:59.921990516Z time="2026-02-14T03:46:59+01:00" level=debug msg="Log entry queued for batching" entries_count=82 fields.level=info legacy_template=true message="Removing image" notify=no
2026-02-14T02:46:59.921993998Z time="2026-02-14T03:46:59+01:00" level=info msg="Removing image" image_id=6fd3965e232f image_name="lscr.io/linuxserver/sonarr:latest" notify=yes
2026-02-14T02:47:00.453488628Z time="2026-02-14T03:47:00+01:00" level=debug msg="Cleaned up old image" image_id=6fd3965e232f image_name="lscr.io/linuxserver/sonarr:latest"
2026-02-14T02:47:00.453523687Z time="2026-02-14T03:47:00+01:00" level=debug msg="Image removal details" deleted="6fd3965e232f, c5076d568ca0, 61b969639768, 2fe40b2febba, ac40d3ce912b, 3c7a50e96f3f, 21b016d76280, ad1ffbc0df8c, fcb28ae1530b, 27c36e51b0c6" image_id=6fd3965e232f image_name="lscr.io/linuxserver/sonarr:latest" untagged="lscr.io/linuxserver/sonarr@sha256:6f73bbba3339"
2026-02-14T02:47:00.453528936Z time="2026-02-14T03:47:00+01:00" level=debug msg="Cleaned up old image" image_id=6fd3965e232f image_name="lscr.io/linuxserver/sonarr:latest"
2026-02-14T02:47:00.453531736Z time="2026-02-14T03:47:00+01:00" level=debug msg="Removed old image" image_id=6fd3965e232f image_name="lscr.io/linuxserver/sonarr:latest"
2026-02-14T02:47:00.453534291Z time="2026-02-14T03:47:00+01:00" level=debug msg="Attempting to remove image" container_id=cfd32aac3736604fbf0e5a8a59932d499835c54868bb829cccb0b194addcbd1f image_id="sha256:67698e9f0df1babef6cb4cbd5d6e55e3e8273919a56141dd512907eef540823e" image_name="ghcr.io/cleanuparr/cleanuparr:latest"
2026-02-14T02:47:00.453537182Z time="2026-02-14T03:47:00+01:00" level=debug msg="Log entry queued for batching" entries_count=83 fields.level=info legacy_template=true message="Removing image" notify=no
2026-02-14T02:47:00.453539715Z time="2026-02-14T03:47:00+01:00" level=info msg="Removing image" image_id=67698e9f0df1 image_name="ghcr.io/cleanuparr/cleanuparr:latest" notify=yes

Additional Context

No response

[nicholas-fedor]

Watchtower is currently designed to globally persist its client from startup.
In the situation that you described, it would be unsurprising to see Watchtower exhibit errant behavior.

Support for socket proxies has not been a traditional focal point in Watchtower's design (including in the upstream Containrrr version). Docker socket disconnection events, whether from the proxy restarting/disconnecting or even Docker updates not properly restarting the Watchtower container, currently require a user-initiated restart of Watchtower.

Resolving this will require adding detection and recovery mechanisms to appropriately handle this particular error state. In the meantime, restarting the Watchtower container is currently the solution to a disconnection between Watchtower and socket-proxy.

[bondskin]

Thank you for your thoughts and detailed explanation, @nicholas-fedor

It's clearly a pattern as I have updated the watchtower socket proxy without restarting watchtower itself (I forgot it) and the update last night did not went well.

Restarting watchtower myself after socket update is perfectly fine for me.
Thanks again