Docker Hub rate limiting while pulling BuildKit image during builder bootstrap

[steebchen]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

Every now and then, twice this week already, we are running into this:

Error: ERROR: Error response from daemon: Head "https://registry-1.docker.io/v2/moby/buildkit/manifests/buildx-stable-1": toomanyrequests: too many failed login attempts for username or IP address

It was either that or some 5xx server error.

full details + workflow here: https://github.com/theopenco/llmgateway/actions/runs/23655760618/job/68912490505

Expected behaviour

no error

Actual behaviour

it errors 💀

Repository URL

https://github.com/theopenco/llmgateway

Workflow run URL

https://github.com/theopenco/llmgateway/actions/runs/23655760618/job/68912490505

YAML workflow

name: images

on:
  release:
    types: [published]
  push:
    branches:
      - main
      - hotfix
  pull_request:
    paths:
      - "**/package.json"
      - "**/tsconfig.json"
      - "pnpm-lock.yaml"
      - "infra/**"
      - ".github/workflows/images.yml"

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      image_tag: ${{ steps.set-tag.outputs.IMAGE_TAG }}
      short_sha: ${{ steps.set-tag.outputs.SHORT_SHA }}
    steps:
      - name: Checkout
        uses: actions/checkout@v6

      - name: Set image tag
        id: set-tag
        run: |
          SHORT_SHA=$(git rev-parse --short HEAD)
          echo "SHORT_SHA=${SHORT_SHA}" >> $GITHUB_OUTPUT
          if [[ "${{ github.event_name }}" == "push" ]]; then
            echo "IMAGE_TAG=v0.0.0-${SHORT_SHA}" >> $GITHUB_OUTPUT
          elif [[ "${{ github.event_name }}" == "pull_request" ]]; then
            # For PRs, use a local tag that won't be pushed
            echo "IMAGE_TAG=pr-${SHORT_SHA}" >> $GITHUB_OUTPUT
          else
            # Validate that the tag only contains allowed characters
            if [[ ! "${{ github.event.release.tag_name }}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9_-]+)*$ ]]; then
              echo "Invalid tag name for Docker: ${{ github.event.release.tag_name }}"
              exit 1
            fi
            echo "IMAGE_TAG=${{ github.event.release.tag_name }}" >> $GITHUB_OUTPUT
          fi
  build-split:
    needs: setup
    if: |
      (github.event_name != 'pull_request' && needs.setup.result == 'success') ||
      (github.event_name == 'pull_request')
    runs-on: ${{ matrix.platform == 'linux/arm64' && 'ubuntu-24.04-arm' || 'ubuntu-latest' }}
    strategy:
      fail-fast: false
      matrix:
        app: [api, gateway, playground, ui, docs, worker, admin, code]
        platform: ${{ github.event_name == 'pull_request' && fromJSON('["linux/amd64"]') || fromJSON('["linux/amd64", "linux/arm64"]') }}
    permissions:
      contents: read
      packages: write
    steps:
      - name: Prepare
        run: |
          platform=${{ matrix.platform }}
          echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
      - name: Checkout
        uses: actions/checkout@v6

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v4

      - name: Login to GitHub Container Registry (main branch only)
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v6
        with:
          images: ghcr.io/${{ github.repository }}-${{ matrix.app }}

      - name: Build and conditionally push by digest
        id: build
        uses: docker/build-push-action@v7
        with:
          context: .
          file: infra/split.dockerfile
          platforms: ${{ matrix.platform }}
          labels: ${{ steps.meta.outputs.labels }}
          tags: ghcr.io/${{ github.repository }}-${{ matrix.app }}
          target: ${{ matrix.app }}
          outputs: ${{ github.event_name == 'pull_request' && 'type=image,push=false,load=true' || 'type=image,push-by-digest=true,name-canonical=true,push=true' }}
          cache-from: type=gha,scope=${{ matrix.app }}-${{ env.PLATFORM_PAIR }}
          cache-to: type=gha,mode=max,scope=${{ matrix.app }}-${{ env.PLATFORM_PAIR }}
          build-args: |
            APP_VERSION=${{ github.event_name == 'pull_request' && 'pr-test' || needs.setup.outputs.image_tag }}
      - name: Export digest (main branch only)
        if: github.event_name != 'pull_request'
        run: |
          mkdir -p ${{ runner.temp }}/digests
          digest="${{ steps.build.outputs.digest }}"
          touch "${{ runner.temp }}/digests/${digest#sha256:}"
      - name: Upload digest (main branch only)
        if: github.event_name != 'pull_request'
        uses: actions/upload-artifact@v7
        with:
          name: digests-${{ matrix.app }}-${{ env.PLATFORM_PAIR }}
          path: ${{ runner.temp }}/digests/*
          if-no-files-found: error
          retention-days: 1

  merge-split:
    runs-on: ubuntu-latest
    needs:
      - setup
      - build-split
    if: needs.setup.result == 'success' && github.event_name != 'pull_request'
    strategy:
      matrix:
        app: [api, gateway, playground, ui, docs, worker, admin, code]
    permissions:
      contents: read
      packages: write
    steps:
      - name: Download digests
        uses: actions/download-artifact@v8
        with:
          path: ${{ runner.temp }}/digests
          pattern: digests-${{ matrix.app }}-*
          merge-multiple: true

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v4

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v6
        with:
          images: ghcr.io/${{ github.repository }}-${{ matrix.app }}
          tags: |
            type=raw,value=${{ needs.setup.outputs.image_tag }}
            type=raw,value=latest
      - name: Create manifest list and push
        working-directory: ${{ runner.temp }}/digests
        run: |
          docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
            $(printf 'ghcr.io/${{ github.repository }}-${{ matrix.app }}@sha256:%s ' *)
      - name: Inspect image
        run: |
          docker buildx imagetools inspect ghcr.io/${{ github.repository }}-${{ matrix.app }}:${{ needs.setup.outputs.image_tag }}
  test-split:
    runs-on: ubuntu-latest
    needs:
      - setup
      - merge-split
      - build-split
    if: |
      always() &&
      (
        (github.event_name != 'pull_request' && needs.setup.result == 'success' && needs.merge-split.result == 'success') ||
        (github.event_name == 'pull_request' && needs.setup.result == 'success' && needs.build-split.result == 'success')
      )
    permissions:
      contents: read
      packages: read
    steps:
      - name: Checkout
        uses: actions/checkout@v6

      - name: Free up disk space
        run: |
          echo "Disk space before cleanup:"
          df -h
          # Remove unnecessary tools and files
          sudo rm -rf /usr/share/dotnet
          sudo rm -rf /usr/local/lib/android
          sudo rm -rf /opt/ghc
          sudo rm -rf /opt/hostedtoolcache/CodeQL
          # Clean up docker
          docker system prune -af --volumes
          echo "Disk space after cleanup:"
          df -h
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v4

      - name: Login to GitHub Container Registry (for non-PR builds)
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build all split Docker images for testing
        run: |
          # Build all split images sequentially
          for app in api gateway playground ui docs worker admin code; do
            echo "Building $app image..."
            docker build \
              -f infra/split.dockerfile \
              --target $app \
              --build-arg APP_VERSION=${{ github.event_name == 'pull_request' && 'pr-test' || needs.setup.outputs.image_tag }} \
              -t ghcr.io/${{ github.repository }}-$app:${{ github.event_name == 'pull_request' && 'pr-test' || needs.setup.outputs.image_tag }} \
              .
            # Clean up intermediate build layers to free space
            docker image prune -f
            echo "Disk space after building $app:"
            df -h / | tail -1
          done
      - name: Test split Docker images
        run: |
          chmod +x .github/test-split-docker.sh
          .github/test-split-docker.sh ghcr.io/${{ github.repository }} ${{ github.event_name == 'pull_request' && 'pr-test' || needs.setup.outputs.image_tag }}
  build-unified:
    needs: setup
    if: |
      (github.event_name != 'pull_request' && needs.setup.result == 'success') ||
      (github.event_name == 'pull_request')
    runs-on: ${{ matrix.platform == 'linux/arm64' && 'ubuntu-24.04-arm' || 'ubuntu-latest' }}
    strategy:
      fail-fast: false
      matrix:
        platform: ${{ github.event_name == 'pull_request' && fromJSON('["linux/amd64"]') || fromJSON('["linux/amd64", "linux/arm64"]') }}
    permissions:
      contents: read
      packages: write
    steps:
      - name: Prepare
        run: |
          platform=${{ matrix.platform }}
          echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
      - name: Checkout
        uses: actions/checkout@v6

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v4

      - name: Login to GitHub Container Registry (main branch only)
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v6
        with:
          images: ghcr.io/${{ github.repository }}-unified

      - name: Build and conditionally push by digest
        id: build
        uses: docker/build-push-action@v7
        with:
          context: .
          file: infra/unified.dockerfile
          platforms: ${{ matrix.platform }}
          labels: ${{ steps.meta.outputs.labels }}
          tags: ghcr.io/${{ github.repository }}-unified
          outputs: ${{ github.event_name == 'pull_request' && 'type=image,push=false,load=true' || 'type=image,push-by-digest=true,name-canonical=true,push=true' }}
          cache-from: type=gha,scope=unified-${{ env.PLATFORM_PAIR }}
          cache-to: type=gha,mode=max,scope=unified-${{ env.PLATFORM_PAIR }}
          build-args: |
            APP_VERSION=${{ github.event_name == 'pull_request' && 'pr-test' || needs.setup.outputs.image_tag }}
      - name: Export digest (main branch only)
        if: github.event_name != 'pull_request'
        run: |
          mkdir -p ${{ runner.temp }}/digests
          digest="${{ steps.build.outputs.digest }}"
          touch "${{ runner.temp }}/digests/${digest#sha256:}"
      - name: Upload digest (main branch only)
        if: github.event_name != 'pull_request'
        uses: actions/upload-artifact@v7
        with:
          name: digests-unified-${{ env.PLATFORM_PAIR }}
          path: ${{ runner.temp }}/digests/*
          if-no-files-found: error
          retention-days: 1

  merge-unified:
    runs-on: ubuntu-latest
    needs:
      - setup
      - build-unified
    if: needs.setup.result == 'success' && github.event_name != 'pull_request'
    permissions:
      contents: read
      packages: write
    steps:
      - name: Download digests
        uses: actions/download-artifact@v8
        with:
          path: ${{ runner.temp }}/digests
          pattern: digests-unified-*
          merge-multiple: true

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v4

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v6
        with:
          images: ghcr.io/${{ github.repository }}-unified
          tags: |
            type=raw,value=${{ needs.setup.outputs.image_tag }}
            type=raw,value=latest
      - name: Create manifest list and push
        working-directory: ${{ runner.temp }}/digests
        run: |
          docker buildx imagetools create $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
            $(printf 'ghcr.io/${{ github.repository }}-unified@sha256:%s ' *)
      - name: Inspect image
        run: |
          docker buildx imagetools inspect ghcr.io/${{ github.repository }}-unified:${{ needs.setup.outputs.image_tag }}
  test-unified:
    runs-on: ubuntu-latest
    needs:
      - setup
      - merge-unified
      - build-unified
    if: |
      always() &&
      (
        (github.event_name != 'pull_request' && needs.setup.result == 'success' && needs.merge-unified.result == 'success') ||
        (github.event_name == 'pull_request' && needs.setup.result == 'success' && needs.build-unified.result == 'success')
      )
    permissions:
      contents: read
      packages: read
    steps:
      - name: Checkout
        uses: actions/checkout@v6

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v4

      - name: Login to GitHub Container Registry (for non-PR builds)
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build unified Docker image for testing
        uses: docker/build-push-action@v7
        with:
          context: .
          file: infra/unified.dockerfile
          platforms: linux/amd64
          tags: test-unified-image:latest
          push: false
          load: true
          cache-from: type=gha,scope=unified-linux-amd64
          build-args: |
            APP_VERSION=${{ github.event_name == 'pull_request' && 'pr-test' || needs.setup.outputs.image_tag }}
      - name: Test unified Docker image
        run: |
          chmod +x .github/test-unified-docker.sh
          .github/test-unified-docker.sh test-unified-image:latest
  trigger-infra-update:
    runs-on: ubuntu-latest
    needs:
      - setup
      - merge-split
    if: github.event_name == 'push' && needs.merge-split.result == 'success'
    steps:
      - name: Trigger infra image update
        env:
          GH_TOKEN: ${{ secrets.GH_INFRA_TOKEN }}
          COMMIT_MSG: ${{ github.event.head_commit.message }}
        run: |
          FIRST_LINE=$(echo "$COMMIT_MSG" | head -n1)
          gh workflow run update-image.yml \
            --repo theopenco/llmgateway-infra \
            -f repository="${{ github.repository }}" \
            -f git_sha="${{ needs.setup.outputs.short_sha }}" \
            -f commit_message="$FIRST_LINE"

Workflow logs

Run docker/setup-buildx-action@v4
  with:
    driver: docker-container
    use: true
    keep-state: false
    cache-binary: true
    cleanup: true
  env:
    PLATFORM_PAIR: linux-amd64
Docker info
  /usr/bin/docker version
  Client: Docker Engine - Community
   Version:           28.0.4
   API version:       1.48
   Go version:        go1.23.7
   Git commit:        b8034c0
   Built:             Tue Mar 25 15:07:16 2025
   OS/Arch:           linux/amd64
   Context:           default
  
  Server: Docker Engine - Community
   Engine:
    Version:          28.0.4
    API version:      1.48 (minimum version 1.24)
    Go version:       go1.23.7
    Git commit:       6430e49
    Built:            Tue Mar 25 15:07:16 2025
    OS/Arch:          linux/amd64
    Experimental:     false
   containerd:
    Version:          v2.2.2
    GitCommit:        301b2dac98f15c27117da5c8af12118a041a31d9
   runc:
    Version:          1.3.4
    GitCommit:        v1.3.4-0-gd6d73eb8
   docker-init:
    Version:          0.19.0
    GitCommit:        de40ad0
  /usr/bin/docker info
  Client: Docker Engine - Community
   Version:    28.0.4
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.32.1
      Path:     /usr/libexec/docker/cli-plugins/docker-buildx
    compose: Docker Compose (Docker Inc.)
      Version:  v2.38.2
      Path:     /usr/libexec/docker/cli-plugins/docker-compose
  
  Server:
   Containers: 0
    Running: 0
    Paused: 0
    Stopped: 0
   Images: 6
   Server Version: 28.0.4
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: systemd
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
   Swarm: inactive
   Runtimes: runc io.containerd.runc.v2
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: 301b2dac98f15c27117da5c8af12118a041a31d9
   runc version: v1.3.4-0-gd6d73eb8
   init version: de40ad0
   Security Options:
    apparmor
    seccomp
     Profile: builtin
    cgroupns
   Kernel Version: 6.17.0-1008-azure
   Operating System: Ubuntu 24.04.4 LTS
   OSType: linux
   Architecture: x86_64
   CPUs: 4
   Total Memory: 15.61GiB
   Name: runnervmrg6be
   ID: 26c853c4-ebf5-403c-a446-44e39e017c08
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: githubactions
   Experimental: false
   Insecure Registries:
    ::1/128
    127.0.0.0/8
   Live Restore Enabled: false
  
Buildx version
  /usr/bin/docker buildx version
  github.com/docker/buildx v0.32.1 d3bfb3f4e48a67dda56e957a6636f4fab6c5fcb2
Inspecting default docker context
  [
    {
      "Name": "default",
      "Metadata": {},
      "Endpoints": {
        "docker": {
          "Host": "unix:///var/run/docker.sock",
          "SkipTLSVerify": false
        }
      },
      "TLSMaterial": {},
      "Storage": {
        "MetadataPath": "<IN MEMORY>",
        "TLSPath": "<IN MEMORY>"
      }
    }
  ]
Creating a new builder instance
  /usr/bin/docker buildx create --name builder-b988b152-692a-45d1-9f1d-6513238b6b38 --driver docker-container --buildkitd-flags --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host --use
  builder-b988b152-692a-45d1-9f1d-6513238b6b38
Booting builder
  /usr/bin/docker buildx inspect --bootstrap --builder builder-b988b152-692a-45d1-9f1d-6513238b6b38
  #1 [internal] booting buildkit
  #1 pulling image moby/buildkit:buildx-stable-1
  #1 pulling image moby/buildkit:buildx-stable-1 0.3s done
  #1 ERROR: Error response from daemon: Head "https://registry-1.docker.io/v2/moby/buildkit/manifests/buildx-stable-1": toomanyrequests: too many failed login attempts for username or IP address
  ------
   > [internal] booting buildkit:
  ------
  ERROR: Error response from daemon: Head "https://registry-1.docker.io/v2/moby/buildkit/manifests/buildx-stable-1": toomanyrequests: too many failed login attempts for username or IP address
Error: ERROR: Error response from daemon: Head "https://registry-1.docker.io/v2/moby/buildkit/manifests/buildx-stable-1": toomanyrequests: too many failed login attempts for username or IP address

BuildKit logs

Additional info

No response

[crazy-max]

This looks like the same class of problem as the retry requests already discussed in sibling actions, for example:

• Track workflow-level retry support for transient registry login failures login-action#750
• Intermittent failures on logging in with Google Artifact Registry login-action#885
• "You have reached your unauthenticated pull rate limit" After login with the action login-action#862
• Feature Request: Automated Recovery build-push-action#719
• The driver-opts of env.http_proxy can not work. build-push-action#1241

In this case the failure happens while the Docker daemon is pulling the BuildKit image during builder bootstrap:

Head "https://registry-1.docker.io/v2/moby/buildkit/manifests/buildx-stable-1": toomanyrequests

I don't think adding retry logic in setup-buildx-action is the right direction.

This is not really a Buildx setup feature, it's a generic workflow reliability problem. The same class of transient failures shows up across login, bootstrap, pulls, cache operations, auth, and pushes, depending on the runner, registry, and network path involved. Adding action-specific retry logic here would mean each action has to guess which errors are safe to retry across different registries and failure modes, which gets brittle quickly.

It's also the wrong abstraction layer. This kind of behavior belongs in GitHub runner/workflow primitives, for example first-class retry support for uses: steps, rather than each action inventing its own retry semantics. That would be reusable across actions, more consistent for users, and much more future proof.

For rate limiting specifically, action-local retries are not even obviously correct. Retrying from inside the action can just add more pressure and make the failure mode worse.

So I'd rather keep this tracked under the broader retry discussions above than add action-specific retry or backoff logic to this action.

[steebchen]

Fair enough I guess