Toolchain being resolved for a wrong platform in Bazel 8

[mariusz-wlodarczyk]

Description of the bug:

We have a situation where we're using a Go binary (courtesy of rules_go's go_binary) in files of a CC toolchain (as a wrapper for the actual compiler driver as it's a proprietary toolchain with some non-standard behaviors, e.g. licensing requirements).

It looks like this:

# toolchains/wrapper/BUILD
go_binary(
    name = "wrapper",
    embed = [":wrapper_lib"],
    visibility = ["//visibility:public"],
)

# toolchains/custom_toolchain/BUILD
toolchain(
    name = "toolchain",
    exec_compatible_with = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    target_compatible_with = [
        "//:custom_cpu",
    ],
    toolchain = ":cc_toolchain",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
    tags = ["manual"],
)

filegroup(
    name = "all_files",
    srcs = [
        "//toolchains/wrapper",
    ],
    tags = ["manual"],
)

cc_toolchain(
    name = "cc_toolchain",
    all_files = ":all_files",
    ar_files = ":all_files",
    compiler_files = ":all_files",
    dwp_files = ":all_files",
    linker_files = ":all_files",
    objcopy_files = ":all_files",
    strip_files = ":all_files",
    tags = ["manual"],
    toolchain_config = ":cc_toolchain_config",
)

cc_toolchain_config(
    name = "cc_toolchain_config",
    driver_executable = "//toolchains/wrapper",
    tags = ["manual"],
)

# BUILD
constraint_value(
    name = "custom_cpu",
    constraint_setting = "@platforms//cpu:cpu",
)

platform(
    name = "custom_platform",
    constraint_values = [
        ":custom_cpu",
    ],
)

cc_library(
    name = "my_lib",
    srcs = ["lib.c"],
)

So, the thing is that with Bazel 7 (any minor version) the Go toolchain used for building the go_binary target is selected for the host platform (that is, when I select a different target platform with --platforms):

$ bazel build :my_lib --platforms=:custom_platform --toolchain_resolution_debug='.*'
...
INFO: ToolchainResolution: Target platform //:custom_platform: Selected execution platform @@platforms//host:host,
INFO: ToolchainResolution: Performing resolution of @@rules_go~//go:toolchain for target platform @@platforms//host:host
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_aix_ppc64-impl; mismatching values: aix, ppc
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_android_386-impl; mismatching values: android, x86_32
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_android_amd64-impl; mismatching values: android
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_android_arm-impl; mismatching values: android, armv7
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_android_arm64-impl; mismatching values: android, aarch64
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_darwin_386-impl; mismatching values: osx, x86_32
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_darwin_amd64-impl; mismatching values: osx
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_darwin_arm-impl; mismatching values: osx, armv7
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_darwin_arm64-impl; mismatching values: osx, aarch64
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_dragonfly_amd64-impl; mismatching values: dragonfly
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_freebsd_386-impl; mismatching values: freebsd, x86_32
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_freebsd_amd64-impl; mismatching values: freebsd
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_freebsd_arm-impl; mismatching values: freebsd, armv7
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_freebsd_arm64-impl; mismatching values: freebsd, aarch64
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_illumos_amd64-impl; mismatching values: illumos
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_ios_amd64-impl; mismatching values: ios
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_ios_arm64-impl; mismatching values: ios, aarch64
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_js_wasm-impl; mismatching values: js, wasm
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_linux_386-impl; mismatching values: x86_32
      ToolchainResolution:   Toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_linux_amd64-impl is compatible with target plaform, searching for execution platforms:
      ToolchainResolution:     Compatible execution platform @@platforms//host:host
      ToolchainResolution:   All execution platforms have been assigned a @@rules_go~//go:toolchain toolchain, stopping
      ToolchainResolution: Recap of selected @@rules_go~//go:toolchain toolchains for target platform @@platforms//host:host:
      ToolchainResolution:   Selected @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_linux_amd64-impl to run on execution platform @@platforms//host:host
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@rules_go~//go:toolchain -> toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_linux_amd64-impl
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host,
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host,
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host,
INFO: ToolchainResolution: Performing resolution of @@rules_go~//go:toolchain for target platform @@platforms//host:host
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_aix_ppc64-impl; mismatching values: aix, ppc
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_android_386-impl; mismatching values: android, x86_32
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_android_amd64-impl; mismatching values: android
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_android_arm-impl; mismatching values: android, armv7
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_android_arm64-impl; mismatching values: android, aarch64
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_darwin_386-impl; mismatching values: osx, x86_32
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_darwin_amd64-impl; mismatching values: osx
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_darwin_arm-impl; mismatching values: osx, armv7
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_darwin_arm64-impl; mismatching values: osx, aarch64
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_dragonfly_amd64-impl; mismatching values: dragonfly
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_freebsd_386-impl; mismatching values: freebsd, x86_32
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_freebsd_amd64-impl; mismatching values: freebsd
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_freebsd_arm-impl; mismatching values: freebsd, armv7
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_freebsd_arm64-impl; mismatching values: freebsd, aarch64
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_illumos_amd64-impl; mismatching values: illumos
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_ios_amd64-impl; mismatching values: ios
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_ios_arm64-impl; mismatching values: ios, aarch64
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_js_wasm-impl; mismatching values: js, wasm
      ToolchainResolution:   Rejected toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_linux_386-impl; mismatching values: x86_32
      ToolchainResolution:   Toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_linux_amd64-impl is compatible with target plaform, searching for execution platforms:
      ToolchainResolution:     Compatible execution platform @@platforms//host:host
      ToolchainResolution:   All execution platforms have been assigned a @@rules_go~//go:toolchain toolchain, stopping
      ToolchainResolution: Recap of selected @@rules_go~//go:toolchain toolchains for target platform @@platforms//host:host:
      ToolchainResolution:   Selected @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_linux_amd64-impl to run on execution platform @@platforms//host:host
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host, type @@rules_go~//go:toolchain -> toolchain @@rules_go~~go_sdk~go_toolchain_wrapper__download_0//:go_linux_amd64-impl
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host,
...
INFO: Build completed successfully, 9 total actions

But, using Bazel 8 (any minor version) resuls in Bazel searching for the Go toolchain for the target platform, not the execution platform (as was the case with Bazel 7):

$ bazel build :my_lib --platforms=:custom_platform --toolchain_resolution_debug='.*'
INFO: ToolchainResolution: Target platform //:custom_platform: Selected execution platform @@platforms//host:host,
INFO: ToolchainResolution: Target platform @@platforms//host:host: Selected execution platform @@platforms//host:host,
INFO: ToolchainResolution: Performing resolution of @@rules_go+//go:toolchain for target platform //:custom_platform
      ToolchainResolution:   Rejected toolchain @@rules_go++go_sdk+go_toolchain_wrapper__download_0//:go_aix_ppc64-impl; mismatching values: aix, ppc
      ...
      ToolchainResolution:   Rejected toolchain @@rules_go++go_sdk+rules_go__download_0_windows_arm64//:go_windows_arm64-impl; mismatching values: windows, aarch64
      ToolchainResolution: No @@rules_go+//go:toolchain toolchain found for target platform //:custom_platform.
ERROR: /home/mario/ttt/toolchains/wrapper/BUILD:10:10: While resolving toolchains for target //toolchains/wrapper:wrapper (b5bf0ec): No matching toolchains found for types:
  @@rules_go+//go:toolchain
To debug, rerun with --toolchain_resolution_debug='@@rules_go+//go:toolchain'
For more information on platforms or toolchains see https://bazel.build/concepts/platforms-intro.
ERROR: Analysis of target '//:my_lib' failed; build aborted: Analysis failed
INFO: Elapsed time: 22.037s, Critical Path: 0.04s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully

Essentially, Bazel 7 does:

Performing resolution of @@rules_go~//go:toolchain for target platform @@platforms//host:host

whereas Bazel 8 does:

Performing resolution of @@rules_go+//go:toolchain for target platform //:custom_platform

which, obviously, does not work as there is no Go toolchain for a custom CPU (@platforms//cpu:cpu).

Changing vesions of rules_go, platforms and rules_cc does not influence this behavior.

Is there something that could be done to work around that? I know we could use a custom constraint setting (rather than @platforms//cpu:cpu) to select our CC toolchain but that's not a nice solution as it'd require a lot of changes in our codebase.

Which category does this issue belong to?

Core

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Please find the reproducible example below:

bazel_8_go_toolchain_resolution_issue.zip

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 7.x.x/release 8.x.x

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

n/a

What's the output of git remote get-url origin; git rev-parse HEAD ?

n/a

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

n/a

Have you found anything relevant by searching the web?

Nothing in GH issues, nor SO.

Any other information, logs, or outputs that you want to share?

n/a

[pcjanzen]

See also bazelbuild/rules_cc#299

There's not currently a perfect way to solve this problem. In situations where there's a twisty maze of toolchains and exec platforms, it's not possible to always get the tool built for the correct exec platform, because native.cc_toolchain() doesn't have a dedicated cfg=exec type attribute to shovel tools into.

However, in many cases the exec platform resolution for targets that depend on the cc_toolchain is going to be identical to the default exec platform resolution. If that is true you can workaround by providing some intermediate rule that takes the tool in the exec config and forwards it.

You might even be able to re-use your existing cc_toolchain_config target for this purpose, e.g.,

diff --git a/toolchains/custom_toolchain/BUILD b/toolchains/custom_toolchain/BUILD
index 75a9e0e..7d929a7 100755
--- a/toolchains/custom_toolchain/BUILD
+++ b/toolchains/custom_toolchain/BUILD
@@ -19,7 +19,7 @@ toolchain(
 filegroup(
     name = "all_files",
     srcs = [
-        "//toolchains/wrapper",
+        ":cc_toolchain_config",
     ],
     tags = ["manual"],
 )
diff --git a/toolchains/custom_toolchain/cc_toolchain_config.bzl b/toolchains/custom_toolchain/cc_toolchain_config.bzl
index 1db0bbe..ffce274 100755
--- a/toolchains/custom_toolchain/cc_toolchain_config.bzl
+++ b/toolchains/custom_toolchain/cc_toolchain_config.bzl
@@ -123,7 +123,9 @@ def _impl(ctx):
         cpp_link_static_library_action,
     ]
 
-    return [cc_common.create_cc_toolchain_config_info(
+    return [
+        DefaultInfo(files=depset(ctx.files.driver_executable)),
+        cc_common.create_cc_toolchain_config_info(
         ctx = ctx,
         features = features,
         toolchain_identifier = "local",

[fmeum]

@pcjanzen's workaround is the best you can do right now. @dzbarsky and I are working on a proper fix for the rules-based toolchain approach: bazelbuild/rules_cc#555

[mariusz-wlodarczyk]

Thank you for your insights and suggestions @pcjanzen and @fmeum.

For anyone interested, I ended up writing a separate helper rule that allows multiple targets to be requested for the exec configuration:

def _exec_cfg_files_impl(ctx):
    return [DefaultInfo(files = depset(ctx.files.srcs))]

exec_cfg_files = rule(
    implementation = _exec_cfg_files_impl,
    attrs = {
        "srcs": attr.label_list(
            allow_files = True,
            mandatory = True,
            allow_empty = False,
            cfg = "exec",
            doc = "Targets whose default outputs should be built with the `exec` configuration",
        ),
    },
    doc = "Ensures that given dependencies are built for the `exec` configuration.",
)

I use it to wrap my wrapper's go_binary target:

# toolchains/wrapper/BUILD
...
go_binary(
    name = "wrapper",
    ...
)

exec_cfg_files(
    name = "wrapper_exec_files",
    srcs = [":wrapper"],
    visibility = ["//visibility:public"],
)

and then pass it to the *_files attributes of the cc_toolchain rule:

# toolchains/custom_toolchain/BUILD
...
filegroup(
    name = "all_files",
    srcs = [
        "//toolchains/wrapper:wrapper_exec_files",
    ],
    tags = ["manual"],
)

cc_toolchain(
    name = "cc_toolchain",
    all_files = ":all_files",
    ar_files = ":all_files",
    ...
    toolchain_config = ":cc_toolchain_config",
)