Production Kubeflow deployment on GKE with Intel Xeon C3 instances — automated bootstrap, Anthos Config Controller, Istio service mesh, and Identity-Aware Proxy.
- Overview
- Why Intel Xeon C3?
- Architecture
- Repository Structure
- Prerequisites
- Step-by-Step Deployment
- Configuration Reference
- Choosing an Intel C3 Instance Type
- What Gets Deployed
- Known Issues and Fixes
- Troubleshooting
- Cleanup
- Cost Estimate
- Alternative: Terraform-only Deployment
This repository automates the deployment of Kubeflow — the ML platform for Kubernetes — on Google Kubernetes Engine using Intel Xeon Scalable (C3) instances. It handles the full lifecycle:
- Enabling all required GCP APIs
- Deploying an Anthos Config Controller management cluster
- Deploying a Kubeflow workload cluster with Intel C3 nodes
- Automatically patching known upstream YAML bugs (v1.7.0)
- Configuring IAP-secured access to the Kubeflow UI
- Teardown in the correct dependency order
You can have a fully working Kubeflow installation in ~25 minutes with a single command sequence.
Google Cloud's C3 machines run on 4th Gen Intel Xeon Scalable processors and offer several advantages for ML workloads:
| Feature | Benefit |
|---|---|
| AVX-512 VNNI | Accelerates INT8 inference — up to 4× vs AVX2 |
| Intel IPU (Infrastructure Processing Unit) | Offloads networking/storage from the CPU, freeing cores for ML |
| High memory bandwidth | Critical for data loading in training loops |
| Up to 192 vCPUs / 1.5TB RAM | Large-scale distributed training without GPUs |
| DDR5 memory | ~50% higher bandwidth vs DDR4 in N2 instances |
For CPU-only ML workloads (inference, feature engineering, small model training), C3 instances often deliver better price-performance than GPUs for batch jobs.
┌──────────────────────────────────────────────────────┐
│ GCP Project │
│ │
│ ┌─────────────────────────────────┐ │
│ │ Management Cluster │ │
│ │ (Anthos Config Controller) │ │
│ │ │ │
│ │ • Provisions GCP resources │ │
│ │ via Kubernetes CRDs │ │
│ │ • 3× e2-standard-2 nodes │ │
│ └──────────────┬──────────────────┘ │
│ │ provisions │
│ ▼ │
│ ┌─────────────────────────────────┐ │
│ │ Kubeflow Cluster │ │
│ │ Intel Xeon C3 node pool │ │
│ │ │ │
│ │ ┌──────────┐ ┌─────────────┐ │ │
│ │ │ Istio │ │ IAP │ │ │
│ │ │ (mesh) │ │ (auth) │ │ │
│ │ └────┬─────┘ └──────┬──────┘ │ │
│ │ └────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌──────────────────────────┐ │ │
│ │ │ Kubeflow UI │ │ │
│ │ ├──────────────────────────┤ │ │
│ │ │ Pipelines │ Notebooks │ │ │
│ │ │ KFServe │ Katib │ │ │
│ │ └──────────────────────────┘ │ │
│ └─────────────────────────────────┘ │
└──────────────────────────────────────────────────────┘
Two-cluster model explained:
The management cluster runs Anthos Config Controller, which is a hosted version of Config Connector (KCC). It manages GCP resources (VPCs, GKE clusters, IAM bindings) via Kubernetes CRDs — meaning your infrastructure is declared as Kubernetes objects and reconciled continuously. The Kubeflow cluster is what KCC provisions, and it hosts all ML workloads.
kubeflow-intel-gke/
│
├── scripts/
│ ├── bootstrap.sh # Enable APIs, init ASM, configure IAM
│ ├── deploy-mgmt.sh # Deploy Anthos Config Controller cluster
│ ├── deploy-kubeflow.sh # Deploy Kubeflow on Intel C3 nodes
│ └── cleanup.sh # Teardown in correct dependency order
│
├── management/
│ ├── env.sh.example # Template — copy to env.sh (gitignored)
│ └── kpt-set.sh # Apply kpt setters to management manifests
│
├── kubeflow/
│ ├── env.sh.example # Template — copy to env.sh (gitignored)
│ └── kpt-set.sh # Apply kpt setters to Kubeflow manifests
│
├── terraform/ # Alternative IaC path (optional)
│ ├── main.tf # VPC, GKE cluster, Intel C3 node pool
│ ├── variables.tf
│ └── outputs.tf
│
└── .github/
└── workflows/
└── validate.yml # CI: shellcheck + terraform validate
| Tool | Minimum Version | Install |
|---|---|---|
gcloud CLI |
latest | Install guide |
kubectl |
≥ 1.27 | gcloud components install kubectl |
kpt |
≥ 1.0 | gcloud components install kpt |
kustomize |
≥ 5.0 | brew install kustomize or releases |
git |
any | pre-installed on most systems |
Tip: Google Cloud Shell has all of these pre-installed. If you want zero local setup, use Cloud Shell at shell.cloud.google.com.
- A GCP project with Owner role
- Billing enabled on the project
- Quota for C3 instances in your target region (
us-central1by default). Request a quota increase atIAM & Admin → Quotasif needed.
gcloud auth login
gcloud auth application-default logingit clone https://github.com/ashiq-ali/kubeflow-intel-gke.git
cd kubeflow-intel-gke
# Create management cluster config
cp management/env.sh.example management/env.sh
# Create Kubeflow cluster config
cp kubeflow/env.sh.example kubeflow/env.shEdit management/env.sh:
export MGMT_PROJECT=my-gcp-project # Your GCP project ID
export MGMT_NAME=kubeflow-mgmt # Name for the management cluster
export LOCATION=us-central1 # Must support Anthos Config ControllerLeave kubeflow/env.sh for now — you'll fill in CLIENT_ID and CLIENT_SECRET after step 3.
bash scripts/bootstrap.shThis script:
- Sets your active GCP project
- Enables all 10 required APIs (takes ~2 minutes)
- Initialises Anthos Service Mesh
- Grants the Cloud Build service account required roles
- Prints instructions for the manual OAuth step
Kubeflow uses Google OAuth + Identity-Aware Proxy to secure its UI. This step requires the GCP Console:
-
Go to APIs & Services → OAuth Consent Screen
- User Type: External
- Fill in app name, support email
- Add authorised domain:
<YOUR_PROJECT_ID>.cloud.goog
-
Go to APIs & Services → Credentials → Create Credentials → OAuth 2.0 Client ID
- Application type: Web application
- Add authorised redirect URI:
https://iap.googleapis.com/v1/oauth/clientIds/<CLIENT_ID>:handleRedirect - Save the Client ID and Client Secret
-
Add them to
kubeflow/env.sh:export CLIENT_ID=<paste-client-id> export CLIENT_SECRET=<paste-client-secret>
-
Optionally change the instance type in
kubeflow/env.sh:export KF_INSTANCE_TYPE=c3-standard-8 # See sizing guide below
bash scripts/deploy-mgmt.shThis will:
- Clone
kubeflow-distributionat tagv1.7.0 - Apply kpt setters for your project/region
- Create the Anthos Config Controller cluster via
gcloud anthos config controller create - Configure kubectl context
- Grant owner permissions to the Config Connector service account
⏱ Expected time: 8–12 minutes
You will see output like:
==> Creating Anthos Config Controller cluster (this takes ~10 minutes)...
...
✓ Management cluster deployed: kubeflow-mgmt
Next: bash scripts/deploy-kubeflow.sh
bash scripts/deploy-kubeflow.shThis will:
- Pull upstream Kubeflow manifests
- Apply kpt setters for your project, zone, OAuth credentials
- Automatically patch 3 known v1.7.0 upstream YAML bugs (see Known Issues)
- Override the node pool machine type to your chosen Intel C3 instance
- Apply KCC resources (creates VPC, GKE cluster in GCP)
- Wait for GKE cluster to reach
RUNNINGstate - Deploy Kubeflow workloads — retries up to 3× on webhook timeout errors
- Grant your account IAP access
- Print the Kubeflow URL
⏱ Expected time: 15–20 minutes
Once deployment completes, get your URL:
kubectl -n istio-system get ingressOutput:
NAME CLASS HOSTS ADDRESS PORTS AGE
envoy-ingress <none> kubeflow.endpoints.my-project.cloud.goog 34.x.x.x 80 5m
Open https://kubeflow.endpoints.<YOUR_PROJECT_ID>.cloud.goog in your browser. You will be prompted to sign in with your Google account (the one granted IAP access).
Note: DNS propagation can take 2–5 minutes after the ingress is created.
To grant access to additional users:
gcloud projects add-iam-policy-binding "${KF_PROJECT}" \
--member="user:colleague@example.com" \
--role=roles/iap.httpsResourceAccessorVerify all pods are running:
kubectl -n kubeflow get podsExpected output (all pods Running or Completed):
NAME READY STATUS RESTARTS AGE
admission-webhook-deployment-xxx 1/1 Running 0 10m
centraldashboard-xxx 1/1 Running 0 10m
jupyter-web-app-deployment-xxx 1/1 Running 0 10m
katib-controller-xxx 1/1 Running 0 10m
kfp-api-server-xxx 2/2 Running 0 10m
kfp-ui-xxx 2/2 Running 0 10m
...
| Variable | Default | Description |
|---|---|---|
MGMT_PROJECT |
— | GCP project ID for the management cluster |
MGMT_NAME |
kubeflow-mgmt |
Name of the Anthos Config Controller cluster |
LOCATION |
us-central1 |
Region (must support Anthos CC — see supported regions) |
| Variable | Default | Description |
|---|---|---|
KF_PROJECT |
— | GCP project ID for Kubeflow |
KF_NAME |
kubeflow |
Name of the Kubeflow GKE cluster |
ZONE |
us-central1-a |
Zone for the GKE cluster |
REGION |
us-central1 |
Region |
CLIENT_ID |
— | OAuth 2.0 Client ID (from GCP Console) |
CLIENT_SECRET |
— | OAuth 2.0 Client Secret |
KF_INSTANCE_TYPE |
c3-standard-8 |
Intel Xeon C3 machine type |
KF_NODE_COUNT |
2 |
Number of nodes in the Kubeflow node pool |
MGMTCTXT |
kubeflow-mgmt |
kubectl context name for the management cluster |
| Machine Type | vCPUs | RAM | Use Case |
|---|---|---|---|
c3-standard-4 |
4 | 16 GB | Dev/test, light workloads |
c3-standard-8 |
8 | 32 GB | Recommended starting point |
c3-standard-22 |
22 | 88 GB | Mid-size model training |
c3-standard-44 |
44 | 176 GB | Large model training |
c3-standard-88 |
88 | 352 GB | Distributed training, LLM fine-tuning |
c3-standard-176 |
176 | 704 GB | Largest single-node ML jobs |
All C3 machines include AVX-512 VNNI for accelerated INT8 inference and are backed by Intel IPUs that offload network/storage processing from the main CPU cores.
Set your choice in kubeflow/env.sh:
export KF_INSTANCE_TYPE=c3-standard-22Once complete, your Kubeflow cluster includes:
| Component | Description |
|---|---|
| Central Dashboard | Unified web UI for all Kubeflow components |
| Jupyter Notebooks | Managed notebook servers with configurable CPU/memory |
| Kubeflow Pipelines | DAG-based ML workflow orchestration |
| KFServe | Model serving — REST/gRPC endpoints for trained models |
| Katib | Hyperparameter tuning and neural architecture search |
| Training Operators | TFJob, PyTorchJob, MXJob for distributed training |
| Istio | Service mesh — mTLS, traffic management, observability |
| IAP | Identity-Aware Proxy — Google account authentication |
deploy-kubeflow.sh automatically applies these patches to kubeflow-distribution v1.7.0:
| File | Bug | Fix Applied |
|---|---|---|
common/cluster/upstream/cluster.yaml |
Deprecated metadata.clusterName field causes apply failure |
Field removed |
common/cluster/upstream/nodepool.yaml |
Same deprecated field | Field removed |
apps/profiles/upstream/crd/bases/kubeflow.org_profiles.yaml |
Spurious creationTimestamp: null causes CRD rejection |
Line removed |
These patches are applied in-place before make apply runs. They have been upstreamed — later versions of kubeflow-distribution should not require them.
Symptom:
Error from server (Timeout): error when creating "...": context deadline exceeded
Cause: Admission webhooks are not yet ready when Kubeflow resources are applied — a race condition on first install.
Fix: deploy-kubeflow.sh retries up to 3 times automatically. If it still fails, run manually:
cd ~/kubeflow-distribution/kubeflow && make applySymptom: Browser shows Error 403: access_denied after OAuth login.
Cause: Your Google account hasn't been granted the IAP role.
Fix:
gcloud projects add-iam-policy-binding "${KF_PROJECT}" \
--member="user:your-email@gmail.com" \
--role=roles/iap.httpsResourceAccessorSymptom: kubectl -n kubeflow get pods shows pods in Pending state.
Diagnosis:
kubectl -n kubeflow describe pod <pod-name> | grep -A5 EventsCommon causes:
| Error | Fix |
|---|---|
Insufficient cpu |
Increase KF_NODE_COUNT or use larger KF_INSTANCE_TYPE |
Insufficient memory |
Same as above |
no nodes available |
Check node pool is RUNNING: gcloud container node-pools list --cluster=${KF_NAME} --zone=${ZONE} |
Symptom: scripts/bootstrap.sh shows unexpected output from the ASM init endpoint.
Cause: The project may need a temporary cluster before ASM can be initialised.
Fix: Create and immediately delete a temporary cluster:
gcloud container clusters create temp-cluster --zone us-central1-a --num-nodes=1
gcloud container clusters delete temp-cluster --zone us-central1-a --quiet
# Then re-run bootstrap.shSymptom: Node pool creation fails with quota or availability error.
Fix: Check C3 availability in your target zone:
gcloud compute machine-types list --filter="name~c3" --zones=us-central1-aTry an adjacent zone (us-central1-b, us-central1-c) or request a quota increase.
bash scripts/cleanup.shThe cleanup script tears down resources in the correct order:
- Deletes the
kubeflownamespace and waits for termination - Runs
make deleteto remove KCC-managed GCP resources (VPC, GKE cluster) - Removes the project namespace from the management cluster
- Deletes the Anthos Config Controller management cluster
Cost note: GKE clusters continue to incur charges until fully deleted. Verify deletion in the GCP Console under
Kubernetes Engine → Clusters.
Approximate costs for a standard deployment in us-central1 (USD/month):
| Resource | Spec | Est. Cost |
|---|---|---|
| Management cluster | 3× e2-standard-2 | ~$75 |
| Kubeflow node pool | 2× c3-standard-8 | ~$280 |
| Persistent disks | ~200 GB SSD | ~$35 |
| Load balancer | 1× L7 LB | ~$20 |
| Total | ~$410/month |
Costs scale linearly with node count and machine type. For development, use c3-standard-4 with KF_NODE_COUNT=1 to reduce costs to ~$200/month.
For teams that prefer pure Terraform IaC, the terraform/ directory provisions the VPC and GKE cluster with an Intel C3 node pool directly — without Anthos Config Controller:
cd terraform
# Create a terraform.tfvars file
cat > terraform.tfvars << EOF
project_id = "my-gcp-project"
region = "us-central1"
zone = "us-central1-a"
machine_type = "c3-standard-8"
node_count = 2
EOF
terraform init
terraform plan
terraform applyAfter the cluster is created, follow steps 3, 5 (skipping make apply-kcc), and 6 to deploy Kubeflow workloads.
Compute: Intel Xeon C3 (AVX-512 VNNI · DDR5 · custom IPU offload) Platform: GKE · Anthos Config Controller · Anthos Service Mesh Networking: Istio · Identity-Aware Proxy · Cloud Load Balancing ML Platform: Kubeflow Pipelines · KFServe · Katib · Training Operators · Jupyter Hub IaC: kpt · kustomize · Config Connector · Terraform