Every Karta must define a root component:
- Must use full Kubernetes GVK (Group, Version, Kind)
- Must include a
statusDefinition
rootComponent:
name: "pytorchjob"
kind:
group: "kubeflow.org"
version: "v1"
kind: "PyTorchJob"
statusDefinition:
statusMappings:
running:
- byConditions:
- type: "Running"
status: "True"For resources owned by the root component:
- Must include
ownerRef(points to parent component) - Usually include a
specDefinition - All paths must be absolute from the CRD root
childComponents:
- name: "worker"
ownerRef: "pytorchjob"
kind:
group: "apps"
version: "v1"
kind: "StatefulSet"
specDefinition:
podTemplateSpecPath: ".spec.pytorchReplicaSpecs.Worker.template"All paths provided in a Karta are written in jq syntax. The jq query language provides both path navigation and various query capabilities, and is widely used in the k8s ecosystem. Make sure to provide a matching jq query type for every property (path/query), and to provide default values where necessary.
There are multiple, mutually exclusive, types of specDefinitions:
| Pattern | Use Case | Example |
|---|---|---|
| PodTemplateSpecPath | CRD embeds a full pod template | .spec.pytorchReplicaSpecs.Master.template |
| PodSpecPath & MetadataPath | CRD directly embeds a podSpec and/or an objectMeta | .spec.jobTemplate.spec .spec.jobTemplate.metadata |
| FragmentedPodSpecDefinition | Pod fields scattered across CRD | See below |
specDefinition:
fragmentedPodDefinition:
labelsPath: ".spec.labels"
annotationsPath: ".spec.annotations"
resourcesPath: ".spec.resources"
schedulerNamePath: ".spec.schedulerName"Component’s spec definition might point to multiple instances specs (in map/array format). In those cases it’s crucial to be able to distinguish between each instance of that component. In order to do so, users must define an instanceIdPath - the location in the spec where we can find the name of each instance. For example:
- Array of specs:
instanceIdPath: ".spec.jobs[].name"- Map of specs: (extract the map keys that represent the instance id)
instanceIdPath: ".spec.jobs | to_entries[] | .key"PodSelector defines how to identify pods belonging to a component / an instance of a component. A component can define both type and instance selectors. All selectors of each kind (component, instance) must be mutually exclusive within themselves.
Paths in pod selectors are referring to paths on the pod’s manifest
- Component type selector - key (and value) selector that associates pod to the current component.
podSelector:
componentTypeSelector:
keyPath: '.metadata.labels["training.kubeflow.org/replica-type"]'
value: "master"If value is not provided, only key existence is checked.
- Component instance selector - a path on the pod that holds its matching instance id.
podSelector:
componentInstanceSelector:
idPath: '.metadata.labels["jobset.sigs.k8s.io/replicatedjob-name"]'- Mapping the described CRD’s conditions or phases to the Karta generic statuses. Must be based on the actual conditions/phases used by the described CRD.
- For each generic status, the user can provide a definition based on conditions or phases. If both are provided, both are validated when evaluating the status.
- If using definition by conditions/phase, you first must include
conditionsDefinition/phaseDefinition - Multiple, separate, definitions can be provided for each generic status.
- When providing a definition
byConditions, all must exist (AND logic) - Required for the root component
statusDefinition:
conditionsDefinition:
path: ".status.conditions"
typeFieldName: "type"
statusFieldName: "status"
statusMappings:
initializing:
- byConditions:
- type: "Created"
status: "True"
- type: "Running"
status: "False"
running:
- byConditions:
- type: "Running"
status: "True"
- type: "Succeeded"
status: "False"
- type: "Failed"
status: "False"
completed:
- byConditions:
- type: "Succeeded"
status: "True"
failed:
- byConditions:
- type: "Failed"
status: "True"List any GVK of objects created or managed by the CRD that are not mentioned explicitly by any child component. This is essential for permission management so that your CRD can be managed correctly.
additionalChildKinds:
- group: apps
version: v1
kind: Deployment
- group: leaderworkerset.x-k8s.io
version: v1
kind: LeaderWorkerSetUsed for scheduling.
Paths in the instruction are referring to paths on the pod’s yaml/json
Currently supported instructions:
gangScheduling: instruct the scheduler how to group pods. Each pod-group definition contains a list of the included members. Each defined member can provide a list of distinct keys to group pods by and a list of filters to determine which pods should be included.
optimizationInstructions:
gangScheduling:
podGroups:
- name: "job"
members:
- componentName: "master"
groupByKeyPaths:
- '.metadata.labels["training.kubeflow.org/job-name"]'
- componentName: "worker"
groupByKeyPaths:
- '.metadata.labels["training.kubeflow.org/job-name"]'Grouping examples:
- Different hierarchy: The following are equivalent (given that master and worker has the same value for that label)
optimizationInstructions:
gangScheduling:
podGroups:
- name: "job"
members:
- componentName: "master"
groupByKeyPaths:
- '.metadata.labels["training.kubeflow.org/job-name"]'
- componentName: "worker"
groupByKeyPaths:
- '.metadata.labels["training.kubeflow.org/job-name"]'
optimizationInstructions:
gangScheduling:
podGroups:
- name: "job"
members:
- componentName: "job"
groupByKeyPaths:
- '.metadata.labels["training.kubeflow.org/job-name"]'- Using default values: when multiple groups are possible, use default value to cover cases where single group is used (the used pattern: -{name}-{index})
optimizationInstructions:
gangScheduling:
podGroups:
- name: "group"
members:
- componentName: "group"
groupByKeyPaths:
- '.metadata.labels["leaderworkerset.sigs.k8s.io/name"]'
- '.metadata.labels["leaderworkerset.sigs.k8s.io/group-index"] // "0"'
3. Filters: when different components are not named components (array/map of specs in the same component) you can use filters to form different pod groups.
In this example, for a CRD that defines multiple jobs but all under the “job” component, we use jq query as filter to identify pods that use Nvidia GPUs, and queries with hard-coded values as grouping keys.
optimizationInstructions:
gangScheduling:
podGroups:
- name: "gpu-jobs"
members:
- componentName: "job"
filters:
- 'any(.spec.jobs[].spec.containers[]; (.resources.limits["nvidia.com/gpu"] // 0) > 0)'
groupByKeyPaths:
- 'gpu'
- name: "no-gpu-jobs"
members:
- componentName: "job"
filters:
- 'any(.spec.jobs[].spec.containers[]; (.resources.limits["nvidia.com/gpu"] // 0) == 0)'
groupByKeyPaths:
- 'no-gpu'- Always use full GVK for kinds (group, version, kind)
- Absolute paths only in spec definitions
- Avoid duplication: don’t list explicitly defined components in
additionalChildKinds - Mutually exclusive pod selectors in multi-component workloads
- Null-safe JQ expressions (e.g.
// 0defaults) - Target actual components in optimization instructions, not just the root CRD
Example 1: Kserve inference service
{
"spec": {
"structureDefinition": {
"rootComponent": {
"name": "inferenceservice",
"kind": {
"group": "serving.kserve.io",
"version": "v1beta1",
"kind": "InferenceService"
},
"specDefinition": {
"fragmentedPodSpecDefinition": {
"resourcesPath": ".spec.domain.resources",
"priorityClassNamePath": ".spec.priorityClassName",
"nodeAffinityPath": ".spec.affinity.nodeAffinity"
}
},
"statusDefinition": {
"conditionsDefinition": {
"path": ".status.conditions",
"typeFieldName": "type",
"statusFieldName": "status"
},
"statusMappings": {
"running": [
{
"byConditions": [
{
"type": "PredictorReady",
"status": "True"
},
{
"type": "RoutesReady",
"status": "True"
},
{
"type": "LatestDeploymentReady",
"status": "True"
}
]
}
],
"failed": [
{
"byConditions": [
{
"type": "PredictorReady",
"status": "False"
},
{
"type": "PredictorConfigurationReady",
"status": "False"
},
{
"type": "RoutesReady",
"status": "False"
}
]
}
]
}
}
},
"childComponents": [
{
"name": "predictor",
"kind": {
"group": "apps",
"version": "v1",
"kind": "Deployment"
},
"ownerRef": "inferenceservice",
"specDefinition": {
"podSpecPath": ".spec.predictor",
"metadataPath": ".spec.predictor",
"fragmentedPodSpecDefinition": {
"containerPath": ".spec.predictor.model"
}
},
"scaleDefinition": {
"minReplicasPath": ".spec.predictor.minReplicas",
"maxReplicasPath": ".spec.predictor.maxReplicas"
},
"podSelector": {
"componentTypeSelector": {
"keyPath": ".metadata.labels[\"component\"]",
"value": "predictor"
}
}
},
{
"name": "transformer",
"kind": {
"group": "apps",
"version": "v1",
"kind": "Deployment"
},
"ownerRef": "inferenceservice",
"specDefinition": {
"podSpecPath": ".spec.transformer",
"metadataPath": ".spec.transformer"
},
"scaleDefinition": {
"minReplicasPath": ".spec.transformer.minReplicas",
"maxReplicasPath": ".spec.transformer.maxReplicas"
},
"podSelector": {
"componentTypeSelector": {
"keyPath": ".metadata.labels[\"component\"]",
"value": "transformer"
}
}
}
]
},
"optimizationInstructions": {
"gangScheduling": {
"podGroups": [
{
"name": "service",
"members": [
{
"componentName": "predictor",
"groupByKeyPaths": [
".metadata.labels[\"serving.kserve.io/inferenceservice\"]"
]
},
{
"componentName": "transformer",
"groupByKeyPaths": [
".metadata.labels[\"serving.kserve.io/inferenceservice\"]"
]
}
]
}
]
}
}
}
}Before submitting, confirm:
- All kinds use full GVK
- root component has statusDefinition
- All JQ paths are absolute and null-safe (and are paths in the correct resource - root CRD/pod)
- No duplicated child kinds
- Pod selectors are mutually exclusive
- Status conditions match real framework APIs
- All child component has ownerRef directed to existing components and there are no ownership cycles
Use these as starting points when creating new workload type definitions. 👉 With these templates, you can register new workload types by simply filling in the blanks instead of starting from scratch.
A single-component workload with pods defined directly in its spec.
rootComponent:
name: "job"
kind:
group: "batch"
version: "v1"
kind: "Job"
statusDefinition:
statusMappings:
running:
- byConditions:
- type: "Running"
status: "True"
completed:
- byConditions:
- type: "Complete"
status: "True"
failed:
- byConditions:
- type: "Failed"
status: "True"
specDefinition:
podTemplateSpecPath: ".spec.template"A controlling resource with generated ReplicaSets.
rootComponent:
name: "deployment"
kind:
group: "apps"
version: "v1"
kind: "Deployment"
statusDefinition:
statusMappings:
running:
- byConditions:
- type: "Available"
status: "True"
childComponents:
- name: "replicaset"
ownerRef: "deployment"
kind:
group: "apps"
version: "v1"
kind: "ReplicaSet"
specDefinition:
podTemplateSpecPath: ".spec.template"Multi-component workload with role-based pods.
rootComponent:
name: "pytorchjob"
kind:
group: "kubeflow.org"
version: "v1"
kind: "PyTorchJob"
statusDefinition:
statusMappings:
running:
- byConditions:
- type: "Running"
status: "True"
completed:
- byConditions:
- type: "Succeeded"
status: "True"
failed:
- byConditions:
- type: "Failed"
status: "True"
childComponents:
- name: "master"
ownerRef: "pytorchjob"
kind:
group: "apps"
version: "v1"
kind: "ReplicaSet"
specDefinition:
podTemplateSpecPath: ".spec.pytorchReplicaSpecs.Master.template"
podSelector:
componentTypeSelector:
keyPath: '.metadata.labels["training.kubeflow.org/replica-type"]'
value: "master"
- name: "worker"
ownerRef: "pytorchjob"
kind:
group: "apps"
version: "v1"
kind: "ReplicaSet"
specDefinition:
podTemplateSpecPath: ".spec.pytorchReplicaSpecs.Worker.template"
podSelector:
componentTypeSelector:
keyPath: '.metadata.labels["training.kubeflow.org/replica-type"]'
value: "worker"Workload that owns a secondary component (Revision).
rootComponent:
name: "service"
kind:
group: "serving.knative.dev"
version: "v1"
kind: "Service"
statusDefinition:
statusMappings:
running:
- byConditions:
- type: "Ready"
status: "True"
childComponents:
- name: "revision"
ownerRef: "service"
kind:
group: "serving.knative.dev"
version: "v1"
kind: "Revision"
specDefinition:
podTemplateSpecPath: ".spec.template"The minimal Karta must contain a rootComponent with name, full GVK (group, version, kind) and statusDefinition.
For example:
rootComponent:
name: "minimal"
kind:
group: "minimal.org"
version: "v1"
kind: "Minimal"
statusDefinition:
statusMappings:
running:
- byConditions:
- type: "Running"
status: "True"-
Start from the closest template to your workload type.
-
Replace the GVK (group, version, kind) with your CRD’s.
-
Verify status conditions in the CRD source code or documentation.
-
Add child/referenced components only if they matter for scheduling or optimization.