[P0] RabbitMQ AsyncActor with scaling.enabled=false can stay at 0 replicas and stall queued tasks

[Gsbreddy]

Summary

In the RabbitMQ + Crossplane path, an AsyncActor with spec.scaling.enabled=false and spec.replicas=1 can reconcile without a ScaledObject, yet the rendered Deployment can still sit at spec.replicas=0. The gateway accepts work and RabbitMQ receives it, but nothing consumes the queue until the deployment is manually scaled.

Why this matters

This breaks the fixed-replica contract for actors with autoscaling disabled and makes a fresh install look healthy while real work stalls.

Expected behavior

• With scaling.enabled=false, no ScaledObject or KEDA HPA should exist for the actor.
• The deployment replica count should follow spec.replicas.
• Status should not report Napping unless KEDA is actually present and has scaled the workload to zero.
• A task submitted through gateway should be consumed without any manual kubectl scale step.

Observed behavior on a real GKE install

Actor under test:

• transport: RabbitMQ
• actor: test-echo (example actor from smoke test)
• spec.scaling.enabled=false
• spec.replicas=1

What was observed before manual intervention:

• AsyncActor/test-echo existed and showed Ready=True / Synced=True
• status.phase was Napping
• status.infrastructure.workload.replicas=0
• status.infrastructure.workload.readyReplicas=0
• no ScaledObject existed for test-echo
• no HPA existed for test-echo
• the rendered Deployment/test-echo had spec.replicas=0
• gateway accepted a real task and RabbitMQ queued it, but no worker pod existed to consume it

Concrete task evidence:

• gateway created a task successfully
• nothing processed it while the actor remained at 0 replicas
• after kubectl scale deployment/test-echo --replicas=1, the actor immediately consumed the queued message, forwarded to x-sink, and mesh updated final status to succeeded

Why this looks like a bug / contract mismatch

The intended contract appears clear from code, docs, and tests:

1. deploy/helm-charts/asya-crossplane/templates/composition-rabbitmq.yaml

   • render-deployment sets deployment replicas from xrSpec.replicas when scaling.enabled=false
   • patch-status-and-derive-phase treats Napping as the case where replicas are 0, but that phase is documented as the KEDA scale-to-zero state

2. docs/reference/components/core-crossplane.md

   • says Ready means workload has replicas > 0
   • says Napping means KEDA scaled the workload to zero

3. testing/e2e/tests/test_keda_scaling.py::test_scaledobject_not_created_when_scaling_disabled

   • expects no ScaledObject when scaling is disabled

4. testing/e2e/tests/test_crossplane_e2e.py::test_asyncactor_replicas_update_scaling_disabled

   • expects deployment replicas to follow the XR when scaling is disabled

So the live RabbitMQ behavior did not match the documented or tested contract.

Suspected scope

At least one of these needs attention:

• RabbitMQ composition reconciliation for scaling.enabled=false
• drift restoration of the composed deployment replica count
• status/phase derivation allowing Napping when KEDA is absent
• missing end-to-end coverage that proves a RabbitMQ actor with scaling disabled actually consumes work after install

Acceptance criteria

• RabbitMQ-backed AsyncActor with scaling.enabled=false and replicas=1 results in Deployment.spec.replicas=1
• no ScaledObject or HPA is created for that actor
• task submission through gateway is consumed without manual scaling
• status never reports Napping when KEDA is absent for that actor
• CI contains a regression test for RabbitMQ + Crossplane + scaling.enabled=false that proves the actor really processes a queued message

Related

• Related to [P0] RabbitMQ queue lifecycle and AsyncActor readiness must reflect reality #372, but distinct.
• [P0] RabbitMQ queue lifecycle and AsyncActor readiness must reflect reality #372 is about RabbitMQ queue lifecycle/readiness being reported as healthy when queues may not exist.
• This issue is about a disabled-scaling actor remaining at 0 replicas even when the queue exists and a task is queued.

[Gsbreddy]

Retest update — 2026-03-23 (fresh clean deploy, full destroy+redeploy)

After a full destroy and clean redeploy, this issue is not reproducing.

On a fresh install, test-echo AsyncActor (with spec.scaling.enabled: false and spec.replicas: 1) deployed correctly:

• Deployment.spec.replicas = 1 ✅
• No ScaledObject for test-echo ✅
• Actor phase: Ready with readyReplicas: 1 ✅
• Task processed successfully after queue sync ✅

The Crossplane composition template (render-deployment step) correctly reads $xrSpec.replicas when scaling.enabled=false and sets Deployment.spec.replicas accordingly.

The original 0-replica observation on 2026-03-23 was likely caused by stale Crossplane state from a prior partial deploy. On a clean cluster this race does not appear.

Recommend: Keep open to add a CI regression test (scaling.enabled=false + message submission → task succeeds without manual scaling), but lower priority from P0 to P1 given it is not reproducing on clean deploys.