test: add an e2e case for a client-less CacheRuntime topology - #6175
test: add an e2e case for a client-less CacheRuntime topology#6175btxu-db wants to merge 2 commits into
Conversation
Mooncake has no POSIX mount semantics, so its CacheRuntimeClass declares only master and worker. No existing e2e case covers that shape: curvine ships a client component, so the client-less path through the controller is untested. The case pins down the behaviour that path is expected to have: - the controller does not panic and the Dataset reaches Bound with the client component omitted (a regression guard for the nil pointer dereference fixed in fluid-cloudnative#6157); - no client DaemonSet or client pods are created, and status.client.phase stays empty; - the ReportSummary script populates status.cacheStates, and cached reflects data written through the cache system's own client; - the Dataset PVC reaches Bound but cannot be mounted by application pods, which is what the docs' FAQ describes. The image is built in-repo from test/gha-e2e/mooncake/image rather than pulled from an external registry, for the same reason as the jindo oss-emulator: e2e runs on every PR, an external image going away turns the whole pipeline red, and the two scripts Fluid invokes inside the image have to be reviewable. The base image is pinned by digest and every resolved pip version is pinned explicitly, installed from wheels only so no package build script runs at image build time. The CacheRuntimeClass template deliberately declares no container resources. A template that sets them while the CacheRuntime does not currently has them overwritten with an empty value on the first reconcile (fluid-cloudnative#6161), and if the resulting rollout flips the Dataset to Failed it does not recover on its own (fluid-cloudnative#6160). Neither is what this case is meant to cover, so it stays clear of both until fluid-cloudnative#6165 lands. Signed-off-by: btxu-db <btxu-db@outlook.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @btxu-db. Thanks for your PR. I'm waiting for a fluid-cloudnative member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| restartPolicy: Never | ||
| automountServiceAccountToken: false | ||
| containers: | ||
| - name: app |
| restartPolicy: Never | ||
| automountServiceAccountToken: false | ||
| containers: | ||
| - name: app |
| # python:3.12.13-slim — slim is required: the full variant carries an extra | ||
| # ~700MB build toolchain, growing the image from ~470MB to ~1.9GB, and e2e also | ||
| # has to kind-load it onto the node and keep a second copy there. | ||
| FROM python:3.12.13-slim@sha256:229a2c5bfa27522db7815ea81f9bed70af17ccb9de9fc7ad142b1877b5830d36 |
| exec mooncake_client \ | ||
| --host="$WORKER_HOST" \ | ||
| --port=50052 \ | ||
| --global_segment_size="$SEGMENT_SIZE" \ | ||
| --master_server_address="$MASTER_ADDR" \ | ||
| --metadata_server="$METADATA_ADDR" \ |
Three of the four findings are fixed here. The negative-case pod declares no memory or ephemeral-storage limits. It is never expected to start, so the values are nominal, but a reviewer's cluster may enforce a policy that rejects a pod without them, and the case would then fail for a reason that has nothing to do with what it tests. The image ran as root. Every listening port is above 1024, the components write nothing to the container filesystem, and the tiered store quota reaches Mooncake as a process memory segment rather than a directory, so nothing needs root. Verified by running mooncake_master from the built image as uid 10001: it binds 50051, 8080 and 9003 and keeps serving. The remaining finding, on clear-text protocols, has no code fix. Mooncake ships no TLS variant for either connection: the master's built-in metadata server (--enable_http_metadata_server) only speaks HTTP, and the transfer engine offers only "tcp" and "rdma". Both connections stay inside the cluster, are addressed by service DNS, and carry cache blocks between components of this runtime only. The entrypoint documents this at the call site, and the finding needs a reviewer to mark it as safe. Signed-off-by: btxu-db <btxu-db@outlook.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6175 +/- ##
=======================================
Coverage 65.19% 65.19%
=======================================
Files 486 486
Lines 34150 34150
=======================================
Hits 22263 22263
Misses 10136 10136
Partials 1751 1751 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
2 participants
Ⅰ. Describe what this PR does
Adds an e2e case covering a client-less CacheRuntime topology — a
CacheRuntimeClasswhosetopologydeclares onlymasterandworker, with noclientcomponent.No existing case covers that shape: the curvine case ships a client component, so the client-less path through
cacheruntime-controlleris currently untested. Mooncake is used as the concrete cache system because it genuinely has no POSIX mount semantics — applications talk to the cache service through its own client library rather than a mount point.test/gha-e2e/mooncake/— the case itself (CacheRuntimeClass/Dataset/CacheRuntimemanifests, a read/write Job, a negative-case pod, andtest.sh).test/gha-e2e/mooncake/image/— the image build context: a Dockerfile on top ofpython:3.12.13-slimplus the two scripts Fluid invokes (custom-entrypoint.sh,reportSummary.sh)..github/scripts/build-all-images.sh— build and kind-load${IMG_REPO}/mooncake:e2e..github/scripts/gha-e2e.sh— run the new case aftercurvine_e2e.Ⅱ. Does this pull request fix one issue?
NONE
Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
This PR is itself an e2e case. It asserts:
panic:/nil pointer dereferencewhile waiting, so a regression reports its root cause instead of just a Bound timeout.cacheruntime.status.client.phasestays empty. (status.clientdoes exist as{"phase":""}andspec.clientis filled in by CRD defaults, so the empty phase is the actual criterion.)status.cacheStatesgets populated,ufsTotal == cacheCapacityfor this UFS-less system, andcached/fileNumreflect data actually written.volumes/volumeMountsanywhere in the pod.FailedMountevent mentioning the missing FUSE mount point. This is the behaviour the docs' FAQ describes, pinned down so it cannot go stale silently.On failure the case dumps controller logs,
describeoutput for the CacheRuntime / Dataset / stuck pod, job logs and namespace events.Ⅳ. Describe how to verify it
against a kind cluster with Fluid deployed and
fluidcloudnative/mooncake:e2eloaded, which is what.github/scripts/build-all-images.shand.github/scripts/gha-e2e.shwire up for CI. The case was run end to end locally on kind.Ⅴ. Special notes for reviews
Guards fix: avoid nil pointer dereference in CacheRuntime configmap builder #6157. That fix is already merged; without it the controller panics on this topology and the case fails at
wait_dataset_bound, which is precisely the regression this case is here to catch.Related to docs: add a Mooncake CacheRuntime sample for client-less cache systems #6163, which documents this same client-less setup and now carries the same build context under
samples/mooncake/docker/. The two copies are kept in sync deliberately: the sample is meant to be read and modified, while this one is pinned for reproducibility (base image by digest, every resolved pip version explicit, wheels only so no package build script runs at image build time). If reviewers would rather have a single copy, I am happy to make one reference the other once both land.The image is built in-repo rather than pulled from an external registry, following
test/gha-e2e/jindo/oss-emulator: e2e runs on every PR, an external image going away turns the pipeline red, and the two scripts Fluid invokes need to be reviewable. The only external dependency is Mooncake's official PyPI package.python:3.12.13-slimis pinned by digest; the non-slim variant would grow the image from ~470MB to ~1.9GB, which e2e then has to kind-load as well.On the SonarCloud findings. Three are fixed in the second commit: the negative-case pod now declares memory and ephemeral-storage limits, and the image runs as an unprivileged user (uid 10001). The latter was verified rather than assumed, by running
mooncake_masterfrom the built image as that user: it binds 50051, 8080 and 9003 and keeps serving. Nothing needs root here since every port is above 1024, the components write nothing to the container filesystem, and the tiered store quota reaches Mooncake as a process memory segment rather than a directory.The fourth, on clear-text protocols, has no code fix available. Mooncake ships no TLS variant for either connection: the master's built-in metadata server (
--enable_http_metadata_server) only speaks HTTP, and the transfer engine offers onlytcpandrdma. Both connections stay inside the cluster, are addressed by service DNS, and carry cache blocks between components of this runtime only.custom-entrypoint.shdocuments this at the call site; the hotspot needs a reviewer to mark it as safe.One worker replica, since the CI kind cluster is single-node and replica scaling is already covered by the curvine case.
test/gha-e2e/mooncake/cacheruntimeclass.yamldeliberately declares noresourceson its containers, and says why in a comment. A template that sets resources while the CacheRuntime does not currently has them overwritten with an empty value on the first reconcile ([BUG]containerresourcesin CacheRuntimeClass are silently dropped #6161), and if the resulting rollout flips the Dataset toFailedit does not recover on its own ([BUG]CacheRuntime: Dataset staysFailedafter the runtime recovers #6160). Neither is what this case is meant to cover, so it stays clear of both. Once fix(cache): preserve CacheRuntimeClass template resources when unset #6165 lands, resources can be declared here and asserted on.