-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Describe the bug
The HydratedTarget and Release controllers intermittently fail with optimistic concurrency errors when updating status:
failed to update status: Operation cannot be fulfilled on hydratedtargets.solar.opendefense.cloud "xxx": the object has been modified; please apply your changes to the latest version and try again
This error occurs in:
hydratedtarget_controller.gorelease_controller.go
To Reproduce
Steps to reproduce the behavior:
- Run controller tests with
make test testargs="-v ./pkg/controller/" - Observe intermittent failures with "the object has been modified" errors
Expected behavior
Controllers should handle concurrent updates by re-fetching the object and retrying the update.
Possible Fix
Add retry logic when updating status, re-fetching the object if a conflict is detected:
// Re-fetch the object and retry update on conflict
updated := &solarv1alpha1.HydratedTarget{}
if err := r.Get(ctx, req.NamespacedName, updated); err != nil {
return ctrlResult, err
}
// Apply changes to updated object and retryAdditional context
Log example:
ERROR Reconciler error {"controller": "hydratedtarget", "error": "failed to create RenderTask: failed to update status: Operation cannot be fulfilled on hydratedtargets.solar.opendefense.cloud \"test-ht-gen\": the object has been modified; please apply your changes to the latest version and try again"}
ERROR Reconciler error {"controller": "release", "error": "failed to create RenderTask: failed to update status: Operation cannot be fulfilled on releases.solar.opendefense.cloud \"test-release-gen\": the object has been modified; please apply your changes to the latest version and try again"}