Skip to content

HydratedTarget/Release controllers fail with "object has been modified" error #320

@dermorz

Description

@dermorz

Describe the bug
The HydratedTarget and Release controllers intermittently fail with optimistic concurrency errors when updating status:
failed to update status: Operation cannot be fulfilled on hydratedtargets.solar.opendefense.cloud "xxx": the object has been modified; please apply your changes to the latest version and try again

This error occurs in:

  • hydratedtarget_controller.go
  • release_controller.go

To Reproduce
Steps to reproduce the behavior:

  1. Run controller tests with make test testargs="-v ./pkg/controller/"
  2. Observe intermittent failures with "the object has been modified" errors

Expected behavior
Controllers should handle concurrent updates by re-fetching the object and retrying the update.

Possible Fix
Add retry logic when updating status, re-fetching the object if a conflict is detected:

// Re-fetch the object and retry update on conflict
updated := &solarv1alpha1.HydratedTarget{}
if err := r.Get(ctx, req.NamespacedName, updated); err != nil {
    return ctrlResult, err
}
// Apply changes to updated object and retry

Additional context
Log example:
ERROR Reconciler error {"controller": "hydratedtarget", "error": "failed to create RenderTask: failed to update status: Operation cannot be fulfilled on hydratedtargets.solar.opendefense.cloud \"test-ht-gen\": the object has been modified; please apply your changes to the latest version and try again"}
ERROR Reconciler error {"controller": "release", "error": "failed to create RenderTask: failed to update status: Operation cannot be fulfilled on releases.solar.opendefense.cloud \"test-release-gen\": the object has been modified; please apply your changes to the latest version and try again"}

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds-triageIssue that has not been reviewed

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions