Skip to content

Workaround for Windows file locking issues during Message Processor task deletion - #4571

Open
ochnios wants to merge 1 commit into
wso2:masterfrom
ochnios:fix/unable_to_delete_task_file_error_on_windows
Open

Workaround for Windows file locking issues during Message Processor task deletion#4571
ochnios wants to merge 1 commit into
wso2:masterfrom
ochnios:fix/unable_to_delete_task_file_error_on_windows

Conversation

@ochnios

@ochnios ochnios commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Purpose

Resolves #4570

Analysis

I couldn't find any place in code with possible resource leak like unclosed stream or something like that so I assume the issue lies somewhere deeper (some bug on JVM for Windows like this?). After failed deletion I can still see open file handles in Windows Resource Monitor (resmon.exe). Both are from the same Micro Integrator process:
screenshot

A similar workaround exists elsewhere in the codebase to handle "known bug in windows" (here). This PR applies a similar strategy.

Goals

  • Fix file locking - ensure task definition files are released and deleted successfully during undeployment on Windows.
  • Prevent zombie state - ensure Message Processors start cleanly after redeployment, rather than sticking in a PAUSED state due to stale metadata files that couldn't be deleted.

Approach

I introduced a deleteWithRetry method in FileBasedTaskRepository. Inside the retry loop, System.gc() is invoked when deletion of the registry files fails. Triggering GC forces the release of file handles which are kept, allowing the OS to unlock and delete the file. A small delay is added between retries to allow the OS file system to catch up.

I acknowledge that using System.gc() is a "last resort" workaround. However, given the aggressive file locking this was the only solution that consistently resolved the issue in my testing.

Summary by CodeRabbit

  • Bug Fixes
    • Enhanced task file deletion reliability with automatic retry mechanism to handle temporary deletion failures more gracefully.
    • Improved error logging for task management operations to assist with troubleshooting and diagnostics.

✏️ Tip: You can customize this high-level summary in your review settings.

@ochnios
ochnios requested a review from rosensilva as a code owner January 13, 2026 18:28
Comment on lines +359 to +362
private boolean deleteWithRetry(File file) {
if (!file.exists()) {
return false;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log Improvement Suggestion No: 1

Suggested change
private boolean deleteWithRetry(File file) {
if (!file.exists()) {
return false;
}
private boolean deleteWithRetry(File file) {
if (!file.exists()) {
return false;
}
log.info("Attempting to delete file: " + file.getAbsolutePath());

Comment on lines +364 to +367
for (int attempt = 1; attempt <= TASK_DELETE_MAX_ATTEMPTS; attempt++) {
if (file.delete()) {
log.debug("File deleted successfully on attempt " + attempt + ": " + file.getAbsolutePath());
return true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log Improvement Suggestion No: 2

Suggested change
for (int attempt = 1; attempt <= TASK_DELETE_MAX_ATTEMPTS; attempt++) {
if (file.delete()) {
log.debug("File deleted successfully on attempt " + attempt + ": " + file.getAbsolutePath());
return true;
for (int attempt = 1; attempt <= TASK_DELETE_MAX_ATTEMPTS; attempt++) {
if (file.delete()) {
if (log.isDebugEnabled()) {
log.debug("File deleted successfully on attempt " + attempt + ": " + file.getAbsolutePath());
}
return true;

@wso2-engineering wso2-engineering Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Agent Log Improvement Checklist

⚠️ Warning: AI-Generated Review Comments

  • The log-related comments and suggestions in this review were generated by an AI tool to assist with identifying potential improvements. Purpose of reviewing the code for log improvements is to improve the troubleshooting capabilities of our products.
  • Please make sure to manually review and validate all suggestions before applying any changes. Not every code suggestion would make sense or add value to our purpose. Therefore, you have the freedom to decide which of the suggestions are helpful.

✅ Before merging this pull request:

  • Review all AI-generated comments for accuracy and relevance.
  • Complete and verify the table below. We need your feedback to measure the accuracy of these suggestions and the value they add. If you are rejecting a certain code suggestion, please mention the reason briefly in the suggestion for us to capture it.
Comment Accepted (Y/N) Reason
#### Log Improvement Suggestion No: 1
#### Log Improvement Suggestion No: 2

@coderabbitai

coderabbitai Bot commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

The change introduces retry-based deletion logic to FileBasedTaskRepository, replacing direct file deletion calls with a private deleteWithRetry method that attempts up to 3 times with 50ms delays between attempts. This addresses transient file locking issues that prevent task file deletion during redeployment operations.

Changes

Cohort / File(s) Summary
Retry-based task file deletion mechanism
components/mediation/tasks/org.wso2.micro.integrator.ntask.core/src/main/java/org/wso2/micro/integrator/ntask/core/impl/FileBasedTaskRepository.java
Added TASK_DELETE_MAX_ATTEMPTS and TASK_DELETE_DELAY_MS constants. Introduced private deleteWithRetry(File) method implementing exponential retry logic with GC hints and per-attempt logging. Replaced two direct file.delete() invocations with calls to the new method, improving robustness against transient file locks without altering public API.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 A rabbit hops through Windows lock,
Three times we'll try, we'll never stop,
With fifty-millisecond grace,
Task files vanish without a trace!
Files fly free, no more PAUSED woe,
hop hop hop

🚥 Pre-merge checks | ✅ 3 | ❌ 2
❌ Failed checks (1 warning, 1 inconclusive)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ❓ Inconclusive The description covers Purpose, Goals, and Approach sections but omits several required template sections including Automation tests, Security checks, Test environment, and other metadata fields. Complete the description by adding missing sections: Automation tests (unit/integration), Security checks, Test environment details, and other applicable template sections like Documentation and Release note.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The code changes directly implement the solution to issue #4570: introducing a deleteWithRetry method with System.gc() calls to force file handle release and retry logic with delays, matching the stated goals of fixing Windows file locking and preventing stale metadata files.
Out of Scope Changes check ✅ Passed All changes are scoped to FileBasedTaskRepository.java and directly address the file deletion retry mechanism required by issue #4570, with no unrelated modifications present.
Title check ✅ Passed The title accurately describes the main change: a workaround for Windows file locking issues during Message Processor task deletion, which matches the core objective of introducing retry-based deletion logic in FileBasedTaskRepository.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@ochnios ochnios changed the title fix: workaround for Windows file locking issues during MP task deletion Workaround for Windows file locking issues during Message Processor task deletion Jan 13, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
components/mediation/tasks/org.wso2.micro.integrator.ntask.core/src/main/java/org/wso2/micro/integrator/ntask/core/impl/FileBasedTaskRepository.java (1)

376-379: Restore the interrupt status after catching InterruptedException.

When catching InterruptedException, best practice is to restore the interrupt flag so that code higher in the call stack can detect that an interrupt occurred.

Suggested fix
             } catch (InterruptedException e) {
+                Thread.currentThread().interrupt();
                 log.warn("Interrupted while waiting to retry file deletion: " + file.getAbsolutePath());
                 break;
             }
📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6c15a65 and 4cb30d3.

📒 Files selected for processing (1)
  • components/mediation/tasks/org.wso2.micro.integrator.ntask.core/src/main/java/org/wso2/micro/integrator/ntask/core/impl/FileBasedTaskRepository.java
🔇 Additional comments (4)
components/mediation/tasks/org.wso2.micro.integrator.ntask.core/src/main/java/org/wso2/micro/integrator/ntask/core/impl/FileBasedTaskRepository.java (4)

64-65: Constants are reasonable for the retry mechanism.

The values (3 attempts, 50ms delay) provide a sensible balance between resilience and responsiveness, with a maximum additional blocking time of ~150ms. Given this is a workaround for a Windows-specific edge case, hardcoded values are acceptable.


274-276: LGTM!

Clean delegation to the retry helper with correct return value handling.


281-284: LGTM!

Correctly gates the in-memory map cleanup on successful file deletion, preventing state inconsistency.


359-385: Pragmatic workaround for Windows file locking.

The System.gc() hint is a known last-resort approach for releasing file handles held by phantom references on Windows. While non-deterministic (JVM may ignore the hint), the retry loop with delays provides reasonable resilience. Given the PR description notes this matches patterns elsewhere in the codebase and was the only consistently working solution in testing, this is acceptable.

@ochnios

ochnios commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

@rosensilva could you take a look at this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Task deletion fails during CAPP redeployment causing Message Processors to stick in PAUSED state

1 participant