Skip to content

Updating composite engine reference before translog recovery #20449

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bharath-techie merged 1 commit into from
Jan 22, 2026
Merged

Updating composite engine reference before translog recovery #20449

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ...atafusion/src/test/java/org/opensearch/datafusion/DataFusionRemoteStoreRecoveryTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -435,7 +435,7 @@ public void testReplicaPromotionWithTranslogReplay() throws Exception {
.setMapping(mappings).get());
ensureGreen(INDEX_NAME);

int initialDocs = 5;
int initialDocs = randomIntBetween(1, 10);
for (int i = 1; i <= initialDocs; i++) {
client().prepareIndex(INDEX_NAME).setId("initial_doc" + i)
.setSource("{ \"value\": " + (i * 100) + ", \"phase\": \"initial\" }", MediaTypeRegistry.JSON).get();
Expand All @@ -444,7 +444,7 @@ public void testReplicaPromotionWithTranslogReplay() throws Exception {
client().admin().indices().prepareRefresh(INDEX_NAME).get();
ensureGreen(INDEX_NAME);

int uncommittedDocs = 7;
int uncommittedDocs = randomIntBetween(1, 10);
for (int i = 1; i <= uncommittedDocs; i++) {
client().prepareIndex(INDEX_NAME).setId("uncommitted_doc" + i)
.setSource("{ \"value\": " + (i * 200) + ", \"phase\": \"uncommitted\" }", MediaTypeRegistry.JSON).get();
Expand Down
5 changes: 4 additions & 1 deletion server/src/main/java/org/opensearch/index/shard/IndexShard.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5534,6 +5534,7 @@ public void close() throws IOException {

// Only create CompositeEngine for optimized indices
CompositeEngine newCompositeEngine;
CompositeEngine oldCompositeEngine;
if (indexSettings.isOptimizedIndex()) {
// Create NEW CompositeEngine OUTSIDE synchronized block with fresh translog
newCompositeEngine = new CompositeEngine(
Expand All @@ -5545,6 +5546,7 @@ public void close() throws IOException {
LocalCheckpointTracker::new,
TranslogEventListener.NOOP_TRANSLOG_EVENT_LISTENER
);
oldCompositeEngine = currentCompositeEngineReference.getAndSet(newCompositeEngine);
Copy link

@ask-kamal-nayan ask-kamal-nayan Jan 21, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we also add it in try block so that even if some exception comes after setting the newCompositeEngine while translog Recovery and all, still we can close the old engine?


final TranslogRecoveryRunner translogRunner = (snapshot) -> runTranslogRecovery(
newCompositeEngine,
Expand All @@ -5562,6 +5564,7 @@ public void close() throws IOException {
newCompositeEngine.refresh("reset_engine");
} else {
newCompositeEngine = null;
oldCompositeEngine = null;
}

// Create InternalEngine AFTER translog recovery so it reads the updated commit with correct checkpoints
Expand All @@ -5588,7 +5591,7 @@ public void close() throws IOException {

synchronized (engineMutex) {
verifyNotClosed();
IOUtils.close(currentEngineReference.getAndSet(newEngineReference.get()), currentCompositeEngineReference.getAndSet(newCompositeEngine));
IOUtils.close(currentEngineReference.getAndSet(newEngineReference.get()), oldCompositeEngine);

// onNewEngine must be called inside synchronized(engineMutex) block for both optimized and non-optimized indices
// We set active because we are now writing operations to the engine; this way,
Expand Down
Loading