[fix](executor) prevent BE crash when split process throws unexpectedly#62044
Open
eldenmoon wants to merge 2 commits intoapache:masterfrom
Open
[fix](executor) prevent BE crash when split process throws unexpectedly#62044eldenmoon wants to merge 2 commits intoapache:masterfrom
eldenmoon wants to merge 2 commits intoapache:masterfrom
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
There was a problem hiding this comment.
Pull request overview
Prevents backend worker-thread termination (and potential BE crash) when split->process() throws unexpectedly in the time-sharing scan executor by converting thrown exceptions into split failure statuses.
Changes:
- Wrap
PrioritizedSplitRunner::process()invocation in a try/catch to prevent exceptions from escaping the dispatch thread. - Map
doris::Exception(including special-casingMEM_ALLOC_FAILED) and other exceptions to appropriateStatuserrors returned viaResult. - Keep
enable_thread_catch_bad_allocscoped aroundsplit->process()to preserve existing memory-exception behavior.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp
Outdated
Show resolved
Hide resolved
Comment on lines
+556
to
+569
| auto blocked_future_result = [&]() -> Result<SharedListenableFuture<Void>> { | ||
| try { | ||
| doris::enable_thread_catch_bad_alloc++; | ||
| Defer defer {[&]() { doris::enable_thread_catch_bad_alloc--; }}; | ||
| return split->process(); | ||
| } catch (const doris::Exception& e) { | ||
| if (e.code() == doris::ErrorCode::MEM_ALLOC_FAILED) { | ||
| return unexpected(Status::MemoryLimitExceeded( | ||
| "PreCatch error code:{}, {}, __FILE__:{}, __LINE__:{}, " | ||
| "__FUNCTION__:{}", | ||
| e.code(), e.to_string(), __FILE__, __LINE__, __PRETTY_FUNCTION__)); | ||
| } | ||
| return unexpected(e.to_status()); | ||
| } catch (const std::exception& e) { |
There was a problem hiding this comment.
This try/catch block duplicates the exception-to-Status mapping logic already implemented in common/exception.h (including the enable_thread_catch_bad_alloc guard and the "PreCatch" message). To avoid future divergence, consider factoring the conversion into a shared helper (e.g., a function that converts an Exception/std::exception to Status) and reuse it here for the Result<...> error path.
Catch exceptions around split->process() in TimeSharingTaskExecutor and convert them to split failure status. This avoids worker thread termination and BE crash for cases : ``` erminate called after throwing an instance of 'doris::Exception' what(): [E-7412] assert cast err:[E-7412] Bad cast from ... doris::vectorized::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:420 10# doris::vectorized::PrioritizedSplitRunner::process() at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/executor/time_sharing/prioritized_split_runner.cpp:104 11# doris::vectorized::TimeSharingTaskExecutor::_dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/executor/time_sharing/time_sharing_task_executor.cpp:568 12# ``` w , while keeping MEM_ALLOC_FAILED mapped to MemoryLimitExceeded.
eldenmoon
force-pushed
the
exp-safe-executor
branch
from
c1429c7 to
a67425a
Compare
…executor.cpp Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Member
Author
|
/review |
Member
Author
|
run buildall |
TPC-H: Total hot run time: 29125 ms |
TPC-DS: Total hot run time: 179999 ms |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Member
Author
|
skip check_coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Catch exceptions around split->process() in TimeSharingTaskExecutor and
convert them to split failure status.
This avoids worker thread termination and BE crash for cases :
, while keeping MEM_ALLOC_FAILED mapped to
MemoryLimitExceeded.