[fix](memory) track IO layer read buffers via MemTrackerLimiter using PODArray#62032
Open
sollhui wants to merge 1 commit intoapache:masterfrom
Open
[fix](memory) track IO layer read buffers via MemTrackerLimiter using PODArray#62032sollhui wants to merge 1 commit intoapache:masterfrom
sollhui wants to merge 1 commit intoapache:masterfrom
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
sollhui
force-pushed
the
fix_io_memory_tracker
branch
from
994e1b2 to
d65f836
Compare
Contributor
Author
|
run buildall |
sollhui
force-pushed
the
fix_io_memory_tracker
branch
from
d65f836 to
4e9989b
Compare
Contributor
Author
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 29179 ms |
TPC-DS: Total hot run time: 178890 ms |
sollhui
force-pushed
the
fix_io_memory_tracker
branch
from
4e9989b to
7747afe
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 29396 ms |
TPC-DS: Total hot run time: 179812 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
sollhui
force-pushed
the
fix_io_memory_tracker
branch
from
7747afe to
35bb6d5
Compare
Contributor
Author
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Apr 2, 2026
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
yiguolei
approved these changes
Apr 2, 2026
TPC-H: Total hot run time: 29162 ms |
TPC-DS: Total hot run time: 180691 ms |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Several IO layer read buffers were allocated via
std::make_unique<char[]>/new char[], which bypasses Doris's memory tracking system (MemTrackerLimiter).These allocations are invisible to the query memory tracker, leading to
under-reported memory usage and potential OOM surprises under concurrent S3/HDFS
scans.
Affected locations:
PrefetchBuffer::_bufinbuffered_reader— the main S3/HDFS prefetch bufferHttpFileReader::_read_buffer— the 1 MB HTTP read bufferHdfsFileSystem::download_impl— the 1 MB copy buffer for remote→local downloadSolution
Replace raw
unique_ptr<char[]>allocations withPODArray<char>, which usesDoris's
Allocator<..., check_and_tracking_memory=true>internally. Everyalloc/realloc/freecall goes throughconsume_memory/release_memoryon the thread-local
MemTrackerLimiter, so these buffers are now properlyaccounted for.
Key behavioral notes:
PODArraydefault constructor allocates no memory, so the lazy-allocationoptimization in
PrefetchBuffer(introduced to reduce peak memory during TVFscans over many small S3 files) is fully preserved — the buffer is only
allocated on the first actual prefetch call.
PODArraysupports move semantics, so the existing move constructor ofPrefetchBufferworks unchanged.Changes
buffered_reader.h_buf:unique_ptr<char[]>→PODArray<char>; remove eagernew char[]from constructorbuffered_reader.cpp_buf.empty()/_buf.resize);.get()→.data()forPrefetchBufferhttp_file_reader.h_read_buffer:unique_ptr<char[]>→PODArray<char>http_file_reader.cppmake_unique<char[]>→resize;.get()→.data()hdfs_file_system.cppnew char[]→PODArray<char>