doris-4.0.3 Potential Memory Leak: PENDING Load Jobs Not Being Cleaned Up in LoadManager

[477972324wlx]

Environment

• Doris Version: doris-4.0.3-rc03 (Cloud Mode)
• Load Method: HTTP Stream Load (mode=async)
• Key Configuration: label_keep_max_second = 600

Description

I have observed a steady growth in JVM Old Generation memory usage. Upon investigating with Arthas, I found that the LoadManager is holding a massive list of tasks that are stuck in the PENDING state.

On one of my nodes, the number of pending tasks has reached over 4 million:

# Check the count of PENDING tasks in LoadManager
[arthas@2844843]$ ognl '@org.apache.doris.catalog.Env@getCurrentEnv().getLoadManager().idToLoadJob.values().{? #this.getState().name() == "PENDING"}.size()'
@Integer[4048847]

Looking at the task details, many of these PENDING jobs were created several days ago (based on timestamps like 1774695394786) and have not been purged, despite the label_keep_max_second being set to 600 seconds.

# Sampling PENDING tasks (showing State--CreateTimestamp:Label#Id)
[arthas@2844843]$ ognl '#list=@org.apache.doris.catalog.Env@getCurrentEnv().getLoadManager().idToLoadJob.values().{? #this.getState().name() == "PENDING"}, #size=#list.size(), #end=#size > 10 ? 10 : #size, new java.util.ArrayList(#list.subList(0, #end)).{#this.getState().name()+"--"+#this.getCreateTimestamp() + ":" + #this.getLabel()+"#"+#this.getId()}'

@ArrayList[
    @String[PENDING--1774695394786:group_commit_834ce3a6d7aa549c_0ec5c85be1a6128d#1774602956553],
    @String[PENDING--1774695393686:group_commit_1f4c5fa642f4e779_ddcc3cc03a68428a#1774602956552],
    @String[PENDING--1774695395590:group_commit_274264d3e3c52a75_c3abcdab7dc1d7bd#1774602956555],
    ...
]

Source Code Anaylisis

    public void removeOldLoadJob() {
        long currentTimeMs = System.currentTimeMillis();
        removeLoadJobIf(job -> job.isExpired(currentTimeMs));
    }

    public boolean isExpired(long currentTimeMs) {
       // Logic Issue: If the job remains in PENDING state, isCompleted() is false,
        // so the job will NEVER be removed by the cleaner.
        if (!isCompleted()) {
            return false;
        }
        long expireTime = Config.label_keep_max_second;
        if (jobType == EtlJobType.INSERT) {
            expireTime = Config.streaming_label_keep_max_second;
        }

        if (LOG.isDebugEnabled()) {
            LOG.debug("Job ID: {}, DB ID: {}, Label: {}, State: {}, Expire Time: {}, Current Time: {}, "
                      + "Finish Timestamp: {}", id, dbId, label, state, expireTime, currentTimeMs,
                      getFinishTimestamp());
        }
        return (currentTimeMs - getFinishTimestamp()) / 1000 > expireTime;
    }

    public boolean isCompleted() {
        return state == JobState.FINISHED || state == JobState.CANCELLED || state == JobState.UNKNOWN;
    }

Questions

• Design Rationale: Is it intentional that PENDING jobs have no timeout mechanism? Should there be a way to expire tasks that stay in PENDING for too long?
• Root Cause: In the context of group_commit, what scenarios could cause such a high volume of tasks to hang in PENDING without transitioning to another state or being cancelled?