[BUG] JVM memory pressure build due to TransportBulkRequest pile up because of cluster state objects.

[SwethaGuptha]

What is the bug?
Having too many outstanding TransportBulkAction request can lead to memory build up on the node as the each request handler holds a reference to cluster state object.

https://github.com/opensearch-project/OpenSearch/blob/6b5c08aecf496b8fbdeb6314b841713684dc49df/server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java#L596C1-L603C57

        protected void doRun() {
            assert bulkRequest != null;
            final ClusterState clusterState = observer.setAndGetObservedState();
            if (handleBlockExceptions(clusterState)) {
                return;
            }
            final ConcreteIndices concreteIndices = new ConcreteIndices(clusterState, indexNameExpressionResolver);
            Metadata metadata = clusterState.metadata();

OpenSearch/server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java

Lines 936 to 941 in 6b5c08a

	static class ConcreteIndices {
	private final ClusterState state;
	private final IndexNameExpressionResolver indexNameExpressionResolver;
	private final Map<String, Index> indices = new HashMap<>();
	ConcreteIndices(ClusterState state, IndexNameExpressionResolver indexNameExpressionResolver) {

    static class ConcreteIndices {
        private final ClusterState state;
        private final IndexNameExpressionResolver indexNameExpressionResolver;
        private final Map<String, Index> indices = new HashMap<>();


        ConcreteIndices(ClusterState state, IndexNameExpressionResolver indexNameExpressionResolver) {

The high memory JVM pressure was observed on a cluster where there were 500+ outstanding indexing request on a node and the cluster state size was 55MB, a total of 30GB memory was consumed in this case.

500 cluster state objects

[rithin-pullela-aws]

Thanks for raising the issue @SwethaGuptha.
Can we tranfer this bug to OpenSearch Core?

[SwethaGuptha]

Yes, it should be addressed in OpenSearch Core.

[cwperks]

Transferred to the core repo

[andrross]

Catch All Triage - 1 2 3

[batcity]

Hey folks, I did some research and found out that the number of cluster state objects == the number of bulk requests even before we get to this point here: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java#L593

I'm not sure where the root of this issue actually begins

However I've removed references to cluster state from ConcreteIndices objects for now in this PR: #20454

cc: @SwethaGuptha @rithin-pullela-aws @cwperks @andrross

[cwperks]

@batcity how large is the cluster?

May be related to opensearch-project/ml-commons#4362

[batcity]

@cwperks this is just a single node cluster (I'm running it on my local machine), I've just simulated the cluster state changing using this script

for i in $(seq 1 4); do
  echo "Iteration $i"

  # Start bulk asynchronously
  curl -s -XPOST localhost:9200/_bulk \
    -H "Content-Type: application/x-ndjson" \
    --data-binary @bulk.json &

  # Force a cluster state update
  curl -s -XPUT localhost:9200/churn_index_$i \
    -H "Content-Type: application/json" \
    -d '{
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 0
      }
    }' >/dev/null
done

The screenshot below shows that the number of cluster state objects seem to be == to the number of inflight bulk requests (4 in this case, the additional 2 are present even if there are no bulk requests) even after I've removed references to the cluster state objects from concrete indexes