[BUG] cluster state call blocks http worker thread

[bowenlan-amzn]

Describe the bug

When handling /_cluster/state/_all/_all requests, the ClusterState.toXContent() serialization runs synchronously on the Netty HTTP worker thread. For large clusters, this serialization can take 4+ seconds, during which the HTTP worker thread is blocked and cannot process other requests.

Related component

Cluster Manager

To Reproduce

 98.2% (491ms out of 500ms) cpu usage by thread 'opensearch[...][http_server_worker][T#2]'
      org.opensearch.common.settings.Settings.convertMapsToArrays(Settings.java:195)
      org.opensearch.common.settings.Settings.toXContent(Settings.java:602)
      org.opensearch.cluster.metadata.IndexMetadata$Builder.toXContent(IndexMetadata.java:1875)
      org.opensearch.cluster.metadata.Metadata$Builder.toXContent(Metadata.java:1962)
      org.opensearch.cluster.metadata.Metadata.toXContent(Metadata.java:1088)
      org.opensearch.cluster.ClusterState.toXContent(ClusterState.java:535)
      org.opensearch.rest.action.admin.cluster.RestClusterStateAction$1.buildResponse(RestClusterStateAction.java:166)
      ...
      io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
      io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)

Expected behavior

Heavy serialization work should be offloaded to a generic thread pool instead of running on the Netty I/O thread, similar to how other CPU-intensive operations are handled.

Could be fixed by following this pattern, offloading to management thread pool.

OpenSearch/server/src/main/java/org/opensearch/rest/action/admin/indices/RestGetMappingAction.java

Lines 122 to 137 in bbdf73a

	// Process serialization on GENERIC pool since the serialization of the raw mappings to XContent can be too slow to execute
	// on an IO thread
	threadPool.executor(ThreadPool.Names.MANAGEMENT)
	.execute(ActionRunnable.wrap(this, l -> new RestBuilderListener<GetMappingsResponse>(channel) {
	@Override
	public RestResponse buildResponse(final GetMappingsResponse response, final XContentBuilder builder)
	throws Exception {
	if (threadPool.relativeTimeInMillis() - startTimeMs > timeout.millis()) {
	throw new OpenSearchTimeoutException("Timed out getting mappings");
	}
	builder.startObject();
	response.toXContent(builder, request);
	builder.endObject();
	return new BytesRestResponse(RestStatus.OK, builder);
	}
	}.onResponse(getMappingsResponse)));

Additional Details

No response

[msfroh]

Another possible solution (versus using the generic threadpool) would be to distribute the work across the cluster. If the coordinating node knows that it's just going to return the serialized response for all nodes, it can ask each node for a serialized response, then concatenate those together.

I guess it comes down to deciding if we want to override the toXContent() behavior of the cluster-level response or the node-level response.

[sgup432]

@bowenlan-amzn /_cluster/state/_all/_all should be handled by the cluster manager node, right? If yes, it won't block search/indexing but other requests coming to cluster manager(including pings).

Cluster manager already has the complete cluster state, so it would be better for it to serialize the response locally (without distributing the request) but do so on a separate thread?

If we saw this issue on a data node, then I think these cluster state requests should be internally routed to a active cluster manager node.

[bowenlan-amzn]

Execution doesn't seem to take much time, but JSON serialization takes nearly 4s in this case. It affects other http requests because it keeps a netty http worker busy for that long time, so any http requests get assigned to the worker at this period need to wait. This is a data node.

[sgup432]

This is a data node.

Should we also route such requests internally via transport actions to active cluster manager node? Along with doing such work on a separate thread?

[msfroh]

Another possible solution (versus using the generic threadpool) would be to distribute the work across the cluster.

I was completely wrong on this point. I was thinking of _nodes/stats, which does broadcast to all nodes and then aggregates their responses. As @sgup432 called out, this is just getting the cluster state, which the cluster manager (and every node) has in full. (Not having full cluster state on every node is essentially the goal of the "cloud-native" project.)

Should we also route such requests internally via transport actions to active cluster manager node? Along with doing such work on a separate thread?

The TransportClusterStateAction does get routed to the cluster manager node, unless the local=true URL parameter is specified. (Functionally, since every node has the full cluster state at all times, the response should be the same whether you ask the cluster manager or any random node, unless we're in the middle of publishing a new cluster state, in which case the "correct" state is ambiguous anyway.)

The problem is that the transport call gets routed to the cluster manager (or stays on the local node) and the binary cluster state is sent back to the node that received the request. Then that node (via RestClusterStateAction) serializes the response to the appropriate REST response type (usually JSON). That happens on the response thread.

---

I'm absolutely blown away that cluster state -- the thing that gets propagated to every node in a strongly-consistent way -- would get so big as to take 4 seconds to serialize. Given that a big part of software engineering is about sharing only what's needed, that's way too much sharing.

Like, I get it -- the JSON representation of cluster state includes the node-assignment of every copy of every shard twice (once from node -> shard and once from index -> shard -> node), which makes things extra silly, but wow... that cluster state is way too big to essentially be a distributed global variable.

[sgup432]

The problem is that the transport call gets routed to the cluster manager (or stays on the local node) and the binary cluster state is sent back to the node that received the request. Then that node (via RestClusterStateAction) serializes the response to the appropriate REST response type (usually JSON). That happens on the response thread.

Ah this makes sense now. Thanks for the deep dive.

I'm absolutely blown away that cluster state -- the thing that gets propagated to every node in a strongly-consistent way -- would get so big as to take 4 seconds to serialize. Given that a big part of software engineering is about sharing only what's needed, that's way too much sharing.

I think that the cluster state isn’t fully shared between the cluster manager and other nodes in OpenSearch, only the diffs (or new changes) are sent, and each node applies them to its local cluster state. This happens whenever there are new changes in cluster state. Is that what you meant here?
In such API calls, it seems like the cluster manager node is sending full response(full cluster state) to the data node which seems unnecessary and certainly needs a correction.