[BUG] Generated PIT ID is exceptionally long (70kb)

[slppo]

Describe the bug

I have an index over 20TB large, and with the current cluster configuration whenever I try to create a point-in-time index to paginate on, the generated PIT ID is over 70kb long.

I would like to understand if there's something about our cluster of index configuration that may be causing this, or if it's reflective of an internal bug.

Snippet for reference:

Request

POST /my-index/_search/point_in_time?keep_alive=30m

Response

{
  "pit_id": "... < TRUNCATED > ...",
  "_shards": {
    "total": 525,
    "successful": 525,
    "skipped": 0,
    "failed": 0
  },
  "creation_time": 1767992611392
}

Related component

Search:Performance

To Reproduce

Request

POST /my-index/_search/point_in_time?keep_alive=30m

Response

{
  "pit_id": "... < TRUNCATED > ...",
  "_shards": {
    "total": 525,
    "successful": 525,
    "skipped": 0,
    "failed": 0
  },
  "creation_time": 1767992611392
}

Expected behavior

I would expect the generate PIT id to be "small" (much less than 1kb) in size

Additional Details

Plugins
Attached text file

plugins.txt

Host/Environment (please complete the following information):

• OS:
  5 c7g.2xlarge.search master nodes
  60 r7g.8xlarge.search data nodes
• Version OpenSearch_3_1_R20251112-P1

[mch2]

@slppo Thanks for your issue.

PIT Ids are encoded with shard and node IDs. The size of the ID is increasing as your shard count increases which with a 20TB index am assuming is large. Perhaps we could look at better encoding here on the IDs to reduce size.

[msfroh]

For the record, the serialization of SearchContextId is really naive and has a lot of duplicate information. (A PIT ID or a scroll ID is really a serialized SearchContextId.)

There is a Map<ShardId, SearchContextIdForNode>. The ShardId is made up of an Index and shard number. Every single shard for the same index serializes the same Index, which includes the index's name and UUID. Instead, we could store one Index per index in the PIT and encode an index ordinal as a VInt.

Each SearchContextIdForNode has the (string) node ID, a ShardSearchContextId, and an optional string for the cluster alias (in case this is a cross-cluster node). Since we probably have multiple shards on the same node, we can store the nodes in a list and reference the node ordinals. Within the ShardSearchContextId, there's a string sessionId, which is generated once on node startup and a long id, which is incremented on every (initial) shard request.

Taking all of that together, I think we can encode the whole SearchContextId as three tables (or equivalently lists of tuples):

1. Indices: (index name, index uuid).
2. Nodes: (node ID, (optional) cluster alias, sessionId)
3. ShardContexts: (index ordinal, shard num, node ordinal, ShardSearchContextId.id) (all of type VInt).

That should significantly reduce the size.

For the specific case given by @slppo, we'd have two ~22 character UUIDs per data node, so up to 2640 bytes (assuming all nodes get hit). With 525 shards, each with 4 VInts (the first 3 should all be 1-2 bytes, while the last one might be up to 8 bytes), we can reasonably assume a worst case of 14 bytes per shard. So, the PIT ID could be around 10k. It's not the "much less than 1kB" that one might hope for, but it's still much better than 70kB.

[slppo]

I think that'd be a significant improvement -- regardless we may wind up having to implement a lookup table in our service to return in API responses, but good to know that there are optimizations that may be made

[andrross]

@slppo Just curious, I know a >70kb is clunky, but what is the limiting factor for you? Are you exceeding payload size limits somewhere?

[slppo]

It's really just trying to reduce the amount of data sent over the wire to clients -- our API is high RPS (>1k RPS) so reducing the size of the response payload as much as possible is desired.