Fetch and store target cluster meta info at benchmark start by Copilot · Pull Request #2095 · elastic/rally

Copilot · 2026-04-08T16:15:20Z

Rally contacts the target cluster at benchmark start but discards useful identifying metadata. This adds target_id, target_platform, and target_auth_type to every indexed document, making it easier to correlate race results with their target clusters without manual tagging.

Changes

`esrally/telemetry.py` — `ClusterEnvironmentInfo`

Added optional client_options parameter to __init__
In on_benchmark_start, extracts and stores three new meta fields via add_meta_info(MetaInfoScope.cluster, ...) — the same mechanism as user tags — so they appear in the meta.* field of every telemetry document:
- target_id — project ID (serverless), cluster ID (ECH), or configured cluster name (on-prem)
- target_platform — "serverless" (via build_flavor), "hosted" (via x-found-handling-cluster response header from ECH), or "on-prem" (default)
- target_auth_type — "api_key" or "basic" derived from client options; omitted if neither is configured

`esrally/driver/driver.py`

prepare_telemetry() retrieves the default cluster's raw client options from config and passes them to ClusterEnvironmentInfo
Driver.prepare_benchmark() computes target_platform (including hosted detection via response headers) and target_auth_type while the API response is fresh, storing them on DriverActor
PreparationComplete message extended with target_id, target_platform, and target_auth_type

`esrally/racecontrol.py`

on_preparation_complete() receives and sets target_id, target_platform, and target_auth_type on the Race object

`esrally/metrics.py` — `Race`

Added target_id, target_platform, and target_auth_type fields
to_result_dicts() includes target-id, target-platform, and target-auth-type as top-level fields alongside distribution-version and distribution-flavor in race result documents

`tests/telemetry_test.py`

Updated existing assertions to include the new target_platform call
Added tests for target_id storage, ECH header detection, api_key/basic auth type detection, and the no-auth case

- cluster_name from GET / response (project ID for serverless, cluster ID for ECH, cluster name for on-prem) - target_platform: serverless/hosted/on-prem (hosted detected via x-found-handling-cluster response header) - target_auth_type: api_key or basic (from client options) - Pass client options to ClusterEnvironmentInfo from prepare_telemetry - Add new tests for cluster_name, target_platform, and target_auth_type Agent-Logs-Url: https://github.com/elastic/rally/sessions/3c4bd014-8eba-47d8-9ef7-7fdf59a882c4 Co-authored-by: gareth-ellis <14981026+gareth-ellis@users.noreply.github.com>

…_auth tuple test Agent-Logs-Url: https://github.com/elastic/rally/sessions/3c4bd014-8eba-47d8-9ef7-7fdf59a882c4 Co-authored-by: gareth-ellis <14981026+gareth-ellis@users.noreply.github.com>

gareth-ellis

As well as this, could you also add the three values (cluster id / cluster name) (you could refer to this as target_id), target_platform and target_auth_type to each document that is indexed? the same way we add user tags, meta.* etc.

…ace results - Rename cluster_name -> target_id in ClusterEnvironmentInfo meta info (telemetry docs) - Extend PreparationComplete message to carry target_id, target_platform, target_auth_type - Compute target_id, target_platform, target_auth_type in DriverActor._after_track_prepared() - Pass new fields through BenchmarkCoordinator.on_preparation_complete() to Race object - Add target_id, target_platform, target_auth_type to Race class and to_result_dicts() so they appear as top-level fields in race result documents alongside distribution-version Agent-Logs-Url: https://github.com/elastic/rally/sessions/1c4b9405-a245-4d6e-95f0-8358de160d35 Co-authored-by: gareth-ellis <14981026+gareth-ellis@users.noreply.github.com>

…ation to prepare_benchmark - Compute target_platform and target_auth_type in Driver.prepare_benchmark() where cluster_details response object is freshly available (avoids fragile getattr in _after_track_prepared) - Use .all_client_options.get("default", {}) instead of .default for safer client options access - Store target_platform and target_auth_type as DriverActor attributes - Remove unused versions import from driver.py Agent-Logs-Url: https://github.com/elastic/rally/sessions/1c4b9405-a245-4d6e-95f0-8358de160d35 Co-authored-by: gareth-ellis <14981026+gareth-ellis@users.noreply.github.com>

Copilot · 2026-04-08T17:13:35Z

As well as this, could you also add the three values (cluster id / cluster name) (you could refer to this as target_id), target_platform and target_auth_type to each document that is indexed? the same...

Done in commit b664cbb. The three values now appear in every indexed document in two ways:

Telemetry metrics documents (latency, throughput, etc.): target_id, target_platform, and target_auth_type are stored via add_meta_info(MetaInfoScope.cluster, ...) in ClusterEnvironmentInfo.on_benchmark_start(), so they appear in the meta.* field of every telemetry document — the same way user tags and source_revision are added.
Race result documents (from Race.to_result_dicts()): target-id, target-platform, and target-auth-type are added as top-level fields alongside distribution-version, distribution-flavor, etc. These are computed in Driver.prepare_benchmark() and propagated through the PreparationComplete message → BenchmarkCoordinator.on_preparation_complete() → Race object.

Note: the meta key for the cluster name was renamed from cluster_name to target_id as suggested.

gareth-ellis · 2026-04-09T06:46:13Z

@copilot theres some lint issues - can you check and fix?

Agent-Logs-Url: https://github.com/elastic/rally/sessions/1a1a49d5-3b57-4cc8-ae10-ff1b263664cb Co-authored-by: gareth-ellis <14981026+gareth-ellis@users.noreply.github.com>

Copilot · 2026-04-09T06:51:20Z

@copilot theres some lint issues - can you check and fix?

Fixed in commit 12ba93a — pylint line-too-long in racecontrol.py and black formatting in telemetry_test.py.

gareth-ellis

LGTM

inqueue

LGTM

Initial plan

332532d

Copilot AI assigned Copilot and gareth-ellis Apr 8, 2026

Copilot started work on behalf of gareth-ellis April 8, 2026 16:15 View session

Copilot AI linked an issue Apr 8, 2026 that may be closed by this pull request

Fetch and store meta info for target cluster #2094

Closed

Copilot AI and others added 2 commits April 8, 2026 16:25

fix: address code review feedback - fix mock.ANY filter and add basic…

e349173

…_auth tuple test Agent-Logs-Url: https://github.com/elastic/rally/sessions/3c4bd014-8eba-47d8-9ef7-7fdf59a882c4 Co-authored-by: gareth-ellis <14981026+gareth-ellis@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fetch and store meta info for target cluster~~ Fetch and store target cluster meta info at benchmark start Apr 8, 2026

Copilot AI requested a review from gareth-ellis April 8, 2026 16:29

Copilot finished work on behalf of gareth-ellis April 8, 2026 16:29

gareth-ellis requested a review from inqueue April 8, 2026 16:46

gareth-ellis requested changes Apr 8, 2026

View reviewed changes

Copilot started work on behalf of gareth-ellis April 8, 2026 16:57 View session

Copilot AI and others added 2 commits April 8, 2026 17:10

Copilot AI requested a review from gareth-ellis April 8, 2026 17:14

Copilot finished work on behalf of gareth-ellis April 8, 2026 17:14

Copilot started work on behalf of gareth-ellis April 9, 2026 06:46 View session

fix: resolve pylint line-too-long and black formatting issues

12ba93a

Agent-Logs-Url: https://github.com/elastic/rally/sessions/1a1a49d5-3b57-4cc8-ae10-ff1b263664cb Co-authored-by: gareth-ellis <14981026+gareth-ellis@users.noreply.github.com>

Copilot finished work on behalf of gareth-ellis April 9, 2026 06:53

gareth-ellis marked this pull request as ready for review April 10, 2026 06:15

fressi-elastic approved these changes Apr 10, 2026

View reviewed changes

gareth-ellis approved these changes Apr 13, 2026

View reviewed changes

inqueue approved these changes Apr 16, 2026

View reviewed changes

inqueue merged commit 3c72da3 into master Apr 16, 2026
26 checks passed

inqueue deleted the copilot/fetch-store-meta-info branch April 16, 2026 18:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fetch and store target cluster meta info at benchmark start#2095

Fetch and store target cluster meta info at benchmark start#2095
inqueue merged 6 commits into
masterfrom
copilot/fetch-store-meta-info

Copilot AI commented Apr 8, 2026 •

edited

Loading

Uh oh!

gareth-ellis left a comment

Uh oh!

Copilot AI commented Apr 8, 2026

Uh oh!

gareth-ellis commented Apr 9, 2026

Uh oh!

Copilot AI commented Apr 9, 2026

Uh oh!

gareth-ellis left a comment

Uh oh!

inqueue left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Copilot AI commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

esrally/telemetry.py — ClusterEnvironmentInfo

esrally/driver/driver.py

esrally/racecontrol.py

esrally/metrics.py — Race

tests/telemetry_test.py

Uh oh!

gareth-ellis left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Apr 8, 2026

Uh oh!

gareth-ellis commented Apr 9, 2026

Uh oh!

Copilot AI commented Apr 9, 2026

Uh oh!

gareth-ellis left a comment

Choose a reason for hiding this comment

Uh oh!

inqueue left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Apr 8, 2026 •

edited

Loading

`esrally/telemetry.py` — `ClusterEnvironmentInfo`

`esrally/driver/driver.py`

`esrally/racecontrol.py`

`esrally/metrics.py` — `Race`

`tests/telemetry_test.py`