Skip to content
BranchesTags

Repository files navigation

OpenSearch Metadata Reader

A standalone REST server for reading and serving OpenSearch index metadata from remote store. Built on the opensearch-metadata library (libs/metadata), this tool operates independently of the OpenSearch server module.

Overview

OpenSearch stores index metadata as binary blobs in SMILE format with Lucene checksums. When remote cluster state is enabled, OpenSearch writes a cluster metadata manifest to the remote store that indexes all cluster state — including the location of every index's metadata file.

This tool reads that manifest to discover indices, then fetches and deserializes index metadata — the same way OpenSearch itself resolves metadata from remote store. Users provide a remote store URI (e.g., s3://my-bucket) and the system handles the rest.

Use cases include:

  • Inspecting index metadata from remote store without a running OpenSearch cluster
  • Listing all indices in a cluster's remote state
  • Debugging index settings, mappings, and aliases from S3-backed metadata
  • Building tooling that reads OpenSearch metadata programmatically via HTTP

Project Structure

opensearch-metadata-reader-poc/
├── build.gradle
├── settings.gradle
├── README.md
├── TESTING.md
├── scripts/
│   └── publish-opensearch-libs.sh        # Publish deps to Maven Local
└── src/
    ├── main/java/org/opensearch/metadata/reader/
    │   ├── MetadataReaderApp.java                # Entry point
    │   ├── compress/
    │   │   └── SimpleDeflateCompressor.java      # DEFLATE compression
    │   ├── remote/
    │   │   ├── RemoteStoreExplorer.java          # Remote store path navigation
    │   │   └── manifest/
    │   │       ├── ManifestDiscoveryException.java   # Discovery errors
    │   │       ├── ManifestDiscoveryService.java     # Manifest discovery orchestration
    │   │       └── ManifestIndices.java              # Parsed manifest index entries
    │   ├── service/
    │   │   ├── MetadataReaderException.java      # Reader exceptions
    │   │   └── MetadataReaderService.java        # Core read logic
    │   ├── source/
    │   │   ├── MetadataSource.java               # Storage interface
    │   │   └── S3MetadataSource.java             # S3 backend
    │   └── rest/
    │       ├── MetadataReaderServer.java          # HTTP server
    │       ├── exception/
    │       │   └── IndexNotFoundException.java
    │       ├── format/
    │       │   ├── OutputFormatter.java
    │       │   ├── Table.java
    │       │   ├── TableRenderer.java
    │       │   └── Tabulatable.java
    │       ├── handler/
    │       │   ├── RestHandler.java               # Handler interface
    │       │   ├── HealthHandler.java
    │       │   ├── IndexMetadataHandler.java
    │       │   └── ListIndicesHandler.java
    │       └── response/
    │           ├── ErrorResponse.java
    │           ├── GetIndexMetadataResponse.java
    │           ├── HealthResponse.java
    │           ├── IndexAliasesResponse.java
    │           ├── IndexMappingsResponse.java
    │           ├── IndexSettingsResponse.java
    │           └── ListIndicesResponse.java
    └── test/java/org/opensearch/metadata/reader/
        └── rest/
            ├── RestTestBase.java                  # Random model generators
            └── RestTestBaseTests.java             # Test helper validation

Building

Prerequisites

  • Java 21 or later
  • Gradle (or use the included gradlew wrapper)
  • A local clone of the OpenSearch repository

Step 1: Publish OpenSearch Libraries to Maven Local

The opensearch-metadata library (version 3.5.0-SNAPSHOT) and its transitive dependencies must be published to your local Maven repository:

cd /path/to/OpenSearch
./gradlew :libs:opensearch-metadata:publishToMavenLocal
./gradlew :libs:opensearch-common:publishToMavenLocal
./gradlew :libs:opensearch-core:publishToMavenLocal
./gradlew :libs:opensearch-x-content:publishToMavenLocal
./gradlew :libs:opensearch-compress:publishToMavenLocal

Or use the helper script:

./scripts/publish-opensearch-libs.sh /path/to/OpenSearch

Step 2: Build the Application

./gradlew build fatJar

This produces a standalone fat JAR at build/libs/opensearch-metadata-reader-1.0.0-SNAPSHOT-all.jar.

Running the Server

The application plugin provides a built-in run task that forwards system properties:

# Minimal — remote store URI and region
./gradlew run -Dremote.store=s3://my-bucket -Daws.region=us-west-2

# Custom port
./gradlew run -Dremote.store=s3://my-bucket -Daws.region=us-west-2 -Dserver.port=9090

Or run the fat JAR directly:

java -Dremote.store=s3://my-bucket -Daws.region=us-west-2 \
     -jar build/libs/opensearch-metadata-reader-1.0.0-SNAPSHOT-all.jar

Configuration

Resolved in order: system property → environment variable → default.

Config System Property Environment Variable Default
Remote store URI remote.store REMOTE_STORE (none)
AWS region aws.region AWS_REGION (none)
Server port server.port SERVER_PORT 8080

Both remote.store and aws.region can be overridden per-request via query parameters (remote_store, region).

REST API

Endpoints

Endpoint Description
GET /_cat/indices List all indices from the cluster manifest
GET /{index} Full index metadata as JSON
GET /{index}/_aliases Aliases only
GET /{index}/_mappings Mappings only
GET /{index}/_settings Settings only
GET /_health Service health check

Query Parameters

Parameter Required Description
remote_store No Remote store URI override (e.g., s3://other-bucket)
region No AWS region override for this request
pretty No Pretty-print JSON output
format No Output format: json (default), yaml, table
v No Include headers in table output
h No Select specific columns for table output
s No Sort specification for table output

Examples

# List all indices in the cluster
curl "http://localhost:8080/_cat/indices"

# List indices as a table
curl "http://localhost:8080/_cat/indices?format=table&v"

# Full metadata for an index
curl "http://localhost:8080/my-index"

# Pretty-printed
curl "http://localhost:8080/my-index?pretty"

# Aliases only
curl "http://localhost:8080/my-index/_aliases"

# Mappings only
curl "http://localhost:8080/my-index/_mappings"

# Settings only
curl "http://localhost:8080/my-index/_settings"

# Per-request remote store override
curl "http://localhost:8080/my-index?remote_store=s3://other-bucket&region=eu-west-1"

# Health check
curl "http://localhost:8080/_health"

How It Works

The system discovers index metadata the same way OpenSearch does — through the cluster metadata manifest:

HTTP Request (GET /my-index)
    │
    ▼
IndexMetadataHandler
    │  resolve remote_store + region (query param → server default)
    ▼
ManifestDiscoveryService
    │  1. Navigate cluster-state/{base64ClusterName}/cluster-state/{clusterUUID}/manifest/
    │  2. Find latest committed manifest file
    │  3. Download manifest → ChecksumValidator → decompress → parse SMILE
    │  4. Look up index by name → get uploaded_filename
    ▼
MetadataReaderService
    │  Download + deserialize index metadata blob (existing flow)
    ▼
Response (JSON / YAML / table)

OpenSearch remote store S3 path structure:

s3://bucket/
  └── cluster-state/                       ← base_path (repository config)
      └── {base64(clusterName)}/
          └── cluster-state/               ← CLUSTER_STATE_PATH_TOKEN
              └── {clusterUUID}/
                  ├── manifest/
                  │   └── manifest__<inverted_term>__<inverted_version>__C|P__<inverted_timestamp>__<codec>
                  └── index/
                      └── {indexUUID}/
                          └── metadata__<version>__<timestamp>__<codec>

AWS Credentials

The S3 client uses the AWS SDK v2 default credential chain. In order of precedence:

  1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
  2. Java system properties (aws.accessKeyId, aws.secretAccessKey)
  3. AWS credentials file (~/.aws/credentials)
  4. IAM instance profile (EC2) or container credentials (ECS)

Known Limitations

Limitation Detail
rolloverInfos Skipped — requires NamedXContentRegistry for Condition types
customData Skipped — requires DiffableStringMap parser from server module
Per-request S3Client New client per request; acceptable for diagnostics, not high-throughput
S3 listing pagination First page only (1000 objects); sufficient for typical clusters
Single cluster per bucket Uses first cluster name found under cluster-state/ base path

Dependencies

Dependency Version Purpose
opensearch-metadata 3.5.0-SNAPSHOT Index metadata deserialization + ChecksumValidator
AWS SDK v2 (S3, Auth) 2.21.0 S3 object reads and listing
Javalin 6.7.0 Lightweight HTTP server
Jackson Databind 2.20.1 JSON serialization
SLF4J Simple 2.0.17 Logging (Javalin requirement)
randomizedtesting 2.7.1 Test framework (aligned with OpenSearch)

License

This project is licensed under the Apache License 2.0. See LICENSE for details.

About

POC for reading OpenSearch index metadata from remote storage (S3)

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages