JWT authentication for gRPC transport

[finnegancarroll]

Description

These changes provide JWT authentication/authorization over the gRPC transport. All options for configuring JWT/User/Roles on a cluster remain the same with the gRPC transport sharing the auth domains of the REST API. As a result, users provide the same JWT HTTP headers over gRPC as they would on the REST API, with tokens being validated against the same authentication backend. This initial PR adds HTTPJwtAuthenticator support only over gRPC, but we expect to expand auth domain support to be in parity with the REST API.

A couple features of REST authentication are excluded in this initial version:

1. Authentication of the superuser (which is only allowed with client/cert auth) is not supported over gRPC. Configuration changes requiring superuser privileges should use the REST API.
2. Anonymous auth is not supported over gRPC. Requests with anonymous auth headers are rejected as unauthorized.

Implementation

Authentication

The SecurityRestFilter is the security plugin component responsible for providing authentication on the REST layer. At a high level it provides a RestHandler which intercepts an incoming RestRequest, translates it to an internal SecurityRequestChannel object which stores relevant security metadata of the request, and invokes BackendRegistry.authenticate() to authenticate and extract the user from the request credentials. A successfully authenticated user is stashed in the thread context for later authorization.

This implementation mimics the above flow with gRPC. Implementing SecurityGrpcFilter, GrpcRequestChannel which utilize the same BackendRegistry.authenticate() to authenticate requests (with some authentication steps rejected over gRPC). Successfully authenticated users are similarly stashed in the thread context.

Authorization

OpenSearch evaluates authorization on the node-to-node transport layer, evaluating the user stashed in the thread context and their associated roles against the permissions associated with the transport action. As gRPC makes no changes to the transport layer behavior authorization remains unchanged for this transport.

Issues Resolved

#5812

Testing

Integration test suite and manual local cluster testing.

Script to build OpenSearch from Security plugin feature branch:

#!/bin/bash

echo "=== Start OpenSearch Security Plugin w/ Security Enabled ==="

SCRIPT_DIR="<path-with-both-repos-cloned>"

# Build security plugin changes
cd ${SCRIPT_DIR}/security
./gradlew assemble

# Build local OpenSearch feature branch
echo "Building local OpenSearch feature branch..."
cd ${SCRIPT_DIR}/OpenSearch
./gradlew :distribution:archives:darwin-tar:assemble  

OS_VERSION="3.5.0-SNAPSHOT"
OS_SNAPSHOT_VERSION="3.5.0.0-SNAPSHOT"
OS_DIR="../opensearch-${OS_VERSION}"

if [ ! -d "$OS_DIR" ]; then
    echo "Extracting local OpenSearch build..."
    tar -xzf distribution/archives/darwin-tar/build/distributions/opensearch-min-${OS_VERSION}-darwin-x64.tar.gz -C ..
    # Rename the extracted directory to match expected name
    mv ../opensearch-min-${OS_VERSION} ../opensearch-${OS_VERSION}
fi

cd ${SCRIPT_DIR}security

# Install security plugin
echo "Installing security plugin..."
cp build/distributions/opensearch-security-${OS_SNAPSHOT_VERSION}.zip ${OS_DIR}/
cd ${OS_DIR}
bin/opensearch-plugin remove opensearch-security || true
bin/opensearch-plugin install file://$(pwd)/opensearch-security-${OS_SNAPSHOT_VERSION}.zip --batch

Case 1 - REST and gRPC with basic auth admin

export OPENSEARCH_INITIAL_ADMIN_PASSWORD="MyTestPassword123"
plugins/opensearch-security/tools/install_demo_configuration.sh -y -i -s

cat > config/opensearch.yml << EOF
aux.transport.types: [transport-grpc]
EOF

With no JWT configured at all gRPC transport will fail to authenticate

grpcurl -plaintext localhost:9400 list
Failed to list services: rpc error: code = Unauthenticated desc = Authentication finally failed

Case 2 - REST and gRPC with security disabled

Minimal security plugin disabled settings with gRPC enabled

plugins.security.disabled: true
aux.transport.types: [transport-grpc]

REST and gRPC should be available over plaintext with no auth required

curl http://localhost:9200/_cluster/health
{"cluster_name":"opensearch","status":"green"...

grpcurl -plaintext localhost:9400 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
org.opensearch.protobufs.services.DocumentService
org.opensearch.protobufs.services.SearchService

Case 3 - REST and gRPC with TLS in WITHOUT SSL only mode

Default configuration as well as TLS configured for gRPC

export OPENSEARCH_INITIAL_ADMIN_PASSWORD="MyTestPassword123"
plugins/opensearch-security/tools/install_demo_configuration.sh -y -i -s

with gRPC TLS settings

aux.transport.types: [secure-transport-grpc]
plugins.security.ssl.aux.secure-transport-grpc.pemkey_filepath: esnode-key.pem
plugins.security.ssl.aux.secure-transport-grpc.pemcert_filepath: esnode.pem
plugins.security.ssl.aux.secure-transport-grpc.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.aux.secure-transport-grpc.enabled: true

Expect failed authentication

grpcurl -insecure localhost:9400 list
Failed to list services: rpc error: code = Unauthenticated desc = Authentication finally failed

curl -k https://localhost:9200/_cluster/health
Unauthorized%

Case 4 - REST and gRPC with TLS in WITH SSL only mode

Configure only TLS for REST and gRPC

plugins.security.ssl_only: true

aux.transport.types: [secure-transport-grpc]
plugins.security.ssl.aux.secure-transport-grpc.pemkey_filepath: esnode-key.pem
plugins.security.ssl.aux.secure-transport-grpc.pemcert_filepath: esnode.pem
plugins.security.ssl.aux.secure-transport-grpc.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.aux.secure-transport-grpc.enabled: true

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
transport.ssl.enforce_hostname_verification: false

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem

TLS requests should not need auth due to ssl only mode

grpcurl -insecure localhost:9400 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
org.opensearch.protobufs.services.DocumentService
org.opensearch.protobufs.services.SearchService

curl -k https://localhost:9200/_cluster/health
{"cluster_name":"opensearch","status":"green"

[codecov]

Codecov Report

❌ Patch coverage is 80.86420% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.77%. Comparing base (43d4850) to head (14e1b2d).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...opensearch/security/filter/SecurityGrpcFilter.java	65.30%	12 Missing and 5 partials ⚠️
.../org/opensearch/security/auth/BackendRegistry.java	82.00%	2 Missing and 7 partials ⚠️
...opensearch/security/filter/GrpcRequestChannel.java	90.90%	2 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5916      +/-   ##
==========================================
+ Coverage   73.70%   73.77%   +0.06%     
==========================================
  Files         437      439       +2     
  Lines       26749    26883     +134     
  Branches     3961     3979      +18     
==========================================
+ Hits        19716    19832     +116     
- Misses       5150     5160      +10     
- Partials     1883     1891       +8     

Files with missing lines	Coverage Δ
.../opensearch/security/OpenSearchSecurityPlugin.java	85.07% <100.00%> (+0.09%)	⬆️
...ava/org/opensearch/security/auditlog/AuditLog.java	100.00% <100.00%> (ø)
...pensearch/security/auditlog/impl/AuditLogImpl.java	90.36% <ø> (ø)
...search/security/filter/SecurityRequestFactory.java	80.00% <100.00%> (+5.00%)	⬆️
...opensearch/security/filter/SecurityRestFilter.java	88.19% <ø> (ø)
...opensearch/security/filter/GrpcRequestChannel.java	90.90% <90.90%> (ø)
.../org/opensearch/security/auth/BackendRegistry.java	80.00% <82.00%> (+2.51%)	⬆️
...opensearch/security/filter/SecurityGrpcFilter.java	65.30% <65.30%> (ø)

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[nibix]

Thank you for this. I have added a few comments.

I have one high level question regarding the configuration:

Is there any way I can configure JWT auth to be supported for gRPC but not for HTTP - or vice versa? It seems to be there is not such way.

Back when we supported authc for the internal transport protocol, it was possible to selectively enable certain methods for transport and for http (compare the http_enabled and transport_enabled flags in config.yml)

[finnegancarroll]

Thanks for the feedback @nibix !

Is there any way I can configure JWT auth to be supported for gRPC but not for HTTP - or vice versa? It seems to be there is not such way.

There is not currently. Unless security plugin is in ssl only mode if JWT is configured it will be utilized for gRPC requests. As I understand it this is mostly* the case for REST as well? There is this one setting which lets users disable auth entirely for REST under plugins.security.unsupported.disable_rest_auth_initially. I'm unsure what the use case is for this, "unsupported" seems to imply it's for internal use only and it doesn't appear to be documented for users so i'm unsure if we need a similar setting for gRPC?

[nibix]

There is not currently. Unless security plugin is in ssl only mode if JWT is configured it will be utilized for gRPC requests. As I understand it this is mostly* the case for REST as well?

In OpenSearch 1.0, you could do:

config:
  dynamic:    
    authc:   
      jwt_auth_domain:
        http_enabled: false
        transport_enabled: true
...

This would only enable auth for transport requests, but not for HTTP requests. Maybe it would make sense to revive this concept?

[finnegancarroll]

In OpenSearch 1.0, you could do:

config:
  dynamic:    
    authc:   
      jwt_auth_domain:
        http_enabled: false
        transport_enabled: true
...

I see. I didn't realize you could selectively disable auth like this.

Looking at this setting it seems to be a member of authc domain configuration object which is loaded by the dynamic config. The current design doesn't touch this part of the code since auth domains are not built and configured per transport, but created once and shared between gRPC and REST. The setting http_enabled: false will then prevent the auth domain from being created in the first place and applies to REST and gRPC transports.

This means my earlier comment here is false:

There is not currently. Unless security plugin is in ssl only mode if JWT is configured it will be utilized for gRPC requests. As I understand it this is mostly* the case for REST as well?

If we want to allow users to disable auth on all client/server transports it looks like this should already be covered since http_enabled: false applies to both gRPC and REST transports in the same way.

To support something like grpc_enabled: false I think would require a refactor in this part of the code. Auth domains would need to be created always and have a new property which indicates which transports they are enabled on and this information could be checked during authc decisions in the backend registry.

[DarshitChanpura]

Thank you @finnegancarroll for adding this new authn mechanism. I didn't take a deeper look but at a glance they look good.

Left some clarification questions.

[finnegancarroll]

Thanks @DarshitChanpura, updated with your suggestions.

[finnegancarroll]

Testing I notice an additional change required for this PR. The gRPC interceptor does not respect plugins.security.disabled: true The fix is to read settings and return a no-op interceptor if this setting is true.

Waiting on opensearch-project/OpenSearch#20493 to publish a new snapshot before adding the change to this branch.

[RyanL1997]

Thanks for the change @finnegancarroll.

[DarshitChanpura]

Changes look good to me. Can you add a happy path for manual test cases you added in the PR description?

[cwperks]

Let's keep this header in existing files. Please follow-up to add this back in.

[cwperks]

Do we need to actually check if its a gRPC request here? Would the value for sslPrincipal always be null for gRPC requests? If so the existing logic may work as intended.

[cwperks]

I think we can extract this to a separate public class and using dependency injection (@Inject) to avoid having to modify the GuiceHolder. Can you please follow up this PR to see if dep injection can be used here?

dep injection doesn't work with private inner classes, private methods or private fields. Our fork of Guice in the core only works w/ public.

[cwperks]

Approving this PR as my comments are mostly organizational. Please follow-up and address in subsequent PRs.

[cwperks]

This PR raises the existing bar for security for gRPC which is only encryption-in-transit (EIT). Its nice to see this rolled out with a first phase targeting JWT auth. Please post any perf gains that you've seen compared to HTTP for comparison.