feat: add --logging-format CLI option for structured logging

[yashhzd]

Description

Adds a new --logging-format CLI option that enables users to programmatically select structured JSON logging formats without requiring custom Log4j2 configuration files.

This addresses the need described in #9626: users who want structured JSON logging for Elasticsearch, Google Cloud Logging, or Graylog currently have to create custom Log4j2 XML configs and inject them via LOG4J_CONFIGURATION_FILE. This is cumbersome in containerized and cloud environments.

Supported Formats

Format	Description
PLAIN	Traditional pattern-based console logging (default, current behavior)
ECS	Elastic Common Schema JSON format
GCP	Google Cloud Platform JSON format
LOGSTASH	Logstash JSON Event Layout V1
GELF	Graylog Extended Log Format

Usage

besu --logging-format=ECS      # Elastic Common Schema
besu --logging-format=GCP      # Google Cloud Platform
besu --logging-format=LOGSTASH  # Logstash JSON Event Layout
besu --logging-format=GELF     # Graylog Extended Log Format
besu --logging-format=PLAIN    # Default (current behavior)

Implementation Details

• Uses Log4j2's built-in JsonTemplateLayout with standard event templates from the log4j-layout-template-json module (ECS, GCP, GELF, Logstash are all shipped as built-in classpath resources) - no custom template files needed
• Added log4j-layout-template-json dependency (part of the Log4j2 BOM already declared in the project)
• Consolidates LoggingLevelOption into LoggingOptions to host both --logging and --logging-format in a single mixin
• Backward compatible: defaults to PLAIN which preserves current behavior
• Respects existing LOG4J_CONFIGURATION_FILE override (custom configs take priority)
• Applies the chosen format after CLI parsing via LogConfigurator.applyLoggingFormat() to avoid the timing issue where Log4j2 initializes before CLI arguments are parsed

Changes

• New files:
  • LoggingFormat.java - Enum with supported formats and their JsonTemplateLayout URIs
  • LoggingOptionsTest.java - Unit tests for the consolidated option and format enum
• Modified files:
  • BesuCommand.java - Registered the mixin, stores selected format, added static getter
  • LoggingLevelOption.java -> LoggingOptions.java - Consolidated level and format options
  • XmlExtensionConfiguration.java - Creates JsonTemplateLayout when a JSON format is selected
  • Log4j2ConfiguratorUtil.java / LogConfigurator.java - Added applyLoggingFormat() method
  • app/build.gradle / util/build.gradle - Added log4j-layout-template-json dependency
  • BesuCommandTest.java - Integration tests for the new CLI option
  • everything_config.toml - Added logging-format entry for completeness test
  • verification-metadata.xml - Updated checksums for new dependency

Fixed Issue(s)

Closes #9626

Testing

• Unit tests for LoggingFormat enum and LoggingOptions (level + format)
• Integration tests in BesuCommandTest for all format values + invalid input
• tomlThatConfiguresEverythingExceptPermissioningToml test updated
• Existing XmlExtensionConfigurationTest continues to pass

[AliZDev-v0]

Hello @yashhzd! Just a heads-up: I’m not a project maintainer, but I'm looking to contribute to Besu by helping with the review process. I have some comments, please feel free to ignore my comments

[AliZDev-v0]

Could you consider merging this 2 options loggingLevelOption and loggingFormatOption to something like LoggerOptions

[yashhzd]

Good idea! I kept them separate initially since they control independent aspects (verbosity vs format), but I can see the benefit of grouping them under a single LoggingOptions mixin for better organization.

I'll consolidate them in the next push.

[AliZDev-v0]

can we accept a path to config from console instead like it was made for engineJwtKeyFile

[yashhzd]

That's a valid use case. For now, users can already point to a fully custom Log4j2 config via the LOG4J_CONFIGURATION_FILE environment variable (or -Dlog4j.configurationFile system property) — XmlExtensionConfiguration checks for this and skips the console appender setup entirely.

Adding a --logging-config-file CLI flag to wrap that behavior would be a nice usability improvement. I'd prefer to keep this PR focused on the preset formats and add the custom config path as a follow-up — would that work for you?

[AliZDev-v0]

Could you please help me confirm whether BesuCommand has not yet been parsed at this point?

I set the configuration format to ECS, but it appears that the value has not been read, as shown below:

However, when I set a breakpoint inside BesuCommand, the value seems to be read correctly:

My understanding is that this might be happening because the logging configuration is initialized before BesuCommand is parsed. In main, setupLogging() is invoked prior to calling besuCommand.parse(...):

public static void main(final String... args) {
    setupLogging();
    final BesuComponent besuComponent = DaggerBesuComponent.create();
    final BesuCommand besuCommand = besuComponent.getBesuCommand();
    int exitCode =
        besuCommand.parse(
            new RunLast(),
            besuCommand.parameterExceptionHandler(),
            besuCommand.executionExceptionHandler(),
            System.in,
            besuComponent,
            args);

    System.exit(exitCode);
  }

[yashhzd]

Great catch, and thanks for confirming with the debugger screenshots — you're absolutely right.

The root cause is exactly as you described: setupLogging() triggers Log4j2 initialization which calls getLoggingFormat() before PicoCLI has parsed the arguments, so selectedLoggingFormat is still null and defaults to PLAIN.

The current code does call LogConfigurator.reconfigure() inside configureLogging() (which runs after parsing), but relying on the full Log4j2 reconfiguration chain is fragile.

I'm pushing a fix that adds a direct applyLoggingFormat() method on XmlExtensionConfiguration — it grabs the current Log4j2 configuration, replaces the Console appender with the correct layout, and calls updateLoggers(). This way we don't depend on the reconfiguration chain working end-to-end.

Will push shortly.

[yashhzd]

Consolidated LoggingLevelOption and LoggingFormatOption into a single LoggingOptions mixin as suggested. Both --logging and --logging-format now live in one class, and BesuCommand registers it with a single addMixin("Logging", loggingOptions) call. Tests are consolidated into LoggingOptionsTest as well.

Summary of the latest push (15da4bf):

• New: LoggingOptions.java — combines level + format options
• Removed: LoggingLevelOption.java, LoggingFormatOption.java
• Updated: BesuCommand.java — uses single LoggingOptions field
• Tests: LoggingOptionsTest.java replaces both individual test classes
• Updated: ConfigDefaultValueProviderStrategyTest — references new class

[AliZDev-v0]

Could you consider to move it to LogConfigurator?

[yashhzd]

Done — moved applyLoggingFormat() to LogConfigurator (delegating to Log4j2ConfiguratorUtil), so all logging config operations go through the same public API.

The method now takes the eventTemplateUri as a parameter rather than reading from a static field, which also let me remove the currentInstance tracking entirely from XmlExtensionConfiguration.

BesuCommand.configureLogging() now calls:

LogConfigurator.applyLoggingFormat(selectedLoggingFormat.getEventTemplateUri());

instead of XmlExtensionConfiguration.applyLoggingFormat().

[yashhzd]

Done — consolidated LoggingLevelOption and LoggingFormatOption into a single LoggingOptions mixin. Both --logging (level) and --logging-format are now grouped together under one class, registered as commandLine.addMixin("Logging", loggingOptions).

[usmansaleem]

I believe GcpLayout is already part of log4j-layout-template-json, it should work without adding it in resources folder.

[yashhzd]

You're right — I missed that GcpLayout.json is already bundled in log4j-layout-template-json. The built-in version is actually better too since it maps log levels to GCP severity names (WARN to WARNING, TRACE to DEBUG, FATAL to EMERGENCY) and includes trace/span ID support from MDC.

Removed the custom file in c22c6ef. The classpath:GcpLayout.json URI in LoggingFormat now resolves to the built-in template directly.

[usmansaleem]

@yashhzd Your latest commit missed DCO sign-off. If you often forget git -s commit ..., you can add prepare commit hook in your local checkout besu repository (besu/.git/hooks/prepare-commit-msg):

#!/bin/sh

NAME=$(git config user.name)
EMAIL=$(git config user.email)

if [ -z "$NAME" ]; then
    echo "empty git config user.name"
    exit 1
fi

if [ -z "$EMAIL" ]; then
    echo "empty git config user.email"
    exit 1

fi

git interpret-trailers --if-exists doNothing --trailer \
    "Signed-off-by: $NAME <$EMAIL>" \
    --in-place "$1"

This should automatically DCO sign all commits in your Besu project. For now, you need to sign your existing commit and force push your changes.

[usmansaleem]

@yashhzd You are on the right track, however, you may have realized that we have two-phase initialization of logging. The correct way to fix this behavior is to initialize logging using execution strategy of PicoCLI BEFORE any of the further parsing happens BUT after PicoCli has initialized itself. Have a look at the PR that I opened against your branch and let me know if changes make sense? yashhzd#1