Migrate to fully programmatic approach for Log4J configuration#1
Closed
usmansaleem wants to merge 5 commits intoyashhzd:feat/logging-format-cli-optionfrom
Closed
Conversation
This refactoring changes from a two-phase ConfigurationFactory approach to an execution strategy pattern. This fixes timing issues where logging was initialized before CLI arguments were available. Key changes: - Add LoggingConfigurator with fully programmatic Log4j2 configuration - Migrate all log4j2.xml content to Java (filters, Splunk, appenders) - Add execution strategy for logging initialization (correct timing) - Add isHelpOrVersionRequested() to walk subcommand tree - Remove two-phase initialization (ConfigurationFactory + applyFormat) - Remove BesuLoggingConfigurationFactory and XmlExtensionConfiguration - Remove log4j2.xml (all config now in Java) - Deprecate configureLogging() (now just announces via logger) - Remove static selectedLoggingFormat field Benefits: - Single-phase initialization at correct time (after CLI parsing) - No static state required - Fully programmatic (type-safe, refactorable) - Simpler codebase (-108 net lines) All existing features preserved: - --logging / -l flag (log level) - --logging-format flag (PLAIN, ECS, GCP, LOGSTASH, GELF) - --color-enabled flag and NO_COLOR env var - LOGGER=Splunk environment variable - LOG4J_CONFIGURATION_FILE custom config override - All 6 logger filters (DNS, transactions, Bonsai, etc.) Related to PR besu-eth#9803 and issue besu-eth#9626 Signed-off-by: Usman Saleem <usman@usmans.info>
This change migrates Besu from XML-based Log4j2 configuration to a fully programmatic approach using execution strategy pattern, similar to Web3Signer. Key changes: - Add LoggingConfigurator with programmatic configuration for Console and Splunk appenders - Implement execution strategy in BesuCommand for proper initialization timing - Remove evmtool's log4j2.xml which was interfering with programmatic config - Add support for Splunk HTTP Event Collector via LOGGER environment variable Splunk configuration via environment variables: - LOGGER=Splunk enables Splunk appender - SPLUNK_URL: Base URL (e.g., https://localhost:8088) - library appends path internally - SPLUNK_TOKEN: HEC token value - SPLUNK_INDEX: Target index (default: main) - SPLUNK_SOURCE: Event source (default: besu) - SPLUNK_SOURCETYPE: Event sourcetype (default: besu) - SPLUNK_MESSAGE_FORMAT: Message format text/json (default: text) - SPLUNK_BATCH_SIZE_BYTES: Batch size in bytes (default: 65536) - SPLUNK_BATCH_SIZE_COUNT: Batch size in events (default: 1000) - SPLUNK_BATCH_INTERVAL: Batch interval in ms (default: 500) - SPLUNK_SKIPTLSVERIFY: Skip TLS verification (default: false) All log4j2.xml filters migrated to programmatic configuration: - DNS query filters for DNSTimerTask and DNSResolver - Invalid transaction removal marker filter - OpenTelemetry B3 propagation filter - Bonsai worldstate stack trace filter Custom log4j2.xml override still supported via -Dlog4j2.configurationFile Tested with: - Console logging with PLAIN and ECS formats - Custom log4j2.xml override - Splunk HEC integration with real Splunk Enterprise instance Signed-off-by: Usman Saleem <usman@usmans.info>
Changes: - Initialize logging BEFORE plugin registration (was backwards) - Plugins can now log properly during registration - Plugin logs respect --logging-format and LOGGER env var - Fixes issue where plugins used auto-discovered Log4j2 config - Remove suppressInfoLog() hack (no longer needed with correct order) - Previous workaround to prevent plugin logs before help/version - Now unnecessary since logging is initialized after help check - Remove "Starting Besu version" log from initializeLogging() - Prevents log output appearing before --Xhelp (hidden options) - Version still logged elsewhere during normal startup - Keeps help/version output clean Execution order is now: 1. Parse CLI 2. Initialize logging (skip if help/version requested) 3. Register plugins (skip if help/version requested) 4. Execute command Signed-off-by: Usman Saleem <usman@usmans.info>
Issue: Logger filters were hardcoded to Level.INFO, causing them to always log at INFO regardless of --logging setting (e.g., --logging=WARN still showed INFO messages from TransactionPoolFactory). Root cause: During migration from log4j2.xml to programmatic configuration, incorrectly added explicit Level.INFO to logger definitions. Original XML had no level attribute, allowing loggers to inherit from root. Fix: Remove hardcoded Level.INFO from all logger filter configurations: - org.apache.tuweni.discovery.DNSTimerTask - org.apache.tuweni.discovery.DNSResolver - io.vertx.core.dns.DnsException - org.hyperledger.besu.ethereum.eth.transactions - io.opentelemetry.extension.trace.propagation.B3PropagatorExtractorMultipleHeaders - org.hyperledger.besu.ethereum.trie.pathbased.bonsai.storage.BonsaiSnapshotWorldStateKeyValueStorage Loggers now inherit root level while still filtering specific message patterns. Before: --logging=WARN still showed INFO messages from filtered loggers After: --logging=WARN correctly shows only WARN+ messages Signed-off-by: Usman Saleem <usman@usmans.info>
- Fix parseLevel() to use Level.toLevel() with default instead of getLevel() that returns null - Add isSplunkConfigValid() to validate required Splunk environment variables (URL and TOKEN) - Add fallback to console logging with warning when Splunk config is invalid - Track actual appender used vs requested for root logger reference - Make SPLUNK_INDEX optional (only add attribute if set, uses Splunk default otherwise) Signed-off-by: Usman Saleem <usman@usmans.info>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Building on PR besu-eth#9803's
--logging-formatoption, this PR completes the migration by:Changes
1. Programmatic Configuration (
LoggingConfigurator)ConfigurationBuilderAPI2. Execution Strategy Pattern (
BesuCommand)selectedLoggingFormatfieldcreateLoggingInitializationTask()execution strategy3. Cleanup
BesuLoggingConfigurationFactory(two-phase initialization no longer needed)XmlExtensionConfigurationand related testslog4j2.xml(was interfering with programmatic config)applyLoggingFormat()methods from util classesConfiguration
Console Logging (default)