No automated performance regression testing or benchmarking

[sfloess]

Description

Despite having JMH (Java Microbenchmark Harness) as a test dependency, there are no performance benchmarks implemented or automated in the CI/CD pipeline. This means performance regressions can silently creep into the codebase without detection.

Current State

✅ JMH Dependency Present

<!-- pom.xml lines 414-426 -->
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.37</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.37</version>
    <scope>test</scope>
</dependency>

❌ No Benchmarks Implemented

find src/test/java -name "*Benchmark*.java" -o -name "*Perf*.java"
# Returns: (no results)

❌ No Performance Testing in CI/CD

```.github/workflows/main.yml` contains:

• Unit tests ✅
• Coverage checks ✅
• Static analysis ✅
• Mutation testing (PIT) configured ✅
• Performance benchmarks ❌

Why Performance Testing Matters

1. This is a Foundation Library

jcommons is a dependency for Solenopsis SOAP framework:

• Used in hot paths (SOAP API calls)
• String utilities called frequently
• Performance issues multiply across all consumers
• One slow utility affects entire framework

2. Critical Performance Paths

StringUtil.concat() and variants:

• Used for building strings in loops
• Could cause O(n²) issues with large datasets
• StringBuilder reuse impacts allocation rate

SoapUtil header operations:

• Called on every SOAP request
• Performance directly impacts API throughput

Serialization methods (deprecated but still in use):

• Object serialization/deserialization
• Compression operations
• Critical for applications still using these

3. No Performance SLAs

Without benchmarks:

• Don't know if changes cause regressions
• Can't track performance over time
• No baseline to measure against
• Can't make informed optimization decisions

4. Subtle Regressions Go Undetected

Examples of silent performance issues:

• Accidentally adding synchronization
• Changing from ArrayList to LinkedList
• Adding extra String concatenations
• Inefficient regex patterns
• Excessive object allocation

Impact

• Severity: Medium
• Type: Quality Assurance / Performance
• No performance regression detection
• Unknown performance characteristics
• Can't prove performance improvements
• No data-driven optimization

Recommended Fix

1. Create Performance Benchmarks

Create src/test/java/org/flossware/jcommons/benchmarks/:

StringUtilBenchmark.java:

package org.flossware.jcommons.benchmarks;

import org.flossware.jcommons.util.StringUtil;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Thread)
@Fork(value = 1, warmups = 1)
@Warmup(iterations = 3, time = 1)
@Measurement(iterations = 5, time = 1)
public class StringUtilBenchmark {

    private String[] smallArray;
    private String[] largeArray;
    
    @Setup
    public void setup() {
        smallArray = new String[]{"a", "b", "c"};
        largeArray = new String[100];
        for (int i = 0; i < 100; i++) {
            largeArray[i] = "string" + i;
        }
    }
    
    @Benchmark
    public String concatSmallArray() {
        return StringUtil.concat(smallArray);
    }
    
    @Benchmark
    public String concatLargeArray() {
        return StringUtil.concat(largeArray);
    }
    
    @Benchmark
    public String concatWithSeparator() {
        return StringUtil.concatWithSeparator(", ", smallArray);
    }
    
    @Benchmark
    public String requireNonBlank() {
        return StringUtil.requireNonBlank("test string");
    }
    
    public static void main(String[] args) throws Exception {
        Options opt = new OptionsBuilder()
            .include(StringUtilBenchmark.class.getSimpleName())
            .build();
        new Runner(opt).run();
    }
}

SoapUtilBenchmark.java:

package org.flossware.jcommons.benchmarks;

import jakarta.xml.ws.Service;
import org.flossware.jcommons.util.SoapUtil;
import org.openjdk.jmh.annotations.*;

import javax.xml.namespace.QName;
import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Thread)
@Fork(value = 1, warmups = 1)
@Warmup(iterations = 3)
@Measurement(iterations = 5)
public class SoapUtilBenchmark {

    private QName qname;
    
    @Setup
    public void setup() {
        qname = new QName("http://test.namespace", "TestService");
    }
    
    @Benchmark
    public void getSoapFactory() {
        try {
            SoapUtil.getSoapFactory();
        } catch (Exception e) {
            // Expected in benchmark environment
        }
    }
    
    // Add more SOAP-related benchmarks
}

2. Add Performance Profile to pom.xml

<profiles>
    <profile>
        <id>benchmark</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <version>3.1.0</version>
                    <executions>
                        <execution>
                            <id>run-benchmarks</id>
                            <phase>test</phase>
                            <goals>
                                <goal>exec</goal>
                            </goals>
                            <configuration>
                                <executable>java</executable>
                                <arguments>
                                    <argument>-classpath</argument>
                                    <classpath/>
                                    <argument>org.openjdk.jmh.Main</argument>
                                    <argument>.*</argument>
                                </arguments>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

3. Add CI/CD Performance Testing

Add to .github/workflows/performance.yml:

name: Performance Benchmarks

on:
  pull_request:
    branches: [ main ]
  schedule:
    # Run weekly on Sunday at 2am
    - cron: '0 2 * * 0'

jobs:
  benchmark:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup JDK 17
        uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '17'
          cache: 'maven'
      
      - name: Run Benchmarks
        run: mvn clean test -Pbenchmark
      
      - name: Store benchmark results
        uses: benchmark-action/github-action-benchmark@v1
        with:
          tool: 'jmh'
          output-file-path: target/jmh-result.json
          github-token: ${{ secrets.GITHUB_TOKEN }}
          auto-push: true
          # Alert if performance degrades by 200%
          alert-threshold: '200%'
          comment-on-alert: true
          fail-on-alert: false

4. Document Performance Standards

Add to CONTRIBUTING.md:

## Performance Testing

### Running Benchmarks Locally
\`\`\`bash
# Run all benchmarks
mvn clean test -Pbenchmark

# Run specific benchmark
mvn clean test -Pbenchmark -Dbenchmark=StringUtilBenchmark
\`\`\`

### Performance Standards
- String concatenation: < 100ns per operation
- Object validation: < 50ns per call
- SOAP header setting: < 1μs per call

### When to Add Benchmarks
- New utility methods expected to be called frequently
- Changes to hot-path code
- Optimization attempts (prove improvement)
- After performance bug reports

5. Critical Methods to Benchmark

High Priority:

1. StringUtil.concat() - Used in tight loops
2. StringUtil.concatWithSeparator() - String building
3. StringUtil.requireNonBlank() - Validation overhead
4. SoapUtil.setHeader() - Per-request overhead
5. SoapUtil.setUrl() - Configuration cost
6. ArrayUtil.ensureArray() - Validation overhead
7. Objects.requireNonNull() patterns - Validation cost

Medium Priority:
8. FileUtil.newInputStream() - I/O initialization
9. PropertyUtil.fromInputStream() - Config loading
10. LoggerUtil.log() - Logging overhead

Low Priority (but track):
11. Deprecated serialization methods (establish baseline before removal)
12. Reflection utilities (ClassUtil, MethodUtil)

Benefits

1. Catch Regressions Early

Before: StringUtil.concat() - 45ns/op
After:  StringUtil.concat() - 180ns/op  ❌ REGRESSION DETECTED

2. Prove Optimizations Work

Before: requireNonBlank() - 120ns/op
After:  requireNonBlank() - 35ns/op   ✅ 71% IMPROVEMENT

3. Data-Driven Decisions

• "Should we cache the Logger?" → Run benchmark
• "Is StringBuilder faster here?" → Measure it
• "Does this allocation matter?" → Quantify it

4. Performance Documentation

• Benchmark results serve as performance specs
• Users know what to expect
• Can compare alternatives

5. Competitive Analysis

• Compare against Apache Commons equivalents
• Justify library choice with data
• Know trade-offs

Alternative: Manual Performance Testing

If automated benchmarks are too complex initially:

1. Document expected performance in JavaDoc:

/**
 * Concatenates objects to a string.
 * 
 * <p>Performance: O(n) where n is the number of objects.
 * Typical throughput: ~20M ops/sec on modern hardware.
 * 
 * @param objs objects to concatenate
 * @return concatenated string
 */
public static String concat(Object... objs) {

2. Create manual benchmark classes (not automated):

// src/test/java/org/flossware/jcommons/manual/StringUtilPerformance.java
public class StringUtilPerformance {
    public static void main(String[] args) {
        // Manual timing code
    }
}

3. Document in README:

## Performance Characteristics
- String concatenation: 45ns per operation
- Validation methods: < 50ns overhead
- Tested on: JDK 17, Ubuntu 22.04, Intel i7

But automated benchmarks are strongly recommended for production quality.

Related Tools

Consider also:

• JProfiler - For production profiling
• Async-profiler - Low-overhead profiling
• JFR (Java Flight Recorder) - Built-in profiling

[sfloess]

Analysis: Performance Testing - Valid Enhancement

Current State

JMH Dependencies: ✅ Present in pom.xml
Benchmarks: ❌ None implemented
CI/CD: ❌ No performance testing

When Performance Testing Matters

High value for:

1. Utility libraries with hot-path methods
2. String manipulation (StringUtil)
3. I/O operations (FileUtil)
4. Serialization/deserialization

Lower priority when:

• API is stable
• No performance complaints
• Methods are simple wrappers
• No algorithmic complexity

jcommons Context

Methods that would benefit from benchmarks:

• StringUtil.concatWithSeparator() - String building
• StringUtil.toString() - Serialization (deprecated)
• FileUtil.newInputStream() - I/O operations

But:

• Most methods are thin wrappers over JDK
• No complex algorithms
• No performance issues reported
• Used in non-performance-critical paths

Recommendation

Label as enhancement, keep open for future if:

1. Performance issues reported
2. Complex algorithms added
3. Hot-path usage identified
4. Pre-v2.0 optimization phase

Not a priority for current v1.x maintenance.

If implemented:

• Add benchmarks for StringUtil concatenation
• Compare with String.join() (relates to Consider replacing StringUtil concatenation with String.join() or Stream API #68)
• Track trends over releases
• Set up GitHub Pages for results