Skip to content

architecture.md

Latest commit

 

History

History
436 lines (349 loc) · 15.7 KB

architecture.md

File metadata and controls

436 lines (349 loc) · 15.7 KB
Raw

MagicLogger Architecture

Overview

MagicLogger is a high-performance, feature-rich logging library for TypeScript/JavaScript applications. Our goal is to provide styled logs in production environments with minimal performance overhead through efficient I/O patterns and smart caching.

Core Components

1. Logger Classes

Logger (Default)

  • Purpose: Synchronous logging with optional buffering for balanced performance
  • Use Cases: General-purpose logging, development
  • Implementation: Direct transport calls with optional buffering

SyncLogger

  • Purpose: Explicit blocking I/O for guaranteed delivery
  • Use Cases: Audit logging, debugging, crash-resilient logging
  • Trade-offs: Blocks event loop, guarantees write completion

AsyncLogger

  • Purpose: Non-blocking logging with immediate dispatch
  • Implementation:
    • Immediate dispatch to transports (no batching at logger level)
    • Optional worker threads (enable with worker.enabled: true)
    • Style processing in main thread by default (~0.01ms overhead)
    • Direct transport dispatch when workers disabled
  • Use Cases: High-throughput applications, web servers
  • Trade-offs:
    • Main thread: Lower latency, simpler architecture
    • With workers: Better for CPU-intensive styling, adds IPC overhead

2. Transport System

Transport Interface

interface Transport {
  name: string;
  enabled: boolean;
  log(entry: LogEntry): void | Promise<void>;
  flush?(): void | Promise<void>;
  close(): void | Promise<void>;
  shouldLog(entry: LogEntry): boolean;
  logBatch?(entries: LogEntry[]): void | Promise<void>;
  init?(): void | Promise<void>;
}

Built-in Transports

  • ConsoleTransport: Outputs to stdout/stderr with colors
  • FileTransport: High-performance async file writes using sonic-boom (default, alias for AsyncFileTransport)
  • AsyncFileTransport: Explicit async file writes using sonic-boom
  • SyncFileTransport: Synchronous file writes with intelligent batching
  • WorkerFileTransport: Worker thread-based file transport for CPU-intensive processing
  • HTTPTransport: Sends logs to HTTP endpoints with batching
  • NullTransport: Discards logs (for benchmarking)

Transport Manager

  • Coordinates multiple transports
  • Handles batching and routing
  • Implements backpressure handling
  • Provides metrics and health monitoring

3. Dispatch Architecture

Immediate Dispatch Strategy

MagicLogger uses an immediate dispatch architecture for optimal performance:

  1. Logger Level (AsyncLogger):

    • Immediate dispatch to transports
    • No batching at logger level
    • Minimal overhead with timestamp caching

  2. Transport Level:

    • Each transport implements its own batching based on I/O needs
    • FileTransport: Uses sonic-boom's internal 4KB buffer
    • HTTPTransport: Batches 100 entries or 5s timeout for network efficiency
    • ConsoleTransport: No batching (immediate output)

This architecture ensures minimal latency while allowing transports to optimize their I/O patterns.

3. Async I/O Architecture

Default: FileTransport with sonic-boom

Main Thread                    Async I/O
    │                               │
    ├─ Logger                     ├─ sonic-boom
    │   ├─ Format Entry           │   ├─ Internal Buffer
    │   └─ Call Transport         │   ├─ Auto-flush at minLength
    │                             │   └─ fs.write() (non-blocking)
    └─ FileTransport              │
        ├─ logSync()              └─ File System
        └─ Direct to sonic-boom       └─ Disk

Default behavior: Logger uses FileTransport (AsyncFileTransport) with sonic-boom for async I/O - provides the best performance for production applications

Optional: WorkerFileTransport for CPU-intensive workloads

Main Thread                    Worker Thread Pool
    │                               │
    ├─ Logger                     ├─ Worker 1
    │   ├─ Format Entry           │   ├─ Process batch
    │   └─ Send to Worker         │   └─ Write to file
    │                             │
    └─ WorkerTransport            ├─ Worker 2 (if poolSize > 1)
        ├─ Batch logs                 ├─ Process batch
        └─ IPC transfer               └─ Write to file

Worker threads are optional: Only use WorkerTransport when you need true parallelism for CPU-intensive processing

Worker Communication Protocol (When using WorkerTransport)

  • INIT: Initialize worker with transport config
  • LOG_BATCH: Send batch of logs to process
  • FLUSH: Force flush buffered logs
  • SHUTDOWN: Graceful worker termination
  • ACK: Acknowledge batch processing
  • METRICS: Performance metrics updates

Note: This protocol only applies when explicitly using WorkerTransport, not the default AsyncFileTransport

4. MAGIC Schema

The MAGIC (Metadata And Graphics In Console) schema enables portable styled logs:

interface LogEntry {
  id: string;                    // Unique identifier
  timestamp: string;              // ISO 8601 timestamp
  timestampMs: number;            // Unix timestamp in ms
  level: LogLevel;                // Log severity
  message: string;                // Plain text message
  styles?: Array<[number, number, string]>; // Style ranges [start, end, style]
  context?: Record<string, any>;  // Structured metadata
  tags?: string[];                // Categorization tags
  loggerId?: string;              // Logger instance ID
  error?: {                       // Error information
    name: string;
    message: string;
    stack?: string;
    code?: string | number;
  };
}

Performance Characteristics

Synchronous Logging

  • Throughput: ~147K ops/sec plain text, ~50K ops/sec with styles
  • Latency: 0.006ms average blocking time
  • Memory: Minimal buffering
  • Reliability: Guaranteed delivery (blocks until written)

Asynchronous Logging

  • Throughput: ~164K ops/sec plain text, ~120K ops/sec with styles
  • Latency: 0.006ms average (non-blocking, 0.004ms P50)
  • Memory: Minimal with sonic-boom buffering
  • Reliability: Best-effort, requires graceful shutdown for guarantee
  • Note: Styled async significantly outperforms sync styled (120K vs 50K ops/sec)

Architecture Benefits

Default Async (sonic-boom)

  • Non-blocking: Main thread never blocks on I/O
  • Efficient buffering: Automatic flush at configurable thresholds
  • Low overhead: No IPC or thread management costs
  • Production-ready: Battle-tested in Pino ecosystem

Optional Worker Threads (WorkerTransport)

  • True parallelism: CPU-intensive operations run in parallel
  • Isolation: Transport failures don't affect main thread
  • Scalability: Pool size adjustable based on workload
  • Use cases: Heavy transformations, encryption, compression

Recommendation: Start with default AsyncFileTransport. Only use WorkerTransport if you have specific CPU-intensive requirements that benefit from parallelism.

Worker Thread Considerations

Based on Node.js best practices:

When to use worker threads:

  • CPU-intensive transformations (encryption, compression)
  • Complex log formatting requiring heavy computation
  • Isolation requirements (untrusted log processing)

When NOT to use worker threads:

  • Simple I/O operations (file writes, network requests)
  • Basic log formatting and styling
  • Low-volume logging scenarios

Performance trade-offs:

  • Worker creation overhead: ~10-50ms per worker
  • IPC overhead: ~0.1-0.5ms per message batch
  • Memory overhead: ~10MB per worker thread
  • Optimal worker count: Number of CPU cores (typically 2-4)

For most logging scenarios, the default async I/O without workers provides the best balance of performance and simplicity.

Real-World Performance Comparison

Logger Architecture Throughput (ops/sec) Avg Latency Use Case
Pino (Plain) Async I/O 226,046 0.004ms High-throughput, minimal overhead
MagicLogger (Async+Styled) Async I/O + Cache 163,350 0.006ms Styled production logging
Winston (Styled) Multi-stream 153,448 0.006ms Feature-rich ecosystem
MagicLogger (Async) Async I/O 127,402 0.007ms Non-blocking production
MagicLogger (Sync) Direct I/O 67,803 0.014ms Guaranteed delivery
MagicLogger (Sync+Styled) Direct I/O + Styles 24,856 0.040ms Interactive CLI tools

Key insights:

  • Async styled (163K) outperforms sync plain (68K) by 2.4x
  • Styling overhead: 63% in sync mode, -28% in async (faster due to better batching)
  • All metrics from real file I/O with production-like payloads

Design Decisions

1. Async I/O Strategy

  • Default: sonic-boom for efficient async file I/O
  • Rationale: Proven performance, minimal overhead, no IPC costs
  • Optional: WorkerTransport for CPU-intensive workloads requiring parallelism

2. Batching Strategy

  • Default: 1000 entries or 10ms timeout
  • Rationale: Balance between syscall reduction and latency
  • Tunable: Via bufferSize and flushInterval options
  • sonic-boom: Internal buffering with automatic flush at minLength

3. Serialization Format

  • Choice: JSON with MAGIC extensions
  • Rationale: Universal compatibility, structured data
  • Trade-off: Larger payload vs. binary formats

4. Style Processing Architecture

Where Styling Happens

Default (Workers OFF - Recommended):

  • Style extraction occurs in the MAIN THREAD
  • Uses TextStyler.parseBracketsWithExtraction() to parse <style>text</> markup
  • Produces { plainText, styles: [[start, end, style], ...] }
  • Result stored in LogEntry for MAGIC schema compliance
  • Performance: ~0.012ms overhead per styled log (acceptable)

With Workers Enabled (Optional):

  • When worker.enabled: true, style extraction moves to WORKER THREAD
  • AsyncLoggerWorker processes styles after receiving batch
  • Beneficial for heavy styling workloads (4x faster for complex styles)
  • Adds IPC overhead for simple logs (~137% slower)

Performance Characteristics (Measured)

  • Main thread styling: ~0.012ms per styled log (679% overhead vs plain)
  • Simple logs: 888K ops/sec without workers, 375K with workers
  • Styled logs: 95K ops/sec without workers, 405K with workers (4x faster)
  • Optimization: Fast-path detection for non-styled text
  • Recommendation: Use workers only for heavy styling workloads

Memory Management

Buffer Limits

  • Batch Buffer: Max 10,000 entries (configurable)
  • Worker Queue: Max 10 concurrent operations per worker
  • Transport Buffer: Transport-specific limits

Backpressure Handling

  1. Monitor worker utilization
  2. Drop logs when over capacity
  3. Emit warning events
  4. Metrics tracking for monitoring

Error Handling

Worker Failures

  • Automatic fallback to setImmediate mode
  • Error events emitted to main thread
  • Graceful degradation

Transport Failures

  • Individual transport isolation
  • Retry logic (transport-specific)
  • Error aggregation and reporting

Security Considerations

Log Sanitization

  • Redaction of sensitive fields
  • PII detection and masking
  • Configurable sanitization rules

Resource Limits

  • Maximum message size
  • Rate limiting support
  • Memory usage caps

Future Enhancements

Planned Features

  1. Compression: zstd/gzip for large batches
  2. Streaming: Server-sent events for real-time logs
  3. Clustering: Multi-process coordination
  4. Tracing: OpenTelemetry integration

Performance Optimizations

Lock-Free Ring Buffer (Experimental)

  • SharedArrayBuffer: Zero-copy data transfer between threads
  • Atomic Operations: Lock-free synchronization using Atomics API
  • 64KB Ring Buffer: Pre-allocated circular buffer, no GC pressure
  • 150K+ ops/sec: Target performance with async logging
  • Enable with: worker: { enabled: true, useRingBuffer: true }
  • Note: Requires worker-thread.js to be deployed (coming soon)

Timestamp Precision

  • SyncLogger: Uses Date.now() for each log (millisecond precision)
  • AsyncLogger: Uses performance.now() for microsecond precision
    • Ensures unique timestamps even in tight loops
    • Prevents timestamp collision in high-throughput scenarios
    • Base timestamp updated every second to prevent drift

Other Optimizations

  1. Memory Pools: Pre-allocated batch arrays reduce GC pressure
  2. Micro-batching: setTimeout(0) batching for optimal throughput
  3. Zero-Copy Buffers: Direct I/O operations where possible
  4. Smart Caching: Property access caching in hot paths

Best Practices

For Maximum Throughput (150K+ ops/sec)

const logger = new AsyncLogger({
  worker: {
    enabled: true,         // Enable workers
    useRingBuffer: true,   // Use lock-free ring buffer
    poolSize: 1,           // Ring buffer uses single worker
    batchSize: 100,        // Optimized batch size
    flushInterval: 10      // Quick flushes
  },
  enableMetrics: true      // Monitor performance
});
// Achieves 150K+ ops/sec with zero-copy ring buffer

For Low Latency (0.003ms avg)

const logger = new AsyncLogger({
  worker: {
    enabled: false,        // No workers = lowest latency (default)
  },
  buffer: {
    size: 10,              // Small batches
    flushInterval: 1       // Immediate flushes
  }
});
// Achieves 0.003ms average latency with 301K ops/sec

For Reliability

const logger = new SyncLogger({
  file: './audit.log',     // Persistent storage
  forceFlush: true         // fsync after each write
});

Monitoring

Metrics Collection

logger.on('metrics', (metrics) => {
  console.log('Logs processed:', metrics.totalLogs);
  console.log('Worker utilization:', metrics.workerUtilization);
  console.log('Dropped logs:', metrics.droppedLogs);
});

Health Checks

const health = {
  workers: logger.workers.length,
  pending: logger.batch.length,
  metrics: logger.getMetrics()
};

Testing

Unit Tests

  • Mock transports for isolation
  • Deterministic worker behavior
  • Error injection

Integration Tests

  • Real worker threads
  • File I/O verification
  • Performance benchmarks

Load Tests

# Generate high load
npm run test:load -- --rate=10000 --duration=60s

# Monitor metrics
npm run test:metrics -- --watch

Performance Characteristics

Logger Type Throughput Latency Memory Best For
Logger ~120K/s <1ms Low General use
SyncLogger ~53K/s Immediate Lowest Guaranteed delivery
AsyncLogger (no workers) ~148K/s <7ms Low High throughput
AsyncLogger (workers) ~100K/s <10ms Moderate CPU-intensive styling

Performance Optimizations

Timestamp Caching

  • Optimization: Cache Date.now() for 10 consecutive logs
  • Impact: Reduces syscall overhead

Fast Path Optimization

  • Optimization: Check for '<' character before style parsing
  • Impact: ~0.01ms saved per plain text log

Object Pre-sizing

  • Optimization: Pre-allocate all properties with undefined
  • Impact: Faster object creation with V8 optimization

Immediate Dispatch

  • Optimization: Direct dispatch to transports without batching
  • Impact: Minimal latency for log delivery

Conclusion

MagicLogger's architecture balances performance, reliability, and developer experience. The immediate dispatch architecture ensures minimal latency, while the MAGIC schema enables rich, portable logging across platforms. Transport-level batching provides optimal I/O performance without sacrificing responsiveness.