Performance: Intelligent cache preloading and multi-level caching

[sfloess]

Overview

Add intelligent cache preloading to reduce perceived latency and optional disk-based L2 cache for large repositories.

Current State

• ✅ Time-based L1 cache (5-minute TTL)
• ✅ Event-based invalidation (on delete)
• ❌ No preloading (cache miss = full latency)
• ❌ No L2 cache (lose everything on restart)

Performance Score Impact

Current: Performance B+ (88/100)
With #53 + #62 + #64 + #71 + this: Performance A+ (98/100)

1. Intelligent Cache Preloading

Problem

Current behavior:

0:00 - User lists repo (cache miss, slow: 2000ms)
5:00 - User lists repo (cache hit, fast: 10ms)
10:00 - Cache expires (TTL elapsed)
10:01 - User lists repo (cache miss, slow: 2000ms again)

Solution: Background Refresh

New behavior:

0:00 - User lists repo (cache miss, slow: 2000ms)
4:30 - Background refresh starts (before expiry)
5:00 - User lists repo (cache hit, fast: 10ms) [new data already ready]
9:30 - Background refresh (proactive)
10:00 - User lists repo (cache hit, fast: 10ms)

Implementation

New Configuration:

# Enable cache preloading
nexus.cache.preload.enabled=true

# Start refresh N seconds before expiry (default: 30s)
nexus.cache.preload.before.expiry.seconds=30

# Refresh interval for frequently accessed repos (default: 4 minutes)
nexus.cache.preload.interval.seconds=240

Code Changes (NexusClient.java):

private final ScheduledExecutorService preloadExecutor = 
    Executors.newScheduledThreadPool(1, r -> {
        Thread t = new Thread(r, "cache-preload");
        t.setDaemon(true);
        return t;
    });

private final Map<String, ScheduledFuture<?>> preloadTasks = new ConcurrentHashMap<>();

private void schedulePreload(String repository) {
    if (!preloadEnabled) return;
    
    // Cancel existing task
    ScheduledFuture<?> existing = preloadTasks.get(repository);
    if (existing != null) {
        existing.cancel(false);
    }
    
    // Schedule refresh before expiry
    long delay = cacheTTL - preloadBeforeExpiry;
    ScheduledFuture<?> task = preloadExecutor.schedule(() -> {
        try {
            logger.debug("Preloading cache for repository: {}", repository);
            listComponents(repository, true);  // Force refresh
        } catch (Exception e) {
            logger.warn("Cache preload failed for {}: {}", repository, e.getMessage());
        }
    }, delay, TimeUnit.SECONDS);
    
    preloadTasks.put(repository, task);
}

public List<RepoRecord> listComponents(String repository, boolean forceRefresh) {
    List<RepoRecord> result = fetchComponents(repository, forceRefresh);
    
    // Schedule preload for next refresh
    schedulePreload(repository);
    
    return result;
}

Benefits:

• Zero perceived latency after first fetch
• Stays "always fresh" for active repos
• No user-facing delays

2. Multi-Level Cache (L1 + L2)

Problem

Current behavior:

• Cache is memory-only (in-process)
• Restart = lose all cached data
• First request after restart = slow

Solution: Add Disk-Based L2 Cache

┌─────────────────────────────────────────┐
│ L1 Cache (Memory)                       │
│ - Fast (1-10ms)                         │
│ - Volatile (lost on restart)            │
│ - TTL: 5 minutes                        │
└─────────────────────────────────────────┘
                ↓ (on eviction)
┌─────────────────────────────────────────┐
│ L2 Cache (Disk)                         │
│ - Slower (10-50ms)                      │
│ - Persistent (survives restart)         │
│ - TTL: 1 hour                           │
└─────────────────────────────────────────┘
                ↓ (on miss)
┌─────────────────────────────────────────┐
│ Nexus Server (HTTP)                     │
│ - Slowest (500-2000ms)                  │
│ - Source of truth                       │
└─────────────────────────────────────────┘

Implementation

Configuration:

# Enable L2 cache
nexus.cache.disk.enabled=true

# L2 cache directory
nexus.cache.disk.directory=~/.flossware/nexus/cache

# L2 cache TTL (default: 1 hour)
nexus.cache.disk.ttl.seconds=3600

# Max L2 cache size (default: 100 MB)
nexus.cache.disk.max.size.mb=100

Code (CacheManager.java - new class):

public class CacheManager {
    private final ConcurrentMap<String, CacheEntry<List<RepoRecord>>> l1Cache;  // Memory
    private final Path l2CacheDir;  // Disk
    private final long l1TTL;
    private final long l2TTL;
    private final ObjectMapper mapper = new ObjectMapper();
    
    public List<RepoRecord> get(String repository) {
        // Try L1 (memory)
        CacheEntry<List<RepoRecord>> l1Entry = l1Cache.get(repository);
        if (l1Entry != null && !isExpired(l1Entry, l1TTL)) {
            logger.debug("L1 cache hit: {}", repository);
            return new ArrayList<>(l1Entry.data());
        }
        
        // Try L2 (disk)
        List<RepoRecord> l2Data = readFromDisk(repository);
        if (l2Data != null) {
            logger.debug("L2 cache hit: {}", repository);
            // Promote to L1
            l1Cache.put(repository, new CacheEntry<>(l2Data, Instant.now()));
            return l2Data;
        }
        
        logger.debug("Cache miss: {}", repository);
        return null;
    }
    
    public void put(String repository, List<RepoRecord> data) {
        Instant now = Instant.now();
        
        // Store in L1 (memory)
        l1Cache.put(repository, new CacheEntry<>(new ArrayList<>(data), now));
        
        // Store in L2 (disk) asynchronously
        if (l2Enabled) {
            CompletableFuture.runAsync(() -> writeToDisk(repository, data, now));
        }
    }
    
    private List<RepoRecord> readFromDisk(String repository) {
        try {
            Path file = l2CacheDir.resolve(repository + ".json");
            if (!Files.exists(file)) return null;
            
            // Check L2 TTL
            FileTime modified = Files.getLastModifiedTime(file);
            if (Duration.between(modified.toInstant(), Instant.now()).getSeconds() > l2TTL) {
                Files.delete(file);  // Expired
                return null;
            }
            
            // Deserialize
            return mapper.readValue(file.toFile(), 
                new TypeReference<List<RepoRecord>>() {});
        } catch (IOException e) {
            logger.warn("L2 cache read failed: {}", e.getMessage());
            return null;
        }
    }
    
    private void writeToDisk(String repository, List<RepoRecord> data, Instant timestamp) {
        try {
            Files.createDirectories(l2CacheDir);
            Path file = l2CacheDir.resolve(repository + ".json");
            
            // Serialize
            mapper.writeValue(file.toFile(), data);
            
            // Enforce max cache size (LRU eviction)
            enforceMaxCacheSize();
        } catch (IOException e) {
            logger.warn("L2 cache write failed: {}", e.getMessage());
        }
    }
    
    private void enforceMaxCacheSize() throws IOException {
        long totalSize = Files.walk(l2CacheDir)
            .filter(Files::isRegularFile)
            .mapToLong(p -> {
                try { return Files.size(p); }
                catch (IOException e) { return 0; }
            })
            .sum();
        
        long maxSize = maxCacheSizeMB * 1024 * 1024;
        
        if (totalSize > maxSize) {
            // Delete oldest files until under limit
            Files.walk(l2CacheDir)
                .filter(Files::isRegularFile)
                .sorted(Comparator.comparing(p -> {
                    try { return Files.getLastModifiedTime(p); }
                    catch (IOException e) { return FileTime.fromMillis(0); }
                }))
                .limit((long) (totalSize - maxSize) / 10000)  // Rough estimate
                .forEach(p -> {
                    try { Files.delete(p); }
                    catch (IOException ignored) {}
                });
        }
    }
}

Benefits:

• Faster startup (warm cache from disk)
• Reduced server load (L2 hits avoid HTTP)
• Better offline experience (L2 survives disconnects)

3. Cache Statistics

Add metrics to monitor cache effectiveness:

public class CacheStats {
    private final AtomicLong l1Hits = new AtomicLong();
    private final AtomicLong l2Hits = new AtomicLong();
    private final AtomicLong misses = new AtomicLong();
    
    public String summary() {
        long total = l1Hits.get() + l2Hits.get() + misses.get();
        if (total == 0) return "No cache activity";
        
        return String.format(
            "Cache Stats: L1 Hit Rate: %.1f%%, L2 Hit Rate: %.1f%%, Miss Rate: %.1f%%",
            l1Hits.get() * 100.0 / total,
            l2Hits.get() * 100.0 / total,
            misses.get() * 100.0 / total
        );
    }
}

Expose via CLI:

jnexus cache stats
# Output: Cache Stats: L1 Hit Rate: 85.2%, L2 Hit Rate: 12.3%, Miss Rate: 2.5%

Acceptance Criteria

• Preloading configurable (default: disabled for backwards compat)
• Background refresh scheduled before expiry
• L2 cache configurable (default: disabled)
• L2 cache persists to disk (JSON format)
• L2 cache LRU eviction enforces max size
• Cache statistics tracking
• CLI command to view cache stats
• Documentation updated (PERFORMANCE.md)
• Tests for preload logic
• Tests for L2 cache persistence

Priority

Low - Nice optimization but not critical

Related

• Performance: Implement connection pooling and HTTP optimization #64: HTTP pooling (different optimization)
• Performance: Establish performance benchmarks and SLAs #71: Performance benchmarks (would measure this)