Implement fuzzy refresh middleware for caching #31

Copilot · 2026-01-14T06:05:14Z

Original prompt

通过以下需求描述，实现 Caching 中间件 Fuzzy refresh

在大规模缓存系统中，严格基于 TTL / Expires 的刷新策略会导致两个典型问题：
1.缓存雪崩（Cache Stampede）
1.1大量对象在同一时间点过期
2.同时触发回源请求刷新抖动（Refresh Jitter）

热点内容在边缘节点集群中同步过期

上游瞬时负载飙升

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

sendya · 2026-01-14T06:09:25Z

@copilot 通过以下需求描述，实现 Caching 中间件 Fuzzy refresh

在大规模缓存系统中，严格基于 TTL / Expires 的刷新策略会导致两个典型问题：
1.缓存雪崩（Cache Stampede）
1.1. 大量对象在同一时间点过期
1.2. 同时触发回源请求
2.刷新抖动（Refresh Jitter）
2.1. 热点内容在边缘节点集群中同步过期
2.2. 上游瞬时负载飙升

Fuzzy Refresh = 对缓存有效期引入可控随机性 + 提前刷新概率机制

刷新模型
t0 t_soft t_hard
|----------|-------------|
随机概率触发刷新强制刷新

Soft Expire（模糊刷新区间）
Hard Expire（绝对过期点）

名称	含义
`hard_ttl`	HTTP 语义上的绝对过期时间
`soft_ttl`	开始进入模糊刷新阶段的时间

soft_ttl = hard_ttl * fuzzy_ratio

例如：

Cache-Control: max-age=600
fuzzy_ratio = 0.8

soft_ttl = 480s
hard_ttl = 600s

刷新触发策略（概率驱动）

在 soft_ttl <= age < hard_ttl 区间：

命中缓存
以概率 P 触发后台刷新（Async Revalidate）

Co-authored-by: sendya <[email protected]>

Copilot · 2026-01-14T06:21:28Z

@copilot 通过以下需求描述，实现 Caching 中间件 Fuzzy refresh

在大规模缓存系统中，严格基于 TTL / Expires 的刷新策略会导致两个典型问题： ...

I've successfully implemented the fuzzy refresh middleware for caching. Here's a summary of the implementation:

Key Features Implemented

✅ Soft TTL Calculation: soft_ttl = hard_ttl * fuzzy_refresh_rate
✅ Probabilistic Refresh: Linear probability P = (now - soft_ttl) / (hard_ttl - soft_ttl) in the fuzzy zone
✅ Async Background Revalidation: Non-blocking refresh using goroutines
✅ Thread-safe: Uses math/rand/v2 for concurrent safety
✅ Proper Cache Updates: Handles both 304 Not Modified and content changes

Configuration

The feature is enabled with:

fuzzy_refresh: true
fuzzy_refresh_rate: 0.8  # Valid range: (0, 1.0]

Example Behavior

With max-age=600 and fuzzy_refresh_rate=0.8:

soft_ttl = 480s - Fuzzy refresh zone begins
hard_ttl = 600s - Absolute expiration
In the interval [480s, 600s): Cache serves stale content while triggering async refresh with increasing probability

The implementation prevents cache stampede by distributing refresh requests over time and eliminates refresh jitter across edge nodes.

Commit: 33bfe1e

sendya · 2026-01-14T06:25:00Z

@copilot 不要修改 .gitignore

Co-authored-by: sendya <[email protected]>

Copilot · 2026-01-14T06:26:53Z

@copilot 不要修改 .gitignore

Reverted the .gitignore modification. The /tavern line that was added has been removed.

Commit: 3055c28

Copilot

Pull request overview

This PR implements fuzzy refresh middleware for the caching system to prevent cache stampede and refresh jitter issues in large-scale distributed cache scenarios. The implementation adds probabilistic background cache revalidation that occurs before hard expiration, reducing synchronized cache misses and upstream load spikes.

Changes:

Adds fuzzy refresh logic with configurable soft TTL calculation and linear probability-based refresh triggering
Implements async background revalidation with proper handling of 304 Not Modified and 200 OK responses
Adds comprehensive test coverage for soft TTL calculation, probability distribution, and boundary conditions

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
server/middleware/caching/caching_revalidate.go	Implements core fuzzy refresh logic including soft TTL calculation, probability-based refresh triggers, and async background revalidation
server/middleware/caching/caching_fuzzy_test.go	Adds comprehensive unit tests for fuzzy refresh functionality including edge cases and probability distribution validation

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-14T11:48:27Z

server/middleware/caching/caching_revalidate.go

+	if fuzzyRate <= 0 || fuzzyRate > 1 {
+		fuzzyRate = 0.8 // default to 0.8 if invalid
+	}


Silent fallback to default value masks configuration errors. Consider logging a warning when invalid fuzzy_rate values are detected to help with debugging and configuration validation.

Copilot · 2026-01-14T11:48:27Z

server/middleware/caching/caching_revalidate.go

+	elapsed := float64(now - softTTL)
+	probability := elapsed / totalWindow
+
+	// Random trigger based on probability using math/rand/v2 which is thread-safe


While math/rand/v2 is thread-safe for concurrent use, consider documenting this thread-safety assumption in a comment since shouldTriggerFuzzyRefresh is called from multiple goroutines via the async refresh mechanism.

Suggested change

// Random trigger based on probability using math/rand/v2 which is thread-safe

// Random trigger based on probability.

// Note: This relies on math/rand/v2's documented guarantee of being safe for concurrent use,

// because shouldTriggerFuzzyRefresh may be called from multiple goroutines via async refresh.

Copilot · 2026-01-14T11:48:27Z

server/middleware/caching/caching_revalidate.go

+						time.Unix(hardTTL, 0).Format(time.DateTime))
+
+					// Trigger async revalidation in background
+					go r.asyncRevalidate(c, req)


Unbounded goroutine spawning could lead to resource exhaustion under high load. Consider using a worker pool or rate limiting mechanism to cap concurrent background refresh operations per cache key.

Copilot · 2026-01-14T11:48:27Z

server/middleware/caching/caching_revalidate.go

+// asyncRevalidate performs background revalidation for fuzzy refresh
+func (r *RevalidateProcessor) asyncRevalidate(c *Caching, req *http.Request) {
+	// Create a background context with timeout
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+


The 30-second timeout is hardcoded. Consider making this configurable through CachingOptions to allow tuning based on upstream latency characteristics.

Suggested change

// asyncRevalidate performs background revalidation for fuzzy refresh

func (r *RevalidateProcessor) asyncRevalidate(c *Caching, req *http.Request) {

// Create a background context with timeout

ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)

defer cancel()

const defaultAsyncRevalidateTimeout = 30 * time.Second

// asyncRevalidate performs background revalidation for fuzzy refresh

func (r *RevalidateProcessor) asyncRevalidate(c *Caching, req *http.Request) {

// Create a background context with timeout or inherit existing deadline

baseCtx := context.Background()

var (

ctx context.Context

cancel context.CancelFunc

)

if deadline, ok := req.Context().Deadline(); ok {

// Honor any existing deadline from the incoming request

ctx, cancel = context.WithDeadline(baseCtx, deadline)

} else {

// Fall back to the default async revalidation timeout

ctx, cancel = context.WithTimeout(baseCtx, defaultAsyncRevalidateTimeout)

}

defer cancel()

Initial plan

7208eac

Copilot AI assigned Copilot and sendya Jan 14, 2026

Copilot AI requested a review from sendya January 14, 2026 06:05

Copilot stopped work on behalf of sendya due to an error January 14, 2026 06:05
The session was cancelled by the user.

Copilot started work on behalf of sendya January 14, 2026 06:10 View session

Copilot AI and others added 4 commits January 14, 2026 06:15

Implement fuzzy refresh middleware for caching

ffecf64

Co-authored-by: sendya <[email protected]>

Address code review feedback - use math/rand/v2 and allow fuzzy_rate=1.0

a9a86a6

Co-authored-by: sendya <[email protected]>

Remove accidentally committed binary and update .gitignore

d587068

Co-authored-by: sendya <[email protected]>

Improve async revalidate handling and clarify comments

33bfe1e

Co-authored-by: sendya <[email protected]>

Copilot AI changed the title ~~[WIP] Implement fuzzy refresh middleware for caching~~ Implement fuzzy refresh middleware for caching Jan 14, 2026

Copilot finished work on behalf of sendya January 14, 2026 06:22

Copilot started work on behalf of sendya January 14, 2026 06:25 View session

Revert .gitignore modification per user request

3055c28

Co-authored-by: sendya <[email protected]>

Copilot finished work on behalf of sendya January 14, 2026 06:27

sendya marked this pull request as ready for review January 14, 2026 11:47

Copilot AI review requested due to automatic review settings January 14, 2026 11:47

sendya merged commit eea2ff6 into main Jan 14, 2026
1 check passed

sendya deleted the copilot/add-fuzzy-refresh-middleware branch January 14, 2026 11:48

Copilot AI reviewed Jan 14, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement fuzzy refresh middleware for caching #31

Implement fuzzy refresh middleware for caching #31

Uh oh!

Copilot AI commented Jan 14, 2026 •

edited

Loading

Uh oh!

sendya commented Jan 14, 2026

Uh oh!

Copilot AI commented Jan 14, 2026

Uh oh!

sendya commented Jan 14, 2026

Uh oh!

Copilot AI commented Jan 14, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 14, 2026

Uh oh!

Copilot AI Jan 14, 2026

Uh oh!

Copilot AI Jan 14, 2026

Uh oh!

Copilot AI Jan 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-	// Random trigger based on probability using math/rand/v2 which is thread-safe
+	// Random trigger based on probability.
+	// Note: This relies on math/rand/v2's documented guarantee of being safe for concurrent use,
+	// because shouldTriggerFuzzyRefresh may be called from multiple goroutines via async refresh.

-// asyncRevalidate performs background revalidation for fuzzy refresh
-func (r *RevalidateProcessor) asyncRevalidate(c *Caching, req *http.Request) {
-	// Create a background context with timeout
-	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
-	defer cancel()
+const defaultAsyncRevalidateTimeout = 30 * time.Second
+// asyncRevalidate performs background revalidation for fuzzy refresh
+func (r *RevalidateProcessor) asyncRevalidate(c *Caching, req *http.Request) {
+	// Create a background context with timeout or inherit existing deadline
+	baseCtx := context.Background()
+	var (
+		ctx    context.Context
+		cancel context.CancelFunc
+	)
+	if deadline, ok := req.Context().Deadline(); ok {
+		// Honor any existing deadline from the incoming request
+		ctx, cancel = context.WithDeadline(baseCtx, deadline)
+	} else {
+		// Fall back to the default async revalidation timeout
+		ctx, cancel = context.WithTimeout(baseCtx, defaultAsyncRevalidateTimeout)
+	}
+	defer cancel()

Implement fuzzy refresh middleware for caching #31

Implement fuzzy refresh middleware for caching #31

Uh oh!

Conversation

Copilot AI commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sendya commented Jan 14, 2026

Uh oh!

Copilot AI commented Jan 14, 2026

Key Features Implemented

Configuration

Example Behavior

Uh oh!

sendya commented Jan 14, 2026

Uh oh!

Copilot AI commented Jan 14, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Jan 14, 2026 •

edited

Loading