[BUG] A `Remove(MyKey)` during load will keep serving stale data

[zlepper]

Describe the bug

I'm currently attempting to switch our custom cache implementation over to using fusion cache, and i'm hitting an issue where calling Remove with the key of an in-flight request causes the request issued before the Remove to still populate the cache.

Technically this is not strictly wrong, but I figure since we had a unit test for it, that we have had issues with stale values ending up in the cache because of this.

This does actually behave correctly when removing entries by tag. However that would probably cause the tag storage to grow unbounded :/

To Reproduce

Some very convenient NUnit tests to illustrate the issue:

    [Test]
    public async Task FusionCache_RemoveDuringInflightFactory_ConcurrentReaderGetsFreshValue()
    {
        using var cache = new FusionCache(Options.Create(new FusionCacheOptions()));

        var opts = TaskCreationOptions.RunContinuationsAsynchronously;
        var firstFactoryStarted = new TaskCompletionSource(opts);
        var releaseFirstFactory = new TaskCompletionSource(opts);
        var secondFactoryRan = false;

        var first = cache
            .GetOrSetAsync<int>(
                "key",
                async (ctx, ct) =>
                {
                    firstFactoryStarted.SetResult();
                    await releaseFirstFactory.Task;
                    return 1;
                }
            )
            .AsTask();

        await firstFactoryStarted.Task.WaitAsync(TimeSpan.FromSeconds(5));

        await cache.RemoveAsync("key");

        var second = cache
            .GetOrSetAsync<int>(
                "key",
                async (ctx, ct) =>
                {
                    secondFactoryRan = true;
                    await Task.Yield();
                    return 2;
                }
            )
            .AsTask();

        releaseFirstFactory.SetResult();
        var secondResult = await second.WaitAsync(TimeSpan.FromSeconds(5));
        await first.WaitAsync(TimeSpan.FromSeconds(5));

        Assert.That(secondResult, Is.EqualTo(2));
        Assert.That(secondFactoryRan, Is.True);
    }

    [Test]
    public async Task FusionCache_RemoveDuringInflightFactory_DoesNotServeStaleValue()
    {
        using var cache = new FusionCache(Options.Create(new FusionCacheOptions()));

        var opts = TaskCreationOptions.RunContinuationsAsynchronously;
        var factoryStarted = new TaskCompletionSource(opts);
        var releaseFactory = new TaskCompletionSource(opts);
        var secondFactoryRan = false;

        var first = cache
            .GetOrSetAsync<int>(
                "key",
                async (ctx, ct) =>
                {
                    factoryStarted.SetResult();
                    await releaseFactory.Task;
                    return 1;
                }
            )
            .AsTask();

        await factoryStarted.Task.WaitAsync(TimeSpan.FromSeconds(5));

        await cache.RemoveAsync("key");
        releaseFactory.SetResult();
        await first.WaitAsync(TimeSpan.FromSeconds(5));

        var afterFirst = await cache.TryGetAsync<int>("key");
        Assert.That(afterFirst.HasValue, Is.False);

        var second = await cache.GetOrSetAsync<int>(
            "key",
            async (ctx, ct) =>
            {
                secondFactoryRan = true;
                await Task.Yield();
                return 2;
            }
        );

        Assert.That(second, Is.EqualTo(2));
        Assert.That(secondFactoryRan, Is.True);
    }

    [Test]
    public async Task FusionCache_RemoveByTagDuringInflightFactory_InvalidatesStaleValue()
    {
        using var cache = new FusionCache(Options.Create(new FusionCacheOptions()));

        var opts = TaskCreationOptions.RunContinuationsAsynchronously;
        var factoryStarted = new TaskCompletionSource(opts);
        var releaseFactory = new TaskCompletionSource(opts);
        var secondFactoryRan = false;
        string[] tags = ["the-tag"];

        var first = cache
            .GetOrSetAsync<int>(
                "key",
                async (ctx, ct) =>
                {
                    factoryStarted.SetResult();
                    await releaseFactory.Task;
                    return 1;
                },
                tags: tags
            )
            .AsTask();

        await factoryStarted.Task.WaitAsync(TimeSpan.FromSeconds(5));

        await cache.RemoveByTagAsync("the-tag");
        releaseFactory.SetResult();

        Assert.That(await first.WaitAsync(TimeSpan.FromSeconds(5)), Is.EqualTo(1));

        var afterFirst = await cache.TryGetAsync<int>("key");
        Assert.That(afterFirst.HasValue, Is.False);

        var second = await cache.GetOrSetAsync<int>(
            "key",
            async (ctx, ct) =>
            {
                secondFactoryRan = true;
                await Task.Yield();
                return 2;
            },
            tags: tags
        );

        Assert.That(second, Is.EqualTo(2));
        Assert.That(secondFactoryRan, Is.True);
    }

(Whether or not the second callback should run, or if it is fine if the original factory runs again I don't have strong opinions on, it's just simpler to illustrate like this :) )

Do note that 2 of those tests fails, while the one using tags passes.

Expected behavior

I expect any in-flight request issues before a cache clear to be treated as invalid (And at least not stored in the cache, though returning them might still be fine).

Versions

I've encountered this issue on:

• FusionCache version: 2.6.0
• .NET version: 10
• OS version: NixOS 26.05/Linux

[zlepper]

This probably relates to #613, however my case illustrative case i'm purely using MemoryCache, no distributed caching involved.