Add a PooledMemoryStream to avoid allocating by adamhathcock · Pull Request #1275 · adamhathcock/sharpcompress

adamhathcock · 2026-04-04T10:11:31Z

This pull request introduces the new PooledMemoryStream class to the codebase and systematically replaces usages of MemoryStream with PooledMemoryStream in performance-critical paths. This change is aimed at reducing memory allocations and improving performance by reusing buffers. Additionally, comprehensive unit tests for PooledMemoryStream have been added, and a minor helper method was introduced in ThrowHelper. There are also minor dependency downgrades in the project lock file.

Memory Stream Refactoring:

Replaced all instances of MemoryStream with PooledMemoryStream in core archive and compression logic, including TAR, SevenZip, LZMA, PPMd, and Squeezed stream handling (TarArchive.cs, TarArchive.Async.cs, SevenZipFilesInfo.cs, ArchiveReader.cs, Lzma2EncoderStream.cs, PpmdStream.cs, SqueezedStream.cs, SqueezedStream.Async.cs, SevenZipWriter.cs). [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]
Added necessary using SharpCompress.IO; directives to files using PooledMemoryStream. [1] [2] [3] [4] [5]

New Features and Testing:

Added a comprehensive test suite for PooledMemoryStream in PooledMemoryStreamTests.cs, testing buffer management, disposal, capacity adjustments, and API behavior.

Utilities:

Added ThrowIfGreaterThan(long value, long other, string? paramName = null) to ThrowHelper for improved argument validation.

Dependency Updates:

Downgraded Microsoft.NET.ILLink.Tasks package versions in packages.lock.json for .NET 8.0 and .NET 10.0 targets. [1] [2]

Copilot

Pull request overview

Introduces PooledMemoryStream (a MemoryStream-compatible implementation backed by ArrayPool<byte>) and refactors performance-critical code paths to use it instead of allocating new MemoryStream buffers, with accompanying unit tests and a small ThrowHelper addition.

Changes:

Added src/SharpCompress/IO/PooledMemoryStream.cs plus a dedicated test suite in PooledMemoryStreamTests.cs.
Replaced various MemoryStream usages with PooledMemoryStream across TAR, 7z, LZMA2, PPMd, and Squeezed stream handling.
Added ThrowHelper.ThrowIfGreaterThan(long, long, ...) and updated packages.lock.json (ILLink tasks versions).

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tests/SharpCompress.Test/Streams/PooledMemoryStreamTests.cs	Adds behavioral tests for pooled buffer growth, disposal/pool returns, buffer exposure, capacity changes, and disposed behavior.
src/SharpCompress/IO/PooledMemoryStream.cs	Implements pooled/segmented + contiguous modes and overrides key `MemoryStream` APIs to reduce allocations.
src/SharpCompress/Writers/SevenZip/SevenZipWriter.cs	Uses `PooledMemoryStream` for temporary header streams during 7z finalization.
src/SharpCompress/Archives/Tar/TarArchive.cs	Uses `PooledMemoryStream` while decoding TAR header name data.
src/SharpCompress/Archives/Tar/TarArchive.Async.cs	Async equivalent TAR header decoding now uses `PooledMemoryStream`.
src/SharpCompress/Common/SevenZip/SevenZipFilesInfo.cs	Uses `PooledMemoryStream` for buffering 7z file property payloads.
src/SharpCompress/Common/SevenZip/ArchiveReader.cs	Replaces an internal `MemoryStream` allocation with `PooledMemoryStream` for 7z reading.
src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs	Uses `PooledMemoryStream` for temporary LZMA encode/decode buffers.
src/SharpCompress/Compressors/PPMd/PpmdStream.cs	Uses `PooledMemoryStream` in PPMd finalization path (EOF encode).
src/SharpCompress/Compressors/Squeezed/SqueezedStream.cs	Uses `PooledMemoryStream` for Huffman-decoded buffer and invalid-header empty stream.
src/SharpCompress/Compressors/Squeezed/SqueezedStream.Async.cs	Async Squeezed implementation mirrors `PooledMemoryStream` usage.
src/SharpCompress/ThrowHelper.cs	Adds a `long` overload for `ThrowIfGreaterThan`.
src/SharpCompress/packages.lock.json	Downgrades `Microsoft.NET.ILLink.Tasks` resolved/requested versions for net8.0 and net10.0.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

… in PooledMemoryStream Agent-Logs-Url: https://github.com/adamhathcock/sharpcompress/sessions/1f2c3c48-1112-43f1-82ce-efb157fbe0d5 Co-authored-by: adamhathcock <527620+adamhathcock@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…am/pooledmemorystream # Conflicts: # src/SharpCompress/IO/PooledMemoryStream.cs

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 10 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-19T10:54:32Z

+        using var headerStream = new PooledMemoryStream();
        ArchiveHeaderWriter.WriteRawHeader(headerStream, mainStreamsInfo, filesInfo);


headerStream is now a PooledMemoryStream, but the raw-header path later computes a CRC via rawHeaderStream.GetBuffer().AsSpan(0, (int)Length). Since PooledMemoryStream.GetBuffer() allocates a fresh array sized to Capacity, this can introduce extra allocations proportional to capacity and undermine the pooling benefit. Consider switching CRC calculation to use a Length-sized buffer or a non-allocating path.

@copilot apply changes based on this feedback

Fixed in commit f8485b0. Both WriteCompressedHeader and WriteRawHeaderToOutput now compute the CRC by writing the stream's blocks directly into a Crc32Stream(Stream.Null) sink via WriteTo(), which avoids allocating any contiguous buffer.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Agent-Logs-Url: https://github.com/adamhathcock/sharpcompress/sessions/0ce9533f-f529-4c07-b641-6bf0883a0643 Co-authored-by: adamhathcock <527620+adamhathcock@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-19T11:25:01Z

+        using var output = new PooledMemoryStream();
        decoder.Code(input, output, compressedData.Length, uncompressedSize, null);


FindConsumedBytes doesn’t use the decompressed output; it only needs input.Position after decoder.Code(...). Using PooledMemoryStream here adds pooling overhead and can rent/return buffers unnecessarily. Use Stream.Null (or another sink stream) for output instead.

Suggested change

using var output = new PooledMemoryStream();

decoder.Code(input, output, compressedData.Length, uncompressedSize, null);

decoder.Code(input, Stream.Null, compressedData.Length, uncompressedSize, null);

Agent-Logs-Url: https://github.com/adamhathcock/sharpcompress/sessions/77f37cc4-3538-4df1-a816-aa7391065878 Co-authored-by: adamhathcock <527620+adamhathcock@users.noreply.github.com>

adamhathcock added 3 commits April 4, 2026 10:44

First pass of implementation

817bd2a

use pooled memory streams for better performance

a26198a

cleaned up PooledMemoryStream.cs

2bf0075

Copilot AI review requested due to automatic review settings April 4, 2026 10:11

Copilot started reviewing on behalf of adamhathcock April 4, 2026 10:12 View session

Copilot AI reviewed Apr 4, 2026

View reviewed changes

Update src/SharpCompress/IO/PooledMemoryStream.cs

db9bdb2

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 5, 2026 13:10

Update src/SharpCompress/Compressors/PPMd/PpmdStream.cs

6b9a3f8

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot started reviewing on behalf of adamhathcock April 5, 2026 13:10 View session

Update src/SharpCompress/IO/PooledMemoryStream.cs

5e7644b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot started work on behalf of adamhathcock April 5, 2026 13:12 View session

Copilot AI reviewed Apr 5, 2026

View reviewed changes

Comment thread src/SharpCompress/IO/PooledMemoryStream.cs Outdated

Comment thread src/SharpCompress/IO/PooledMemoryStream.cs Outdated

Comment thread src/SharpCompress/IO/PooledMemoryStream.cs

Comment thread tests/SharpCompress.Test/Streams/PooledMemoryStreamTests.cs Outdated

Update tests/SharpCompress.Test/Streams/PooledMemoryStreamTests.cs

c01dd5c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 5, 2026 13:16

Copilot started reviewing on behalf of adamhathcock April 5, 2026 13:17 View session

Copilot AI reviewed Apr 5, 2026

View reviewed changes

Remove try/catch from async overloads and fix CA1725 parameter naming…

8c4fed3

… in PooledMemoryStream Agent-Logs-Url: https://github.com/adamhathcock/sharpcompress/sessions/1f2c3c48-1112-43f1-82ce-efb157fbe0d5 Co-authored-by: adamhathcock <527620+adamhathcock@users.noreply.github.com>

Copilot finished work on behalf of adamhathcock April 5, 2026 13:22

adamhathcock added 2 commits April 5, 2026 14:39

fix PooledMemoryStream.cs

c25fec3

remove net5 from target frameworks to avoid issues

bfe7824

Copilot AI review requested due to automatic review settings April 5, 2026 13:47

Copilot started reviewing on behalf of adamhathcock April 5, 2026 13:48 View session

Copilot AI reviewed Apr 5, 2026

View reviewed changes

Update tests/SharpCompress.Test/Streams/PooledMemoryStreamTests.cs

4f09c6d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 5, 2026 13:57

Update src/SharpCompress/IO/PooledMemoryStream.cs

ec46584

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot started reviewing on behalf of adamhathcock April 5, 2026 13:57 View session

Copilot AI reviewed Apr 5, 2026

View reviewed changes

implement GetBuffer with non-pooled array creation

fb78598

adamhathcock added 4 commits April 5, 2026 15:25

remove contiguous buffer tracking and added overrenting array pool test

d1e6173

Merge remote-tracking branch 'origin/adam/pooledmemorystream' into ad…

2487d89

…am/pooledmemorystream # Conflicts: # src/SharpCompress/IO/PooledMemoryStream.cs

Merge branch 'master' into adam/pooledmemorystream

e29670e

format

b36689a

Copilot AI review requested due to automatic review settings April 19, 2026 10:49

Copilot started reviewing on behalf of adamhathcock April 19, 2026 10:49 View session

Copilot AI reviewed Apr 19, 2026

View reviewed changes

update tests

ad3c5da

Copilot started work on behalf of adamhathcock April 19, 2026 11:18 View session

Update src/SharpCompress/IO/PooledMemoryStream.cs

891d126

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 19, 2026 11:18

Update src/SharpCompress/IO/PooledMemoryStream.cs

e0ccbbb

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot started reviewing on behalf of adamhathcock April 19, 2026 11:19 View session

Add EnsureNotClosed() to Flush and FlushAsync in PooledMemoryStream

49a7e15

Agent-Logs-Url: https://github.com/adamhathcock/sharpcompress/sessions/0ce9533f-f529-4c07-b641-6bf0883a0643 Co-authored-by: adamhathcock <527620+adamhathcock@users.noreply.github.com>

Copilot finished work on behalf of adamhathcock April 19, 2026 11:21

Copilot AI reviewed Apr 19, 2026

View reviewed changes

Copilot started work on behalf of adamhathcock April 19, 2026 11:29 View session

Compute CRC without GetBuffer() in SevenZipWriter to avoid allocation

f8485b0

Agent-Logs-Url: https://github.com/adamhathcock/sharpcompress/sessions/77f37cc4-3538-4df1-a816-aa7391065878 Co-authored-by: adamhathcock <527620+adamhathcock@users.noreply.github.com>

Copilot finished work on behalf of adamhathcock April 19, 2026 11:34

adamhathcock enabled auto-merge April 19, 2026 13:56

adamhathcock merged commit 8bd386c into master Apr 19, 2026
7 checks passed

adamhathcock deleted the adam/pooledmemorystream branch April 19, 2026 14:02

		using var headerStream = new PooledMemoryStream();
		ArchiveHeaderWriter.WriteRawHeader(headerStream, mainStreamsInfo, filesInfo);

		using var output = new PooledMemoryStream();
		decoder.Code(input, output, compressedData.Length, uncompressedSize, null);

	using var output = new PooledMemoryStream();
	decoder.Code(input, output, compressedData.Length, uncompressedSize, null);
	decoder.Code(input, Stream.Null, compressedData.Length, uncompressedSize, null);

Conversation

adamhathcock commented Apr 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

Uh oh!

adamhathcock Apr 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!