Skip to content

support sparse simg from a-i-b #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mangelajo merged 1 commit into from
Jan 12, 2026
Merged

support sparse simg from a-i-b #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 112 additions & 0 deletions docs/sparse-image-support.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# Sparse Image (simg) Support

This document explains how `fls` handles Android Sparse Image format for efficient block device flashing.

## Overview

Android Sparse Images are a compressed format designed for efficient block device flashing. Instead of storing the entire disk image (which may contain large empty regions), sparse images encode only the actual data along with instructions for filling or skipping empty regions.

This allows `fls` to:
- Skip writing to unallocated regions (DONT_CARE chunks)
- Fill large regions with a pattern without transferring the full data (FILL chunks)
- Significantly reduce flash time for images with empty space

`fls` automatically detects sparse images by checking for the magic number `0xED26FF3A` in the first 28 bytes of decompressed data. No configuration is needed.

## Sparse Image Format

The format is compatible with images generated by [automotive-image-builder](https://github.com/osbuild/automotive-image-builder) and follows the Android sparse image v1.0 specification.

### Header (28 bytes)

| Offset | Size | Field | Description |
|--------|------|-------|-------------|
| 0 | 4 | magic | `0xED26FF3A` |
| 4 | 2 | major_version | Must be 1 |
| 6 | 2 | minor_version | Usually 0 |
| 8 | 2 | file_hdr_sz | Header size (28) |
| 10 | 2 | chunk_hdr_sz | Chunk header size (12) |
| 12 | 4 | block_size | Output block size (typically 4096) |
| 16 | 4 | total_blocks | Total blocks in output image |
| 20 | 4 | total_chunks | Number of chunks |
| 24 | 4 | image_checksum | Optional checksum |

### Chunk Types

Each chunk has a 12-byte header followed by optional data:

| Type | Value | Description |
|------|-------|-------------|
| RAW | `0xCAC1` | Raw data blocks - write directly |
| FILL | `0xCAC2` | 4-byte pattern repeated across blocks |
| DONT_CARE | `0xCAC3` | Empty blocks - seek over them |

## How It Works

```text
┌─────────────┐ ┌──────────────┐ ┌────────────────┐ ┌─────────────┐
│ HTTP │───►│ Decompressor │───►│ Format │───►│ Block │
│ Download │ │ (xz/gz/zst) │ │ Detector/Parser│ │ Writer │
└─────────────┘ └──────────────┘ └────────────────┘ └─────────────┘
```

1. **Download**: HTTP stream with retry support
2. **Decompress**: Automatic detection of xz, gzip, zstd, or uncompressed
3. **Detect Format**: First 28 bytes checked for sparse magic
4. **Parse/Write**:
- If sparse: Parse chunks, execute Seek/Write/Fill commands
- If regular: Stream directly to block device

### Auto-Detection

The `FormatDetector` buffers the first 28-64 bytes to identify the format:
- If magic `0xED26FF3A` is found with valid header → parse as sparse
- Otherwise → write as regular raw image

### Streaming Parser

The `SparseParser` (in `simg.rs`) processes data incrementally, emitting `WriterCommand` values:

```rust
// From src/fls/simg.rs
enum WriterCommand {
Seek(u64), // Skip to offset (DONT_CARE)
Write(Vec<u8>), // Write data (RAW)
Fill { pattern: [u8; 4], bytes: u64 }, // Fill with pattern (FILL)
Complete { expected_size: u64 }, // Parsing complete
}
```
Comment on lines +70 to +78
Copy link

@coderabbitai coderabbitai bot Jan 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Inconsistent enum name in documentation.

The code sample shows WriterCommand but the actual implementation in src/fls/simg.rs uses WriteCommand. This inconsistency could confuse readers.

📝 Suggested fix 
-enum WriterCommand {
+enum WriteCommand {
     Seek(u64),                             // Skip to offset (DONT_CARE)
     Write(Vec<u8>),                        // Write data (RAW)
     Fill { pattern: [u8; 4], bytes: u64 }, // Fill with pattern (FILL)
     Complete { expected_size: u64 },       // Parsing complete
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
```rust
// From src/fls/simg.rs
enum WriterCommand {
Seek(u64), // Skip to offset (DONT_CARE)
Write(Vec<u8>), // Write data (RAW)
Fill { pattern: [u8; 4], bytes: u64 }, // Fill with pattern (FILL)
Complete { expected_size: u64 }, // Parsing complete
}
```
🤖 Prompt for AI Agents 
In @docs/sparse-image-support.md around lines 70 - 78, Documentation enum name
is inconsistent: the docs show WriterCommand but the code defines WriteCommand;
update the docs sample to use the actual enum name WriteCommand (or, if you
prefer to standardize on WriterCommand, rename the enum in the implementation to
WriterCommand) and ensure all references in the docs and code (the enum
definition and any matches/usages) are aligned to the same identifier.


## O_DIRECT Considerations

When using `--o-direct`:

- **Write Alignment**: Writes are padded to 4096 bytes with zeros
- **Seek Alignment**: Offsets must be 4096-byte aligned (sparse images naturally align)
- **Buffer Handling**: Internal 1MB buffer handles alignment automatically

## Example: Efficiency Gains

Given a 4GB disk image with 1GB of actual data and 3GB of empty space:

| Method | Data Transferred | Data Written |
|--------|------------------|--------------|
| Raw image | 4GB | 4GB |
| Sparse image | ~1GB compressed | ~1GB |

The sparse format encodes empty regions as DONT_CARE chunks, which `fls` handles by seeking instead of writing zeros.

## Limitations

- **RAW chunk size**: Individual RAW chunks limited to ~4GB (u32) for streaming efficiency
- **Sparse version**: Only version 1.x supported
- **Supported chunks**: Only RAW, FILL, and DONT_CARE chunks are supported; unknown chunk types cause an error

## Related Files

| File | Purpose |
|------|---------|
| `src/fls/simg.rs` | Sparse image parser and types |
| `src/fls/format_detector.rs` | Auto-detection logic |
| `src/fls/from_url.rs` | Main flash loop |
| `src/fls/block_writer.rs` | O_DIRECT-aware block device writer |
Loading