Skip to content

lucuma13/mhl-suite

Repository files navigation

mhl-suite

PyPI Version OS Ruff ty CI codecov

mhl-suite is a toolkit for sealing and verifying MHL files. It consists of three primary executables:

  • mhlver: one tool to verify them all. Recursively verifies every MHL file under a path (both classic flat MHL and ASC-MHL), with a per-file progress bar, reports and optional XSD schema validation. Verifying an ASC MHL history appends a new generation recording the results by default, -R/--read-only verifies without writing anything – useful for locked, WORM or archived media (it flattens the history and verifies the in-memory manifest).
  • simple-mhl: a modern sealing and verification tool, for classic flat MHL files. A successor of the discontinued mhl-tool and backwards compatible with legacy manifests - twice as fast (smart parallel hashing), real support for xxh64, NFC/NFD handling, XSD schema validation features, cleaner output and structured exit codes.
  • advanced-mhl: a chain-of-custody tool, for ASC MHL histories. An actively maintained implementation of the ASC MHL Specification, fast by design: size pre-checks before hashing, smart parallel hashing, NFC/NFD handling, interoperable with the reference implementation — tested against it, and with several of its known verification issues fixed.

🚀 Installation

  1. Install the uv package manager with the official installer, or:
  • macOS: brew install uv
  • Windows: winget install astral-sh.uv
  • Linux (Debian): apt-get install uv
  1. Install the toolkit:
uv tool install mhl-suite
  1. Test the installation (if the command is not recognised try uv tool update-shell and restart your terminal):
mhlver --version; simple-mhl --version; advanced-mhl --version

Alternative (macOS only)

Install with Homebrew: brew install lucuma13/dit/mhl-suite

📖 Usage examples

Verify MHL files (both classic and ASC-MHL):

mhlver path/to/file.mhl
mhlver path/to/directory/                  # every manifest found
mhlver                                     # current directory

Create a report after verification:

mhlver --report path/to/directory/

Quick size-only check:

mhlver --size-only path/to/directory/

Seal a directory:

simple-mhl seal path/to/directory/
simple-mhl seal -a md5 -a xxh64 path/to/directory/         # use both MD5 and xxh64 algorithms
simple-mhl seal -o path/to/output/mhl path/to/directory/   # write the MHL into a parent directory

Create a new verification generation on the ASC MHL history:

advanced-mhl generate path/to/directory/

Validate XML Schema Definition of a file:

mhlver --xsd-schema-check path/to/file

Run any tool with --help to see the full list of options.

🔤 Unicode filenames

Unicode allows the same visible filename to be written with different byte sequences: in rosé.mov the é can be a single precomposed code point (U+00E9 – the NFC form) or a decomposed e followed by a combining accent (U+0301 – the NFD form). Filesystems split on what they store and how they look names up:

Filesystem Storage normalisation Normalisation sensitivity
NTFS, FAT32, exFAT, ext4 preserving sensitive
APFS preserving insensitive
HFS+ forcing insensitive

A name's byte form can therefore change in transit through an offload – HFS+ round-trips rewrite it with no data corruption at all. The picture is fragile even on a single macOS machine, when reading an exFAT volume with colliding Unicode filenames: Finder and ls can disagree about how many files exist, while cp and Finder may copy different data. So the byte form of a filename is not stable enough to carry identity: a file's identity is its contents, not its path.

Verification tools split into two camps over this: some compare manifest paths to disk byte-for-byte, others tolerate normalisation differences. mhl-suite is built for interoperability:

  • Sealing records names verbatim. The exact bytes the filesystem reports land in the manifest. Since recorded bytes are only as meaningful as the filesystem that reported them, we also include seal-context on the manifest (OS, kernel, filesystem, driver).
  • Sealing refuses ambiguous sources. If two names in the tree are Unicode-equivalent, the seal aborts before any hashing and names both byte forms. Such trees would silently merge, overwrite, or lose one file's data on the first copy to a normalisation-insensitive volume, and no manifest could tell the resulting entries apart – refusing early, while the source media is still in hand, is the only moment this is fixable.
  • Verifying matches bytes first, equivalence second. A name whose normalisation form drifted still verifies via Unicode equivalence – but only when the match cannot guess wrong. If equivalent forms coexist on disk or in the manifest, no fallback is attempted and the discrepancy is reported as plain missing/unknown findings.

📊 Performance

All tools use smart parallel hashing. They measure real sequential hashing throughput, the aggregate read speed (across several concurrent streams), and the in-memory speed of the selected hash algorithm to always achieve the highest performance in any volume. They parallelise only when it pays off, which is when a hash can't keep the storage busy on its own.

Storage Algorithm Bottleneck Ref. implementation mhl-suite Result
SSD xxh64 disk 3.14 GB/s 3.04 GB/s
MD5 hash 0.93 GB/s 1.86 GB/s
12-bay NAS xxh64 read concurrency 0.72 GB/s 1.06 GB/s 1.5×
MD5 read concurrency 0.54 GB/s 1.10 GB/s

Measured over an 80 GB media set with the OS page cache evicted before every pass and the hash algorithm pinned. The reference implementations are ascmhl and mhl-tool.

About

Toolkit for sealing and verifying MHL files

Topics

Resources

License

Stars

1 star

Watchers

0 watching

Forks

Contributors

Languages