Skip to content

Commit 075abe0

Browse files
d33bsMikeLippincott
d33bs
and
MikeLippincott
authored
Add per-shard CLI and Metadata_Imaging_ImageID propagation (#50)
* Add per-shard CLI and Metadata_Imaging_ImageID propagation ZedProfiler is the feature extractor the NF1 pipeline dispatches per well/FOV shard via SLURM sbatch. This commit adds the command that process runs and the identifier column that makes shards warehouse-joinable. CLI (src/zedprofiler/cli.py): - "ZedProfiler run" subcommand (argparse): repeatable --image/--label NAME=PATH flags, --anisotropy-spacing Z Y X, identifier fields (--patient-tumor/--plate/--well/--field), --out-dir, a --features selector, repeatable --feature TYPE[,key=value,...] advanced requests, --skip-existing, and --force. - Reuses the six compute_* featurizers; builds the shared loader via ImageSetLoader.from_image_dict with the identifier fields. - Restartable/idempotent: deterministic output paths, --skip-existing filters before any image is read (a finished shard re-run skips I/O entirely), and atomic writes (temp + os.replace) so a crashed shard never leaves a partial file that --skip-existing would mistake for complete. - Fixes the orphaned/mis-cased console script to zedprofiler.cli:trigger. Identifiers (src/zedprofiler/identifiers.py): - build_image_id(patient_tumor, plate, well, field) -> deterministic Metadata_Imaging_ImageID; single source of truth for the format. Loaders (src/zedprofiler/IO/loading_classes.py): - ImageSetConfig carries patient_tumor/plate/well/field with an image_id property; ImageSetLoader exposes image_id (falls back to image_set_name). - New from_image_dict classmethod builds a multi-channel loader from an in-memory {key: ndarray} dict (the path the CLI needs); it derives compartments/image names directly from the declared label keys so it is self-contained and correct independent of get_compartments. Featurizers (6 modules): - Each emits Metadata_Imaging_ImageID before Metadata_Experiment_ImageSet. Feature values are unchanged; only a metadata column is added. Feature writing: - save_features_as_parquet gains an opt-in atomic flag used by the CLI. End-to-end CLI tests self-skip when the CellProfiler 3D tutorial data is absent (it lands via a separate data commit), so the CLI test module stays green everywhere and runs in full wherever the data is available. * Add per-shard CLI and Metadata_Imaging_ImageID propagation ZedProfiler is the feature extractor the NF1 pipeline dispatches per well/FOV shard via SLURM sbatch. This commit adds the command that process runs and the identifier column that makes shards warehouse-joinable. CLI (src/zedprofiler/cli.py): - "ZedProfiler run" subcommand (argparse): repeatable --image/--label NAME=PATH flags, --anisotropy-spacing Z Y X, identifier fields (--patient-tumor/--plate/--well/--field), --out-dir, a --features selector, repeatable --feature TYPE[,key=value,...] advanced requests, --skip-existing, and --force. - Reuses the six compute_* featurizers; builds the shared loader via ImageSetLoader.from_image_dict with the identifier fields. - Restartable/idempotent: deterministic output paths, --skip-existing filters before any image is read (a finished shard re-run skips I/O entirely), and atomic writes (temp + os.replace) so a crashed shard never leaves a partial file that --skip-existing would mistake for complete. - Fixes the orphaned/mis-cased console script to zedprofiler.cli:trigger. Identifiers (src/zedprofiler/identifiers.py): - build_image_id(patient_tumor, plate, well, field) -> deterministic Metadata_Imaging_ImageID; single source of truth for the format. Loaders (src/zedprofiler/IO/loading_classes.py): - ImageSetConfig carries patient_tumor/plate/well/field with an image_id property; ImageSetLoader exposes image_id (falls back to image_set_name). - New from_image_dict classmethod builds a multi-channel loader from an in-memory {key: ndarray} dict (the path the CLI needs); it derives compartments/image names directly from the declared label keys so it is self-contained and correct independent of get_compartments. Featurizers (6 modules): - Each emits Metadata_Imaging_ImageID before Metadata_Experiment_ImageSet. Feature values are unchanged; only a metadata column is added. Feature writing: - save_features_as_parquet gains an opt-in atomic flag used by the CLI. Tests: - test_real_world_data.py colocalization loader switched to from_image_dict so the new image_id column is populated for the existing colocalization end-to-end test. - CLI end-to-end tests run against the CellProfiler 3D tutorial data present on main and self-skip if that data is absent on other branches. * test coverage * fov labeling Co-Authored-By: Mike Lippincott <58147848+MikeLippincott@users.noreply.github.com> * add cli docs to readme Co-Authored-By: Mike Lippincott <58147848+MikeLippincott@users.noreply.github.com> * feature specs Co-Authored-By: Mike Lippincott <58147848+MikeLippincott@users.noreply.github.com> * cli spec Co-Authored-By: Mike Lippincott <58147848+MikeLippincott@users.noreply.github.com> --------- Co-authored-by: Mike Lippincott <58147848+MikeLippincott@users.noreply.github.com>
1 parent b44f5d3 commit 075abe0

25 files changed

Lines changed: 1735 additions & 33 deletions

‎README.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,63 @@ Accepted image formats (order matters):
3131

3232
- Single channel: `(z, y, x)`
3333

34+
## Command-line interface
35+
36+
`ZedProfiler run` extracts features for a single well/field-of-view (FOV) shard: it loads one image set from explicit file paths, runs a selected subset of featurizers, and writes one Parquet per feature table to an output directory. It is the command a workflow manager (for example Nextflow via SLURM `sbatch`) dispatches once per shard.
37+
38+
After `uv sync` (or `pip install .`) the `ZedProfiler` console script is available; from a checkout you can also use `uv run ZedProfiler run ...`. Run `ZedProfiler run --help` for the authoritative, up-to-date list of arguments.
39+
40+
```bash
41+
ZedProfiler run \
42+
--image=DNA=/path/to/channel1.tif \
43+
--label=Nuclei=/path/to/nuclei_mask.tiff \
44+
--anisotropy-spacing 1.0 1.0 1.0 \
45+
--patient-tumor NF0014_T1 \
46+
--plate PLATE01 \
47+
--well A1 \
48+
--fov 1 \
49+
--out-dir ./shard_output \
50+
--features Intensity
51+
```
52+
53+
### Arguments
54+
55+
| Argument | Required | Description |
56+
| -------------------------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
57+
| `--image=NAME=PATH` | yes (>=1, repeatable) | A channel image as `NAME=PATH`. Repeat for multi-channel shards. |
58+
| `--label=NAME=PATH` | yes (>=1, repeatable) | A compartment label mask as `NAME=PATH`. Repeat for multiple compartments. |
59+
| `--anisotropy-spacing Z Y X` | yes | Z, Y, X voxel spacing (three floats). |
60+
| `--patient-tumor` | yes | Patient-tumor identifier (e.g. `NF0014_T1`). |
61+
| `--plate` | yes | Plate identifier. |
62+
| `--well` | yes | Well identifier (e.g. `A1`). |
63+
| `--fov` | yes | Field-of-view index or identifier. |
64+
| `--out-dir` | yes | Shard output directory (created if needed). |
65+
| `--features` | no | Comma-separated feature types to run (selector). With no `--feature` flags, runs these types over the channel x compartment cross-product; with `--feature` flags, restricts those requests by type. Default: all single-channel types, plus `Colocalization` when >=2 channels are declared. |
66+
| `--feature=TYPE[,key=value,...]` | no (repeatable) | An explicit feature request, e.g. `Intensity,channel=DNA,compartment=Nuclei` or `Colocalization,channel1=DNA1,channel2=DNA2,compartment=Nuclei,fast_costes=Faster`. |
67+
| `--skip-existing` | no | Skip a feature request whose output Parquet already exists. |
68+
| `--force` | no | Overwrite even when the output exists (writes are still atomic). |
69+
70+
### Feature types
71+
72+
`VolumeSizeShape`, `Intensity`, `Neighbors`, `Texture`, and `Granularity` are single-channel features run per channel x compartment. `Colocalization` is a two-channel feature run per ordered channel pair x compartment.
73+
74+
### Outputs
75+
76+
Each request writes `{compartment}_{channel}_{feature_type}_cpu_features.parquet` into `--out-dir`. Every table carries `Metadata_Imaging_ImageID` (deterministically built from the patient-tumor, plate, well, and FOV coordinates) and `Metadata_Experiment_ImageSet` so downstream tables can rejoin. Writes are atomic (temp file + replace), so a crashed shard never leaves a partial file that `--skip-existing` would mistake for a complete one.
77+
78+
### Two-channel colocalization example
79+
80+
```bash
81+
ZedProfiler run \
82+
--image=DNA1=/path/to/channel1.tif \
83+
--image=DNA2=/path/to/channel2.tif \
84+
--label=Nuclei=/path/to/nuclei_mask.tiff \
85+
--anisotropy-spacing 1.0 1.0 1.0 \
86+
--patient-tumor NF0014_T1 --plate PLATE01 --well A1 --fov 1 \
87+
--out-dir ./shard_output \
88+
--feature=Colocalization,channel1=DNA1,channel2=DNA2,compartment=Nuclei,fast_costes=Faster
89+
```
90+
3491
## Quality Gates
3592

3693
We lint and format code with our pre-commit configuration.

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ dependencies = [
4242
]
4343
urls.Homepage = "https://zedprofiler.readthedocs.io"
4444
urls.Repository = "https://github.com/WayScience/ZedProfiler"
45-
scripts.ZedProfiler = "ZedProfiler.cli:trigger"
45+
scripts.ZedProfiler = "zedprofiler.cli:trigger"
4646

4747
[dependency-groups]
4848
dev = [

‎src/zedprofiler/IO/feature_writing_utils.py‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
from __future__ import annotations
77

88
import dataclasses
9+
import os
910
import pathlib
1011

1112
import pandas
@@ -181,6 +182,7 @@ def save_features_as_parquet(
181182
parent_path: pathlib.Path,
182183
df: pandas.DataFrame,
183184
metadata: FeatureMetadata,
185+
atomic: bool = False,
184186
) -> pathlib.Path:
185187
"""Save features as parquet files in a consistent way.
186188
@@ -196,6 +198,12 @@ def save_features_as_parquet(
196198
metadata : FeatureMetadata
197199
Metadata for the feature output (compartment, channel, feature_type,
198200
cpu_or_gpu).
201+
atomic : bool
202+
When True, write to a sibling ``.tmp`` file then atomically replace
203+
the destination via ``os.replace``. This prevents a crashed write
204+
from leaving a partial parquet file that a restartable caller (using
205+
``--skip-existing``) could mistake for a complete one. Default False
206+
preserves the existing direct-write behavior.
199207
200208
Returns
201209
-------
@@ -210,5 +218,10 @@ def save_features_as_parquet(
210218
metadata.cpu_or_gpu,
211219
)
212220
save_path = parent_path / f"{output_prefix}_features.parquet"
213-
validated_df.to_parquet(save_path, index=False)
221+
if atomic:
222+
tmp_path = save_path.with_suffix(save_path.suffix + ".tmp")
223+
validated_df.to_parquet(tmp_path, index=False)
224+
os.replace(tmp_path, save_path)
225+
else:
226+
validated_df.to_parquet(save_path, index=False)
214227
return save_path

‎src/zedprofiler/IO/loading_classes.py‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
from beartype import beartype
1313

1414
from zedprofiler.contracts import ImageArrayModel
15+
from zedprofiler.identifiers import build_image_id
1516

1617
logging.basicConfig(level=logging.INFO)
1718

@@ -42,6 +43,15 @@ class ImageSetConfig:
4243
image_set_name: str | None = None
4344
label_key_name: list[str] | None = None
4445
raw_image_key_name: list[str] | None = None
46+
# Imaging-coordinate identifier fields used to build a deterministic
47+
# ``Metadata_Imaging_ImageID``. All four must be set for ``image_id`` to be
48+
# populated; when any is None, ``image_id`` is None and the loader falls
49+
# back to ``image_set_name`` for the emitted metadata column (see
50+
# ``ImageSetLoader.image_id``).
51+
patient_tumor: str | None = None
52+
plate: str | None = None
53+
well: str | None = None
54+
field_of_view: int | str | None = None
4555

4656
# validate the arg types
4757
def __post_init__(self) -> None:
@@ -52,12 +62,42 @@ def __post_init__(self) -> None:
5262
raise TypeError("label_key_name must be a list of strings or None")
5363
if not isinstance(self.raw_image_key_name, (list, type(None))):
5464
raise TypeError("raw_image_key_name must be a list of strings or None")
65+
if not isinstance(self.patient_tumor, (str, type(None))):
66+
raise TypeError("patient_tumor must be a string or None")
67+
if not isinstance(self.plate, (str, type(None))):
68+
raise TypeError("plate must be a string or None")
69+
if not isinstance(self.well, (str, type(None))):
70+
raise TypeError("well must be a string or None")
71+
if not isinstance(self.field_of_view, (int, str, type(None))):
72+
raise TypeError("field_of_view must be an int, str, or None")
5573

5674
if self.label_key_name is None:
5775
self.label_key_name = []
5876
if self.raw_image_key_name is None:
5977
self.raw_image_key_name = []
6078

79+
@property
80+
def image_id(self) -> str | None:
81+
"""Deterministic ``Metadata_Imaging_ImageID`` value, or None if unset.
82+
83+
Returns ``build_image_id(...)`` when all four coordinate fields are
84+
set, otherwise ``None`` (the loader then falls back to
85+
``image_set_name``).
86+
"""
87+
if (
88+
self.patient_tumor is not None
89+
and self.plate is not None
90+
and self.well is not None
91+
and self.field_of_view is not None
92+
):
93+
return build_image_id(
94+
patient_tumor=self.patient_tumor,
95+
plate=self.plate,
96+
well=self.well,
97+
field_of_view=self.field_of_view,
98+
)
99+
return None
100+
61101

62102
class _LazyImageSetDict(dict): # type: ignore[type-arg]
63103
"""Dictionary that loads image arrays on first access."""
@@ -179,6 +219,13 @@ def __init__( # noqa: PLR0913
179219
self.anisotropy_factor = self.anisotropy_spacing[0] / self.anisotropy_spacing[1]
180220
self.image_set_name = config.image_set_name
181221
self.label_set_path = label_set_path
222+
# Deterministic imaging identifier for the warehouse join key
223+
# (``Metadata_Imaging_ImageID``). When identifier fields are not
224+
# provided (legacy/library use), fall back to the image set name so
225+
# the emitted metadata column always has a value.
226+
self.image_id = (
227+
config.image_id if config.image_id is not None else config.image_set_name
228+
)
182229
self._load_path_based_images(
183230
channel_mapping=channel_mapping,
184231
channel_tokens=channel_tokens,
@@ -195,6 +242,92 @@ def __init__( # noqa: PLR0913
195242
self.get_image_names()
196243
self.get_unique_objects_in_compartments()
197244

245+
@classmethod
246+
def from_image_dict( # noqa: PLR0913
247+
cls,
248+
image_dict: dict[str, numpy.ndarray],
249+
*,
250+
anisotropy_spacing: tuple[float, float, float],
251+
image_set_name: str | None = None,
252+
label_key_names: list[str] | None = None,
253+
patient_tumor: str | None = None,
254+
plate: str | None = None,
255+
well: str | None = None,
256+
field_of_view: int | str | None = None,
257+
) -> ImageSetLoader:
258+
"""Build an ImageSetLoader from an in-memory channel/label dict.
259+
260+
Existing constructors only accept a directory glob (path-based) or a
261+
single array (array-based). A well/FOV shard carries multiple channels
262+
and multiple compartments as distinct arrays, so this classmethod
263+
builds the ``image_set_dict`` directly from a pre-loaded
264+
``{key: ndarray}`` mapping. It formalizes the ``ImageSetLoader.__new__``
265+
workaround previously used in the colocalization test helper.
266+
267+
Parameters
268+
----------
269+
image_dict : dict[str, numpy.ndarray]
270+
Mapping of channel names and compartment names to their arrays.
271+
Each array is validated through ``ImageArrayModel``.
272+
anisotropy_spacing : tuple[float, float, float]
273+
(z_spacing, y_spacing, x_spacing).
274+
image_set_name : str | None
275+
Optional image set name (emitted as ``Metadata_Experiment_ImageSet``).
276+
label_key_names : list[str] | None
277+
Keys in ``image_dict`` that are compartment labels (not channels).
278+
Used by ``get_compartments`` to distinguish compartments from
279+
raw channels.
280+
patient_tumor, plate, well, field_of_view : optional
281+
Imaging-coordinate identifier fields. When all four are set, the
282+
loader's ``image_id`` is the deterministic
283+
``Metadata_Imaging_ImageID``; otherwise it falls back to
284+
``image_set_name``.
285+
286+
Returns
287+
-------
288+
ImageSetLoader
289+
A fully initialized loader (compartments, image names, and unique
290+
compartment objects populated).
291+
292+
"""
293+
self = cls.__new__(cls)
294+
self.image_set_dict = _LazyImageSetDict()
295+
for key, array in image_dict.items():
296+
# Run through pydantic validation to ensure each array is valid,
297+
# mirroring ``_load_array_based_images``.
298+
self.image_set_dict[key] = ImageArrayModel(array=array).array
299+
self._label_key_names = list(label_key_names or [])
300+
self.anisotropy_spacing = anisotropy_spacing
301+
self.anisotropy_factor = self.anisotropy_spacing[0] / self.anisotropy_spacing[1]
302+
self.image_set_name = image_set_name
303+
self.label_set_path = None
304+
config = ImageSetConfig(
305+
image_set_name=image_set_name,
306+
label_key_name=list(label_key_names or []),
307+
raw_image_key_name=[
308+
key for key in image_dict if key not in (label_key_names or [])
309+
],
310+
patient_tumor=patient_tumor,
311+
plate=plate,
312+
well=well,
313+
field_of_view=field_of_view,
314+
)
315+
self.image_id = (
316+
config.image_id if config.image_id is not None else config.image_set_name
317+
)
318+
# Set compartments and image names directly from the declared label
319+
# keys rather than calling ``get_compartments``/``get_image_names``.
320+
# Those methods' compartment heuristic differs across repo revisions
321+
# (it was corrected in a later bugfix commit), but this classmethod
322+
# already knows which keys are labels, so deriving the split here keeps
323+
# it self-contained and correct on any base.
324+
self.compartments = list(self._label_key_names)
325+
self.image_names = [
326+
key for key in image_dict if key not in self._label_key_names
327+
]
328+
self.get_unique_objects_in_compartments()
329+
return self
330+
198331
@staticmethod
199332
def _validate_input_sources(
200333
image_set_path: pathlib.Path | None,

0 commit comments

Comments
 (0)