Skip to content

feat: adds option to pass file regex pattern#30

Merged
jtyoung84 merged 4 commits intomainfrom
feat-29-filename-regex
Sep 23, 2025
Merged

feat: adds option to pass file regex pattern#30
jtyoung84 merged 4 commits intomainfrom
feat-29-filename-regex

Conversation

@jtyoung84
Copy link
Contributor

@jtyoung84 jtyoung84 commented Sep 20, 2025

Closes #29

  • Removes setuptools-scm from build-system, which has been causing issues in github actions workflows
  • Adds parameter to allow user to better control which files in a directory will be processed. This is needed for the chronic ephys data collection where one hour chunks are added to a directory, and we want to process one hour at a time.
  • Adds test to verify filter works

Copy link
Collaborator

@galenlynch galenlynch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You probably want re.search not re.match.

Keys are Paths and values are argument sets. Default is an empty
dictionary.
file_filter_pattern : str | None
If set, will filter file names based on this pattern. Default is None.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If set, will filter file names based on this pattern. Default is None.
If set, will filter file names based on this regex pattern. Default is None.

ffmpeg_thread_cnt: int = Field(
default=0, description="Number of threads per ffmpeg compression job."
)
file_filter: Optional[str] = Field(
Copy link
Collaborator

@galenlynch galenlynch Sep 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
file_filter: Optional[str] = Field(
file_filter: str | None = Field(

ruff UP007

from os import symlink, walk
from os.path import relpath
from pathlib import Path
from typing import Optional
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from typing import Optional

Not needed if you follow str | None

output_dir: Path,
arg_set,
overrides=dict(),
file_filter_pattern: Optional[str] = None,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
file_filter_pattern: Optional[str] = None,
file_filter_pattern: str | None = None,

UP007


for file_name in files:
file_path = Path(root) / file_name
if file_filter_pattern and not re.match(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if file_filter_pattern and not re.match(
if file_filter_pattern and not re.search(

re.match only matches at the beginning of the string, which I imagine was not the intent. re.search matches anywhere in the string. If it was the intent, clarify that in the documentation for this field, since it would be surprising to me.

@jtyoung84 jtyoung84 merged commit 6d989dd into main Sep 23, 2025
4 checks passed
@jtyoung84 jtyoung84 deleted the feat-29-filename-regex branch September 23, 2025 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a job_setting param to allow users to filter which files to process.

3 participants