Skip to content

Make JSON Lines sink support ".tar.gz" formatting. #505

@DnlRKorn

Description

@DnlRKorn

Is your feature request related to a problem? Please describe.
I would like to be able to run a KGX transform where JSON Lines files are exported as a singular tar.gz file.
At this moment if run monarch-ingest and cat-merge to get some TSVs and then run a KGX transform command like

poetry run kgx transform --input-format tsv --input-compression tar.gz --output output/mokg-jsonl --output-format jsonl --output-compression tar.gz output/mokg.tar.gz
The "output-compression" parameter will end up ignored and the files will be outputted from this command will be two files; output/mokg-jsonl_nodes.jsonl.tar.gz and output/mokg-json_edges.jsonl.tar.gz.

Describe the solution you'd like
Allow the JSONL writer to be able to handle tar.gz as an output compression.

Describe alternatives you've considered
Make JSONL writer fail when passing in invalid compression parameter.

Additional context
Code pointing to TSV Sink Compression code right now.

kgx/kgx/sink/tsv_sink.py

Lines 138 to 149 in f5c8ab9

if self.mode:
archive_basename = f"{self.basename}.{archive_format[self.mode]}"
archive_name = os.path.join(
self.dirname if self.dirname else "", archive_basename
)
with tarfile.open(name=archive_name, mode=self.mode) as tar:
tar.add(self.nodes_file_name, arcname=self.nodes_file_basename)
tar.add(self.edges_file_name, arcname=self.edges_file_basename)
if os.path.isfile(self.nodes_file_name):
os.remove(self.nodes_file_name)
if os.path.isfile(self.edges_file_name):
os.remove(self.edges_file_name)

Code pointing to JSON Lines Sink Compression code right now.

if compression == "gz":
nodes_filename += f".{compression}"
edges_filename += f".{compression}"
NFH = gzip.open(nodes_filename, "wb")
self.NFH = jsonlines.Writer(NFH)
EFH = gzip.open(edges_filename, "wb")
self.EFH = jsonlines.Writer(EFH)

Notice that JSONL doesn't support tar.gz compression format.

compression: Optional[str]
The compression type (``gz``)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions