-
Notifications
You must be signed in to change notification settings - Fork 37
Description
Is your feature request related to a problem? Please describe.
I would like to be able to run a KGX transform where JSON Lines files are exported as a singular tar.gz file.
At this moment if run monarch-ingest and cat-merge to get some TSVs and then run a KGX transform command like
poetry run kgx transform --input-format tsv --input-compression tar.gz --output output/mokg-jsonl --output-format jsonl --output-compression tar.gz output/mokg.tar.gz
The "output-compression" parameter will end up ignored and the files will be outputted from this command will be two files; output/mokg-jsonl_nodes.jsonl.tar.gz and output/mokg-json_edges.jsonl.tar.gz.
Describe the solution you'd like
Allow the JSONL writer to be able to handle tar.gz as an output compression.
Describe alternatives you've considered
Make JSONL writer fail when passing in invalid compression parameter.
Additional context
Code pointing to TSV Sink Compression code right now.
Lines 138 to 149 in f5c8ab9
| if self.mode: | |
| archive_basename = f"{self.basename}.{archive_format[self.mode]}" | |
| archive_name = os.path.join( | |
| self.dirname if self.dirname else "", archive_basename | |
| ) | |
| with tarfile.open(name=archive_name, mode=self.mode) as tar: | |
| tar.add(self.nodes_file_name, arcname=self.nodes_file_basename) | |
| tar.add(self.edges_file_name, arcname=self.edges_file_basename) | |
| if os.path.isfile(self.nodes_file_name): | |
| os.remove(self.nodes_file_name) | |
| if os.path.isfile(self.edges_file_name): | |
| os.remove(self.edges_file_name) |
Code pointing to JSON Lines Sink Compression code right now.
Lines 49 to 55 in f5c8ab9
| if compression == "gz": | |
| nodes_filename += f".{compression}" | |
| edges_filename += f".{compression}" | |
| NFH = gzip.open(nodes_filename, "wb") | |
| self.NFH = jsonlines.Writer(NFH) | |
| EFH = gzip.open(edges_filename, "wb") | |
| self.EFH = jsonlines.Writer(EFH) |
Notice that JSONL doesn't support tar.gz compression format.
Lines 23 to 24 in f5c8ab9
| compression: Optional[str] | |
| The compression type (``gz``) |