Data Package Bundling (and maybe compression)

[sabas]

Updated: 2016-11-17

We want a way to "bundle" a data package into a single file for transmission. In addition it may be compressed at the same time.

Note also that individual resources can be compressed in themselves - see #290

Desired Features

• Widely supported in client systems
• Ability to access data within the bundle easily and without downloading the entire bundle (e.g. to stream resources from the bundle)

Original Description

As other packaging types use compression for distributing each package (JAR is a ZIP archive), there should be a section proposing a way to deal with compressed data packages.

[rufuspollock]

@sabas do you have a specific suggestion? I think you are right this is useful.

/cc @paulfitz

[sabas]

I was thinking a specification which would tell how to intepret a zipped package on the fly, in the same way a JAR is executed by Java.
So I could expect:

• the compression algorithm: gzip?
• which files are needed for correct decompression or reading on the fly (like the zcat and similar cli tools)
• how to compress the datapackage
• which file extension or MIME type to use

[rufuspollock]

@sabas i think this makes a lot of sense. Do you want to start speccing something out?

[sabas]

See #198

[rufuspollock]

There was a lot of discussion in the PR. The PR basically suggested tar + gzip. Subsequent discussion in the PR suggested reviewing existing best practice more and using zip. Main excerpts:

@mfenner wrote:

In the spirit of keeping things simple I wouldn't provide two options (.dp and .dpz). And in the spirit of not reinventing the wheel I would look at https://researchobject.github.io/specifications/bundle/, which uses Universal Container Format (UCF). Or for a software packacking example Chrome extensions: https://developer.chrome.com/extensions/packaging

Excerpt from Research Object bundle spec:

A UCF container is based on the ZIP compression file format [ZIP], enforcing additional restrictions. The most important restrictions are:

• Reserved filenames in the root directory: mimetype and META-INF
• Filenames must be encoded in UTF-8
• Compression must be Uncompressed or Flate
• may use Zip64 extensions, but should only do so when required
• The first file must be the uncompressed mimetype and without any extra attributes

UCF says about mimetype:

The first file in the Zip container must be a file with the ASCII name of mimetype, which holds the MIME type for the Zip container (application/epub+zip as an ASCII string; no padding, white-space, or case change).

@tfmorris wrote:

I'll second @mfenner 's suggestion to exhaust all possible existing alternatives before defining a new format. If you are forced to define something new, I'd strongly consider using zip instead of tar, since every other container format in the world from JAR to EPUB to Research Object Bundle has settled on it. There's an old overview of a bunch of the zip-based formats here: http://broadcast.oreilly.com/2009/01/packaging-formats-of-famous-ap.html

[rufuspollock]

@mfenner would you be interested in taking a bit of editorship here? You were a strong proponent of introducing this (and I'm +1 too). In addition, this should be very simple and short spec to write once we decide what to do.

[mfenner]

Let me think about how to approach this.

[rufuspollock]

@mfenner any further thoughts? /cc @danfowler

I am increasingly thinking that "bundling" a data package into one file (compressed) is an important use case and would love your suggestions here.

[mfenner]

@rgrp sorry for not following up on this. I want a standard zip compression, and hadn't found the time to spec out the details.

Bundling a data package into one file is an important use case for me.

[pwalsh]

For reference (although not directly related to a spec for compression) we went ahead and added zip support to the recently upgraded Python lib for DataPackage, based on very clear use cases in the CKAN integration, and, in general, that it is sensible and reasonable :). @vitorbaptista developed and led on that initiative.

For reference:

• https://github.com/okfn/datapackage-py/blob/master/datapackage/datapackage.py#L231
• https://github.com/okfn/datapackage-py/blob/master/datapackage/datapackage.py#L277

[rufuspollock]

@mfenner i imagine this can be super simple. Would you be able to start a draft and drop it in an issue here?

@vitorbaptista useful to get outline of what you did.

[vitorbaptista]

The requirements for my ZIP file loading were to be able to load both ZIPs that follow the pattern:

./datapackage.json
./data/resource.csv

and also

./my-datapackage/datapackage.json
./my-datapackage/data/resource.csv

This is because we wanted to support the ZIP files generated by GitHub (i.e. https://github.com/datasets/gdp/archive/master.zip), which have all contents inside a folder.

The actual code checks that the ZIP file has only and only one datapackage.json file and loads it. All paths in the datapackage are then relative to the datapackage.json, then. This allows any folder structure inside the ZIP file, as long as there is a single datapackage.json. It was easier to code this way 👍