Expose a max upload size for stores

[mkeen]

StoreDrivers should expose a provider-relevant max upload size. These could be directly passed through statically from the relevant SDK.

This would prevent leaning on huge numbers or hard-coded values when the upload size is unknown.

[MarcusSorealheis]

@mkeen Thank you for the issue and the awesome contribution. It's a great idea. Hard-coded values and knobs are generally good/bad for different reasons. Is there any optimization opportunity you see by making the max upload size configurable?

[mkeen]

You raise a good point about tradeoffs between knobs/static values.

I looked into it and sadly the S3 SDK (the only API I looked at) for Rust (and all other languages for that matter) don't include static values for file/chunk upload limits that I could see. Adding a hard-coded one would be a bit odious as it would add a maintenance requirement, so without upstream support, my idea is probably dead.

Really wish aws_sdk_s3::Client or a similarly namespaced construct from the aws_sdk crate (as well as gcs_client, etc) exposed a value like that, because then we could simply use a trait to provide a reasonable default value and override with simple static use calls and pass-through to store callers.

We'd be able to avoid doing:

let fast_store_fut = self.fast_store.update(
            key.borrow(),
            fast_rx,
            match known_reader_stream_size {
                Some(size) => UploadSizeInfo::ExactSize(size),
                None => UploadSizeInfo::MaxSize(u64::MAX),
            },
        );

And instead have something like this:

let fast_store_fut = self.fast_store.update(
            key.borrow(),
            fast_rx,
            match known_reader_stream_size {
                Some(size) => UploadSizeInfo::ExactSize(size),
                None => self.slow_store.max_upload_size(key.borrow()),
            },
        );

I'm ok if you close this, but perhaps we could open issues with cloud providers asking for these exports. If they'd manage the values, we wouldn't have to maintain it past the initial implementation. Maybe wishful thinking though, and probably not even medium priority.

There's not a performance benefit at all either. But it would be a tighter implementation with something like that.