How to handle authenticated URLs?

[subdigital]

Our CI is locally hosted and accessible by VPN, but still requires authentication. If I click to download a zip file in my browser it works (because I have a session cookie), but Tophat won't have this when it goes to download.

How can we utilize Tophat in this scenario?

[lfroms]

Great question. This definitely depends on your CI provider and where your artifacts are hosted.

You may have noticed that Tophat suggests installing the Google Cloud SDK. This is because Tophat supports unauthenticated Google Cloud Storage URLs under the assumption that you have logged in to the CLI using gcloud auth login. However, this actually isn't the recommended approach because it requires the Google Cloud SDK to be installed, and also assumes that your artifacts are hosted in Google Cloud Storage.

When you say your CI is hosted locally, does this mean the artifacts produced in CI are also hosted locally?

You could potentially use signed (authenticated) URLs (with appropriate expiration for security) and then provide Tophat with those signed URLs. Not sure if that's possible with your infrastructure, however.

[subdigital]

CI is Jenkins. I can publish artifacts from a build (so the .ipa or .app.zip for simulator). I'm sure I could do something with AWS S3 signed urls, however I was hoping there was something obvious I was missing.

Thanks for the reply. Looks like a really useful tool!

[lfroms]

Good to know! Something I've been wanting to do but haven't had the time to is to allow Tophat to persist a session by handing authentication challenges automatically and prompting for login if needed. That would simplify a lot of the setup process.

[subdigital]

Prompting to authenticate would be really slick. Right now it fails as it downloads an HTML file containing an error page, then tries to unzip this with ZIPFoundation. I think this is probably Jenkins' fault, maybe returning a 200 for this request, but rendering an error page 🙃. But it could perhaps be detected at runtime ZIPFoundation gives you an unrecognized format error.

[lfroms]

Artifact downloads are handled by URLSession. I haven't looked too deeply into this, but it looks like URLSessionTask has built-in support for handling authentication challenges. Or perhaps something that can be done with ASWebAuthenticationSession.

In any case, possibly something that can be detected when the response is returned and handle appropriately.

[jaredh]

A very flexible option might be to also support a general Exec source type, in addition to Direct Link and Tophat API. Tophat could arbitrarily execute it, maybe providing it some context like platform, build type, URL, with an expected destination path, and take action based on exit code. Consumers would then be free to write scripts integrating it into any sort of system or auth method, as long as there's a valid artifact or URL available at the end of it.

This would be particularly useful when using Bitrise APIs for resolving authenticated artifact URLs from various workflows.

[lfroms]

Wanted to provide a bit of an update here to mention that there are some big changes that will be shipping with Tophat 2 very soon that I believe should address this question. Tophat will have an extension mechanism. In the current iteration, extensions will be responsible for handling the download process (but not unpacking) of artifacts. Extensions can register artifact providers which take care of the download process. Out of the box, Tophat ships with a basic HTTP artifact provider (default behaviour for unauthenticated endpoints), a shell script artifact provider (allows you to write a shell script to fetch the artifact however you want), and a set of Bitrise providers for those who use Bitrise.

For custom integrations you'll now have the option to either use a shell script, or write your own Swift-based Tophat extension using the TophatKit SDK (which you can deploy in a private macOS app or ship directly to this repo if you want the world to have it).

This means that you can now handle the authentication of protected resources yourself.

There is some new API documentation in the README, worth a look for when the upcoming release ships!

[subdigital]

I have a working POC that uses an AWS lambda to sign requests to S3 where our builds are. It redirects to Tophat w/ the signed urls and works. All of this is behind Okta, so when you click on a link you're authenticated first.

In the new extension system, will this workflow change?

[lfroms]

That should still work; Tophat will still have the ability to handle plain HTTP URLs using the new /install/http route. With the new system you'll just have a few more options when it comes to handling authenticated resources since you can provide the full implementation of the download mechanism if desired.

[lfroms]

Tophat 2 is now available to download from the Releases page.

[rckim77]

Do HTTPS URLs work as well? I'm noticing that in URLReader some params in a signed URL are getting parsed incorrectly. For example, the following gets parsed incorrectly I think because of the use of & as a separator:

https://s3.us-east-1.amazonaws.com/artifacts/ios/example.app.zip?X-Amz-Algorithm%3DAWS4-HMAC-SHA256&X-Amz-Credential=...&X-Amz-Date=...&X-Amz-Expires=...&X-Amz-SignedHeaders=...

[lfroms]

Sorry, @rckim77, I missed this message. We continued in the issue that you opened.

Circling back here just to add that the AWS URL needs to be fully URL-encoded before passing it as a query param.