Enabling experimental triton onnx runtime in vespa today

[tronikelis2]

Hello,

We saw that you are supporting the triton service as an alternative to the embedded onnx runtime.

I understand it's not production ready, but nonetheless we would like to try it out sooner and see if it would solve our needs of gpu batching.

As there are no docs for it yet, could we get some guidelines on how to enable/configure the triton inference? 🙏

Thanks

[glebashnik]

Hello, Triton integration is not quite ready for production yet so no docs.

[tronikelis2]

Yeah, I understand, but is it truly not possible to enable it yet, just to test the difference between the embedded onnx runtime?

[glebashnik]

Currently it can only be enabled in Vespa Cloud with a feature flag.
We did some benchmarking to compare it with embedded ONNX runtime.
Even without batching it improves performance on GPU, more so with batching.
We will publish the results when it is production ready.

[tronikelis2]

Do you perhaps have an estimate when it will be available for self hosted users?

[kkraune]

Good question! We focus on the Vespa Cloud deployment first, as this lets us easily observe the feature and make improvements and fixes. This is a lot about configuration and orchestration. Then we will look at making this available for Vespa Enterprise but I am sorry I have no estimate for this yet.