New ORT_TRT port?

[hooke007]

microsoft/onnxruntime#24456 was merged.
Would you plan to add the new ORT_TRT wrapper in mlrt to test its performance?

[WolframRhodium]

I have no interest in using the backend from ONNX Runtime.

The backend can be added without the wrapper code from ORT. A problem is that the tensorrt_lean backend removes the feature of building engines at runtime, so a vs-mlrt distribution may need to include pre-compiled engines (which is highly inflexible); another solution is to use TensorRT-Cloud, but this is not for public use currently.

Another problem is that the lean runtime may be actually slower than the default runtime, not faster

It may be slower than an engine built for the default runtime.

Therefore, there is no planned update in vs-mlrt for this feature. I can modify the existing C++ code to enable the use tensorrt_lean later.

[hooke007]

Therefore, there is no planned update in vs-mlrt for this feature. I can modify the existing C++ code to enable the use tensorrt_lean later.

Considering this new TRT binary is not public yet, some questions here ... We might need to upload the private TRT binary to our internal blob storage and let new CI to fetch from there.

It may be different with lean version as recorded in the nvidia docs. We can wait for more details when it was released.

[WolframRhodium]

It may be different with lean version as recorded in the nvidia docs. We can wait for more details when it was released.

The new TensorRT for RTX is going to have:

Much smaller footprint
Much faster model compile/load times.
Better usability in terms of use of cached models across multiple RTX GPUs.

Performance is not a target.

[hooke007]

https://developer.nvidia.com/tensorrt-rtx

Offering peak performance for PC AI workloads such as CNN, Transformers, Speech & Diffusion models

Perhaps it's a marketing term?