SAM3 (Sam3Image) with TensorRT EP on RTX 4090 still ~1.1s/image (600×500) via our thin usls-server wrapper — expected larger speedup

[likebean]

Summary
We are running usls::models::Sam3Image on an RTX 4090 with ONNXRuntime TensorRT EP enabled. TensorRT is clearly being used (engine serialization + DryRun ... on TensorRT:0), but end-to-end latency for a single 600×500 JPEG image is still ~1.1s after warm-up. This is only ~20% faster than CUDA EP on the same machine (~1.4s). We would like to confirm whether this performance is expected for SAM3 and what the recommended optimization path is.

Environment
OS: Ubuntu 24.04
GPU: NVIDIA GeForce RTX 4090 (24GB)
Driver: 580.126.09
CUDA toolkit: 13.0 (nvcc --version = 13.0.88)
TensorRT: 10.14.1.48 (10.14.1.48-1+cuda13.0)
ONNXRuntime: via ort-download-binaries (pyke dfbin cache)
Runner: our own thin Rust HTTP wrapper usls-server

What is usls-server (context)
This is our own small Rust axum service that:
accepts multipart form (image + JSON params)
decodes image bytes
calls usls::Runtime::forward(...)
returns COCO-style RLE masks grouped by class name
We will attach usls-server/src/main.rs and usls-server/Cargo.toml to this issue (for reference only).

Reproduction

1. Start service (TensorRT + CUDA image processor):
   export ORT_DYLIB_PATH=/root/ai-monitor/usls-server/target/releaseexport LD_LIBRARY_PATH=$ORT_DYLIB_PATH:$LD_LIBRARY_PATHUSLS_DEVICE=tensorrt:0 \USLS_PROCESSOR_DEVICE=cuda:0 \USLS_POOL_SIZE=1 \USLS_TEXT_CACHE_SIZE=4096 \USLS_DTYPE=fp32 ./target/release/usls-server
   On first start we see logs like:
   Initial model serialization with TensorRT may require a wait...
   DryRun ... on TensorRT:0(NVIDIA)
2. Measure latency (single image, 600×500 JPEG):
   curl -s -o /dev/null -w "usls_http=%{time_total}\n" \ -X POST http://127.0.0.1:8080/v1/sam3-image \ -F "image=@/root/001105.jpg" \ -F 'params={"prompts":["person","head","helmet"],"confs":0.5,"return":"rle","independent":false}'
   Observed (warm): usls_http ≈ 1.126s
   Notes:
   independent=false (batch prompts in one forward call)
   USLS_PROCESSOR_DEVICE=cpu:0 was slightly slower than cuda:0
   Same machine with CUDA EP was ~1.4s (so TensorRT is only ~20% faster)

Expected / Question
Is ~1.1s per 600×500 image expected for Sam3Image even with TensorRT EP on an RTX 4090? If not, what are the recommended steps/knobs to optimize latency?

main.rs.txt
Cargo.toml.txt

[jamjamjon]

@likebean Sorry for the late reply.

1. About end-to-end latency

First, could you clarify what end-to-end latency refers to in your measurement?

• If it is pure Sam3Image inference, it should only include the Runtime::forward() call.
• If it is HTTP end-to-end latency, please note that this also includes multipart parsing, JPEG decoding, post-processing (COCO RLE encoding), and response serialization.

TensorRT only accelerates execution inside the ONNX graph.

2. DType and TensorRT behavior

For TensorRT EP, please make sure:

• USLS_DTYPE=fp32 is used

usls will automatically handle FP32 → FP16 conversion internally when TensorRT is enabled.
Using FP32 input typically gives the best performance and compatibility with TensorRT.

3. Sam3Image TensorRT configuration

The default sam3_image configuration is intentionally general-purpose and supports both text and geometry prompts, with relatively wide dynamic shape ranges.

If your use case only involves text prompts, you can tighten the TensorRT optimization space:

.with_textual_encoder_batch_min_opt_max(1, 1, 1)  // text-encoder
.with_encoder_ixx(0, 1, (1, 1, 1))  // geometry-encoder

Default config

impl Config {
    /// SAM3 Image Predictor configuration (text + geometry prompts)
    pub fn sam3_image() -> Self {
        Self::sam3()
            .with_task(Task::Sam3Image)
            // Batch sizes: vision=1, text=4, geometry=8, decoder=1
            .with_visual_encoder_batch_min_opt_max(1, 1, 8)
            .with_textual_encoder_batch_min_opt_max(1, 4, 16)
            .with_encoder_batch_min_opt_max(1, 8, 16) // geometry encoder
            .with_decoder_batch_min_opt_max(1, 1, 8) // decoder (memory intensive)
            // Shape configurations
            .with_visual_encoder_ixx(0, 1, 3) // vision-encoder: channels
            .with_visual_encoder_ixx(0, 2, 1008) // vision-encoder: height
            .with_visual_encoder_ixx(0, 3, 1008) // vision-encoder: width
            .with_textual_encoder_ixx(0, 1, 32) // text-encoder: sequence length
            .with_encoder_ixx(0, 1, (1, 2, 8)) // geometry-encoder: num_boxes
            .with_encoder_ixx(1, 1, (1, 2, 8)) // geometry-encoder: num_boxes
            .with_decoder_ixx(4, 1, (1, 34, 60)) // decoder: prompt_len
            .with_decoder_ixx(5, 1, (1, 34, 60)) // decoder: prompt_len
            // Tokenizer configs
            .with_model_max_length(32) // CLIP max length, enables auto padding/truncation
            .with_tokenizer_file("sam3/tokenizer.json")
            .with_tokenizer_config_file("sam3/tokenizer_config.json")
            .with_special_tokens_map_file("sam3/special_tokens_map.json")
            .with_config_file("sam3/config.json")
            // Model files
            .with_visual_encoder_file("vision-encoder.onnx")
            .with_textual_encoder_file("text-encoder.onnx")
            .with_encoder_file("geometry-encoder.onnx")
            .with_decoder_file("decoder.onnx")
    }
}

⚠️ Note: Sam3Image fundamentally supports only one text prompt per inference.
While usls allows multiple text prompts for API consistency, internally this results in multiple inference passes.

4. Input resolution and expected performance

The exported Sam3Image model uses a default resolution of 1008×1008, and the vision encoder is expected to dominate inference time.

For this class of ViT / attention-heavy models, TensorRT typically provides a 10%–40% speedup over CUDA EP.
A ~20% improvement is therefore within the expected range, especially for batch size = 1 and end-to-end HTTP measurements.

To better isolate the model cost, I recommend:

• Running the sam3_image example directly in usls (file-based, non-HTTP)
• Comparing per-stage timing rather than HTTP latency

5. Reference comparison

As an additional reference point, you may want to test Sam3Image in transformers on the same GPU and compare:

• pure inference latency
• input resolution
• batch size

6. Input resolution and preprocessing behavior

Sam3Image requires a fixed input resolution of 1008 × 1008.

This means that regardless of whether the original image is 600×500 or any other size, the preprocessing step will always resize / pad the input to 1008×1008, and the vision encoder will run on that resolution.

You are already using the CUDA ImageProcessor, and in practice the GPU-side image preprocessing for a 1008×1008 input usually takes ~1 ms or less, depending on the GPU.
Therefore, preprocessing is not the primary performance bottleneck here.

7. About exporting non-1008×1008 Sam3Image models

It is theoretically possible to export a Sam3Image ONNX model with a non-1008×1008 input resolution.
However, this requires modifying the positional encoding logic in the model.

At the moment, the export scripts I provide only support 1008×1008, which matches the official configuration.
I may revisit non-1008 exports in the future, but for now you may want to check whether alternative pre-exported variants already exist.

This can help determine whether the observed performance is model-inherent or pipeline-related.

PS

Sam3Image has not yet undergone deep inference-level optimization.
Further performance improvements are planned in future updates.

[likebean]

Thanks for your reply. Following your suggestions, I optimized the code accordingly. This is the updated main.rs.

I added the following changes:

// Optimize: tighten vision-encoder batch (single image per inference)
.with_visual_encoder_batch_min_opt_max(1, 1, 1)
// Optimize: tighten text-encoder batch (assuming single prompt per inference)
.with_textual_encoder_batch_min_opt_max(1, 1, 1)
// Optimize: tighten geometry-encoder (if not using geometry prompts)
.with_encoder_ixx(0, 1, (1, 1, 1))
.with_encoder_ixx(1, 1, (1, 1, 1))

I also added logging to measure the inference time.

With the same environment, image, and prompt as before, the measured inference time is 568.4 ms. This is still quite slow and is far from the expected performance of ~30 FPS.

curl -X POST http://127.0.0.1:8080/v1/sam3-image
-F "image=@/root/001105.jpg"
-F 'params={"prompts":["person"],"confs":0.5,"return":"rle","independent":true}'

log:

026-01-25T13:31:16.890659Z INFO Pure inference time (independent, prompt='person'): 568.401877ms
2026-01-25T13:31:16.891195Z INFO Total pure inference time (independent mode, 1 prompts): 568.401877ms
2026-01-25T13:31:16.891202Z INFO === Performance Statistics (independent mode) ===

🚀 usls Performance Analysis Chart
═══════════════════════════════════════════════════════════════

🤖 Sam3Image │████████████████████████████████████████ 638.321ms (100.00%)
├─ 🔵 decoder │█████████████████████ 342.745ms
├─ 🟠 vision-encoder │████████ 106.922ms
├─ 🔵 vision-fpn-feats │████████ 68.338ms
├─ 🟣 image-processor │████████ 13.390ms
└─ 🔴 geometry-encoder │████████ 4.673µs
└────────────────────────────────────────
0 638.321ms

Using the same image, I tested it with “running the sam3_image example directly in usls” (file-based, non-HTTP). The results are shown below. However, I don’t understand why there is such a large difference between these two approaches.

(base) root@easyaikyrhgltb:~/usls# RUST_LOG=info stdbuf -oL -eL cargo run --release --features tensorrt-cuda-13000,vlm --example open-set-segmentation -- sam3-image --visual-encoder-dtype fp32 --visual-encoder-device tensorrt:0 --textual-encoder-dtype fp32 --textual-encoder-device tensorrt:0 --geo-encoder-dtype fp32 --geo-encoder-device tensorrt:0 --decoder-dtype fp32 --decoder-device tensorrt:0 --processor-device cpu:0 --visual-encoder-batch 1 --textual-encoder-batch 1 --geo-encoder-batch 1 --decoder-batch 1 --source /root/001105.jpg -p person 2>&1 | tee perf_test.log
Finished release profile [optimized] target(s) in 0.25s
Running target/release/examples/open-set-segmentation sam3-image --visual-encoder-dtype fp32 --visual-encoder-device 'tensorrt:0' --textual-encoder-dtype fp32 --textual-encoder-device 'tensorrt:0' --geo-encoder-dtype fp32 --geo-encoder-device 'tensorrt:0' --decoder-dtype fp32 --decoder-device 'tensorrt:0' --processor-device 'cpu:0' --visual-encoder-batch 1 --textual-encoder-batch 1 --geo-encoder-batch 1 --decoder-batch 1 --source /root/001105.jpg -p person
2026-01-25T21:37:35.265683109+08:00 INFO usls::ort::config: External data file is not enabled for model /root/.cache/usls/sam3/vision-encoder.onnx
2026-01-25T21:37:35.266104704+08:00 INFO usls::ort::config: External data file is not enabled for model /root/.cache/usls/sam3/text-encoder.onnx
2026-01-25T21:37:35.266559413+08:00 INFO usls::ort::config: External data file is not enabled for model /root/.cache/usls/sam3/decoder.onnx
2026-01-25T21:37:35.266941397+08:00 INFO usls::ort::config: External data file is not enabled for model /root/.cache/usls/sam3/geometry-encoder.onnx
2026-01-25T21:37:37.790624084+08:00 INFO engine_init{spec=sam3/vision-encoder}: usls::ort::engine: Initial model serialization with TensorRT may require a wait...
2026-01-25T21:37:49.099226324+08:00 INFO engine_init{spec=sam3/vision-encoder}: usls::ort::engine: Minimum Supported Ort Version: 1.23.x, Opset Version: 17, Device: TensorRT:0(NVIDIA), Parameters: 438.66 MiB, Memory: 1.71 GiB
2026-01-25T21:37:49.702824258+08:00 INFO engine_init{spec=sam3/text-encoder}: usls::ort::engine: Initial model serialization with TensorRT may require a wait...
2026-01-25T21:37:54.856744305+08:00 INFO engine_init{spec=sam3/text-encoder}: usls::ort::engine: Minimum Supported Ort Version: 1.23.x, Opset Version: 17, Device: TensorRT:0(NVIDIA), Parameters: 336.84 MiB, Memory: 1.32 GiB
2026-01-25T21:37:54.872700289+08:00 INFO engine_init{spec=sam3/geometry-encoder}: usls::ort::engine: Initial model serialization with TensorRT may require a wait...
2026-01-25T21:38:24.657920930+08:00 INFO engine_init{spec=sam3/geometry-encoder}: usls::ort::engine: Minimum Supported Ort Version: 1.23.x, Opset Version: 17, Device: TensorRT:0(NVIDIA), Parameters: 7.71 MiB, Memory: 30.84 MiB
2026-01-25T21:38:24.706788731+08:00 INFO engine_init{spec=sam3/decoder}: usls::ort::engine: Initial model serialization with TensorRT may require a wait...
2026-01-25T21:38:25.843473983+08:00 INFO engine_init{spec=sam3/decoder}: usls::ort::engine: Minimum Supported Ort Version: 1.23.x, Opset Version: 17, Device: TensorRT:0(NVIDIA), Parameters: 22.84 MiB, Memory: 91.33 MiB
2026-01-25T21:38:25.954512927+08:00 INFO usls::dataloader::r#impl: Found 1 tasks, total estimated items: 1 (images: 1, video frames: 0)
2026-01-25T21:38:26.185838450+08:00 INFO usls::dataloader::image: Saved image to: "/root/usls/runs/open-set-segmentation/sam3-image/20260125213826162263115.jpg"

🚀 usls Performance Analysis Chart
═══════════════════════════════════════════════════════════════

🔍 Data Loading │███ 9.268ms (3.22%)
🤖 Sam3Image │████████████████████████████████████████ 231.393ms (80.47%)
├─ 🟣 decoder │██████████████ 84.444ms
├─ 🟡 vision-encoder │██████████ 60.313ms
├─ 🔵 vision-fpn-feats │████████ 14.507ms
├─ 🟢 image-processor │████████ 6.863ms
├─ 🟤 text-encoder │████████ 4.944ms
└─ 🔵 geometry-encoder │████████ 8.194µs
🎨 Visualization │████████ 46.896ms (16.31%)
└────────────────────────────────────────
0 287.556ms

main.rs.txt

[jamjamjon]

@likebean I will optimize the performance of the sam3image code as soon as possible.

[likebean]

Thanks! If it’s convenient, could you also share the HTTP server code and usage instructions?
Looking forward to your optimization — hope it can reach around 30 FPS. Thanks!

[jamjamjon]

Thanks! If it’s convenient, could you also share the HTTP server code and usage instructions?
Looking forward to your optimization — hope it can reach around 30 FPS. Thanks!

Try the current main branch. I’ve simplified the sam3image model, but the input is still 1008×1008. Reducing the resolution causes a significant performance drop. You can test it on your GPU and see what FPS you can reach.

[likebean]

test result:
1、http server：

curl -X POST http://127.0.0.1:8080/v1/sam3-image
-F "image=@/root/001105.jpg"
-F 'params={"prompts":["person"],"confs":0.5,"return":"rle","independent":true}'

result：
2026-01-26T02:03:24.885609Z INFO Pure inference time (independent, prompt='person'): 139.134075ms
2026-01-26T02:03:24.886151Z INFO Total pure inference time (independent mode, 1 prompts): 139.134075ms
2026-01-26T02:03:24.886158Z INFO === Performance Statistics (independent mode) ===

🚀 usls Performance Analysis Chart
═══════════════════════════════════════════════════════════════

🤖 Sam3Image │████████████████████████████████████████ 275.124ms (100.00%)
├─ 🟢 encode-images │███████████████ 106.470ms
├─ 🟠 encode-images/vision-encoder │████████████ 83.022ms
├─ 🟡 decode-all-prompts │████████ 32.643ms
├─ ⚫️ decoder │████████ 30.564ms
├─ 🔧 encode-images/preprocess │████████ 20.355ms
├─ 📦 postprocess │████████ 2.065ms
├─ ⚪️ encode-text-prompts │████████ 4.745µs
└─ 🔴 encode-images/fpn │████████ 1.672µs
└────────────────────────────────────────
0 275.124ms

running the sam3_image example directly in usls

(base) root@easyaikyrhgltb:~/usls# RUST_LOG=info stdbuf -oL -eL cargo run --release --features tensorrt-cuda-13000,vlm --example open-set-segmentation -- sam3-image --visual-encoder-dtype fp32 --visual-encoder-device tensorrt:0 --textual-encoder-dtype fp32 --textual-encoder-device tensorrt:0 --decoder-dtype fp32 --decoder-device tensorrt:0 --processor-device cpu:0 --visual-encoder-batch 1 --textual-encoder-batch 1 --decoder-batch 1 --source /root/001105.jpg -p person 2>&1 | tee perf_test.log
Finished release profile [optimized] target(s) in 0.22s
Running target/release/examples/open-set-segmentation sam3-image --visual-encoder-dtype fp32 --visual-encoder-device 'tensorrt:0' --textual-encoder-dtype fp32 --textual-encoder-device 'tensorrt:0' --decoder-dtype fp32 --decoder-device 'tensorrt:0' --processor-device 'cpu:0' --visual-encoder-batch 1 --textual-encoder-batch 1 --decoder-batch 1 --source /root/001105.jpg -p person
2026-01-26T10:12:49.736592448+08:00 INFO usls::ort::config: External data file is not enabled for model /root/.cache/usls/sam3/vision-encoder.onnx
2026-01-26T10:12:49.736991049+08:00 INFO usls::ort::config: External data file is not enabled for model /root/.cache/usls/sam3/geo-encoder-mask-decoder.onnx
2026-01-26T10:12:49.737340403+08:00 INFO usls::ort::config: External data file is not enabled for model /root/.cache/usls/sam3/text-encoder.onnx
2026-01-26T10:12:49.939103061+08:00 INFO engine_init{spec=sam3/geo-encoder-mask-decoder}: usls::ort::engine: Initial model serialization with TensorRT may require a wait...
2026-01-26T10:12:52.317111130+08:00 INFO engine_init{spec=sam3/geo-encoder-mask-decoder}: usls::ort::engine: Minimum Supported Ort Version: 1.23.x, Opset Version: 17, Device: TensorRT:0(NVIDIA), Parameters: 30.54 MiB, Memory: 122.15 MiB
2026-01-26T10:12:54.623191775+08:00 INFO engine_init{spec=sam3/vision-encoder}: usls::ort::engine: Initial model serialization with TensorRT may require a wait...
2026-01-26T10:13:07.450385806+08:00 INFO engine_init{spec=sam3/vision-encoder}: usls::ort::engine: Minimum Supported Ort Version: 1.23.x, Opset Version: 17, Device: TensorRT:0(NVIDIA), Parameters: 438.66 MiB, Memory: 1.71 GiB
2026-01-26T10:13:08.085213906+08:00 INFO engine_init{spec=sam3/text-encoder}: usls::ort::engine: Initial model serialization with TensorRT may require a wait...
2026-01-26T10:13:14.721134798+08:00 INFO engine_init{spec=sam3/text-encoder}: usls::ort::engine: Minimum Supported Ort Version: 1.23.x, Opset Version: 17, Device: TensorRT:0(NVIDIA), Parameters: 336.84 MiB, Memory: 1.32 GiB
2026-01-26T10:13:14.827314160+08:00 INFO usls::dataloader::r#impl: Found 1 tasks, total estimated items: 1 (images: 1, video frames: 0)
2026-01-26T10:13:15.015417919+08:00 INFO open_set_segmentation: ys: [Y { Hbbs: [Hbb { xyxy: [72.265625, 45.18872, 427.73438, 682.333], id: 0, name: "person", confidence: 0.9543366 }], Masks: [Mask { dimensions: (500, 683), id: 0, name: "person", confidence: 0.9543366 }] }]
2026-01-26T10:13:15.061708384+08:00 INFO usls::dataloader::image: Saved image to: "/root/usls/runs/open-set-segmentation/sam3-image/20260126101315037953943.jpg"

🚀 usls Performance Analysis Chart
═══════════════════════════════════════════════════════════════

🔍 Data Loading │███ 9.195ms (2.22%)
🤖 Sam3Image │████████████████████████████████████████ 362.312ms (87.34%)
├─ 🔵 encode-images │████████████ 117.117ms
├─ 🟡 encode-images/vision-encoder │████████████ 109.851ms
├─ 🟢 decode-all-prompts │████████ 51.592ms
├─ ⚪️ decoder │████████ 49.384ms
├─ ⚪️ encode-text-prompts │████████ 12.563ms
├─ 🟠 textual-encoder │████████ 12.357ms
├─ 🔧 encode-images/preprocess │████████ 7.252ms
├─ 📦 postprocess │████████ 2.189ms
└─ 🔵 encode-images/fpn │████████ 6.585µs
🎨 Visualization │████ 43.335ms (10.45%)
└────────────────────────────────────────
0 414.843ms

[jamjamjon]

@likebean
The tests above are all based on single-image results. You should test with --source /path/to/image-folder, because warm-up is not performed by default. However, based on a simple analysis, it is almost impossible to reach 30 FPS on your GPU.

1. From encode-images/vision-encoder │████████████ 83.022ms, we can see that the Vision Encoder alone takes 83 ms. In addition, there are the text encoder and the geometry-encoder–mask-decoder. The text encoder caches the text prompt by default.
2. encode-images/preprocess │████████ 7.252ms corresponds to image preprocessing. Generally speaking, when the image processor runs on CUDA, it should take less than 1 ms.

Your machine may also have other tasks using the GPU and CPU. But even so, a single image inference on Sam3Image takes around 260 ms+, so the FPS should be only about ~4.

In addition, usls does not use IO binding to optimize host and device memory transfers. While this would require a significant amount of engineering effort, the performance gains would be negligible. You may consider using other open-set segmentation models, such as YOLOE-v8 / v11 / v26, which are much faster.

[likebean]

I want to choose a high-accuracy open-set segmentation model. Do you have any good recommendations?

SAM3’s accuracy is indeed quite good based on my previous tests. Are there any other good alternatives?

Do you think YOLOE-26 would be a good option?

[darkSuperman]

I've tried many models, including the latest YOLOE26, but their accuracy (confidence) is far inferior to SAM3. SAM3 is the best model I've seen for non-fixed object categories, but its model architecture makes it too computationally expensive. It seems the only way to optimize it currently is to distill a smaller model, like in this project (which is still under development): https://github.com/SimonZeng7108/efficientsam3. @jamjamjon If this project releases a model, will you integrate it into your system?

[jamjamjon]

@likebean @darkSuperman

YOLOE-26 indeed does not perform well in our tests. In usls, the open-set segmentation task can also be implemented through a pipeline of open-set detection + SAM1/2. For example, using YOLO / GroundingDINO / LLMDet together with SAM / MobileSAM / EdgeSAM, where the former provides box prompts and the latter is responsible for segmentation.

I will keep an eye on this repository and add it to usls once the weights are released in a future update.

[likebean]

@jamjamjon https://github.com/leon0514/trt-sam3/tree/main this project cost around 50ms.

[jamjamjon]

@jamjamjon https://github.com/leon0514/trt-sam3/tree/main this project cost around 50ms.

He applied PTQ quantization to the vision encoder, which is the most time-consuming module in SAM3Image inference. This will undoubtedly reduce inference latency significantly. In addition, for maximum performance optimization, you can use TensorRT + CUDA directly instead of ONNX Runtime with the TensorRT Execution Provider, as this relates to the capability boundaries of ONNX Runtime.