Deformable Instance MetalSplatter

This project is forked from an amazing work MetalSplatter. It provides support for running deformable splats as well as instance segmented splats. This is also a dedicated GUI for the paper TRASE adapted from its original GUI based on DearPyGui and CUDA.

Render deformable 3D Gaussian Splats using Metal on Apple platforms. Tested on :

• Iphone 15, Iphone 17 Air
• Ipad Pro (M1)
• Macbook Pro (M5)

Installation

Please follow the steps in original README to setup the project in xcode.

TODOS

• Feature : switch to newer mobileclip2 model
• Feature: Segmentation Threshold as a configurable setting

Usage

By selecting a folder in the startup page, the app loads the weights.bin, clusters.bin and point_cloud.ply inside the directory. You can download example scenes from here trained with TRASE.

There is a scroll bar for adjusting the time but you can also let it play by deactivating the manual time setting. You can toggle the additional dropdown to unlock additional options such as the TRASE based instance segmentation "Show Clusters" and depth based visualisation "Depth" (based on camera viewpoint). Clicking on clusters will isolate them in the visualisation, showing the whole scene again can be done via "Show all" button.

The gestures for X/Y Panning, Orbit, Zoom in/out are also implemented. As some scenes (such as sear-stake) may have a flipped coordinate system we add some buttons to change the coordinate axis in the dropdown.

Using your own Scenes

Step 1: Create env

conda create -n trase_model_converter python=3.10 -y
conda activate trase_model_converter
pip install torch "numpy<2" torchinfo plyfile

Step 2: For dynamic splat scenes

Output should be the base path of your input folder, where point_cloud.ply is stored.

Export the deform.pth via :

python export_deform_weights.py --model <path-to-deform.pth> --output <path to deform output.bin>

Step 3 TRASE clustering: Optionally for clusters (store in same path as point_cloud.ply just like for 1) :

Train your scene with TRASE and run the export script :

python export_clusters_bin.py --model <clusters.pt> --output <path to clusters output.bin>

Step 4 MobileCLIP querying: Optionally for cluster scenes (perform step 3 first)

Download the Mobileclip coreml models from our google drive link and put it in DeformableMetalSplatter/ (whole path is then DeformableMetalSplatter/coreml_models/)

You can then move to a certain view and run the Encode Clusters CLIP button to encode the features, after which the search bar can be used to query clusters with text. topk setting can be used to decide which amount of clusters can be queried.

Step 5 : Optional dynamic vs static splat speedup (perform step 3 first)

Generate the deformation mask, make sure params are named as such :

PLY_PATH="$SCENE_DIR/point_cloud.ply"
MODEL_PATH="$SCENE_DIR/deform.pth"
OUTPUT_PATH="$SCENE_DIR/mask.bin"

python generate_deformation_mask.py \"$PLY_PATH\" \"$MODEL_PATH\" \"$OUTPUT_PATH\" --smooth --threshold 0.1 --max-t 1.0 --num-samples 10

Demo

Dynamic Splats + Instances on iPhone 17 Air

Iphone_sear_stake.mp4

Dynamic Splats + Instances on iPad Pro (M1)

ipad_split_cookie.mp4

Dynamic Splats + Instances on iPhone 15

as_novel_view_iPhone15.mp4

MobileCLIP Encoding and Instance Querying on iPhone 17 Air

ScreenRecording_02-14-2026.1-03-50.PM_1.mov

MobileCLIP Encoding and Instance Querying on Macbook Pro M5

sear.steak.mov

painting.mov

Optional Static Splat Masking Speedup on Macbook Pro M5

-> notice the deformation network FPS speedup in the top right corner

My.Movie.mp4

Optional Static Splat Masking Speedup on iPhone 17 Air

-> notice the deformation network FPS speedup in the top right corner

ScreenRecording_02-20-2026.10-12-31.PM_1.mov

Acknowledgments

This project is a fork of MetalSplatter created by Sean Cier.

Original code is licensed under the MIT License (Copyright © 2023 Sean Cier).

Modifications and new features are licensed under MIT License (Copyright © 2026 Jim Li).

References

If you find this useful for your own work, particularly the 3D segmentation please consider taking a look at TRASE's code, paper, and website. Give us a star if you find it interesting :) Also feel free to open an issue or contribute to our project if you have an idea for a useful feature or developed on yourself :)

@article{li2024trase,
    title={TRASE: Tracking-free 4D Segmentation and Editing},
    author={Li, Yun-Jin and Gladkova, Mariia and Xia, Yan and Cremers, Daniel},
    journal={arXiv preprint arXiv:2411.19290},
    year={2024}
}

Contributors