GHOST - Geometric Human Out-of-sync Spatio Temporal reconstruction

Overview

This repository contains the code for GHOST, a novel approach that leverages state-of-the-art models, time warping algorithms and a transformer based geometric fusion in order to reconstruct human meshes from casual, unsynchronized and multi-view videos.

GHOST receives as input videos containing humans and employs SAM3 and SAM 3D Body to detect people and extract per-view SMPL-X parameters. Such human parameters are then used to temporally align videos and extract initial relative camera poses.

Such initial parameters are then processed by the fusion module, which aligns shape, pose and camera parameters into a unique world representation. Such a representation is obtained exclusively through geometric constraints and confidence masks.

Quick start

Installation

This code leverages several external repositories. For SAM3 and SAM 3D Body (make sure to download the DINOv3 version).

# From inside /ghost:
git clone https://github.com/facebookresearch/sam3.git
git clone https://github.com/facebookresearch/sam-3d-body.git
git clone git@github.com:facebookresearch/MHR.git
cd MHR
curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip
unzip assets.zip

git clone https://github.com/nghorbani/human_body_prior.git

Then make sure you have pixi installed, since the code uses pixi for environment management.

x86-64 (standard workstation / GPU node)

pixi install
pixi run setup-cuda
pixi run download-model  # download SAM3 weights
pixi run install-hbp

On a login node without a GPU driver (e.g. a SLURM cluster), prefix every command with CONDA_OVERRIDE_CUDA=12.6:

CONDA_OVERRIDE_CUDA=12.6 pixi install
CONDA_OVERRIDE_CUDA=12.6 pixi run setup-cuda
CONDA_OVERRIDE_CUDA=12.6 pixi run download-model
CONDA_OVERRIDE_CUDA=12.6 pixi run install-hbp

aarch64 (e.g. CSCS Alps cluster)

pymomentum, torchvision (CUDA ops), and decord have no aarch64 binary wheels and must be built from source. Two intermediate C++ libraries (Apache Arrow and the Rerun C++ SDK) must be compiled first as build dependencies for pymomentum.

Run the following sequence after pixi install and setup-cuda:

CONDA_OVERRIDE_CUDA=12.6 pixi run build-arrow        # ~5 min  — Arrow 18 static lib
CONDA_OVERRIDE_CUDA=12.6 pixi run build-rerun-sdk    # ~2 min  — Rerun C++ SDK
CONDA_OVERRIDE_CUDA=12.6 pixi run install-pymomentum # ~20 min — compiles ~550 C++ files
CONDA_OVERRIDE_CUDA=12.6 pixi run build-torchvision  # ~30 min — torchvision CUDA .so files
CONDA_OVERRIDE_CUDA=12.6 pixi run install-decord     # ~5 min  — video decoder from source
cd vggt-omega && pixi run python -m pip install -e . && cd ..

# DROID-SLAM CUDA extensions (lietorch, pytorch_scatter, droid_backends)
CONDA_PREFIX=.pixi/envs/default
PATH=$CONDA_PREFIX/nvvm/bin:$CONDA_PREFIX/bin:$PATH FORCE_CUDA=1 CUDA_HOME=$CONDA_PREFIX/targets/sbsa-linux \
    $CONDA_PREFIX/bin/python -m pip install --no-build-isolation DROID-SLAM/thirdparty/lietorch
PATH=$CONDA_PREFIX/nvvm/bin:$CONDA_PREFIX/bin:$PATH FORCE_CUDA=1 CUDA_HOME=$CONDA_PREFIX/targets/sbsa-linux \
    $CONDA_PREFIX/bin/python -m pip install --no-build-isolation DROID-SLAM/thirdparty/pytorch_scatter
cd DROID-SLAM && PATH=$CONDA_PREFIX/nvvm/bin:$CONDA_PREFIX/bin:$PATH FORCE_CUDA=1 CUDA_HOME=$CONDA_PREFIX/targets/sbsa-linux \
    $CONDA_PREFIX/bin/python -m pip install --no-build-isolation -e . && cd ..

Notes:

• build-arrow and build-rerun-sdk write to /tmp, which is wiped on reboot. Re-run them before install-pymomentum if the node reboots.
• pymomentum is installed from TommasoNanni/momentum@aarch64-no-rasterizer, a fork that gates the drjit-based rasterizer behind a cmake flag (drjit 1.3.x removed the API it uses).
• build-torchvision and install-pymomentum internally call bash scripts (build_torchvision_aarch64.sh, install_pymomentum_aarch64.sh) to work around pixi's lack of for loop support in task definitions.
• import torch must appear before import pymomentum in any script (the conda env contains a CPU-only libc10.so that gets loaded first otherwise). This is already the case everywhere in the ghost pipeline.
• SLURM scripts use #!/bin/bash -l so they source ~/.bash_profile → ~/.bashrc, making pixi available on compute nodes without manual PATH setup.

This project uses Python 3.12 and PyTorch 2.7.1 with CUDA 12.8 support.

Moreover, make sure to have SMPLX and SMPL body models installed in ghost/body_models/. Download the SMPLX_NEUTRAL.pkl and SMPL_NEUTRAL.pkl body models from SMPL-X.

ghost/
├── main.py                     # End-to-end pipeline entry point
├── data/
│   ├── video_dataset.py        # Lazy video / scene dataset (EgoExoSceneDataset)
│   ├── segmentation.py         # PersonSegmenter (GDINO + SAM2)
│   └── parameters_extraction.py # BodyParameterEstimator (SAM3D Body)
├── synchronize_videos/
│   └── synchronizer.py         # Temporal alignment via weighted DTW
├── utilities/                  # Offline helper scripts
├── bash_jobs/                  # SLURM job scripts
├── test/                       # Unit and integration tests
├── sam-3d-body/                # SAM3D Body submodule
├── MHR/                        # MHR / SMPL conversion tools
├── checkpoints/                # Model weights (not tracked)
└── body_models/                # SMPL body model files (not tracked)

Repo modifications

Two key modifications have to be done to the external repositories:

1. In sam3/pyproject.toml, make sure to remove the numpy<2 dependency. This causes conflicts with our new versions of pytorch and doesn't cause issues in SAM3 usage
2. SAM 3D Body doesn't have a pyproject.toml file. We created a minimal one that contains the dependencies needed to install it in our repo. Run:

   cd sam-3d-body
   cat <<EOF > pyproject.toml
   [build-system]
   requires = ["hatchling"]
   build-backend = "hatchling.build"
   
   [project]
   name = "sam-3d-body"
   version = "0.1.0"
   requires-python = ">=3.11"
   
   [tool.hatch.build.targets.wheel]
   packages = ["notebook", "sam_3d_body", "tools"]
   
   EOF

Dataset

This repo has been trained using the RICH dataset. To replicate training using rich, follow the instructions on their website for downloading the dataset. The dataset downloads images in .bmp format, taking approximately 30 Mb each. To overcome the problem of such an extreme space usage, we provide a script that converts the images to .jpg format.

To run it, go in bash_jobs/convert_rich_bmp_to_jpeg.sh and change the --root argument to your rich directory. Then launch

bash_jobs/convert_rich_bmp_to_jpeg.sh

Notably, our pipeline resize images to approximately 1000 x 1000 resolution and saves them inside a /frames folder in the data directory. If higher resolution images are not needed, they can be deleted and this will save even more space.

Also make sure to go to configuration/config.py and update your data directories.

The expected dataset layouts are the ones that the scripts automatically download.

Usage

pixi run python main.py \
    --data_root /path/to/egoexo/takes \
    --output_dir /path/to/output \
    [--slice N]              # process only the first N scenes \
    [--detection_step 50]    # run GDINO every N frames \
    [--sam3d_step 1]         # run SAM3D every N frames \
    [--smooth]               # temporal smoothing for body params \
    [--vis]                  # save annotated segmentation videos \
    [--device cuda]

Output layout

output_dir/
    <scene_id>/
        <video_id>/
            frames/                  # extracted JPEGs (can be deleted)
            mask_data.npz            # compressed per-frame masks (uint16)
            json_data/               # per-frame instance metadata
            body_data/
                person_<id>.npz      # per-person body parameters
                body_params_summary.json
            segmentation.mp4         # (optional) visualisation video
        cross_video_id_mapping.json

Running on the cluster (SLURM)

A reference SLURM script is provided:

sbatch bash_jobs/rich_pipeline.sh

Logs are written to logs/<job_name>_<job_id>.{out,err}.

Key design decisions

• Multi-GPU parallelism: segmentation distributes videos across all available GPUs using torch.multiprocessing.Pool; each worker loads its own model instances.
• Incremental processing: already-segmented videos are skipped automatically (detected by the presence of mask_data.npz).
• Mask storage: per-frame .npy files are merged into a single .npz after segmentation (typically 20–50× compression).
• Synchronization: pairwise DTW offsets between all camera pairs are combined in a global least-squares solve, giving robust start times even with missing pairs.