2603.07604.md

EmbedTalk: Triplane-Free Talking Head Synthesis using Embedding-Driven Gaussian Deformation

Real-time talking head synthesis increasingly relies on deformable 3D Gaussian Splatting (3DGS) due to its low latency. Tri-planes are the standard choice for encoding Gaussians prior to deformation, since they provide a continuous domain with explicit spatial relationships. However, tri-plane representations are limited by grid resolution and approximation errors introduced by projecting 3D volumetric fields onto 2D subspaces. Recent work has shown the superiority of learnt embeddings for driving temporal deformations in 4D scene reconstruction. We introduce $\textbf{EmbedTalk}$, which shows how such embeddings can be leveraged for modelling speech deformations in talking head synthesis. Through comprehensive experiments, we show that EmbedTalk outperforms existing 3DGS-based methods in rendering quality, lip synchronisation, and motion consistency, while remaining competitive with state-of-the-art generative models. Moreover, replacing the tri-plane encoding with learnt embeddings enables significantly more compact models that achieve over 60 FPS on a mobile GPU (RTX 2060 6 GB). Our code will be placed in the public domain on acceptance.

由于延迟低，实时说话人头合成越来越多地依赖可形变的三维高斯喷溅。三平面通常被用作高斯形变前的编码方式，因为它提供了具有显式空间关系的连续表示域。然而，三平面表示会受到网格分辨率限制，并且将三维体场投影到二维子空间时会引入近似误差。近期研究已经表明，学习得到的嵌入在驱动四维场景重建中的时间形变方面具有优势。我们提出 EmbedTalk，展示了如何利用这类嵌入来建模说话人头合成中的语音驱动形变。通过全面实验，我们证明 EmbedTalk 在渲染质量、口型同步和运动一致性方面优于现有基于 3DGS 的方法，同时与最先进的生成模型相比也具有竞争力。此外，用学习嵌入替代三平面编码后，模型可显著压缩，并在移动 GPU，具体为 RTX 2060 6 GB 上实现超过 60 FPS 的运行速度。代码将在论文录用后公开。