Abstract (WIP)

Integrating 2D diffusion models with 3D Gaussian splatting for text-guided generation of individual 3D objects and editing Gaussian scenes are recently receiving increasing attention. However, current methods for single object generation normally require additional constraints, such as masks and normal maps, which limit their applicability in handling complex spaces. Furthermore, current techniques for text-guided 3D Gaussian editing typically rely on Iterative Dataset Update (IDU) methods based on the instruct nerf2nerf, which are also unsuitable for large spatial manipulations. In response to these challenges, we propose a new method, GaussianSpace, which enables effective text-guided editing of large space in 3D Gaussian Splatting. The key innovation of this method lies in its consideration of both RGB loss from the ground-true images and Score Distillation Sampling (SDS) loss based on the diffusion model during the iterative process. Additionally, we have introduced an automatic weighted loss technique to steadily descend the overall loss, ensuring that the edited Gaussian scene adapts to text instructions while retaining its original structural features. This marks the first successful implementation of text-guided 3D Gaussian large spatial manipulations.

Results (Preview)

Source of Original Space: 3D gaussain scene trained from 502 frames extracted from video taken in Landeszeughaus Graz Museum by iphone 14.
Prompt of Fantasy toys Museum: "anime painting of fantasy toys museum in the style of Studio Ghibli, surreal, maximalism, trending on artstation, 8K, HD, ultra details"
Prompt of Cyberpunk Arsenal: "An robotic arsenal of future weapons in the style of neon cyberpunk, cgsociety, trending on artstation, 8K, HD"
Prompt of Abandoned Biological Museum: "An abandoned futurism biological museum reclaimed by tropical rainforest with beautiful flowers, surreal, trending on artstation, 8K, HD, ultra details"

Optimization Progress

Optimization process （Video in 100X acceleration）: Resume from the trained 3D Gaussian scene of the original space (30000 iterations), and re-trained for 20000 iterations on a singel RTX 3090 GPU for one hour.

@article{Meng2024gaussianspace,
  title={GaussianSpace: Text guide 3D Gaussian Splatting Large Spatial Manipulations},
  author={Meng, Shengyu and Zhong, Ximing and Yu, Fujia},
  year={2024}
}