EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing

Kaizhi Zheng1, Xiaotong Chen3, Xuehai He1, Jing Gu1, Linjie Li2, Zhengyuan Yang2, Kevin Lin2, Jianfeng Wang2, Lijuan Wang2, Xin Eric Wang1,
1University of California, Santa Cruz, 2Microsoft, 3University of Michigan, Ann Arbor
Paper arXiv Code

Figure 1. Editing Pipeline with EditRoom. EditRoom is a unified language-guided 3D scene layout editing framework that can automatically execute all layout editing types with natural language commands, which includes the command parameterizer for natural language comprehension and the scene editor for editing execution. Given a source scene and natural language commands, it can generate a coherent and appropriate target scene.

Abstract

Given the steep learning curve of professional 3D software and the timeconsuming process of managing large 3D assets, language-guided 3D scene editing has significant potential in fields such as virtual reality, augmented reality, and gaming. However, recent approaches to language-guided 3D scene editing either require manual interventions or focus only on appearance modifications without supporting comprehensive scene layout changes. In response, we propose EditRoom, a unified framework capable of executing a variety of layout edits through natural language commands, without requiring manual intervention. Specifically, EditRoom leverages Large Language Models (LLMs) for command planning and generates target scenes using a diffusion-based method, enabling six types of edits: rotate, translate, scale, replace, add, and remove. To address the lack of data for language-guided 3D scene editing, we have developed an automatic pipeline to augment existing 3D scene synthesis datasets and introduced EditRoom-DB, a large-scale dataset with 83k editing pairs, for training and evaluation. Our experiments demonstrate that our approach consistently outperforms other baselines across all metrics, indicating higher accuracy and coherence in language-guided scene layout editing.

Unified Scene Layout Editing

  • We leverage the pretrained multimodal large language model (GPT-4o) as command parameterizer and graph diffusion-based method as scene editor to create a unified scene layout editing pipeline.
  • Larget language model can convert natural language commands into breakdown commands by given source scene information.
  • SceneEditor take the breakdown commands and source scene as input, and it can first generate the abstract target scene graph, then it will estimate the accurate target scene layout.

Figure 2. Scene Editor aims to provide accurate, coherent editing results according to the given source scene and language commands. It consists of two graph transformer-based conditional diffusion models. One diffusion model generates semantic target scene graphs. Another diffusion model can estimate accurate poses and size information for each object inside the generated target scene graphs. All diffusion processes are conditioned on the source scene and breakdown command.

Qualitative Comparison

Qualitative examples from EditRoom and baselines on single- and multi-operation editing. From the comparisons, we can find the EditRoom can provide more accurate and coherent editing results than other baselines, and it can generalize to multi-operation editing tasks without training on such data.

Figure 3. Comparison with other baselines on single-operation editing.

Figure 4. Comparison with other baselines on multi-operation editing.

BibTeX

@inproceedings{
      zheng2025editroom,
      title={EditRoom: {LLM}-parameterized Graph Diffusion for Composable 3D Room Layout Editing},
      author={Kaizhi Zheng and Xiaotong Chen and Xuehai He and Jing Gu and Linjie Li and Zhengyuan Yang and Kevin Lin and Jianfeng Wang and Lijuan Wang and Xin Eric Wang},
      booktitle={The Thirteenth International Conference on Learning Representations},
      year={2025}
      }