MUSE - Agentic 3D Scene Authoring

Abstract

Text-driven 3D scene generation is a promising technique for digital content creation, embodied AI simulation, and interactive design, yet practical workflows often require refining, extending, or correcting existing scenes while preserving non-target content. Existing methods can produce realistic and structurally plausible scenes, but they generally lack editability with requirement-level state tracking, so part-level failures often lead to full-scene regeneration or manual intervention. To tackle this challenge, we formulate controllable 3D scene authoring as incremental requirement satisfaction, unifying construction and editing. In this paper, we present MUSE, a memory-grounded multi-agent framework in which an Architect compiles instructions into structured requirements, a Sculptor executes local scene operations, and an Inspector verifies each step while updating Working, Scene, and Skill Memory.

To evaluate requirement-level controllability and preservation-aware editing, we introduce AuthorBench, offering 145 constrained construction cases and a 1,584-case preservation-aware editing pool paired with external structured checks. On full construction cases, MUSE improves All-Goal success from 37.9 to 80.7 and surface-constraint fulfillment from 35.0 to 92.6 over the strongest baseline. On a stratified 240-case editing test split, MUSE achieves 49.6 All-Goal success, 99.9 preservation rate, and only 0.6 unintended change rate. Beyond automated metrics, human evaluations on compared local-editing baselines support stronger alignment with user intent, and downstream navigation-proxy tests indicate stronger spatial stability. Combined with ablations validating our memory designs, these results establish MUSE as an effective framework for controllable 3D scene authoring.

Method

MUSE turns scene creation and editing into a closed-loop authoring process over explicit requirements and persistent memory.

MUSE pipeline overview — The Architect compiles instructions into a structured requirement program, the Sculptor performs focused local actions with protection-aware repair, and the Inspector verifies progress while consolidating Working and Scene Memory.

AuthorBench

AuthorBench evaluates requirement-level scene authoring across construction and preservation-aware editing, using structured external checks rather than internal model state.

145 construction cases

1,584 editing pool cases

240 fixed editing split

720 preservation checks

Results

MUSE improves completion while preserving non-target scene content and maintaining physical validity.

Construction

80.7%

All-Goal success, compared with 37.9% for the strongest one-shot baseline.

Surface Constraints

92.6%

Surface fulfillment, addressing a major weakness of prior layout methods.

Editing Locality

99.9%

Preservation rate with only 0.6% unintended change rate.

Complexity Robustness

96.6%

Construction retention from easy to dense complex cases.

Main Construction Results

Method	Obj.	Rel.	Surf.	Zone	Macro	All-Goal	OOB	Col.
LayoutGPT	98.4	83.8	35.0	79.0	74.1	37.9	1.4	2.3
Holodeck	83.3	57.6	21.5	30.9	48.3	14.5	0.0	0.0
LayoutVLM	98.0	69.7	22.1	59.3	62.3	26.9	0.0	0.0
ReSpace	71.9	36.4	8.6	32.1	37.3	11.0	0.4	0.5
SceneReVis	46.9	17.2	9.2	8.6	20.5	2.1	0.3	0.9
MUSE	99.3	93.9	92.6	82.7	92.1	80.7	0.0	0.1

Main Editing Results

Method	Obj.	Rel.	Surf.	Macro	All-Goal	PR	UCR	OOB	Col.
LayoutGPT-E	86.0	21.1	19.3	42.2	36.2	25.1	95.7	1.8	5.2
Edit-As-Act	30.6	14.1	5.0	16.6	5.4	84.9	17.8	0.0	0.0
SceneReVis	33.6	28.2	4.2	22.0	10.4	92.6	5.9	0.8	2.1
ReSpace-E	46.3	19.7	5.9	24.0	11.2	100.0	1.9	1.2	3.5
MUSE	76.0	47.9	83.2	69.0	49.6	99.9	0.6	0.1	0.3

Completion-locality trade-off plot — MUSE occupies the preferred region of higher fulfillment and lower unintended changes.

Qualitative Comparisons

MUSE produces organized layouts in construction and performs precise local edits without rewriting protected scene context.

Main qualitative construction and editing comparison — Main qualitative comparisons showing MUSE's ability to construct organized layouts and perform precise preservation-aware edits.

Additional construction examples — Additional construction comparisons under multi-object spatial and support constraints.

Additional preservation-aware editing examples — Additional preservation-aware editing cases with coupled replacement, movement, and insertion.

Citation

@misc{xu2026museagentic3dscene,
      title={MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction},
      author={Ruijie Xu and Xinnan Zhu and Jiayu Ying and Daoguo Dong and Yuzhou Ji and Xin Tan},
      year={2026},
      eprint={2606.14168},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.14168},
}