Toward Improving the Generation Quality of Autoregressive Slot VAEs

Patrick Emami, Pan He, Sanjay Ranka, Anand Rangarajan

Research output: Contribution to journalArticlepeer-review


Unconditional scene inference and generation are challenging to learn jointly with a single compositional model. Despite encouraging progress on models that extract object-centric representations (“slots”) from images, unconditional generation of scenes from slots has received less attention. This is primarily because learning the multiobject relations necessary to imagine coherent scenes is difficult. We hypothesize that most existing slot-based models have a limited ability to learn object correlations. We propose two improvements that strengthen object correlation learning. The first is to condition the slots on a global, scene-level variable that captures higher-order correlations between slots. Second, we address the fundamental lack of a canonical order for objects in images by proposing to learn a consistent order to use for the autoregressive generation of scene objects. Specifically, we train an autoregressive slot prior to sequentially generate scene objects following a learned order. Ordered slot inference entails first estimating a randomly ordered set of slots using existing approaches for extracting slots from images, then aligning those slots to ordered slots generated autoregressively with the slot prior. Our experiments across three multiobject environments demonstrate clear gains in unconditional scene generation quality. Detailed ablation studies are also provided that validate the two proposed improvements.
Original languageAmerican English
Pages (from-to)858-896
Number of pages39
JournalNeural Computation
Issue number5
StatePublished - 2024

NREL Publication Number

  • NREL/JA-2C00-90373


  • object correlation learning
  • scene generation
  • slot-based models


Dive into the research topics of 'Toward Improving the Generation Quality of Autoregressive Slot VAEs'. Together they form a unique fingerprint.

Cite this