"Are scene graphs still a good way to represent and understand scenes?"

[ChocoWu]

Scene graphs are a form of explicit scene representation. But with the rise of implicit scene representations, is this approach still effective? Which representation is more promising moving forward?

[ChocoWu]

Explicit Scene Representation is primarily a post-processing approach that mirrors how humans understand scenes, representing them in a way that reflects human behavioral patterns and reductions. A common example of this is the use of various scene graphs. The major advantage of this method is its high interpretability; once we understand the composition of a scene and the semantic relationships between its components, reasoning about the scene becomes much easier. However, a significant drawback of this approach is that it diverges from one of the core strengths of deep learning-based representations: their superior performance, despite being difficult to interpret.

On the other hand, Implicit Representation refers to latent representations learned through self-supervised or unsupervised methods, often leveraging deep learning networks. These representations, also known as neural scene representations, are more difficult to visualize and interpret directly. We can't easily verify whether the model truly understands the scene content, but can instead observe the model's behavior through tasks like predicting camera poses or tracking points within a given scene. For further insights, one can refer to Scene Representations.

Personally, I believe Implicit Representation holds greater potential as a key research direction in the future. From a human perspective, we tend to store information about the world in a more abstract and highly compressed manner, rather than representing everything as raw pixels or textual data, which are simply more visual and directly observable mediums. For example, our understanding of a "table" is abstract; when asked about it, we can convey that concept through various forms such as text, images, or 3D pixels, offering great flexibility in representation.

This remains an area that requires further exploration.

[ChocoWu]

Recently, I have explored several intriguing studies on concept learning, which raised an interesting question:

Is scene understanding fundamentally about acquiring and grounding the abstract concepts present in a scene?

Such concept acquisition may be key to enabling models to generalize like humans — for instance, understanding the semantic rationale behind why an object is identified as a desk, or why we describe the bowl as being on the table rather than in the table.

Some interesting work:

• The Neuro-Symbolic Concept Learner
• Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners
• Neuro-Symbolic Concepts

[ChocoWu]

I just thought of an issue: it seems that scene graphs are not well-suited for representing fine-grained content. While we can identify that there is a book in the scene, how do we represent the content and layout of the book?

[qiyueordinary]

Currently trained large models are not able to input implicit representations, and I think scene graphs are still a good way to help large models understand spatial relationships at this stage