What is the future development direction of scene graph?

[ChocoWu]

Currently, the definition of a scene graph involves object nodes and the relationships between these nodes. What are the drawbacks of this approach? In the future, how might the modeling techniques and applications in downstream tasks evolve to address these challenges?

[ChocoWu]

Let's review the current work on defining scene graphs. In fact, scene graphs are not limited to the graph representation defined by Prof. Li Fei-Fei. They also exist in 3D scenes, such as in the concept of ConceptGraph. Additionally, a new concept called Scene Language has also been defined.

[yuchen2199]

Hi Shengqiong,

Thanks for organizing this repo. I’ve learned a lot. I’d like to expand the discussion by sharing some recent progress on scene graph applications in the AD/ITS community, which might be helpful.

In AD/ITS, some works are exploring explicit scene graph construction through spatial location rules (for example, using BEV rules to derive triplets like <ego-vehicle, isin, lane-3>, <ego-vehicle, to right of, pedestrian-1>, <ego-vehicle, very near, pedestrian-2>). Guided by these graphs, GNNs are used to model agent interactions and tackle AD tasks like collision prediction and self-driving decisions.

This differs from the "scene graph generation" pipeline in the CV community. These types of scene graphs represent space as a closed set, which provides more accurate spatial relationships, but also limits its representational capabilities.

Below are some papers for your reference:

Hktsg: A hierarchical knowledge-guided traffic scene graph representation learning framework for intelligent vehicles
roadscene2vec: A tool for extracting and embedding road scene-graphs
Scene-graph augmented data-driven risk assessment of autonomous vehicle decisions

Yuchen Zhou

[ChocoWu]

Thank you for joining the discussion! I’ve updated several papers in the article list and also added the Traffic Scene Graph dataset (TSG-451) to the dataset collection. Indeed, TSG differs from general image/video scene graphs in its focus—particularly in terms of relationships and entities.

I’m curious about your thoughts:

Do you think scene graphs will remain a relevant tool for autonomous driving (AD) and intelligent transportation systems (ITS), especially with the rise of large vision-language models (LVLMs)?

Looking forward to your reply!

[ChocoWu]

A scene graph is an abstract semantic representation built upon pixel images. Previous work in scene graph construction has focused on modeling objects and relationships within a scene as a finite set of categories. However, this approach inherently lacks fine-grained understanding of the scene. By "fine-grained" I mean a true recognition and understanding of the object and an accurate comprehension of the relationship between objects. For example, consider an irregular-shaped object, which can be classified as a "desk", while also understanding why this object is larger than another in spatial terms.

Current scene graphs still support a relatively coarse-grained understanding. In the future, scene graphs should enable understanding at any granularity. They should support coarse-grained, holistic understanding at one scale, while also being able to provide fine-grained, localized comprehension when the scene is zoomed in.

Furthermore, based on this coarse-grained scene understanding, it is challenging to reconstruct the original concrete scene information from the scene graph. This leads to a loss of information in existing scene graphs, which remains inadequate for more rigorous domains, such as robotics, where details like texture and precise shapes are critical.

Therefore, we envision future scene graphs capable of recovering the raw, original information. Such scene graphs will maintain the essence of abstraction, while also possessing the ability to restore concrete details. Additionally, they should support flexible granularity in understanding and exhibit the capacity for compositional generalization.

[yuchen2199]

Hi Shengqiong,

From my perspective, in the era of LVLMs, scene graphs remain a powerful tool. Especially in autonomous driving scenarios, which inherently have explicit topological structures (such as traffic lights -> lanes, etc.), these are naturally well-matched to the structure of scene graphs. I still believe that scene graphs serve as a useful representation or data structure. There have also been some recent works in the community that might be of interest to you:

[1] CVPR2025 T2SG: Traffic Topology Scene Graph for Topology Reasoning in Autonomous Driving
[2] ECCV2024 DriveLM: Driving with Graph Visual Question Answering

Moreover, beyond autonomous driving, I believe scene graphs would also be a natural representation or data structure in certain robotics domains (such as household robots, vision-language navigation, and drones). Additionally, I think the concept you mentioned of scene graphs with any granularity (or rather, scene graphs with infinite/flexible granularity or hierarchical granularity) will be an important direction for the future. This could further advance the practical application of scene graphs.

[yuchen2199]

Additionally, there is another work that proposes multi-layered scene graphs: [3] ICRA2024 Collaborative Dynamic 3D Scene Graphs for Automated Driving