RECAPBuilding 3D Semantic Scene Graphs
with Relational Evidence Capture and Pooling

A persistent memory of objects and their relationships, built from multi-view observations for language-based robot goals.

Compare videos Method GitHub Code coming soon

See the difference in 3D.

FROSS vs. RECAP (Ours) on Replica. Inspect the highlighted differences below each clip.

Enlarged FROSS bed and pillow region from the comparison video's first frame. — **Bed and pillows:** inspect fragmented geometry and incorrectly merged instances.
**Wall picture:** compare which objects have graph nodes.
**Relation links:** follow the light and window connections.

View fullscreen ↗On a phone, rotate to landscape for small labels.

Enlarged still: bed and pillow geometry

**FROSS**

**RECAP (Ours)**

Matching RECAP bed and pillow region from the same comparison frame. — **Bed and pillows:** inspect fragmented geometry and incorrectly merged instances.
**Wall picture:** compare which objects have graph nodes.
**Relation links:** follow the light and window connections.

View fullscreen ↗On a phone, rotate to landscape for small labels.

Enlarged still: bed and pillow geometry

**FROSS**

**RECAP (Ours)**

Enlarged FROSS chair and relation region from the comparison video's first frame. — **Chair instances:** compare the physical chairs with their graph nodes.
**Table–chair relations:** inspect missing links between nearby objects.
**Long-range arrows:** check the endpoints, not just the number of edges.

View fullscreen ↗Pause or use fullscreen to inspect individual relations.

Enlarged still: chair nodes and relation links

**FROSS**

**RECAP (Ours)**

Matching RECAP chair and relation region from the same comparison frame. — **Chair instances:** compare the physical chairs with their graph nodes.
**Table–chair relations:** inspect missing links between nearby objects.
**Long-range arrows:** check the endpoints, not just the number of edges.

View fullscreen ↗Pause or use fullscreen to inspect individual relations.

Enlarged still: chair nodes and relation links

**FROSS**

**RECAP (Ours)**

These clips compare the complete systems. See the L1 and L2 examples for the mechanisms, and controlled ablations for their individual contributions.

RECAP pipeline: offline supervision above five mapping and target-selection stages, from RGB keyframes to a persistent scene graph. — **RECAP overview.** Offline supervision adapts the compact observer and trains L1/L2. During mapping, observation and relation scoring share the same frozen compact-model weights. Select any figure to view it at full resolution.

01ObservePredict object names, boxes, and relation descriptions.
02Lift to 3DUse masks and geometry to locate observations in space.
03ConsolidateAssociate repeated observations with persistent entities.
04Fuse relationsWeight evidence by reliability and viewpoint diversity.
05Select a goalRetrieve an entity using its names and stored relations.

Read the full abstract

We present RECAP, a framework for constructing persistent 3D semantic scene graphs that support natural-language object queries and robot navigation. Frame-wise predictions can represent the same object as multiple nodes and produce inconsistent relations, while repeated viewpoints can reinforce these errors. To obtain open-vocabulary scene observations, we fine-tune a compact vision–language model using structured pseudo-labels generated offline by a larger model. After mapping observations into 3D, Entity Consolidation (L1) uses learned cross-frame associations to form persistent entities while retaining view-specific object names.

Relation Belief Fusion (L2) scores relations between these entities using the same frozen compact model, then aggregates multi-view predictions with learned reliability weights while discounting repeated viewpoints. Experiments on 3DSSG and zero-shot ReplicaSSG show higher relation recall than FROSS and DeWorldSG. With RGB-D inputs and ground-truth poses, the relative gains over DeWorldSG are 27.55% and 18.25% on the two datasets, respectively. Controlled ablations show reduced entity fragmentation and improved target selection for relational queries. Downstream evaluation examines query-selected goals through shortest-path endpoints on a ground-truth navigation mesh and physical-robot navigation through historical image goals in previously scanned environments.

Method and evidence

Keep identities. Accumulate relation evidence.

L1 associates repeated observations with persistent entities. L2 combines relation predictions at those shared endpoints.

L1 · Entity Consolidation

One object, across partial views.

L1 schematic: candidate generation, learned association, joint assignment, and persistent entities with retained name evidence.

Partial views produce different shapes and names for the same object. L1 combines geometric, semantic, temporal, quality, and relational-context evidence to associate them. Local surface continuity handles walls and floors; retained names support later language queries.

Compact-object fragmentation: 2.83 → 1.54.
A 45.6% reduction in the controlled 3DSSG L1 ablation, relative to FROSS-style association with shared observations and readout. See the full trade-off.

Replica comparison: FROSS retains a bed-corner fragment labeled pillow at the top; RECAP consolidates the bed into a persistent entity below. — **Why the bed fragments.** FROSS's same-class association leaves the pillow-labeled corner separate from the bed. RECAP combines spatial and semantic evidence to consolidate inconsistent observations into a persistent entity. FROSS above; RECAP (Ours) below.

L2 · Relation Belief Fusion

Recover missing links. Limit repeated errors.

L2 schematic: geometric pair selection, frozen relation scoring, reliability and view-density weighting, and weighted log-opinion pooling.

Geometry and joint visibility select ordered entity pairs, including pairs without an initial relation statement. The shared frozen VLM scores each pair; learned reliability weights and view-density discounting control how its predictions accumulate.

Replica relation comparison with missed and spurious relations highlighted: FROSS on the left and RECAP on the right. — **Inspect missing and spurious relations.** FROSS shows ceiling-light links and misses table–chair relations. Pair rescoring lets L2 recover omitted evidence; reliability weighting and view discounting limit poor and repeated observations. FROSS left; RECAP (Ours) right.

Relational target accuracy: 31.84% → 38.79%.
Majority vote versus full L2 on Replica queries, with full-L1 endpoints and per-view predictions fixed. See the controlled ablation.

Observation model and supervision

RECAP samples one in five RGB frames. Structured pseudo-label SFT adapts Qwen3-VL-2B-Instruct; SAM3 masks support 3D lifting. RGB geometry comes from VGGT-Ω with Depth Anything 3 scale alignment. L1 and L2 use 3DSSG instance and relation supervision, respectively. Object names remain open vocabulary; relation scoring uses a fixed task-specific codebook including NONE.

Experiments

From prediction quality to target selection.

Scene-graph benchmarks measure the resulting representation. Controlled ablations isolate observation quality, entity association, and relation aggregation.

+27.55%

3DSSG relation recall

Relative gain over DeWorldSG · RGB-D with GT poses

+18.25%

Zero-shot ReplicaSSG relation recall

Relative gain over DeWorldSG · RGB-D with GT poses

Overall scene-graph performance (%)
Method	Input	Relation recall	Object recall	Predicate recall	Object mRecall	Predicate mRecall
3DSSG
FROSS	RGB-D	27.90	62.40	33.00	63.80	18.00
DeWorldSG	RGB-D	50.20	75.00	57.30	70.60	36.00
RECAP	RGB	58.84	68.78	65.37	67.30	52.40
RECAP	RGB-D	64.03	74.42	70.60	75.20	56.50
ReplicaSSG · zero-shot
FROSS	RGB-D	22.30	26.10	27.80	28.80	20.40
DeWorldSG	RGB-D	38.20	35.40	45.30	38.10	27.30
RECAP	RGB	41.46	48.12	47.08	48.30	38.21
RECAP	RGB-D	45.17	51.42	50.08	52.29	43.88

mRecall: mean recall. RGB uses estimated depth and camera poses; RGB-D uses sensor depth and ground-truth poses. FROSS and DeWorldSG numbers are quoted from their papers under the same benchmark evaluation protocol. RECAP uses 37 3DSSG test scans and all 11 ReplicaSSG test scenes.

Structured observations · fewer empty outputs, higher relation precision

Structured SFT is compared with the base observer and non-empty prompting on the same images. Empty-output rates use answerable frames; precision measures image-supported emitted facts.

Compact VLM output quality (%)
Observation model	Object empty ↓	Relation empty ↓	Object precision ↑	Relation precision ↑
Base 2B	20.10	10.70	98.10	88.20
+ non-empty prompting	0.00	10.40	97.50	85.90
+ structured SFT	0.00	0.00	97.70	92.40

The audit contains 8,339 object-answerable images, including 8,256 relation-answerable images. All variants use the same format repair and single retry at temperature 0.6. Reported metrics use final outputs after this procedure.

L1 · persistent entities and correct-instance selection

All variants share observations, geometry, and max-prototype semantic readout. Fragmentation counts predicted entities per represented GT instance (ideal: 1). False merge measures parents containing multiple GT identities; correct instance@1 measures object-query target selection.

Entity Consolidation ablation
Entity memory	Compact frag. ↓	Surface frag. ↓	False merge (%) ↓	Correct instance@1 (%) ↑
Shared observations + FROSS-style association	2.83	13.17	3.11	59.23
L1 without surface continuity	1.50	12.33	4.59	65.74
Full L1	1.54	3.14	4.04	69.17

Memory metrics use the 37-scan 3DSSG RGB-D setting; target accuracy uses the 120 Replica object queries. Both report unweighted scene means.

L2 · reliable aggregation and relational target selection

Compared methods share full-L1 endpoints, the frozen scorer, per-view predictions, and geometry-only candidates. Presence F1 distinguishes a listed relation from NONE; relational target@1 counts correct subject selections over all relational queries.

Relation Belief Fusion ablation (%)
Edge aggregation	Predicate mRecall ↑	Presence F1 ↑	Relational target@1 ↑
Majority vote	45.15	62.10	31.84
Unweighted log pooling	50.70	64.58	32.86
+ reliability weighting	54.29	67.96	36.27
+ view-density discounting (full L2)	56.50	71.14	38.79

Relation-recognition metrics use 3DSSG; target accuracy uses the 265 Replica relational queries. Both report unweighted scene means.

Relational goal localization · shortest-path endpoint evaluation

Across 11 Replica test scenes, 265 relational queries yield 1,325 query–start trials. The predicted graph selects a target; a shared GT NavMesh supplies traversable space and ideal shortest paths. The endpoint is evaluated without action-level execution.

Relational goal localization (%)
Persistent memory	Endpoint SR@1m ↑
Legacy association + majority relations	23.01
L1 + majority relations	28.07
Full RECAP	33.65
Full RECAP + closed-vocabulary name projection	18.59

SR@1m uses Euclidean distance from the endpoint to the intended object’s GT 3D bounding box. Failed predictions, goal snapping, and paths remain in the denominator. Scores are averaged equally across scenes.

Physical-robot evaluation

From a language query
to a physical target.

8 indoor environments · 80 distinct queries · 160 attempted episodes

A teleoperated RGB scan builds the persistent scene memory and an ordered image map. MiniLM matches query phrases to retained entity names, with L2 disambiguating relational descriptions. The selected entity supplies a historical image goal for a frozen ViNT navigator. Depth observations provide obstacle protection during execution.

Query after the teleoperated scanFind the white chair next to the computer monitor.

Recorded execution · accelerated playback. After target selection, frozen ViNT follows the historical image goal. Depth observations provide obstacle protection.

Query
Selected entity
Historical image goal
Frozen ViNT

The selected white chair outlined in green as Target A; other chair instances are marked as distractors. — **Selected entity.** The query resolves to the chair marked Target A.

Historical image of the selected white chair beside the computer monitor, used as the navigation goal. — **Historical image goal.** A stored observation supplies the navigator's target image.

One physical query episode: query and candidate objects, the selected entity and execution path, and the robot endpoint measured 0.72 meters from the intended object. — **One query, from selection to arrival.** The three panels show the same episode. The endpoint is 0.72 m from the intended target by laser measurement.

70.00%

Correct-instance selection

112 / 160 attempted episodes

58.13%

Physical SR@1m

93 / 160 attempted episodes

54.38%

Correct instance + SR@1m

87 / 160 attempted episodes

Physical SR@1m requires no execution failure and a laser-measured distance of at most 1 m from the robot’s front-edge center to the intended object’s nearest measurable surface. Rejections, unavailable image goals, collisions, timeouts, and manual takeovers count as failures. Repeated episodes do not add distinct queries.

Code coming soon.

The official RECAP repository is hosted at 3DAgentWorld.

Visit the repository

RECAPBuilding 3D Semantic Scene Graphs
with Relational Evidence Capture and Pooling

See the difference in 3D.

Persistent entities across partial views

Relations supported across views

Keep identities. Accumulate relation evidence.

One object, across partial views.

Recover missing links. Limit repeated errors.

From prediction quality to target selection.

From a language query
to a physical target.

Code coming soon.

RECAPBuilding 3D Semantic Scene Graphswith Relational Evidence Capture and Pooling

See the difference in 3D.

Persistent entities across partial views

Relations supported across views

Keep identities. Accumulate relation evidence.

One object, across partial views.

Recover missing links. Limit repeated errors.

From prediction quality to target selection.

From a language queryto a physical target.

Code coming soon.

RECAPBuilding 3D Semantic Scene Graphs
with Relational Evidence Capture and Pooling

From a language query
to a physical target.