DualMap Benchmark — semantic mapping for Line 2

SUMMARYWhat was measured

DualMap (Eku127/DualMap, RA-L 2025, Apache-2.0) is the open-vocabulary semantic mapper selected for the speech-driven navigation line. This report benchmarks it end-to-end on a 6 GB laptop GPU: 5 numeric precisions × 9 scenes × 2 datasets, geometric accuracy against the Replica GT mesh, a face-level ground-truth audit of its semantic labels, and a head-to-head against VLFM's BLIP-2 scorer.

SETUPConfiguration

• Hardware: XPS 15 (i9 hybrid), RTX 4050 Laptop 6 GB, torch 2.5.1+cu121.
• Model stack: YOLO-World-L (naming) + FastSAM-s (segmentation) + MobileSAM + MobileCLIP-S2 (features), all local, no cloud APIs.
• Data: Replica ×8 scenes (nice-slam rendering, 200 keyframes each, stride 10) + TUM fr1_desk (real handheld RGB-D camera, 60 keyframes).
• Precision hook: a small patch exposing DUALMAP_PRECISION (fp32 | fp16 | bf16 | int8 | fp4); int8/fp4 = bitsandbytes quantization of the CLIP visual tower's Linear layers (MLP/projections — conv nets cannot be bnb-quantized; text tower stays fp16).

RESULT 1The semantic map itself

RESULT 2Precision sweep — fp16 wins outright

Why int8 is slower, not faster

Per-component timing over all 9 scenes: CLIP 0.118→0.126 s (+7%), full frame 0.492→0.519 s (+6%). bitsandbytes pays a quant/dequant transform on every forward pass; at DualMap's real workload (per-object crops, batch 1–10) that overhead exceeds the int8 arithmetic gain, while fp16 rides the tensor cores at full speed. int8's only win is model residency (0.43→0.34 GB). On Jetson the correct speed path is TensorRT (compile-time INT8 calibration), not runtime bnb.

int8 map quality is genuinely equivalent though: 382 vs 376 total objects across 9 scenes, and the only class flip in room0 was one object oscillating kettle ↔ coffee maker — semantic neighbors at the same location.

RESULT 3Generalization — 9/9 scenes, including a real camera

The class distributions track scene type — bedrooms yield nightstands and pillows, offices yield a dozen chairs and desks, and the real TUM desk sequence yields exactly what sits on that desk: keyboards, monitors, a mouse. The real-camera run (sensor noise, hand-held trajectory) was also the fastest and had one of the lowest unknown rates — the D455 integration risk is low.

RESULT 4Geometric accuracy — sub-centimeter

Every object point cloud checked against the Replica GT mesh (954k-vertex KDTree): sofa, table, vase, lamp, chair, pillows all land at median 0.6 cm / p90 0.8 cm from the true surface. For a navigation stack whose goal tolerance is 25 cm and whose robot is 70 cm long, the semantic map is the most accurate link in the chain — the error budget is dominated by SLAM drift and planner tolerance, not by DualMap.

RESULT 5Ground-truth label audit — the honest numbers

Using the face-level Replica semantic mesh (92 GT instances, via the vMAP data package), each DualMap object was assigned its true class by nearest-face vote. Two findings:

An earlier self-consistency estimate (84% top-1) overestimated relabeling quality: named objects are the easy samples YOLO already recognizes, while unknowns are hard by construction (small, occluded, background-contaminated crops — e.g. a ceiling lamp crop matching "beam"). The corrected claim: zero-cost relabeling recovers ~4 in 10 unknowns at top-1, ~6 in 10 with top-3 aliases.

A follow-up grading of the 14 "wrong" pairs shows the errors are overwhelmingly co-located confusions (vase→indoor-plant, ceiling→lamp, pillows→sofa — position-anchored grading means mistakes can only occur between things at the same spot): only one pair (coffee maker→candle) would misdirect a navigation query, putting navigation-equivalent accuracy near 96% against 69% strict. Crucially, all 19 unknowns sit at 0.01 m from their GT instance — the errors are purely in naming, never in position. With top-3 aliases stored in the grounding registry, a query for the right concept still resolves to the right place in most cases. Raising strict label accuracy further requires per-object captioning (Florence-2-class, +1 GB VRAM) — queued as a workstation/Orin upgrade, and the reason ConceptGraphs-style pipelines cost the VRAM they do.

RESULT 6Versus VLFM's scorer

• Frame-level scoring accuracy: a wash between the two models — but MobileCLIP is 18× faster (2 s vs 37 s for 500 scores).
• Object-level (map) queries beat frame-level scoring regardless of model — fused evidence localizes what single views cannot (the coffee-maker case).
• VLFM's full stack fits the 6 GB GPU (5.43/6.09 GB with grounding-dino-tiny) but cannot co-reside with DualMap. Its frontier value-map idea remains valuable for finding unmapped objects — future plan: run it powered by DualMap's already-loaded MobileCLIP instead of BLIP-2.

RESULT 7Voice-to-navigation demo — and what is real vs. oracle

The full spoken-language loop was assembled and exercised in simulation (MolmoSpaces procedural house, MuJoCo, the product Nav2/safety chain):

mic (5 s windows) → faster-whisper STT (zh+en, 2.3 s CPU) → /semantic_nav/query
  → grounding node (registry lookup + standoff) → /goal_pose
  → Nav2 MPPI → mux → stack_safety → robot

Honest architecture note: which layer is oracle

In this demo the semantic layer is an oracle: object positions come from a registry generated offline from scene ground truth — the robot never "saw" the dining table. The geometric layer is real: the occupancy map is built live from the simulated Mid-360, and Nav2 plans through unknown space and avoids obstacles it discovers en route. The demo therefore exercises stages 2–3 of the real pipeline:

• Stage 1 — mapping (replaced by oracle today): walk the robot around; DualMap builds the semantic object map from D455 RGB-D + FAST-LIVO2 pose. This whole report is the evidence that stage 1 works (62 objects, 0.6 cm, real-camera TUM run).
• Stage 2 — query (real): speech → grounding → object position → standoff goal.
• Stage 3 — navigation (real): live occupancy mapping + Nav2 through the safety gate.

Swapping the registry for live DualMap queries (runner_ros.py) makes stage 1 real; a target not yet mapped then returns unmatched — which is exactly where the VLFM-style frontier exploration mode (Result 6) takes over.

Compute verdict → offline demo required

At full LiDAR fidelity the XPS 15 cannot host the concurrent demo: the MolmoSpaces run's occupancy map produced zero messages in 150 s while odometry was ready at 6 s — the 1,411-geom house × 20k-ray CPU raycast starved the mapping pipeline. Follow-up fix: reducing the simulated Mid-360 to 5,000 rays/frame (the Demo5 precedent) restored a steady 10 Hz map at load 6/20 cores — the laptop can run the demo in this "lite" mode (slower Fast-LIO convergence, readiness ~140 s). The official demo still targets the lab workstation at full fidelity; checklist: repo + data/benchmarks/molmospaces (7.5 GB) + composed world + registry + faster-whisper-small (460 MB, pre-downloaded) + a USB microphone; all sim-only, no robot required.

RESULT 8SCAN-Planner vs Nav2 in the same house — a layer-by-layer autopsy

Control experiment: the same MolmoSpaces house, the same goals, both navigation backends. Nav2 MPPI reached the bed (SUCCEEDED); SCAN-Planner refused the dining-table goal outright (zero output, no log) and crawled 18 cm in 60 s toward an open-space goal. SCAN itself is healthy — in its home arena (demo4) the identical command moves the robot within 5 s. Sparse LiDAR is not the cause (18.5k healthy points/frame at 5k rays). The divergence is architectural:

Why demo4/demo5 never exposed this: those arenas contain no floating surfaces — walls, floors, ramps only — so z_down=0.4 was free safety margin there and is lethal in a furnished home. The fix direction is known (z_down 0.4→0.15, radius 0.25→0.20) but the real lesson is fit-for-terrain: flat furnished floors are Nav2's home turf; SCAN's 3D traversability earns its keep on stairs, ramps and multi-level scenes (Demo5). The voice demo therefore stays on the Nav2 backend.

RESULT 9DualMap on REAL robot data — the oracle gap closes

The Go2W's own recordings (pit_turn bag, 2026-08-10: Mid-360 + D455 RGB, 39 s) were pushed through the full pipeline offline: FAST-LIVO2 replay produced 459 poses + registered world cloud; since no depth stream was recorded, depth was synthesized by projecting the accumulated LiDAR cloud (317k pts) into each camera frame through the koide3 extrinsics — the projection lands on RGB contours (stairs, railing, mats) confirming the whole calibration chain. 457 synthetic RGB-D frames → DualMap dataset mode: 92 keyframes in 45 s, 6 map objects.

Honest notes: only 6 objects (a 39 s turning clip sees little; 4 are unknowns — lab clutter outside the class list), and this is offline replay, not live. But stage-1 "oracle" status is now demonstrably closable with data the robot already collects; the remaining step is the live runner_ros wiring. Next recording session should enable the D455 depth stream (one launch flag) to skip synthesis entirely.

NEXTDeployment recipe & open items

• Recipe: fp16 everywhere · scene-appropriate class list · offline relabel with top-3 aliases · structural-fragment filter. Orin speed path = TensorRT, not bnb.
• Wire runner_ros.py to the D455 + FAST-LIVO2 pose TF for online mapping; swap the grounding node's registry backend for live DualMap queries.
• Label-quality upgrade: per-object Florence-2 captions on the workstation.
• Full Table-II reproduction (mAcc/F-mIoU) now unblocked — the vMAP package provides the semantic GT locally.