Project
DimOS · Agent Evals · Pointcloud Encoding
Best node
exp_0012
Date
2026-08-09
Status
optimizing

Teaching the agent to read pointclouds

Same model, same 40 spatial-QA evals over frozen Memory2 recordings. The only change is PointCloud2.agent_encode() — designed by overnight autoresearch, 14 experiments, four anti-gaming gates.

01

Result

0.136
before · str()
0.957
after · exp_0012
0.15
blind control (gate ≤ 0.35)
02

Before / after

Input, before — one line

PointCloud2(frame_id='world',
            num_points=28955)
"How far is the nearest obstacle?" → "Cannot be determined."
"What is the horizontal extent?" → "unknown"
Every geometry family scored 0.000; sighted equalled blind.

Input, after — three layers, ~2.7 kB

"exact_stats": { x_range, y_range, z_range,
   horizontal_extent_m: 6.35, vertical_span_m: 1.6,
   occupied_floor_footprint_m2: 29.8 }   ← read, don't estimate

"centroid_xy_m": [-3.46, -3.81]            ← one point per frame ⇒ motion
"compass": "|dx| > 2.41·|dy| → east/west …"  ← trig-free rule

"body_height_occupancy": { cell_m: 0.25, rows: [
   "y=-2.05|-5.32:-4.62,-3.08:-2.58,-1.02:-0.93", … ],
   clearance: dx = max(0, min−qx, qx−max) }  ← exact meters
03

Is it 3D? The 2.5-D decision

The cloud is not flattened wholesale. Full 3-D information is kept as exact scalars (z_range, vertical_span — computed over every point), while spatial detail is spent where the robot's questions live: the body-height slab z ∈ [0.15, 1.0] m is projected into per-row x-intervals with exact world-meter endpoints. Height inside the slab is deliberately collapsed — that is the compression that buys sub-cell precision in x/y within a ~2.7 kB budget. A 20,000×3 float cloud becomes ~40 text rows the model can do arithmetic on.

3D pipeline: raw cloud with body-height slab, then encoded interval volumes
FIG. 01The transformation in 3-D (go2_bigoffice, t=80 s). Left: raw cloud; the tinted planes bound the body-height slab. Right: the encoding's interval rows drawn as occupied volumes — walls, desks and door gaps survive; ceiling/floor detail is carried by scalars instead of geometry. robot.
04

Sensor view vs agent view

go2_bigoffice top-down: raw cloud and encoded intervals
FIG. 02go2_bigoffice @ t=80 s — 28,955 pts (left, colored by height in the slab) vs the interval rows the model reads (right). ✕ centroid · robot (odom).
go2_short top-down: raw cloud and encoded intervals
FIG. 03go2_short @ t=40 s — accumulated single-room map, same encoding.
05

Per-family scores

World geometry (extent, z-span, area) · metric distance (nearest, from the robot's own pose) · spatiotemporal (shift, compass, area trend over sequences).

extentbbox size, m
0.00 → 1.00
zspanvertical span, m
0.00 → 1.00
areafloor footprint, m²
0.00 → 1.00
shiftmap-center motion, m
0.00 → 1.00
direction8-way compass
0.00 → 1.00
areatrendgrow / shrink / same
0.50 → 1.00
nearestobstacle clearance, m
0.38 → 0.81
06

Source — agent_encode(), as shipped in exp_0012

The full method plus its helper. Transformations: full-cloud stats (min/max/centroid, 0.2 m footprint cells) → body-height mask → adaptive row binning (0.25→3.2 m ladder) → per-row sorted x-runs split at gaps > cell → interval strings with selective @y refinement.

    def agent_encode(self) -> dict[str, object]:
        """Compact, spatially structured encoding for LLM consumption.

        World-frame meters throughout. Carries the cloud's horizontal centroid
        (so consecutive frames reveal motion of the mapped region) and a
        per-row interval map of the body-height slice with exact world-meter
        x endpoints (so clearance around a given world position is readable
        at sub-cell precision).
        """
        pts = self.points_f32()
        n = int(pts.shape[0])
        out: dict[str, object] = {"frame_id": self.frame_id, "num_points": n}
        if n == 0:
            return out
        xy = pts[:, :2]
        cx, cy = xy.mean(axis=0)
        out["centroid_xy_m"] = [round(float(cx), 2), round(float(cy), 2)]
        mins = pts.min(axis=0)
        maxs = pts.max(axis=0)
        # Distinct occupied 0.2 m x-y cells of THIS frame's cloud alone.
        floor_cells = np.unique(np.floor(xy / 0.2).astype(np.int64), axis=0)
        out["exact_stats"] = {
            "note": "exact full-cloud values in meters; for numeric extent/span/area "
            "answers use these, not the body-height interval map below",
            "x_range": [round(float(mins[0]), 2), round(float(maxs[0]), 2)],
            "y_range": [round(float(mins[1]), 2), round(float(maxs[1]), 2)],
            "z_range": [round(float(mins[2]), 2), round(float(maxs[2]), 2)],
            "horizontal_extent_m": round(float(max(maxs[0] - mins[0], maxs[1] - mins[1])), 2),
            "vertical_span_m": round(float(maxs[2] - mins[2]), 2),
            "occupied_floor_footprint_m2": round(float(floor_cells.shape[0]) * 0.2 * 0.2, 1),
            "footprint_note": "mapped floor area of this frame's cloud only (distinct "
            "occupied 0.2 m x-y cells); use it, not bbox area, for floor coverage. For "
            "area trends compare each frame's own footprint value across frames -- it "
            "can decrease as well as increase; do not accumulate coverage over frames",
        }
        out["compass"] = (
            "8-way direction of motion (dx,dy = last minus first): if |dx|>2.41*|dy| "
            "then east (dx>0) or west (dx<0); if |dy|>2.41*|dx| then north (dy>0) or "
            "south (dy<0); otherwise diagonal by signs (northeast, northwest, "
            "southeast, southwest). For any question about which direction the map "
            "moved or gained coverage, take dx,dy from centroid_xy_m of the last "
            "minus the first frame; range edges are too noisy for direction"
        )
        z = pts[:, 2]
        band = xy[(z >= 0.15) & (z <= 1.0)]
        grid = self._body_height_occupancy(band)
        if grid is not None:
            out["body_height_occupancy"] = grid
        return out

    @staticmethod
    def _body_height_occupancy(xy: np.ndarray, max_cells: int = 28) -> dict[str, object] | None:
        """Per-row occupied x-intervals of body-height points, world meters.

        ponytail: y binned into rows, x kept exact via interval endpoints;
        coarsens row height until the row count fits max_cells.
        """
        if xy.shape[0] == 0:
            return None
        lo = xy.min(axis=0)
        hi = xy.max(axis=0)
        span = float(max(hi[0] - lo[0], hi[1] - lo[1]))
        cell = next((c for c in (0.25, 0.4, 0.8, 1.6, 3.2) if span / c < max_cells), 6.4)
        iy = np.floor((xy[:, 1] - lo[1]) / cell).astype(int)
        rows = []
        for r in range(int(iy.max()), -1, -1):
            sel = xy[iy == r]
            yc = lo[1] + (r + 1 / 2) * cell
            label = f"y={yc:.2f}|"
            if sel.shape[0] == 0:
                rows.append(label)
                continue
            sel = sel[np.argsort(sel[:, 0])]
            rx = sel[:, 0]
            breaks = np.flatnonzero(np.diff(rx) > cell)
            starts = np.concatenate(([0], breaks + 1))
            ends = np.concatenate((breaks, [rx.size - 1]))
            parts = []
            for s, e in zip(starts, ends, strict=False):
                a, b = f"{rx[s]:.2f}", f"{rx[e]:.2f}"
                run = a if a == b else f"{a}:{b}"
                ym = float(sel[s : e + 1, 1].mean())
                if abs(ym - yc) > cell / 4:
                    run += f"@{ym:.2f}"
                parts.append(run)
            rows.append(label + ",".join(parts))
        return {
            "desc": "Occupied x-intervals at body height (z 0.15..1.0 m), world "
            "frame, one row per cell_m-tall band of y (label = band-center y, "
            "+y north, top=northmost). Interval min:max = exact x (m, +x east) "
            "of points in that band (a lone x = a thin obstacle); gaps wider "
            "than cell_m are free space. @y, when present, is the exact mean y "
            "of that interval's points. Horizontal clearance from a query "
            "point (qx,qy) = min over all intervals of hypot(dx,dy), where "
            "dx = max(0, min-qx, qx-max) (zero only when qx lies inside the "
            "interval), and dy = qy minus the interval's @y (or minus the row "
            "label y if no @y).",
            "cell_m": cell,
            "rows": rows,
        }
07

How it was found

exp_00000.136baseline — bare str(), point count only
exp_00010.622global scalar stats (nearest still 0)
exp_00020.487ASCII occupancy grid + world centroid
exp_00030.638+ compass legend, footprint count
exp_00040.741trig-free compass ratio rule
exp_00050.876exact_stats merge + signal routing
exp_00070.926centroid-delta steer — direction 9/9
exp_00060.793branch: interval rows, exact x in meters
exp_00080.823branch: selective @y + clearance recipe
exp_00120.957lanes merged + closed-form clearance — current best
08

Findings that generalize

Closed-form beats branchy. The model skipped an if/else clearance recipe; max(0, min−qx, qx−max) can't be short-circuited.
Decision procedures, not trig. "atan2, round to 45°" gave off-by-one sectors every time; a compare rule went 9/9.
Every signal needs routing. New fields silently broke direction answers until a note said which signal serves which question.
Exact coordinates beat grids. Meter-valued interval endpoints de-quantized extent and nearest; three ideators converged here independently.