Same model, same 40 spatial-QA evals over frozen Memory2 recordings.
The only change is PointCloud2.agent_encode() — designed by overnight
autoresearch, 14 experiments, four anti-gaming gates.
PointCloud2(frame_id='world',
num_points=28955)
"exact_stats": { x_range, y_range, z_range, horizontal_extent_m: 6.35, vertical_span_m: 1.6, occupied_floor_footprint_m2: 29.8 } ← read, don't estimate "centroid_xy_m": [-3.46, -3.81] ← one point per frame ⇒ motion "compass": "|dx| > 2.41·|dy| → east/west …" ← trig-free rule "body_height_occupancy": { cell_m: 0.25, rows: [ "y=-2.05|-5.32:-4.62,-3.08:-2.58,-1.02:-0.93", … ], clearance: dx = max(0, min−qx, qx−max) } ← exact meters
The cloud is not flattened wholesale. Full 3-D information is kept as exact scalars (z_range, vertical_span — computed over every point), while spatial detail is spent where the robot's questions live: the body-height slab z ∈ [0.15, 1.0] m is projected into per-row x-intervals with exact world-meter endpoints. Height inside the slab is deliberately collapsed — that is the compression that buys sub-cell precision in x/y within a ~2.7 kB budget. A 20,000×3 float cloud becomes ~40 text rows the model can do arithmetic on.
World geometry (extent, z-span, area) · metric distance (nearest, from the robot's own pose) · spatiotemporal (shift, compass, area trend over sequences).
The full method plus its helper. Transformations: full-cloud stats (min/max/centroid, 0.2 m footprint cells) → body-height mask → adaptive row binning (0.25→3.2 m ladder) → per-row sorted x-runs split at gaps > cell → interval strings with selective @y refinement.
def agent_encode(self) -> dict[str, object]:
"""Compact, spatially structured encoding for LLM consumption.
World-frame meters throughout. Carries the cloud's horizontal centroid
(so consecutive frames reveal motion of the mapped region) and a
per-row interval map of the body-height slice with exact world-meter
x endpoints (so clearance around a given world position is readable
at sub-cell precision).
"""
pts = self.points_f32()
n = int(pts.shape[0])
out: dict[str, object] = {"frame_id": self.frame_id, "num_points": n}
if n == 0:
return out
xy = pts[:, :2]
cx, cy = xy.mean(axis=0)
out["centroid_xy_m"] = [round(float(cx), 2), round(float(cy), 2)]
mins = pts.min(axis=0)
maxs = pts.max(axis=0)
# Distinct occupied 0.2 m x-y cells of THIS frame's cloud alone.
floor_cells = np.unique(np.floor(xy / 0.2).astype(np.int64), axis=0)
out["exact_stats"] = {
"note": "exact full-cloud values in meters; for numeric extent/span/area "
"answers use these, not the body-height interval map below",
"x_range": [round(float(mins[0]), 2), round(float(maxs[0]), 2)],
"y_range": [round(float(mins[1]), 2), round(float(maxs[1]), 2)],
"z_range": [round(float(mins[2]), 2), round(float(maxs[2]), 2)],
"horizontal_extent_m": round(float(max(maxs[0] - mins[0], maxs[1] - mins[1])), 2),
"vertical_span_m": round(float(maxs[2] - mins[2]), 2),
"occupied_floor_footprint_m2": round(float(floor_cells.shape[0]) * 0.2 * 0.2, 1),
"footprint_note": "mapped floor area of this frame's cloud only (distinct "
"occupied 0.2 m x-y cells); use it, not bbox area, for floor coverage. For "
"area trends compare each frame's own footprint value across frames -- it "
"can decrease as well as increase; do not accumulate coverage over frames",
}
out["compass"] = (
"8-way direction of motion (dx,dy = last minus first): if |dx|>2.41*|dy| "
"then east (dx>0) or west (dx<0); if |dy|>2.41*|dx| then north (dy>0) or "
"south (dy<0); otherwise diagonal by signs (northeast, northwest, "
"southeast, southwest). For any question about which direction the map "
"moved or gained coverage, take dx,dy from centroid_xy_m of the last "
"minus the first frame; range edges are too noisy for direction"
)
z = pts[:, 2]
band = xy[(z >= 0.15) & (z <= 1.0)]
grid = self._body_height_occupancy(band)
if grid is not None:
out["body_height_occupancy"] = grid
return out
@staticmethod
def _body_height_occupancy(xy: np.ndarray, max_cells: int = 28) -> dict[str, object] | None:
"""Per-row occupied x-intervals of body-height points, world meters.
ponytail: y binned into rows, x kept exact via interval endpoints;
coarsens row height until the row count fits max_cells.
"""
if xy.shape[0] == 0:
return None
lo = xy.min(axis=0)
hi = xy.max(axis=0)
span = float(max(hi[0] - lo[0], hi[1] - lo[1]))
cell = next((c for c in (0.25, 0.4, 0.8, 1.6, 3.2) if span / c < max_cells), 6.4)
iy = np.floor((xy[:, 1] - lo[1]) / cell).astype(int)
rows = []
for r in range(int(iy.max()), -1, -1):
sel = xy[iy == r]
yc = lo[1] + (r + 1 / 2) * cell
label = f"y={yc:.2f}|"
if sel.shape[0] == 0:
rows.append(label)
continue
sel = sel[np.argsort(sel[:, 0])]
rx = sel[:, 0]
breaks = np.flatnonzero(np.diff(rx) > cell)
starts = np.concatenate(([0], breaks + 1))
ends = np.concatenate((breaks, [rx.size - 1]))
parts = []
for s, e in zip(starts, ends, strict=False):
a, b = f"{rx[s]:.2f}", f"{rx[e]:.2f}"
run = a if a == b else f"{a}:{b}"
ym = float(sel[s : e + 1, 1].mean())
if abs(ym - yc) > cell / 4:
run += f"@{ym:.2f}"
parts.append(run)
rows.append(label + ",".join(parts))
return {
"desc": "Occupied x-intervals at body height (z 0.15..1.0 m), world "
"frame, one row per cell_m-tall band of y (label = band-center y, "
"+y north, top=northmost). Interval min:max = exact x (m, +x east) "
"of points in that band (a lone x = a thin obstacle); gaps wider "
"than cell_m are free space. @y, when present, is the exact mean y "
"of that interval's points. Horizontal clearance from a query "
"point (qx,qy) = min over all intervals of hypot(dx,dy), where "
"dx = max(0, min-qx, qx-max) (zero only when qx lies inside the "
"interval), and dy = qy minus the interval's @y (or minus the row "
"label y if no @y).",
"cell_m": cell,
"rows": rows,
}| exp_0000 | 0.136 | baseline — bare str(), point count only |
| exp_0001 | 0.622 | global scalar stats (nearest still 0) |
| exp_0002 | 0.487 | ASCII occupancy grid + world centroid |
| exp_0003 | 0.638 | + compass legend, footprint count |
| exp_0004 | 0.741 | trig-free compass ratio rule |
| exp_0005 | 0.876 | exact_stats merge + signal routing |
| exp_0007 | 0.926 | centroid-delta steer — direction 9/9 |
| exp_0006 | 0.793 | branch: interval rows, exact x in meters |
| exp_0008 | 0.823 | branch: selective @y + clearance recipe |
| exp_0012 | 0.957 | lanes merged + closed-form clearance — current best |