Research Log · Wednesday, October 28, 2026

Video becomes the robot's training data

Daily working notes on autonomous-vehicle AI and frontier AI research.

Top developments

Liquid AI ships LFM3: six open-weight 2B-class models on one shared backbone. Text, vision, audio, omni, and two decision variants all branch from a 1.7B base pretrained on 22T tokens, with a 256K context window and a text model that fits in about 1 GB at 4-bit precision. Liquid claims the 1.7B text model beats its own larger LFM2.5-2.6B with 35% fewer parameters and decodes 1.8 to 2.8x faster than the next model on its tested on-device setups, and the vision model adds video input while the decision models return calibrated answers in a single forward pass. Genuinely useful for on-device and edge agents where a single GPU fine-tune and a tiny footprint matter; the vendor benchmarks need independent confirmation, and the Jetson action-chunk claim (under 50 ms) is the one to test for robotics. (Liquid AI, MarkTechPost)

Anthropic's Managed Agents architecture, explained: separate the brain from the hands. In a Latent Space episode with Anthropic's engineering and product leads, the key design point is that the agent session and harness persist separately from the sandbox that executes code, so a failed sandbox does not kill the session, and sandboxes never receive service credentials (attached instead through a proxy with network controls). The same separation is credited with lower latency and better reliability versus putting the whole agent in a sandbox behind an API, and "dreaming" (memory) is implemented as one agent inspecting another's prior work. This is a genuinely useful mental model for anyone building agent runtimes, and it directly answers the recent sandbox-credential incidents: keep secrets out of the execution environment by construction, not by policy. (Latent Space)

Baidu is reportedly developing a world model, with autonomous driving as a possible application. A TechNode report citing STAR Market Daily (Oct 28) says Baidu will share research progress soon; no technical details, and no disclosed relationship to Apollo Go, which now operates in 28 cities. Treat as a watch item only: Baidu's fleet data makes it a credible entrant, but until there is a paper or demo there is nothing to evaluate, and this item remains an unverified report. (TechNode)

Frontier AI labs

Follow-up on Anthropic's S-1: the compute commitments are the story. Coverage of the Oct 26 filing now puts the non-cancellable cloud commitments at roughly $266B in total ($114.5B to Google through July 2033, $110B to Amazon through April 2036, $44.9B to Microsoft through May 2033), against $16.3B in H1 2026 revenue, plus a separate SpaceX compute ceiling raised to $120.5B that is mostly terminable on 90 days' notice. The distinction matters: the hyperscaler deals are a fixed-cost floor, the SpaceX deal is reserved optionality on scarce GPUs. Useful context for why inference efficiency and model cost curves dominate lab strategy right now. (Particle, a16z/ainvest analysis)

Google Cloud detailed its A5X instances, powered by NVIDIA's Vera Rubin platform. They scale from a single 72-GPU rack to 80,000 GPUs in one data center, linked by sixth-generation NVLink, aimed at frontier training, RL, physical AI agents, and real-time inference. Note the timing carefully: Google's own announcement puts general availability next month, not this week, so treat any "already available" framing as premature. Infrastructure news, but the scale claim and the explicit physical-AI positioning make it relevant to training economics for driving and robotics workloads. (Google Cloud Blog)

AV / embodied-AI research

RynnVLA-Latent scales VLA pretraining with 3D-grounded latent actions learned from video, not robot controls. The RynnLAM encoder learns continuous latent actions from visual transitions across 30.1K hours of human, robot, and simulation video (745.5M latent-action descriptors), using residual future-feature prediction, camera-compensated 3D flow, and camera-motion decoupling so the representation captures control-relevant motion rather than appearance change. The resulting 4B policy hits 98.8% on LIBERO, 84.7% on LIBERO-Plus without benchmark-specific adaptation, 51.9% across five VLABench tracks, and 88.6% on RoboTwin 2.0 bimanual, plus the best compared success on six real-world manipulation tasks. This is the most promising attack yet on the action-data bottleneck: if latent actions from ordinary video transfer this well, the data ceiling for manipulation policies moves substantially. (arXiv)

Video world models need predictable latents. Scardecchia and Yang show that training the autoencoder's latent geometry for temporal predictability (temporal objectives plus a past-conditioned decoder) gives up to 5x cheaper inference, over 7x faster training, and 15 to 24% lower pixel and perceptual errors out of distribution and over long horizons, at the same training budget, with world-model rollouts agreeing more closely with real policy evaluation. A concrete design rule for anyone building video world models for driving or manipulation: optimize the latent space for dynamics, not just reconstruction. (arXiv)

Zero-HLM: humanoid loco-manipulation with no real robot data. A real-to-sim-to-real pipeline turns smartphone captures plus simulated source demonstrations into 4,800 physically validated demonstrations, with Gaussian-guided agentic scene reconstruction and motion-environment co-adaptation from rollout feedback. Simulation-trained GR00T N1.7 reaches 66.7% mean full-task success versus 46.7% for real-data training, and expanded simulation training lifts pooled success on generalized configurations from 23% to 69%. Strong evidence for sim-first humanoid data engines, with the usual caveat that zero-shot transfer claims need the real-robot details checked. (arXiv)

Research / papers worth reading

RynnVLA-Latent (above): arXiv. Video World Models Need Predictable Latents (above): arXiv. Zero-HLM (above): arXiv.

Pretraining Knows What, Simulation Teaches How (Scholar Inbox digest, Oct 26 paper): compositional generalization in a sim-trained robot agent; pairs naturally with Zero-HLM as the sim-data-engine theme of the week. No citable source surfaced for this item in this pass beyond the Scholar Inbox digest, so it is left unlinked here.

Agents / tools / models

LFM3 (above) is the model release of the day for edge agents; 1Password for Agents is rolling out for Muse and Grok Bot, letting an agent sign into a site with user approval without seeing the password. The latter is a small but real piece of the credential-isolation story from the Anthropic episode: secrets should be brokered, not handed to the agent. (1Password)

Offline coding agents are practical for bounded tasks. A Latent Space guide runs OpenCode with Ollama and Qwen3.5 9B entirely on a laptop, keeping explain, refactor, and test-investigation workflows working with no network once models, dependencies, and tools are downloaded. Modest in scope and honest about it (no new packages, no web lookup), but a useful template for disconnected or air-gapped development. (Latent Space)

Other notable items

A quieter day overall. No new frontier model release beyond LFM3, no new AV deployment news, and no Bluesky posts from either tracked account in the window. The durable threads to keep watching are latent-action scaling for VLAs, predictable latents for world models, and the fixed-cost compute economics revealed by Anthropic's filing.