This one is still on the stove, and it is the most ambitious thing in the lab: physical and spatial agentic intelligence for 3D synthetic environments.
The short version: I am building a set of MCPs that train AI agents to learn synthetic 3D worlds the way a mind would, not as flat pixels but as objects with mass, motion, and consequences.
What that means in practice:
- Smart object detection inside a synthetic 3D scene - the agent identifies what things are, not just where they are.
- Accurate physics detection and calculation fused with that perception, so the agent reasons about collisions, trajectories, forces, and outcomes.
- A self-learning, automated loop - agents harness information from the environment and improve their understanding without a human hand-feeding every label.
The goal is an agent that can be dropped into a synthetic 3D world and genuinely understand it: what is there, how it moves, and what happens next. The use cases I am aiming at are orbital and spatial - environmental objects and physics detection where precision is everything and there is no room to guess.
Synthetic worlds are the perfect training ground. Build the environment, let the agents learn it, and the intelligence transfers to the real thing.
The stack
I am building this on top of the strongest tooling in the space:
- NVIDIA Isaac Lab - the simulation and reinforcement-learning framework for training agents in physically accurate worlds.
- USD Code MCP and kit-usd-agents - agentic control over OpenUSD scenes, so agents can read, reason about, and edit the 3D world programmatically.
- ovphysx - Omniverse PhysX for high-fidelity collision, rigid-body, and dynamics simulation.
- Epic Unreal Engine - real-time rendering and high-quality synthetic environment generation.
- Blender - asset creation, scene authoring, and procedural geometry.
- Adobe Substance Suite - physically based materials and textures that make synthetic worlds look and behave like the real one.
More as it comes out of the kitchen.