Evaluating Agentic Coding for 4D Scene Generation in Blender
The greybox scenes in this video were made by GPT-6 Astra (max effort); the photorealistic footage was generated by Seedance 2.5 from those greyboxes.
Can a coding agent turn a script, a few photos, a first frame or real footage into an executable 4D scene that a director can use?
Each task hands the agent a brief and Blender. It writes and runs code, renders previews, inspects its own frames, and delivers a greybox scene: who stands where, who moves when, how every shot is framed and where it cuts. We check the exported scene fact by fact against hidden ground truth, and a VLM judge watches the rendered proxy.
Every task is labelled with the scene it is set in, by the place that defines it. Pick a scene to re-rank the agents; ▲/▼ shows where each agent does better or worse than its own average.
Two cases per scene type, each with all 11 agents: the input, every agent's proxy video, and its scene as an interactive 3D file. Picked as typical cases, where the agents rank much as they do on the whole leaderboard; not a random sample.
Check what the scene does, then whether it reads on screen.
Writes and runs Blender code; hands in the scene file and its proxy video.
Does the code run, does it render, does the scene open? (below)
Re-executed in a fresh sandbox; the scene state is dumped frame by frame.
The rule score checks the scene; the VLM score watches the video.
All files handed in, valid syntax, runs standalone, tagging toolkit untouched.
Every frame rendered; the video decodes with the specified resolution, fps, duration and codec.
The .blend reopens in Blender with the task's fps, resolution and frame range.
We re-judged every video with GPT-6.1 Sol and Qwen3.8 Omni Flash using the same questions, and compared all judges with human pairwise preferences. The judges differ in absolute level, but they rank the agents almost identically, and each tracks human judgement closely.
| Comparison | Spearman ρ | Pearson r |
|---|---|---|
| Gemini 4 vs GPT-6.1 Sol2,628 runs | 0.96 | 0.98 |
| Gemini 4 vs Qwen3.8 Omni Flash2,628 runs | 0.94 | 0.97 |
| GPT-6.1 Sol vs Qwen3.8 Omni Flash2,628 runs | 0.96 | 0.99 |
| Human vs Gemini 4242 pairs, one per task | 0.92 | 0.94 |
| Human vs GPT-6.1 Sol242 pairs, one per task | 0.92 | 0.91 |
| Human vs Qwen3.8 Omni Flash242 pairs, one per task | 0.90 | 0.93 |
Between judges: the 2,628 runs that all three judges scored (excluding the 14 failed runs and 20 that GPT-6.1 Sol declined under its content policy). For each judge we take every agent's mean VLM score over these runs, then correlate across the 11 agents. GPT-6.1 Sol and Qwen score about 14 points below Gemini 4 overall, so we compare rankings and linear agreement, not the raw scores.
Against humans: one pair of proxy videos from two different agents per task, 242 pairs in all; the annotator picked the one that is easier to understand (or a tie). Each agent's human win rate (ties count half) is correlated with each judge's mean score on the same videos.