Agentic Video

Most AI video tools today are reactive. You give a prompt, you get a clip. Useful, but limited. Each output is an island.

Agentic video is different. The AI understands context, makes decisions, orchestrates multiple steps. You might say "find the best moments from this interview and cut a highlight reel" and it actually does it, watching the footage, making editorial choices, assembling the result.

It's the difference between asking someone to hand you a tool and asking them to help you build something.

There's a useful distinction going around: tools like Sora or Runway are black box. You prompt, you get a video. If the cat has three arms, you re-roll and hope. With programmatic video and an LLM that writes code, you get a glass box. You can see the instructions. You can change the colour yourself. That's why the describe-it, get-code, get-video pipeline matters to me.

So what? I'm betting on glass-box, code-first workflows. The bottleneck is shifting from execution to direction.

A few things have converged to make agentic video possible in the last year or two. None of them is the whole picture, but together they unlock something new.

• Vision models that understand video. New multimodal models can process long footage, generate timestamped analyses, and distinguish A-roll from B-roll or identify story beats. An editing agent has to watch and understand the material before it can cut it.
• Longer context and memory. LLMs can hold a full transcript or storyboard. The agent can remember what was said in minute 2 while working on minute 15.
• Agents that use tools and APIs. Function calling, code-driven pipelines like Remotion. The AI can actually perform edits, not just suggest them.
• Better generative models. Clip quality and consistency are improving. You can fill gaps with generated B-roll or synthetic shots when needed.
• Multimodal integration. Audio, visual, and text in one pipeline. Transcribe, trim filler, re-sync. Editing is inherently multi-modal.
• Hardware and software. WebGPU, WebCodecs, browser-based editing. Heavy tasks can run server-side while the agent orchestrates; we're getting to the point where an AI editor in the browser is feasible.

Motion design involves a lot of repetitive work: keyframing the same animation again, rendering variants, exporting formats. What if that could be delegated to a system that understands your style? The goal isn't to replace the craft. It's to spend less time pushing pixels and more time guiding the idea.

• Automating the grind. Most time in video goes into editing; an agent could speed that up.
• Lowering the barrier. Agentic tools give anyone a reasonable first cut on demand; you become more of a director.
• Quality and taste. Agents might learn what makes a video good, not just how to cut; we're not fully there yet.
• Scale and personalisation. Same promo in five aspect ratios and three languages; brand consistency at volume.
• New possibilities. Creative collaborator: feed hours of footage, ask for a storyline; sketch a concept, AI fills in details.
• Accessibility. Small teams and indies could produce polished video without a big budget.

It's still early, but we can already see capability buckets that agentic systems are tackling. Think of these as building blocks of a full pipeline.

• Content analysis and triage. Ingest footage, find best takes, summarize, annotate for search. Classify A-roll vs B-roll, detect when multiple angles cover the same scene. Some tools do this as a first step before any editing.
• Autonomous editing and orchestration. Rough cuts from intent. Multi-step coordination of models: generate a clip, insert it, add transitions. Context across the edit so the AI remembers earlier choices.
• Polishing and post-production. Color correction, audio cleanup, filler removal, consistency. One-click to fix an entire timeline's look, or strip "um" and "uh" from speech.
• Multi-format adaptation. Resize and reframe for platforms (16:9 to 9:16). Cut a longer video into shorter variants. Localisation: translate, new voiceover, adjust timings.
• Optimization and creative suggestion. Goal-driven edits (e.g. maximise retention). "Agents with taste" that suggest creative flourishes. Iterative refinement: you give high-level feedback, the agent executes and delivers a new version.