The New Video Stack: From Traditional to Agentic

There is growing confusion in the video and motion design world around terms like procedural, data driven, programmatic, agentic, and AI generative.

Part of the confusion is that these words do not all describe the same thing. Some describe how a video is built. Some describe what drives it. Some describe how the pipeline runs.

I have found it helpful to think about video systems in two phases: before AI, and after AI. Before the current wave of LLMs and diffusion models, creators were already using procedural, data driven, and programmatic systems. AI added agentic workflows and generative models, but it did not replace the older categories.

The difference is in where control lives and how the pipeline is assembled.

Five layers in the modern video stack

These are not sealed boxes. They are different layers of control that often coexist in the same pipeline.

Traditional

Built manually on a timeline, shot by shot and edit by edit.

Procedural

Built from rules and systems.

Data-driven

Built from templates that respond to external data.

Programmatic

Built or rendered through code rather than a traditional timeline.

AI generative

A model synthesizes new imagery or video from prompts or learned patterns.

Agentic

A system orchestrates the pipeline and makes decisions about what to generate, render, or assemble.

Summary

Why these categories overlap and why the distinction matters.

Traditional Video

Traditional video is the baseline most people already understand.

You shoot footage, place clips on a timeline, edit shots by hand, animate manually, and render a finished piece. Even when the work is highly skilled or complex, the output is usually crafted directly rather than generated from a reusable system.

Traditional video is therefore about manually authoring a specific piece of media.

Procedural Video

Procedural video or motion design is built from rules and systems rather than individual keyframes.

Instead of manually animating every object, the creator builds a structure that generates motion automatically. For example:

• A grid that distributes 200 shapes
• A falloff that controls which shapes move
• A value behaviour that drives animation across the system

Once the structure exists, the animation emerges from the rules.

Tools like Cavalry, Houdini, and parts of Blender operate heavily in this paradigm. After Effects too, but it was not designed that way from the beginning and much of it still relies on scripting or expressions.

The key idea is that the artist defines the system, not the final frames.

If you change the rules, the entire animation updates.

Procedural video is therefore about structure and behaviour.

If you want to learn more about Cavalry, one of the most interesting procedural motion tools out there, you can check out some of my tutorials or listen to an interview I did with Cavalry's founder and CTO, Ian Waters.

Data Driven Video

Data driven video connects visual systems to real data.

Instead of animating arbitrary shapes, the motion responds to inputs such as:

• business metrics
• color values
• text values
• API responses
• product data
• analytics streams

The animation becomes a visual interface for information.

Spotify Wrapped is a useful example. The format stays broadly consistent, but the rankings, text, and visuals change with the data.

This often happens in traditional tools like After Effects or Cavalry, where a template is already designed and data flows into it. It can also happen in code-based systems. The important distinction is that the visual system already exists. Data updates the output, but it does not invent the structure from scratch.

Data driven video is therefore about mapping information to visuals.

Check out the two resources below: a conversation with Arie Stavchansky, founder of Dataclay, and a Cavalry tutorial on visualising data from Google Sheets.

Programmatic Video

Programmatic video is built or rendered through code rather than a traditional timeline.

Instead of opening a composition and moving layers by hand, the creator defines scenes, timing, layouts, and rendering logic in code.

A common example is Remotion. It treats video like a software project. React components define scenes, props control variations, and renders can run locally or in the cloud.

Programmatic video can also be procedural or data driven. Code might describe a visual system. It might also pull in external data and render a new output for every record.

Programmatic video is therefore about authoring and rendering through code.

Listen

Jonny Burger on Remotion

The creator of Remotion on programmatic video with React, the Claude skills wave, and where video-as-code is heading.

AI Generative Video

AI generative video works very differently.

Instead of building systems or mapping data, the creator describes the desired result and a model generates the frames.

For example:

• text to video prompts
• image to video models
• diffusion video models

The model learns visual patterns from massive datasets and produces new frames that match the prompt.

In this paradigm, control moves away from explicit systems and toward probabilistic generation.

The creator specifies intent, but the model determines the exact output.

AI generative video is therefore about synthesis rather than structure.

FYI: These terms are not neat buckets. One project can belong to several at once. Something like Spotify Wrapped could use a data-driven template, a programmatic renderer, an agentic orchestration layer, or a mix of all three. The output alone does not tell you which one you are looking at.

Agentic Video

Agentic video sits one layer above programmatic and data driven systems.

It is not just about using code. It is about a system orchestrating the pipeline and making decisions about what to generate, render, or assemble.

LLMs are often part of this, but they are not required. An agentic system can be driven by language models, rules, or other orchestration logic. What makes it agentic is that the system is coordinating the work rather than simply executing a fixed render.

In this model, a system might:

• pull data from APIs
• choose scenes or assets
• call tools like Remotion, After Effects, or Runway
• render multiple variations
• assemble final outputs automatically

Here the creator designs the production workflow and decision-making layer rather than a single template or composition.

Agentic video is therefore about orchestration and autonomy across the video pipeline.

Why These Terms Overlap

These are not mutually exclusive categories.

A system can be:

• procedural and data driven
• data driven and programmatic
• programmatic and agentic
• agentic and AI generative
• all of the above at once

Something like Spotify Wrapped could be built in several different ways. From the outside, the output might look the same. The difference is in where control lives and how the pipeline is assembled.

Why This Distinction Matters

These terms matter because they tell you where the creative and technical control lives.

• Procedural systems give creators precise structural control.
• Data driven systems let visuals respond to information.
• Programmatic systems move authorship and rendering into code.
• AI generative systems produce new imagery from learned patterns.
• Agentic systems orchestrate the entire production pipeline.

These are not just categories. They are layers of control in the modern video stack. One layer defines structure, another connects data, another governs authoring and rendering, another generates imagery, and another coordinates the whole system.

The future of video production will likely combine all five, and begin to look increasingly agentic.