I was just thinking about this casually, but I wonder if the future of AI creation will be less about one giant model doing everything and more about specialized models working together.
For example:
- Model A: Understand an input video and describe the scene
- Model B: Apply motion or animation changes
- Model C: Generate voice or clone a character's voice
- Model D: Assemble everything into the final video
Something like a small AI production team.
Currently, we have many impressive models:
- Vision models that can understand images and videos
- Video generation and animation models
- TTS and voice models
- Image generation models
But the missing part seems to be the editor or the coordinator.
Humans do not create movies with one person doing everything. A director, animator, voice actor, and editor each have their own role. Maybe AI systems could work in a similar way.
The difficult part is not only generating each component, but making them cooperate:
- How does one model pass information to another?
- How do we keep character identity consistent?
- How do we synchronize voice, motion, and video timing?
- Who decides what needs to be changed?
Maybe instead of creating a single huge multimodal model, we could create a system where smaller specialized models communicate and complete tasks together.
I don't know how much research has been done in this direction, but it seems like an interesting area for open-source AI development.
This is already a fairly active research area, not just a hunch: there are several 2025-2026 papers doing exactly this (FilmAgent, VideoGen-of-Thought, CineAGI, StoryAgent, AniME), all decomposing video production into specialized agents (script/storyboard, character consistency, animation, TTS, assembly) coordinated by an orchestrator. Multi-agent frameworks including FilmAgent, Kubrick, and VideoGen-of-Thought decompose production into specialized roles but fail to coordinate complex narrative, visual, and temporal constraints, so your instinct about “the missing part is the editor/coordinator” lines up with what these papers identify as the actual bottleneck too.
The character-identity and sync problems you listed are the two hardest parts in practice. One approach (CineAGI) tackles identity consistency through a decoupled character-centric pipeline using instance-level tracking, and reports meaningful gains in character and subject consistency over baselines. On the coordination side specifically, most early systems used simple waterfall pipelines (one agent’s output feeds the next) but these waterfall pipelines are susceptible to cascading errors, and newer work is moving toward a formal orchestrator that manages exploration/exploitation across the creative decisions instead of just chaining fixed steps. Worth searching arXiv for “agentic video generation” or “multi-agent video storytelling” if you want to dig into any of these, it’s a good sign this is being taken seriously as a research direction rather than just a hobbyist idea.