FAPL Trains Video Models to Swap Characters and Actions Independently

On October 22, researchers from HKUST, Tencent Video, and École Polytechnique released a paper describing FAPL, a training method for generating film-like video. The goal is to control characters, scenes, actions, and camera work separately. The work also focuses on taking sparse inputs, something closer to a screenplay, and turning them into scenes that unfold coherently over time.

Can you change only the element you want?

FAPL trains on pairs of videos that differ in exactly one element. For each pair, the researchers specify in advance which of the four factors, character, scene, action, or camera, is being changed, and the model learns to alter only that element while leaving everything else untouched. The team used 25,000 pairs per element, 100,000 pairs in total. The underlying video generation model has 13 billion parameters.

The problem the researchers identify with existing approaches is that what the language model understands does not get used enough during video generation. A language model that handles both text and images carries knowledge that is useful for understanding a scene. Yet a video generation model, they explain, may lean more heavily on its own built-in text or reference-video input pathways than on that knowledge.

Early in training, FAPL uses only what the language model has distilled as the conditioning signal for the video. Next comes training in which per-element information is swapped around, teaching the model what to change and what to preserve. In the final stage, the video generation model is frozen while the language model and the connector between the two models are trained together. The point of this process is to make the system infer the actions and temporal flow of the target video without being told what to do shot by shot.

How much better are action ordering and face consistency?

In the researchers’ own measurements, action ordering and editing accuracy both improved. Compared with UniVideo, the action ordering score rose from 48% to 61%, and per-element editing accuracy went from 42% to 58%, gains of 13 and 16 percentage points respectively. Face consistency across multiple shots, meaning separate continuous takes assembled into a sequence, also rose from 0.451 to 0.523, the researchers reported.

These metrics look at whether actions follow in the right order, whether only the specified element actually changes, and whether the same face is maintained across shots. They are exactly the things a creator cares about when stitching scenes together or revising only part of a sequence. That said, the researchers explain that the ability to interpret complex camera movement is bound by the limits of the language model, while the ability to portray realistic acting is bound by the limits of the underlying video generation model.

When can you try it yourself?

The project repository lists training and inference code, trained model checkpoints, and the AI prompts used for evaluation all as coming soon. Anyone who wants to run the paper’s method and compare results firsthand will have to wait for that release.

FAPL feeds what the language model understands into video generation and trains the system to change only the elements you select. The improvements the researchers report span action ordering, editing accuracy, and face consistency. What remains to be seen is the release schedule for the code and models, and whether the same gains hold up in real production settings.