Evaling Video Slop — Maor Bril, Character.ai
Video generation eval methods lag far behind generation quality, missing narrative coherence entirely.
“the hard part was never how to make video. The hard part was how do we generate good enough video and how do we judge if the video is good enough?”
A Character.ai engineer argues that while AI video generation (Kling, Sora, VEO) has become impressively capable, evaluation tooling is stuck in the text/image era — CLIP score and LPIPS measure per-frame quality and prompt alignment but cannot assess whether a video tells a coherent story. The gap means high-quality output still requires human judges, creating a bottleneck. This is a useful practitioner framing of the eval gap but is a niche engineering insight rather than a major industry signal.