Multimodal Judge AI & Quality Gates
Zero-Defect Autonomous Video Publishing on GitHub Actions
How we eliminate silent rendering defects—black frames, audio desync, illegible subtitle overlays, and hallucinated visual artifacts—across an autonomous 5-channel YouTube Shorts media fleet using Gemini Multimodal Judge AI, FFmpeg hardware filters, and dynamic sidechain audio ducking.
Gemini Vision Rubric
FFmpeg Hardware Gate
Dynamic Sidechain
100% GHA Serverless
Why Traditional Programmatic Video Pipelines Fail Silently
When you scale an automated video generation engine across multiple niche channels (dispatching 16+ high-retention vertical videos daily), human editorial review is mathematically impossible without defeating the economic advantage of automation.
However, naive unattended publishing is hazardous to channel algorithmic standing. Generative pipelines experience four catastrophic silent failure modes:
- Black Frame Dropouts: WebGL canvas or Remotion worker threads timing out mid-render, leaving 2–4 seconds of pitch-black video.
- Audio/Video Phase Desync: Background music and TTS neural voiceovers drifting due to variable sample rate mismatch (44.1kHz vs 48kHz).
- Subtitle Occlusion: Subtitles rendering outside YouTube Shorts safe-zones, covered by platform UI elements (like buttons, comments, channel tags).
- AI Visual Hallucinations: Midjourney or SDXL generating anatomical deformities or corrupted visual frames that degrade viewer trust.
The Three-Tier Verification Engine: Code & Vision
We implemented a cascading validation pipeline that evaluates rendered MP4 files prior to invoking the YouTube Data API v3 or Instagram Graph API:
TIER 1: FFMPEG HARDWARE DETECTORS (SUB-SECOND PASS)
We scan the compiled 1080p60 stream with native FFmpeg filtergraphs before consuming any AI tokens. If black frames or silent intervals exceed 500ms, the build is aborted immediately:
# 1. Detect any black sequences longer than 0.5s with 10% pixel threshold ffmpeg -i output.mp4 -vf "blackdetect=d=0.5:pix_th=0.10" -an -f null - 2>&1 | grep black_start # 2. Detect silent audio dropouts below -45dB longer than 1.5s ffmpeg -i output.mp4 -af "silencedetect=noise=-45dB:d=1.5" -vn -f null - 2>&1 | grep silence_start
TIER 2: DYNAMIC SIDECHAIN AUDIO DUCKING (-18 dB)
Instead of static background music volume that muddles spoken words, we route the TTS voice track through an audio sidechain compressor. The music dynamically drops by -18dB when speaking occurs and smoothly ramps back up in 200ms:
ffmpeg -i voice.wav -i bgm.mp3 -filter_complex \ "[1:a]asplit[bgm1][bgm2]; \ [0:a][bgm1]sidechaincompress=threshold=0.08:ratio=4:attack=5:release=200[ducked_bgm]; \ [ducked_bgm][0:a]amix=inputs=2:weights=1 1" -c:a aac -b:a 192k mixed_audio.m4a
TIER 3: GEMINI MULTIMODAL JUDGE AI (≥ 85/100 THRESHOLD)
We extract 5 keyframes across the timeline (0%, 25%, 50%, 75%, 100%) and pass them along with the narration script to Gemini 1.5/2.0 Flash with a strict evaluation rubric:
- Visual Framing & Resolution (25 pts): Zero pixelation, correct 9:16 aspect ratio, subject properly centered.
- Text Legibility & Contrast (25 pts): Subtitle colors maintain >4.5:1 contrast against dynamic backgrounds.
- Script Relevance (25 pts): Visual scenes accurately depict the topic discussed in the narration.
- Platform Safe-Zone Adherence (25 pts): Crucial UI elements remain untouched by Shorts overlays.
Programmatic Re-Prompting When Scores Drop Below 85
If the Gemini Judge AI returns an aggregate score below 85, the automated publishing webhook is cancelled. The runner parses the JSON critique, flags the deficient timestamps, and automatically adjusts the Remotion layout properties (e.g., bumping font stroke shadow or swapping the SDXL background prompt) for an automated rebuild.
Related Autonomous AI & Video Pipelines
Need Autonomous Video Pipelines for Your Business?
We build dedicated Remotion, PyTorch, and multimodal AI pipelines that produce broadcast-grade social video at zero marginal cloud cost.