Project HELIOS: Literature Review
Editing Algorithms, Computer Vision, Local Generative Models & Compiler Theory
Project HELIOS: Chapter 2 — Literature Review
2.1 Video & Film Editing Algorithms
Historically, automated video editing software relied on heuristic, rule-based algorithms. Early academic systems in the 1990s and 2000s utilized simple audio energy thresholds:
to identify quiet gaps in spoken tracks and apply automated jump cuts. While effective at removing dead air, these algorithms possessed zero awareness of visual narrative flow, facial pacing, or cinematic grammar.
With the advent of deep learning in the late 2010s, tools introduced transcript-aligned timeline pruning (e.g., matching Whisper speech-to-text outputs against timeline clips). However, existing systems fundamentally misunderstand cinematic language. Professional film editing relies on rhythmic pacing, subjective character reaction framing, emotional beats, and narrative tension—elements that cannot be captured by simple text transcript matching.
---
2.2 Computer Vision & Temporal Motion Tracking
The state of local computer vision in 2026 is defined by zero-shot foundation models capable of continuous spatial tracking across video sequences:
Meta's SAM 2.1 (Segment Anything Model 2.1)
Released as a definitive upgrade over SAM 1, SAM 2.1 introduced a streaming memory module that stores temporal feature embeddings across frames. This solved one of post-production's hardest problems: automated zero-shot rotoscoping without manual keyframing.
+-----------------------+ +-----------------------+ +-----------------------+
| Natural Language | ---> | Grounding DINO / | ---> | SAM 2.1 Streaming | ---> Pixel Alpha
| "Rotoscope subject" | | Florence-2 BBox | | Memory Tracking | Matte Mask
+-----------------------+ +-----------------------+ +-----------------------+
Open-Vocabulary Pre-Processors: Grounding DINO & Florence-2
Grounding DINO and Florence-2 act as textual-visual translators. When the HELIOS Orchestrator issues a command like "Find the main speaker in the blue jacket", Grounding DINO converts the query into bounding box coordinates, which are fed into SAM 2.1 for frame-accurate mask extraction.
---
2.3 Local Generative Video & Audio Models
The paradigm shifted dramatically in late 2025 and 2026 toward high-fidelity generative models capable of running locally on 12GB–24GB VRAM GPUs:
- Wan 2.2 & HunyuanVideo 1.5: Set the baseline for localized cinematic B-roll generation. Wan 14B utilizes aggressive offloading (
t5_cpu) to run on consumer GPUs while maintaining visual fidelity. - LTX-Video 2.3: A breakthrough 22B parameter Diffusion Transformer (DiT). Recognized as the first open-weights model capable of generating synchronized video and audio in a single pass, essential for autonomous B-roll generation.
- DeepFilterNet3 & CosyVoice 3.0: DeepFilterNet3 delivers sub-20ms real-time neural noise suppression, removing room echo and background hum. CosyVoice 3.0 provides zero-shot Voice Cloning (TTS) from 3-second audio samples for seamless narrative voiceover pickups.
---
2.4 Compiler Design & Operating System Theory
Applying compiler theory to multi-modal video compositing represents a major architectural leap. Existing 3D scene standards like Pixar's Universal Scene Description (USD) provide schemas for 3D asset exchange, but lack intent-driven compilation logic.
HELIOS bridges multi-modal LLM agent frameworks (AutoGPT, BabyAGI, LangChain) with classical compiler construction principles (Lexical Analysis $\rightarrow$ AST $\rightarrow$ EIR $\rightarrow$ Machine Code), establishing video editing as a formal code compilation problem.