Vivix Model Infra: Turning Video into a Real-Time Interactive Medium

The past year has been a remarkable one for video generation. Models now produce longer, more stable, and more controllable clips, while character consistency, cinematography, and multimodal understanding continue to improve rapidly. But the moment you shift the question from "generate a video" to "interact with a video in real time," the field hits a wall.

That wall lives at the systems layer.

Offline video generation can afford to wait tens of seconds, sometimes minutes; real-time interaction only has a sub-second window. Voice-only interaction transmits only a few kilobytes of audio per second; multimodal video moves megabytes per second—roughly three orders of magnitude more data. Data-center GPUs can readily power compelling demos, but at the scale of a large online service, cost, concurrency, and stability must all hold simultaneously before the system is truly production-ready.

Here's the hard part: making generative video interaction behave like a full-duplex, multimodal video call that continuously responds to speech, conversational timing, emotion, and context.

Here's Vivix Model Infra in a sentence:

A sub-second, fully streaming, highly cost-efficient real-time video generation stack—pushing consumer-facing real-time video inference into the same price band as mainstream digital entertainment.

Video generation is starting to move away from the "submit a job, wait for the result" model. It's becoming a medium you can converse with in real time.

1. Why real-time video generation is an infra problem, not just a model problem

Traditional video-generation systems are built on a simple set of assumptions: one prompt in, one complete inference run, and one video out. The pipeline is batch-oriented, compute is monolithic, and all user interaction occurs before inference begins.

Real-time video interaction overturns each of those assumptions. The system has to start responding while input is still arriving, start transmitting before the video is finished, and accept interruption before inference is done. The metric that matters is no longer "total generation time" but time to first reaction (TTFR)—how long before the user sees the video start to react.

That splits the work across three layers:

• The model has to stream — it can't wait for the whole context to land before it starts generating.
• The compute layer has to squeeze every cycle out of the hardware—from single-GPU kernels and multi-GPU communication down to memory management and launch overhead.
• The system layer has to rebuild the perception and reasoning modules → audio-visual generation → VAE → streaming chain, turning a serial pipeline into a real-time system that supports interruption, preemption, and overlap.

Real-time video generation isn't just a faster video model. It's about welding diffusion models, real-time media transport, inference engines, and distributed systems together into a single low-latency production line.

Vivix Model Infra uses a disaggregated encoder–decoder architecture, splitting the system into two subsystems: a hard-real-time decoder loop, called the Real-Time Decoder Loop, and a low-latency Encoder Service. They are deployed separately so that each can be sized for its own workload, improving overall utilization. Inside the Real-Time Decoder Loop, a natively streaming scheduler reduces response latency to user input to approximately 300 ms, giving us model-native interruption instead of a wrapper bolted on top. On the Encoder Service side, low-latency batching maintains high utilization while keeping latency under control, even under heavy concurrent load.

2. Single-GPU compute: getting everything out of the hardware

Every FLOP, memory access, and kernel launch is treated as a cost to minimize.

Low-precision GEMM. We have deployed low-precision matrix multiplication across all performance-critical operators, including the Q/K/V/O projections and FFN layers, ranging from FP8 and INT8 to first-generation NVFP4 GEMM support on NVIDIA Blackwell GPUs. We correct for quantization errors that accumulate during the video denoising process—without retraining—enabling FP4 inference with negligible quality degradation, so throughput goes up and memory footprint drops at the same time. For real-time video, that can determine whether the model fits on a consumer GPU at all.

Sparse attention. Alongside low-precision attention, we built a sparse attention variant purpose-built for video streams—we call it Vivix Sparse Attention (VSA). Video frames carry huge amounts of natural redundancy; allocating the same amount of compute to every frame and token wastes most of it on information the model already has. VSA concentrates compute on what actually affects the next generated output.

Memory and cache. KV cache prefetching, asynchronous transfer, near-lossless quantization, cross-session context sharing, memory offloading, and Ring Auto-Offload — which uses ring-based scheduling to move cold KV cache blocks to host memory while retaining only hot data on the GPU — let real-time video inference, long the exclusive turf of data-center GPUs, run on RTX-class consumer hardware.

CUDA Graphs and kernel fusion. Once real-time video generation enters its streaming phase, the bottleneck shifts away from raw compute and toward CPU-side kernel launches and fragmented scheduling. We rewrote most of the inference kernels and tightly fused GEMM with the surrounding quantization, LayerNorm, and gating operations—leaving essentially only GEMM and attention on the critical path. CUDA Graphs then consolidate hundreds of kernel launches per step into a single submission, so CPU scheduling falls off the critical path. On H100-class hardware and above, we go further by fusing communication and compute within the same kernel, pushing actual streaming throughput even closer to the hardware's theoretical limit.

In short, our optimization spans the entire stack, from individual operators and GEMM kernels to attention, KV cache management, memory movement, and kernel-launch overhead. Every layer has waste worth squeezing out.

3. Multi-GPU communication: co-designed with compute

Video-generation models are already computationally intensive, and real-time generation increases the demands further. Beyond a certain point, single-GPU optimization runs out of headroom, making multi-GPU parallelism unavoidable. The classic failure mode is that compute never reaches saturation because communication becomes the bottleneck first.

We treat communication as part of the compute graph. Parallelism strategy, wire format, and synchronization are all designed around end-to-end throughput.

We run hybrid sequence parallelism, mixing Ring, Ulysses, and other strategies depending on the hardware topology and the model's workload profile. Attention communication runs in FP8 out of the box, cutting the bytes transmitted in half with negligible precision loss. That matters most on consumer GPUs, where the RTX line has no NVLink and GPUs communicate over PCIe at a small fraction of data-center bandwidth. Without compressing the wire format, a multi-GPU setup quickly becomes bandwidth-bound.

The bigger lever is overlap. Attention communication, memory offloading, and disaggregated-deployment transfers are each assigned a dedicated CUDA stream, with synchronization explicitly managed so communication can overlap with compute wherever possible. From a distance, this looks like "multi-GPU parallelism"; up close, it's GPU, PCIe, and CPU scheduling all doing useful work at the same time, with no link in the chain sitting idle.

The final scorecard for a multi-GPU system is simple: does scaling still buy you near-linear returns?

4. Inference pipeline: take the shortest path

The most common failure in real-time interaction is a beautifully engineered pipeline whose end-to-end path is too long.

Every individual module can look fast on its own. But once perception and reasoning modules, audiovisual generation, VAE, and streaming are scattered across regions, services, and scheduling queues, system overhead quietly eats into the end-to-end latency budget. Users don't care about single-module benchmarks—they notice that the video is sluggish and the interaction doesn't feel connected.

So we optimize the architecture for the shortest possible end-to-end inference path.

We consolidated the distributed inference modules into a single inference cluster, cutting out cross-region and cross-service hops. We split perception and reasoning modules, DIT, and VAE along responsibility lines, keeping only what genuinely has to be synchronous on the critical path. To our knowledge, Vivix is the first to apply prefill–decode (PD) disaggregation to a real-time video-generation pipeline, together with a redesigned KV cache transport layer that allows intermediate state to be reused and streamed between stages instead of being recomputed.

In Vivix's production streaming model, this architecture keeps end-to-end TTFR below 1.7 seconds and video response latency—the time from user input to the first visible reaction—below 0.6 seconds. As a result, the video begins visibly reacting to user input much sooner.

The experience begins to feel genuinely live: users are no longer waiting for an offline job to finish; they are watching the video respond continuously to their input.

5. Production requires both capability and cost efficiency

The sections above cover specific optimizations across the model, multi-GPU communication, and the inference pipeline. This section focuses on what happens when these optimizations are combined. Long-term competition in real-time video generation will not be decided by any single operator or optimization. It comes down to whether these capabilities can be reliably integrated into a single production-grade online serving pipeline.

Cost is part of the moat. Vivix has reduced the per-compute-hour cost of real-time video inference for consumer video formats by orders of magnitude. That's aggressive by video-generation standards: when commercial video-generation APIs are converted from per-second pricing into an hourly usage rate, they tend to land in the hundreds of dollars, while face-only digital-human platforms sit well above that line. What we're delivering is full-body, half-body, and close-up shots with free camera movement — capabilities well beyond lip-sync and face-only animation. Our aim is to keep pushing until cost sits in the same band as cloud gaming, music subscriptions, and streaming memberships, not to buy our way into real-time with more expensive hardware. That is what determines whether the system can sustain long-running, always-on consumer interactions.

The capability moat comes from the whole stack working together. A single low-precision kernel eventually gets matched, a single attention optimization eventually gets open-sourced, a single deployment strategy inevitably gets copied. The moat comes from combining all of them into one pipeline that can operate reliably in production:

• Model-native streaming inference;
• Memory and low-precision engineering on consumer GPUs;
• Co-designed multi-GPU communication, memory offloading, and compute–communication overlap;
• Shortest end-to-end path across perception and reasoning modules, audio-visual generation, VAE, and streaming;
• An agent orchestration layer on top of the inference pipeline, supporting synchronous and asynchronous director agents, dual fast- and slow-reasoning modes, and multi-loop instruction orchestration over shared context, together with model-level streaming preemption and priority scheduling, so that interruptions, follow-ups, and mid-response redirects take effect within milliseconds;
• The engineering system that keeps all of this running in production and iterating quickly.

That's what Vivix Model Infra exists for: to serve today's avatar product, and to support the more demanding agents, interactive entertainment, virtual companionship, and productivity workflows that come next.

A video agent can't just be a text or voice agent wearing a video wrapper. Nor can it remain merely a video-generation tool. It has to see, listen, hold context, wait, be interrupted, and keep expressing itself — continuously, at low cost, for long stretches of time. Infrastructure is what moves all of this beyond the demo stage and into stable production.

6. From usable to widely deployed

Vivix Model Infra has taken real-time video inference from "runs a demo" to "runs in production." Several priorities come next.

Push cost toward CDN scale. The next milestone is to further reduce per-stream hourly cost, bringing it closer to CDN-scale delivery costs so video agents can be deployed the way CDNs are: always on, unmetered by the hour, and embedded as a default capability in any interactive product.

Push model–system co-design deeper. Real-time video isn't purely a model problem. Next-generation sparse attention, more aggressive low-precision formats, and KV cache reuse strategies all require the model architecture and inference engine to be designed together. New scenarios (multiple characters on screen, longer conversation context, more elaborate camera work) will in turn place demands on training objectives and sampling strategies. How far quality, latency, and cost can improve together depends on how far we can push the quality–latency–cost Pareto frontier.

Turn production experience into a real iteration loop. In real-time video generation, offline benchmarks don't settle the score. Stalls, interruption response, lip-sync drift, and visual consistency issues often surface only in production. We're building finer-grained online evaluation and incident replay capabilities to identify these failure modes faster, reproduce them reliably, and guide improvements across the model, inference stack, and serving system.

Back to the wall we opened with. For two years, real-time generative video has repeatedly reached the demo stage without becoming truly shippable. Vivix Model Infra is taking that wall apart, piece by piece, until real-time video generation can respond in under a second, support interruption, run on consumer hardware, and operate at costs comparable to mainstream digital entertainment.

Once all of those hold, video stops being the delayed output of a submitted job. It becomes what voice and text already are: a first-class channel in how humans and machines talk to each other.