Video Diffusion Models

AI绘画模型—

One of our main innovations is a new conditional generation method for unconditional diffusion models. Our new conditioning method, which refer to as the gradient method, modifies the sampling procedure of the model to improve a conditioning loss on denoised data using gradient-based optimization. We find that the gradient method is more capable than existing methods in ensuring consistency of the generated samples with the conditioning information.

We use the gradient method to autoregressively extend our models to more timesteps and higher resolutions.

Frames from our gradient method (left) and a baseline "replacement" method (right) for autoregressive extension. Videos sampled using the gradient method attain superior temporal coherence compared to the baseline method.

We show that high quality videos can be generated by essentially the standard formulation of the Gaussian diffusion model, with little modification other than straightforward architectural changes to accommodate video data within memory constraints of deep learning accelerators. We train models that generate a block of a fixed number of frames of a video, and to generate videos longer than that number of frames, we additionally show how to repurpose a trained model to act as a model which is block-autoregressive over frames. We test our methods on an unconditional video generation benchmark, where we achieve state-of-the-art sample quality scores, and we also show promising results on text-conditioned video generation.