LVCD: Reference-based Lineart Video Colorization with Diffusion Models
ACM Transactions on Graphics & SIGGRAPH Asia 2024
\(^*\) : Corresponding author
ArXiv paper |
Code is available now! |
Supplementary (120 demo clips)
The first row is from Princess Mononoke directed by Hayao Miyazaki, the second and fourth rows are from Big Fish & Begonia by Bi An Tian (Beijing) Culture Co., Ltd., and the third row is from Mr. Miao by Gu Dong Animation Studio.
Abstract
We propose the first video diffusion framework for reference-based lineart video colorization. Unlike previous works that rely solely on image generative models to colorize lineart frame by frame, our approach leverages a large-scale pretrained video diffusion model to generate colorized animation videos. This approach leads to more temporally consistent results and is better equipped to handle large motions. Firstly, we introduce Sketch-guided ControlNet which provides additional control to finetune an image-to-video diffusion model for controllable video synthesis, enabling the generation of animation videos conditioned on lineart. We then propose Reference Attention to facilitate the transfer of colors from the reference frame to other frames containing fast and expansive motions. Finally, we present a novel scheme for sequential sampling, incorporating the Overlapped Blending Module and Prev-Reference Attention, to extend the video diffusion model beyond its original fixed-length limitation for long video colorization. Both qualitative and quantitative results demonstrate that our method significantly outperforms state-of-the-art techniques in terms of frame and video quality, as well as temporal consistency. Moreover, our method is capable of generating high-quality, long temporal-consistent animation videos with large motions, which is not achievable in previous works.
Methodology
We aim to design a video diffusion framework for reference-based lineart video colorization, capable of producing temporally consistent long sequences of animations with large motions. First, we propose the sketch-guided ControlNet and Reference Attention, which enable the model to generate animations with fast and expansive movements guided by lineart sketches. After modifying the model architecture, we finetune it using animation videos to perform our task. During inference, we extend the original SVD to produce long, temporally consistent animations through sequential sampling, incorporating the Overlapped Blending Module and Prev-Reference Attention.
Comparisons
Qualitative comparison with five methods: ACOF [Yu et al . 2024], TCVC [Thasarathan et al. 2019], CNet+Refonly [Zhang et al . 2023], EISAI [Chen and Zwicker 2022], and SEINE [Chen et al . 2024].