CoDeF: Content Deformation Fields for Temporally Consistent Video Processinghttps://github.com/qiuyu96/CoDeF
CoDeF is a new video representation involving a canonical content field and a temporal deformation field. The canonical content field captures the static contents of the entire video, while the temporal deformation field records the transformations from the canonical image to each frame. By optimizing these fields together, CoDeF can reconstruct a target video and support the application of image algorithms to the canonical image, which can then be propagated to the entire video using the temporal deformation field. This allows for video-to-video translation and keypoint tracking without training, resulting in improved cross-frame consistency compared to existing methods. CoDeF can also be applied to other tasks like point-based tracking, segmentation-based tracking, and video super-resolution.