ControlNet, a neural network structure that adds extra conditions to diffusion models to control them. ControlNet copies the weights of neural network blocks into a "locked" copy and a "trainable" copy, allowing the "trainable" copy to learn the condition while preserving the production-ready diffusion models. The "zero convolution" is used to prevent distortion during training, and no layer is trained from scratch, making it safe to use with small-scale or personal devices. The text also explains how ControlNet can be used with Stable Diffusion to reuse the SD encoder as a powerful backbone for learning diverse controls. The efficacy of the SD encoder as a backbone is validated through various evidences.

Tips for using ControlNet include adding negative prompts and ignoring canvas height/width. The addon have also T2I-Adapter support and experimental features such as CFG-based ControlNet, Guess Mode, and Multi-ControlNet/Joint Conditioning. The weight and guidance strength/start/end are factors that affect the influence of ControlNet on the original SD Unet.