Cones: Concept Neurons in Diffusion Models for Customized Generationhttps://arxiv.org/abs/2303.05125v1
The study explores if modern deep neural networks exhibit similar patterns to human brains in responding to semantic features of presented stimuli with different neurons. The researchers identify a small cluster of neurons in a diffusion model that correspond to a particular subject, which they call concept neurons. These neurons can be identified by statistics of network gradients to a stimulation connected with the given subject and demonstrate magnetic properties in interpreting and manipulating generation results. Shutting them can directly yield the related subject contextualized in different scenes, and concatenating multiple clusters of concept neurons can generate all related concepts in a single image. The concept neurons are environmentally friendly as they only need to store a sparse cluster of int index instead of dense float32 values of the parameters, which reduces storage consumption by 90%. The study shows the superiority of the method in interpreting and manipulating diffusion models through extensive qualitative and quantitative studies on diverse scenarios.