Show-1 is a hybrid model for text-to-video generation that combines pixel-based and latent-based VDMs. It addresses the limitations of relying solely on either approach. Show-1 first uses pixel-based VDMs to generate a low-resolution video with accurate text-video alignment. It then employs latent-based VDMs to upsample the low-resolution video to high resolution using a novel expert translation method. This combination allows Show-1 to produce high-quality videos with precise text-video alignment while being more efficient in terms of GPU memory usage compared to pixel-based VDMs.

Kosmos-G is a model that leverages Multimodal Large Language Models (MLLMs) to generate images in context from generalized vision-language inputs. It aligns the output space of the MLLM with CLIP using the textual modality as an anchor. Kosmos-G's unique capability includes zero-shot multi-entity subject-driven generation and seamless integration with various U-Net techniques. The model consists of an MLLM for multimodal perception and an AlignerNet that connects the MLLM to the image decoder. The training pipeline involves pre-training the MLLM, aligning the image decoder, and fine-tuning through instruction tuning. Kosmos-G demonstrates its effectiveness in diverse image generation tasks and offers potential applications with customized image decoder variants.

This paper aims to understand how neural networks work by breaking them down into smaller components that are easier to comprehend. However, it turns out that individual neurons are not easy to understand because they respond to a mix of unrelated inputs. One reason for this is superposition, where a neural network represents more features than it has neurons. The paper explores different approaches to finding interpretable features hidden by superposition, such as sparse autoencoders. By using sparse autoencoders, the researchers were able to extract meaningful features from a neural network and analyze its behavior. The paper provides detailed investigations, global analyses, and visualizations to support their findings.

PIXART-α is a low-cost and efficient text-to-image (T2I) model that produces high-quality images. By utilizing a Transformer-based diffusion model, PIXART-α achieves state-of-the-art image generation standards and supports high-resolution synthesis. The authors propose three core designs, including a decomposed training strategy, an efficient T2I Transformer with cross-attention modules, and the use of high-informative data for auto-labeling. As a result, PIXART-α significantly reduces training time, saving costs and reducing CO2 emissions. The model demonstrates excellent image quality, artistry, and semantic control, making it a valuable tool for the AIGC community and startups seeking to build their own affordable and high-quality generative models.

The Gsgen project introduces a novel approach called Gaussian Splatting based text-to-3D GENeration (Gsgen) that generates high-quality, multi-view consistent 3D assets. Previous methods lacked accurate geometry and fidelity, so Gsgen leverages 3D Gaussian Splatting as a representation technique to address these limitations. Their approach involves a progressive optimization strategy, including a geometry optimization stage and an appearance refinement stage. The geometry optimization establishes a coarse representation under a 3D geometry prior, while the appearance refinement iteratively refines the obtained Gaussians to enhance details. The method proves effective in generating 3D assets with accurate geometry and delicate details

InstaFlow is an ultra-fast, one-step image generator that overcomes computational constraints of diffusion models in text-to-image generation. It achieves image quality similar to Stable Diffusion but requires significantly lesser computational resources due to the use of a Rectified Flow technique. This technique enables training of probability flows with straight trajectories, requiring just a single step for quick inference. InstaFlow's advantages include ultra-fast inference, up to 90% faster than Stable Diffusion, high-quality image generation, and a simple and efficient training process. It produces results equivalent to top text-to-image GANs, like StyleGAN-T, and with the help of a pre-trained Stable Diffusion model, InstaFlow-0.9B requires only 199 A100 GPU days to train.

NeoPrompt is a tool designed to make the creation of AI art more accessible and less time-consuming by providing a comprehensive framework for generating unique prompts. Catering to both beginners and expert artists, it features a repository of imaginative words and phrases and offers explanatory insights into its prompt generation methods. It's compatible with popular AI art platforms like Midjourney and Stable Diffusion, and the team is continuously working on enhancing its compatibility and accuracy.

Fondant is an open-source framework designed to simplify and accelerate large-scale data processing. It allows for the reuse of containerized components across pipelines and execution environments, enabling sharing within the community. Fondant provides plug-and-play pipelines for various tasks such as AI image generation model fine-tuning, large language model fine-tuning, and code generation model fine-tuning. It also offers a library of reusable components for tasks such as data extraction, filtering, removal of unwanted content, data transformation, data tuning, and data enrichment. The framework supports multimodal capabilities, standardized Python/Pandas-based custom component creation, and production-ready scalable deployment. It also integrates with multiple cloud platforms. Fondant's main goal is to give users control over their data and simplify the building of pipelines for large-scale data processing. The text provides information on how to get started with Fondant and showcases example pipelines for tasks like filtering a creative commons image dataset and fine-tuning models like ControlNet and Stable Diffusion.

https://github.com/ml6team/fondant

Collage Diffusion is a novel interface for interacting with image generation models. It allows you to specify the composition of an image in a familiar Photoshop-like interface. Our modified version of Stable Diffusion takes the layers in and produces a harmonized image, ensuring that everything from perspectives to lighting are plausible. Unlike text prompting supported by traditional diffusion interfaces, Layered Diffusion allows you to precisely outline how a scene should be composed—from where objects are relative to each other to what they look like.

TokenFlow is a framework for text-driven video editing using a text-to-image diffusion model. The framework aims to generate high-quality videos that adhere to a target text while preserving the spatial layout and dynamics of the input video. The method leverages the observation that consistency in the edited video can be achieved by enforcing consistency in the diffusion feature space. This is done by propagating edited features across frames based on inter-frame correspondences. The framework does not require training or fine-tuning and can be used with any text-to-image editing method. Experimental results demonstrate state-of-the-art video editing on real-world videos. The method involves inverting input video frames, extracting tokens, and using nearest-neighbor search for feature correspondences. Denoising is performed by replacing generated tokens with propagated tokens from the original video.

IP-Adapter, an effective and lightweight adapter to achieve image prompt capability for the pre-trained text-to-image diffusion models. An IP-Adapter with only 22M parameters can achieve comparable or even better performance to a fine-tuned image prompt model. IP-Adapter can be generalized not only to other custom models fine-tuned from the same base model, but also to controllable generation using existing controllable tools. Moreover, the image prompt can also work well with the text prompt to accomplish multimodal image generation.

https://github.com/cubiq/ComfyUI_IPAdapter_plus

With the advance of text-to-image models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an affordable cost. Subsequently, there is a great demand for image animation techniques to further combine generated static images with motion dynamics. In this report, we propose a practical framework to animate most of the existing personalized text-to-image models once and for all, saving efforts in model-specific tuning. At the core of the proposed framework is to insert a newly initialized motion modeling module into the frozen text-to-image model and train it on video clips to distill reasonable motion priors. Once trained, by simply injecting this motion modeling module, all personalized versions derived from the same base T2I readily become text-driven models that produce diverse and personalized animated images. We conduct our evaluation on several public representative personalized text-to-image models across anime pictures and realistic photographs, and demonstrate that our proposed framework helps these models generate temporally smooth animation clips while preserving the domain and diversity of their outputs.

In Relation Inversion the goal is to capture and apply a relation from exemplar images to synthesize new scenes. While existing methods focus on capturing object appearances, this work explores the inversion of object relations. The proposed ReVersion framework learns a relation prompt from exemplar images using a pre-trained text-to-image diffusion model. The key insight is the "preposition prior," where real-world relation prompts can be activated with a set of basis prepositional words. The framework incorporates relation-steering contrastive learning and relation-focal importance sampling to enforce capturing object interactions and disentangling from appearances. The paper also introduces the ReVersion Benchmark, providing diverse exemplar images with different relations. Experimental results validate the superiority of the proposed approach in inverting relations compared to existing methods.

ToolLLM is a project that aims to create a large-scale dataset for training language models with tool-use capabilities. They collect instructions involving real-world APIs and develop a new annotation approach to improve efficiency. The project provides the ToolLLaMA model, which performs well in handling single-tool and complex multi-tool instructions. They also release the ToolLLaMA-7b, ToolLLaMA-7b-LoRA, and ToolLLaMA-2-7b models, along with a tool retriever. They evaluate the models using pass rate and preference metrics, showing good performance compared to other models. Overall, ToolLLM empowers language models to understand and use real-world tools effectively.

CoDeF is a new video representation involving a canonical content field and a temporal deformation field. The canonical content field captures the static contents of the entire video, while the temporal deformation field records the transformations from the canonical image to each frame. By optimizing these fields together, CoDeF can reconstruct a target video and support the application of image algorithms to the canonical image, which can then be propagated to the entire video using the temporal deformation field. This allows for video-to-video translation and keypoint tracking without training, resulting in improved cross-frame consistency compared to existing methods. CoDeF can also be applied to other tasks like point-based tracking, segmentation-based tracking, and video super-resolution.

Cheetor, a Transformer-based multi-modal large language model equipped with controllable knowledge re-injection. Cheetor demonstrates strong capabilities in reasoning over complicated interleaved vision-language instructions. It can identify connections between images, infer causes and reasons, understand metaphorical implications, and comprehend absurd objects through multi-modal conversations with humans.

Perfusion is a text-to-image personalization method that creatively portrays personalized objects with significant changes in appearance while maintaining their identity. It introduces a novel mechanism called "Key-Locking" to maintain high visual fidelity and allow creative control, combine personalized concepts, and keep a small model size. Perfusion achieves this through dynamic rank-1 updates to the underlying text-to-image model and locks new concepts' cross-attention Keys to their superordinate category. It also uses a gated rank-1 approach to control the influence of learned concepts during inference and combine multiple concepts, enabling a trade-off between visual fidelity and textual alignment. With just a 100KB trained model, Perfusion covers different operating points across the Pareto front without additional training. It outperforms strong baselines in both qualitative and quantitative terms, enabling personalization even in one-shot settings and displaying personalized object interactions in novel ways. The method also allows for efficient control of visual-textual alignment and demonstrates different variations of key-locking. Additionally, Perfusion concepts trained with a vanilla diffusion model can generalize to fine-tuned variants.

Pytorch implementation : https://github.com/lucidrains/perfusion-pytorch

A novel approach to personalizing text-to-image models using diffusers. The current methods for personalization face challenges like long training times, high storage requirements, and loss of identity. To address these issues, the proposed method utilizes an encoder-based domain-tuning approach. This approach involves underfitting on a large set of concepts from a given domain to improve generalization and enable quick addition of novel concepts from the same domain. The method consists of two components: an encoder that maps a target concept image to a word embedding representing the concept, and regularized weight-offsets for the text-to-image model that effectively incorporate additional concepts. By leveraging these components, the model can be personalized with a single image and as few as 5 training steps, significantly reducing personalization time while maintaining quality.

GLIGEN is a new approach that enhances text-to-image generation models by allowing them to be conditioned on additional grounding inputs. It achieves impressive performance in generating images from text by including caption and bounding box condition inputs. GLIGEN takes advantage of a pre-trained model by freezing its weights and adding trainable layers to incorporate grounding information. This enables the model to have a better understanding of spatial relationships and concepts. GLIGEN outperforms other models in generating images based on specific instructions and can handle counterfactual scenarios. It can also use reference images for more detailed and stylistic output. Additionally, GLIGEN has the capability to inpaint images based on provided bounding boxes. Overall, GLIGEN improves the controllability and versatility of text-to-image generation models.

Composer introduces a generative model that addresses the limited controllability of existing large-scale models. By decomposing images into representative factors and utilizing a diffusion model to recompose them with these factors as conditions, Composer allows for flexible control over the output image while maintaining quality and creativity. It supports various levels of conditions, such as text description, depth map, sketch, and color histogram. Furthermore, Composer serves as a general framework for classical generative tasks without requiring retraining. The composition results demonstrate the ability to combine elements like text and depth, masked image and text, sketch, depth, and embedding, and more. Additionally, the model facilitates image variations, interpolations, reconfigurations, and region-specific editing, as well as tasks like image translation, style transfer, pose transfer, and virtual try-on.