PIXART-α is a low-cost and efficient text-to-image (T2I) model that produces high-quality images. By utilizing a Transformer-based diffusion model, PIXART-α achieves state-of-the-art image generation standards and supports high-resolution synthesis. The authors propose three core designs, including a decomposed training strategy, an efficient T2I Transformer with cross-attention modules, and the use of high-informative data for auto-labeling. As a result, PIXART-α significantly reduces training time, saving costs and reducing CO2 emissions. The model demonstrates excellent image quality, artistry, and semantic control, making it a valuable tool for the AIGC community and startups seeking to build their own affordable and high-quality generative models.

InstaFlow is an ultra-fast, one-step image generator that overcomes computational constraints of diffusion models in text-to-image generation. It achieves image quality similar to Stable Diffusion but requires significantly lesser computational resources due to the use of a Rectified Flow technique. This technique enables training of probability flows with straight trajectories, requiring just a single step for quick inference. InstaFlow's advantages include ultra-fast inference, up to 90% faster than Stable Diffusion, high-quality image generation, and a simple and efficient training process. It produces results equivalent to top text-to-image GANs, like StyleGAN-T, and with the help of a pre-trained Stable Diffusion model, InstaFlow-0.9B requires only 199 A100 GPU days to train.

NeoPrompt is a tool designed to make the creation of AI art more accessible and less time-consuming by providing a comprehensive framework for generating unique prompts. Catering to both beginners and expert artists, it features a repository of imaginative words and phrases and offers explanatory insights into its prompt generation methods. It's compatible with popular AI art platforms like Midjourney and Stable Diffusion, and the team is continuously working on enhancing its compatibility and accuracy.

Fondant is an open-source framework designed to simplify and accelerate large-scale data processing. It allows for the reuse of containerized components across pipelines and execution environments, enabling sharing within the community. Fondant provides plug-and-play pipelines for various tasks such as AI image generation model fine-tuning, large language model fine-tuning, and code generation model fine-tuning. It also offers a library of reusable components for tasks such as data extraction, filtering, removal of unwanted content, data transformation, data tuning, and data enrichment. The framework supports multimodal capabilities, standardized Python/Pandas-based custom component creation, and production-ready scalable deployment. It also integrates with multiple cloud platforms. Fondant's main goal is to give users control over their data and simplify the building of pipelines for large-scale data processing. The text provides information on how to get started with Fondant and showcases example pipelines for tasks like filtering a creative commons image dataset and fine-tuning models like ControlNet and Stable Diffusion.

https://github.com/ml6team/fondant

IP-Adapter, an effective and lightweight adapter to achieve image prompt capability for the pre-trained text-to-image diffusion models. An IP-Adapter with only 22M parameters can achieve comparable or even better performance to a fine-tuned image prompt model. IP-Adapter can be generalized not only to other custom models fine-tuned from the same base model, but also to controllable generation using existing controllable tools. Moreover, the image prompt can also work well with the text prompt to accomplish multimodal image generation.

https://github.com/cubiq/ComfyUI_IPAdapter_plus

With the advance of text-to-image models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an affordable cost. Subsequently, there is a great demand for image animation techniques to further combine generated static images with motion dynamics. In this report, we propose a practical framework to animate most of the existing personalized text-to-image models once and for all, saving efforts in model-specific tuning. At the core of the proposed framework is to insert a newly initialized motion modeling module into the frozen text-to-image model and train it on video clips to distill reasonable motion priors. Once trained, by simply injecting this motion modeling module, all personalized versions derived from the same base T2I readily become text-driven models that produce diverse and personalized animated images. We conduct our evaluation on several public representative personalized text-to-image models across anime pictures and realistic photographs, and demonstrate that our proposed framework helps these models generate temporally smooth animation clips while preserving the domain and diversity of their outputs.

Perfusion is a text-to-image personalization method that creatively portrays personalized objects with significant changes in appearance while maintaining their identity. It introduces a novel mechanism called "Key-Locking" to maintain high visual fidelity and allow creative control, combine personalized concepts, and keep a small model size. Perfusion achieves this through dynamic rank-1 updates to the underlying text-to-image model and locks new concepts' cross-attention Keys to their superordinate category. It also uses a gated rank-1 approach to control the influence of learned concepts during inference and combine multiple concepts, enabling a trade-off between visual fidelity and textual alignment. With just a 100KB trained model, Perfusion covers different operating points across the Pareto front without additional training. It outperforms strong baselines in both qualitative and quantitative terms, enabling personalization even in one-shot settings and displaying personalized object interactions in novel ways. The method also allows for efficient control of visual-textual alignment and demonstrates different variations of key-locking. Additionally, Perfusion concepts trained with a vanilla diffusion model can generalize to fine-tuned variants.

Pytorch implementation : https://github.com/lucidrains/perfusion-pytorch

A novel approach to personalizing text-to-image models using diffusers. The current methods for personalization face challenges like long training times, high storage requirements, and loss of identity. To address these issues, the proposed method utilizes an encoder-based domain-tuning approach. This approach involves underfitting on a large set of concepts from a given domain to improve generalization and enable quick addition of novel concepts from the same domain. The method consists of two components: an encoder that maps a target concept image to a word embedding representing the concept, and regularized weight-offsets for the text-to-image model that effectively incorporate additional concepts. By leveraging these components, the model can be personalized with a single image and as few as 5 training steps, significantly reducing personalization time while maintaining quality.

GLIGEN is a new approach that enhances text-to-image generation models by allowing them to be conditioned on additional grounding inputs. It achieves impressive performance in generating images from text by including caption and bounding box condition inputs. GLIGEN takes advantage of a pre-trained model by freezing its weights and adding trainable layers to incorporate grounding information. This enables the model to have a better understanding of spatial relationships and concepts. GLIGEN outperforms other models in generating images based on specific instructions and can handle counterfactual scenarios. It can also use reference images for more detailed and stylistic output. Additionally, GLIGEN has the capability to inpaint images based on provided bounding boxes. Overall, GLIGEN improves the controllability and versatility of text-to-image generation models.

Composer introduces a generative model that addresses the limited controllability of existing large-scale models. By decomposing images into representative factors and utilizing a diffusion model to recompose them with these factors as conditions, Composer allows for flexible control over the output image while maintaining quality and creativity. It supports various levels of conditions, such as text description, depth map, sketch, and color histogram. Furthermore, Composer serves as a general framework for classical generative tasks without requiring retraining. The composition results demonstrate the ability to combine elements like text and depth, masked image and text, sketch, depth, and embedding, and more. Additionally, the model facilitates image variations, interpolations, reconfigurations, and region-specific editing, as well as tasks like image translation, style transfer, pose transfer, and virtual try-on.

A new algorithm called Restart has been developed to improve the speed and quality of generative processes. These processes use complex differential equations, which usually has a trade-off between quick results and accuracy. Existing methods like ODE and SDE samplers either work fast but reach a performance limit or offer better quality at a slower rate. The Restart algorithm outperforms these existing methods by better managing these sampling errors. In tests, it produced faster and higher quality results, surpassing SDE results on different platforms and maintaining better balance in large-scale model testing.

AI company Optic has developed a web tool called AI or Not, which aims to combat misinformation spread through AI-generated images. The tool scans images and quickly determines whether they were generated by artificial intelligence or by humans. Optic claims that its algorithms provide highly accurate results with a precision rate of 95%. However, users may have concerns about privacy when uploading images to the tool. Optic states that uploaded images and URLs are not stored on its servers longer than necessary and that they adhere to data protection regulations. By analyzing images and detecting signs of AI generation, the company aims to improve its algorithms and machine learning techniques.

A new method for inverting Deep Diffusion Probabilistic Models (DDPMs) by using noise maps to encode image structure and facilitate image editing. The authors demonstrate that this method can be used for text-based image editing tasks and can generate diverse results for any given image and text. The article compares this method to other image editing techniques such as P2P, DDIM inversion, and Plug-and-Play (PnP) and shows how this new method can perform better in terms of fidelity, adherence to target prompts, and support of diversity among generated outputs.

Demo

Stable Diffusion Sketch is an Android app that allows users to create colorful sketches and enhance them using various modes of Stable Diffusion. Users can create new paintings using blank canvas, camera capture, Stable Diffusion txt2img, and images shared from other apps. The app also offers preset modes, custom modes, painting tools, undo/redo functionality, and options to adjust prompt prefix, postfix, and negative prompt values. There are three canvas aspect ratios available: landscape, portrait, and square. Users can also enlarge the image using the upscaler feature or delete them from the main screen. Grouping related sketches is also possible, and the app supports multiple ControlNet.

DreamSim is a new algorithm designed to compare images holistically by assessing images on their mid-level attributes such as object layout, position, and semantic context. Unlike standard image comparison algorithms that only compare images based on colors and textures, DreamSim evaluates an image on its overall appearance and composition, giving it a more well-rounded comparison. The algorithm uses synthetic data to train its network but can still perform well on real images. A key point is that DreamSim showed better results than existing comparison algorithms in identifying objects and overall semantic context within an image. In summary, DreamSim is a valuable image comparison tool that captures features beyond low-level colors and textures while providing a stronger foundation for visual analysis.

Stability AI has released an open-source version of its DreamStudio text-to-image consumer application called StableStudio. The company intends to work with the broader community to create a world class user interface for generative AI that users fully control. DreamStudio was first conceived as an animation studio that shifted its focus to image generation with the arrival of Stable Diffusion in the summer of 2022. StableStudio enable local-first development through WebGPU and a desktop installation of its Stable Diffusion tool. It is also compatible with ControlNet tools and local inference through AUTOMATIC1111 stable-diffusion-webui tool.

aiNodes is a Python-based AI image/motion picture generator node engine that facilitates creativity in the creation of images and videos. The engine is fully modular and can download node packs on runtime. It also features RIFE and FILM interpolation integration, coloured background drop, and node-creation with IDE annotations. The installation process requires Python 3.10, Git, and an nVidia GPU with CUDA and drivers installed. AiNodes engine is an open-source desktop AI-based image/motion generator that supports various features such as Deforum, Stable Diffusion, Upscalers, Kandinsky, ControlNet, LORAs, Ti Embeddings, Hypernetworks, Background Separation, Human matting/masking, and Compositing, among others.

(Kind of) Realtime Stable Diffusion from a screen area using Automatic1111's API.

ProFusion is a new framework for customized text-to-image generation that preserves fine-grained image details without using regularization, as proposed in the paper. ProFusion includes PromptNet, an encoder network, and Fusion Sampling, a method that generates customized images based on a single user-provided image and text requirements. The paper explains how ProFusion works and provides experiments demonstrating its superior performance compared to existing approaches, while still meeting additional user-defined requirements.

We propose the Asymmetric VQGAN, to preserve the information of conditional image input. Asymmetric VQGAN involves two core designs compared with the original VQGAN as shown in the figure. First, we introduce a conditional branch into the decoder of the VQGAN which aims to handle the conditional input for image manipulation tasks. Second, we design a larger decoder for VQGAN to better recover the losing details of the quantized codes.