LLaVA-Plus introduces a versatile multimodal assistant that extends LLaVA by incorporating a diverse set of external tools for real-world task fulfillment. It maintains a skill repository of vision and vision-language pre-trained models, activating relevant tools based on users' multimodal inputs to compose execution results dynamically. The process involves humans providing a task instruction related to an image, the assistant analyzing the input, selecting and executing a tool, and finally providing the aggregated result.

Fooocus is a free and offline image-generating tool inspired by Stable Diffusion and Midjourney designs. It simplifies the creative process by eliminating the need for manual adjustments, letting users focus on prompts and images. With inner optimizations, it takes care of technical details, making it easy for anyone to use. Installation is straightforward, needing less than three clicks, and it works with just 4GB of GPU memory. Fooocus aims to make creative exploration enjoyable, allowing users to discover new ideas and expand their imagination effortlessly.

Kosmos-G is a model that leverages Multimodal Large Language Models (MLLMs) to generate images in context from generalized vision-language inputs. It aligns the output space of the MLLM with CLIP using the textual modality as an anchor. Kosmos-G's unique capability includes zero-shot multi-entity subject-driven generation and seamless integration with various U-Net techniques. The model consists of an MLLM for multimodal perception and an AlignerNet that connects the MLLM to the image decoder. The training pipeline involves pre-training the MLLM, aligning the image decoder, and fine-tuning through instruction tuning. Kosmos-G demonstrates its effectiveness in diverse image generation tasks and offers potential applications with customized image decoder variants.

PIXART-α is an innovative Transformer-based text-to-image diffusion model introduced in this paper to overcome the substantial training costs associated with advanced models in the AIGC community. Offering competitive image generation quality akin to leading generators like Imagen and Midjourney, PIXART-α stands out for its efficiency, supporting high-resolution synthesis up to 1024px with significantly reduced training expenses. Its key features include a decomposed training strategy, an efficient Transformer with cross-attention modules, and an emphasis on high-informative data. PIXART-α's training speed surpasses existing models, saving costs (10.8% of Stable Diffusion v1.5's training time) and reducing CO2 emissions by 90%. With exceptional image quality, artistry, and semantic control, PIXART-α presents potential insights for startups and the AIGC community to build high-quality, low-cost generative models.

Collage Diffusion is a novel interface for interacting with image generation models. It allows you to specify the composition of an image in a familiar Photoshop-like interface. Our modified version of Stable Diffusion takes the layers in and produces a harmonized image, ensuring that everything from perspectives to lighting are plausible. Unlike text prompting supported by traditional diffusion interfaces, Layered Diffusion allows you to precisely outline how a scene should be composed—from where objects are relative to each other to what they look like.

With the advance of text-to-image models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an affordable cost. Subsequently, there is a great demand for image animation techniques to further combine generated static images with motion dynamics. In this report, we propose a practical framework to animate most of the existing personalized text-to-image models once and for all, saving efforts in model-specific tuning. At the core of the proposed framework is to insert a newly initialized motion modeling module into the frozen text-to-image model and train it on video clips to distill reasonable motion priors. Once trained, by simply injecting this motion modeling module, all personalized versions derived from the same base T2I readily become text-driven models that produce diverse and personalized animated images. We conduct our evaluation on several public representative personalized text-to-image models across anime pictures and realistic photographs, and demonstrate that our proposed framework helps these models generate temporally smooth animation clips while preserving the domain and diversity of their outputs.

"Magic123" is a two-stage solution for generating high-quality 3D meshes from single images. It uses 2D and 3D priors to optimize a neural radiance field in the first stage, creating a coarse geometry. The second stage utilizes a memory-efficient mesh representation to produce a high-resolution mesh with appealing texture. Through reference view supervision and diffusion priors, the approach generates novel views. The system incorporates a tradeoff parameter for controlling the balance between exploration and precision in the generated geometry.

UniControl is a new generative model that combines many condition-to-image (C2I) tasks into one framework while still allowing for language prompts. This enables precise image generation where visual conditions influence the generated structures and language prompts guide the style and context. UniControl can handle diverse visual conditions through pretrained text-to-image diffusion models and a task-aware HyperNet that modulates these models. UniControl outperforms single-task models of similar sizes and marks a significant advancement in controlled visual generation.

Composable Diffusion (CoDi) is a new generative model that can create different types of outputs (like language, images, videos, or audio) from various inputs. It can generate multiple outputs at the same time and is not limited to specific types of inputs. Even without specific training data, CoDi aligns inputs and outputs to generate any combination of modalities. It uses a unique strategy to create a shared multimodal space, allowing synchronized generation of intertwined modalities.

Demo video

DeepFloyd IF is a text-to-image model that utilizes the large language model T5-XXL-1.1 as a text encoder to generates intelligible and coherent image alongside with text. The model is capable of incorporating text into images, generating a high degree of photorealism and the ability to generate images with non-standard aspect ratios. It can also modify style, patterns, and details in images without the need for fine-tuning. DeepFloyd IF is modular, cascaded, and works in pixel space, utilizing diffusion models that inject random noise into data before reversing the process to generate new data samples from the noise.

Demo

Making sure that diffusion model-generated images are safe from undesirable content and copyrighted material is a serious concern. Previous methods for stopping this content could be easily bypassed, but this new technique can fine-tune the model weights to erase concepts permanently, without having to retrain the entire model. The technique works by using the model's knowledge to guide the output away from the targeted concept. The authors have tested this technique on artistic style erasure and erasing nudity from images, and found it to be more effective than previous methods at erasing targeted concepts. However, for large concepts, there may be a trade-off between complete erasure and interference with other concepts.

Paella is an easy-to-use text-to-image model that can turn text into pictures. It was inspired by earlier models but has simpler code for training and sampling. During training, it "noises" images by randomly replacing visual elements with others from a library, and then tries to predict the original elements. During sampling, the model creates a distribution over each element and then selects one at random to build up the final image. Paella is designed to make text-to-image models more accessible to non-experts in the field by simplifying the technical components.

This stable-diffusion-webui extension aims to limit the influence of certain tokens in language models by rewriting them as padding tokens. This is important because the position vector of certain tokens can affect other tokens in unexpected ways, and the goal is to remove this influence while still preserving the overall meaning of the sentence. The extension achieves this by generating a new prompt with the specified token replaced by a padding token, and overwriting the position vectors corresponding to the original token with those generated from the new prompt. The Cutoff option allows the user to control the level of influence limitation, with "off" being the default setting.

ComfyUI is a powerful and modular stable diffusion GUI and backend that enables users to design and execute advanced stable diffusion pipelines using a graph/nodes/flowchart-based interface. This software supports SD1.x and SD2.x, and provides an asynchronous queue system with many optimizations. It can be used to load ckpt, safetensors, and diffusers models/checkpoints, as well as standalone VAEs and CLIP models. Other features include embeddings/textual inversion, area composition, inpainting with both regular and inpainting models, ControlNet and T2I-Adapter, upscale models, unCLIP models, and more. ComfyUI starts up quickly and works fully offline without downloading anything. Users can also save and load workflows as Json files, and the nodes interface can be used to create complex workflows.

https://github.com/BlenderNeko/ComfyUI_Cutoff

Long Stable Diffusion is a pipeline of generative models that can be used to illustrate a full story. Currently, Stable Diffusion can only take in a short prompt, but Long Stable Diffusion can generate images for a long-form text. The process involves starting with a long-form text, asking GPT-3 for several illustration ideas for the beginning, middle, and end of the story, translating the ideas to "prompt-English," and then putting them through Stable Diffusion to generate the images. The images and prompts are then dumped into a .docx file for easy copy-pasting. The purpose of this pipeline is to automate the process of generating illustrations for AI-generated stories.

The study aims to improve the ability of large text-to-image models to express customized concepts without excessive computation or memory burden. The researchers propose a learning-based encoder consisting of global and local mapping networks. The global mapping network projects the hierarchical features of a given image into multiple "new" words in the textual word embedding space, including one primary word for well-editable concepts and other auxiliary words to exclude irrelevant disturbances. The local mapping network injects the encoded patch features into cross-attention layers to provide omitted details without sacrificing the editability of primary concepts. The study compares the proposed method with prior optimization-based approaches on a variety of user-defined concepts and demonstrates that it enables more high-fidelity inversion and robust editability with a significantly faster encoding process.

Gradio demo

The paper proposes Custom Diffusion, an efficient method for augmenting existing text-to-image models to quickly acquire new concepts given a few examples. The method optimizes only a few parameters in the text-to-image conditioning mechanism to represent new concepts while enabling fast tuning. The fine-tuned model can generate variations of multiple new concepts in novel unseen settings. The method is fast and has low storage requirements for each additional concept model apart from the pretrained model. The paper shows single-concept and multi-concept results, as well as sample qualitative comparisons with concurrent works. The method has limitations in difficult compositions, such as composing pets of different species, and composing three or more concepts together.

The strength of modern generative models lies in their ability to be controlled through text-based prompts. Typical "hard" prompts are made from interpretable words and tokens, and must be hand-crafted by humans. There are also "soft" prompts, which consist of continuous feature vectors. These can be discovered using powerful optimization methods, but they cannot be easily interpreted, re-used across models, or plugged into a text-based interface.
We describe an approach to robustly optimize hard text prompts through efficient gradient-based optimization. Our approach automatically generates hard text-based prompts for both text-to-image and text-to-text applications. In the text-to-image setting, the method creates hard prompts for diffusion models, allowing API users to easily generate, discover, and mix and match image concepts without prior knowledge on how to prompt the model. In the text-to-text setting, we show that hard prompts can be automatically discovered that are effective in tuning LMs for classification.

This space can either generate a text fragment that describes your image, or it can shorten an existing text prompt. This space is using OpenCLIP-ViT/H, the same text encoder used by Stable Diffusion V2. After you generate a prompt, try it out on Stable Diffusion here, here or on Midjourney. For a quick PEZ demo, try clicking on one of the examples at the bottom of this page.

Note: Generation with 1000 steps takes ~60 seconds with a T4. Don't want to wait? You can also run on Google Colab. Or, you can reduce the number of steps.

More example here : https://bird.trom.tf/tomgoldsteincs/status/1623358917110538240#m

Code on Github

Versatile Diffusion (VD), the first unified multi-flow multimodal diffusion framework, as a step towards Universal Generative AI. VD can natively support image-to-text, image-variation, text-to-image, and text-variation, and can be further extended to other applications such as semantic-style disentanglement, image-text dual-guided generation, latent image-to-text-to-image editing, and more. Future versions will support more modalities such as speech, music, video and 3D.

Xingqian Xu, Atlas Wang, Eric Zhang, Kai Wang, and Humphrey Shi [arXiv] [GitHub]

The paper introduces a technique for text-based editing of real images using the Stable Diffusion model. In order to modify an image using text, it must first be inverted with a meaningful text prompt into the pretrained model's domain. The proposed inversion technique consists of two novel components: pivotal inversion for diffusion models, which uses a single pivotal noise vector for each timestamp and optimizes around it, and null-text optimization, which modifies only the unconditional textual embedding used for classifier-free guidance, rather than the input text embedding. This allows for prompt-based editing while avoiding the cumbersome tuning of the model's weights. The proposed null-text inversion technique is extensively evaluated on a variety of images and various prompt editing, showing high-fidelity editing of real images.