SDXL-Turbo stands out as a high-speed generative text-to-image model capable of producing photorealistic images from a given text prompt in a single network evaluation. It represents a distilled variant of SDXL 1.0, specifically trained for real-time synthesis. The model adopts Adversarial Diffusion Distillation (ADD), a novel training method detailed in the technical report, allowing efficient sampling of foundational image diffusion models in 1 to 4 steps while maintaining high image quality. Score distillation is employed to leverage large-scale off-the-shelf image diffusion models as a teacher signal, coupled with an adversarial loss to ensure superior image fidelity, even in scenarios with one or two sampling steps.

Download safe tensors

LLaVA-Plus introduces a versatile multimodal assistant that extends LLaVA by incorporating a diverse set of external tools for real-world task fulfillment. It maintains a skill repository of vision and vision-language pre-trained models, activating relevant tools based on users' multimodal inputs to compose execution results dynamically. The process involves humans providing a task instruction related to an image, the assistant analyzing the input, selecting and executing a tool, and finally providing the aggregated result.

Phind introduces its latest model, Phind Model V7, which is faster and outperforms GPT-4 in coding tasks. It achieves a HumanEval score of 74.7%, fine-tuned with 70B+ tokens of high-quality code. Notably, it runs five times faster than GPT-4, thanks to NVIDIA's TensorRT-LLM library. The model supports up to 16k tokens, allowing 12k tokens for user input on the website

Zero123++ is an advanced image-conditioned diffusion model highlighted in this paper, focusing on generating 3D-consistent multi-view images from a single input view. The model minimizes the effort of fine-tuning by leveraging pre-trained 2D generative priors, particularly from StableDiffusion. Noteworthy improvements include tiling six views into a single image, shifting the noise schedule, implementing scaled reference attention for local conditioning, and introducing FlexDiffuse for global conditioning. Through these enhancements, Zero123++ excels in producing high-quality, consistent multi-view images, overcoming issues like texture degradation and geometric misalignment. The paper showcases the model's effectiveness through qualitative and quantitative comparisons with leading models, emphasizing its potential for various applications. Additionally, a depth-controlled version of Zero123++ is introduced, demonstrating superior performance with ControlNet and highlighting the model's versatility.

StableSwarm introduces a user-friendly Modular Stable Diffusion Web-User-Interface, designed for easy access to powerful creative tools with high performance and expandability in projects. This interface, accessible on Google Colab for hands-on experience, facilitates efficient creation of dynamic visuals through features such as image editing tools, LoRAs, ControlNets, simplified TIs handling, streamlined text prompts, seamless translation, LAN reliability, mobile browser compatibility, a self-contained mini-backend, organized batch grids, and model usage guidance. StableSwarmUI aims to provide a comprehensive and accessible platform for creative endeavors, ensuring a smooth and organized workflow.

3D-GPT is a pioneering framework that simplifies 3D asset modeling in the metaverse era by utilizing large language models (LLMs). Developed collaboratively by teams from the Australian National University, University of Oxford, and Beijing Academy of Artificial Intelligence, 3D-GPT breaks down complex 3D modeling tasks into manageable segments, employing LLMs as adept problem-solvers. The framework consists of three key agents – task dispatch, conceptualization, and modeling – working together to enhance initial scene descriptions and seamlessly integrate procedural generation. Demonstrating reliability and effective collaboration with human designers, 3D-GPT not only streamlines traditional 3D modeling but also integrates smoothly with Blender, expanding manipulation possibilities. This innovative approach underscores the substantial potential of language models in shaping the future of 3D modeling, particularly in scene generation and animation.

Fooocus is a free and offline image-generating tool inspired by Stable Diffusion and Midjourney designs. It simplifies the creative process by eliminating the need for manual adjustments, letting users focus on prompts and images. With inner optimizations, it takes care of technical details, making it easy for anyone to use. Installation is straightforward, needing less than three clicks, and it works with just 4GB of GPU memory. Fooocus aims to make creative exploration enjoyable, allowing users to discover new ideas and expand their imagination effortlessly.

RAVE introduces a pioneering zero-shot video editing approach, leveraging pre-trained text-to-image diffusion models to achieve high-quality results with advanced user control. Addressing the lag in visual quality and user manipulation seen in video editing models, RAVE employs a novel noise shuffling strategy, exploiting spatio-temporal interactions between frames for faster and temporally consistent video production. Notably efficient in memory usage, RAVE handles longer videos while accommodating a broad spectrum of edits, from local attribute adjustments to shape transformations. The method's versatility is demonstrated through a comprehensive video evaluation dataset spanning diverse scenes, including object-focused, human activities, and dynamic scenarios. Rigorous quantitative and qualitative experiments underscore RAVE's superior performance in diverse video editing scenarios compared to existing methods, marking a substantial advancement in the realm of video editing with diffusion models.

Paint3D introduces a groundbreaking generative framework for creating high-resolution, lighting-less, and diverse 2K UV texture maps for untextured 3D meshes, conditioned on text or image inputs. Addressing the challenge of generating quality textures without embedded illumination information, the method employs a coarse-to-fine approach. It initially utilizes a pre-trained depth-aware 2D diffusion model for multi-view texture fusion, producing a coarse texture map. To overcome incomplete areas and illumination artifacts, separate UV Inpainting and UVHD diffusion models are trained for shape-aware refinement. The resulting process yields high-quality 2K UV textures with semantic consistency, allowing for significant advancements in texturing 3D objects and providing flexibility for re-lighting and editing within modern graphics pipelines.

OneTrainer is a comprehensive training solution catering to stable diffusion models, including Stable Diffusion 1.5, 2.0, 2.1, SDXL, Würstchen-v2, and inpainting models. Boasting diverse features such as masked training, automatic backups, image augmentation, and multi-resolution training, OneTrainer simplifies the fine-tuning process. The platform offers a user-friendly interface with preset configurations for quick setup, separate workspaces for distinct training runs, and advanced settings for caching, tensorboard integration, and model specifications. Its flexibility extends to input/output model settings, data configurations, and training parameters, making it a versatile tool for efficient and tailored diffusion model training. Additionally, the backup and continuation features facilitate a seamless training experience, allowing users to regularly back up and convert model data for continuous progress.

Show-1 is a hybrid model for text-to-video generation that combines pixel-based and latent-based VDMs. It addresses the limitations of relying solely on either approach. Show-1 first uses pixel-based VDMs to generate a low-resolution video with accurate text-video alignment. It then employs latent-based VDMs to upsample the low-resolution video to high resolution using a novel expert translation method. This combination allows Show-1 to produce high-quality videos with precise text-video alignment while being more efficient in terms of GPU memory usage compared to pixel-based VDMs.

Kosmos-G is a model that leverages Multimodal Large Language Models (MLLMs) to generate images in context from generalized vision-language inputs. It aligns the output space of the MLLM with CLIP using the textual modality as an anchor. Kosmos-G's unique capability includes zero-shot multi-entity subject-driven generation and seamless integration with various U-Net techniques. The model consists of an MLLM for multimodal perception and an AlignerNet that connects the MLLM to the image decoder. The training pipeline involves pre-training the MLLM, aligning the image decoder, and fine-tuning through instruction tuning. Kosmos-G demonstrates its effectiveness in diverse image generation tasks and offers potential applications with customized image decoder variants.

This paper aims to understand how neural networks work by breaking them down into smaller components that are easier to comprehend. However, it turns out that individual neurons are not easy to understand because they respond to a mix of unrelated inputs. One reason for this is superposition, where a neural network represents more features than it has neurons. The paper explores different approaches to finding interpretable features hidden by superposition, such as sparse autoencoders. By using sparse autoencoders, the researchers were able to extract meaningful features from a neural network and analyze its behavior. The paper provides detailed investigations, global analyses, and visualizations to support their findings.

PIXART-α is an innovative Transformer-based text-to-image diffusion model introduced in this paper to overcome the substantial training costs associated with advanced models in the AIGC community. Offering competitive image generation quality akin to leading generators like Imagen and Midjourney, PIXART-α stands out for its efficiency, supporting high-resolution synthesis up to 1024px with significantly reduced training expenses. Its key features include a decomposed training strategy, an efficient Transformer with cross-attention modules, and an emphasis on high-informative data. PIXART-α's training speed surpasses existing models, saving costs (10.8% of Stable Diffusion v1.5's training time) and reducing CO2 emissions by 90%. With exceptional image quality, artistry, and semantic control, PIXART-α presents potential insights for startups and the AIGC community to build high-quality, low-cost generative models.

The Gsgen project introduces a novel approach called Gaussian Splatting based text-to-3D GENeration (Gsgen) that generates high-quality, multi-view consistent 3D assets. Previous methods lacked accurate geometry and fidelity, so Gsgen leverages 3D Gaussian Splatting as a representation technique to address these limitations. Their approach involves a progressive optimization strategy, including a geometry optimization stage and an appearance refinement stage. The geometry optimization establishes a coarse representation under a 3D geometry prior, while the appearance refinement iteratively refines the obtained Gaussians to enhance details. The method proves effective in generating 3D assets with accurate geometry and delicate details

InstaFlow is an ultra-fast, one-step image generator that overcomes computational constraints of diffusion models in text-to-image generation. It achieves image quality similar to Stable Diffusion but requires significantly lesser computational resources due to the use of a Rectified Flow technique. This technique enables training of probability flows with straight trajectories, requiring just a single step for quick inference. InstaFlow's advantages include ultra-fast inference, up to 90% faster than Stable Diffusion, high-quality image generation, and a simple and efficient training process. It produces results equivalent to top text-to-image GANs, like StyleGAN-T, and with the help of a pre-trained Stable Diffusion model, InstaFlow-0.9B requires only 199 A100 GPU days to train.

NeoPrompt is a tool designed to make the creation of AI art more accessible and less time-consuming by providing a comprehensive framework for generating unique prompts. Catering to both beginners and expert artists, it features a repository of imaginative words and phrases and offers explanatory insights into its prompt generation methods. It's compatible with popular AI art platforms like Midjourney and Stable Diffusion, and the team is continuously working on enhancing its compatibility and accuracy.

Fondant is an open-source framework designed to simplify and accelerate large-scale data processing. It allows for the reuse of containerized components across pipelines and execution environments, enabling sharing within the community. Fondant provides plug-and-play pipelines for various tasks such as AI image generation model fine-tuning, large language model fine-tuning, and code generation model fine-tuning. It also offers a library of reusable components for tasks such as data extraction, filtering, removal of unwanted content, data transformation, data tuning, and data enrichment. The framework supports multimodal capabilities, standardized Python/Pandas-based custom component creation, and production-ready scalable deployment. It also integrates with multiple cloud platforms. Fondant's main goal is to give users control over their data and simplify the building of pipelines for large-scale data processing. The text provides information on how to get started with Fondant and showcases example pipelines for tasks like filtering a creative commons image dataset and fine-tuning models like ControlNet and Stable Diffusion.

https://github.com/ml6team/fondant

Collage Diffusion is a novel interface for interacting with image generation models. It allows you to specify the composition of an image in a familiar Photoshop-like interface. Our modified version of Stable Diffusion takes the layers in and produces a harmonized image, ensuring that everything from perspectives to lighting are plausible. Unlike text prompting supported by traditional diffusion interfaces, Layered Diffusion allows you to precisely outline how a scene should be composed—from where objects are relative to each other to what they look like.

TokenFlow is a framework for text-driven video editing using a text-to-image diffusion model. The framework aims to generate high-quality videos that adhere to a target text while preserving the spatial layout and dynamics of the input video. The method leverages the observation that consistency in the edited video can be achieved by enforcing consistency in the diffusion feature space. This is done by propagating edited features across frames based on inter-frame correspondences. The framework does not require training or fine-tuning and can be used with any text-to-image editing method. Experimental results demonstrate state-of-the-art video editing on real-world videos. The method involves inverting input video frames, extracting tokens, and using nearest-neighbor search for feature correspondences. Denoising is performed by replacing generated tokens with propagated tokens from the original video.