BLIP-Diffusion is a new model for generating and editing images based on text prompts and subject images. Unlike previous models, it uses a pre-trained multimodal encoder to represent the subject, allowing for efficient fine-tuning and better preservation of subject details. The model enables the generation of new images based on text prompts and subject images, even without prior training on specific subjects. It also supports image manipulation, style transfer, and editing guided by subject images. The model is trained in two stages to learn subject representation and can be combined with other techniques for more control over the generation and editing process. Overall, BLIP-Diffusion provides a flexible and efficient approach to generate and edit images with specific subjects.

DreamGPT, the first GPT-based solution that uses hallucinations from LLMs for divergent thinking to generate new innovative ideas. Hallucinations are often seen as a negative thing, but what if they could be used for our advantage? dreamGPT is here to show you how. The goal of dreamGPT is to explore as many possibilities as possible, as opposed to most other GPT-based solutions which are focused on solving specific problems.

Tree-of-Thought (ToT) aims to enhance the problem-solving capabilities of large language models (LLMs) like GPT-4. The framework utilizes a deliberate 'System 2' tree search approach to tackle complex and general problems that LLMs struggle with. The author demonstrates significant improvements on three tasks: the game of 24, creative writing, and crosswords, which GPT-4 and CoT (chain of thought, another approach) find challenging due to the need for planning and searching. The limitations of token-by-token decoding, which lacks lookahead, backtrack, and global exploration, are highlighted as the reason for these difficulties. ToT achieves a tenfold performance boost by leveraging the LLM's ability to generate diverse intermediate thoughts, self-evaluate them through deliberate reasoning, and employ search algorithms like breadth-first search (bfs) or depth-first search (dfs) to systematically explore the problem space.

Alternative repositorie

Composable Diffusion (CoDi) is a new generative model that can create different types of outputs (like language, images, videos, or audio) from various inputs. It can generate multiple outputs at the same time and is not limited to specific types of inputs. Even without specific training data, CoDi aligns inputs and outputs to generate any combination of modalities. It uses a unique strategy to create a shared multimodal space, allowing synchronized generation of intertwined modalities.

Demo video

A comprehensive guide to ControlNet v1.1, a Stable Diffusion model that allows users to control image compositions and human poses based on reference images. The guide covers various aspects of ControlNet, including its installation on different platforms (Windows, Mac, Google Colab), settings, and common use cases.

The tutorial provides a comprehensive guide on creating consistent characters using Stable Diffusion (SD) and a Textual Inversion embedding. It outlines a five-step process, including generating input images, filtering them based on desired attributes, tagging them for training, training the embedding, and selecting a validated iteration. The tutorial emphasizes the importance of generating high-quality input images, filtering out unwanted variations, and fine-tuning the selection to achieve consistency. By following this tutorial, users can learn how to generate consistent characters with SD and create an embedding that reliably recreates the desired character across different poses, hairstyles, body types, and prompts.

SmartGPT is an experimental program meant to provide LLMs (particularly GPT-3.5 and GPT-4) with the ability to complete complex tasks without user input by breaking them down into smaller problems, and collecting information using the internet and other external sources.

DragGAN is a novel approach for controlling generative adversarial networks (GANs) in order to synthesize visual content that meets users' needs. It offers precise and flexible controllability over the pose, shape, expression, and layout of generated objects. Unlike existing methods that rely on manual annotations or 3D models, DragGAN enables interactive control by allowing users to "drag" any points of an image to reach desired positions. The approach consists of two main components: feature-based motion supervision and a point tracking approach utilizing discriminative GAN features. By utilizing DragGAN, users can manipulate diverse categories of images, such as animals, cars, humans, and landscapes, with realistic outputs even in challenging scenarios like occluded content and deforming shapes.

OP Vault is a versatile platform that allows users to upload various document types through a simple react frontend, enabling the creation of a customized knowledge base. It leverages advanced algorithms to provide accurate and relevant answers based on the content of the uploaded documents. Users can gain insights into the answers by viewing filenames and specific context snippets. The user-friendly interface of OP Vault makes it easy to explore the capabilities of the OP Stack, a powerful combination of OpenAI and Pinecone Vector Database. Moreover, OP Vault supports large-scale uploads, making it possible to load entire libraries' worth of books, thus expanding the scope of knowledge accessible through the platform. To ensure smooth operation, certain manual dependencies such as node (v19), go (v1.18.9 darwin/arm64), and poppler are required. With its diverse features, OP Vault offers a convenient solution for document upload, accurate retrieval of answers, and efficient exploration of information.

Demo

The paper focuses on a vision-language model called InstructBLIP and explores the process of instruction tuning. The authors collect 26 datasets and categorize them for instruction tuning and zero-shot evaluation. They also introduce a method called instruction-aware visual feature extraction. The results show that InstructBLIP achieves the best performance among all models, surpassing BLIP-2 and Flamingo. When fine-tuned for specific tasks, InstructBLIP exhibits exceptional accuracy, such as 90.7% on the ScienceQA IMG task. Through qualitative comparisons, the study highlights InstructBLIP's superiority over other multimodal models, demonstrating its importance in the field of vision-language tasks.

Demo

The SD-CN-Animation project offers automated video stylization and text-to-video generation using StableDiffusion and ControlNet. It provides the ability to stylize videos automatically and generate new videos from text input, using various Stable Diffusion models as backbones. The project incorporates the 'RAFT' optical flow estimation algorithm to maintain animation stability and generate occlusion masks for frame generation. In text-to-video mode, it utilizes the 'FloweR' method for predicting optical flow from previous frames. The ControlNet model is recommended for better results in vid2vid mode.

ImageBind, the first AI model capable of binding data from six modalities at once, without the need for explicit supervision. By recognizing the relationships between these modalities — images and video, audio, text, depth, thermal and inertial measurement units (IMUs) — this breakthrough helps advance AI by enabling machines to better analyze many different forms of information, together.

UnpromptedControl is a tool used for guiding StableDiffusion models in image restoration and object removal tasks. By leveraging a simple hack, it allows for the restoration or removal of objects without requiring user prompts, leading to enhanced process efficiency. The tool uses ControlNet and StableDiffusionInpaintPipeline models to guide the inpainting process and restore the image to a more natural-looking state. However, the algorithm currently has limitations in processing images of people's faces and bodies.

Promptsandbox.io is a node-based visual canvas used for creating chatbots powered by OpenAI APIs. The platform has an intuitive drag-and-drop interface for creating dynamic chains of nodes that perform specific operations as part of the workflow. It is built using React and has various features such as integration with OpenAI APIs, document upload and retrieval, support for various block types, debugging tools, a chatbot gallery, and extensibility with additional node types. Promptsandbox.io provides a seamless experience for users to work with OpenAI APIs and build more complex chatbots.

An experiment where an autonomous GPT (Generative Pre-trained Transformer) agent is given access to a browser to perform tasks. For example adding text to a webpage and making restaurant reservations. gpt-assistant requires Node.js, an OpenAI API key, and a Postgres database.

DeepFloyd IF is a text-to-image model that utilizes the large language model T5-XXL-1.1 as a text encoder to generates intelligible and coherent image alongside with text. The model is capable of incorporating text into images, generating a high degree of photorealism and the ability to generate images with non-standard aspect ratios. It can also modify style, patterns, and details in images without the need for fine-tuning. DeepFloyd IF is modular, cascaded, and works in pixel space, utilizing diffusion models that inject random noise into data before reversing the process to generate new data samples from the noise.

Demo

Inpaint Anything is an innovative tool that seamlessly inpaints images, videos, and 3D scenes by allowing users to remove, fill, or replace objects with just a few clicks. It leverages advanced vision models like Segment Anything Model (SAM), LaMa, and Stable Diffusion (SD) to achieve these tasks. With support for multiple aspect ratios and resolutions up to 2K, Inpaint Anything offers a user-friendly interface for various modalities, including images, videos, and 3D scenes. The tool is continuously improving with new features and functionalities, making it an accessible and powerful solution for users seeking advanced inpainting capabilities.

WizardLM is a pre-trained language model that can follow complex instructions using Evol-Instruct - a method that uses language models instead of humans to automatically produce open-domain instructions of various difficulty levels. WizardLM is still in development and will continue to improve by training on larger scales, adding more training data, and innovating more advanced large-model training methods. To fine-tune WizardLM model, alpaca_evol_instruct_70k.json containing 70K instruction-following data generated from Evol-Instruct was used. In terms of human evaluation, WizardLM achieved significantly better results than Alpaca and Vicuna-7b models on diverse user-oriented instructions including difficult coding generation, debugging, math, reasoning, complex formats, academic writing, and extensive disciplines. Additionally, in the high-difficulty section of the human evaluation test set, WizardLM even outperforms ChatGPT, indicating its significant potential to handle complex instructions.

Demo Link
Demo Backup 1

Hugging Face, a company and AI community that provides free open source tools for machine learning and AI apps, has released HuggingChat, an open source ChatGPT clone that is available for anyone to use or download. The app is based on the Open Assistant Conversational AI Model by Large-scale Artificial Intelligence Open Network (LAION), a global non-profit organization dedicated to democratizing ML research and its applications. HuggingChat was trained with the OpenAssistant Conversations Dataset (OASST1), which was collected up to April 12, 2023, and used reinforcement learning from human feedback methodology to create a high quality human-annotated dataset. The dataset is the product of a worldwide crowdsourcing effort by over 13,000 volunteers.

Token Merging (ToMe) is a technique used to speed up transformers by merging redundant tokens, which helps reduce the workload for the transformer without compromising quality. The technique is applied to the underlying transformer blocks in Stable Diffusion, minimizing quality loss while preserving the speed-up and memory benefits. It works without training and can be used for any Stable Diffusion model, reducing the workload by up to 60%. ToMe for SD is not another efficient reimplementation of transformer modules, but an actual reduction of the total workload required to generate an image. The results of ToMe for SD show that it produces images similar to the originals, while being faster and using less memory, making it an efficient tool for image generation.