With the advance of text-to-image models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an affordable cost. Subsequently, there is a great demand for image animation techniques to further combine generated static images with motion dynamics. In this report, we propose a practical framework to animate most of the existing personalized text-to-image models once and for all, saving efforts in model-specific tuning. At the core of the proposed framework is to insert a newly initialized motion modeling module into the frozen text-to-image model and train it on video clips to distill reasonable motion priors. Once trained, by simply injecting this motion modeling module, all personalized versions derived from the same base T2I readily become text-driven models that produce diverse and personalized animated images. We conduct our evaluation on several public representative personalized text-to-image models across anime pictures and realistic photographs, and demonstrate that our proposed framework helps these models generate temporally smooth animation clips while preserving the domain and diversity of their outputs.

In Relation Inversion the goal is to capture and apply a relation from exemplar images to synthesize new scenes. While existing methods focus on capturing object appearances, this work explores the inversion of object relations. The proposed ReVersion framework learns a relation prompt from exemplar images using a pre-trained text-to-image diffusion model. The key insight is the "preposition prior," where real-world relation prompts can be activated with a set of basis prepositional words. The framework incorporates relation-steering contrastive learning and relation-focal importance sampling to enforce capturing object interactions and disentangling from appearances. The paper also introduces the ReVersion Benchmark, providing diverse exemplar images with different relations. Experimental results validate the superiority of the proposed approach in inverting relations compared to existing methods.

ToolLLM is a project that aims to create a large-scale dataset for training language models with tool-use capabilities. They collect instructions involving real-world APIs and develop a new annotation approach to improve efficiency. The project provides the ToolLLaMA model, which performs well in handling single-tool and complex multi-tool instructions. They also release the ToolLLaMA-7b, ToolLLaMA-7b-LoRA, and ToolLLaMA-2-7b models, along with a tool retriever. They evaluate the models using pass rate and preference metrics, showing good performance compared to other models. Overall, ToolLLM empowers language models to understand and use real-world tools effectively.

CoDeF is a new video representation involving a canonical content field and a temporal deformation field. The canonical content field captures the static contents of the entire video, while the temporal deformation field records the transformations from the canonical image to each frame. By optimizing these fields together, CoDeF can reconstruct a target video and support the application of image algorithms to the canonical image, which can then be propagated to the entire video using the temporal deformation field. This allows for video-to-video translation and keypoint tracking without training, resulting in improved cross-frame consistency compared to existing methods. CoDeF can also be applied to other tasks like point-based tracking, segmentation-based tracking, and video super-resolution.

Cheetor, a Transformer-based multi-modal large language model equipped with controllable knowledge re-injection. Cheetor demonstrates strong capabilities in reasoning over complicated interleaved vision-language instructions. It can identify connections between images, infer causes and reasons, understand metaphorical implications, and comprehend absurd objects through multi-modal conversations with humans.

Perfusion is a text-to-image personalization method that creatively portrays personalized objects with significant changes in appearance while maintaining their identity. It introduces a novel mechanism called "Key-Locking" to maintain high visual fidelity and allow creative control, combine personalized concepts, and keep a small model size. Perfusion achieves this through dynamic rank-1 updates to the underlying text-to-image model and locks new concepts' cross-attention Keys to their superordinate category. It also uses a gated rank-1 approach to control the influence of learned concepts during inference and combine multiple concepts, enabling a trade-off between visual fidelity and textual alignment. With just a 100KB trained model, Perfusion covers different operating points across the Pareto front without additional training. It outperforms strong baselines in both qualitative and quantitative terms, enabling personalization even in one-shot settings and displaying personalized object interactions in novel ways. The method also allows for efficient control of visual-textual alignment and demonstrates different variations of key-locking. Additionally, Perfusion concepts trained with a vanilla diffusion model can generalize to fine-tuned variants.

Pytorch implementation : https://github.com/lucidrains/perfusion-pytorch

A novel approach to personalizing text-to-image models using diffusers. The current methods for personalization face challenges like long training times, high storage requirements, and loss of identity. To address these issues, the proposed method utilizes an encoder-based domain-tuning approach. This approach involves underfitting on a large set of concepts from a given domain to improve generalization and enable quick addition of novel concepts from the same domain. The method consists of two components: an encoder that maps a target concept image to a word embedding representing the concept, and regularized weight-offsets for the text-to-image model that effectively incorporate additional concepts. By leveraging these components, the model can be personalized with a single image and as few as 5 training steps, significantly reducing personalization time while maintaining quality.

GLIGEN is a new approach that enhances text-to-image generation models by allowing them to be conditioned on additional grounding inputs. It achieves impressive performance in generating images from text by including caption and bounding box condition inputs. GLIGEN takes advantage of a pre-trained model by freezing its weights and adding trainable layers to incorporate grounding information. This enables the model to have a better understanding of spatial relationships and concepts. GLIGEN outperforms other models in generating images based on specific instructions and can handle counterfactual scenarios. It can also use reference images for more detailed and stylistic output. Additionally, GLIGEN has the capability to inpaint images based on provided bounding boxes. Overall, GLIGEN improves the controllability and versatility of text-to-image generation models.

Composer introduces a generative model that addresses the limited controllability of existing large-scale models. By decomposing images into representative factors and utilizing a diffusion model to recompose them with these factors as conditions, Composer allows for flexible control over the output image while maintaining quality and creativity. It supports various levels of conditions, such as text description, depth map, sketch, and color histogram. Furthermore, Composer serves as a general framework for classical generative tasks without requiring retraining. The composition results demonstrate the ability to combine elements like text and depth, masked image and text, sketch, depth, and embedding, and more. Additionally, the model facilitates image variations, interpolations, reconfigurations, and region-specific editing, as well as tasks like image translation, style transfer, pose transfer, and virtual try-on.

"Magic123" is a two-stage solution for generating high-quality 3D meshes from single images. It uses 2D and 3D priors to optimize a neural radiance field in the first stage, creating a coarse geometry. The second stage utilizes a memory-efficient mesh representation to produce a high-resolution mesh with appealing texture. Through reference view supervision and diffusion priors, the approach generates novel views. The system incorporates a tradeoff parameter for controlling the balance between exploration and precision in the generated geometry.

The gpt-prompt-engineer tool is a powerful solution for prompt engineering, enabling users to experiment and find the optimal prompt for GPT-4 and GPT-3.5-Turbo language models. It generates a variety of prompts based on the provided use-case and test cases, and then tests and ranks them using an ELO rating system. Additionally, there is a specific classification version that evaluates test case correctness and provides scores for each prompt. The tool also supports optional logging to Weights & Biases, allowing for tracking of configurations and prompt performance.

A new algorithm called Restart has been developed to improve the speed and quality of generative processes. These processes use complex differential equations, which usually has a trade-off between quick results and accuracy. Existing methods like ODE and SDE samplers either work fast but reach a performance limit or offer better quality at a slower rate. The Restart algorithm outperforms these existing methods by better managing these sampling errors. In tests, it produced faster and higher quality results, surpassing SDE results on different platforms and maintaining better balance in large-scale model testing.

AI company Optic has developed a web tool called AI or Not, which aims to combat misinformation spread through AI-generated images. The tool scans images and quickly determines whether they were generated by artificial intelligence or by humans. Optic claims that its algorithms provide highly accurate results with a precision rate of 95%. However, users may have concerns about privacy when uploading images to the tool. Optic states that uploaded images and URLs are not stored on its servers longer than necessary and that they adhere to data protection regulations. By analyzing images and detecting signs of AI generation, the company aims to improve its algorithms and machine learning techniques.

A new method for inverting Deep Diffusion Probabilistic Models (DDPMs) by using noise maps to encode image structure and facilitate image editing. The authors demonstrate that this method can be used for text-based image editing tasks and can generate diverse results for any given image and text. The article compares this method to other image editing techniques such as P2P, DDIM inversion, and Plug-and-Play (PnP) and shows how this new method can perform better in terms of fidelity, adherence to target prompts, and support of diversity among generated outputs.

Demo

Stable Diffusion Sketch is an Android app that allows users to create colorful sketches and enhance them using various modes of Stable Diffusion. Users can create new paintings using blank canvas, camera capture, Stable Diffusion txt2img, and images shared from other apps. The app also offers preset modes, custom modes, painting tools, undo/redo functionality, and options to adjust prompt prefix, postfix, and negative prompt values. There are three canvas aspect ratios available: landscape, portrait, and square. Users can also enlarge the image using the upscaler feature or delete them from the main screen. Grouping related sketches is also possible, and the app supports multiple ControlNet.

DreamSim is a new algorithm designed to compare images holistically by assessing images on their mid-level attributes such as object layout, position, and semantic context. Unlike standard image comparison algorithms that only compare images based on colors and textures, DreamSim evaluates an image on its overall appearance and composition, giving it a more well-rounded comparison. The algorithm uses synthetic data to train its network but can still perform well on real images. A key point is that DreamSim showed better results than existing comparison algorithms in identifying objects and overall semantic context within an image. In summary, DreamSim is a valuable image comparison tool that captures features beyond low-level colors and textures while providing a stronger foundation for visual analysis.

WizardCoder is a new model that utilizes complex instruction fine-tuning to enhance Code Large Language Models (Code LLMs) for code-related tasks. This model has been tested on four benchmarks and has shown superior performance compared to other Code LLMs and even outperforms the largest closed LLMs on some benchmarks.

UniControl is a new generative model that combines many condition-to-image (C2I) tasks into one framework while still allowing for language prompts. This enables precise image generation where visual conditions influence the generated structures and language prompts guide the style and context. UniControl can handle diverse visual conditions through pretrained text-to-image diffusion models and a task-aware HyperNet that modulates these models. UniControl outperforms single-task models of similar sizes and marks a significant advancement in controlled visual generation.

Stability AI has released an open-source version of its DreamStudio text-to-image consumer application called StableStudio. The company intends to work with the broader community to create a world class user interface for generative AI that users fully control. DreamStudio was first conceived as an animation studio that shifted its focus to image generation with the arrival of Stable Diffusion in the summer of 2022. StableStudio enable local-first development through WebGPU and a desktop installation of its Stable Diffusion tool. It is also compatible with ControlNet tools and local inference through AUTOMATIC1111 stable-diffusion-webui tool.

aiNodes is a Python-based AI image/motion picture generator node engine that facilitates creativity in the creation of images and videos. The engine is fully modular and can download node packs on runtime. It also features RIFE and FILM interpolation integration, coloured background drop, and node-creation with IDE annotations. The installation process requires Python 3.10, Git, and an nVidia GPU with CUDA and drivers installed. AiNodes engine is an open-source desktop AI-based image/motion generator that supports various features such as Deforum, Stable Diffusion, Upscalers, Kandinsky, ControlNet, LORAs, Ti Embeddings, Hypernetworks, Background Separation, Human matting/masking, and Compositing, among others.