记要点

TypeError("'ObjectId' object is not iterable") 解决方法

Wed, 05 Jun 2024 21:43:54 +0800

‘ObjectId’ object is not iterable

使用fastapi，当返回结果里面包含mongodb的id，也就是ObjectId类型的时候，就会报错： TypeError("‘ObjectId’ object is not iterable")。

直接google搜索下，可以得到几种方法：

先使用str()方法将ObjectId转换成字符串
使用bson内置的json_util.dumps()方法将ObjectId转换成字符串
删除ObjectId字段
定义一个JSONEncoder类，将ObjectId转换成字符串

import json
from bson import ObjectId

class JSONEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, ObjectId):
            return str(o)
        return json.JSONEncoder.default(self, o)

JSONEncoder().encode(analytics)

json.dumps(my_obj, default=str)
如果是老版本的fastapi

1
2
3

import pydantic
from bson import ObjectId
pydantic.json.ENCODERS_BY_TYPE[ObjectId]=str

看起来，第6种方法是比较优雅的，但是，对于没有使用pydantic的返回结果，就不适用了。而且，新版本的pydantic也不是这样的使用方法了。

其实，不管什么类型，只要是json不支持的，都会报错，比如datetime类型也会报错。但是为什么fastapi返回datetime类型的时候不会报错呢？因为fastapi内部已经做了处理，将datetime类型转换成了字符串类型。

通过报错信息。

│ /xxxx/site-packages/fastapi/encoders.py:332 in          │
│ jsonable_encoder                                                                                 │
│                                                                                                  │
│   329 │   │   │   data = vars(obj)                                                               │
│   330 │   │   except Exception as e:                                                             │
│   331 │   │   │   errors.append(e)                                                               │
│ ❱ 332 │   │   │   raise ValueError(errors) from e                                                │
│   333 │   return jsonable_encoder(                                                               │
│   334 │   │   data,

我们可以看到是在fastapi的encoders.py里面报错的。

DesignEdit: Multi-Layered Latent Decomposition and Fusion for Unified & Accurate Image Editing

Fri, 15 Mar 2024 22:31:50 +0800

以下是论文翻译

摘要

近年来,如何实现精确的图像编辑越来越受到关注,特别是考虑到文本到图像生成模型的显著成功。为了将各种空间感知图像编辑能力统一到一个框架中,我们采用了设计领域的层概念,通过各种操作灵活地操作对象。关键的洞见是将空间感知图像编辑任务转换为两个子任务的组合:多层潜在分解和多层潜在融合。

首先,我们将源图像的潜在表示分割为多个层,其中包括几个对象层和一个需要可靠修复的不完整背景层。为了避免额外的调整,我们进一步探索了自注意力机制中的内在修复能力。我们引入了一种键掩码自注意力方案,该方案可以将周围的上下文信息传播到掩码区域,同时减轻其对掩码外区域的影响。

其次,我们提出了一种指令引导的潜在融合,将多层潜在表示粘贴到画布上。我们还在潜伏空间中引入了伪影抑制方案,以提高修复质量。由于这种多层表示固有的模块化优势,我们可以实现准确的图像编辑,并证明我们的方法始终优于最新的空间编辑方法,如Self-Guidance和DiffEditor。

最后,我们展示了我们的方法是一个统一的框架,可以在六个以上的不同编辑任务上支持各种准确的图像编辑任务。

图 1：视觉设计图像编辑实例。我们的方法借助无需训练的统一框架推动了一系列图像编辑操作，从而实现设计图像的精准空间感知式编辑。我们的方法能够同步操作不同对象，并同时施行各种操作。所有结果均通过一种扩散去噪过程生成。

1. 简介

图 2. 我们的方法与 Self-Guidance 和 DiffEditor 之间的对比。我们在（a）中汇报了图像质量和编辑准确性的胜率对比。针对每次对比，我们选取了 10 个包含移动和调整大小等多种操作的示例。要求用户就图像质量和编辑准确性这两方面进行投票。“平局”选项表示效果相同。我们收集了 73 名用户的答案，每个指标总计有 1460 票。

尽管通过训练大规模的文本到图像扩散模型在图像生成方面取得了巨大成就[18、23、27、26、10、15]，正如近期具有开创性的研究，包括 SDXL[21]、DALL·E3[19、3]和 Ideogram1 所展示的那样，这些模型面临着需要具备数字能力或空间排列能力的提示所带来的挑战。例如，图 1（a）展示了由 DALL·E3 生成的一幅引人入胜的故事书设计图像，其文字提示描述了“三只小猪”的故事。我们发现图中有四只猪，这与文字提示中的“三只猪”不相符。为了克服这些限制，前沿的努力[9、17、28、16]已经致力于开发精确的空间感知图像编辑技术，旨在弥合用户期望与初始生成结果之间的差距。

与之前的方法[9、17、28、16]需要结合为不同编辑任务设计的多种编辑指导方案，并通过额外的反向传播来更新潜在表示不同，我们为精确的空间感知图像编辑任务提出了一种无需训练、仅向前、且统一的框架。我们的方法将大多数具有代表性的空间感知编辑任务转化为一个双重过程。该过程首先根据精确的用户指令和层分割掩码来分解源图像的多层潜在表示，然后按照准确的布局排列将这些表示集成到目标图像中。为了确保多个图像层的精确空间感知编辑质量，我们根据目标布局排列明确融合多层潜在，以形成目标潜在表示。此外，我们支持利用 GPT-4V[34]的推理和视觉规划能力来协助制定用户指令并生成（和完善）准确的布局安排。

我们明确了执行多层潜在分解与融合过程中的关键挑战，并提出了以下三个非平凡的技术贡献：

首先，我们发现执行多层潜在分解的关键挑战之一在于生成高品质的背景层。该层不仅要对原始层保持忠实，还需修复分解对象层中的不完整区域。我们没有采用现有的修复方法，而是引入了一种极为简单却更可靠的self-attention[31]key-masking方法，其能一直达成更佳的修复质量。
其次，我们需要解决的另一个挑战是修复区域可能会受到一些无关区域的负面影响，从而产生瑕疵。因此，我们提出了一种瑕疵抑制方案以进一步提升修复质量。
第三，我们通过将各种图像编辑任务分解为两个基本子任务，引入了一个统一的框架：多层潜在分解与多层潜在融合。

我们开展了广泛的用户研究，以评估我们方法的图像编辑质量，并将其与 Self-Guidance[9]和 DiffEditor[17]的最新进展进行对比。结果在图 2 中展示，呈现了在两个关键维度上的胜率：图像质量和编辑保真度。我们的研究结果表明，我们的方法在各类编辑任务，如对象移动和尺寸调整上，明显优于这两种基准方法。

此外，我们还将我们的方法应用于一系列具有挑战性的设计图像编辑任务，例如对象移除、尺寸调整、移动、重复、翻转、相机平移、缩小、合成多个图像以及编辑版式或装饰等。我们期望能推动更精确的空间感知图像编辑技术的进一步发展。

图 3：展示我们方法的整体框架：在多层分解阶段，给定用户的编辑指令和源图像，我们首先使用 GPT-4V 执行指令规划，生成一组详细的分层编辑指令。接着，我们将源图像分割成多个图像层，包括需要额外进行修复的背景层（由新颖的关键掩码自关注方案来实现）以及要操作对象的其他对象层。

对于多层融合阶段，我们依据层的顺序和逐层指令的顺序，依次将它们粘贴到潜在空间的画布上。我们进一步应用多个去噪步骤来协调融合的多层潜在表示。此外，我们还进行瑕疵抑制以提升背景修复质量。

2. 相关工作

2.1 潜在扩散模型

潜在扩散模型[24]（LDM）通过在压缩的潜在空间操作，而非在图像层面操作，为生成式建模领域引入了一种开创性的方法。此方法加快了生成过程，降低了计算需求。近来，采用潜在扩散模型架构并经过大量数据训练的大规模条件扩散模型[24, 21, 27]，能够生成细节丰富且视觉上引人注目的图像。像混合潜在扩散[2]等图像编辑方法表明，在潜在空间操作可实现比在图像层面操作[1]更快的推理速度和更高的精度，来完成局部图像调整。在我们的工作中，我们采用了最先进的大规模文本到图像 LDM，即具有 U-Net 结构[25]的稳定扩散[24, 21]，以进一步探索用于空间感知图像编辑的潜在操作。

2.2 引导驱动的空间感知图像编辑

空间编辑是指通过考虑图像内的空间上下文和关系来修改图像。与就地编辑方法[12, 5, 13, 4]不同。受扩散模型分类器引导策略的启发，无训练布局控制[7]和 Boxdiff[32]利用位置信息损失来约束潜在空间，以实现带有布局控制的空间感知图像生成。自我引导[9]将分类器引导引入基于扩散的图像编辑，以完成如对象移动和调整大小等任务。受 DragGAN[20]启发的 DragonDiffusion[16]，将基于拖动的图像编辑任务融入扩散模型，扩展到更多空间感知的编辑任务，如使用对象蒙版等图像提示的对象移动和调整大小。DiffEditor[17]改进了 DragonDiffusion，在精确图像编辑任务中达到了最先进的结果。

WURSTCHEN

Fri, 15 Mar 2024 17:31:50 +0800

论文原文

ABSTRACT
We introduce Wurstchen, a novel architecture for text-to-image synthesis that ¨
combines competitive performance with unprecedented cost-effectiveness for largescale text-to-image diffusion models. A key contribution of our work is to develop
a latent diffusion technique in which we learn a detailed but extremely compact
semantic image representation used to guide the diffusion process. This highly
compressed representation of an image provides much more detailed guidance
compared to latent representations of language and this significantly reduces the
computational requirements to achieve state-of-the-art results. Our approach also
improves the quality of text-conditioned image generation based on our user
preference study. The training requirements of our approach consists of 24,602
A100-GPU hours – compared to Stable Diffusion 2.1’s 200,000 GPU hours. Our
approach also requires less training data to achieve these results. Furthermore,
our compact latent representations allows us to perform inference over twice as
fast, slashing the usual costs and carbon footprint of a state-of-the-art (SOTA)
diffusion model significantly, without compromising the end performance. In
a broader comparison against SOTA models our approach is substantially more
efficient and compares favorably in terms of image quality. We believe that this
work motivates more emphasis on the prioritization of both performance and
computational accessibility
1 INTRODUCTION
State-of-the-art diffusion models (Ho et al., 2020; Saharia et al., 2022; Ramesh et al., 2022) have
advanced the field of image synthesis considerably, achieving remarkable results that closely approxi-
∗
equal contribution
1
arXiv:2306.00637v2 [cs.CV] 29 Sep 2023
Figure 1: Text-conditional generations using Wurstchen. Note the various art styles and aspect ratios. ¨
mate photorealism. However, these foundation models, while impressive in their capabilities, carry
a significant drawback: they are computationally demanding. For instance, Stable Diffusion (SD)
1.4, one of the most notable models in the field, used 150,000 GPU hours for training (Rombach &
Esser, 2022). While more economical text-to-image models do exist (Ding et al., 2021; 2022; Tao
et al., 2023; 2022), the image quality of these models can be considered inferior in terms of lower
resolution and overall aesthetic features.
The core dilemma for this discrepancy is that increasing the resolution also increases visual complexity
and computational cost, making image synthesis more expensive and data-intensive to train. Encoderbased Latent Diffusion Models (LDMs) partially address this by operating on a compressed latent
space instead of directly on the pixel-space (Rombach et al., 2022), but are ultimately limited by how
much the encoder-decoder model can compress the image without degradation (Richter et al., 2021a).
Against this backdrop, we propose a novel three-stage architecture named ”Wurstchen”, which ¨
drastically reduces the computational demands while maintaining competitive performance. We
achieve this by training a diffusion model on a very low dimensional latent space with a high
compression ratio of 42:1. This very low dimensional latent-space is used to condition the second
generative latent model, effectively helping it to navigate a higher dimensional latent space of a
Vector-quantized Generative Adversarial Network (VQGAN), which operates at a compression ratio
of 4:1. More concretely, the approach uses three distinct stages for image synthesis (see Figure 2):
initially, a text-conditional LDM is used to create a low dimensional latent representation of the
image (Stage C). This latent representation is used to condition another LDM (Stage B), producing
a latent image in a latent space of higher dimensionality. Finally, the latent image is decoded by a
VQGAN-decoder to yield the full-resolution output image (Stage A).
Training is performed in reverse order to the inference (Figure 3): The initial training is carried out
on Stage A and employs a VQGAN to create a latent space. This compact representation facilitates
learning and inference speed (Rombach et al., 2022; Chang et al., 2023; Rampas et al., 2023). The
next phase (Stage B) involves a first latent diffusion process (Rombach et al., 2022), conditioned on
the outputs of a Semantic Compressor (an encoder operating at a very high spatial compression rate)
and on text embeddings. This diffusion process is tasked to reconstruct the latent space established
by the training of Stage A, which is strongly guided by the detailed semantic information provided by
the Semantic Compressor. Finally, for the construction of Stage C, the strongly compressed latents
of the Semantic Compressor from Stage B are used to project images into the condensed latent
space where a text-conditional LDM (Rombach et al., 2022) is trained. The significant reduction in
space dimensions in Stage C allows for more efficient training and inference of the diffusion model,
considerably reducing both the computational resources required and the time taken for the process.
Our proposed Wurstchen model thus introduces a thoughtfully designed approach to address the ¨
high computational burden of current state-of-the-art models, providing a significant leap forward
in text-to-image synthesis. With this approach we are able to train a 1B parameter Stage C textconditional diffusion model within approximately 24,602 GPU hours, resembling a 8x reduction in
computation compared to the amount SD 2.1 used for training (200,000 GPU hours), while showing
similar fidelity both visually and numerically. Throughout this paper, we provide a comprehensive
evaluation of Wurstchen’s efficacy, demonstrating its potential to democratize the deployment & ¨
training of high-quality image synthesis models.
2
Figure 2: Inference architecture for text-conditional image generation.
Our main contributions are the following:
1. We propose a novel three-stage architecture for text-to-image synthesis at strong compression
ratio, consisting of two conditional latent diffusion stages and a latent image decoder.
2. We show that by using a text-conditional diffusion model in a strongly compressed latent
space we can achieve state-of-the-art model performance at a significantly reduced training
cost and inference speed.
3. We provide comprehensive experimental validation of the model’s efficacy based on automated metrics and human feedback.
4. We are publicly releasing the source code and the entire suite of model weights.
2 RELATED WORK
2.1 CONDITIONAL IMAGE GENERATION
The field of image generation guided by text prompts has undergone significant progression in recent
years. Initial approaches predominantly leveraged Generative Adversarial Networks (GANs) (Reed
et al., 2016; Zhang et al., 2017). More recently, however, a paradigm shift in the field of image
generation towards diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) has occurred. These
approaches, in some cases, have not only met but even exceeded the performance of GANs in both
conditional and unconditional image generation (Dhariwal & Nichol, 2021). Diffusion models put
forth a score-based scheme that gradually eliminates perturbations (e.g., noise) from a target image,
with the training objective framed as a reweighted variational lower-bound. Next to diffusion models,
another dominant choice for training text-to-image models is transformers. In their early stages,
transformer-based models utilized an autoregressive approach, leading to a significant slowdown
in inference due to the requirement for each token to be sampled individually. Current strategies,
however, employ a bidirectional transformer (Ding et al., 2022; Chang et al., 2022; 2023) to address
the challenges that traditional autoregressive models present. As a result, image generation can be
executed using fewer steps, while also benefiting from a global context during the generative phase.
Other recent work has shown that convolution-based approaches for image generation can yield
similar results (Rampas et al., 2023).
2.2 COMPRESSED LATENT SPACES
The majority of approaches in the visual modality of generative models use some way to train at a
smaller space, followed by upscaling to high resolutions, as training at large pixel resolutions can
become exponentially more expensive with the size of images. For text-conditional image generation,
there are two established categories of approaches: encoder-based and upsampler-based. LDMs
(Rombach et al., 2022), DALL-E (Ramesh et al., 2021), CogView (Ding et al., 2021; 2022), MUSE
(Chang et al., 2023) belong to the first category and employ a two-stage training process. Initially,
an autoencoder (Rumelhart et al., 1985) is trained to provide a lower-dimensional, yet perceptually
equivalent, representation of the data. This representation forms the basis for the subsequent training
of a diffusion or transformer model. Eventually, generated latent representations can be decoded
with the decoder branch of the autoencoder to the pixel space. The result is a significant reduction in
computational complexity for the diffusion/sampling process and efficient image decoding from the
latent space using a single network pass. On the contrary, upsampler-based methods generate images
at low resolution in the pixel space and use subsequent models for upscaling the images to higher
3
resolution. UnClip (Ramesh et al., 2022) and Imagen (Saharia et al., 2022) both generate images
at 64x64 and upscale using two models to 256 and 1024 pixels. The former model is the largest in
terms of parameter count, while the latter models are smaller due to working at higher resolution and
only being responsible for upscaling.
2.3 CONDITIONAL GUIDANCE
The conditional guidance of models in text-based scenarios is typically facilitated through the
encoding of textual prompts via a pretrained language model. Two major categories of text encoders
are employed: contrastive text encoders and uni-modal text encoders. Contrastive Language-Image
Pretraining (CLIP) (Radford et al., 2021) is a representative of the contrastive multimodal models
that strives to align text descriptions and images bearing semantic resemblance within a common
latent space. A host of image generation methodologies have adopted a frozen CLIP model as their
exclusive conditioning method in recent literature. The hierarchical DALL-E 2 by Ramesh et al.
(2022) specifically harnesses CLIP image embeddings as input for their diffusion model, while a
’prior’ performs the conversion of CLIP text embeddings to image embeddings. SD (Rombach et al.,
2022), on the other hand, makes use of un-pooled CLIP text embeddings to condition its LDM. In
contrast, the works of Saharia et al. (2022), Liu et al. (2022a) and Chang et al. (2023) leverage a
large, uni-modal language model such as T5 (Raffel et al., 2020) or ByT5 (Xue et al., 2022) that can
encode textual prompts with notable accuracy, leading to image generations of superior precision in
terms of composition, style, and layout.
3 METHOD
Our method comprises three stages, all implemented as deep neural networks. For image generation,
we first generate a latent image at a strong compression ratio using a text-conditional LDM (Stage C).
Subsequently, this representation is transformed to a less-compressed latent space by the means of a
secondary model which is tasked for this reconstruction (Stage B). Finally, the tokens that comprise
the latent image in this intermediate resolution are decoded to yield the output image (Stage A).
The training of this architecture is performed in reverse order, starting with Stage A, then following
up with Stage B and finally Stage C (see Figure 3). Text conditioning is applied on Stage C using
CLIP-H (Ilharco et al., 2021). Details on the training procedure can be found in Appendix E.
3.1 STAGE A AND B
It is a known and well-studied technique to reduce the computational burden by compressing data
into a smaller representation(Richter et al., 2021a;b; Chang et al., 2022). Our approach follows
this paradigm, too, and makes use of Stages A & B to achieve a notably higher compression than
usual. Let H × W × C be the dimensions of images. A spatial compression maps images to a
latent representation with a resolution of h × w × z with h = H/f, w = W/f, where f defines the
compression rate. Common approaches for modeling image synthesis use a one-stage compression
between f4 and f16 (Esser et al., 2021; Chang et al., 2023; Rombach et al., 2022), with higher factors
usually resulting in worse reconstructions. Our Stage A consists of a f4 VQGAN (Esser et al., 2021)
with parameters Θ and initially encodes images X ∈ R
3×1024×1024 into 256 × 256 discrete tokens
from a learned codebook of size 8,192.
Xq = fΘ(X)
The network is trained as described by Esser et al. and tries to reconstruct the image based on the
quantized latents, so that:
f
−1
Θ (fΘ (X)) = f
−1
Θ (Xq) ≈ X
where f
−1
Θ resembles the decoder part of the VQGAN.
Afterward, the quantization is dropped from Stage A, and Stage B is trained in the unquantized
latent space of the Stage A-encoder as a conditioned LDM. In stage B, we utilize a Semantic
Compressor, i.e., an encoder-type network that is tasked to create latent representations at a strong
spatial compression rate that can be used to create a latent representation to guide the diffusion
process. The unquantized image embeddings are noised following an LDM training procedure. The
4
Figure 3: Training objectives of our model. Initially, a VQGAN is trained. Secondly, Stage B is trained
as a diffusion model inside Stage A’s latent space. Stage B is conditioned on text-embeddings and the
output of the Semantic Compressor, which produces strongly downsampled latent representations
of the same image. Finally, Stage C is trained on the latents of the Semantic Compressor as a
text-conditional LDM, effectively operating on a compression ratio of 42 : 1.
noised representation X˜
t, together with the visual embeddings from the Semantic Compressor, Csc,
text conditioning Ctext and the timestep t are given to the model.
The highly compressed visual embeddings extracted by the Semantic Compressor will act as an
interface for Stage C, which will be trained to generate them. The embeddings will have a shape
of R
1280×24×24 obtained by encoding images with shape X ∈ R
3×786×786. We use simple bicubic
interpolation for the resizing of the images from 1024×1024 to 786×786, which is a sufficiently high
resolution to fully utilize the parameters of the Semantic Compressor (Richter et al., 2023; Richter &
Pal, 2022), while also reducing the latent representation size. Moreover, we further compress the
latents with a 1 × 1 convolution that normalizes and projects the embeddings to Csc ∈ R
16×24×24
.
This compressed representation of the images is given to the Stage B decoder as conditioning to
guide the decoding process.
X¯
0 = fϑ(X˜
t, Csc, Ctext, t)
By conditioning Stage B on low-dimensional latent representations, we can effectively decode images
from a R
16×24×24 latent space to a resolution of X ∈ R
3×1024×1024, resulting in a total spatial
compression of 42:1.
We initialized the Semantic Compressor with weights pre-trained on ImageNet, which, however, does
not capture the broad distribution of images present in large text-image datasets and is not well-suited
for semantic image projection, since it was trained with an objective to discriminate the ImageNet
categories. Hence we updated the weights of the Semantic Compressor during training, establishing a
latent space with high-precision semantic information. We use Cross-Attention (Vaswani et al., 2017)
for conditioning and project Csc (flattened) to the same dimension in each block of the model and
concatenate them. Furthermore, during training Stage B, we intermittently add noise to the Semantic
Compressor’s embeddings, to teach the model to understand non-perfect embeddings, which is likely
to be the case when generating these embeddings with Stage C. Lastly, we also randomly drop Csc to
be able to sample with classifier-free-guidance (Ho & Salimans, 2022) during sampling.
5
3.2 STAGE C
After Stage A and Stage B were trained, training of the text-conditional last stage started. In our
implementation, Stage C consists of 16 ConvNeXt-block (Liu et al., 2022b) without downsampling,
text and time step conditionings are applied after each block via cross-attention. We follow a standard
diffusion process, applied in the latent space of the finetuned Semantic Compressor. Images are
encoded into their latent representation Xsc = Csc, representing the target. The latents are noised by
using the following forward diffusion formula:
Xsc,t =
√
α¯t · Xsc +
√
1 − α¯t · ϵ
where ϵ represents noise from a zero mean unit variance normal distribution. We use a cosine schedule
(Nichol & Dhariwal, 2021) to generate α¯t and use continuous timesteps. The diffusion model takes
in the noised embeddings Xsc,t, the text conditioning Ctext and the timestep t. The model returns
the prediction for the noise in the following form:
ϵ¯ =
Xsc,t − A
| 1 − B | +1e−5
with
A, B = fθ(Xsc,t, Ctext, t)
We decided to formulate the objective as such, since it made the training more stable. We hypothesize
this occurs because the model parameters are initialized to predict 0 at the beginning, enlarging the
difference to timesteps with a lot of noise. By reformulating to the A & B objective, the model
initially returns the input, making the loss small for very noised inputs. We use the standard meansquared-error loss between the predicted noise and the ground truth noise. Additionally, we employ
the p2 loss weighting (Choi et al., 2022):
p2(t)· || ϵ − ϵ¯ ||2
where p2(t) is defined as 1−α¯t
1+ ¯αt
, making higher noise levels contribute more to the loss. Text
conditioning Ctext are dropped randomly for 5% of the time and replaced with a null-label in order
to use classifier-free-guidance (Ho & Salimans, 2022)
3.3 IMAGE GENERATION (SAMPLING)
A depiction of the sampling pipeline can be seen in Figure 2. Sampling starts at Stage C, which is
primarily responsible for image-synthesis (see Appendix D), from initial random noise Xsc,τC =
N (0, I). We use the DDPM (Ho et al., 2020) algorithm to sample the Semantic Compressor latents
conditioned on text-embeddings. To do so, we run the following operation for τC steps:
Xˆ
sc,t−1 =
1
√
αt
· (Xˆ
sc,t −
1 − αt √
1 − α¯t
ϵ¯) + r
(1 − αt)
1 − α¯t−1
1 − α¯t
ϵ
We denote the outcome as X¯
sc which is of shape 16 × 24 × 24. This output is flattened to a shape
of 576 × 16 and given as conditioning, along with the same text embeddings used to sample X¯
sc,
to Stage B. This stage operates at 4 × 256 × 256 unquantized VQGAN latent space. We initialize
Xq,τB to random tokens drawn from the VQGAN codebook. We sample X˜ for τB steps using the
standard LDM scheme.
X˜
t−1 = fϑ(X˜
t, Csc, Ctext, t)
Finally X˜ is projected back to the pixel space using the decoder f
−1
Θ of the VQGAN (Stage A):
X¯ = f
−1
Θ (X˜ )
6
3.4 MODEL DECISIONS
Theoretically, any feature extractor could be used as backbone for the Semantic Compressor. However,
we hypothesize that it is beneficial to use a backbone that already has a good feature representation of
a wide variety of images. Furthermore, having a small Semantic Compressor makes training of Stage
B & C faster. Finally, the feature dimension is vital. If it is excessively small, it may fail to capture
sufficient image details or will underutilize parameters (Richter & Pal, 2022); conversely, if it is overly
large, it may unnecessarily increase computational requirements and extend training duration (Richter
et al., 2021a). For this reason, we decided to use an ImageNet1k pre-trained EfficientV2 (S) as the
backbone for our Semantic Compressor, as it combines high compression with well generalizing
feature representations and computational efficiency.
Furthermore, we deviate in Stage C from the U-Net standard architecture. As the image is already
compressed by a factor of 42, and we find further compression harmful to the model quality. Instead,
the model is a simple sequence of 16 ConvNeXt blocks (Liu et al., 2022b) without downsampling.
Time and text conditioning is applied after each block.
4 EXPERIMENTS AND EVALUATION
To demonstrate Wurstchen’s capabilities on text-to-image generation, we trained an 18M parameter ¨
Stage A, a 1B parameter Stage B and a 1B parameter Stage C. We employed an EfficientNet2-Small
as Semantic Compressor (Tan & Le, 2020) during training. Stage B and C are conditioned on
un-pooled CLIP-H (Ilharco et al., 2021) text-embeddings. The setup is designed to produce images
of variable aspect ratio with up to 1538 pixels per side. All stages were trained on subsets of the
improved-aesthetic LAION-5B (Schuhmann et al., 2022) dataset.
All the experiments use the standard DDPM (Ho et al., 2020) algorithm to sample latents in Stage B
and C. Both stages also make use of classifier-free-guidance (Ho & Salimans, 2022) with guidance
scale w. We fix the hyperparameters for Stage B sampling to τB = 12 and w = 4, Stage C uses
τC = 60 for sampling. Images are generated using a 1024 × 1024 resolution.
Baselines To better assess the efficacy of our architecture, we additionally train a U-Net-based 1B
parameter LDM on SD 2.1 first stage and text-conditioning model. We refer to this model as Baseline
LDM, it is trained for ≈ 25,000 GPU-hours (same as Stage C) using an 512 × 512 input resolution.
Additionally, we evaluate our model against various state-of-the-art models that were publicly
available at the time of writing (see Tables 1 and Table 2). All these models were used in their
respective default configuration for text-to-image synthesis. Whenever possible, the evaluation
metrics published by the original authors were used.
Evaluation Metrics We used the Frechet Inception Distance (FID) (Heusel et al., 2018) and ´
Inception Score (IS) to evaluate all our models on COCO-30K, similar to (Tao et al., 2023; Ding
et al., 2021; 2022). For evaluating the FID score, all images were downsampled to 256 × 256 pixels
to allow for a fair comparison between other models in the literature. However, both metrics suffer
from inconsistencies and are known to be not necessarily well correlated with the aesthetic quality
perceived by humans (Podell et al. (2023); Ding et al. (2021; 2022), see also Appendix B). For this
reason, we chose PickScore (Kirstain et al., 2023) as our primary automated metric. PickScore is
designed to imitate human preferences, when selecting from a set of images given the same prompt.
We applied PickScore to compare Wurstchen to various other models on various datasets. We provide ¨
the percentage of images, where PickScore preferred the image of Wurstchen over the image of the ¨
other model. To also evaluate the environmental impact of our model we estimated the carbon emitted
during training based on the work of (Lacoste et al., 2019).
Finally, we also conducted a study with human participants, where the participants chose between
two images from two different models given the prompt.
Datasets To assess the zero-shot text-to-image capabilities of our model, we use three distinct sets
of captions alongside their corresponding images. The COCO-validation is the de-facto standard
dataset to evaluate the zero-shot performance for text-to-image models. For MS COCO we generate
30,000 images based on prompts randomly chosen from the validation set. We refer to this set of
7
Figure 4: Inference time for 1024 × 1024 images on an A100-GPUs. Left plot shows performance
without specific optimization, right plot shows performance using torch.compile().
Table 1: Evaluation of Image Quality on MS-COCO and Localized Narratives (Pont-Tuset et al.,
2020) using the PickScore (Kirstain et al., 2023) to binary select images generated from the same
captions by two different models. Wurstchen outperforms all models of equal and smaller size, ¨
despite Stable Diffusion models using a significantly higher compute budget.
PickScore(COCO-30k) ↑
Model Baseline LDM (ours) DF-GAN GALIP SD 1.4 SD 2.1 SD XL
(train cost) (≈25,000 gpu-h) - - (150.000 gpu-h) (200.000 gpu-h) -
Wurstchen ¨
96.5% 99.8% 98.1% 78.1% 64.4% 39.4% (24,602 gpu-h)
PickScore (Localized Narratives-COCO-5K) ↑
Model Baseline LDM (ours) DF-GAN GALIP SD 1.4 SD 2.1 SD XL
(train cost) (≈25,000 gpu-h) - - (150.000 gpu-h) (200.000 gpu-h) -
Wurstchen ¨
96.6% 98.0% 95.5% 79.9% 70.0% 39.1% (24,602 gpu-h)
PickScore (Parti-prompts) ↑
Model Baseline LDM (ours) DF-GAN GALIP SD 1.4 SD 2.1 SD XL
(train cost) (≈25,000 gpu-h) - - (150.000 gpu-h) (200.000 gpu-h) -
Wurstchen ¨
98.6% 99.6% 97.9% 82.1% 74.6% 39.0% (24,602 gpu-h)
images as COCO30K. Since the prompts of MS COCO are quite short and frequently lack detail, we
also generate 5,000 images from the Localized Narrative MS COCO subset, we refer to his dataset
as Localized Narratives-COCO-5K. Finally, we also use Parti-prompts (Yu et al., 2022b), a highly
diverse set of 1633 captions, which closely reflects the usage scenario we intend for our model.
4.1 AUTOMATED TEXT-TO-IMAGE EVALUATION
We evaluate the quality of the generated images using automated metrics in comparison to other,
publicly available models (see Appendix A for random examples). The PickScores in Table 1 paint
a consistent picture over the three datasets the models were evaluated on. Wurstchen is preferred ¨
very significantly over smaller models like DF-GAN and GALIP, which is expected. The LDM is
outperformed dramatically in all cases, highlighting that the architecture had a significant impact on
the model’s computational training efficiency. Wurstchen is also preferred in all three scenarios ¨
over SD 1.4 and 2.1, despite their significantly higher compute-budget at a similar modelcapacity. While SD XL is still superior in image quality, our inference speed is significantly faster
(see Figure 4). This comparison is not entirely fair, as it’s a higher capacity model and its data and
compute budget is unknown. For this reason, we are omitting SD XL from the following experiments.
While we achieve a higher Inception Score (IC) on COCO30K compared to all other models in our
broader comparison in Table 2 also shows a relatively high FID on the same dataset. While still
outperforming larger models like CogView2 (Ding et al., 2022) and our Baseline LDM, the FID
is substantially lower compared to other state-of-the-art models. We attribute this discrepancy to
high-frequency features in the images. During visual inspections we find that images generates by
Wurstchen tend smoother than in other text-to-image models. This difference is most noticeable in ¨
real-world images like COCO, on which we compute the FID-metric.
8
Würstchen Stable
Diffusion 2.1
Both Equal
0
20
40
60
80
100
% Preference
41.3% 40.6%
18.1%
49.5%
32.8%
17.7%
Images from MS COCO Captions
Images from Parti-prompts
(a) Overall Preference
Würstchen Stable
Diffusion 2.1
Both Equal
0
20
40
60
80
100
% Preferred by Participants
44.44%
27.78% 27.78%
72.22%
5.56%
22.22%
Images from MS COCO Captions
Images from Parti-prompts
(b) Individual Preference
Individuals
0
50
100
150
200
250
300
350
# Comparisons (Parti-prompts)
50th Percentile,
preference statistic
cutoff
(c) Histogram (MS COCO)
Individuals
0
50
100
150
200
250
300
350
# Comparisons (MS COCO)
50th Percentile,
preference statistic
cutoff
(d) Histogram (Parti)
Figure 5: Overall human preferences (left) and by users (middle). The preference by users considered
only users with a large number of comparisons (right).
4.2 HUMAN PREFERENCE EVALUATION
While most metrics evaluated in the previous section are correlated with human preference (Kirstain
et al., 2023; Heusel et al., 2018; Salimans et al., 2016), we follow the example of other works
and also conducted two brief studies on human preference. To simplify the evaluation, we solely
compared Wurstchen against ¨ SD 2.1, its closest capacity and performance competitor, and evaluated
the human preference between these two models following the setup of Kirstain et al. (2023). In total,
we conducted two studies using the generated images from Parti-prompts and COCO30K images.
Participants were presented randomly chosen pairs of images in randomized order. For each pair the
participants selected a preference for one or neither of the images (see Appendix C for details). In
total, 3343 (Parti-prompts) and 2262 (COCO Captions) comparisons by 90 participants were made.
We evaluate results in two distinct ways. First, by counting the total number of preferences independent of user-identity. In Figure 5 (a) we can see that images generated by our model on Parti-prompts
were clearly preferred. This is important to us, since Parti-prompt closely reflects the intended use
case of the model. However, for MS-COCO this statistic is inconclusive. We hypothesize that this is
due to the vague prompts generating a more diverse set of images, making the preference more subject
to personal taste, biasing this statistics towards users that completed more comparisons (Figure 5 (c,
d)). For this reason, we conducted a second analysis, where we evaluated the personally preferred
model for each individual. In an effort to only include participants that completed a representative
number of comparisons, we only include users from the upper 50th percentile and above. By doing
so, we include only individuals with at least 30 (MS-COCO) and 51 (Parti-prompts) comparisons in
the statistic. Under these circumstances, we observed a light preference for MS-COCO in favor of
Wurstchen and a strong preference for our model on Parti-prompts (Figure 16 (b)). In summary, the ¨
human preference experiments confirm the observation made in the PickScore experiments. While
the real-world results were in-part less decisive, the image generation quality of Wurstchen was ¨
overall preferred by the participants of both studies over SD 2.1.
Table 2: Comparison to other architectures. ∗
computed from own evaluation. † based on official
model cards (Rombach & Esser, 2022; Rombach et al., 2023).
Model Params Sampling
Steps
FID ↓
@2562
IS ↑
@2992
Open
Source
GPU Hours
@ A100 ↓
Train ↓
Samples
Est. Carbon Em.
[kg CO2 eq.]
GLIDE (Nichol et al., 2021) 3.5B 250 12.24 – – – –
Make-A-Scene (Gafni et al., 2022) 4B 1024 11.84 – – – –
Parti (Yu et al., 2022a) 20B 1024 7.23 – – – –
CogView (Ramesh et al., 2021) 4B 1024 27.1 22.4 ✓ – – –
CogView2 (Ding et al., 2022) 6B - 24.0 25.2 - – – –
DF-GAN (Tao et al., 2022) 19M - 19.3 18.6 ✓ – – –
GALIP (Tao et al., 2023) 240M - 12.5 26.3* ✓ – – –
DALL-E (Ramesh et al., 2021) 12B 256 17.89 17.9 – – – –
LDM (Rombach et al., 2022) 1.45B 250 12.63 30.3 ✓ – – –
Baseline LDM (ours) 0.99B 60 43.5* 20.1* - ≈25,000 ≈2,300
Wurstchen (ours) ¨ 0.99B 60 23.6* 40.9* ✓ 24,602 1.42B 2,276
SD 1.4 (Rombach et al., 2022) 0.8B 50 16.2* 40.6* ✓ 150,000 † 4.8B † 11,250 †
SD 2.1 (Rombach et al., 2022) 0.8B 50 15.1* 40.1* ✓ 200,000 † 3.9B † 15,000 †
SD XL (Podell et al., 2023) 2.6B 50 > 18 – ✓ – – –
9
4.3 EFFICIENCY
Table 2 shows the computational costs for training Wurstchen compared to the original ¨ SD 1.4 and
2.1. Based on the evaluations in Section 4.1, it can be seen that the proposed setup of decoupling
high-resolution image projection from the actual text-conditional generation can be leveraged even
more as done in the past (Esser et al., 2021; Saharia et al., 2022; Ramesh et al., 2022), while still
staying on-par or outperforming in terms of quality, fidelity and alignment. Stage C, being the most
expensive stage to train from scratch, required only 24,602 GPU hours, compared to 200,000 GPU
hours (Rombach et al., 2023) for SD 2.1, making it a 8x improvement. Additionally, SD 1.4 and 2.1
processed significantly more image samples. The latter metric is based on the total number of steps
of all trainings and finetunings and multiplied with the respective batch sizes. Even when accounting
for 11,000 GPU hours and 318M train samples used for training Stage B, Wurstchen is significantly ¨
more efficient to train than the SD models. Moreover, although needing to sample with both Stage A
& B to generate the VQGAN latents ¯xq, the total inference is still significantly faster than SD 2.1 and
XL (see Figure 4).
5 CONCLUSION
In this work, we presented our text-conditional image generation model Wurstchen, which employs a ¨
three stage process of decoupling text-conditional image generation from high-resolution spaces. The
proposed process enables to train large-scale models efficiently, substantially reducing computational
requirements, while at the same time providing high-fidelity images. Our trained model achieved
comparable performance to models trained using significantly more computational resources, illustrating the viability of this approach and suggesting potential efficient scalability to even larger model
parameters. We hope our work can serve as a starting point for further research into a more sustainable
and computationally more efficient domain of generative AI and open up more possibilities into
training, finetuning & deploying large-scale models on consumer hardware. We will provide all of
our source code, including training-, and inference scripts and trained models on GitHub.

论文笔记：

看论文：Self-Rewarding Language Models

Tue, 30 Jan 2024 11:45:08 +0800

概述

语言模型通常的训练方法是先收集一大堆人类的反馈,然后基于这些反馈教模型“说话”。但这种依赖外部信号的机制缺点也很明显,模型的能力受限于人类反馈的数据指令。

所以论文提出,我们得让模型自己动手试错、自我完善。具体想法是让模型给自己当老师,让它边生成回复边给自己打分。这样模型就可以根据自己的评价,找出好和不好的回答,进而再基于这些评分来改进模型。

论文里面迭代模型的过程是这样的:

Model0: 没有微调的预训练模型
Model1: 基于人类反馈数据的微调模型,使用SFT的方法微调
Model2: 使用Model1生成的回复,然后使用Model1对回复进行打分,选出好的和不好的结果,用这些结果使用DPO的方法对Model2进行微调
Model3: 使用Model2生成的回复,然后使用Model2对回复进行打分,选出好的和不好的结果,用这些结果使用DPO的方法对Model3进行微调

这样,就可以不断的迭代下去,直到模型的能力达到预期的水平。

模型迭代细节

Model0:原始预训练模型
Model1:基于人类反馈数据的微调模型,使用SFT的方法微调

Model2:基于Model1自评分微调

生成新的指令,具体的方法参考Aligning Language Models with Self-Generated Instructions和Tuning Language Models with (Almost) No Human Labor
基于生成的指令,让Model1给每个输入生成N个回复

使用Model1对每个回复进行打分,返回的分数是0-5分。使用如下的Prompt:

Review the user’s question and the corresponding response using the additive 5-point scoring system described below. Points are accumulated based on the satisfaction of each criterion: 
- Add 1 point if the response is relevant and provides some information related to the user’s inquiry, even if it is incomplete or contains some irrelevant content. 
- Add another point if the response addresses a substantial portion of the user’s question, but does not completely resolve the query or provide a direct answer. 
- Award a third point if the response answers the basic elements of the user’s question in a useful way, regardless of whether it seems to have been written by an AI Assistant or if it has elements typically found in blogs or search results. 
- Grant a fourth point if the response is clearly written from an AI Assistant’s perspective, addressing the user’s question directly and comprehensively, and is well-organized and helpful, even if there is slight room for improvement in clarity, conciseness or focus. 
- Bestow a fifth point for a response that is impeccably tailored to the user’s question by an AI Assistant, without extraneous information, reflecting expert knowledge, and demonstrating a high-quality, engaging, and insightful answer. 

User: <INSTRUCTION_HERE> 

<response><RESPONSE_HERE></response> 

After examining the user’s instruction and the response: 
- Briefly justify your total score, up to 100 words.
- Conclude with the score using the format: “Score: <total points>” 

Remember to assess from the AI Assistant perspective, utilizing web search knowledge as necess

翻译成中文：

多卡训练：DP vs DDP

Sat, 20 May 2023 17:45:08 +0800

今天聊聊数据并行（DP）和分布式数据并行（DDP）这两个常用的方法。

如果你有2个GPU，那你就可以简单的通过DP和DDP实现更快的训练速度。

Pytorch已经内置这两种方法，官方建议使用DDP。

数据并行（DP）

数据并行是一种简单且常见的方法，它让我们可以在多个GPU上同时进行模型训练。

这个方法的核心思想就是：把一个大批量的数据分成几个小批量，然后让每个GPU处理一个小批量数据。处理完成后，我们把各个GPU计算出来的梯度汇总，然后更新模型权重。

优点：

实现简单：数据并行很容易理解和实现。
加速训练：因为数据是在多个GPU上同时处理的，所以训练速度会比单个GPU快很多。

缺点：

扩展性有限：当GPU数量增加时，通信和同步开销也会增加，这会限制训练速度的提升。
只适用于较小模型：如果模型太大，无法放入单个GPU的内存中，那么数据并行就不适用了。

分布式数据并行（DDP）

分布式数据并行是对数据并行的一种改进。在这个方法中，我们不仅把数据切分成小批量，而且还把模型参数在各个GPU上分片存储。这样一来，每个GPU都处理一部分数据，同时也只更新模型的一部分参数。

这种方法的关键在于，我们需要在所有GPU之间同步梯度和模型参数。

优点：

更好的扩展性：DDP的通信和同步开销相对较小，因此在大量GPU上训练时，它能提供更好的扩展性。
支持更大模型：由于模型参数在各个GPU上分片存储，DDP可以支持无法放入单个GPU内存的大型模型。

缺点：

实现更复杂：与数据并行相比，DDP需要更多的设置和细节处理。
依赖高速网络：DDP需要在各个GPU之间同步梯度和模型参数，这要求有高速的网络连接。

DP与DDP的区别

通常，DDP比DP更快，但并非总是如此，比如显卡之间不支持 nv-link 的时候。

DP基于Python线程，而DDP基于多进程，因此DDP没有诸如全局解释器锁（GIL）之类的Python线程限制。
在GPU卡之间的连接速度较慢时，DDP的实际运行速度可能会更慢。

以下是两种模式之间的主要差异：

DDP：

在开始时，主进程将模型从GPU 0复制到其它GPU
然后，对于每个批次：
- 每个GPU直接计算自己的小批量数据
- 在反向传播过程中，一旦本地梯度计算好了，它们就会在所有进程之间取平均值

DP：

对于每个批次：
- GPU 0读取数据，然后将一个小批量发送给每个GPU
- 从GPU 0将最新的模型复制到每个GPU
- 执行推理并将结果从每个GPU发送到GPU 0，计算loss
- 将loss从GPU 0分散到所有GPU，进行反向传播
- 将梯度从每个GPU发送到GPU 0并取平均值
因此，DDP每个批次只需要进行梯度发送，而DP需要进行5次不同的数据交换。

DP通过Python线程在进程内复制数据，而DDP通过torch.distributed复制数据。

在DP下，GPU 0的工作量远大于其他GPU，导致GPU的利用率降低。

DDP可以在多台机器上使用，但DP则不行。虽然DP和DDP之间还有其他差异，但它们与本讨论无关。

结论

数据并行（DP）和分布式数据并行（DDP）是在深度学习中实现多GPU训练的两种有效方法。通常情况下，DDP比DP更快，但具体差异取决于GPU之间需要同步的数据量。当需要同步的数据越多时，慢速连接可能导致整体运行速度变慢。

在选择使用哪种方法时，要考虑你的硬件条件和实际需求。希望这篇文章能帮助你更好地了解DP和DDP的概念、优缺点以及区别，从而为你的深度学习项目选择合适的多GPU训练策略。

参考

Efficient Training on Multiple GPUs (huggingface.co)

多卡训练：如何选择合适的并行策略

Sat, 22 Apr 2023 17:45:08 +0800

前言

当我们训练深度学习模型时，有时会遇到问题：单个GPU速度太慢,或者模型权重放不进单个GPU里。这时候，我们就需要考虑使用多个GPU来进行训练。

现在，有很多方法可以实现并行，比如数据、张量和流水线并行。不过，并没有一个通用的解决方案能适应所有情况。

注意：单GPU的许多策略（比如混合精度训练或梯度累积）都是通用的，也适用于大部分模型训练。

主要概念

下面是一些主要概念的简要概述：

数据并行（DataParallel）：就是把模型参数放到所有卡上，每张卡都有完整的模型参数，每张卡都会处理一部分数据。处理过程是同时进行的，然后在每次训练步骤结束时同步所有结果。
张量并行（TensorParallel）：这个方法是把每个张量切成好几块，所以不是把整个张量放在一个GPU上，而是让张量的每个部分都放在不同的GPU上。在处理过程中，每个部分都在不同的GPU上同时进行处理，然后在步骤结束时同步结果。这就是所谓的水平并行，因为切分是在水平层面进行的。
流水线并行（PipelineParallel）：这个方法是把模型垂直（按层级）切分到不同的GPU上，这样单个GPU上只放置模型的一部分或几个层。每个GPU同时处理流水线的不同阶段，一次处理一小部分批次数据。
零冗余优化器（ZeRO）：这个方法也对张量进行分片，但与TP不同的是，在进行正向或反向计算时，整个张量会在适当时候重建。因此，模型不需要进行修改。它还支持各种offloading技术以弥补不足的GPU内存。
分片数据并行（Sharded DDP）：是ZeRO的另一个名称。

如何选择合适的并行策略

在训练大模型的时候，为了能够训练，或者提高训练速度，我们可以采用各种并行策略。

以下将针对单GPU、单节点多GPU以及多节点多GPU等不同场景，介绍如何选择合适的并行策略。

一、单GPU

模型可以完全加载到单个GPU中
- 在这种情况下，可以正常使用单GPU进行训练。
模型无法完全加载到单个GPU中
- ZeRO + Offload CPU方案，并可选使用NVMe作为辅助存储。
- 如果模型中最大的层无法放入单个GPU中，可以考虑使用Memory Centric Tiling, MCT技术。MCT可以通过自动分割并顺序执行大型层来运行任意大小的层。这种方式目前很少用，需要手动覆盖torch.nn.Linear来实现。

二、单节点多GPU场景

模型可以完全加载到单个GPU中
- 可以使用DDP（Distributed Data Parallel，分布式数据并行）策略
- ZeRO（Zero Redundancy Optimizer，零冗余优化器）策略也可以作为一种选择。
模型无法完全加载到单个GPU中
- PP（Pipeline Parallelism）
- ZeRO
- TP（Tensor Parallelism）
当节点内部，卡与卡之间有NVLINK或NVSwitch连接时，这三种策略的性能基本相当；

若没有，PP通常比TP或ZeRO更快。不同程度的TP对性能的影响可能不同，所以最好通过实验来确定在特定设备上的最佳策略。
最大层无法放入单个GPU中
- ZeRO
- 如果无法使用ZeRO策略，则只能选择TP。

三、多节点多GPU场景

具有快速节点间连接时
- 选择使用ZeRO策略，因为它几乎不需要对模型进行修改。
- 还可以考虑使用PP+TP+DP（Data Parallel，数据并行）组合策略，该策略减少了通信量，但需要对模型进行大量修改。
节点间连接速度较慢，且GPU内存仍然不足时
- 可以尝试使用DP+PP+TP+ZeRO-1组合策略

参考文献

Efficient Training on Multiple GPUs (huggingface.co)

关于本站

Mon, 01 Jan 0001 00:00:00 +0000

about