结合论文描述，给出从视频文件中提取视频帧和音频特征的最佳方法，给出最优代码实现，不要用生成临时文件的方法。下面是论文内容：

Loopy

3 Method

In this section, we will introduce our method, Loopy. First, we will provide an overview of the framework, including the input, output, and key components of Loopy. Second, we will focus on the design of the inter-/intra-temporal modules, including the temporal segment module. Third, we will detail the implementation of the audio condition module. Finally, we will describe the implementation details during the training and testing of Loopy.

Figure 2: The framework of Loopy. it removes the commonly used face locator and speed layer modules found in existing methods. Instead, it achieves flexible and natural motion generation through the proposed inter/intra-clip temporal layer and audio-to-latents module.

3.1 Framework

Our method is built upon Stable Diffusion (SD) and uses its initialization weights. SD is a textto-image diffusion model based on the Latent Diffusion Model (LDM) (Rombach et al., 2022). It employs a pretrained VQ-VAE (Kingma, 2013; Van Den Oord et al., 2017) E to transform images from pixel space to latent space. During training, images are first converted to latents, i.e., z0 = E(I).

Gaussian noise ϵ is then added to the latents in the latent space based on the Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., 2020) for t steps, resulting in a noisy latent zt. The denoising net takes zt as input to predict ϵ. The training objective can be formulated as follows:

$L=\mathbb{E}_{z_{t},c,\epsilon\sim{\mathcal{N}}(0,1),t}\left[\|\epsilon-\epsilon_{\theta}(z_{t},t,c)\|_{2}^{2}\right],$
$(1)$
, (1)
where ϵθ represents the denoising net, including condition-related attention modules, which is the main part Loopy aims to improve. c represents the text condition embedding in SD, and in Loopy, it includes audio, motion frames, and other additional information influencing the final generation.

During testing, the final image is obtained by sampling from Gaussian noise and removing the noise based on DDIM (Song et al., 2020) or DDPM.

As demonstrated in Figure 2, in the Loopy, the inputs to the denoising net include noisy latents, i.e., the VQ-VAE encoded image latents zt. Unlike original SD, where the input is a single image, here it is a sequence of images representing a video clip. The inputs also include reference latents cref (encoded reference image latents via VQ-VAE), audio embedding caudio (audio features of the current clip), motion frames cmf (image latents of the M frames sequence from the preceding clips), and timestep t. During training, additional facial movement-related features are involved:
clmk (facial keypoint sequence of the current clip), cmov (head motion variance of the current clip), and cexp (expression variance of the current clip). The output is the predicted noise ϵ. The denoising network employs a Dual U-Net architecture (Hu, 2024; Zhu et al., 2023). This architecture includes an additional reference net module, which replicates the original SD U-Net structure but utilizes reference latents cref as input. The reference net operates concurrently with the denoising U-Net.

During the spatial attention layer computation in the denoising U-Net, the key and value features from corresponding positions in the reference net are concatenated with the denoising U-Net's features along the token dimension before proceeding with the attention module computation. This design enables the denoising U-Net to effectively incorporate reference image features from the reference net.

Additionally, the reference net also takes motion frames latents cmf as input for feature extraction, allowing these features to be utilized in subsequent temporal attention computations.

3.2 Inter/Intra- Clip Temporal Layers Design

Here, we introduce the design of the proposed inter-/intra-clip temporal modules. Unlike existing methods (Tian et al., 2024; Xu et al., 2024a; Chen et al., 2024; Wang et al., 2024) that process motion frame latents and noisy latents features simultaneously through a single temporal layer, Loopy employs two temporal attention layers: the inter-clip temporal layer and the intra-clip temporal layer.

The inter-clip temporal layer first handles the cross-clip temporal relationships between motion frame latents and noisy latents, while the intra-clip temporal layer focuses on the temporal relationships within the noisy latents of the current clip.

First, we introduce the inter-clip temporal layer, initially ignoring the temporal segment module in Figure 2. We first collect the m image latents from the preceding clip, referred to as motion frames latents cmf . Similar to cref , these latents are processed frame-by-frame through the reference network for feature extraction. Within each residual block, the features xcmf obtained from the reference network are concatenated with the features x from the denoising U-Net along the temporal dimension. To distinguish the types of latents, we add learnable temporal embeddings. Subsequently, self-attention is computed on the concatenated tokens along the temporal dimension, referred to as temporal attention. The intra-clip temporal layer differs in that its input does not include features from motion frames latents, it only processes features from the noisy latents of the current clip. By separating the dual temporal layers, the model can better handle the aggregation of different semantic temporal features in cross-clip temporal relationships.

Figure 3: The illustration of the temporal segment module

and the inter/intra- clip temporal layers. The former allows us to expand the motion frame to cover over 100 frames, while the later enables the modeling of long-term motion dependency.

Due to the independent design of the inter-clip temporal layer, Loopy is better equipped to model the motion relationships among clips. To enhance this capability, we introduce the temporal segment module before cmf enters the reference network. This module not only extends the temporal range covered by the inter-clip temporal layer but also accounts for the variations in information due to the varying distances of different clips from the current clip, as illustrated in Figure 3. The temporal segment module divides the original motion frame into multiple segments and extracts representative motion frames from each segment to abstract the segment. Based on these abstracted motion frames, we recombine them to obtain motion frame latents for subsequent computations. For the segmentation process, we define two hyperparameters: stride s and expand ratio r. The stride s represents the number of abstract motion frames in each segment, while the expand ratio r is used to calculate the length of the original motion frames in each segment. The number of frames in the i-th segment, ordered from closest to farthest from the current clip, is given by s × r
(i−1). For example, with a stride s = 4 and an expand ratio r = 2, the first segment would contain 4 frames, the second segment would contain 8 frames, and the third segment would contain 16 frames. For the abstraction process after segmentation, we default to uniform sampling within each segment. In the experimental section, we investigate different segmentation parameters and abstraction methods. Since segmentation and abstraction directly affect the learning of long-term motion dependency, different methods have a significant impact on the results. The output of the temporal segment module, c o mf , can be defined as:
where start_indexi =Pk−1 j=0 r j· s and end_indexi = start_indexi + r k· s, with k =is
. c o mf,i represents the mean value of the i-th element of the output, calculated from the sequence cmf .

The temporal segment module rapidly expands the temporal coverage of the motion frames input to the inter-clip temporal layer while maintaining acceptable computational complexity. For closer frames, a lower expansion rate retains more details, while for distant frames, a higher expansion rate covers a longer duration. This approach helps the model better capture motion style from long-term information and generate temporally natural motion without spatial templates.

3.3 Audio Condition Module

For the audio condition, we first use wav2vec (Baevski et al., 2020; Schneider et al., 2019) for audio feature extraction. Following the approach in EMO, we concatenate the hidden states from each layer of the wav2vec network to obtain multi-scale audio features. For each video frame, we concatenate the audio features of the two preceding and two succeeding frames, resulting in a 5-frame audio feature as audio embeddings caudio for the current frame. Initially, in each residual block, we use cross-attention with noisy latents as the query and audio embedding caudio as the key and value to compute an attended audio feature. This attended audio feature is then added to the noisy latents feature obtained from self-attention, resulting in a new noisy latents feature. This provides a preliminary audio condition.

Additionally, as illustrated in Figure 4, we introduce the Audio-to-Latents module. This module maps conditions that have both strong and weak correlations with portrait motion, such as audio, to a shared motion latents space. These mapped conditions serve as the final condition features, thereby enhancing the modeling of the relationship between audio and portrait movements based on motion latents. Specifically, we maintain a set of learnable embeddings. For each input condition, we map it to a query feature using a fully connected (FC) layer, while the learnable embeddings serve as key and value features for attention computation to obtain a new feature based on the learnable embeddings. These new features, referred to as motion latents, replace the input condition in subsequent computations. These motion latents are then dimensionally transformed via an FC layer and added to the timestep embedding features for subsequent network computation.

During training, we sample an input condition for the Audio-to-Latents module with equal probability from audio embeddings and facial movement-related features such as landmarks, face absolute motion variance, and face expression motion variance. During testing, we only input audio to generate motion latents. By leveraging features strongly correlated with portrait movement, the model utilizes learnable embeddings to control motion. Consequently, transforming audio embeddings into motion latents can also allows for more direct influence on portrait motion.

Figure 4: The audio-to-latents module.

3.4 Training Strategies

Conditions Mask and Dropout. In the Loopy framework, various conditions are involved, including the reference image cref , audio features caudio, preceding frame motion frames cmf , and motion latents cml representing audio and facial motion conditions. Due to the overlapping nature of the information contained within these conditions, to better learn the unique information specific to each condition, we used distinct masking strategies for the conditions during the training process. During training, caudio and motion latents are masked to all-zero features with a 10% probability. For cref and cmf , we design specific dropout and mask strategies due to their conflicting relationship. cmf also provides appearance information and is closer to the current clip compared to cref , leading the model to heavily rely on motion frames rather than the reference image. This can cause color shifts and artifacts in long video sequences during inference. To address this, cref has a 15% probability of being dropped, meaning the denoising U-Net will not concatenate features from the reference network during self-attention computation. When cref is dropped, motion frames are also dropped, meaning the denoising U-Net will not concatenate features from the reference network during temporal attention computation. Additionally, motion frames have an independent 40% probability of being masked to all-zero features.

Multistage Training. Following AnimateAnyone (Hu, 2024) and EMO (Tian et al., 2024), we employ a 2-stage training process. In the first stage, the model is trained without temporal layers and the audio condition module. The inputs to the model are the noisy latents of the target single-frame image and reference image latents, focusing the model on an image-level pose variations task. After completing the first stage, we proceed to the second stage, where the model is initialized with the reference network and denoising U-Net from the first stage. We then add the inter-/intra-clip temporal layers and the audio condition module for full training to obtain the final model. Inference. During Inference, we perform class-free guidance (Ho & Salimans, 2022) using multiple conditions. Specifically, we conduct three inference runs, differing in whether certain conditions are dropped. The final noise ef inal is computed as:
ef inal = audio_ratio × (eaudio − eref ) + ref_ratio × (eref − ebase) + ebase where eaudio includes all conditions cref , caudio, cmf , cml, eref but mask caudio to all-zero features, and ebase masks caudio to all-zero features and removes the concatenation of reference network features in the denoising U-Net's self-attention. This approach allows us to control the model's final output in terms of adherence to the reference image and alignment with the audio. The audio ratio is set to 5 and the reference ratio is set to 3. We use DDIM sampling with 25 denoising steps to complete the inference.

3.5 Experiments

Datasets. For training Data, We collected talking head video data from the internet, excluding videos with low lip sync scores, excessive head movement, extreme rotation, or incomplete head exposure.

This resulted in 160 hours of cleaned training data. Additionally, we supplemented the training data with public datasets such as HDTF (Zhang et al., 2021). For the test set, we randomly sampled 100 videos from CelebV-HQ (Zhu et al., 2022) (a public high-quality celebrity video dataset with mixed scenes) and RAVDESS (Kaggle) (a public high-definition indoor talking scene dataset with rich emotions). To test the generalization ability of diffusion-based models, we also collected 20 portrait test images, including real people, anime, side face, and humanoid crafts of different materials, along with 20 audio clips, including speeches, singing, rap and emotionally rich speech. We refer to this test set as the openset test set. Implementation Details. We trained our model using 24 Nvidia A100 GPUs with a batch size of 24, using the AdamW (Loshchilov & Hutter, 2017) optimizer with a learning rate of 1e-5. The generated video length was set to 12 frames, and the motion frame length was set to 124 frames, representing the preceding 124 frames of the current 12-frame video. After temporal squeezing, this was compressed to 20 motion frame latents. During training, the reference image was randomly selected from a frame within the video clip. For the facial motion information required to train audo-to-motion module, we used DWPose (Yang et al., 2023) to detect facial keypoints for the current 12 frames. The variance of the absolute position of the nose tip across these 12 frames was used as the absolute head motion variance. The variance of the displacement of the upper half of the facial keypoints (37 keypoints)
relative to the nose tip across these 12 frames was used as the expression variance. The training videos were uniformly processed at 25 fps and cropped to 512x512 portrait videos.

Metrics. We assess image quality with the IQA metric (Wu et al., 2023), video motion with VBench's smooth metric (Huang et al., 2024), and audio-visual synchronization using SyncC and SyncD (Prajwal et al., 2020). For the CelebvHQ and RAVDESS test sets, which have corresponding ground truth videos, we will also compute FVD (Unterthiner et al., 2019), E-FID (Tian et al., 2024)
and FID metrics for comparison. Additionally, to compare the global motion (denoted as Glo) and dynamic expressions (denoted as Exp) of the portrait, we calculated the variance values based on key points of the nose and the upper face, specifically excluding the mouth area. We also computed the values for the ground truth video as a reference for comparison. For the openset testset, which lacks ground truth video references, we conducted subjective evaluations. Ten experienced users were invited to assess six key dimensions: identity consistency, video synthesis quality, audio-emotion