bivddlZddlmZmZmZmZmZmZddlZ ddl Z ddl m Z m Z mZmZmZmZmZmZmZmZmZddlmZddlmZddlmZmZmZmZm Z ddl!m"Z"dd l#m$Z$m%Z%d d l&m'Z'm(Z(d d l)m*Z*m+Z+erddl,Z,ddlm-Z-e-r ddl.m/cm0Z1dZ2ndZ2ejfe4Z5dZ6 ddZ7Gdde(Z8y)N)AnyCallableDictListOptionalUnion) ClapFeatureExtractor ClapModelGPT2LMHeadModelRobertaTokenizerRobertaTokenizerFastSpeechT5HifiGanT5EncoderModel T5TokenizerT5TokenizerFast VitsModel VitsTokenizer) AutoencoderKL)KarrasDiffusionSchedulers)is_accelerate_availableis_accelerate_versionis_librosa_availableloggingreplace_example_docstring)is_transformers_version)empty_device_cache randn_tensor)AudioPipelineOutputDiffusionPipeline)AudioLDM2ProjectionModelAudioLDM2UNet2DConditionModel)is_torch_xla_availableTFah Examples: ```py >>> import scipy >>> import torch >>> from diffusers import AudioLDM2Pipeline >>> repo_id = "cvssp/audioldm2" >>> pipe = AudioLDM2Pipeline.from_pretrained(repo_id, torch_dtype=torch.float16) >>> pipe = pipe.to("cuda") >>> # define the prompts >>> prompt = "The sound of a hammer hitting a wooden surface." >>> negative_prompt = "Low quality." >>> # set the seed for generator >>> generator = torch.Generator("cuda").manual_seed(0) >>> # run the generation >>> audio = pipe( ... prompt, ... negative_prompt=negative_prompt, ... num_inference_steps=200, ... audio_length_in_s=10.0, ... num_waveforms_per_prompt=3, ... generator=generator, ... ).audios >>> # save the best audio sample (index 0) as a .wav file >>> scipy.io.wavfile.write("techno.wav", rate=16000, data=audio[0]) ``` ``` #Using AudioLDM2 for Text To Speech >>> import scipy >>> import torch >>> from diffusers import AudioLDM2Pipeline >>> repo_id = "anhnct/audioldm2_gigaspeech" >>> pipe = AudioLDM2Pipeline.from_pretrained(repo_id, torch_dtype=torch.float16) >>> pipe = pipe.to("cuda") >>> # define the prompts >>> prompt = "A female reporter is speaking" >>> transcript = "wish you have a good day" >>> # set the seed for generator >>> generator = torch.Generator("cuda").manual_seed(0) >>> # run the generation >>> audio = pipe( ... prompt, ... transcription=transcript, ... num_inference_steps=200, ... audio_length_in_s=10.0, ... num_waveforms_per_prompt=2, ... generator=generator, ... max_new_tokens=512, #Must set max_new_tokens equa to 512 for TTS ... ).audios >>> # save the best audio sample (index 0) as a .wav file >>> scipy.io.wavfile.write("tts.wav", rate=16000, data=audio[0]) ``` c H| |ddddf}||||jddS)N use_cache) inputs_embedsattention_maskpast_key_valuesr()get)r)r*r+kwargss k/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/audioldm2/pipeline_audioldm2.pyprepare_inputs_for_generationr/s; "%af- '(*ZZ ,  c06eZdZdZdededeeefde de dee e fdee eefd ed ed ed effd ZdZdZd3deedeej4effdZ d4dej:defdZ d5deej:deej:deej:deej:deej>deej>deefdZ dZ!dZ"d Z# d6d!Z$d7d"Z%ejLe'e( d8d#eee)efd$eee)efd%ee*d&ed'e*d(eeee)efd)eed*e*d+eeejVe)ejVfd,eej:deej:deej:deej:deej:deej>deej>deed-e,d.ee-eeej:gdfd/eed0ee.ee/fd1eef,d2Z0xZ1S)9AudioLDM2Pipelinea Pipeline for text-to-audio generation using AudioLDM2. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, running on a particular device, etc.). Args: vae ([`AutoencoderKL`]): Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations. text_encoder ([`~transformers.ClapModel`]): First frozen text-encoder. AudioLDM2 uses the joint audio-text embedding model [CLAP](https://huggingface.co/docs/transformers/model_doc/clap#transformers.CLAPTextModelWithProjection), specifically the [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused) variant. The text branch is used to encode the text prompt to a prompt embedding. The full audio-text model is used to rank generated waveforms against the text prompt by computing similarity scores. text_encoder_2 ([`~transformers.T5EncoderModel`, `~transformers.VitsModel`]): Second frozen text-encoder. AudioLDM2 uses the encoder of [T5](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5EncoderModel), specifically the [google/flan-t5-large](https://huggingface.co/google/flan-t5-large) variant. Second frozen text-encoder use for TTS. AudioLDM2 uses the encoder of [Vits](https://huggingface.co/docs/transformers/model_doc/vits#transformers.VitsModel). projection_model ([`AudioLDM2ProjectionModel`]): A trained model used to linearly project the hidden-states from the first and second text encoder models and insert learned SOS and EOS token embeddings. The projected hidden-states from the two text encoders are concatenated to give the input to the language model. A Learned Position Embedding for the Vits hidden-states language_model ([`~transformers.GPT2Model`]): An auto-regressive language model used to generate a sequence of hidden-states conditioned on the projected outputs from the two text encoders. tokenizer ([`~transformers.RobertaTokenizer`]): Tokenizer to tokenize text for the first frozen text-encoder. tokenizer_2 ([`~transformers.T5Tokenizer`, `~transformers.VitsTokenizer`]): Tokenizer to tokenize text for the second frozen text-encoder. feature_extractor ([`~transformers.ClapFeatureExtractor`]): Feature extractor to pre-process generated audio waveforms to log-mel spectrograms for automatic scoring. unet ([`UNet2DConditionModel`]): A `UNet2DConditionModel` to denoise the encoded audio latents. scheduler ([`SchedulerMixin`]): A scheduler to be used in combination with `unet` to denoise the encoded audio latents. Can be one of [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. vocoder ([`~transformers.SpeechT5HifiGan`]): Vocoder of class `SpeechT5HifiGan` to convert the mel-spectrogram latents to the final audio waveform. vae text_encodertext_encoder_2projection_modellanguage_model tokenizer tokenizer_2feature_extractorunet schedulervocoderc t ||j||||||||| | |  t|ddr5dt |j j jdz z|_yd|_y)N) r3r4r5r6r7r8r9r:r;r<r=r3rr") super__init__register_modulesgetattrlenr3configblock_out_channelsvae_scale_factor) selfr3r4r5r6r7r8r9r:r;r<r= __class__s r.rAzAudioLDM2Pipeline.__init__s  %)-)#/  W^^bdikoVpc$((//*L*L&MPQ&Q Rvwr0c8|jjy)z Enable sliced VAE decoding. When this option is enabled, the VAE will split the input tensor in slices to compute decoding in several steps. This is useful to save some memory and allow larger batch sizes. N)r3enable_slicingrHs r.enable_vae_slicingz$AudioLDM2Pipeline.enable_vae_slicings !r0c8|jjy)z Disable sliced VAE decoding. If `enable_vae_slicing` was previously enabled, this method will go back to computing decoding in one step. N)r3disable_slicingrLs r.disable_vae_slicingz%AudioLDM2Pipeline.disable_vae_slicings   "r0Ngpu_iddevicec trtddrddlm}n t dt j |}|j}| |td|d|d |j|j}|}|s |jr|d |xs |j}t j |}|j jd k7r(|jd d t|j|jj|jj|j|j |j"|j$|j&|j(|jg }d} |D]} || || \} } | |_y)a Offloads all models to CPU using accelerate, reducing memory usage with a low impact on performance. Compared to `enable_sequential_cpu_offload`, this method moves one whole model at a time to the GPU when its `forward` method is called, and the model remains in GPU until the next model runs. Memory savings are lower than with `enable_sequential_cpu_offload`, but performance is much better due to the iterative execution of the `unet`. z>=z 0.17.0.dev0r)cpu_offload_with_hookzC`enable_model_cpu_offload` requires `accelerate v0.17.0` or higher.NzYou have passed both `gpu_id`=z4 and an index as part of the passed device `device`=zwCannot pass both. Please make sure to either not define `gpu_id` or not pass the index as part of the device: `device`=:cpuT)silence_dtype_warnings)prev_module_hook)rr acceleraterT ImportErrortorchrRindex ValueErrortypetorr4 text_modeltext_projectionr5r6r7r;r3r=final_offload_hook) rHrQrRrT torch_device device_index device_type device_strmodel_sequencehookcpu_offloaded_model_s r.enable_model_cpu_offloadz*AudioLDM2Pipeline.enable_model_cpu_offloads # $)>t])S 8cd d||F+ #))  ,":08lmsltJKWK\K\J]^  #''  \''& !+ .2?2E2Ea2H !, /.5d