biwNddlZddlmZmZmZmZmZmZddlZ ddl Z ddl m Z m Z mZmZmZmZddlmZmZddlmZddlmZmZmZmZmZddlmZmZm Z d d l!m"Z"m#Z#m$Z$m%Z%erddl&Z&dd lm'Z'e'r ddl(m)cm*Z+d Z,nd Z,ejZe.Z/dZ0Gdde#e$e%Z1y)N)AnyCallableDictListOptionalUnion)ClapFeatureExtractor ClapModelClapTextModelWithProjectionRobertaTokenizerRobertaTokenizerFastSpeechT5HifiGan) AutoencoderKLUNet2DConditionModel)KarrasDiffusionSchedulers)is_accelerate_availableis_accelerate_versionis_librosa_availableloggingreplace_example_docstring)empty_device_cache get_device randn_tensor)AudioPipelineOutputDeprecatedPipelineMixinDiffusionPipelineStableDiffusionMixin)is_torch_xla_availableTFae Examples: ```py >>> from diffusers import MusicLDMPipeline >>> import torch >>> import scipy >>> repo_id = "ucsd-reach/musicldm" >>> pipe = MusicLDMPipeline.from_pretrained(repo_id, torch_dtype=torch.float16) >>> pipe = pipe.to("cuda") >>> prompt = "Techno music with a strong, upbeat tempo and high melodic riffs" >>> audio = pipe(prompt, num_inference_steps=10, audio_length_in_s=5.0).audios[0] >>> # save the audio sample as a .wav file >>> scipy.io.wavfile.write("techno.wav", rate=16000, data=audio) ``` c$eZdZdZ dedeeefdeee fde e de de deffd Z d#d e ej"d e ej"fd ZdZdZdZ d#dZd$dZd%dZej2ee d&deeeefde ededede eeeefde edede eej@eej@fde ej"d e ej"d e ej"de!de e"eeej"gd fde ed e e#ee$fd!e ef d"Z%xZ&S)'MusicLDMPipelinez0.33.1vae text_encoder tokenizerfeature_extractorunet schedulervocoderc t||j|||||||t|ddr5dt |j j jdz z|_yd|_y)N)r#r$r%r&r'r(r)r#r) super__init__register_modulesgetattrlenr#configblock_out_channelsvae_scale_factor) selfr#r$r%r&r'r(r) __class__s i/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/pipelines/musicldm/pipeline_musicldm.pyr.zMusicLDMPipeline.__init__jsx  %/  W^^bdikoVpc$((//*L*L&MPQ&Q RvwN prompt_embedsnegative_prompt_embedsc |t|trd}n-|t|tr t|}n|jd}|:|j |d|j j dd} | j} | j} |j |dd j} | jd | jd k\rtj| | sj|j j| dd|j j dz d f} tjd |j j d | |jj| j!|| j!| }|j!|jj"j$|}|j\}}|j'd|}|j)||z|}|r||dg|z}nt+|t+|ur$t-dt+|dt+|dt|tr|g}n1|t|k7r!t/d|dt|d|d|d |}|jd}|j |d|dd}|jj!|}|jj!|} |jj|| }|r~|jd}|j!|jj"j$|}|j'd|}|j)||z|}tj0||g}|S)a` Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded device (`torch.device`): torch device num_waveforms_per_prompt (`int`): number of waveforms that should be generated per prompt do_classifier_free_guidance (`bool`): whether to use classifier free guidance or not negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the audio generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. Nr+r max_lengthTpt)paddingr< truncationreturn_tensorslongest)r>r@z\The following part of your input was truncated because CLAP can only handle sequences up to z tokens: )attention_mask)dtypedevicez?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) isinstancestrlistr1shaper%model_max_length input_idsrCtorchequal batch_decodeloggerwarningr$get_text_featuresto text_modelrDrepeatviewtype TypeError ValueErrorcat)r5promptrEnum_waveforms_per_promptdo_classifier_free_guidancenegative_promptr9r: batch_size text_inputstext_input_idsrCuntruncated_ids removed_textbs_embedseq_len uncond_tokensr< uncond_inputuncond_input_idss r7_encode_promptzMusicLDMPipeline._encode_promptsD  *VS"9J  Jvt$<VJ&,,Q/J  ..$>>::# )K)22N(77N"nnVYW[n\ffO$$R(N,@,@,DDU[[N $~~::#At~~'F'F'JR'O$OP  778 ,Q !--??!!&)-008@M &((t/@/@/K/K/Q/QZ`(a     &,,Q0HI %**86N+NPWX  '+A+I&!#z 1 fT/%::UVZ[jVkUlmV ~Q(OS1!0 1 s?33 )/)::J3K_J`ax/ |<33 !0 &,,Q/J>>$%# *L ,5588@ )88;;FCN%)%6%6%H%H -&I& " ',2215G%;%>%>TEVEVEaEaEgEgpv%>%w "%;%B%B1F^%_ "%;%@%@NfAfho%p " "II'=}&MNMr8c|jdk(r|jd}|j|}|jj }|S)Nr+)dimsqueezer)cpufloat)r5mel_spectrogramwaveforms r7mel_spectrogram_to_waveformz,MusicLDMPipeline.mel_spectrogram_to_waveformsJ    A %-55a8O<<0<<>'')r8ctstjd|S|j|dd}t j |j |jjj|jj}|jt|d|jjjj||d<|j|}|jd i|j }t#j$|dd ddd|f} t#j&|d | j)d j+}|S) NaAutomatic scoring of the generated audio waveforms against the input prompt text requires the `librosa` package to resample the generated waveforms. Returning the audios in the order they were generated. To enable automatic scoring, install `librosa` with: `pip install librosa`.r=T)r@r>)orig_sr target_sr)r@ sampling_rateinput_featuresr+)rm descendingrrB)rrQinfor%librosaresamplenumpyr)r2rwr&rJrxrXrTr$logits_per_textrNargsort index_selectreshapero) r5textaudior]rErDinputsresampled_audiorindicess r7score_waveformsz MusicLDMPipeline.score_waveformss/#% KKi  LT4H!** KKM4<<#6#6#D#DPTPfPfPtPt $(#9#9  !$dF\F\FjFj$:$ .e  6",$++5f5EE--Q4HLeMeLeIef""5!W__R-@-D-D-FG r8cVdttj|jjj j v}i}|r||d<dttj|jjj j v}|r||d<|S)Neta generator)setinspect signaturer(step parameterskeys)r5rr accepts_etaextra_step_kwargsaccepts_generators r7prepare_extra_step_kwargsz*MusicLDMPipeline.prepare_extra_step_kwargs*s s7#4#4T^^5H5H#I#T#T#Y#Y#[\\  '* e $(3w/@/@ATAT/U/`/`/e/e/g+hh -6 k *  r8c||jz}||krtd|d|d|jjj|jzdk7r:td|jjjd|jd||0t |t r|dkrtd|dt|d||td |d |d | | td |7t |ts't |tstd t|||td|d|d |C|@|j|jk7r&td|jd|jdyyy)NzH`audio_length_in_s` has to be a positive value greater than or equal to z , but is rGrzwThe number of frequency bins in the vocoder's log-mel spectrogram has to be divisible by the VAE scale factor, but got z bins and a scale factor of z5`callback_steps` has to be a positive integer but is z of type zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` ) r4rZr)r2 model_in_dimrHintrXrIrJrK) r5r\audio_length_in_svocoder_upsample_factorcallback_stepsr_r9r:min_audio_length_in_ss r7 check_inputszMusicLDMPipeline.check_inputs<s!8$:O:O O 4 4Z[pZqr'(+  <<   + +d.C.C Cq H--1\\-@-@-M-M,NNj((),   "  & >30OSaefSfGGWX(),   -";08N}o^00 ^ 5w  FC)@TZ\`IaQRVW]R^Q_`a a  &+A+M9/9JK*++]_   $)?)K""&<&B&BB --:-@-@,AB.445Q8C*L $r8c||t||jzt|jjj|jzf}t |t r)t||k7rtdt|d|d|t||||}n|j|}||jjz}|S)Nz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.)rrErD) rr4r)r2rrHrJr1rZrrTr(init_noise_sigma) r5r`num_channels_latentsheightrDrErlatentsrKs r7prepare_latentsz MusicLDMPipeline.prepare_latentsws  K400 0  ##00 1T5J5J J   i &3y>Z+GA#i.AQR&<'gi  ?"5IfTYZGjj(GDNN;;;r8ctrtddrddlm}n t dt }t j|d|}|jjdk7r|jdd t|jj|jj|j|j|j |jg}d }|D]}|||| \}}||_y ) a Offloads all models to CPU using accelerate, reducing memory usage with a low impact on performance. Compared to `enable_sequential_cpu_offload`, this method moves one whole model at a time to the accelerator when its `forward` method is called, and the model remains in accelerator until the next model runs. Memory savings are lower than with `enable_sequential_cpu_offload`, but performance is much better due to the iterative execution of the `unet`. z>=z 0.17.0.dev0r)cpu_offload_with_hookzC`enable_model_cpu_offload` requires `accelerate v0.17.0` or higher.:roT)silence_dtype_warningsN)prev_module_hook)rr accelerater ImportErrorrrNrErXrTrr$rUtext_projectionr'r#r)final_offload_hook) r5gpu_idr device_typerEmodel_sequencehookcpu_offloaded_model_s r7enable_model_cpu_offloadz)MusicLDMPipeline.enable_model_cpu_offloads # $)>t])S 8cd d l  Qvh78 ;;  u $ GGE$G 7      ( (    - - II HH LL     #1 ` +,?Z^_GAt `#'r8r\rnum_inference_stepsguidance_scaler_r]rrr return_dictcallbackrcross_attention_kwargs output_typec > tj|jjj|jjj z }|0|j jj|jz|z}t||z }t||jjj z}||jzdk7rZttj||jz |jz}tjd|d||zd|d|j|||||| | |t|trd}n-|t|t r t#|}n| j$d}|j&}|dkD}|j)|||||| | } |j*j-|| |j*j.}|j jj0}|j3||z||| j4||| } |j7||}t#|||j*j8zz }|j;| 5}t=|D]5\}}|rt?j@| gd zn| }|j*jC||}|j ||d| |d d}|r|jEd \} }!| ||!| z zz}|j*jF||| fi|jH} |t#|dz k(s'|dz|kDr]|dz|j*j8zdk(r>|jK| ,||zdk(r$|tM|j*ddz}"| |"|| tNs"tQjR8 ddd|jU|dk(sLd|jVjjXz | z} |jVj[| j\}#n t_| S|ja|#}$|$ddd|f}$|dkDr"| |jc||$||| j4}$|dk(r|$je}$| s|$fSt_|$S#1swYxYw)u The call function to the pipeline for generation. Args: prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide audio generation. If not defined, you need to pass `prompt_embeds`. audio_length_in_s (`int`, *optional*, defaults to 10.24): The length of the generated audio sample in seconds. num_inference_steps (`int`, *optional*, defaults to 200): The number of denoising steps. More denoising steps usually lead to a higher quality audio at the expense of slower inference. guidance_scale (`float`, *optional*, defaults to 2.0): A higher guidance scale value encourages the model to generate audio that is closely linked to the text `prompt` at the expense of lower sound quality. Guidance scale is enabled when `guidance_scale > 1`. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide what to not include in audio generation. If not defined, you need to pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`). num_waveforms_per_prompt (`int`, *optional*, defaults to 1): The number of waveforms to generate per prompt. If `num_waveforms_per_prompt > 1`, the text encoding model is a joint text-audio model ([`~transformers.ClapModel`]), and the tokenizer is a `[~transformers.ClapProcessor]`, then automatic scoring will be performed between the generated outputs and the input text. This scoring ranks the generated waveforms based on their cosine similarity to text input in the joint text-audio embedding space. eta (`float`, *optional*, defaults to 0.0): Corresponds to parameter eta (η) from the [DDIM](https://huggingface.co/papers/2010.02502) paper. Only applies to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor is generated by sampling using the supplied random `generator`. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, text embeddings are generated from the `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not provided, `negative_prompt_embeds` are generated from the `negative_prompt` input argument. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.AudioPipelineOutput`] instead of a plain tuple. callback (`Callable`, *optional*): A function that calls every `callback_steps` steps during inference. The function is called with the following arguments: `callback(step: int, timestep: int, latents: torch.Tensor)`. callback_steps (`int`, *optional*, defaults to 1): The frequency at which the `callback` function is called. If not specified, the callback is called at every step. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the [`AttentionProcessor`] as defined in [`self.processor`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). output_type (`str`, *optional*, defaults to `"np"`): The output format of the generated audio. Choose between `"np"` to return a NumPy `np.ndarray` or `"pt"` to return a PyTorch `torch.Tensor` object. Set to `"latent"` to return the latent diffusion model (LDM) output. Examples: Returns: [`~pipelines.AudioPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.AudioPipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated audio. NrzAudio length in seconds z is increased to z; so that it can be handled by the model. It will be cut to z after the denoising process.r+g?)r9r:)rE)totalrF)encoder_hidden_states class_labelsrrorderlatent)audios)rrr]rErDnp)3rprodr)r2upsample_ratesrwr' sample_sizer4rceilrQr{rrHrIrJr1rK_execution_devicerjr( set_timesteps timesteps in_channelsrrDrr progress_bar enumeraterNr[scale_model_inputchunkr prev_sampleupdater0 XLA_AVAILABLExm mark_stepmaybe_free_model_hooksr#scaling_factordecodesamplerrsrr~)%r5r\rrrr_r]rrrr9r:rrrrrrroriginal_waveform_lengthr`rEr^rrrnum_warmup_stepsritlatent_model_input noise_prednoise_pred_uncondnoise_pred_textstep_idxrqrs% r7__call__zMusicLDMPipeline.__call__sh#%''$,,*=*=*L*L"MPTP\P\PcPcPqPq"q  $ $ 0 0 < ,?$..BVBV,VV   %8  9 #\!), #1A\UYYy1}%=bi"%)^^%E%EFXZ[%\""YY&*.!.+A % ' /9C9I9I!9L6%!2^YjGj5k!kJ.$..--j!WZHYZffI**A9I/IqSTuX\XfXfXlXlNlpqNq '')+N0Ba0G#$(K#K 1g6 LLN? # #D ##%h&$((//8887BG"hhoog6==O&g6 600Aa22223 $a 'F,>(()A#)) )E $ KKME8O"%00 # #s/D,RRR)NNN)N)r)NNg@Nr+gNNNNTNr+Nr)'__name__ __module__ __qualname___last_supported_versionrrr r r r rr rrrr.rNTensorrjrsrrrrrno_gradrEXAMPLE_DOC_STRINGrIrrpr Generatorboolrrrr __classcell__)r6s@r7r"r"Osz&2x x7BCx)+??@ x $$89 x # x-x!x:049=C  -C!) 6CL2!0#8v,"'HU]]_12)--1#& #;?23MQ*.049= GK();?%)#Z1c49n%Z1$E?Z1! Z1  Z1 "%T#Y"78 Z1#+3-Z1Z1E%//43H"HIJZ1%,,'Z1  -Z1!) 6Z1Z18S#u||$rs== :3NMrr,))MM   H %(~1.0ACW~1r8