bi(^ddlZddlmZmZmZmZddlZddlmZm Z m Z ddl m Z ddl mZmZeeZdZd Zd Ze xr e d d Zer dd lmZddlmZGddeZGddeZ d!dej8j:dej<dej<deeeedffdeeeej8j:dfde ddfdZ! d"dej8j:dej<dej<deeedfdeeeej8j:dfde deddfdZ"dej8j:dej<dej<de ddf dZ#dej8j:de fdZ$dej8j:ddfd Z%y)#N)OptionalTupleTypeUnion) get_loggeris_peft_availableis_peft_version)_GO_LC_SUPPORTED_PYTORCH_LAYERS) HookRegistry ModelHooklayerwise_castingpeft_autocast_disable) pos_embed patch_embednormz ^proj_in$z ^proj_out$>z0.14.0)disable_input_dtype_casting)BaseTunerLayerc$eZdZdZdZdej dej deddfdZd ejjfd Z d ejjfd Z d ejjfd Z d ejjfd Zy)LayerwiseCastingHookz A hook that casts the weights of a module to a high precision dtype for computation, and to a low precision dtype for storage. This process may lead to quality loss in the output, but can significantly reduce the memory footprint. F storage_dtype compute_dtype non_blockingreturnNc.||_||_||_yN)rrr)selfrrrs \/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/hooks/layerwise_casting.py__init__zLayerwiseCastingHook.__init__1s**(modulecT|j|j|j|SN)dtypertorrrr#s r initialize_hookz$LayerwiseCastingHook.initialize_hook6# **9J9J K r"ctd)NapLayerwiseCastingHook does not support deinitialization. A model once enabled with layerwise casting will have casted its weights to a lower precision dtype for storage. Casting this back to the original dtype will lead to precision loss, which might have an impact on the model's generation quality. The model should be re-initialized and loaded in the original dtype.)NotImplementedErrorr)s r deinitalize_hookz%LayerwiseCastingHook.deinitalize_hook:s! B  r"cX|j|j|j||fSr%)r(rrrr#argskwargss r pre_forwardz LayerwiseCastingHook.pre_forwardBs( **9J9J KV|r"cT|j|j|j|Sr%r')rr#outputs r post_forwardz!LayerwiseCastingHook.post_forwardFr+r")__name__ __module__ __qualname____doc__ _is_statefultorchr&boolr!nnModuler*r.r3r6r"r rr(s L)ekk)%++)]a)fj) ehhoo uxx %((//588??r"rcDeZdZdZdej j fdZy)PeftInputAutocastDisableHookas A hook that disables the casting of inputs to the module weight dtype during the forward pass. By default, PEFT casts the inputs to the weight dtype of the module, which can lead to precision loss. The reasons for needing this are: - If we don't add PEFT layers' weight names to `skip_modules_pattern` when applying layerwise casting, the inputs will be casted to the, possibly lower precision, storage dtype. Reference: https://github.com/huggingface/peft/blob/0facdebf6208139cbd8f3586875acb378813dd97/src/peft/tuners/lora/layer.py#L706 - We can, on our end, use something like accelerate's `send_to_device` but for dtypes. This way, we can ensure that the inputs are casted to the computation dtype correctly always. However, there are two goals we are hoping to achieve: 1. Making forward implementations independent of device/dtype casting operations as much as possible. 2. Performing inference without losing information from casting to different precisions. With the current PEFT implementation (as linked in the reference above), and assuming running layerwise casting inference with storage_dtype=torch.float8_e4m3fn and compute_dtype=torch.bfloat16, inputs are cast to torch.float8_e4m3fn in the lora layer. We will then upcast back to torch.bfloat16 when we continue the forward pass in PEFT linear forward or Diffusers layer forward, with a `send_to_dtype` operation from LayerwiseCastingHook. This will be a lossy operation and result in poorer generation quality. r#c|t|5|jj|i|cdddS#1swYyxYwr)rfn_reforiginal_forwardr0s r new_forwardz(PeftInputAutocastDisableHook.new_forward`s< ( 0 A/4;;//@@ A A As2;N)r7r8r9r:r<r>r?rFr@r"r rBrBKs(A%((//Ar"rBr#rrskip_modules_pattern.skip_modules_classesrrcv|dk(rt}||t||||yt||||||t|y)a Applies layerwise casting to a given module. The module expected here is a Diffusers ModelMixin but it can be any nn.Module using diffusers layers or pytorch primitives. Example: ```python >>> import torch >>> from diffusers import CogVideoXTransformer3DModel >>> transformer = CogVideoXTransformer3DModel.from_pretrained( ... model_id, subfolder="transformer", torch_dtype=torch.bfloat16 ... ) >>> apply_layerwise_casting( ... transformer, ... storage_dtype=torch.float8_e4m3fn, ... compute_dtype=torch.bfloat16, ... skip_modules_pattern=["patch_embed", "norm", "proj_out"], ... non_blocking=True, ... ) ``` Args: module (`torch.nn.Module`): The module whose leaf modules will be cast to a high precision dtype for computation, and to a low precision dtype for storage. storage_dtype (`torch.dtype`): The dtype to cast the module to before/after the forward pass for storage. compute_dtype (`torch.dtype`): The dtype to cast the module to during the forward pass for computation. skip_modules_pattern (`Tuple[str, ...]`, defaults to `"auto"`): A list of patterns to match the names of the modules to skip during the layerwise casting process. If set to `"auto"`, the default patterns are used. If set to `None`, no modules are skipped. If set to `None` alongside `skip_modules_classes` being `None`, the layerwise casting is applied directly to the module instead of its internal submodules. skip_modules_classes (`Tuple[Type[torch.nn.Module], ...]`, defaults to `None`): A list of module classes to skip during the layerwise casting process. non_blocking (`bool`, defaults to `False`): If `True`, the weight casting operations are non-blocking. autoN)DEFAULT_SKIP_MODULES_PATTERNapply_layerwise_casting_hook_apply_layerwise_casting_disable_peft_input_autocast)r#rrrGrHrs r apply_layerwise_castingrOesTbv%;#(<(D$V]M<X !(r"_prefixc t|duxr t||xs|duxrtfd|D}|rtjddyt|tr(tjddt ||||y|j D]"\}} rd|n|} t| |||||| $y)Nc3JK|]}tj|ywr)research).0patternrPs r z+_apply_layerwise_casting..s0qQX7G1L0qs #z&Skipping layerwise casting for layer ""z%Applying layerwise casting to layer ".)rP) isinstanceanyloggerdebugr rLnamed_childrenrM) r#rrrGrHrrP should_skipname submodule layer_names ` r rMrMs(t3` 6K_8`D(qS0q\p0q-q =gYaHI&9: r?r&strr=rOrMrLrnrNr@r"r rzs# // BB4* H .5^'8&9&\ocS[>\#&87 9 FA9A<9?HL @) HHOO@);;@);;@) U38_ 45 @) #5ehhoo)>)C#DE @)  @) @)N7;HL HHOO ;; ;; #5c?3  #5ehhoo)>)C#DE       D: HHOO:,1KK:HM :cg: :*TFFTFr"