biz54dZddlZddlmZddlmZddlmZmZmZm Z ddl m Z er ddl Z ddl mZerddlZerddlZdd lmZdd lmZmZe j,eZ dd Zdd Zddd ZdZ ddZ ddZdeeeffdZ y)z Adapted from https://github.com/huggingface/transformers/blob/c409cd81777fb27aadc043ed3d8339dbc020fb3b/src/transformers/integrations/bitsandbytes.py N) signature)Union)is_accelerate_availableis_bitsandbytes_availableis_torch_availablelogging)QuantizationMethod)init_empty_weights)add_hook_to_moduleremove_hook_from_modulec  |jD]9\}}|g}|j|t|tjr||vrdj | t fd|Dst5|j}|j}|jdk(rTtjj|||jdu|j|j|j |<d}n|j"||j"vrndt%t'tjj(j*vrd|j,ini} tjj(|||jdu|j.f|j0|j2d| |j |<d}t5||j |_|j |j9d dddt;t%|j=d kDrt?||||| \} }|jAd <||fS#1swYYxYw) z Private method that wraps the recursion for module replacement. Returns the converted model and a boolean that indicates if the conversion has been successful or not. N.c3:K|]}|dzvxs|k(ywrN.0keycurrent_key_name_strs b/home/cdr/jupyterlab/.venv/lib/python3.12/site-packages/diffusers/quantizers/bitsandbytes/utils.py z+_replace_with_bnb_linear..>.Y\s22T?S8STllm_int8)has_fp16_weights thresholdT quant_storage)compress_statistics quant_typeFr)has_been_replaced)!named_childrenappend isinstancennLinearjoinanyr in_features out_featuresquantization_methodbnb Linear8bitLtbiasllm_int8_has_fp16_weightllm_int8_threshold_modulesllm_int8_skip_moduleslistr Linear4bit parametersbnb_4bit_quant_storagebnb_4bit_compute_dtypebnb_4bit_use_double_quantbnb_4bit_quant_typetype source_clsrequires_grad_lenchildren_replace_with_bnb_linearpop) modelmodules_to_not_convertcurrent_key_namequantization_configr"namemoduler+r, extra_kwargs_rs @rrArA*sU,,.;! f  #! % fbii (T9O-O#&88,<#= `v()&?"("4"4K#)#6#6L*>>@JN/2vv/B/B'("KKt3-@-Y-Y&9&L&L 0C0t,-1)0EEQ $(;(Q(Q Q $3d9SVVEVEV;W;b;b6c#c"12E2\2\ ]%') 47663D3D + , & 4 7 3 J J 4 5H4a4a+>+R+R 4#/4ENN4015-6:6lENN4(3NN4(77>M&?N tFOO%& '! +#;& #"3 $ A  R w;!x # ##c&?&?s >E=II ct||||\}}td|jD}|stj d|S)a Helper function to replace the `nn.Linear` layers within `model` with either `bnb.nn.Linear8bit` or `bnb.nn.Linear4bit` using the `bitsandbytes` library. References: * `bnb.nn.Linear8bit`: [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://huggingface.co/papers/2208.07339) * `bnb.nn.Linear4bit`: [QLoRA: Efficient Finetuning of Quantized LLMs](https://huggingface.co/papers/2305.14314) Parameters: model (`torch.nn.Module`): Input model or `torch.nn.Module` as the function is run recursively. modules_to_not_convert (`List[`str`]`, *optional*, defaults to `[]`): Names of the modules to not convert in `Linear8bitLt`. In practice we keep the `modules_to_not_convert` in full precision for numerical stability reasons. current_key_name (`List[`str`]`, *optional*): An array to track the current key of the recursion. This is used to check whether the current key (part of it) is not in the list of modules to not convert (for instances modules that are offloaded to `cpu` or `disk`). quantization_config ('transformers.utils.quantization_config.BitsAndBytesConfig'): To configure and manage settings related to quantization, a technique used to compress neural network models by reducing the precision of the weights and activations, thus making models more efficient in terms of both storage and computation. c3K|]E\}}t|tjjtjjfGywN)r&r.r'r6r/rrJreplaced_modules rrz*replace_with_bnb_linear..s< A ?SVV%6%68K8K$LMsA A zYou are loading your model in 8bit or 4bit but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.)rAr* named_modulesloggerwarning)rCrDrErFrJr"s rreplace_with_bnb_linearrSusY4(/EGWYlmHE1"'"5"5"7   Lct|tjjst dt |d|j j}|dvr|S|dk(rtjj|j|j}d|jd}|r#d|jd|}|j|}tj!||S|j"|j"|_t%tjd r5tjj'|j|j"}n,|j|j"j)d d zd z}|r|j|}|S) z Helper function to dequantize 4bit or 8bit bnb weights. If the weight is not a bnb quantized weight, it will be returned as is. z1Input weight should be of type nn.Parameter, got z instead) Params4bit Int8ParamsrVz(The model is going to be dequantized in z - if you want to upcast it to another dtype, make sure to pass the desired dtype when quantizing the model through `bnb_4bit_quant_type` argument of `BitsAndBytesConfig`z.The model is going to be first dequantized in z and type-casted to int8_vectorwise_dequantr#g@ ?)r&torchr' Parameter TypeErrorr< __class____name__r. functionaldequantize_4bitdata quant_statedtypetorQ warning_onceSCBhasattrrXview)weightstaterccls_name output_tensormsg dequantizeds rdequantize_bnb_weightrosX fehh00 1KDQWL>Yabcc((H33 <66v{{FDVDVW 89L9L8MNxy B=CVCVBWWklqkrsC),,U3MC  yyJJ s~~89nn<.rrr0)r0_hf_hookTr)rcrDrErFr"r#)"r-r.r'r/r6r$r%r&r)r*rqridevicer rZr(r+r,rjr[ror0rgrrrr rdr3r?r5r@_dequantize_and_replacerB)rCrcrDrErFr" quant_method target_clsrGrHr0r new_modulerjrxr~rJrs @rrrs'::.)s(9KO ?EHHOO4s13zkSome linear modules were not dequantized. This could lead to unexpected behaviour. Please check your model.)rrcr*rPrQrR)rCrDrFrJr"s rdequantize_and_replacers` ' kk5/ HE1 OTObObOd  y  LrTreturnct|dxr,|jxrt|ddtjk(}t|dxr,|j xrt|ddtjk(}|xs|||fS)Nis_loaded_in_4bitr-is_loaded_in_8bit)rgrrqr BITS_AND_BYTESr)rHis_loaded_in_4bit_bnbis_loaded_in_8bit_bnbs r_check_bnb_statusr4s+, ^  $ $ ^ F14 8rs dd4-L   H %   H$V'V"J(  E$T , htTz!2 hrT