hz "ddlZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z ddlmZddlZddlmZddlmZddlmZddldd lmZmZd d lmZd d lmZd d lm Z d dl!m"Z"d dl#m$Z$d dl%m&Z&d dl'm(Z(d dl)m*Z*d dl+m,Z,d dl-m.Z.dZ/Gdde0eZ1GddZ2e3jhdddgdfde e3de e0de e0de e0de e e5f d Z6e3jhdddgdfde e3de e0de e0de e0de e e5f d!Z7d"Z8y)#N)Enum)AnyDictListOptionalTupleUnion) BaseModel)verbose_logger)CACHED_STREAMING_CHUNK_DELAY)ModelParamHelper)*)EmbeddingResponseall_litellm_params)AzureBlobCache) BaseCache) DiskCache) DualCache) InMemoryCache)QdrantSemanticCache) RedisCache)RedisClusterCache)RedisSemanticCache)S3Cachec tj|tjr t |yy#t $rYywxYw)N)r debuglitellm set_verboseprint Exception)print_statements S/var/www/Befach/backend/env/lib/python3.12/site-packages/litellm/caching/caching.py print_verboser$'s= _-    / "     s04 AAceZdZdZdZy) CacheMode default_on default_offN)__name__ __module__ __qualname__r'r(r#r&r&0s JKr-r&cEFeZdZejej ddddddddgddddddddddddddddddddddddf"deedeedeedeed eed eed ee d ee d ee dee dee e deedeedeedeedeedee dee e efdeedeedeedeedeedeededeedeed ee d!eed"eed#eed$eed%eed&efDd'Zd(efd)Zd*ed+ed(eefd,Zd+ed(efd-Zd.ed/eed(eefd0Zd+ed(efd1Zd(eefd2Zd3ed(dfd4Zed5ed(efd6Zd7ed(efd8Zd9Zd:eed;ee fd<Zd=Zd>Zd?Z d@Z!dAZ"dBedCeed(e#fdDZ$ dPdEe%dFed+edGed(e&eeeff dHZ'dIZ(dJZ)dKZ*dLZ+dMZ,dNZ-d(e fdOZ.y)QCacheN completion acompletion embedding aembeddingatranscription transcriptionatext_completiontext_completionarerankrerankTztext-embedding-ada-002typemodehostportpassword namespacettldefault_in_memory_ttldefault_in_redis_ttlsimilarity_thresholdsupported_call_typesazure_account_urlazure_blob_containers3_bucket_names3_region_names3_api_version s3_use_ssl s3_verifys3_endpoint_urls3_aws_access_key_ids3_aws_secret_access_keys3_aws_session_token s3_configs3_path$redis_semantic_cache_embedding_modelredis_semantic_cache_index_nameredis_flush_sizeredis_startup_nodesdisk_cache_dirqdrant_api_baseqdrant_api_keyqdrant_collection_nameqdrant_quantization_config%qdrant_semantic_cache_embedding_modelc# d|tjk(r0|rtd |||||d|#|_nt d ||||d|#|_n|tj k(rt d |||| ||d|#|_n|tjk(rt||| | |!|"|_n|tjk(rt|_nz|tjk(rtd |||||||||||d |#|_nJ|tjk(rt| | |_n$|tjk(rt!||_dt"j$vrt"j$j'ddt"j(vrt"j*j-ddt"j.vrt"j*j1d| |_||_||_||_||_|xst<j>|_ |j4tjk(r |||_|j4tjk(s|j4tj k(r | | |_|j67tC|jtr|j6|j_y y y ) a Initializes the cache based on the given type. Args: type (str, optional): The type of cache to initialize. Can be "local", "redis", "redis-semantic", "qdrant-semantic", "s3" or "disk". Defaults to "local". # Redis Cache Args host (str, optional): The host address for the Redis cache. Required if type is "redis". port (int, optional): The port number for the Redis cache. Required if type is "redis". password (str, optional): The password for the Redis cache. Required if type is "redis". namespace (str, optional): The namespace for the Redis cache. Required if type is "redis". ttl (float, optional): The ttl for the Redis cache redis_flush_size (int, optional): The number of keys to flush at a time. Defaults to 1000. Only used if batch redis set caching is used. redis_startup_nodes (list, optional): The list of startup nodes for the Redis cache. Defaults to None. # Qdrant Cache Args qdrant_api_base (str, optional): The url for your qdrant cluster. Required if type is "qdrant-semantic". qdrant_api_key (str, optional): The api_key for the local or cloud qdrant cluster. qdrant_collection_name (str, optional): The name for your qdrant collection. Required if type is "qdrant-semantic". similarity_threshold (float, optional): The similarity threshold for semantic-caching, Required if type is "redis-semantic" or "qdrant-semantic". # Disk Cache Args disk_cache_dir (str, optional): The directory for the disk cache. Defaults to None. # S3 Cache Args s3_bucket_name (str, optional): The bucket name for the s3 cache. Defaults to None. s3_region_name (str, optional): The region name for the s3 cache. Defaults to None. s3_api_version (str, optional): The api version for the s3 cache. Defaults to None. s3_use_ssl (bool, optional): The use ssl for the s3 cache. Defaults to True. s3_verify (bool, optional): The verify for the s3 cache. Defaults to None. s3_endpoint_url (str, optional): The endpoint url for the s3 cache. Defaults to None. s3_aws_access_key_id (str, optional): The aws access key id for the s3 cache. Defaults to None. s3_aws_secret_access_key (str, optional): The aws secret access key for the s3 cache. Defaults to None. s3_aws_session_token (str, optional): The aws session token for the s3 cache. Defaults to None. s3_config (dict, optional): The config for the s3 cache. Defaults to None. # Common Cache Args supported_call_types (list, optional): List of call types to cache for. Defaults to cache == on for all call types. **kwargs: Additional keyword arguments for redis.Redis() cache Raises: ValueError: If an invalid cache type is provided. Returns: None. Cache is set as a litellm param )r=r>r?rU startup_nodes)r=r>r?rU)r=r>r?rDembedding_model index_name)rXrYcollection_namerDquantization_configr_) rHrIrJrKrLrMrNrOrPrQrR) account_url container)rWcacheNr,)"LiteLLMCacheTypeREDISrrerREDIS_SEMANTICrQDRANT_SEMANTICrLOCALrS3r AZURE_BLOBrDISKrrinput_callbackappendsuccess_callbacklogging_callback_manageradd_litellm_success_callback_async_success_callback"add_litellm_async_success_callbackrEr;r@rUrAr&r'r< isinstance)$selfr;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\kwargss$ r#__init__zCache.__init__7sD #)) )"(9)%%5"5 )  ) (%%5    %44 4+!%9 D: DJ%55 5, /- 6%9$> E DJ%++ +&DJ %(( ( ---%# /%9)A%9#  DJ%00 0'-.DJ%** *".ADJ '00 0  " " ) )' 2 '22 2  , , I I' R '99 9  , , O OPW X$8! " 0#;y';'; 99(.. .3H3T,DH II)// /yy,;;;".+DH >> %*TZZ*L#'>>DJJ +M %r-returnc 2d}|jdi|}|tjd||Stj}t }|D]w}||vr2|j ||}||t|dt|z }9||vs>tjdusQ||W||}|t|dt|z }ytjd|tj|}|j|fi|}|jdd|i||S)a  Get the cache key for the given arguments. Args: **kwargs: kwargs to litellm.completion() or embedding() Returns: str: The cache key generated from the arguments, or None if no cache key could be generated. z Returning preset cache key: %sz: Tz Created cache key: %spreset_cache_keyr,)!_get_preset_cache_key_from_kwargsr rr _get_all_llm_api_paramsr_get_param_valuestrr3enable_caching_on_provider_specific_optional_paramsr/_get_hashed_cache_key_add_namespace_to_cache_key_set_preset_cache_key_in_kwargs) rvrw cache_keyr|combined_kwargslitellm_param_kwargsparam param_valuehashed_cache_keys r# get_cache_keyzCache.get_cache_keysR B4AAKFK  '  !CEU V# #*BBD1E'-1-B-B5&-Q *CJ >vg>r-rrc\|jdg}|r|D]}||vst|cSy)Ncaching_groups)rr)rvrrrgroups r#rzCache._get_caching_group:s9*26F)K '%'u:%(r-c|jd}|jdi}|jdi}|jdxs3t|ddxs$|jdxs|jdS)zo Handles getting the value for the 'file' param from kwargs. Used for `transcription` requests rrr file_checksumnameN file_name)rgetattr)rvrwrrrs r#rzCache._get_file_param_valueDs{zz&!::j"-$4b9 LL ) /tVT* /||K( /!!+.  r-c :|rd|vr|djddSy)a  Get the preset cache key from kwargs["litellm_params"] We use _get_preset_cache_keys for two reasons 1. optional params like max_tokens, get transformed for bedrock -> max_new_tokens 2. avoid doing duplicate / repeated work rr|N)r)rvrws r#r}z'Cache._get_preset_cache_key_from_kwargsRs, 6)./334FMMr-r|c $|rd|vr ||dd<yyy)z Set the calculated cache key in kwargs This is used to avoid doing duplicate / repeated work Placed in kwargs["litellm_params"] rr|Nr,)rvr|rws r#rz%Cache._set_preset_cache_key_in_kwargs`s* 6)?O'();<* r-rctj|j}|j}t j d||S)z Get the hashed cache key for the given cache key. Use hashlib to create a sha256 hash of the cache key Args: cache_key (str): The cache key to hash. Returns: str: The hashed cache key. zHashed cache key (SHA-256): %s)hashlibsha256encode hexdigestr r)r hash_objecthash_hexs r#rzCache._get_hashed_cache_keyls?nnY%5%5%78 ((*=xHr-rc |jdi}|jdxs/|jdijdxs |j}|r|d|}tjd||S)a If a redis namespace is provided, add it to the cache key Args: hash_hex (str): The hashed cache key. **kwargs: Additional keyword arguments. Returns: str: The final hashed cache key with the redis namespace. rer@rredis_namespace:zFinal hashed key: %s)rr@r r)rvrrwdynamic_cache_controlr@s r#rz!Cache._add_namespace_to_cache_keys~6r-c#Kd}tdt||D]-}ddd||||zdigitjt/yw)Nrchoicesdelta assistant)rolecontent)rangelentimesleepr )rvr chunk_sizeis r#generate_streaming_contentz Cache.generate_streaming_contentsa q#g, 3A$/'.q1z>'B"   JJ3 44sA A  cached_resultmax_agecB|wt|trgd|vrc|d}tj}||z }|||kDry|jd} t|tr |St j |} |S|S#t $rtj|}Y|SwxYw)zL Common get cache logic across sync + async implementations N timestampresponse) rudictrrjsonloadsr!ast literal_eval)rvrrr current_time response_agecached_responses r#_get_cache_logiczCache._get_cache_logics  %=$/},%k2I99;L()3L"|g'=,// ;O Dot4# " '+jj''O # " D"%"2"2?"C" " DsA<"A<<BBc  |jd i|dury|jdg}d|vr|d}n|jd i|}||jdi}|jdxs|jdxs td}|jj || }|jj || }|j || Sy#t$r#td tjYywxYw) a& Retrieves the cached result for the given arguments. Args: *args: args to litellm.completion() or embedding() **kwargs: kwargs to litellm.completion() or embedding() Returns: The cached result if it exists, otherwise None. TNmessagesrres-maxage s-max-ageinf)rrrAn exception occurred: r,) should_use_cacherrfloatre get_cacherr!r$ traceback format_exc)rvrwrrcache_control_argsrrs r#rzCache.get_caches& $t$$.v.d:zz*b1Hf$";/ .D..88 $:@**Wb:Q"&**:6$)--k:$U| !% 4 4Y 4 R $ 4 4Y 4 R ,,"/-%  3I4H4H4J3KL M sCB?C)DDc K |jd i|dury|jdgd|vr|d}n|jd i|}|u|jdi}|jd|jdtd}|jj |fi|d{}|j || Sy7#t$r#td tjYywxYww) zd Async get cache implementation. Used for embedding calls in async wrapper TNrrrerrrrrr,) rrrrreasync_get_cacherr!r$rr)rvrwrrrrs r#rzCache.async_get_caches $t$$.v.d: JJz2 &f$";/ .D..88 $%+ZZ%<",00!3!7!7 E%L!Q'Adjj&@&@&Uf&U U ,,"/- % !V  3I4H4H4J3KL M sEC0CC0B C&B?'C=C0?C)C-*C0,C--C0c  d|vr|d}n|jdi|}|t|tr|j}|j|j|d<|j dd}t|t r#|jD]\}}|dk(s ||d<tj|d}|||fStd#t$r}|d}~wwxYw)zO Common implementation across sync + async add_cache functions rNrAre)rrzcache key is Noner,) rrur model_dump_jsonrArritemsrr!) rvresultrwr _cache_kwargskv cached_dataes r#_add_cache_logiczCache._add_cache_logic s f$";/ .D..88 $fi0#335F88'$(HHF5M & 7D 9 mT2 - 3 3 51:,-F5M!6-1IIKVL  +v55 344 G s$BC"C< C CCCc  |jdi|dury|jdd|i|\}}}|jj||fi|y#t$r+}t j dt|Yd}~yd}~wwxYw)z Adds a result to the cache. Args: *args: args to litellm.completion() or embedding() **kwargs: kwargs to litellm.completion() or embedding() Returns: None TNr#LiteLLM Cache: Excepton add_cache: r,)rrre set_cacher!r exceptionrrvrrwrrrs r# add_cachezCache.add_cache)s U$t$$.v.d:-BT-B-B..!'. *I{F !DJJ K B6 B U  $ $'J3q6(%S T T UsA6A B!A==BcK |jdi|dury|jdk(r(|j|j|fi|d{y|jdd|i|\}}}|j j ||fi|d{y7D7#t$r+}tjdt|Yd}~yd}~wwxYww)z3 Async implementation of add_cache TNredisrrr,) rr;rUbatch_cache_writerreasync_set_cacher!r rrrs r#async_add_cachezCache.async_add_cache>s U$t$$.v.d:yyG#(=(=(I,d,,V>v>>>1F1F1F2!2%+2. ;1djj00KR6RRR ? S U  $ $'J3q6(%S T T UshC BC 1B B BC 9B B BC BB C!C=C CC embedding_responserc t|tr4|jd|jd|jd|dSt|drD|j }|jd|jd|jd|dSt |}|jd|jd|jd|dS#t $r}td|d}~wwxYw)zh Convert any embedding response into the standardized CachedEmbedding TypedDict format. r3indexobject)r3rrr model_dumpz,Missing expected key in embedding response: N)rurrhasattrrvarsKeyError ValueError)rvrrdatars r#_convert_to_cached_embeddingz"Cache._convert_to_cached_embeddingQs Q,d3!3!7!7 !D/33G<044X>"  +\:)446!%+!6!XXg."hhx0" ./!%+!6!XXg."hhx0"   QKA3OP P Qs%ACAC>C C1C,,C1rinputidx_in_result_datac|jdii|d|i}||d<|j|}|j}|j||}|jdd|i|\} } }| | |fS)Nrrrr,)rrrrr) rvrrrwrr|r model_nameembedding_dictrrs r#add_embedding_response_to_cachez%Cache.add_embedding_response_to_cacheqs.4--K0J60J7E0JK.{#[[);<\\ *.*K*KL^`j*k)>)>)>* !* * & ;+v--r-cZK |jdi|dury|j|j|d<g}t|dtrBt |dD]0\}}|j ||||\}}}|j ||f2n@t|dtr-|j ||d|\}}}|j ||f|jjdd|i|d{y7#t$r+}tjdt|Yd}~yd}~wwxYww)z Async implementation of add_cache for Embedding calls Does a bulk write, to prevent using too many clients TNrAr cache_listrr,) rrArulist enumeraterrorreasync_set_cache_pipeliner!r r) rvrrwridxrrrrs r#async_add_cache_pipelinezCache.async_add_cache_pipelines; U$t$$.v.d:xx# $u J&/40'w8FC <? 9F7OS1151U1UF7OV2. ;!!9k":;5$**55VVvV V V U  $ $'J3q6(%S T T UsKD+C4D+CC4,C2-C41D+2C44 D(=!D#D+#D((D+c |jtjk(ry|jdd}t j d|||r%t |tr|jddduryy)z Returns true if we should use the cache for LLM API calls If cache is default_on then this is True If cache is default_off then this is only true when user has opted in to use cache TreNz(should_use_cache: kwargs: %s; _cache: %sz use-cacheF)r<r&r'rr rrur)rvrw_caches r#rzCache.should_use_cachesd 99 ,, ,GT*GQWX j.zz+u-5r-cK|jdd|i|\}}}|jj||fi|d{y7w)Nrr,)rrer)rvrrwrrs r#rzCache.batch_cache_writesI)>)>)>)Wf)WPV)W& ;*djj**9kLVLLLs:AAAc^Kt|jd}|r|d{Sy7w)Npingrre)rv cache_pings r#r z Cache.pings,TZZ0 #% %&s #-+-c`Kt|jd}|r||d{Sy7w)Ndelete_cache_keysr )rvkeyscache_delete_cache_keyss r#rzCache.delete_cache_keyss1")$**6I"J "066 67s $.,.cKt|jdr#|jjd{yy7w)N disconnect)rrerrvs r#rzCache.disconnects2 4::| ,**'') ) ) - )s 4?=?cX|jr|jtjk(ryy)z Internal method to check if the cache type supports async get/set operations Only S3 Cache Does NOT support async operations FT)r;rfrkrs r#_supports_asynczCache._supports_asyncs# 99&6&9&99r-)r)/r)r*r+rfrjr&r'rrrrCachingSupportedCallTypesboolr rintrxrrrrrrr}r staticmethodrrrrrrrrrCachedEmbeddingrrrrrrrr rrrr,r-r#r/r/6se,<+A+A  """&#'#150404 K ,0.2(,(,(,%)04)-.226.2#'!%4L9=*..2(,)-(,04485Mcv2'(v2  v2 sm v2smv23-v2C=v2e_v2 (v2'uov2'uov2't,E'FGv26$C=7v28'sm9v2:! ;v2<! =v2>! ?v2@TNAv2BE$),-Cv2D"#Ev2F'smGv2H#+3-Iv2J'smKv2LC=Mv2N#Ov2P/2Qv2R*2#Sv2T#3-Uv2V&d^Wv2X! Yv2Z"#[v2\! ]v2^!) _v2`%-SMav2b03cv2p* * X    # ?T?c?"+3C= #  D  S   Xc]  P PRV P$Cc, 5%}%%%N!F:<U*U&QsQ8TW=Q]lQJ#$ .!.. .  . sD$  .*&UP$M  *  r-r/r0r;r=r>r?rEc tddtjvrtjjddtjvrtj j ddtjvrtj jdtjtd|||||d|t_ tdtjtdttjy)a Enable cache with the specified configuration. Args: type (Optional[Literal["local", "redis", "s3", "disk"]]): The type of cache to enable. Defaults to "local". host (Optional[str]): The host address of the cache server. Defaults to None. port (Optional[str]): The port number of the cache server. Defaults to None. password (Optional[str]): The password for the cache server. Defaults to None. supported_call_types (Optional[List[Literal["completion", "acompletion", "embedding", "aembedding"]]]): The supported call types for the cache. Defaults to ["completion", "acompletion", "embedding", "aembedding"]. **kwargs: Additional keyword arguments. Returns: None Raises: None zLiteLLM: Enabling CachereNr;r=r>r?rEz&LiteLLM: Cache enabled, litellm.cache=LiteLLM Cache: r,) r$rrnrorprqrrrsrtrer/rr;r=r>r?rErws r# enable_cachersJ+,g,,,%%g.g...((EEgNg555((KKGT}} !5     :7==/JKOD$7#89:r-c tdtd|||||d|t_tdtjtdt tjy)a Update the cache for LiteLLM. Args: type (Optional[Literal["local", "redis", "s3", "disk"]]): The type of cache. Defaults to "local". host (Optional[str]): The host of the cache. Defaults to None. port (Optional[str]): The port of the cache. Defaults to None. password (Optional[str]): The password for the cache. Defaults to None. supported_call_types (Optional[List[Literal["completion", "acompletion", "embedding", "aembedding"]]]): The supported call types for the cache. Defaults to ["completion", "acompletion", "embedding", "aembedding"]. **kwargs: Additional keyword arguments for the cache. Returns: None zLiteLLM: Updating Cacherz&LiteLLM: Cache Updated, litellm.cache=rNr,)r$r/rrerrs r# update_cacher!sgF+,   1   GM:7==/JKOD$7#89:r-cpddlm}td|t5tj j dtjj dtjj dddddt_ tdtjy#1swY1xYw)aF Disable the cache used by LiteLLM. This function disables the cache used by the LiteLLM module. It removes the cache-related callbacks from the input_callback, success_callback, and _async_success_callback lists. It also sets the litellm.cache attribute to None. Parameters: None Returns: None rsuppresszLiteLLM: Disabling CachereNz'LiteLLM: Cache disabled, litellm.cache=) contextlibr$r$rrrnremoverprsrer#s r# disable_cacher'Js$,- * 8%%g.  ''0''..w78 GM;GMM?KL 88s AB,,B5)9rrrrrenumrtypingrrrrrr pydanticr rlitellm._loggingr litellm.constantsr -litellm.litellm_core_utils.model_param_helperr litellm.types.cachinglitellm.types.utilsrrazure_blob_cacher base_cacher disk_cacher dual_cacherin_memory_cacherqdrant_semantic_cacher redis_cacherredis_cluster_cacherredis_semantic_cachers3_cacherr$rr&r/rfrjrrr!r'r,r-r#r:sN  ::+:J#E,!!!*6#24  T g g V(8'='=" G 7; # $7; 3-7; 3-7;sm 7; #4(A#BC 7;v(8'='=" G -; # $-; 3--; 3--;sm -; #4(A#BC -;`Mr-