hAdZddlZddlZddlZddlZddlmZmZmZm Z m Z m Z ddl Z ddl mZddlmZddlmZddlmZGd d eZy) a Redis Semantic Cache implementation for LiteLLM The RedisSemanticCache provides semantic caching functionality using Redis as a backend. This cache stores responses based on the semantic similarity of prompts rather than exact matching, allowing for more flexible caching of LLM responses. This implementation uses RedisVL's SemanticCache to find semantically similar prompts and their cached responses. N)AnyDictListOptionalTuplecast) print_verbose)get_str_from_messages)EmbeddingResponse) BaseCachecDeZdZUdZdZeed< ddeedeedeedeed eed ed eefd Z d ee fdZ ded e efdZ ded efdZdeded dfdZded efdZded e efdZdeded dfdZded efdZd eeeffdZde eeefd dfdZy)RedisSemanticCachea Redis-backed semantic cache for LLM responses. This cache uses vector similarity to find semantically similar prompts that have been previously sent to the LLM, allowing for cache hits even when prompts are not identical but carry similar meaning. litellm_semantic_cache_indexDEFAULT_REDIS_INDEX_NAMENhostportpassword redis_urlsimilarity_thresholdembedding_model index_namec "ddlm} ddlm} | |j}t d|| t d||_d|z |_||_ |Q |xstjd}|xstjd }|xstjd }d|d|d|}t d|| |j} | ||| |jd|_y#t$r'} | jd} t d | d | d | d} ~ wwxYw)a- Initialize the Redis Semantic Cache. Args: host: Redis host address port: Redis port password: Redis password redis_url: Full Redis URL (alternative to separate host/port/password) similarity_threshold: Threshold for semantic similarity (0.0 to 1.0) where 1.0 requires exact matches and 0.0 accepts any match embedding_model: Model to use for generating embeddings index_name: Name for the Redis index ttl: Default time-to-live for cache entries in seconds **kwargs: Additional arguments passed to the Redis client Raises: Exception: If similarity_threshold is not provided or required Redis connection information is missing r) SemanticCache)CustomTextVectorizerNz*Redis semantic-cache initializing index - z2similarity_threshold must be provided, passed Noner REDIS_HOST REDIS_PORTREDIS_PASSWORDz&Missing required Redis configuration: z . Provide z or redis_url.z redis://:@:z Redis semantic-cache redis_url: F)namer vectorizerdistance_threshold overwrite)redisvl.extensions.llmcacherredisvl.utils.vectorizerrr ValueErrorrr#rosenvironKeyErrorargs_get_embeddingllmcache)selfrrrrrrrkwargsrre missing_varcache_vectorizers `/var/www/Befach/backend/env/lib/python3.12/site-packages/litellm/caching/redis_semantic_cache.py__init__zRedisSemanticCache.__init__'sP< >@  66JB:,OP 'QR R%9! #$&:":.   7rzz,77rzz,7#Crzz2B'C$H:QtfAdV; sAC D'"D  Dreturnc B|jd}| t|}|S)a Get the TTL (time-to-live) value for cache entries. Args: **kwargs: Keyword arguments that may contain a custom TTL Returns: Optional[int]: The TTL value in seconds, or None if no TTL should be applied ttl)getint)r.r/r7s r3_get_ttlzRedisSemanticCache._get_ttlxs%jj ?c(C promptc tttj|j|ddd}|ddd}|S)z Generate an embedding vector for the given prompt using the configured embedding model. Args: prompt: The text to generate an embedding for Returns: List[float]: The embedding vector Tzno-storezno-cachemodelinputcachedatar embedding)rr litellmrDr)r.r<embedding_responserDs r3r,z!RedisSemanticCache._get_embeddingsP"    **#'T:  'v.q1+> r;cached_responsecD||St|tr|jd} tj|}|S#tj $rN t j|}Y|S#ttf$r"}tdt|Yd}~Yyd}~wwxYwwxYw)z Process the cached response to prepare it for use. Args: cached_response: The raw cached response Returns: The processed cache response, or None if input was None Nzutf-8zError parsing cached response: ) isinstancebytesdecodejsonloadsJSONDecodeErrorast literal_evalr' SyntaxErrorr str)r.rGr0s r3_get_cache_logicz#RedisSemanticCache._get_cache_logics  "" " ou --44W=O "jj9O##  "%"2"2?"C   ,  ?AxHI  s->BA**B9BBBBkeyvaluec td|d} |jdg}|s tdyt|}t|}|jdi|}|(|j j ||t|y|j j ||y#t$r(}td|xs|dt|Yd}~yd}~wwxYw) a> Store a value in the semantic cache. Args: key: The cache key (not directly used in semantic caching) value: The response value to cache **kwargs: Additional arguments including 'messages' for the prompt and optional 'ttl' for time-to-live z(Redis semantic-cache set_cache, kwargs: Nmessages)No messages provided for semantic caching)r7zError setting z in the Redis semantic cache: ) r r8r rRr:r-storer9 Exception) r.rTrUr/ value_strrWr<r7r0s r3 set_cachezRedisSemanticCache.set_caches @IJ#'  zz*b1HIJ*84FE I $--)&)C ##FI3s8#D ##FI6   !3e 44RSVWXSYRZ[   s#B!AB!B!! C*C  Cc td| |jdg}|s tdyt|}|jj |}|sy|d}t |d}d|z }|d } |d } td |j d |d |d| |j| S#t$r!} tdt| Yd} ~ yd} ~ wwxYw)aN Retrieve a semantically similar cached response. Args: key: The cache key (not directly used in semantic caching) **kwargs: Additional arguments including 'messages' for the prompt Returns: The cached response if a semantically similar prompt is found, else None z(Redis semantic-cache get_cache, kwargs: rW.No messages provided for semantic cache lookupN)r<rvector_distancer r<response!Cache hit: similarity threshold: , actual similarity: , current prompt: , cached prompt: rGz,Error retrieving from Redis semantic cache: ) r r8r r-checkfloatrrSr[rR) r.rTr/rWr<results cache_hitr` similarity cached_promptrGr0s r3 get_cachezRedisSemanticCache.get_caches @IJ$ Szz*b1HNO*84Fmm)))8G  I#I.?$@AO _,J%h/M' 3O 3D4M4M3NO&&0\2##)(+""/2 (((I I S HQQ R R Ss#B/)B/AB// C8CCc NKddlm}m}||Dcgc]}|d c}ng} ||j|vrq|j dij dd}|j |j|ddd |d|j dij d dd  d{}n-t j |j|ddd  d{}|dddScc}w7B7#t$r4} tdt| tdt| | d} ~ wwxYww)a Asynchronously generate an embedding for the given prompt. Args: prompt: The text to generate an embedding for **kwargs: Additional arguments that may contain metadata Returns: List[float]: The embedding vector r)llm_model_list llm_routerN model_namemetadata user_api_keyTr>trace_id)rszsemantic-cache-embeddingru)r@rArBrrr?rCrDz"Error generating async embedding: zFailed to generate embedding: ) litellm.proxy.proxy_serverrorprr8 aembeddingrEr[r rRr') r.r<r/rorpmrouter_model_namesrsrFr0s r3_get_async_embeddingz'RedisSemanticCache._get_async_embeddingsP J )'5 5Q|_ 5   O%$*>*>BT*T%zz*b9==nbQ +5+@+@.. '+>(448$*JJz2$>$B$B:t$T ,A , &",3+=+=.. '+>,&"&f-a0= =9 6 && O >s1vhG H=c!fXFGQ N OsWD% CD%A;C%C!-C% C# C%D%!C%#C%% D"./DD""D%cKtd| |jdg}|s tdyt|}t|}|j|fi|d{}|j di|}|(|j j||||d{y|j j|||d{y7g7.7 #t$r!} tdt| Yd} ~ yd} ~ wwxYww) aM Asynchronously store a value in the semantic cache. Args: key: The cache key (not directly used in semantic caching) value: The response value to cache **kwargs: Additional arguments including 'messages' for the prompt and optional 'ttl' for time-to-live z.Async Redis semantic-cache set_cache, kwargs: rWrXN)vectorr7)r|zError in async_set_cache: rY) r r8r rRrzr:r-astorer[) r.rTrUr/rWr<r\prompt_embeddingr7r0s r3async_set_cachez"RedisSemanticCache.async_set_cache?s FvhOP Azz*b1HIJ*84FE I&?T%>%>v%P%PP  $--)&)Cmm**+ +mm**++ Q   A 6s1vh? @ @ AszC9C C9,C C:C CC C9!C C C C9C C C C6C1,C91C66C9c Ktd| |jdg}|s!tdd|jdid<yt|}|j|fi|d{}|j j ||d{}|sd|jdid<y|d }t|d }d |z } |d } |d } | |jdid<td|jd| d|d| |j| S77#t$r6} tdt| d|jdid<Yd} ~ yd} ~ wwxYww)a] Asynchronously retrieve a semantically similar cached response. Args: key: The cache key (not directly used in semantic caching) **kwargs: Additional arguments including 'messages' for the prompt Returns: The cached response if a semantically similar prompt is found, else None z.Async Redis semantic-cache get_cache, kwargs: rWr_grrzsemantic-similarityN)r<r|rr`r r<rarbrcrdrerfzError in async_get_cache: ) r r8 setdefaultr rzr-acheckrhrrSr[rR) r.rTr/rWr<r~rirjr`rkrlrGr0s r3async_get_cachez"RedisSemanticCache.async_get_cachejs FvhOP/ Kzz*b1HNOKN!!*b12GH*84F&?T%>%>v%P%PP !MM00GW0XXG!!*b1) I#I.?$@AO _,J%h/M' 3OHRF  j" -.C D 3D4M4M3NO&&0\2##)(+""/2 (((I IC QY> K 6s1vh? @GJF  j" -.C D KskE4DE!D(D)$D DD)E*A)DEDD E!,E EEEcK|jjd{}|jd{S77w)z Get information about the Redis index. Returns: Dict[str, Any]: Information about the Redis index N)r-_get_async_indexinfo)r.aindexs r3 _index_infozRedisSemanticCache._index_infos5}}5577[[]""8"sA=A?AA cache_listcK g}|D]+}|j|j|d|dfi|-tj|d{y7#t$r!}t dt |Yd}~yd}~wwxYww)z Asynchronously store multiple values in the semantic cache. Args: cache_list: List of (key, value) tuples to cache **kwargs: Additional arguments rr Nz#Error in async_set_cache_pipeline: )appendrasynciogatherr[r rR)r.rr/tasksvalr0s r3async_set_cache_pipelinez+RedisSemanticCache.async_set_cache_pipelines~ JE! 1T11#a&#a&KFKL"..%( ( ( J ?AxH I I JsABAA A ABA A=A83B8A==B)NNNNNztext-embedding-ada-002N)__name__ __module__ __qualname____doc__rrR__annotations__rrhr4r9r:rr,rrSr]rmrzrrrrrrrYr;r3rrs%CcB#""&#'047$(O smO smO 3- O C= O 'uo O O SMO b HSM ST%[,: S  4 D1SS1Ss1Sf.O.O4;.O`)A)AS)At)AV<K<K3<K|#4S>#JuS#X/J Jr;r)rrOrrLr(typingrrrrrrrElitellm._loggingr 8litellm.litellm_core_utils.prompt_templates.common_utilsr litellm.types.utilsr base_cacher rrYr;r3rsA   99*2!fJfJr;