hddlZddlZddlZddlZddlZddlmZmZddlmZm Z m Z m Z ddl Z ddl m Z mZmZmZmZmZddlZddlmZddlmZddlmZddlmZmZmZmZmZm Z m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'm(Z(m)Z)m*Z*e Z+e+jYd d gee#gdefdZ-e+jYdd gee#gee#e j\dfde de e de/ffdZ0de1fdZ2dZ3 dOde/de4de4de5de e/d e e/f d!Z6 dOd"Z7e+jYd#d gee#gee#e j\dd$e j\dd%fde d&e e/d e e/fd'Z8e+jYd(d gee#gee#e j\dd)e j\dd*e j\d+d,d-d./e j\dd0d1fde d&e e/d2e e/d3e9d4e9f d5Z:e+jYd6d gee#gee#fde fd7Z;d8ejxd9a=d:Z>e+jYd;d gee#ge+jYdZ@e+jYd?d gee#gd@ZAe+jYdAd gBe+jYdCd gBdDZBe+jd?d gee#gdEZDe+jdAd gBe+jdCd gBdFZEe+jdGd gee#ge jdHdIe jddJee#fdedKe e dLdMede fdNZHy)PN)datetime timedelta)DictLiteralOptionalUnion) APIRouterDepends HTTPExceptionRequestResponsestatus)verbose_proxy_logger)HEALTH_CHECK_TIMEOUT_SECONDS)AsyncHTTPHandler) AlertTypeCallInfoLitellm_EntityTypeProxyErrorTypesProxyExceptionUserAPIKeyAuth WebhookEvent)user_api_key_auth)PrismaDBExceptionHandler)_clean_endpoint_data'_update_litellm_params_for_health_checkperform_health_checkrun_with_timeoutz/testhealth)tags dependenciesrequestc:Kd|jjiSw)a [DEPRECATED] use `/health/liveliness` instead. A test endpoint that pings the proxy server to check if it's healthy. Parameters: request (Request): The incoming request. Returns: dict: A dictionary containing the route of the request URL. route)urlpath)r"s l/var/www/Befach/backend/env/lib/python3.12/site-packages/litellm/proxy/health_endpoints/_health_endpoints.py test_endpointr('s$ W[[%% &&sz/health/serviceszSpecify the service being hit.) descriptionuser_api_key_dictservice) slack_budget_alertslangfuseslack openmeterwebhookemail braintrustdatadog generic_apic K ddlm}m}m}|t dddi|dvrt ddd |d d gi|d k(s!|d k(s|dk(s|t j vr>|dk7r9t jddddgddd{}ddj|dS|dk(rk(rd,} np| t:j@k(rd-} nZ| t:j0k(rd.} nD| t:jBk(rd/} n.| t:jDk(rd0} n| t:jFk(rd1} nd.} |jI| d2| 3d{n*|jId4d2t:j03d{|ZtKjL|j4jOtKjL|j4jQ|j4j8xsg}tS|}d|d5d6St d7dd8j|i|d9k(rtUd:t,j.d;|j$xsd#d }|j4j[|?d{dd@dSy77m77g7=7#t\$r}t_j`dAjtc|t_jdtgjhtk|trYtmto|dBdCtc|dDtpjrto|dEdFto|dGttjvHtk|tlr|tmdItc|ztpjrto|dEdFttjvHd}~wwxYww)Jz Use this admin-only endpoint to check if the service is healthy. Example: ``` curl -L -X GET 'http://0.0.0.0:4000/health/services?service=datadog' -H 'Authorization: Bearer sk-1234' ``` r)general_settings prisma_clientproxy_logging_objNierrorzService must be specified. status_codedetail) r,r1r-r.r/r0r2otelcustom_callback_api langsmithr3r4z!Service must be in list. Service=z. List=r,r/r2r4r-z"openai/litellm-mock-response-modeluserzHey, how's it going?)rolecontentzlitellm:/health/serviceszThis is a mock response)modelmessagesr@ mock_responsesuccessz!Mock LLM request made - check {}.)rmessager3) DataDogLoggerr unhealthy error_messagezDatadog is healthy)LangFuseLoggerz'Mock LLM request made - check langfuse.r0)tokenspend max_budgetuser_id key_aliasteam_id event_group user_budget)type user_infor.alertingzdefault test messagezLLM Exception test alertzLLM Too Slow test alertzLLM Requests Hanging test alertzBudget Alert test alertzDB Exception test alertz!Outage Alert Exception test alertzDaily Reports test alertLow)rGlevel alert_typez"This is a test slack alert messagezBMock Slack Alert sent, verify Slack Alert Received on your channel)r alert_typesrGiz@"{}" not in proxy config: general_settings. Unable to test this.r1 key_createdzTest Email AlertzNEmail Test key (This is only a test alert key. DO NOT USE THIS IN PRODUCTION.)TEST_EMAIL_ADDRESS) eventrT event_messagerNrRrOrPrQ user_emailrS) webhook_eventz2Mock Email Alert sent, verify Email Alert ReceivedzMlitellm.proxy.proxy_server.health_services_endpoint(): Exception occured - {}r<zAuthentication Error()paramNoner;)rGrVrdcodezAuthentication Error, )<litellm.proxy.proxy_serverr6r7r8r litellmsuccess_callback acompletionformat$litellm.integrations.datadog.datadogrHasync_health_check&litellm.integrations.langfuse.langfuserKLangfuse auth_check completionrrNrQrRrSrKEY budget_alertsgetslack_alerting_instancealert_to_webhook_urlr\rllm_exceptions llm_too_slowllm_requests_hanging db_exceptions outage_alerts daily_reportsalerting_handlerasyncio create_tasksend_monthly_spend_reportsend_weekly_spend_reportlistrosgetenv&send_key_created_or_user_invited_email Exceptionrr9strdebug traceback format_exc isinstancergetattrr auth_errorrHTTP_500_INTERNAL_SERVER_ERROR)r*r+r6r7r8_rHdatadog_loggerresponserKlangfuse_loggerrWr[ test_messager\rbes r'health_services_endpointr<s>T   ?2N(O     @ RgQhPij  { ",&-'7333:8M)):#)6LMN/7 A$>EEgN  ! J*_N+>>@@H"8, )[8_-  .   " M,.O  $ $ / / 1"":#)6LMN/7 A$D  i  '--3)11+55)11.22I$11"#2   g ,A!A*..z2>>&==RR +BBWW#.EEQQ#'( *#4#L#L#X#X!Y%'= %)A)AA+EL'9+A+AA+DL'9+I+II+LL'9+B+BB+DL'9+B+BB+DL'9+B+BB+NL'9+B+BB+EL+DL/@@$0*A9X@,<< D##,#:#:= !,'')AA[[]'')AAZZ\ &==IIOR#;/ '#.c $ #!c!j!j#" g (#.220'--3j)1199%9:)11 M$;;bb+c   $O ' AAF V^   "" [ b bA  ""9#7#7#9: a ' 8/DSVHA-NO$//a&1Q v/T/TU   > *G,s1v5 ++!Wf-66   ! sUA;P8?P'P8U%P8=P*>P8UP8UA P8&U'A'P8P-D:P8 P0 ,P86P37B P8UBP8P6P8%U'P8*P8-P80P83P86P88 UDUUUreturnc |j|j|j|j|j|j |j |j|j|j|jr|jjnd|jr|jjd Sdd S)z9Convert health check database record to dictionary formatN) health_check_id model_namemodel_idr healthy_countunhealthy_countrJresponse_time_msdetails checked_by checked_at created_at) rrrrrrrJrrrr isoformatr)checks r'_convert_health_check_to_dictr2s!00&&NN,,,, 00,,!22==&&6;6F6Fe&&002D6;6F6Fe&&002  MQ  cPddlm}|ttjddi|S)zIHelper to check if prisma_client is available and raise appropriate errorrr7r9zDatabase not initializedr:)rgr7r rrrs r'_check_prisma_clientrDs28==78   rrhealthy_endpointsunhealthy_endpoints start_timerQrc lK |r(|djdrt|ddddnd}|j|||rdndt|t||t j|z dzd| d{y7#t $r%}t jd |d |Yd}~yd}~wwxYww) z8Helper function to save health check results to databaserr9NihealthyrI) rrrrrrJrrrz2Failed to save health check to database for model : )rtrsave_health_check_resultlentimerrwarning) r7rrrrrQrrJdb_errors r'_save_health_check_to_dbrPst#':1'='A'A''J #A&w/ 0# 6  44! 19{/0 34'"iikJ6$>5   t$$'YZdYeeghpgq%rsstsAB4A7B;B<BB4B B1 B,'B4,B11B4c Kt||||d{\}} |-||n|} | %tjt|| || ||||| t |t | dS7Ow)zDHelper function to perform health check and save results to database) model_list cli_modelrCrNr)rrrr)rr~rrr) r target_modelrrr7rrQrrrmodel_name_for_dbs r'_perform_health_check_and_saverrs4H 4.** )2)>IL  (    8!!#!! /2./23  -.sA'A%AA'z/healthz!Specify the model name (optional)zSpecify the model ID (optional)rCc Kddlm}m}m}m}m}m}m} ddl} | j} |} |ri|sg|D |j|} | | j} n!ttjdd|din!ttjdd|di |D|%t#gd||| | |j$d d{Sttj&dd it)j*|}t-|j.dkDrn |r|St#|| d|| | |j$| d{S#t$rA}tj d |d |ttjdd|did}~wwxYw77S#t$r[}tj d j1t3|tj4t7j8|d}~wwxYww)u 🚨 USE `/health/liveliness` to health check the proxy 🚨 See more 👉 https://docs.litellm.ai/docs/proxy/health Check the health of all the endpoints in config.yaml To run health checks in the background, add this to config.yaml: ``` general_settings: # ... other settings background_health_checks: True ``` else, the health checks will be run on models when /health is called. r)health_check_detailshealth_check_resultsllm_model_list llm_routeruse_background_health_checks user_modelr7Nrr9zModel with ID z not foundr:z&Error getting deployment for model_id r)rrrrr7rrQrzModel list not initializedzHlitellm.proxy.proxy_server.py::health_endpoint(): Exception occured - {})rgrrrrrrr7rget_deploymentrr rHTTP_404_NOT_FOUNDrrr9rrQrcopydeepcopyrmodelsrkrrrr)r*rCrrrrrrrr7rrr deploymentr_llm_model_lists r'health_endpointrs44JL  ! '666I )#-#8#8L'$*$=$= '>(:)NO! "55>(:!FG  ,  !%;!!%(0"/)-55!    "AA!=> --7  '' (1 ,   '' '7*),+%)11!   O $**-ST\S]]_`a_b+cd# & 9 9#~hZz%JK  .  "" V ] ]A  ""9#7#7#9:s0G.AD65"G.$F<F=FG.AFG. F1F2F5G.6 F?>' ?  KRR5e<R R* .    S  ""%I!#MN==HQQR   sJ B,AAAAAB,AA B)#AB$$B))B,z/health/latestc Kt} |jd{}|Dcic]0}|jr |jn |jt |2}}|t |dS7Mcc}w#t $rF}tjd|ttjddt|id}~wwxYww)z Get the latest health check status for all models Returns the most recent health check result for each model. N)latest_health_checks total_modelsz$Error getting latest health checks: r9z)Failed to retrieve latest health checks: r:) rget_all_latest_health_checksrrrrrrr9r rrr)r*r7 latest_checksr checks_datars r'latest_health_checks_endpointr;s)*M +HHJJ '  %~~U^^53C3CFcdiFj j  %0 ,  K   ""%I!#MN==HQQR   sJ CA6A/A65A1A6.C/A61A66 C?ACCCunknownr last_updatedcKddlm} tjtdz }tddk7r|t dkrtS|dtjd atS|j d{d tjd atS7!#t$r%}tj|tcYd}~Sd}~wwxYww) Nrrrrr)minutes disconnectedr connected) rgr7rnowdb_health_cacher health_checkrrhandle_db_exception)r7 time_diffrs r'_db_health_readiness_checkr^s8 LLN_^%DD 8 $ 1i)TUBV6V" "  )7XO" "((***%0(,,.Q +  44Q7s^C>BC B'C(B;B< BCB C (CC CC  Cz /settingsz/active/callbacksc rKddlm}m}t|j d}t j Dcgc] }t|}}t jDcgc] }t|}}t jDcgc] }t|}}t jDcgc] }t|}}t jDcgc] }t|}}t jDcgc] }t|} }t jDcgc] }t|} }||z|z|z|z| z| z} |j} d} | rt| tr t!| } ||||||| | | t!| | t j"d Scc}wcc}wcc}wcc}wcc}wcc}wcc}ww)au Returns a list of litellm level settings This is useful for debugging and ensuring the proxy server is configured correctly. Response schema: ``` { "alerting": _alerting, "litellm.callbacks": litellm_callbacks, "litellm.input_callback": litellm_input_callbacks, "litellm.failure_callback": litellm_failure_callbacks, "litellm.success_callback": litellm_success_callbacks, "litellm._async_success_callback": litellm_async_success_callbacks, "litellm._async_failure_callback": litellm_async_failure_callbacks, "litellm._async_input_callback": litellm_async_input_callbacks, "all_litellm_callbacks": all_litellm_callbacks, "num_callbacks": len(all_litellm_callbacks), "num_alerting": _num_alerting, "litellm.request_timeout": litellm.request_timeout, } ``` r)r6r8rX) rXzlitellm.callbackszlitellm.input_callbackzlitellm.failure_callbackzlitellm.success_callbackzlitellm._async_success_callbackzlitellm._async_failure_callbackzlitellm._async_input_callbackall_litellm_callbacks num_callbacks num_alertingzlitellm.request_timeout)rgr6r8rrtrh callbacksinput_callbackfailure_callbackri_async_success_callback_async_failure_callback_async_input_callbackrXrrrrequest_timeout)r6r8 _alertingxlitellm_callbackslitellm_input_callbackslitellm_failure_callbackslitellm_success_callbackslitellm_async_success_callbackslitellm_async_failure_callbackslitellm_async_input_callbacksrrX _num_alertings r'active_callbacksrvsFO$((45I*1):):;AQ;;/6/E/EF!s1vFF181I1I JAQ J J181I1I JAQ J J7>7V7V&W!s1v&W#&W7>7V7V&W!s1v&W#&W5<5R5R$SSV$S!$S  ! " # $ $ $ *  * *  * (  (!))HMJx.H ."9$=$=+J+J)F!623%#*#:#:  /<F J J&W&W$SsY5F7F F7F/F7FF7)F#;F7F(!F75F-F7F2-B F7ct|tr|S |jS#t$r6 |jjcYS#t$rt|cYcYSwxYwwxYwN)rr__name__AttributeError __class__)callbacks r' callback_namer sa(C !   ! !%%.. . !x=  !!s+ AAAAAAAz/health/readinessc Kddlm}m} g} tjDcgc] }t |}}d}tjzddlm }tjj}ttjj|r6 tjjjd{}||d}|/td{}dd|||t!j"d |Sdd |||t!j"d Scc}w#t $rtj}YwxYw7y#t$r}dt|z}Yd}~d}~wwxYw7#t$r}t%d d t|d d}~wwxYww)zJ Unprotected endpoint for checking if worker can receive requests r)r7versionN)RedisSemanticCachezindex does not exist - error: )rV index_inforr)rdbcachelitellm_versionsuccess_callbacksuse_aiohttp_transportz Not connectedizService Unhealthy (rcr:)rgr7r rhrir rrlitellm.caching.cachingr rVr _index_inforrrr_should_use_aiohttp_transportr ) r7r success_callback_namesr cache_typer rrdb_health_statuss r'health_readinessrsB4U!# >+2*B*B&%& a & "& == $ B ++J'----/ABK'.}}':':'F'F'H!HJ'1 K   $%?%AA #!##*%;)9)W)W)Y # $%##*%;)9)W)W)Y  I& >%,%=%= " >"I K!ACF!JJK B& U6I#a&QR4STTUs FEDD DAE +D37D18D3<EE E1F2E F DD.+E-D..E1D33 E<E EEE F"E<<FFz/health/liveliness)r z/health/livenessc Kyw)z> Unprotected endpoint for checking if worker is alive z I'm alive!rrr'health_livelinessrs  sc0Kdddd}t|dSw)z6 Options endpoint for health/readiness check. GET, OPTIONS*AllowzAccess-Control-Allow-MethodszAccess-Control-Allow-Headersheadersr;r response_headerss r'health_readiness_optionsr)#s) (6(+ ,# >>c0Kdddd}t|dSw)z7 Options endpoint for health/liveliness check. rr r!r#r$r&r's r'health_liveliness_optionsr,4s) (6(+ ,# >>r*z/health/test_connectionchatzThe mode to test the model withzIParameters for litellm.completion, litellm.embedding for the health checkmode) r-rq embedding audio_speechaudio_transcriptionimage_generationbatchrerankrealtimelitellm_paramsc K ti|}|xs|jdd}ttj||ddgt d{}t i||d}d|vrdnd |d S7 #t$rv}tjd t|tjtjttj dd t|i d}~wwxYww)a Test a direct connection to a specific model. This endpoint allows you to verify if your proxy can successfully connect to a specific model. It's useful for troubleshooting model connectivity issues without going through the full proxy routing. Example: ```bash curl -X POST 'http://localhost:4000/health/test_connection' \ -H 'Authorization: Bearer sk-1234' \ -H 'Content-Type: application/json' \ -d '{ "litellm_params": { "model": "gpt-4", "custom_llm_provider": "azure_ai", "litellm_credential_name": null, "api_key": "6xxxxxxx", "api_base": "https://litellm8397336933.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-10-21", }, "mode": "chat" }' ``` Returns: dict: A dictionary containing the health check result with either success information or error details. ) model_infor6r.Nztest from litellm) model_paramsr.promptinputT)rr9rF)rresultzMlitellm.proxy.health_endpoints.test_model_connection(): Exception occurred - zFailed to test connection: r:)rpoprrh ahealth_checkrrrrr9rrrrr rr)r"r.r6r*r<cleaned_resultrs r'test_model_connectionr@Hsh# @) 7~))&$7'  ! !+**+   )  . (~ ( ($  ")F!2g $   (  ""[\_`a\b[c d  ""9#7#7#9:==:3q6(CD   s<C7AA5A3A52C73A55 C4>A1C//C44C7r)Ir~rrrrrrtypingrrrrfastapir r r r r rrhlitellm._loggingrlitellm.constantsr&litellm.llms.custom_httpx.http_handlerrlitellm.proxy._typesrrrrrrr$litellm.proxy.auth.user_api_key_authr"litellm.proxy.db.exception_handlerrlitellm.proxy.health_checkrrrrrouterrtr(Queryrrdictrrrfloatrrrintrrrrrrr rroptionsr)r,postBodyr@rrr'rRsN (11PP1:CCG  +,-  ' '  '  +,-  )00A(B  "BCn %n      n  n bD$ &#ttt t  t c] tsmtT&RIXJg>O6P5QR(/0A(B(7== =,gmm ; l%l C=l sm lSl^ XJgFW>X=YZ(/0A(B(7== 9$17== @$ 61 '-- 2q, %, C=, C= ,  ,  , [, ^ H:WEV=W'  G0 +,-   +,-  B  BJ ! +,-  :U  :Uz       +,- ?  ?   ?   ? +,- &  V)JK'7<< _)00A(B'R R    R R &&'R  R r