o h1)@sddlZddlZdejd<ddlmZddlmZmZddlm Z m Z m Z m Z m Z ddlTddlTddlTddlTdd lmZdd lmZdd lmZddlTddlZddlZejd d dGdddZdS)NfalseTOKENIZERS_PARALLELISM)Path)UrlModel CrawlResult)init_dbget_cached_url cache_urlDB_PATHflush_db)*)List)ThreadPoolExecutor)WebScrapingStrategyignorezBField "model_name" has conflict with protected namespace "model_".)messagec@s8eZdZd%dededefddZdd Zedd edddde f d e d e d e dede dedede de defddZedd edddde f dee d e d e dedede dede de deefddZede ddddd fde de de dede dede defddZde de d e d!ede de de deded"edefd#d$ZdS)& WebCrawlerNFcrawler_strategyalways_by_pass_cacheverbosecCsh|pt|d|_||_tjtdtd|_ tj |j ddtj |j dddt d|_ dS)N)rCRAWL4_AI_BASE_DIRECTORYz .crawl4aiT)exist_okz/cacheF) LocalSeleniumCrawlerStrategyrrospathjoingetenvrhomecrawl4ai_foldermakedirsrready)selfrrrr#Q/var/www/Befach/backend/venv/lib/python3.10/site-packages/crawl4ai/web_crawler.py__init__s zWebCrawler.__init__cCs0td|jddtdddd|_tddS)Nu([LOG] 🌤️ Warming up the WebCrawlerzhttps://google.com/F)urlword_count_thresholdextraction_strategy bypass_cacherTu'[LOG] 🌞 WebCrawler is ready to crawl)printrunNoExtractionStrategyr!r"r#r#r$warmups zWebCrawler.warmupT url_modelprovider api_tokenextract_blocks_flag css_selector screenshotuse_cached_htmlr)chunking_strategyreturnc Ks*|j|j|| p t| f|j||d| S)N)r*r4r5)r,r'r-forced) r"r0r1r2r3r(r4r5r6r)r7kwargsr#r#r$ fetch_page*szWebCrawler.fetch_page url_modelsc  s| pt} fdd} tP} t| j| ||gt||gt||gt||gt||gt||gt||gt|| gt|| gt|g | gt|R}Wd|S1sawY|S)Ncsj|g|Ri|S)N)r;)r0argsr:r.r#r$fetch_page_wrapperSsz2WebCrawler.fetch_pages..fetch_page_wrapper)r-rlistmaplen)r"r<r1r2r3r(r6r4r5r)r7r:r>executorresultsr#r.r$ fetch_pagesDs0             zWebCrawler.fetch_pagesr'r* user_agentc  Ksz|pt}| |_t|tstdt|tstdt|t}d} d} d} |s/|js/t |} | ddr;|j s;WdS| rSt | d}t | d} |rS| d} | sSd} | rW|s|r_|j |t}t |j j|fi| }t}| rtd|d t|d ||d d |r|j } |j||| ||||| | t| f i| }t||_|WSty}z$t|d st||_td|d|jt|dd|jdWYd}~Sd}~ww)NzUnsupported extraction strategyzUnsupported chunking strategyr/Tr u[LOG] 🚀 Crawling done for z , success: , time taken: .2f secondsmsgu[ERROR] 🚫 Failed to crawl z , error: F)r'htmlsuccess error_message)r-r isinstanceExtractionStrategy ValueErrorChunkingStrategymaxMIN_WORD_THRESHOLDrr getr!sanitize_input_encoderupdate_user_agenttimecrawlr+booltake_screenshot process_htmlrN ExceptionhasattrstrrKr)r"r'r(r)r7r*r4r5rErr:cachedscreenshot_dataextracted_contentrMt1t2 crawl_resulter#r#r$r,jsT        & (   zWebCrawler.runrMrcr( is_cachedc  Kst} zFt} t}dd| D}|j||f||| dd| dtd|}| r>td|dt| d d |durItd |Wnty\}ztt |d}~wwt |d d }t |dd }|dg}|dg}|di}|dur| rtd|d|j | |}| ||}tj|dt dd}| rtd|dt| d d|sdn|}| st|||||dt|t|t||d t||t|||||||dd d S)NcSsi|] \}}|dvr||qS)) only_text$image_description_min_word_thresholdr#).0kvr#r#r$ sz+WebCrawler.process_html..riFrj)r(r4rirju![LOG] 🚀 Content extracted for z, success: True, time taken: rIrJz,Failed to extract content from the website: cleaned_htmlrLmarkdownmedialinksmetadatau*[LOG] 🔥 Extracting semantic blocks for z , Strategy: rF)indentdefault ensure_asciiu[LOG] 🚀 Extraction done for rHz seconds.T)r5) r'rMrorprqrrrsr5rcrNrO)rYritemsscraprV$IMAGE_DESCRIPTION_MIN_WORD_THRESHOLDr+rRInvalidCSSSelectorErrorr`rWnamechunkr,jsondumpsr r format_html)r"r'rMrcr(r)r7r4r5rrhr:trdscrapping_strategy extra_paramsresultrgrorprqrrrssectionsr#r#r$r]s            zWebCrawler.process_html)NFF)__name__ __module__ __qualname__CrawlerStrategyr[r%r/DEFAULT_PROVIDERrU RegexChunkingrr`rQrSrr;rrDr,intr]r#r#r#r$rs         )  =    r)rrYenvironpathlibrmodelsrrdatabaserr r r r utilsr7r)rtypingrconcurrent.futuresrcontent_scraping_strategyrconfigwarningsr}filterwarningsrr#r#r#r$s"