o h8@svddlZddlZddlZddlZddlmZddlmZmZm Z m Z ddl m Z ddl mZmZmZddlZddlZddlmZddlmZmZdd lmZdd lTdd lTdd lTdd lmZmZm Z dd l!m"Z"m#Z#m$Z$dd l%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.ddl/m0Z0m1Z1m2Z2ddl3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:ddl;Z;ddldS)N)Enum)initForeBackStyle)Path)OptionalListUnion)asynccontextmanager) CrawlResultMarkdownGenerationResult)async_db_manager)*)AsyncCrawlerStrategyAsyncPlaywrightCrawlerStrategyAsyncCrawlResponse) CacheMode CacheContext_legacy_to_cache_mode)DefaultMarkdownGeneratorMarkdownGenerationStrategy)WebScrapingStrategy) AsyncLogger) BrowserConfigCrawlerRunConfig)MIN_WORD_THRESHOLD$IMAGE_DESCRIPTION_MIN_WORD_THRESHOLDURL_LOG_SHORTEN_LENGTH)sanitize_input_encodeInvalidCSSSelectorError format_htmlfast_format_htmlcreate_box_message)urlparse) __version__c@seZdZdZiZddddeede dfde e de e de de e d ed e f d d Zd dZddZddZddZddZeddZdededddddddddddfdede edededede ede d e d!e d"e d#ed$e d%e d&ed'efd(d)Zded*ed+eded$ed,ed-e d'efd.d/Zdededdddddddf d0e ede edededede ede d#ed$e d%e d&ed'e efd1d2Z!d3d4Z"d5d6Z#d7d8Z$dS)9AsyncWebCrawleraB Asynchronous web crawler with flexible caching capabilities. There are two ways to use the crawler: 1. Using context manager (recommended for simple cases): ```python async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://example.com") ``` 2. Using explicit lifecycle management (recommended for long-running applications): ```python crawler = AsyncWebCrawler() await crawler.start() # Use the crawler multiple times result1 = await crawler.arun(url="https://example.com") result2 = await crawler.arun(url="https://another.com") await crawler.close() ``` Migration Guide: Old way (deprecated): crawler = AsyncWebCrawler(always_by_pass_cache=True, browser_type="chromium", headless=True) New way (recommended): browser_config = BrowserConfig(browser_type="chromium", headless=True) crawler = AsyncWebCrawler(config=browser_config) Attributes: browser_config (BrowserConfig): Configuration object for browser settings. crawler_strategy (AsyncCrawlerStrategy): Strategy for crawling web pages. logger (AsyncLogger): Logger instance for recording events and errors. always_bypass_cache (bool): Whether to always bypass cache. crawl4ai_folder (str): Directory for storing cache. base_directory (str): Base directory for storing cache. ready (bool): Whether the crawler is ready for use. Methods: start(): Start the crawler explicitly without using context manager. close(): Close the crawler explicitly without using context manager. arun(): Run the crawler for a single source: URL (web, local file, or raw HTML). awarmup(): Perform warmup sequence. arun_many(): Run the crawler for multiple sources. aprocess_html(): Process HTML content. Typical Usage: async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://example.com") print(result.markdown) Using configuration: browser_config = BrowserConfig(browser_type="chromium", headless=True) async with AsyncWebCrawler(config=browser_config) as crawler: crawler_config = CrawlerRunConfig( cache_mode=CacheMode.BYPASS ) result = await crawler.arun(url="https://example.com", config=crawler_config) print(result.markdown) NFCRAWL4_AI_BASE_DIRECTORYcrawler_strategyconfigalways_bypass_cachealways_by_pass_cachebase_directory thread_safec  s(|}|durtfdddDr|jjdddnt}||_ttj |dd |jj d d |_d d D} |pGt d||jd| |_ |j jsR|j|j _|durhddrdtjdtdd||_n||_|rqtnd|_tj |d|_tj|jddtj|jdddd|_dS)ax Initialize the AsyncWebCrawler. Args: crawler_strategy: Strategy for crawling web pages. If None, will create AsyncPlaywrightCrawlerStrategy config: Configuration object for browser settings. If None, will be created from kwargs always_bypass_cache: Whether to always bypass cache (new parameter) always_by_pass_cache: Deprecated, use always_bypass_cache instead base_directory: Base directory for storing cache thread_safe: Whether to use thread-safe operations **kwargs: Additional arguments for backwards compatibility Nc3s|]}|vVqdSN).0kkwargsr0V/var/www/Befach/backend/venv/lib/python3.10/site-packages/crawl4ai/async_webcrawler.py z+AsyncWebCrawler.__init__..) browser_typeheadlessviewport_widthviewport_heightz`Both browser_config and legacy browser parameters provided. browser_config will take precedence.WARNINGmessagetagz .crawl4aiz crawler.log )log_fileverbose tag_widthcSsi|] \}}|dvr||qS))browser_congigloggerr0r1r2vr0r0r5 sz,AsyncWebCrawler.__init__..)browser_configrEwarningTz'always_by_pass_cache' is deprecated and will be removed in version 0.5.0. Use 'always_bypass_cache' instead. Pass warning=False to suppress this warning. stacklevel)exist_okz/cacheFr0)anyrErJr from_kwargsrIrospathjoinrBitemsrr)getwarningswarnDeprecationWarningr+asyncioLock_lockcrawl4ai_foldermakedirsready) selfr)r*r+r,r-r.r4rIparamsr0r3r5__init__ksP    zAsyncWebCrawler.__init__cs$|jIdH|IdH|S)a Start the crawler explicitly without using context manager. This is equivalent to using 'async with' but gives more control over the lifecycle. This method will: 1. Initialize the browser and context 2. Perform warmup sequence 3. Return the crawler instance for method chaining Returns: AsyncWebCrawler: The initialized crawler instance N)r) __aenter__awarmupr_r0r0r5starts zAsyncWebCrawler.startcs|jdddIdHdS)a Close the crawler explicitly without using context manager. This should be called when you're done with the crawler if you used start(). This method will: 1. Clean up browser resources 2. Close any open pages and contexts N)r) __aexit__rdr0r0r5closes zAsyncWebCrawler.closecs|IdHSr/)rerdr0r0r5rbszAsyncWebCrawler.__aenter__cs|IdHdSr/)rg)r_exc_typeexc_valexc_tbr0r0r5rfszAsyncWebCrawler.__aexit__cs"|jjdtddd|_dS)z Initialize the crawler with warm-up sequence. This method: 1. Logs initialization info 2. Sets up browser configuration 3. Marks the crawler as ready z Crawl4AI INIT)r?TN)rEinfocrawl4ai_versionr^rdr0r0r5rcs zAsyncWebCrawler.awarmupcCs dVdS)u异步空上下文管理器Nr0rdr0r0r5 nullcontexts zAsyncWebCrawler.nullcontextTurlextraction_strategychunking_strategycontent_filter cache_mode bypass_cache disable_cache no_cache_readno_cache_write css_selector screenshotpdf user_agentreturnc"s"|}t|tr |std|jp|4IdHz|dur$|}n||||||| | | | | ||d |}t|}t|| | | gr_|ddrQt j dt dd|j dur_t | || | d |_ |j durhtj|_ t||j |j}d}d}d}d}d}t}|rt|IdH}|rt|j}t|jpd }|r|d krdn|}|j}|j}|jr| r|jr|sd}|jj|jt|t|d d |r|s{t}|r|j !||j j"||dIdH}t|j}|j}|j#}t}|jj|jt|||d d |j$d(|||||||j%|&drdndd|IdH}|j'|_'|j(|_(|j)|_)|j*|_*t||_+t,|dd|_-|jj+dd|j|j+t|ddd|j+rSt.j/nt.j0t.j1dd|2rnt|snt3|IdH|WWdIdHS|jj+dd|jdt|dddt.j/t.j1ddt||_+t,|dd|_-|WWdIdHSt4y}zFt5t67} d| dd| dd| dd t|d!| d" }!|jj8|t9|!d#d$d%d&t:|d d|!d'WYd}~WdIdHSd}~ww1IdHs wYdS))a Runs the crawler for a single source: URL (web, local file, or raw HTML). Migration Guide: Old way (deprecated): result = await crawler.arun( url="https://example.com", word_count_threshold=200, screenshot=True, ... ) New way (recommended): config = CrawlerRunConfig( word_count_threshold=200, screenshot=True, ... ) result = await crawler.arun(url="https://example.com", crawler_config=config) Args: url: The URL to crawl (http://, https://, file://, or raw:) crawler_config: Configuration object controlling crawl behavior [other parameters maintained for backwards compatibility] Returns: CrawlResult: The result of crawling and processing z4Invalid URL, make sure the URL is a non-empty stringN) word_count_thresholdrprqrrrsrtrurvrwrxryrzrBrJTztCache control boolean flags are deprecated and will be removed in version 0.5.0. Use 'cache_mode' parameter instead.rKrL)rurtrvrwz[]FETCH)rosuccesstimingr?)r*zraw:F)rohtmlextracted_contentr*rypdf_datarB is_raw_html session_idz1{url:.50}... | Status: {status} | Total: {timing}COMPLETE.2fs)rostatusr)rrr>r?r`colorsz'Unexpected error in _crawl_web at line line_noz in functionz (filenamez ): Error: z Code context: code_contexterror)typeERROR)rorr?)rorr error_messager0); isinstancestr ValueErrorr[rnrrPrOrUrVrWrXrsrrENABLEDrr+time perf_counter should_readraget_cached_urlr rrryrzrE url_status display_urlboolr)update_user_agentcrawlr aprocess_htmlrB startswith status_coderesponse_headersdownloaded_filesssl_certificatergetattrrrGREENREDYELLOW should_write acache_url Exceptionget_error_contextsysexc_info error_statusr$r )"r_ror*r}rprqrrrsrtrurvrwrxryrzr{rBr4crawler_config config_kwargs cache_contextasync_response cached_resultscreenshot_datarr start_timert1t2 crawl_resulte error_contextrr0r0r5arunsH3             0 B    BzAsyncWebCrawler.arunrrrrBcszD|dds |nd} t} t|jd} dd|Dfdd|D| j||fi} | durDt d |Wn)t yW} zt t | d} ~ wt yn} z t d |d t | d} ~ wwt | d d }t | d d }t | dd }| dg}| dg}| di}|jpt}|j||d}|}t |j}|jjdd| tt| ddd|dur,|jr,|jr,t|jts,t} |jj}|d kr|js|jjddd| idd}|||jd||}|dkrtn|j}||}|j||}tj|dt dd}|jjd d| t| dd|s1dn|}|s8dn|}|j rBt!|}t"|||||||||||||d!d d"S)#a Process HTML content using the provided configuration. Args: url: The URL being processed html: Raw HTML content extracted_content: Previously extracted content (if any) config: Configuration object controlling processing behavior screenshot: Screenshot data (if any) pdf_data: PDF data (if any) verbose: Whether to enable verbose logging **kwargs: Additional parameters for backwards compatibility Returns: CrawlResult: Processed result containing extracted and formatted content rFzRaw HTML)rEcSsi|] \}}|dvr||qS))ror0rFr0r0r5rH$sz1AsyncWebCrawler.aprocess_html..cs"i|] \}}|vr||qSr0)keysrFr`r0r5rH&"Nz:Process HTML, Failed to extract content from the website: z , error: cleaned_htmlr~ fit_markdownfit_htmlmedialinksmetadata)rbase_urlz)Processed {url:.50}... | Time: {timing}msSCRAPEi)rorr>r?r`zGFit markdown requested but not available. Falling back to raw markdown.EXTRACTromarkdown)rrrr)indentdefault ensure_asciiz,Completed for {url:.50}... | Time: {timing}sT)rorr markdown_v2rrrrrrryrzrrr)#rUrrrrEto_dictrTupdatescraprr!rrr markdown_generatorrgenerate_markdown raw_markdownrlintrprqrNoExtractionStrategy input_formatrrJIdentityChunkingchunkrunjsondumps prettiifyr#r )r_rorrr*ryrrBr4_urlrscrapping_strategyresultrrrrrrrrmarkdown_resultrrcontent_formatcontentchunkingsectionsrr0rr5rs             zAsyncWebCrawler.aprocess_htmlurlsc sV}|dur#tdd|||||| | | fDr jjddd|n||||||| | | | d |}t||rP|dd rGtjd td d j durPt j _ j pTd }t |fddjjdddt|idt}fdd|D}t j|dd iIdH}t}jjddt|||ddddtjiddd|DS)a Runs the crawler for multiple URLs concurrently. Migration Guide: Old way (deprecated): results = await crawler.arun_many( urls, word_count_threshold=200, screenshot=True, ... ) New way (recommended): config = CrawlerRunConfig( word_count_threshold=200, screenshot=True, ... ) results = await crawler.arun_many(urls, crawler_config=config) Args: urls: List of URLs to crawl crawler_config: Configuration object controlling crawl behavior for all URLs [other parameters maintained for backwards compatibility] Returns: List[CrawlResult]: Results for each URL Ncss|]}|duVqdSr/r0)r1paramr0r0r5r6r7z,AsyncWebCrawler.arun_many..zXBoth crawler_config and legacy parameters provided. crawler_config will take precedence.r<r=) r}rprqrrrsrtrxryrzrBrJTz'bypass_cache' is deprecated and will be removed in version 0.5.0. Use 'cache_mode=CacheMode.BYPASS' instead. Pass warning=False to suppress this warning.rKrLc st|j}t}jjddd|idj}j}|jvr;|j|}||kr;|t d|}t |IdH|j|<4IdHj |dIdHWdIdHS1IdHsbwYdS)NzStarted task for {url:.50}...PARALLELrorr)rr{) r%netlocrrEdebug mean_delay max_range_domain_last_hitrandomuniformrYsleepr)rodomain current_timerrtime_since_lastdelay)r*r_ semaphorer{r0r5crawl_with_semaphores.    0z7AsyncWebCrawler.arun_many..crawl_with_semaphorez0Starting concurrent crawling for {count} URLs...rkcountrcsg|]}|qSr0r0)r1ro)rr0r5 $sz-AsyncWebCrawler.arun_many..return_exceptionszEConcurrent crawling completed for {count} URLs | Total time: {timing}rrr)rrrrcSs"g|] }t|ts |nt|qSr0)rrr)r1rr0r0r5r5r)rOrErJrrPrUrVrWrXrsrBYPASSsemaphore_countrY Semaphorerllenrrgatherrrr)r_rr*r}rprqrrrsrtrxryrzr{rBr4rrrrtasksresultsend_timer0)r*rr_rr{r5 arun_manysr.           zAsyncWebCrawler.arun_manyctIdHdS)zClear the cache database.N)rcleanuprdr0r0r5 aclear_cache7zAsyncWebCrawler.aclear_cachecr)zFlush the cache database.N)r aflush_dbrdr0r0r5 aflush_cache;r zAsyncWebCrawler.aflush_cachecstIdHS)z%Get the total number of cached items.N)raget_total_countrdr0r0r5aget_cache_size?szAsyncWebCrawler.aget_cache_size)%__name__ __module__ __qualname____doc__rrrQgetenvrhomerrrrrarergrbrfrcr rnr RegexChunkingrExtractionStrategyChunkingStrategyRelevantContentFilterrr rrr rr r rr0r0r0r5r')s? R        %       r')?rQrrrVenumrcoloramarrrrpathlibrtypingrr r rrY contextlibr modelsr rasync_databaserrqcontent_filter_strategyrpasync_crawler_strategyrrrrrrrmarkdown_generation_strategyrrcontent_scraping_strategyr async_loggerr async_configsrrr*rrrutilsr r!r"r#r$ urllib.parser%rr&rmr'r0r0r0r5s6