o h$@sddlmZmZddlmZmZmZmZddlm Z ddl m Z ddl m Z mZddlZddlmZed Zd ed ed efd dZGdddeZGdddeZdS))ABCabstractmethod)OptionalDictAnyTuple)MarkdownGenerationResult)CustomHTML2Text)RelevantContentFilterBM25ContentFilterN)urljoinz+!?\[([^\]]+)\]\(([^)]+?)(?:\s+"([^"]*)")?\)baseurlreturncCsD|dr|S|dr|dr|dd|S||St||S)z"Fast URL joining for common cases.)http://https://mailto:z///N) startswithendswithr )rrrb/var/www/Befach/backend/venv/lib/python3.10/site-packages/crawl4ai/markdown_generation_strategy.py fast_urljoin s    rc@steZdZdZddeedeeeeffddZ e    dd ed ed eeeefdeed e d e f ddZ dS)MarkdownGenerationStrategyz7Abstract base class for markdown generation strategies.Ncontent_filteroptionscCs||_|pi|_dSN)rrselfrrrrr__init__sz#MarkdownGenerationStrategy.__init__T cleaned_htmlbase_urlhtml2text_options citationsrcKsdS)z$Generate markdown from cleaned HTML.Nr)r r#r$r%rr&kwargsrrrgenerate_markdowns z,MarkdownGenerationStrategy.generate_markdownNN)r"NNT)__name__ __module__ __qualname____doc__rr rstrrr!rboolr r(rrrrrs*$rcseZdZdZddeedeeeefffdd Z dded ed e eeffd d Z     dded edeeeefdeeeefdeede d e fddZZS)DefaultMarkdownGeneratoraw Default implementation of markdown generation strategy. How it works: 1. Generate raw markdown from cleaned HTML. 2. Convert links to citations. 3. Generate fit markdown if content filter is provided. 4. Return MarkdownGenerationResult. Args: content_filter (Optional[RelevantContentFilter]): Content filter for generating fit markdown. options (Optional[Dict[str, Any]]): Additional options for markdown generation. Defaults to None. Returns: MarkdownGenerationResult: Result containing raw markdown, fit markdown, fit HTML, and references markdown. Nrrcst||dSr)superr!r __class__rrr!9sz!DefaultMarkdownGenerator.__init__r"markdownr$rcCsji}i}g}d}d}t|D]{}|||||\} } } |r9| ds9| |vr5t|| || <|| } | |vrdg} | rF| | | rQ| | krQ| | || r[dd| ndf|| <|d7}|| d} ||dds|| d| d nd | d| d | }q|||d d|}d g}| ddt | dddD|d|fS)a" Convert links in markdown to citations. How it works: 1. Find all links in the markdown. 2. Convert links to citations. 3. Return converted markdown and references markdown. Note: This function uses a regex pattern to find links in markdown. Args: markdown (str): Markdown text. base_url (str): Base URL for URL joins. Returns: Tuple[str, str]: Converted markdown and references markdown. rr)rrrz: z - r"!⟨u⟩z![u⟩]Nz ## References css.|]\}\}}d|d||dVqdS)r6u⟩  Nr).0rnumdescrrr os   zFDefaultMarkdownGenerator.convert_links_to_citations..cSs |ddS)Nrrr)xrrrqs zEDefaultMarkdownGenerator.convert_links_to_citations..)key) LINK_PATTERNfinditerappendstartgroupsrrjoingroupendextendsorteditems)r r4r$link_map url_cachepartslast_endcountermatchtextrtitler:r9converted_text referencesrrrconvert_links_to_citations<s8 8   z3DefaultMarkdownGenerator.convert_links_to_citationsTr#r%r&c Kszt|d}ddddddddd} |r| |n|r!| |n |jr*| |j|jdi| |s7d}n t|ts@t|}z||} Wnty`} z dt| } WYd} ~ nd} ~ ww| d d } | } d} |rz | | |\} } Wnty} z| } d t| } WYd} ~ nd} ~ wwd}d}|s|j rz|p|j }| |}d d d|D}||}Wnty} zdt| }d}WYd} ~ nd} ~ wwt | pd| pd| pd|pd|pddWSty } zdt| }t ||ddddWYd} ~ Sd} ~ ww)a Generate markdown with citations from cleaned HTML. How it works: 1. Generate raw markdown from cleaned HTML. 2. Convert links to citations. 3. Generate fit markdown if content filter is provided. 4. Return MarkdownGenerationResult. Args: cleaned_html (str): Cleaned HTML content. base_url (str): Base URL for URL joins. html2text_options (Optional[Dict[str, Any]]): HTML2Text options. options (Optional[Dict[str, Any]]): Additional options for markdown generation. content_filter (Optional[RelevantContentFilter]): Content filter for generating fit markdown. citations (bool): Whether to generate citations. Returns: MarkdownGenerationResult: Result containing raw markdown, fit markdown, fit HTML, and references markdown. )baseurlrFT) body_widthignore_emphasis ignore_links ignore_images protect_linkssingle_line_break mark_code escape_snobr"z#Error converting HTML to markdown: Nz ```z```zError generating citations: r7css|]}d|VqdS)z
{}
N)format)r8srrrr;sz=DefaultMarkdownGenerator.generate_markdown..zError generating fit markdown: ) raw_markdownmarkdown_with_citationsreferences_markdown fit_markdownfit_htmlzError in markdown generation: r)r updater update_params isinstancer.handle ExceptionreplacerTrfilter_contentrDr )r r#r$r%rrr&r'hdefault_optionsr`erarbrc filtered_html error_msgrrrr(vs          z*DefaultMarkdownGenerator.generate_markdownr))r")r"NNNT)r*r+r,r-rr rr.rr!rrTr/r r( __classcell__rrr2rr0(s0( <r0)abcrrtypingrrrrmodelsr html2textr content_filter_strategyr r re urllib.parser compiler?r.rrr0rrrrs