o ›Œh‰ã@sÄddlmZmZddlZddlmZddlZddlmZddl TGdd„deƒZ Gd d „d e ƒZ Gd d „d e ƒZ Gd d„de ƒZ Gdd„de ƒZGdd„de ƒZGdd„de ƒZGdd„de ƒZdS)é)ÚABCÚabstractmethodN)ÚCounteré©Úload_nltk_punkt)Ú*c@s&eZdZdZededefdd„ƒZdS)ÚChunkingStrategyz6 Abstract base class for chunking strategies. ÚtextÚreturncCsdS)zº Abstract method to chunk the given text. Args: text (str): The text to chunk. Returns: list: A list of chunks. N©©Úselfr r r úW/var/www/Befach/backend/venv/lib/python3.10/site-packages/crawl4ai/chunking_strategy.pyÚchunks zChunkingStrategy.chunkN)Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚstrÚlistrr r r rr sr c@s"eZdZdZdedefdd„ZdS)ÚIdentityChunkingzJ Chunking strategy that returns the input text as a single chunk. r r cCs|gS©Nr r r r rr szIdentityChunking.chunkN)rrrrrrrr r r rrsrc@s,eZdZdZd dd„Zdedefdd„ZdS) Ú RegexChunkingzR Chunking strategy that splits text based on regular expression patterns. NcKs|durdg}||_dS)zŸ Initialize the RegexChunking object. Args: patterns (list): A list of regular expression patterns to split text. Nz\n\n)Úpatterns)rrÚkwargsr r rÚ__init__(s zRegexChunking.__init__r r cCs:|g}|jD]}g}|D] }| t ||¡¡q |}q|Sr)rÚextendÚreÚsplit)rr Ú paragraphsÚpatternÚnew_paragraphsÚ paragraphr r rr3s zRegexChunking.chunkr©rrrrrrrrr r r rr$s  rc@s*eZdZdZdd„Zdedefdd„ZdS) ÚNlpSentenceChunkingz\ Chunking strategy that splits text into sentences using NLTK's sentence tokenizer. cKs tƒdS)z< Initialize the NlpSentenceChunking object. Nr)rrr r rrAs zNlpSentenceChunking.__init__r r cCs.ddlm}||ƒ}dd„|Dƒ}tt|ƒƒS)Nr)Ú sent_tokenizecSsg|]}| ¡‘qSr )Ústrip)Ú.0Úsentr r rÚ Qóz-NlpSentenceChunking.chunk..)Ú nltk.tokenizer&rÚset)rr r&Ú sentencesÚsensr r rrHs  zNlpSentenceChunking.chunkNr$r r r rr%=sr%c@sPeZdZdZddd„Zdedefdd„Zdedefd d „Zdedefd d „Z d S)ÚTopicSegmentationChunkingzà Chunking strategy that segments text into topics using NLTK's TextTilingTokenizer. How it works: 1. Segment the text into topics using TextTilingTokenizer 2. Extract keywords for each topic segment écKsddl}|j ¡|_||_dS)z³ Initialize the TopicSegmentationChunking object. Args: num_keywords (int): The number of keywords to extract for each topic segment. rN)ÚnltkÚtokenizeÚTextTilingTokenizerÚ tokenizerÚ num_keywords)rr6rÚnlr r rr_s  z"TopicSegmentationChunking.__init__r r cCs|j |¡}|Sr)r5r3)rr Úsegmented_topicsr r rrjs zTopicSegmentationChunking.chunkcsHddl‰ˆj |¡}‡fdd„|Dƒ}t|ƒ}dd„| |j¡Dƒ}|S)Nrcs0g|]}|ˆjj d¡vr|tjvr| ¡‘qS)Úenglish)ÚcorpusÚ stopwordsÚwordsÚstringÚ punctuationÚlower)r(Útoken©r7r rr*ss0z>TopicSegmentationChunking.extract_keywords..cSsg|]\}}|‘qSr r )r(ÚwordÚfreqr r rr*wr+)r2ÚtoknizeÚ word_tokenizerÚ most_commonr6)rr ÚtokensÚ freq_distÚkeywordsr rArÚextract_keywordsos  z*TopicSegmentationChunking.extract_keywordscs ˆ |¡}‡fdd„|Dƒ}|S)Ncsg|] }|ˆ |¡f‘qSr )rJ)r(Úsegment©rr rr*~sz?TopicSegmentationChunking.chunk_with_topics..)r)rr ÚsegmentsÚsegments_with_topicsr rLrÚchunk_with_topicszs z+TopicSegmentationChunking.chunk_with_topicsN)r1) rrrrrrrrrJrOr r r rr0Vs    r0c@s,eZdZdZd dd„Zdedefdd„Zd S) ÚFixedLengthWordChunkingzÉ Chunking strategy that splits text into fixed-length word chunks. How it works: 1. Split the text into words 2. Create chunks of fixed length 3. Return the list of chunks édcKs ||_dS)zµ Initialize the fixed-length word chunking strategy with the given chunk size. Args: chunk_size (int): The size of each chunk in words. N)Ú chunk_size)rrRrr r rr‹s z FixedLengthWordChunking.__init__r r cs*| ¡‰‡‡fdd„tdtˆƒˆjƒDƒS)Ncs$g|]}d ˆ||ˆj…¡‘qS)ú )ÚjoinrR)r(Úi©rr<r rr*–s$z1FixedLengthWordChunking.chunk..r)rÚrangeÚlenrRr r rVrr”s"zFixedLengthWordChunking.chunkN)rQr$r r r rrP‚s  rPc@ó,eZdZdZd dd„Zdedefdd „Zd S) ÚSlidingWindowChunkingzÈ Chunking strategy that splits text into overlapping word chunks. How it works: 1. Split the text into words 2. Create chunks of fixed length 3. Return the list of chunks rQé2cKó||_||_dS)a Initialize the sliding window chunking strategy with the given window size and step size. Args: window_size (int): The size of the sliding window in words. step (int): The step size for sliding the window in words. N)Ú window_sizeÚstep)rr]r^rr r rr¢ó zSlidingWindowChunking.__init__r r cCs–| ¡}g}t|ƒ|jkr|gStdt|ƒ|jd|jƒD]}d ||||j…¡}| |¡q||jt|ƒkrI| d ||j d…¡¡|S)NrrrS)rrXr]rWr^rTÚappend)rr r<ÚchunksrUrr r rr®s  zSlidingWindowChunking.chunkN)rQr[r$r r r rrZ™s  rZc@rY) ÚOverlappingWindowChunkinga Chunking strategy that splits text into overlapping word chunks. How it works: 1. Split the text into words using whitespace 2. Create chunks of fixed length equal to the window size 3. Slide the window by the overlap size 4. Return the list of chunks éèrQcKr\)a) Initialize the overlapping window chunking strategy with the given window size and overlap size. Args: window_size (int): The size of the window in words. overlap (int): The size of the overlap between consecutive chunks in words. N)r]Úoverlap)rr]rdrr r rrÉr_z"OverlappingWindowChunking.__init__r r cCs‚| ¡}g}t|ƒ|jkr|gSd}|t|ƒkr?||j}d |||…¡}| |¡|t|ƒkr4 |S||j}|t|ƒks|S)NrrS)rrXr]rTr`rd)rr r<raÚstartÚendrr r rrÕs     þ ø zOverlappingWindowChunking.chunkN)rcrQr$r r r rrb¿s  rb)ÚabcrrrÚ collectionsrr=Ú model_loaderrÚutilsr rrr%r0rPrZrbr r r rÚs  ,&