o
h
@ s d dl Z d dlZd dlmZ d dlmZ d dlmZ d dlm Z d dl
m Z de de fd d
Z
d e de fd d
Zde defddZde defddZdd ZdS ) N)urlparse)
BeautifulSoup)async_playwright)
stealth_async) human_delay product_urlreturnc
C sR zt d| }d}|r|d}d| W S ty( } ztd| d}~ww )zE
Convert a product detail URL to the description iframe URL.
z_([0-9]+)\.html zMhttps://www.alibaba.com/product-detail/description/descIframe.html?productId=z Invalid product URL format: N)researchgroup Exception
ValueError)r match
product_ide r 7/var/www/Befach/backend/bot/utils/scrape_description.pyconvert_to_description_url s
r urlc s t 4 I dH j}|jjddg dddI dH }|jddd d
d d I dH }| I dH }t|I dH td
| |j| ddI dH tdd td |j dddI dH |
I dH }| I dH |W d I dH S 1 I dH sxw Y dS )zS
Use Playwright and stealth to get HTML content after waiting for .
NTchrome)z --no-sandboxz---disable-blink-features=AutomationControlledz--disable-extensions2 )headlesschannelargsslow_moz[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36i i )widthheightzen-US)
user_agentviewportlocaleu ⏳ Loading domcontentloaded)
wait_untilr
u" ⌛ Waiting for to load...tablei0u )timeout) r chromiumlaunch new_contextnew_pager printgotor wait_for_selectorcontentclose)r pbrowsercontextpagehtmlr r r fetch_html_with_tbody_wait s0
0r5 r4 c C sz t | d}|d}|stdi }|dD ]#}|d}t|dkr:|d jdd }|d
jdd }|r:|||< q|S ) zO
Extract table data (class 'all magic-1') into a key-value dictionary.
z html.parserr% z)Table with class 'all magic-1' not found.trtdr$ r T)stripr
)r findr find_alllenget_text)r4 soupr% datarowcellskeyvaluer r r parse_table_to_dict@ s
rC c s$ t | }t|I dH }t|}|S )zp
Convert product detail URL to description URL, load it,
and extract table data as key-value pairs.
N)r r5 rC )r description_urlr4
table_datar r r extract_product_description_dataT s
rF c s" i fdd}t | S )Nc
s z$t I d H } td | D ]\}}t| d| | |< qW d S ty? } ztd| W Y d }~d S d }~ww )Nu ✅ Extracted Table Data:z: u ❌ Error: )rF r+ itemsr )r> kvr discriptionr r r maina s
z!extract_description..main)asynciorun)r rL r rJ r extract_description^ s
rO )rM r urllib.parser bs4r playwright.async_apir playwright_stealth.stealthr bot.utils.scraper_utilsr strr r5 dictrC rF rO r r r r s %