h
d dl Z d dlZd dlmZ d dlmZ d dlmZ d dlm Z d dl
m Z de de fd Z
d
e de fd Zd e defd
Zde defdZd Zy) N)urlparse)
BeautifulSoup)async_playwright)
stealth_async) human_delay product_urlreturnc t j d| }d}|r|j d }d| S # t $ r}t d| d}~ww xY w)zE
Convert a product detail URL to the description iframe URL.
z_([0-9]+)\.html zMhttps://www.alibaba.com/product-detail/description/descIframe.html?productId=z Invalid product URL format: N)researchgroup Exception
ValueError)r match
product_ides 7/var/www/Befach/backend/bot/utils/scrape_description.pyconvert_to_description_urlr s_ = ,k:
Q J^_i^jkk =7s;<<=s /2 AA Aurlc K t 4 d{ }|j j ddg dd d{ }|j ddd d
d d{ }|j d{ }t | d{ t
d
| |j | d d{ t dd t
d |j dd d{ |j d{ }|j d{ |cddd d{ S 7 7 7 7 7 7 7 T7 >7 (7 # 1 d{ 7 sw Y yxY ww)zS
Use Playwright and stealth to get HTML content after waiting for .
NTchrome)z --no-sandboxz---disable-blink-features=AutomationControlledz--disable-extensions2 )headlesschannelargsslow_moz[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36i i )widthheightzen-US)
user_agentviewportlocaleu ⏳ Loading domcontentloaded)
wait_untilr u" ⌛ Waiting for to load...tablei0u )timeout) r chromiumlaunch new_contextnew_pager printgotor wait_for_selectorcontentclose)r pbrowsercontextpagehtmls r fetch_html_with_tbody_waitr7 sT ! Q
))
*
++t #s3 ,
%%''D!!!
SE"#ii(:i;;;Aq
23$$We$<<<\\^#mmo =
(! < =# ; s ED'E%D<D* D<D,D<1D.2D<D0(D<-D2.1D< D4 D<7D68D<D8D< E!D:"E*D<,D<.D<0D<2D<4D<6D<8D<:E<EE E
Er6 c . t | d }|j d }|s t d i }|j d D ]T }|j d }t | dk\ s#|d j d }|d
j d }|sP|||< V |S ) zO
Extract table data (class 'all magic-1') into a key-value dictionary.
z html.parserr' z)Table with class 'all magic-1' not found.trtdr& r T)stripr )r findr find_alllenget_text)r6 soupr' datarowcellskeyvalues r parse_table_to_dictrF @ s } -D IIg E DEE
D~~d# T" u:?(##$#/C!H%%D%1E !S
$ K c d K t | }t | d{ }t | }|S 7 w)zp
Convert product detail URL to description URL, load it,
and extract table data as key-value pairs.
N)r r7 rF )r description_urlr6
table_datas r extract_product_description_datarK T s4
1=O+O< .maina sy %9#>>D - .
12aSk" A % ?
% Ks# $ $ %s> A7A Ar^ sl ! 1 4 / =C =C = "# "# "Jc d ( rG