o
jhl+ @ s d dl Z d dlZd dlZd dlZd dlmZ d dlZd dlmZ ddl m
Z
e edg Z g dZ
ded efd
d Zd ed efd
dZd ed efddZd ed efddZdded ed efddZded efddZdS ) N)
BeautifulSoup)settings ) parse_proxy_url_for_requests
PROXY_LIST)zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36zuMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36zPMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/121.0urlreturnc C s t tddddddddd d
d }d }tr%t t}t|}td
| ztj| ||ddd}| |j W S tj
yR } ztd| W Y d }~d S d }~ww )zK
Fetch product page using requests with proper headers and proxies
zJtext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8zen-US,en;q=0.5zgzip, deflate, brz
keep-alive1documentnavigatenonez?1z max-age=0) z
User-AgentAcceptzAccept-LanguagezAccept-Encoding
ConnectionzUpgrade-Insecure-RequestszSec-Fetch-DestzSec-Fetch-ModezSec-Fetch-SitezSec-Fetch-Userz
Cache-ControlNz[Proxy] Using: T)headersproxiestimeoutallow_redirectszError fetching page: ) randomchoice USER_AGENTSr r printrequestsgetraise_for_statustextRequestException)r r r proxyresponsee r 3/var/www/Befach/backend/bot/utils/scrape_details.pyfetch_product_page s@
r" htmlc
C s zQd}t || t j}|r(|d}t dd|}t dd|}t|}|W S d}t || t j}|rO|d}t dd|}t dd|}t|}|W S W i S tjtt j fyp } zt
d| W Y d }~i S d }~ww )
zX
Extract product and seller data from script tags containing window.detailData.
z$window\.detailData\s*=\s*({.*?})\s*;r z,\s*}}z,\s*]]z detailData\s*:\s*({.*?})\s*,zError parsing detailData: N) researchDOTALLgroupsubjsonloadsJSONDecodeErrorAttributeErrorerrorr )r# detail_data_patternmatch script_data detail_data alt_pattern alt_matchr r r r! extract_detail_data_from_script8 s0
r6 c
C s z5d}t || t j}|si W S |d }t| }|s i W S td| |d ddkr3|d W S W i S tyW } zddl }|
td| W Y d}~i S d}~ww )z<
Extract JSON-LD data from script tags in the HTML.
z2r zdata isz@typeProductNz Error extracting JSON-LD data: ) r& findallr( r+ r, stripr r Exception traceback print_exc)r# json_ld_patternmatchesr1 datar r; r r r! extract_json_ld_dataZ s*
r@ c C s0 d}t d| t jr d}t d| t jrd}|S )zE
Extract product type from HTML content without full parsing
PRE_ORDERz add to cartIN_STOCKz send inquiryINQUIRY)r& r'
IGNORECASE)r# product_typer r r! extract_product_type_from_htmls s rF r3 c
C s i }z| s |W S | di }| di }|r| di }| di }| dg }| di dg }dd |D }d
d |D } i d | d d
| dd| dd| dd| dd| dd| dd| di di dd| di di d i gd d d | d d!g d"| d#d$| d%|d&| d'i d(i d)i d*d+| d+d,| d-d.|rt|nd/i|d< | d
i }
i } |
r |
d0i |d d1< |
d2g } | D ]1}
|
d3}|| vrg | |< |
d4g D ]}| | | d5| d6| d3| d7d8 qq| |d9< | d'i d:i d)i d;g }d<d=d |D }||d>< | d?|d?< | d@i }|rw| dA| dB| dC| dD| dE| dF| dG| dHdI|d@< | dJg }dKdL |D |dM< |rt|}|r| dNi dO|dP< | dQi d3|dR< | dS|dS< |W S ty } ztdT| i W Y dU}~S dU}~ww )Vz_
Parse the extracted detailData into a structured format with product and seller info.
globalDataproductpriceproductRangePrices
mediaItems customPriceproductLadderPricesc S s* g | ]}| d | d| ddqS )minmaxrI )rN rO rI r .0itemr r r!
s
z%parse_detail_data..c S s0 g | ]}| d i dr| d i dqS )imageUrlbigrP rQ r r r! rT s 0 titlesubjectsku productId price_mindollarPriceRangeLow price_maxdollarPriceRangeHigh min_ordermoqunit package_typetrade logisticInfounitSize lead_time leadTimeInfoladderPeriodListr
processPeriodport deliverPlace
payment_termssupply_ability
supplyAbilityimages ladder_pricesample_availablenodeMapmodule_actions privateDataisSampleTradablesample sample_info
sampleInforE UNKNOWN
skuInfoMapvariantsskuAttrsnamevaluesidtype
largeImage)r r r} image
attributesmodule_breadcrumb
breadCrumbz > c S s g | ] }| d r| d qS )r} rP rQ r r r! rT s categoryreviewseller companyNameaccountIsGoldPlusSupplierhasPassAssessmentbaoAccountIsServicecompanyBusinessTypecompanyJoinYearscompanyTotalRevenueTitle
totalStaffNum) company_name
gold_supplierverified_suppliertrade_assurance
business_typeyear_establishedannual_revenue employeesproductBasicPropertiesc S s* i | ]}d |v rd|v r|d |d qS )attrName attrValuer )rR attrr r r!
s * z%parse_detail_data..featuresoffersr
detail_urlbrand
brand_name descriptionzError parsing detail data: N)r rF appendjoinr@ r: r )r3 r# result global_data product_info
price_info range_prices media_itemsrp ro sku_info sku_attrsattrsr attr_namevalue
breadcrumbr seller_infor json_ld_datar r r r! parse_detail_data s
(
"
$
r product_urlc C s t | }|s
ddiS t|}|rt||}|r|S t|}|rF|d|dt|dtr8|di dn|dd|di dS dd iS )
zY
Main function to extract product data from Alibaba product page using requests.
r/ z Failed to fetch product pager} r r )r} r r r )rH r zNo product data found)r" r6 r r@ r
isinstancedict)r r# r3 parsed_datar r r r! extract_product_info s"
*
r )N)r r& r+ r bs4r time django.confr
scraper_utilsr getattrr r strr" r r6 r@ rF r r r r r r! s &"p