🕵️ Scraper Service

A robust background scraping service built with Playwright, BullMQ, and Chromium to scrape hotel metadata, real-time prices, Genius discounts, and guest reviews from Booking.com.

🌟 Key Capabilities

Deep Schema & Metadata Extraction:
- Unpacks nested Schema.org JSON-LD scripts (including @graph arrays and accommodation schemas: Hotel, Resort, Apartment, Hostel, LodgingBusiness, etc.).
- Multi-tier DOM and OpenGraph meta tag fallbacks for title, address, hero images, and ratings.
- Automatic URL slug parser (extractHotelNameFromUrl) as an ultimate fallback if client-side rendering is obstructed.
AWS WAF Challenge & Anti-Bot Evasion:
- URL Sanitization (cleanBookingUrl): Automatically cleans tracking tokens and expired challenge parameters (chal_t, force_referer, sid, srpvid, srepoch).
- Browser Fingerprint Masking: Emulates standard desktop Chrome headers, window.chrome runtime objects, navigator.plugins, and German language headers.
- Active WAF Challenge Resolution: Automatically detects AWS WAF challenge pages and waits for client-side JavaScript token generation and redirection.
Overlay & Banner Handling:
- Auto-dismisses OneTrust cookie banners, modal dialogs, and selection popups before evaluating DOM data.
Dual-Mode Scraping:
- Quick Mode (Stammdaten & Prices): Scrapes name, star rating, review count, address, hero image, room size, and prices in ~3–5 seconds and posts directly to Next.js webhook without LLM overhead.
- Full AI Review Mode: Sorts reviews by newest first, pages through reviews, filters for western travelers from the last 2 years, pulls Google Maps bathroom photos via SerpApi, and sends reviews to n8n for LLM summarization.
Debugging & Inspection:
- Automatically captures post-render rendered DOM (src/hotel-debug.html) and screenshots (src/hotel-debug.png) when DEBUG_HTML=true or when requested via ?debug=true.

⚙️ Configuration (`.env` / Docker Environment)

Variable	Description	Default
`REDIS_HOST`	Redis host or container name	`redis`
`REDIS_PORT`	Redis port	`6379`
`DEBUG_HTML`	Enable saving `hotel-debug.html` and `hotel-debug.png`	`true`
`SERPAPI_KEY`	(Optional) SerpApi key for Google Maps bathroom photos	`null`

🚀 Running with Docker Compose

docker compose up -d --build

Viewing Logs

docker logs -f scraper-app

Inspecting Scraped DOM Debug Files

When debugging hotel pages, the rendered DOM and screenshot are saved directly to:

src/hotel-debug.html
src/hotel-debug.png
src/review-debug.png

2.7 KiB Raw Blame History Unescape Escape