🕵️ Scraper Service

A robust background scraping service built with Playwright, BullMQ, and Chromium to scrape hotel metadata, real-time prices, Genius discounts, and guest reviews from Booking.com.

---

🌟 Key Capabilities

• Deep Schema & Metadata Extraction:
  • Unpacks nested Schema.org JSON-LD scripts (including @graph arrays and accommodation schemas: Hotel, Resort, Apartment, Hostel, LodgingBusiness, etc.).
  • Multi-tier DOM and OpenGraph meta tag fallbacks for title, address, hero images, and ratings.
  • Automatic URL slug parser (extractHotelNameFromUrl) as an ultimate fallback if client-side rendering is obstructed.
• AWS WAF Challenge & Anti-Bot Evasion:
  • URL Sanitization (cleanBookingUrl): Automatically cleans tracking tokens and expired challenge parameters (chal_t, force_referer, sid, srpvid, srepoch).
  • Browser Fingerprint Masking: Emulates standard desktop Chrome headers, window.chrome runtime objects, navigator.plugins, and German language headers.
  • Active WAF Challenge Resolution: Automatically detects AWS WAF challenge pages and waits for client-side JavaScript token generation and redirection.
• Overlay & Banner Handling:
  • Auto-dismisses OneTrust cookie banners, modal dialogs, and selection popups before evaluating DOM data.
• Dual-Mode Scraping:
  • Quick Mode (Stammdaten & Prices): Scrapes name, star rating, review count, address, hero image, room size, and prices in ~3–5 seconds and posts directly to Next.js webhook without LLM overhead.
  • Full AI Review Mode: Sorts reviews by newest first, pages through reviews, filters for western travelers from the last 2 years, pulls Google Maps bathroom photos via SerpApi, and sends reviews to n8n for LLM summarization.
• Debugging & Inspection:
  • Automatically captures post-render rendered DOM (src/hotel-debug.html) and screenshots (src/hotel-debug.png) when DEBUG_HTML=true or when requested via ?debug=true.

---

⚙️ Configuration (.env / Docker Environment)

Variable	Description	Default
REDIS_HOST	Redis host or container name	redis
REDIS_PORT	Redis port	6379
DEBUG_HTML	Enable saving hotel-debug.html and hotel-debug.png	true
SERPAPI_KEY	(Optional) SerpApi key for Google Maps bathroom photos	null

---

🚀 Running with Docker Compose

docker compose up -d --build

Viewing Logs

docker logs -f scraper-app

Inspecting Scraped DOM Debug Files

When debugging hotel pages, the rendered DOM and screenshot are saved directly to:

• src/hotel-debug.html
• src/hotel-debug.png
• src/review-debug.png