55 lines
2.7 KiB
Markdown
55 lines
2.7 KiB
Markdown
# 🕵️ Scraper Service
|
||
|
||
A robust background scraping service built with **Playwright**, **BullMQ**, and **Chromium** to scrape hotel metadata, real-time prices, Genius discounts, and guest reviews from Booking.com.
|
||
|
||
---
|
||
|
||
## 🌟 Key Capabilities
|
||
|
||
- **Deep Schema & Metadata Extraction:**
|
||
- Unpacks nested Schema.org JSON-LD scripts (including `@graph` arrays and accommodation schemas: `Hotel`, `Resort`, `Apartment`, `Hostel`, `LodgingBusiness`, etc.).
|
||
- Multi-tier DOM and OpenGraph meta tag fallbacks for title, address, hero images, and ratings.
|
||
- Automatic URL slug parser (`extractHotelNameFromUrl`) as an ultimate fallback if client-side rendering is obstructed.
|
||
- **AWS WAF Challenge & Anti-Bot Evasion:**
|
||
- **URL Sanitization (`cleanBookingUrl`):** Automatically cleans tracking tokens and expired challenge parameters (`chal_t`, `force_referer`, `sid`, `srpvid`, `srepoch`).
|
||
- **Browser Fingerprint Masking:** Emulates standard desktop Chrome headers, `window.chrome` runtime objects, `navigator.plugins`, and German language headers.
|
||
- **Active WAF Challenge Resolution:** Automatically detects AWS WAF challenge pages and waits for client-side JavaScript token generation and redirection.
|
||
- **Overlay & Banner Handling:**
|
||
- Auto-dismisses OneTrust cookie banners, modal dialogs, and selection popups before evaluating DOM data.
|
||
- **Dual-Mode Scraping:**
|
||
- **Quick Mode (Stammdaten & Prices):** Scrapes name, star rating, review count, address, hero image, room size, and prices in ~3–5 seconds and posts directly to Next.js webhook without LLM overhead.
|
||
- **Full AI Review Mode:** Sorts reviews by newest first, pages through reviews, filters for western travelers from the last 2 years, pulls Google Maps bathroom photos via SerpApi, and sends reviews to n8n for LLM summarization.
|
||
- **Debugging & Inspection:**
|
||
- Automatically captures post-render rendered DOM (`src/hotel-debug.html`) and screenshots (`src/hotel-debug.png`) when `DEBUG_HTML=true` or when requested via `?debug=true`.
|
||
|
||
---
|
||
|
||
## ⚙️ Configuration (`.env` / Docker Environment)
|
||
|
||
| Variable | Description | Default |
|
||
|---|---|---|
|
||
| `REDIS_HOST` | Redis host or container name | `redis` |
|
||
| `REDIS_PORT` | Redis port | `6379` |
|
||
| `DEBUG_HTML` | Enable saving `hotel-debug.html` and `hotel-debug.png` | `true` |
|
||
| `SERPAPI_KEY` | (Optional) SerpApi key for Google Maps bathroom photos | `null` |
|
||
|
||
---
|
||
|
||
## 🚀 Running with Docker Compose
|
||
|
||
```bash
|
||
docker compose up -d --build
|
||
```
|
||
|
||
### Viewing Logs
|
||
```bash
|
||
docker logs -f scraper-app
|
||
```
|
||
|
||
### Inspecting Scraped DOM Debug Files
|
||
When debugging hotel pages, the rendered DOM and screenshot are saved directly to:
|
||
- `src/hotel-debug.html`
|
||
- `src/hotel-debug.png`
|
||
- `src/review-debug.png`
|
||
|