fix(scraper): enable reliable multi-page review dialog extraction with lazy-scroll and exact country parsing

AGENTS.md

@@ -0,0 +1,21 @@
+# Engineering & Scraping Guidelines
+
+## 1. Git Commit & Repository Hygiene
+- **Conventional Commits:** Write clear, concise commit messages prefixed with standard types (e.g., `feat:`, `fix:`, `refactor:`, `docs:`).
+- **Exclude Transient Artifacts:** Never stage or commit generated debug files (e.g., `src/*.html`, `src/*.png`, large logs) to version control. Always maintain `.gitignore` to keep git packs lean and prevent remote unpacker errors on Gitea/GitHub.
+
+## 2. Web Scraping & DOM Inspection Strategy
+- **Rendered DOM Dumping for Debugging:** When debugging extraction failures on dynamic or anti-bot protected pages:
+  - Capture the post-render HTML via `await page.content()` and save it to `src/hotel-debug.html`.
+  - Capture a synchronized screenshot (`src/hotel-debug.png`).
+  - Inspect the raw rendered DOM to verify if the page was served as standard SSR, an A/B test SPA variant, or intercepted by a challenge (e.g. AWS WAF).
+- **Multi-Tier Extraction Hierarchy:** Always implement fallbacks in this order:
+  1. **Deep JSON-LD Parsing:** Traverse all `<script type="application/ld+json">` tags, including `@graph` arrays and varied accommodation schemas (`Hotel`, `Resort`, `Apartment`, `Hostel`, `LodgingBusiness`, etc.).
+  2. **DOM Selectors:** Fallback to standard test IDs, classes, and element hierarchy (`data-testid`, header titles, score components).
+  3. **Meta Tags:** Fallback to OpenGraph and Twitter meta tags (`og:title`, `og:image`, `og:description`).
+  4. **URL Slug Extraction:** As an ultimate safety net, derive clean human-readable names from the URL path if the HTML body was obstructed.
+- **Anti-Bot & WAF Handling:**
+  - Sanitize input URLs by stripping transient tokens (`chal_t`, `force_referer`, session IDs).
+  - Emulate full browser properties (`window.chrome`, `navigator.plugins`, `navigator.languages`).
+  - Actively detect WAF challenge interstitial pages and wait for client-side JavaScript token computation and automatic redirection before evaluating the DOM.
+