Engineering & Scraping Guidelines

1. Git Commit & Repository Hygiene

Conventional Commits: Write clear, concise commit messages prefixed with standard types (e.g., feat:, fix:, refactor:, docs:).
Exclude Transient Artifacts: Never stage or commit generated debug files (e.g., src/*.html, src/*.png, large logs) to version control. Always maintain .gitignore to keep git packs lean and prevent remote unpacker errors on Gitea/GitHub.

Rendered DOM Dumping for Debugging: When debugging extraction failures on dynamic or anti-bot protected pages:
- Capture the post-render HTML via await page.content() and save it to src/hotel-debug.html.
- Capture a synchronized screenshot (src/hotel-debug.png).
- Inspect the raw rendered DOM to verify if the page was served as standard SSR, an A/B test SPA variant, or intercepted by a challenge (e.g. AWS WAF).
Multi-Tier Extraction Hierarchy: Always implement fallbacks in this order:
1. Deep JSON-LD Parsing: Traverse all <script type="application/ld+json"> tags, including @graph arrays and varied accommodation schemas (Hotel, Resort, Apartment, Hostel, LodgingBusiness, etc.).
2. DOM Selectors: Fallback to standard test IDs, classes, and element hierarchy (data-testid, header titles, score components).
3. Meta Tags: Fallback to OpenGraph and Twitter meta tags (og:title, og:image, og:description).
4. URL Slug Extraction: As an ultimate safety net, derive clean human-readable names from the URL path if the HTML body was obstructed.
Anti-Bot & WAF Handling:
- Sanitize input URLs by stripping transient tokens (chal_t, force_referer, session IDs).
- Emulate full browser properties (window.chrome, navigator.plugins, navigator.languages).
- Actively detect WAF challenge interstitial pages and wait for client-side JavaScript token computation and automatic redirection before evaluating the DOM.