From 8cf101dee599dad5c830bbfa39dbf00a38050516 Mon Sep 17 00:00:00 2001 From: gitea Date: Fri, 21 Aug 2026 21:11:06 +0000 Subject: [PATCH] fix(scraper): enable reliable multi-page review dialog extraction with lazy-scroll and exact country parsing --- .gitignore | 1 + AGENTS.md | 21 ++++ src/index.js | 226 +++++++++++++++++++++---------------------- src/review-debug.png | Bin 667757 -> 0 bytes 4 files changed, 134 insertions(+), 114 deletions(-) create mode 100644 AGENTS.md delete mode 100644 src/review-debug.png diff --git a/.gitignore b/.gitignore index 2b7d3d2..7712c76 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,4 @@ src/*.png debug/ .DS_Store + diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..1d284f2 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,21 @@ +# Engineering & Scraping Guidelines + +## 1. Git Commit & Repository Hygiene +- **Conventional Commits:** Write clear, concise commit messages prefixed with standard types (e.g., `feat:`, `fix:`, `refactor:`, `docs:`). +- **Exclude Transient Artifacts:** Never stage or commit generated debug files (e.g., `src/*.html`, `src/*.png`, large logs) to version control. Always maintain `.gitignore` to keep git packs lean and prevent remote unpacker errors on Gitea/GitHub. + +## 2. Web Scraping & DOM Inspection Strategy +- **Rendered DOM Dumping for Debugging:** When debugging extraction failures on dynamic or anti-bot protected pages: + - Capture the post-render HTML via `await page.content()` and save it to `src/hotel-debug.html`. + - Capture a synchronized screenshot (`src/hotel-debug.png`). + - Inspect the raw rendered DOM to verify if the page was served as standard SSR, an A/B test SPA variant, or intercepted by a challenge (e.g. AWS WAF). +- **Multi-Tier Extraction Hierarchy:** Always implement fallbacks in this order: + 1. **Deep JSON-LD Parsing:** Traverse all `