Automation & Web Scraping
Automation service built as a technical challenge. Built entirely by Paulo.
Automation service built as a technical challenge: navigate a public portal via Playwright, extract structured data and serve the results via REST API. The challenge is not in the scraping itself — it is in making it reliable in production, where the environment differs from local and timeouts are real.
Public portals have undocumented behaviours: analytics scripts that never finish loading, WAFs that add variable latency, strings that are substrings of one another, clicks intercepted by overlapping elements. Each of these cases manifested differently in production than in the local environment.
The most valuable aspect of this project is not the code that worked — it is the record of the six problems that only appeared in production and how each was isolated and fixed.
FastAPI receives requests and validates input/output via Pydantic. Async Playwright executes the navigation and data extraction — async because the operation is I/O-bound (network/DOM waiting), not CPU-bound. Docker containerises the service with Chromium. Nginx sits in front as a reverse proxy with adjusted proxy_read_timeout. EC2 hosts the service. Swagger UI automatically generated by FastAPI documents all endpoints. Three-layer error handling: input validation (Pydantic), execution with explicit timeout (Playwright), and guaranteed cleanup in finally (closing the browser even if extraction fails).
Async Playwright — async because I/O-bound, not CPU-bound
- Problem
- Web scraping is a waiting operation: waiting for network loading, DOM rendering, JavaScript execution. Using synchronous threads would block FastAPI's event loop for the entire browser wait time — potentially up to 3 minutes per request.
- Alternatives
- Multiprocessing: suitable for CPU-bound operations (compression, heavy parsing). Adds IPC and memory overhead. Synchronous threads: blocks the event loop. For I/O-bound work, async is the right choice — it frees the event loop while the browser waits for the network.
- Choice
- playwright.async_api with async/await. FastAPI keeps the event loop free while Playwright waits. Multiple requests can be processed concurrently while one scraping operation awaits DOM loading.
- Result
- Responsive service even with long operations in progress. The async approach naturally covers long-timeout cases without blocking the server.
Pydantic separating input schema from output schema
- Problem
- Without explicit separation between what the API receives and what it returns, input validation and output formatting get mixed into the same model — or worse, into the scraping code. Hard to test and to document.
- Choice
- Two distinct Pydantic schemas: SearchRequest (validates and normalises input before scraping begins) and SearchResult (defines the exact structure of the response). FastAPI uses these schemas for automatic validation and to generate Swagger UI without manual annotation.
- Result
- Input errors are rejected before opening the browser. The response has schema-guaranteed structure. Swagger UI documents the API automatically.
Three-layer error handling with finally guaranteeing cleanup
- Problem
- If scraping fails mid-execution — timeout, element not found, unexpected exception — the browser stays open consuming memory on the EC2. In production, this accumulates and degrades the service over time.
- Choice
- Layer 1 (Pydantic): validates input before any operation. Layer 2 (try/except in Playwright): captures navigation and timeout errors with a structured error message. Layer 3 (finally): closes the browser and Playwright context regardless of outcome — success or failure. The finally is not optional: it is the cleanup guarantee.
- Result
- No browser process leaks in production. Failures return structured errors. EC2 does not accumulate orphaned processes over time.
networkidle never resolved — analytics scripts blocked the wait
- Symptom
- page.goto() with wait_until='networkidle' hung indefinitely. Playwright waited for 500ms of network silence, but the portal's analytics scripts and trackers kept network requests in a continuous loop. Scraping never advanced.
- Cause
- networkidle waits for all network connections to be closed. Third-party scripts (analytics, beacons) never stop making requests — this state is never reached on portals with tracking.
- Fix
- Replaced networkidle with domcontentloaded as wait_until. A ready DOM is the correct signal for scraping — not network silence. Completely independent of third-party scripts.
30s timeout insufficient on EC2 — WAF added variable latency
- Symptom
- Locally, scraping completed in 8–12 seconds. On EC2, the same flow consistently exceeded 30 seconds and returned a timeout error. Playwright reached the page but the response took much longer than expected.
- Cause
- The portal's WAF inspects requests from datacenter IPs differently from residential IPs. This inspection overhead adds variable latency that does not exist in the local environment.
- Fix
- Playwright timeout adjusted to 90 seconds to cover the actual latency observed on EC2. The value was established empirically — not arbitrarily — after measuring the 95th percentile of response times in production.
Nginx returning 502 before scraping completed
- Symptom
- Even with Playwright configured for 90s, long requests returned 502 Bad Gateway before scraping finished. Playwright completed, but Nginx had already closed the connection.
- Cause
- Nginx's default proxy_read_timeout is 60 seconds. I had already configured 120s in the location block, but the longer flow — with extra navigation steps — still exceeded that limit, and Nginx closed the connection with the client before FastAPI returned the Playwright response.
- Fix
- proxy_read_timeout: 60s default → 120s (still insufficient) → 300s, in the Nginx location block. The Nginx timeout must be greater than the Playwright timeout — not equal to it. Reloading Nginx was enough, no container rebuild.
False positive: '0 results' was a substring of '10,000 results'
- Symptom
- The code checked if the page text contained '0 results' to detect empty searches. Searches with 10,000 results were incorrectly marked as empty — '0 results' exists as a substring inside '10,000 results'.
- Cause
- Unanchored substring check. Any number ending in 0 (10, 100, 1,000, 10,000) contains the sequence '0 result' inside the results string.
- Fix
- Replaced the substring check with an anchored regex: a pattern that verifies '0 results' as a complete expression, not as part of another string. Tested against the edge cases that triggered the bug.
Click blocked — overlapping label intercepted the event
- Symptom
- page.click() on a form element failed with 'element is not clickable at point'. Playwright located the element correctly, but the click was not received by the expected element.
- Cause
- A label tag with absolute CSS positioning was overlaid on the input. The click hit the label, not the input directly — different behaviour from what the automation expected.
- Fix
- Replaced page.click() with page.locator().dispatch_event('click'). dispatch_event delivers the event directly to the element in the DOM, bypassing visual overlap. Alternative: click the label (which is the real user's correct behaviour), but dispatch_event was more explicit about intent.
Unstable text extraction — page.evaluate() with TreeWalker
- Symptom
- Text extraction from table cells via innerText returned inconsistent content depending on how the browser had rendered the DOM. Values from adjacent cells appeared concatenated or truncated.
- Cause
- innerText is sensitive to the rendering layout — what is considered 'visible text' varies with CSS state. Elements with display:none or visibility:hidden can be included or excluded unpredictably.
- Fix
- Replaced with page.evaluate() using TreeWalker to traverse the DOM explicitly: select only text nodes, filter by node type, concatenate with a controlled separator. Deterministic extraction independent of rendering state.
Async scraping service operating in production 24/7 on AWS EC2. Six specific production problems identified, isolated and resolved — each with a documented root cause.
API documented via Swagger UI automatically generated by FastAPI. Three-layer error handling with guaranteed resource cleanup. No browser process leaks during continuous operation.
Datacenter environments behave differently from local environments. WAFs add latency. networkidle never ends on portals with analytics. Timeouts must be measured in production, not estimated locally. The value of a scraper in production lies in the cases that broke — not the ones that worked on the first try.