Context

A small e-commerce operator was paying a SaaS vendor to track competitor pricing. The vendor dropped two of their target sites and support stopped responding. They wanted something they owned.

The problem

Two of the five target sites had moved to client-side rendering, so the vendor simple HTTP fetch returned empty HTML. Price changes were also noisy: a 1% rounding difference was triggering alerts every day, so the team had muted them entirely.

Approach & decisions

  1. Split the job into fetch and parse stages with independent retry and backoff. Most of the failures were transient network issues on one site, not parse bugs — coupling them meant a single timeout threw away the whole crawl.
  2. Used a headless browser only for the two client-rendered sites, plain HTTP for the other three. Running everything through a browser tripled runtime and memory for no benefit.
  3. Normalised all prices to a base currency with a daily exchange rate snapshot before comparing, so a currency swing did not look like a competitor price change.
  4. Added a per-SKU rolling median and only alerted on moves beyond 3% of that median. This was the change that actually made the alerts useful again — the team had turned them off before this.

Result

Runs daily for 1,800 SKUs in about four minutes. Scraping cost dropped from $400/month to roughly $6/month in compute. False-positive alerts went from roughly 30 per day to under 2 per week, and the team turned notifications back on.

The part that took longest

Not the scraping. The parser for the two client-rendered sites broke four times in the first month because the target site kept changing its markup. The fix was not a smarter parser — it was storing the raw HTML of every failed parse and alerting on three consecutive failures, so a break was caught the same morning it happened instead of silently producing empty prices.