A client asked me to constantly scrape hotel prices for pricing decision support, so I spent one weekend building this system.
The current state of my scraper
scraping script is written in playwright, and orchestrated using apscheduler
prices for 4 different hotels are scraped every hour
it dumps raw data into a MongoDB Atlas instance
it runs on my old Windows PC at home (which is now permanently on)
this data is served from my FastAPI backend deployed on Google Cloud Run in a Docker container
This scraping system has been behaving itself, but it wasn’t always like this — I had to waste a bunch of time and take a bunch of detours to get here.
For what it’s worth, I believe that reflecting on dumb mistakes I made can be beneficial to anyone who might be building similar-ish systems
1) Overspending time on sites that are hard to scrape
Some websites are harder to scrape than others — they lazyload data using JavaScript, put random stuff in iframes everywhere, and introduce captchas and other stuff that stops non-humans from easily getting data.
