Python Project
Price Tracker Web Scraper
A price tracker teaches HTTP, HTML parsing, scheduling and persistence in one small project, and it produces something you will actually run. The interesting part is not the scraping — it is handling the page changing shape, the request failing, and doing all of it without hammering someone else's server.
The brief
Track the price of a product page over time. Store every observation, detect a meaningful drop, and notify yourself. Respect robots.txt and rate-limit yourself — a scraper that behaves badly is a liability, not a portfolio piece.
What it demonstrates
That you can work with someone else's unreliable output, schedule recurring work, and think about the ethics of automation.
What "done" looks like
Build all of these and the project is finished. Anything past that is in the stretch goals.
- Fetch a product page and extract the current price
- Store every observation with a timestamp
- Detect a drop past a threshold you set
- Notify — email, a desktop notification, or a webhook
- Run on a schedule without supervision
- Fail loudly and clearly when the page structure changes
How to build it
- 1
Check what you are allowed to do
Read robots.txt and the terms. Record in your README that you checked — it is part of the work, not a formality.
- 2
Fetch the page politely
Set a real User-Agent, add a timeout, and rate-limit yourself. Retry with backoff rather than hammering on failure.
- 3
Extract the price
Parse with BeautifulSoup. Check for a JSON-LD block first — structured data is far more stable than a CSS selector.
- 4
Normalise it
Strip currency symbols and thousands separators, and store as an integer number of cents. Floats and money do not mix.
- 5
Store the history
SQLite with a row per observation. History is what makes this a tracker rather than a checker.
- 6
Detect and notify
Compare against a rolling minimum, not just the previous value, so a brief spike does not trigger a false alert.
- 7
Schedule it
A cron job or a scheduled workflow. Log every run so a silent failure is visible the next time you look.
Once it works
Only after the definition of done is met. Half-finished ambition reads worse than a small finished thing.
- Track several products and render a price-history chart
- Detect selector breakage automatically and alert on it
- Add a small web UI to manage the watch list
Frequently Asked Questions
Is web scraping legal?
It depends on the site, the jurisdiction and what you do with the data. Read robots.txt and the terms of service, scrape only public pages, rate-limit yourself, and never republish scraped content. For a portfolio project, pick a site that permits it and say in your README that you checked.
What happens when the site changes its HTML?
Your selectors break — and handling that is the actual engineering. Fail loudly with a clear message rather than silently recording a null price, and keep selectors in one place so a fix is a one-line change. Every real scraper has this problem.
Should I use Selenium?
Only if the price is rendered by JavaScript, and check first — many sites include it in the initial HTML or in a JSON blob you can parse directly. Selenium is much slower and much heavier, so it is a fallback rather than a starting point.