Build a Web Scraper With Python and BeautifulSoup

1What Web Scraping Is

Web scraping is the automated extraction of data from web pages. In Python the standard approach is to download the page's HTML with the requests library and then parse it with BeautifulSoup, navigating the document to pull out exactly the values you want.

It is an excellent project because it connects real skills — HTTP requests, HTML structure, CSS selectors, and data storage — to an immediately useful outcome. Just as important, it teaches responsibility: scraping done carelessly can overload servers or break rules, so ethical technique is part of the craft.

2Setting Up the Tools

You need two libraries: requests to fetch pages and beautifulsoup4 to parse them. Install both with pip, ideally inside a virtual environment so your project dependencies stay isolated.

BeautifulSoup can use Python's built-in html.parser, or the faster lxml parser if you install it. For most beginner scrapes the built-in parser is perfectly fine.

• python -m venv venv && source venv/bin/activate
• pip install requests beautifulsoup4
• # optional faster parser:
• pip install lxml

3Fetching a Page

The first step is downloading the HTML. requests.get returns a response object whose .text holds the page source. Always check the status code and set a descriptive User-Agent header so the server knows who is calling.

Wrap the request with a timeout so your scraper does not hang forever on a slow server, and raise for status to catch errors like 404 or 500 early.

• import requests
• headers = {'User-Agent': 'MyScraper/1.0 (learning project)'}
• resp = requests.get(url, headers=headers, timeout=10)
• resp.raise_for_status()
• html = resp.text

4Parsing HTML With BeautifulSoup

Once you have the HTML string, pass it to BeautifulSoup to get a searchable tree. From there, find returns the first matching element and find_all returns a list of all matches. The select method accepts CSS selectors, which many people find the most natural way to target elements.

Before writing selectors, open the page in your browser, right-click the data you want, and choose Inspect. DevTools reveals the exact tags and class names, so your selectors match reality instead of guesswork.

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'html.parser')
titles = soup.select('h2.product-title')
for t in titles: print(t.get_text(strip=True))
link = soup.find('a', class_='next')['href']

5Extracting Structured Data

Real value comes from turning scattered elements into structured rows. Loop over repeating containers — such as each product card or article — and pull multiple fields from within each one into a dictionary.

Collect those dictionaries into a list, then write them to CSV or JSON. This container-per-record pattern keeps related fields together and mirrors how the page is actually laid out.

• rows = []
• for card in soup.select('.product-card'):
• rows.append({
• 'name': card.select_one('.title').get_text(strip=True),
• 'price': card.select_one('.price').get_text(strip=True),
• })

6Scraping Ethically and Legally

Just because data is visible does not mean you may scrape it freely. Check the site's robots.txt file (at /robots.txt) to see which paths are disallowed, and read the terms of service. Many sites offer an official API that is faster and sanctioned — prefer it when available.

Be a good citizen technically too: add delays between requests so you do not hammer the server, identify yourself with a real User-Agent, and cache pages during development so you do not re-fetch the same URL dozens of times while debugging.

7Handling JavaScript-Rendered Pages

Sometimes requests returns almost-empty HTML even though the page looks full in your browser. That happens when the content is rendered by JavaScript after load. Because requests does not run JavaScript, BeautifulSoup sees only the initial shell.

For these pages, use a browser-automation tool like Selenium or Playwright, which drive a real browser, execute the scripts, and then hand you the fully rendered HTML to parse with BeautifulSoup as usual.

• Check whether requests.text actually contains your target data.
• If it does not, the page is likely JavaScript-rendered.
• Switch to Playwright or Selenium to load and render it.
• Extract page_source or content(), then parse with BeautifulSoup.
• Or look for an underlying JSON API the page itself calls.

8Common Mistakes to Avoid

Beginners hit the same handful of problems when starting out with scraping.

• Not checking robots.txt or terms of service before scraping.
• Sending requests too fast and getting blocked or rate-limited.
• Assuming elements always exist — missing tags return None and crash your code.
• Using requests on a JavaScript-heavy page and getting empty results.
• Hardcoding fragile selectors that break the moment the site changes markup.

9Key Takeaways

Scraping combines HTTP, HTML parsing, and good manners.

• requests fetches the HTML; BeautifulSoup parses and selects elements from it.
• Inspect the page in DevTools to find the right tags and classes.
• Use find, find_all, and select with CSS selectors for precise extraction.
• Respect robots.txt, set a User-Agent, and rate-limit to scrape ethically.
• JavaScript-rendered pages need Selenium or Playwright, not plain requests.

10Frequently Asked Questions

Q: Is web scraping legal? A: It depends on the site and the data. Public, non-personal data is generally lower risk, but you must check robots.txt and the terms of service, avoid overloading servers, and steer clear of copyrighted or login-protected content. When in doubt, use an official API.

Q: Why does my scraper return empty results even though the page has content? A: The page is probably rendered by JavaScript after load. requests does not execute JavaScript, so it only sees the initial HTML shell. Use Playwright or Selenium to render the page first.

Q: What is the difference between find and select? A: find and find_all locate elements by tag name and attributes, while select uses CSS selectors like '.class' or 'div > p'. Both are valid; choose whichever reads more clearly for your target.

Q: How do I avoid getting blocked? A: Set a realistic User-Agent, add delays between requests, respect the site's rate limits and robots.txt, and cache responses during development so you do not repeatedly hit the same URL.