Web Scraping

Chapter 1: Introduction to Web Scraping

1.1 What Is Web Scraping?

Web scraping is the process of automatically collecting information from websites using a program instead of a human manually copying and pasting data.

When you visit a website in your browser, you see formatted text, images, and links. Behind the scenes, however, the website is built using HTML (HyperText Markup Language). Web scraping works by downloading that HTML and extracting specific pieces of information from it.

In simple terms:

A web scraper is a program that reads a webpage and pulls out the parts you care about.

For example, a scraper might collect:

  • Product prices

  • News headlines

  • Quotes

  • Job listings

  • Research data

Web scraping is a form of automation. Instead of repeating the same manual task over and over, you write code once and let the computer do the work.

 

1.2 The Legal and Ethical Side of Web Scraping

Before writing any code, it is important to understand that web scraping must be done responsibly.

1. Check the Website’s Terms of Service

Many websites state whether automated data collection is allowed. Always review their terms.

2. Respect robots.txt

Most websites provide a file called:

https://example.com/robots.txt

This file tells automated programs which parts of the site can or cannot be accessed.

3. Do Not Overload Servers

Sending too many requests too quickly can harm a website’s performance. Responsible scrapers:

  • Add delays between requests

  • Avoid making excessive requests

  • Do not scrape private or login-protected content

4. Use Public Practice Sites When Learning

For this tutorial, we will use:

https://quotes.toscrape.com

This website was specifically created for learning web scraping.

1.3 How Web Scraping Works

A basic web scraper follows three main steps:

Step 1: Send a Request

The scraper asks the website for its HTML content.

Step 2: Receive the HTML

The website responds with the page’s source code.

Step 3: Extract the Data

The scraper searches through the HTML and pulls out specific elements (such as quotes or titles).

That is the entire concept. Everything else is refinement and structure.

 

Chapter 2: Preparing Your Environment

2.1 Install Python

  1. Go to https://python.org

  2. Download Python 3

  3. During installation, check:

    Add Python to PATH

To confirm installation, open Command Prompt and type:

python --version

If you see a version number, Python is ready.

2.2 Install Required Libraries

We need two tools:

  • requests → to download webpages

  • BeautifulSoup → to read and search HTML

In Command Prompt, type:

pip install requests beautifulsoup4

Once installed, you are ready to build your first scraper.

Chapter 3: Building Your First Web Scraper

3.1 Create a New Python File

Create a new file named:

scraper.py

Open it in a text editor or IDE.

3.2 Write the Basic Program

Below is the complete beginner version of a web scraper that:

  • Downloads a webpage

  • Extracts quotes and authors

  • Cleans the text

  • Saves the results into a text file

import requests
from bs4 import BeautifulSoup

# Step 1: Define the website URL
url = “https://quotes.toscrape.com”

# Step 2: Send a request to the website
response = requests.get(url)

# Step 3: Check if request was successful
if response.status_code == 200:

# Step 4: Parse the HTML content
soup = BeautifulSoup(response.text, “html.parser”)

# Step 5: Find all quote containers
quotes = soup.find_all(“div”, class_=”quote”)

# Step 6: Open a file to save results
with open(“quotes.txt”, “w”, encoding=”utf-8″) as file:

# Step 7: Loop through each quote
for quote in quotes:

# Extract the quote text
text = quote.find(“span”, class_=”text”).get_text(strip=True)

# Extract the author
author = quote.find(“small”, class_=”author”).get_text(strip=True)

# Clean and format output
cleaned_output = f”{text} — {author}”

# Save to file
file.write(cleaned_output + “\n”)

print(“Scraping complete. Data saved to quotes.txt”)

else:
print(“Failed to retrieve the webpage.”)

Chapter 4: Understanding the Program

Let us examine what each section does.

Importing Libraries

import requests
from bs4 import BeautifulSoup

These provide the tools needed to download and analyze the webpage.

Sending the Request

response = requests.get(url)

This asks the website to send its HTML.

Checking the Response

if response.status_code == 200:

Status code 200 means the request was successful.

Parsing HTML

soup = BeautifulSoup(response.text, "html.parser")

This converts raw HTML into something Python can search.

Finding Elements

quotes = soup.find_all("div", class_="quote")

This finds every quote container on the page.

Extracting Clean Text

.get_text(strip=True)

This removes extra spaces and formatting.

Writing to a File

with open("quotes.txt", "w", encoding="utf-8") as file:

This creates a clean text file containing only the information you extracted.

Chapter 5: Running the Program

  1. Save scraper.py

  2. Open Command Prompt

  3. Navigate to the file location

  4. Run:

python scraper.py

After it finishes, you will see:

quotes.txt

Open it. You have successfully scraped and cleaned website data.

Chapter 6: What You Have Learned

By completing this tutorial, you have learned:

  • What web scraping is

  • The ethical considerations involved

  • How websites deliver HTML

  • How to request a webpage using Python

  • How to extract specific elements

  • How to clean extracted data

  • How to save results to a file

This is the foundation of automation, data extraction, and large-scale information gathering.

From here, you can expand into:

  • Scraping multiple pages

  • Saving data as CSV

  • Using automation tools for dynamic sites

  • Building dashboards

  • Integrating with databases