Menu Data Pipeline

Pulls data from 17 restaurant websites, cleans it, and updates a live Google Sheet.

Code on GitHub ↗

Status: Finished
Built with: Python, httpx (HTTP/2), Pydantic, pandas, Google Sheets API, Tenacity

What this demonstrates for your project

Discuss a data pipeline →Discuss data pipelines →

Problem: Keeping a comparison spreadsheet current across 17 restaurant websites required repeated manual checks and copy-paste.
What was built: A concurrent Python pipeline using httpx (HTTP/2) across 17 sources, Pydantic schema validation, pandas deduplication, and direct sync to Google Sheets API.
What it proves: 17 concurrent extraction sources with Pydantic validation at the ingestion boundary; exponential backoff on HTTP 429/500 errors.
Status & context: Completed project. Public repository on GitHub.

Overview

Problem

Keeping a menu comparison spreadsheet current across 17 different restaurant websites by hand means revisiting each site whenever menus change.

This pipeline collects menu items from 17 restaurant chains in Islamabad, validates and normalizes every field, and writes the unified dataset directly to a live Google Sheet, turning a manual, error-prone task into a single command.

What I built

Key decision: Every scraped item is passed through a strict Pydantic schema before it's allowed anywhere near the output: prices must parse as numbers, categories get normalized, and malformed records are caught at the boundary instead of corrupting the sheet.

A thread pool runs all 17 sources concurrently over HTTP/2 (via httpx), cutting a 15+ minute sequential run down to a few minutes. Requests that get throttled or fail (HTTP 429/500), including a 250+ page catalog from one chain, retry automatically with exponential backoff via Tenacity. Collection stays within each source's published access rules and runs at a conservative, rate-aware pace. Once collected, pandas deduplicates and standardizes messy categories (merging near-duplicates like "Add-ons" and "Addons") before the final push to Sheets.

The 17 sources come in three shapes, and each gets its own scraper: 14 chains through Foodpanda's vendor API, Broadway Pizza and Cheezious through their own menu APIs, and Tehzeeb Bakers through its Magento store. When Foodpanda's bot protection blocks a session, that chain stops with a named error instead of retrying into the block. The master sheet is then rebuilt from every chain's last saved file, so a blocked run reports what failed without wiping the rows collected before.

menus_master.csv

chain	branch	category	item_name	price_pkr
Savour Foods	Blue Area	Deals & Combos	Hot Deal	2605
Chaaye Khana	F-6	Breakfast	Sunny Side Up Eggs (2 Eggs)	1049
Howdy	F-7	Deals & Combos	Favorite Combo 1	1248.72
Subway	F11 Markaz	Sandwiches	Veggie Delite (6 Inches)	580
Pizza Hut	New Blue Area	Appetizers	Wings (4 Pieces)	499
McDonald's	F9 Park	Deals & Combos	McChicken w/o Cheese with Regular Drink	422.74
Tehzeeb Bakers	Blue Area	Drinks	Water	100

Seven rows of the unified output, one from each of seven chains, with the columns that matter for a comparison. Every row carries its source and scrape time too.

Rows per chain, 1,930 in all

Domino's Pizza309
McDonald's234
Tehzeeb Bakers206
Hardee's154
Cheezious121
Chaaye Khana118
Subway108
OPTP97
Pizza Hut89
Ranchers85
KFC76
14th Street Pizza65
Kababjees Fried Chicken63
Tayto60
Broadway Pizza53
Savour Foods47
Howdy45

Every chain in the sheet and how many menu rows it brought. Fourteen came through one delivery platform; three came from their own websites.

Bottled water, cheapest listing per chain, PKR

CheeziousWater Small60
Savour FoodsAquafina Water - 500 ml74
Hardee'sMineral Water90
OPTPMineral Water90
14th Street PizzaMineral Water Small99
SubwayAquafina Water - 500 ml100
Tehzeeb BakersWater100
McDonald'sWater (500ml)104.35
Kababjees Fried ChickenPremium Drinking Water 500 Ml120
Chaaye KhanaMineral Water - 500 ml175

One filter on the sheet: the cheapest water on ten menus runs from 60 to 175 rupees. Item names are as each chain lists them, and not every listing gives a size.

Impact / Learnings

The core lesson: normalization is most of the work. Getting the HTML off the page is the easy 20%: validating, deduplicating, and reconciling inconsistent categories across 17 independently-run sites is the part that actually determines whether the output is usable.

Architecture

Pipeline Design

17 sources: 14 via Foodpanda, 2 chain APIs, 1 Magento store
ThreadPoolExecutor: 17 concurrent jobs over httpx (HTTP/2)
Tenacity retries: exponential backoff on 429/500
Pydantic MenuItem model: type + field validation
pandas: dedupe, normalize categories, patch gaps
Google Sheets API: live unified output

Runtime

Run: python run_all.py --upload --gsheets-creds key.json
Export: python build_xlsx.py (styled local Excel alongside the sheet)
Auth: Google service account, scoped to the target Sheet

Code

run_all.py

# run_all.py (simplified) - 17 sources -> validated -> live sheet
# Concurrent fetch, strict schema, normalized export

from concurrent.futures import ThreadPoolExecutor, as_completed
from pydantic import BaseModel, field_validator
import pandas as pd

class MenuItem(BaseModel):
    chain: str
    category: str
    name: str
    price: float

    @field_validator("category")
    @classmethod
    def normalize_category(cls, v: str) -> str:
        return v.strip().title().replace("Add-Ons", "Addons")

def scrape_all(sources: list[str]) -> list[MenuItem]:
    items: list[MenuItem] = []
    with ThreadPoolExecutor(max_workers=len(sources)) as pool:
        futures = {pool.submit(scrape_one, s): s for s in sources}
        for future in as_completed(futures):
            items.extend(future.result())  # retried internally w/ backoff
    return items

def to_sheet(items: list[MenuItem]) -> pd.DataFrame:
    df = pd.DataFrame([i.model_dump() for i in items])
    return df.drop_duplicates(subset=["chain", "name"])

Want many sites in one live sheet?

Send a brief →

Need something like this? See Web Scraping, Monitoring & Alerting Automation.