Menu Data Pipeline

Keeping a menu comparison spreadsheet current across 17 different restaurant websites by hand is a full afternoon of copy-paste, every time a menu changes.

This pipeline collects menu items from 17 restaurant chains in Islamabad, validates and normalizes every field, and writes the unified dataset directly to a live Google Sheet, turning a manual, error-prone task into a single command.

A thread pool runs all 17 sources concurrently over HTTP/2 (via httpx), cutting a 15+ minute sequential run down to a few minutes. Requests that get throttled or fail (HTTP 429/500), including a 250+ page catalog from one chain, retry automatically with exponential backoff via Tenacity. Collection stays within each source's published access rules and runs at a conservative, rate-aware pace. Once collected, pandas deduplicates and standardizes messy categories (merging near-duplicates like "Add-ons" and "Addons") before the final push to Sheets.

The core lesson: normalization is most of the work. Getting the HTML off the page is the easy 20%: validating, deduplicating, and reconciling inconsistent categories across 17 independently-run sites is the part that actually determines whether the output is usable.