Marketplace Monitoring & Alerting

New listings on a job board, tender portal, or marketplace get buried under old ones within hours, checking manually means missing most of them.

This system periodically scans one or more listing feeds, extracts structured fields (title, rate, duration, location, contract type) from each posting, deduplicates against everything already seen, and emails a digest only for what's genuinely new. Built and run against three job and opportunity marketplaces; the architecture generalizes to any page that publishes a changing list of items: tenders, classifieds, inventory, pricing pages.

A resilient scanning layer captures listing and detail pages, a parsing layer resolves structured fields through a fallback chain, and a WAL-journaled SQLite database deduplicates by canonical URL while keeping full snapshot and observation history. An alerting layer sends HTML email digests only after the mail server confirms acceptance, never before. A first-run safety gate refuses to send a flood of alerts for an already-populated feed; it requires an explicit baseline first.

299 unit tests, a 1,000-input property-based fuzz suite, and a dedicated adversarial suite (malformed HTML, split DOM nodes, hidden modals) run in CI across 4 Python versions and 2 operating systems on every push.

The lesson that generalized best: reliability work is mostly about the failure paths, not the happy path: resuming an interrupted scan safely, marking a cycle degraded instead of pretending it succeeded, and never emailing before the send is actually confirmed.