Marketplace Monitor
Watches freelance marketplaces for new listings and emails a structured digest when one appears.
What this demonstrates for your project
- Problem
- Freelance marketplace and job postings get buried within hours; manual checks miss postings and naive scrapers fail silently.
- What was built
- A fail-closed Python monitoring pipeline with SQLite WAL journaling, canonical URL deduplication, and alert digests marked sent only after SMTP acceptance.
- What it proves
- Fails closed on dependency failure instead of reporting empty results; 304 collected tests run in CI across 4 Python versions (3.10 to 3.13) and 2 operating systems (Ubuntu and Windows).
- Status & context
- Completed internship work at Paramount Intelligence (July to August 2026). Proprietary code (private repository).
Overview
Problem
New listings on a job board, tender portal, or marketplace get buried under old ones within hours, making manual checks hard to keep up with.
This system periodically scans one or more listing feeds, extracts structured fields (title, rate, duration, location, contract type) from each posting, deduplicates against everything already seen, and emails a digest only for what's genuinely new. Built and run against three job and opportunity marketplaces; the architecture generalizes to any page that publishes a changing list of items: tenders, classifieds, inventory, pricing pages.
What I built
A resilient scanning layer captures listing and detail pages, a parsing layer resolves structured fields through a fallback chain, and a WAL-journaled SQLite database deduplicates by canonical URL while keeping full snapshot and observation history. An alerting layer marks HTML email digests sent only after the mail server confirms acceptance. A first-run safety gate refuses to send a flood of alerts for an already-populated feed; it requires an explicit baseline first.
Both cycles found nothing. Only one of them looked.
The page loaded as expected
- Fetch listing page
- Load check: listings and fields present
- Parse: 0 new listings
- Valid empty resultRecorded as no new listings
The site changed its layout
- Fetch listing page
- Load check: expected selectors missing
- Parse skipped
- Marked degradedCycle marked degraded, reason logged, no alert, retried later
{
"cycle": {
"baseline": false,
"degraded": false,
"detail_failure": 0,
"detail_success": 1,
"discovered": 22,
"emailed": 1,
"new": 1,
"personalized_feed_status": "ok",
"primary_feed_status": "ok"
},
"mode": "once",
"service": "freelancermap-monitor",
"status": "success",
"updated_at": "2026-08-04T06:51:32+00:00"
}$ python -m pytest -q
........................................................................ [ 23%]
............................................................ [ 43%]
........................................................................ [ 67%]
....................................................... [ 85%]
............................................. [100%]
304 passed, 36 subtests passed in 172.22s (0:02:52)Impact / Learnings
304 collected tests, including a property-based fuzz suite and an adversarial suite (malformed HTML, split DOM nodes, hidden modals), run in CI across 4 Python versions and 2 operating systems on every push.
The lesson that generalized best: reliability work is mostly about the failure paths, not the happy path: resuming an interrupted scan safely, marking a cycle degraded instead of pretending it succeeded, and marking a digest sent only after SMTP acceptance.
Architecture
Pipeline Design
- Listing feed(s)
- Browser session layer: persistent profile, verified sort/auth state
- Parsing layer: DOM + JSON-LD extraction, fallback chain
- SQLite (WAL): canonical-URL dedup, snapshot + observation history
- First-run baseline gate: never floods on an empty DB
- Email digest: mark sent only after SMTP confirms acceptance
Reliability Evidence
- Tests
- 304 collected (pytest, 2026-10-20)
- Fuzz
- randomized malformed-HTML inputs, invariants checked
- Adversarial
- prose pollution, split DOM, hidden modals
- CI matrix
- Python 3.10-3.13 x Ubuntu + Windows (8 jobs)
- Recovery
- resumable scans, bounded retry with backoff, profile-lock cleanup
- Boundaries
- no CAPTCHA/MFA/rate-limit/access-control bypass, by design
When it goes wrong
| What fails | What the system does |
|---|---|
| Feed/browser session fails to load correctly | Verified before use; unverifiable cycles are skipped and the reason is recorded, never silently treated as empty or successful. |
| First run against an already-populated feed | Refuses to alert until an explicit baseline snapshot exists, preventing a false flood of "new" listings on day one. |
| Scan interrupted mid-run (crash, network drop) | Resumable from a safe checkpoint, with bounded retry and backoff, plus profile-lock cleanup on restart. |
| Partial failure inside a cycle | Cycle is marked degraded rather than reported as a silent success. |
| SMTP send fails or is unconfirmed | Digest is only marked sent after the mail server confirms acceptance, never before. |
Need a source tracked and turned into structured alerts?
Send a brief →Need something like this? See Web Scraping, Monitoring & Alerting Automation.