Job-board monitors that fail closed

The two failure modes of a naive scraper

A fail-open scraper can swallow an error in a broad try/except, so the loop continues. When the site redesigns its listing page, the selector stops matching, the except swallows that too, and the run reports an empty list. An operator reading the report believes there were no new postings, when the truth is that nobody looked.

The second failure shows up the first time the monitor works. The initial run records every listing currently on the site, and if each new record triggers an email, the first cycle sends hundreds of stale alerts. Retries can also re-send: a crash after the mail server accepted a message but before anyone recorded it makes the next run send the same alert again. Both failures are structural, so both need structural fixes.

Verify the page loaded before you parse it

Before trusting a fetched page, check that it looks the way a real listing page should: the HTTP status is fine, the selectors that normally hold listings exist, and the extracted fields pass validation. If the checks fail, the fail-closed behaviour is to mark that cycle degraded, log the reason, suppress the alert, and retry later with jitter. Nothing is written as "no new listings" and no lead is marked as seen, so the next run starts from the same state instead of a false clean baseline.

An empty result can also be valid, so the load check must be separate from the "is there anything new" check.

SQLite in WAL mode, with a busy timeout

Background monitors write while the UI or an exporter reads. In SQLite's default rollback-journal mode, lock contention can produce database is locked errors. WAL mode lets readers keep reading while one writer writes. Writers still take turns, so pair it with a busy timeout instead of a bare except.

conn = sqlite3.connect("monitor.db", timeout=30.0)
conn.execute("PRAGMA journal_mode=WAL;")
conn.execute("PRAGMA synchronous=NORMAL;")
conn.execute("PRAGMA busy_timeout=5000;")

WAL persists in the database file, so one connection setting it is enough. busy_timeout is per connection, set it every time. With WAL and synchronous=NORMAL, power loss can roll back a committed transaction. The durability tradeoff and this connection setup are described in the fail-closed scraping post.

A transactional outbox for recording and retrying alerts

An email is an external service, so it cannot be part of the database transaction. The outbox sits between the two: the new listing and its pending alert row are written together in one transaction, and a separate send step works against the outbox, marking a row sent only after the mail server accepts the message.

source page
    |
    v
load verification (HTTP status, expected selectors, required fields)
    |
    v
parser -> schema validation
    |
    v
SQLite in WAL mode (dedupe on canonical URL)
    |
    v
transactional outbox (one pending alert per new listing)
    |
    v
SMTP send -> status stays pending until the server confirms

The next code block is an illustrative sketch of the pattern, written for this guide; it is not the production monitor's source.

import sqlite3, time

SCHEMA = """
CREATE TABLE IF NOT EXISTS listings (
    id TEXT PRIMARY KEY,
    source TEXT NOT NULL,
    title TEXT NOT NULL,
    canonical_url TEXT UNIQUE NOT NULL,
    discovered_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS outbox (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    listing_id TEXT NOT NULL REFERENCES listings(id),
    status TEXT NOT NULL DEFAULT 'pending'
        CHECK (status IN ('pending', 'sending', 'sent', 'failed')),
    attempts INTEGER NOT NULL DEFAULT 0,
    last_error TEXT,
    sent_at TEXT
);
"""

def connect(path: str) -> sqlite3.Connection:
    conn = sqlite3.connect(path, timeout=30.0)
    conn.execute("PRAGMA journal_mode=WAL;")
    conn.execute("PRAGMA synchronous=NORMAL;")
    conn.execute("PRAGMA busy_timeout=5000;")
    conn.executescript(SCHEMA)
    return conn

def record_listing(conn, source, listing, first_run: bool) -> None:
    """Store the listing and, unless first_run, queue one alert.
    The insert and the outbox row share one transaction."""
    with conn:  # commits on success, rolls back on error
        cur = conn.execute(
            "INSERT OR IGNORE INTO listings"
            " (id, source, title, canonical_url) VALUES (?, ?, ?, ?)",
            (listing["id"], source, listing["title"], listing["canonical_url"]),
        )
        if cur.rowcount and not first_run:
            conn.execute(
                "INSERT INTO outbox (listing_id) VALUES (?)", (listing["id"],)
            )

def send_pending(conn, deliver) -> None:
    """Deliver each pending row, then mark it sent only after the
    mail server accepted it. A crash leaves the row pending, so the
    next run retries instead of forgetting it."""
    rows = conn.execute(
        "SELECT id, listing_id FROM outbox"
        " WHERE status IN ('pending', 'sending') ORDER BY id"
    ).fetchall()
    for outbox_id, listing_id in rows:
        listing = conn.execute(
            "SELECT title, canonical_url FROM listings WHERE id = ?",
            (listing_id,),
        ).fetchone()
        deliver(listing)
        with conn:
            conn.execute(
                "UPDATE outbox SET status='sent', sent_at=datetime('now')"
                " WHERE id = ?",
                (outbox_id,),
            )

The duplicate check is the unique canonical_url on the listing, not the raw URL: normalise the URL before insert so one posting reached through two links does not become two alerts. Be honest about the limits: the outbox makes duplicates unlikely and recoverable, not impossible. A process that dies after SMTP accepts but before the status update will resend on retry, and fixing that needs an idempotency key at the mail provider.

First-run baseline protection

The first run after setup should record what is already on the site and send nothing. In the sketch above that is the first_run flag: the listing is stored, no outbox row is queued. From the second cycle on, only genuinely new listings produce alerts. Without this, the monitor greets its owner with every posting that happened to exist this morning.

The same idea protects restarts. Baseline state lives in the database, not in process memory, so a crash or re-deploy cannot turn the history into a first run again.

Test the parsers against a hostile web

Parser code fails quietly, so it needs adversarial tests, not just happy-path ones. The cases worth collecting: malformed HTML, DOM nodes split mid-field, hidden modals, and text that looks like a listing but is not one.

Marketplace Monitor, a monitor I built during my internship at Paramount Intelligence, collects 304 tests and runs in CI on a 4x2 matrix: Python 3.10 through 3.13 on Ubuntu and Windows. Those are collected tests, not a pass rate, and the repository is private, so there is not much more I can show of it than the case study already does. The failure the matrix is shaped for: a parser that works on Monday breaks on Windows, or breaks in a subprocess, and CI is where that shows before a deployment rather than after.

Compliance and scraping boundaries

A monitor you can keep running is one the target site can tolerate. In practice:

• Read the robots.txt and the site's terms before you start. If either forbids automated access to the pages you want, use the official feed or API, or ask.
• Check at a sensible interval, reuse one session instead of opening many, and back off with jitter when responses slow down.
• Handle 429 responses properly: read the Retry-After header, wait at least that long, and stop escalating after repeated 429s rather than rotating identities to get around them.
• No CAPTCHA, login or MFA bypass, and no circumventing access controls of any kind. If a site blocks automation, record the source as degraded and stop; a fail-closed monitor is designed to stop rather than find a way through.

This is the boundary I keep in my own monitoring work, and it is stated in the monitoring service description too. If a source cannot be read within its rules, the honest deliverable is a log entry and a recommendation to use the official channel.

A review checklist for your own monitor

The checks from this guide in one list, to run against a monitor you already have or one you are about to build.

Frequently asked questions