Fail-Closed Web Scraping: Preventing Silent Data Loss in Python Automation

The Danger of "Fail-Open" Automation

A scraper with a broad try/except: pass block can hide a broken selector, a rate limit or a CAPTCHA challenge. The loop keeps running and returns an empty list, leaving the operator unable to distinguish a failed check from a feed with no new listings.

Naive "Fail-Open" Scraper:
[Target Changed] ──> [Selector Fails] ──> [except: pass] ──> silent data loss

Fail-Closed Architecture:
[Target Changed] ──> [Validation Fails] ──> [Cycle Marked Degraded + Reason Logged] ──> operator knows

The fail-closed cycle

The architecture is a straight line on a good day and a stop sign on a bad one:

source page
    |
    v
load verification (HTTP status, expected item count, required selectors)
    |
    v
parser -> Pydantic validation
    |
    v
SQLite in WAL mode (dedupe on canonical URL)
    |
    v
transactional outbox (one row per pending alert)
    |
    v
SMTP send -> mark sent only after confirmation

A failed cycle has its own sequence, and every step is deliberate:

source times out
    -> cycle marked degraded
    -> reason logged
    -> alert suppressed
    -> next cycle retries with jitter

A failed source check sends no digest and does not advance the baseline. The next run retries from the last verified state.

{
  "cycle": {
    "baseline": false,
    "degraded": false,
    "detail_failure": 0,
    "detail_success": 1,
    "discovered": 22,
    "emailed": 1,
    "new": 1,
    "personalized_feed_status": "ok",
    "primary_feed_status": "ok"
  },
  "mode": "once",
  "service": "freelancermap-monitor",
  "status": "success",
  "updated_at": "2026-08-04T06:51:32+00:00"
}

Why SQLite WAL Mode Suits Scrapers

In a background monitor, writes can contend with reads from a UI or exporter. In SQLite's default DELETE journal mode, that contention can produce sqlite3.OperationalError: database is locked when a lock outlasts the configured timeout.

Switching to Write-Ahead Logging (WAL) mode lets readers keep reading while one writer writes. Writers still take turns, so set a busy timeout as well.

Journal mode alone does not determine durability. With WAL and synchronous=NORMAL, SQLite keeps the database consistent, but a power loss or system crash can roll back a committed transaction. Application crashes do not have that durability risk. synchronous=FULL adds a WAL sync at each commit. Choose the setting based on what loss you can tolerate and measure it on your hardware. See SQLite's synchronous documentation.

A Minimal Python Example

Here is a minimal version of the database pattern. It is a simplified example for this post, not the monitor's own code:

import sqlite3
import time
from typing import Optional, Dict, Any

class ResilientDatabase:
    def __init__(self, db_path: str = "monitor.db"):
        self.db_path = db_path
        self._init_db()

    def _get_connection(self) -> sqlite3.Connection:
        conn = sqlite3.connect(self.db_path, timeout=30.0)
        # NORMAL keeps consistency; power loss can roll back recent commits.
        conn.execute("PRAGMA journal_mode=WAL;")
        conn.execute("PRAGMA synchronous=NORMAL;")
        conn.execute("PRAGMA busy_timeout=5000;")
        return conn

    def _init_db(self):
        with self._get_connection() as conn:
            conn.execute("""
                CREATE TABLE IF NOT EXISTS leads (
                    id TEXT PRIMARY KEY,
                    title TEXT NOT NULL,
                    budget REAL,
                    url TEXT UNIQUE NOT NULL,
                    discovered_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
                );
            """)
            conn.execute("CREATE INDEX IF NOT EXISTS idx_leads_url ON leads(url);")

    def insert_lead_atomic(self, lead: Dict[str, Any]) -> bool:
        """Atomic insert with duplicate detection. Returns True if new lead stored."""
        if not lead.get("title") or not lead.get("url"):
            raise ValueError(f"Schema violation: Required fields missing in payload {lead}")

        with self._get_connection() as conn:
            cursor = conn.cursor()
            cursor.execute("""
                INSERT OR IGNORE INTO leads (id, title, budget, url)
                VALUES (?, ?, ?, ?)
            """, (lead["id"], lead["title"], lead.get("budget"), lead["url"]))
            return cursor.rowcount > 0

The transactional outbox

An email is an external dependency, so it cannot be part of the database transaction. The outbox sits between them: the lead and its pending alert are written together, and the send step happens later against the outbox row.

CREATE TABLE IF NOT EXISTS outbox (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    lead_id TEXT NOT NULL REFERENCES leads(id),
    status TEXT NOT NULL DEFAULT 'pending'
        CHECK (status IN ('pending', 'sending', 'sent', 'failed')),
    attempts INTEGER NOT NULL DEFAULT 0,
    last_error TEXT,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    sent_at TIMESTAMP
);

CREATE INDEX IF NOT EXISTS idx_outbox_status ON outbox(status);

def store_lead_and_queue_alert(self, lead: Dict[str, Any]) -> bool:
    """Insert the lead and its pending alert in one transaction."""
    with self._get_connection() as conn:
        cursor = conn.execute(
            """
            INSERT OR IGNORE INTO leads (id, title, budget, url)
            VALUES (?, ?, ?, ?)
            """,
            (lead["id"], lead["title"], lead.get("budget"), lead["url"]),
        )
        if cursor.rowcount == 0:
            return False

        conn.execute(
            "INSERT INTO outbox (lead_id, status) VALUES (?, 'pending')",
            (lead["id"],),
        )
        return True

def mark_alert_sent(self, outbox_id: int) -> None:
    with self._get_connection() as conn:
        conn.execute(
            """
            UPDATE outbox
            SET status = 'sent', sent_at = CURRENT_TIMESTAMP
            WHERE id = ?
            """,
            (outbox_id,),
        )

The sender claims a row, sends it, and calls mark_alert_sent only after the mail server accepts the message. If the process dies before that update, the row stays pending and the next run retries it. The duplicate check is the unique url value on leads; the production monitor normalizes it to a canonical URL before insert, so an insert retry cannot create a second lead or outbox row. A send retry can still deliver a duplicate if SMTP accepted the previous attempt before the process died.

Automated CI Matrix Testing

Marketplace Monitor collects 304 tests. GitHub Actions runs them on a 4x2 matrix:

• Python versions: 3.10, 3.11, 3.12, 3.13
• Operating systems: Ubuntu and Windows
• Fuzz and adversarial cases: malformed HTML, DOM nodes split mid-field, hidden modals and text that looks like a listing but isn't.

These are collected tests, not a published pass rate. The point of the matrix is to catch platform-specific failures before a deployment, not to turn a test count into a quality score.

What fail-closed is not

• It is not CAPTCHA or anti-bot bypass. If a site does not want to be read automatically, a fail-closed monitor reports that and stops. Marketplace Monitor does not bypass CAPTCHAs, MFA, rate limits or access controls.
• It is not exactly-once delivery. The outbox makes duplicates unlikely and recoverable. A crash after SMTP accepts a message but before the sent update can still produce a duplicate on retry. Fixing that needs an idempotency key at the mail provider.
• It is not a substitute for monitoring the monitor. A process that stops running cannot log a degraded cycle. Something outside it has to notice the silence.

If your scraper currently reports an empty result when a source breaks, that is the problem this pattern solves. I build these as the fail-closed monitoring service.

Frequently Asked Questions

What happens when a website changes its HTML layout?

A fail-closed pipeline checks what it parsed before trusting it. If a page that normally lists twenty items suddenly yields none, or required fields come back empty, that cycle is marked degraded and the reason is logged. It is never recorded as "no new items", which is the silent failure you are trying to avoid.

How do you prevent duplicate alerts?

Dedupe on a canonical key, not on the raw URL. In Marketplace Monitor that key is the normalized listing URL, stored with a unique constraint. The lead insert and the pending outbox row happen in one transaction, and the sender marks the row sent only after SMTP confirms. A retry that hits the unique constraint writes nothing and queues no second alert.

How do you enable SQLite WAL mode in Python?

Execute PRAGMA journal_mode=WAL; immediately upon opening the SQLite connection. The setting is stored in the database file, so it persists across connections. Readers no longer block on the writer, though two writers still wait for each other.

How do you avoid getting blocked while monitoring a site?

Scrape politely and within the site's terms: check at a sensible interval, retry with jittered backoff, reuse one session instead of opening many, and respect rate limits. Marketplace Monitor deliberately does not bypass CAPTCHAs, MFA, rate limits or access controls. If a site doesn't want to be read automatically, look for an API or ask.