My Website Has a Test Suite

Deprecations, external dependency shutdowns, infrastructure changes, data changes and shifting standards. Websites rot: each broken piece sits there for months until a reader (or worse, a potential employer) finds it before you do. I got tired of discovering this stuff by accident, so I gave it a test suite.

This post covers a static audit script that reads every page like a very pedantic proofreader, and a headless browser test that loads the site like a very terminal focused nerd. Those are the two halves I started with. Today one Python file, run_checks.py, lists 26 static checks and three browser suites, and GitHub runs all of them on every push and pull request.

The proofreader

The first half is audit_site.py, a Python script with no dependencies beyond the standard library. It asks git for every tracked HTML file, parses each one, and complains about everything it doesn't like. It requires zero errors, zero warnings, on every page, on every commit.

What it checks, roughly in order of how often it has saved me:

• Broken links and images. Every internal link and image source is resolved to an actual file on disk. A renamed page or a missing thumbnail fails the audit immediately.
• Metadata. Every page needs a description of sensible length, a canonical URL that matches its filename, and the social preview tags that make links look respectable when shared. The audit also validates that the preview image actually exists (this caused me so many issues trying to understand how social media sites grab images).
• Structured data. Each blog post carries a machine-readable summary for search engines. The audit parses every one and checks each URL inside it, because invalid structured data fails silently in the real world.
• Accessibility basics. Images without alt text, duplicate element ids, missing or multiple h1 headings, and skipped heading levels all get flagged for accessibility requirements I set.
• The data files. The blog listing is driven by a JSON file: every entry must point at a real page and a real image, every date must be well-formed, and every tag must come from a fixed allowed list. That last one exists because the tag buttons are generated dynamically, so a typo like "phtography" wouldn't error; it would just mint a brand-new filter button and display it.
• The feeds. The RSS feed and the sitemap are parsed and reconciled against the post list, so publishing a post without wiring it up everywhere gets caught. Not much point in publishing a new blog that isn't shown anywhere!

Headless smoke tests

The audit reads files; it cannot tell you whether the page actually works. This site injects its header, footer, comments, and related posts with JavaScript, and it animates each section heading in as you scroll. A one-character mistake in that JavaScript can leave a page technically valid and completely blank, and the audit wouldn't catch it.

So the second half is a smoke test built on Playwright, which drives a real, invisible Chromium. The script serves the repository over a local web server, loads eight representative pages, waits on each one until the things it checks for have turned up (a page still not ready a couple of seconds past its budget fails, instead of being checked half-built), and then asserts on what a visitor would actually see:

# A few of the checks, one page each:
# index.html   - hero present, live-stats skeletons cleared, blog cards visible
# blog.html    - at least 3 post cards and 3 tag filter buttons rendered
# gallery.html - at least 12 photos in the grid
# a blog post  - body text visible (opacity != 0), table of contents built
# series page  - at least 4 parts, each with its part chip
# quotes.html  - at least 500 quote cards
# map.html     - the map initialised with at least 10 markers

The site used to reveal whole sections with a fade-in animation, which meant a JavaScript failure didn't remove content, it left content sitting in the page at opacity zero: present in the HTML, invisible to humans. That is the failure a static checker can never catch. The fade-ins are gone (only a section's heading animates now, on a CSS scroll timeline where the browser has one), but the check stays as a guard. It's a Python tuple holding a line of JavaScript, evaluated inside the live page:

("post body visible", "(() => { const p = document.querySelector('.blog-post > p'); return p && getComputedStyle(p).opacity !== '0'; })()"),

The test also collects console errors, ignoring the third-party noise (analytics, embeds) and failing on anything from my own code. Past those eight pages, it loads every top-level page at phone width (360px) and desktop width (1280px) and fails if anything runs off the right edge, runs every interactive demo and 120 frames of every hover sketch on the blog cards, and replays a list of pins: one check for each bug that shipped and got fixed, each one proved against its bug before I trusted it.

The robot that runs it all

All of it runs in GitHub Actions on every push and pull request. The workflow has two jobs. One installs Python and works through 24 named steps, one per check: the data files against their schemas, the post markup, whether every generated file still agrees with its source (each post's read time, share image and opening header, the related-posts blocks, the feed, the sitemap, the gallery's photo sizes, the colophon's figures), the stylesheet's rules and whether the minified stylesheet is up to date (the CSS build is its own story), the Python and JavaScript unit tests, and the audit itself. The other installs Playwright and runs the three browser suites: the smoke tests, a check that every post renders its template, and a set of performance budgets. The black and white tag check lives in that second job too, because it needs Pillow to read the pixels. One of the steps, --ci-parity, fails if the workflow and the list in run_checks.py ever disagree, because they used to be kept in three places and drifted apart. If anything fails, I get an email and a red X on the commit within a few minutes. The same list runs before a commit, too: the repository carries a pre-commit hook that runs every check needing nothing beyond Python itself, so most mistakes never even reach GitHub.

Is this overkill for a personal site? No: it's because the site is a hobby that I need automation. Nobody is paid to notice when this site breaks (or, you know, paid for writing blogs, creating the site, etc.). There is no QA department, no on-call rota, just me, and I would rather spend my evenings writing posts than manually clicking every link on fifty pages. The test suite is what lets a one-person site behave like it has staff.

The suite also enforces my rules on me. Zero errors and zero warnings sounds strict, but a clean baseline means any non-zero number is news.

What it still misses

The audit checks that every referenced image exists on disk. It does not check that the image is committed to git. I published a post whose hero image existed on my machine but had never been committed, every local check passed, and the live site served a grey void where a seascape should have been. A reader (fine: me) spotted it and fixed it later that day, but that's not great when I already had a few dozen readers.

Each gap you find becomes the next check you write. The fix here is a rule that any local file referenced by a tracked page must itself be tracked, which is one more function in the audit script. Here is that check as it runs today (lightly trimmed):

def check_target(url, line, what="target"):
    local = resolve_local(c.page, url)
    if local is None:
        return
    if not local.exists():
        c.add("ERROR", c.page, line, "broken-link", f"missing {what}: {url}")
    elif c.page_is_tracked:
        try:
            relp = local.resolve().relative_to(ROOT).as_posix()
        except ValueError:
            relp = None
        if relp and relp not in c.tracked:
            c.add("ERROR", c.page, line, "untracked-ref",
                  f"{what} exists locally but is not tracked by git: {url}")

Fittingly, that check fired while I was preparing this very post: the hero image at the top of this page existed on my machine and had never been committed. Test suites are never finished, only extended.