Letting Robots Update Your Homepage
My homepage shows my scrobble count, the book I'm working reading through, the last song I played, and whatever I last posted on Bluesky. This is a static site, meaning it can't just run dynamic scripts on a whim. I set up a system to dynamically update my HTML weekly, however, and it works for free. Part of the series on how this site is built.
Quick jargon guide
- GitHub Actions: GitHub's free automation. You describe a job in a text file, and GitHub runs it on their machines, on a schedule if you like.
- Cron: the venerable syntax for "run this at these times". Mine says
0 6 * * 1: every Monday at 06:00 UTC. - API: a website's machine-readable front door. Last.fm's API answers "what did this user play?" with data instead of a web page.
- API key / secret: the password for that front door, stored in GitHub's encrypted secrets so it never appears in the code.
- Scrobble: one logged song play. My Last.fm account has been counting since 2011.
- Bot commit: a change to the repository made by an automation rather than a person.
The idea: if the site can't fetch, fetch into the site
A normal website with live data has a server that queries things when visitors arrive. A static site can't do that, but it has a loophole: the site is a git repository, and anything that can commit to the repository can change the site. So instead of fetching data per visit, a scheduled job fetches it once a week and commits the results as plain JSON files. The site stays static. Visitors' browsers read those JSON files with an ordinary fetch: same origin, no keys, no rate limits, nothing that can fail separately from the site itself.
The whole pipeline is one workflow file and one Python script. The workflow is short: check out the repository, run the script, and commit if anything changed:
name: Refresh live data
on:
schedule:
- cron: "0 6 * * 1" # weekly, Mondays 06:00 UTC
workflow_dispatch:
permissions:
contents: write
jobs:
refresh:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Fetch live data
env:
LASTFM_API_KEY: ${{ secrets.LASTFM_API_KEY }}
run: python .github/scripts/refresh_now.py
- name: Commit if changed
run: |
FILES="data/lastfm.json data/now.json data/lastfm-history.json data/topalbums.json"
if git diff --quiet $FILES; then
echo "No change."
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add $FILES
git commit -m "Refresh live data (Last.fm, Goodreads)"
git push
The workflow_dispatch line adds a manual "run now" button in GitHub's interface (which you will want the first dozen times), while the diff check means runs where nothing changed produce no commit at all.
What the script gathers
The Python script makes a handful of HTTP requests. From the Last.fm API: my total play count, the most recent track, and my top albums of the past three months, which feed the homepage stat, the "now playing" strip, and the album wall on the music page respectively. From Goodreads, which shut its API years ago but still publishes RSS feeds: my currently-reading shelf, parsed straight out of the XML. Everything goes into a small JSON files under data/, the biggest a few kilobytes.
Each run appends that week's play count to a rolling log, capped at ninety entries, which at one measurement a week is close to two years of history.
What the homepage shows
The strip under the hero is a single row of five things, and only three of them come from the robot. The book and the track are read straight out of now.json. The latest blog post and the latest short story come from posts.json and stories.json, which are written by build scripts when I publish rather than by the Action on a schedule.
Last.fm distinguishes a track that is playing from one that was played, so the label switches between "Now playing" and "Last played" depending on whether I happen to be listening when you open the page, and the first version gets a little animated equaliser beside it. It is not accurate, because it is updated so infrequently, but it gives some flavor to the site and over time it's likely representative of me. Maybe.
The fifth item, my most recent Bluesky post, is in no JSON file at all. Your browser fetches it from Bluesky's public API while the page loads. I've been tempted to remove it, as recently I've only posted new blogs to social media, but I hope I will return to posting random thoughts there again soon, making this a bit more useful than another place to see a new blog post (which is already on the index page!)
Design decisions
The homepage HTML contains a hardcoded scrobble count, and the JavaScript replaces it with the fresh value from the JSON after load. If the fetch fails, or JavaScript is off, the visitor sees a slightly stale number instead of a blank. The robot improves the site but it fails elegantly.
Goodreads' RSS goes down more often than Last.fm's API. When a source fails, the script keeps whatever the previous run wrote rather than blanking it: better yesterday's truth than today's error. A failure in one source never stops the others from updating, of course.
The Last.fm key lives in the repository's encrypted secrets and reaches the script as an environment variable. It appears in no file and no log for security reasons (not that I'd be massively put-out if someone broke into my goodreads or last.fm).
The bot's commits are labelled as bot commits, touch only the data files, and say what they did. My git history has a weekly heartbeat in it now: a tidy row of "Refresh live data" commits, one every Monday, each one the robot clocking in so I don't have to manually update stuff.
What else this pattern is good for
Anything that changes slowly and comes from somewhere with an API or a feed: your latest posts elsewhere, sports scores, weather, stars on a project, prices you're tracking, a "days since" counter for whatever you are currently ashamed of. The recipe is always the same three steps: a script that fetches and writes JSON, a workflow that runs it on a schedule and commits, and a page that reads the JSON with a fallback. If the data changes faster than a schedule can keep up with (live chat, comment counts), a static site is the wrong tool and no robot can easily fix that. Everything slower works just fine though.
Common questions
Doesn't committing data on a schedule bloat the repository?
Slowly, and acceptably. Each commit stores a few kilobytes of changed JSON, so a year of weekly commits comes to a couple of hundred kilobytes.
Why weekly instead of hourly, or on every visit?
I could but it's not exactly vitally important and worth moving out of the free level of GitHub actions.
What happens when the robot itself breaks?
GitHub emails me when a scheduled workflow fails.
Could this update the page instantly when something changes?
Not this pattern. A schedule can only poll. Instant would need the source to push (webhooks) into something always listening, which drags a server back into the picture.
