Ten books is nothing next to his bibliography, and four I've read are short stories or novellas, so this is not the definitive ranking by any means. The Goodreads average is next to each one for comparison. Lowest first.

Goodreads readers and I agree on the best of King's work (in my opinion): The Shining has the highest average of the ten, 4.28. We disagree most about the book in second place on their list. It averages 4.27, and I gave it two stars. Below the top, my ratings mostly sit within half a star of everyone else's, except Carrie and Cujo, which I liked less than most people seem to.

10. It (1986)

Me

2 out of 5

Goodreads

4.27 out of 5

It has a brilliant and terrifying premise: a group of kids in Derry, Maine, unite to confront an ancient, shapeshifting entity that feeds on fear, then return 27 years later as adults to finish the fight. But, sadly, I had to give it two stars. King's writing is atmospheric but lacking something in immersion. I find the mythology of his wider universe really interesting, but I didn't like this one the way I expected to. The last quarter is so abstract, exploring concepts rather than actual events, even in the fight scenes, that I lost interest pretty quickly. And, of course, the scene with the children in the sewers near the end was gross. I get that it fits the theme, but it didn't sit well with me, and regardless of what it conveys conceptually or King's intent in writing it (or how much his face resembled someone who fell into fresh snow at the time of writing), I tend to find an "orgy of children" to be at best uncomfortable, unnecessary and concerning.

I wish there was more exploration of It as a creature, the deep dark mythology of the Lovecraftian monster with its odd diet and fingers in various stories - especially considering the epic length of the book. Speaking of the length, there were various subplots and meandering chapters that felt unfinished and/or unnecessary. All in all, a disappointing read.

9. Cujo (1981)

Me

3 out of 5

Goodreads

3.80 out of 5

A mother and her son trapped in a sweltering car by a rabid Saint Bernard is a great setup, and the survival story at the middle of it is interesting, psychological and tragic. The writing around it is sluggish. It's a disjointed stream-of-consciousness style with no chapters, which keeps breaking the flow (chapterless books are fine but you need to make it work!). The tension also keeps getting derailed by bloated subplots about a failing cereal marketing campaign (???) and an affair, so it's hard to stay lost in the horror. A decent, middle-of-the-road horror novel that must have skipped the editor on the way to publication.

8. Carrie (1974)

Me

3 out of 5

Goodreads

4.00 out of 5

Carrie was King's first novel and it carries a lot of cultural weight (though I think the movie did a lot of that with the visuals), but it reads like a first book: raw, messy and sometimes brilliant. The core of the story is a page-turner for sure. King nails the cruelty of high school and what it's like to live with an abusive, fanatical parent, and you feel for Carrie White so much that the prom, when it comes, is tragic and also super cathartic.

The execution is clunky. The story keeps cutting away to fake news clippings, book excerpts and reports, which kills the suspense right when it builds, at least it did for me. Margaret White and Chris Hargensen are cartoonishly evil, there to push Carrie over the edge and not much else. It's absolutely worth reading to see where the master of horror started.

Mike Flanagan's Carrie series comes to Prime Video on 7 October 2026, all eight episodes at once, with the story moved into the social-media age. As a huge fan of the Flaniverse, I know I'll be watching this.

7. The House on Maple Street (1993)

Me

3 out of 5

Goodreads

3.60 out of 5

The hook is fun: the Bradbury kids work out that their house is swapping its wood and plaster for alien metal and a countdown clock, and they use it to get rid of their awful stepfather by trapping him inside just before liftoff. It's a creative nod to classic sci-fi, it moves fast, and the worry for their mum gives it some weight. But it didn't pull me in. The jump from a family drama to a literal spaceship taking off is too abrupt, and nobody ever explains how the house works, so it reads like a B-movie or a pulp comic, not a world I could sink into.

6. The Jaunt (1981)

Me

4 out of 5

Goodreads

4.16 out of 5

A short story about teleportation, and it tickles a certain trope I adore: the Jaunt isn't instant travel from A to B, it's something darker. In King's future everyone jaunts, but you have to be put under first, because anyone who goes through awake spends an eternity alone in white nothing while their body arrives in an instant. The dread lands on you the same way it lands on the characters.

The father spends a big chunk of the story telling his kids the history of the technology, and the lore is interesting, and if the story were longer it would have a purpose, but as it is the tension of the family's trip just stops for a history lesson. Once the jump happens it drags you straight back in, and the ending leaves with an unsettling idea to consider.

5. Things They Left Behind (2005)

Me

4 out of 5

Goodreads

3.80 out of 5

A short story about survivor's guilt after 9/11. Scott Staley called in sick to his job at the World Trade Center that morning, and a year later things that belonged to his dead coworkers start turning up in his apartment, a pair of sunglasses, a baseball bat. King keeps the supernatural part completely ordinary and it works: the guilt is heavy and Scott can't get out from under it. It loses a star because once he works out he has to take the things back to the families, it turns into the same visit over and over. Tying up a real tragedy with a neat supernatural bow also gets a bit too sentimental for me, and it undoes some of the realism of the first half. Still a great read.

4. Apt Pupil (1982)

Me

4 out of 5

Goodreads

3.92 out of 5

This one is disturbing, and it's about people, not the supernatural. Todd Bowden is an American golden boy who finds out his elderly neighbor is Kurt Dussander, a Nazi war criminal in hiding. Instead of turning him in, Todd blackmails him into telling every gruesome detail of what he did. It's a weird parasitic relationship, with the boy manipulating this old man, who appears to just want to live out his days in peace, leaving his awful past behind. At first, at least. The premise is great and you keep hoping the 'apt pupil' will wake up and be redeemed.

Then the years pass, both of them start killing local transients on their own, and the tension of those talks in Dussander's kitchen turns into so much violence that it numbs more than it scares. The pacing and execution of the second half is a bit off. The ending is bleak and explosive, and there isn't one sympathetic character in the whole thing.

3. The Gunslinger (1982)

Me

4 out of 5

Goodreads

3.94 out of 5

I was surprised to learn as a teenager that Stephen King had written an epic fantasy series, as I had only known him for horror. The Gunslinger is the first Dark Tower book: Roland Deschain chasing the Man in Black across a dying desert, and the heat and the emptiness pull you right in. The world is completely bizarre, high fantasy and post-apocalyptic sci-fi and a brutal spaghetti western, of all things, and King doesn't explain any of it, he just drops you in. It was first published as separate short stories and you can tell. Roland's past in Gilead is fascinating, but the flashbacks keep stopping the chase. The final act is a huge, mind-bending philosophical conversation that leaves you with far more questions than answers - this isn't bad but it does make the pacing a bit disjointed.

2. Pet Sematary (1983)

Me

4 out of 5

Goodreads

4.21 out of 5

I first read Pet Sematary as a kid, and it left a creepy impression on me. It's about how far a parent would go to undo a loss, and King puts you inside Louis Creed's head as he talks himself into each terrible decision on the way to the old burial ground. The walk through the woods past the deadfall is the scariest part of the book, and it makes the supernatural feel real. The slow part is the build-up, with lots of the Creeds' everyday family life and lots of foreshadowing. It makes the loss hurt more when it comes, but the story stalls before the third act.

The film adaptations don't live up to the writing either. They go for shock and gore and miss the tension. The real horror isn't the cat or the toddler, it's grief, and how much pain it takes before you'd do anything to make it stop.

1. The Shining (1977)

Me

5 out of 5

Goodreads

4.28 out of 5

Five stars, the only one. Jack Torrance takes a winter job as caretaker of the Overlook Hotel, up in the Colorado Rockies, and moves his wife and son in with him. The snow cuts them off, and you watch Jack come apart slowly while the boiler runs too hot and the hotel works on him. The haunting is scary because it's tied to real things, his drinking and the way he's hurt his family before. Unlike in It, King actually gives a good amount of space for the supernatural world to be explored and probed, which gives breadth and depth to his multiverse and the magic within it.

Kubrick's film is a perfect piece of cinema in its own right, but it's a different story. His Jack looks halfway to madness from the very first frame. King's Jack is a loving but badly flawed father who gets eaten away by the hotel, and that makes the book a tragedy, with a warmth and heartbreak the film leaves out on purpose. In my opinion it's the best example of "the building is the monster" I've ever read.

There are many books of King's I haven't read, so I can't comment on everything. I should mention I DNF'd The Stand - I had high hopes as I love post-apocalyptic and fantasy books alike, but it felt like a drag and not worth pushing to the end.

I don't describe myself as a King fan, because I find his writing a real mixed bag, and his movies are also mixed with a tendency to be worse. But he is a cultural icon, with the respect of many, and he deserves it for sure. I feel there is an element of Americana in his literature I don't connect with, and I think there's also a generational gap - he writes for a world where kids grew up in the pre-90s, and expects readers to fill in the gaps of the world he draws only an outline for.

Finally, I find his work lacking in depth. There is such potential in the multiverse, but it's so barely explored and only hinted at, though I know I am missing context in much of the Dark Tower series and potentially other books, too.

Hero image: Stephen King's house on Broadway in Bangor, Maine, with its bat-and-spiderweb iron gate. Photograph by Warren LeMay, darkened for this page, via Wikimedia Commons (CC BY-SA 2.0).

In a 2024 paper, Windred and colleagues took more than 10 million hours of accelerometer data from 60,977 people in the UK Biobank and found that how regular someone's sleep was predicted their risk of dying over the following years better than how long they slept^[1]. People are obsessed with sleeping pills, mattresses and spending insane amounts of money and time on these, when there are many much simpler (and healthier) solutions backed by actual science - and you can tell it's not some grift because no one is making money off of it (unlike the pills and mattresses).

I have the same problem with the 'sleep influencers' on YouTube shorts, Instagram reels and TikToks that I have with most self-help books. Their whole business is telling you that you're doing sleep wrong and dying faster because of it, and some of that is built on good science, but is driven by a will to make money off of you, through selling books, sleep accessories, diet plans or 1 on 1 mentoring for some obscene costs.

What actually works

The core of the science is solid, and fairly simple. Sleep is when a lot of important biology happens, memory consolidation for one and, going by more recent work, the brain clearing out metabolic waste. Chronic short sleep goes with worse heart health, worse mood, worse thinking and more accidents, and it keeps showing up that way study after study. The thing that really sold "better sleep" to me is that I've been working on my physical health and wellbeing, hitting the gym a lot - and it turns out good sleep is essential to maximize the gains you reap at the gym. What's the point in putting in all the effort to learn how to play guitar, or memorize content for class, or exhaust yourself at the gym, only to throw away a % of the gains from these things by sleeping poorly? So, here is some high level information I found backed by the science:

• Regular times. Going to bed and waking at the same times every day is one of the best-supported habits there is, and in the Windred study it predicted the risk of dying better than total hours did.
• Light. Bright light in the morning and dim evenings do more for your body clock than most gadgets.
• Apnea. If you snore heavily, wake up unrefreshed, or someone has noticed you stop breathing in your sleep, get assessed. Untreated apnea is a real cardiovascular risk, and getting checked for it is covered by most health insurances (from what I've read: I am not your insurance specialist!)

Oversold

Matthew Walker's Why We Sleep did a lot of good for how seriously people take sleep. It changed my habits more than any other book I've read, and I gave it five stars (which is not particularly rare for me but still). Alexey Guzey went through it and argues that the World Health Organization "sleep loss epidemic" in it doesn't exist (he couldn't find a WHO document declaring one), and that a bar chart on sports injuries, adapted from another study, lost its 5-hour column, the one showing fewer injuries than 6 hours^[2], so some of it is fairly called out, but the central argument very much survives: sleep matters and most of us don't get enough of it.

Tracker "sleep scores" are largely guesswork. A consumer ring or watch estimates your sleep stages from movement and heart rate. These devices are bad at telling light, deep and REM sleep apart compared with polysomnography, and the app still shows you a confident number every morning. It runs largely guess work and validation.

Most sleep supplements have little behind them except marketing. Even melatonin, which does have real uses for shifting your body clock, is routinely sold at doses far higher than the evidence supports and taken as a sedative, which it isn't. In fact, the aforementioned book specifies that melatonin is a wonder-drug - for changing your sleep cycle when traveling - only. It's not intended for, nor good at, aiding people with sleep when they're not changing time zones. If you take it otherwise and find it useful, that's great, but it's likely as useful as a sugar pill, but with potential side effects. Think on it and settle for some of the other tips in this page.

Orthosomnia

Baron and colleagues described orthosomnia in 2017: patients so anxious about getting their tracker's numbers right that the anxiety itself was damaging their sleep^[3]. You lie awake worrying about whether you're getting enough deep sleep. Or you wake up feeling fine, check the app, find out you "slept badly", and feel worse all day than you would have if you'd never looked.

What to do

You don't need to buy anything to sleep well. Roughly in order of how much they help:

1. Routine. Go to bed and get up at the same times every day, weekends included (I know, that's a hard ask). In the Windred study, regularity predicted the risk of dying better than total hours did^[1].
2. Light. Get daylight in the morning, ideally outside, and keep the evenings dim. Light is the main thing that sets your body clock, and a bright screen at midnight tells it it's still daytime.
3. Caffeine. It lasts much longer than it feels like it does. In healthy adults its half-life is between 3 and 7 hours^[5], so half of a 3pm coffee can still be in you at 9pm, and a quarter at bedtime. Drake and colleagues gave people 400mg (two or three big coffees) six hours before bed and it still cut their sleep by more than an hour^[4]. If you drink it at all, drink it in the morning (after all, caffeine is actually great for you cognitively and physically^[9][10], hence the culture around it being a key ingredient in preworkouts, energy drinks and synonymous with studying!).
4. Temperature. Keep the bedroom cool. Under a duvet, being too warm costs more deep and REM sleep than being cold does, and humid heat is worse again^[6].
5. Exercise. Regular exercise has a moderate effect on how well people say they sleep, and small effects on how long they sleep and how fast they drop off^[7]. It's good for everything else too.
6. Hydration. Drink enough through the day, but not a pint right before bed unless you enjoy 3am trips to the toilet. People who slept six hours were more likely to be under-hydrated than people who slept eight^[8].

Everything past that (the scores, the supplements, the doom, the optimising) ranges from "might help" to overthinking and anxiety provoking. Unless your doctor says otherwise.

One major tip that I think lies overarching most of these but isn't directly stated so far: stress will keep you up. There isn't a single pill to fixing stress, but these other things can help regardless, and whether you need to lower your ambition, worry less about things not in your control (therapy? meditation? prayer? journal keeping?) or change your personal circumstances to less stressful ones (new job? cut off that toxic person? move to somewhere warmer or more beautiful?), lowering your stress is also wonderful for your sleep habits, your mental health, and your physical health - even if it's easier said than done.

References

1. Windred, D. P., et al. (2024). Sleep regularity is a stronger predictor of mortality risk than sleep duration. Sleep, 47(1), zsad253. https://doi.org/10.1093/sleep/zsad253
2. Guzey, A. (2019/2022). Matthew Walker's "Why We Sleep" is riddled with scientific and factual errors. https://guzey.com/books/why-we-sleep/ (a critique; treat as a starting point, not a final word, and read the exchanges it prompted)
3. Baron, K. G., Abbott, S., Jao, N., Manalo, N., & Mullen, R. (2017). Orthosomnia: Are some patients taking the quantified self too far? Journal of Clinical Sleep Medicine, 13(2), 351–354. https://doi.org/10.5664/jcsm.6472
4. Drake, C., Roehrs, T., Shambroom, J., & Roth, T. (2013). Caffeine effects on sleep taken 0, 3, or 6 hours before going to bed. Journal of Clinical Sleep Medicine, 9(11), 1195–1200. https://doi.org/10.5664/jcsm.3170
5. Caffeine. Wikipedia (accessed September 2026). https://en.wikipedia.org/wiki/Caffeine
6. Okamoto-Mizuno, K., & Mizuno, K. (2012). Effects of thermal environment on sleep and circadian rhythm. Journal of Physiological Anthropology, 31, 14. https://doi.org/10.1186/1880-6805-31-14
7. Kredlow, M. A., Capozzoli, M. C., Hearon, B. A., Calkins, A. W., & Otto, M. W. (2015). The effects of physical activity on sleep: a meta-analytic review. Journal of Behavioral Medicine, 38(3), 427–449. https://doi.org/10.1007/s10865-015-9617-6
8. Rosinger, A. Y., Chang, A.-M., Buxton, O. M., Li, J., Wu, S., & Gao, X. (2019). Short sleep duration is associated with inadequate hydration: cross-cultural evidence from US and Chinese adults. Sleep, 42(2), zsy210. https://doi.org/10.1093/sleep/zsy210
9. McLellan, T. M., Caldwell, J. A., & Lieberman, H. R. (2016). A review of caffeine's effects on cognitive, physical and occupational performance. Neuroscience & Biobehavioral Reviews, 71, 294–312. https://doi.org/10.1016/j.neubiorev.2016.09.001
10. Guest, N. S., VanDusseldorp, T. A., Nelson, M. T., Grgic, J., Schoenfeld, B. J., Jenkins, N. D. M., Arent, S. M., Antonio, J., et al. (2021). International Society of Sports Nutrition position stand: caffeine and exercise performance. Journal of the International Society of Sports Nutrition, 18(1), 1. https://doi.org/10.1186/s12970-020-00383-4

It has its own vocabulary (registrars, DNS, A records, MX records), the settings pages kind of looks like a programmer's encyclopedia, and one wrong entry feels like it might break the internet for everyone else too. Nothing you do here is hard to undo, and you do most of it once then forget how it's done (until you write a blog post on it, I guess).

Step one: rent the name

Pick a registrar (mine is Bluehost through historical inertia, and Cloudflare, Porkbun and Namecheap are the usual modern recommendations), search for the name, pay. A .co.uk or .com runs roughly £10–15 a year. If you host via GitHub, this is your only expense - pretty cheap, right?

Register it in your own account, not a friend's, not a freelancer's, and definitely not your employer's. Turn auto-renew on and keep the payment card current, because expired domains get bought by squatters within hours (if your website is big enough to warrant that). And treat the registrar account like a bank login (real password, two-factor), because whoever controls it controls everything downstream, including your email!

On the name itself: your own name is a fine default for a personal site (this one is proof), shorter beats cleverer, and any mainstream ending is fine. The exotic endings (.io, .dev, .pizza) work identically but often cost more at renewal, and might make it difficult to convince people of your email address (I've had people ask me if "I'm sure" that my email address really does end with @kenreid.co.uk - yes, it does!). Introductory first-year discounts are the industry's favourite trick.

Step two: point it at your host

Your registrar's DNS page is where the twenty minutes happens, and you add a handful of records that say where web traffic should go. For this site, hosted on GitHub Pages, that means four A records pointing the apex at GitHub's published IP addresses, and one CNAME record pointing www at my GitHub Pages hostname. Every host publishes the records it wants and the shape is always the same few lines. Mine, conceptually:

Then you tell the host which domain to answer for. On GitHub Pages that's a one-line file named CNAME in the repository (mine says www.kenreid.co.uk), or the same setting in the repository's Pages settings, plus a checkbox to enforce HTTPS. The certificate is issued automatically within the hour, free, and renews itself forever. In 2012 the padlock cost real money and real ceremony, and now it is free generally.

DNS changes take anywhere from minutes to a day to spread across the internet - be patient.

Step three (optional, recommended): email at your name

The domain routes email separately from the website, via MX records, so ken@kenreid.co.uk works without GitHub ever seeing an email. I pay Fastmail a few pounds a month to host the mailbox. Adding their MX records at the registrar took five minutes with their guide open in another tab.

An email address you own outlives every provider, every job, and every free service's pivot to enshittification. Mine will follow me to any mail host forever, because the name it lives at is rented in my name.

Whether your site's official address is the bare domain or the www one. Both work (one redirects to the other), but pick one and use it everywhere, because search engines treat them as different sites. This site standardised on www.kenreid.co.uk and every page declares it. The metadata post covers the tags that keep that consistent.

What ongoing maintenance looks like

Approximately none. The domain renews itself annually (you get an email and a small charge), the certificate renews itself without telling you, and the DNS records sit untouched for years. The one day they change is the day you switch host or email provider. You edit a few lines, and everything else follows the name across.

Register it in your own account, turn auto-renew on, and then ignore it for a year. Well worth the minor cost, in my opinion.

Steps, concisely

1. Buy: register your domain and turn on auto-renew.
2. Point: add A and CNAME records at your registrar to send traffic to your host (GitHub's DNS guide).
3. Connect: tell your host your domain name and turn on HTTPS (hosting on GitHub Pages).
4. Email (optional): add MX records to send mail to your provider (Fastmail's custom domain guide).
5. Forget: keep your payment details current, let auto-renew run, and use your chosen address everywhere (the metadata post).

My website has a bunch of different optimisers running in blog posts and a photograph of my face on the front. I think the optimisers are the best thing on this site, and I figured, why not show it right there on the homepage? So the second slide of the hero now runs one too, and what it's optimising itself into drawing the headline "Hi, I'm Ken". Here's how it works, and what it took to make it cheap enough to leave running on a homepage. Part of the series on how this site is built.

Roger Alsing's Mona Lisa came out in December 2008 and is one of the best demos of evolutionary computation. He took 50 semi-transparent polygons, changed one small thing at a time, kept the change only if the picture looked more like the painting than before, and after enough rounds of that he had something you'd recognise as da Vinci's.

Text, therefore, should be an easier target than a face. Letters are high contrast and built from straight-ish strokes, and people will read a word from a rough smear of ink in roughly the right place well before they notice it's made of shapes. So the slide has a canvas, and on the canvas are 78 translucent triangles being nudged toward the pixels of "Hi, I'm Ken."

The whole algorithm

There's no population, no crossover and no selection pressure to tune. It's one candidate and one mutation at a time:

for (var e = 0; e < EVALS_PER_FRAME; e++) {
    var cand = genome.map(cloneShape);
    mutate(cand, scale);
    var s2 = score(cand);
    if (s2 < best) {
        best = s2;
        genome = cand;
    }
}

The fitness function draws the candidate and compares it with the target, pixel by pixel:

function score(g) {
    paint(g, fit, FIT_W, FIT_H);
    var d = fit.getImageData(0, 0, FIT_W, FIT_H).data;
    var err = 0;
    for (var i = 0, p = 0; i < d.length; i += 4, p++) {
        var lum = (d[i] * 0.2126 + d[i + 1] * 0.7152 + d[i + 2] * 0.0722) / 255;
        var diff = lum * (d[i + 3] / 255) - target[p];
        err += diff * diff;
    }
    return err;
}

It's squared error against the brightness of the target, with the alpha folded in because the canvas sits transparent over a photograph, and lower is better. The target is the headline itself, drawn once into an offscreen canvas in the site's own font (Poppins).

Keeping it cheap

I score on a canvas of 176 by 48 pixels that is hidden from users. The canvas you see is up to 560 pixels wide on the page, which is 1,120 real pixels on a retina screen, and scoring at that size would mean reading back over 370,000 pixels per evaluation, 170 evaluations a frame. At 176 by 48 it's 8,448, and the letters are still readable at that size.

I didn't let colour evolve either. Each triangle carries one number that puts it somewhere between the two ends of the site's photo gradient, which keeps the result on palette and takes three dimensions out of the search.

Most mutations get rejected, and a rejected mutation changes nothing anyone can see, so the visible canvas is repainted at most once a frame and only if at least one of that frame's mutations was kept.

The hero is a carousel of five slides, so four times out of five this one isn't on screen, and while it's off screen the loop checks whether its slide is the active one, sees it isn't, and does nothing. Once 1,800 evaluations in a row have failed to improve anything the loop idles, and when the slide comes back round it starts again from a fresh set of triangles, so whoever's looking gets to watch the headline being built instead of arriving at a finished one. The carousel can be paused, too, to give it more time.

What a screen reader gets

I marked the canvas aria-hidden so screen readers skip it, and put the real heading next to it as visually hidden text:

<h2 class="sr-only">Hi, I'm Ken.</h2>
<canvas class="kr-hero-evolve" aria-hidden="true"></canvas>

The document outline doesn't change, a crawler sees a normal heading, and nobody has to sit through a description of an animation they can't see.

Was it worth it

It's one small file, and most of it is comments and setup. It swapped a static line of text for a working demo of evolutionary computation, the field my research is in, inside the first few seconds on the site. I'm normally against homepage animations that are only there to move, because they cost the reader something and give nothing back, but this one is a real optimiser running on your machine and you can read its fitness function further up this page.

If you want the version with sliders and a population and all the knobs, that's Evolution, Live.

A hundred and twenty-six years later, photography looks like a hobby for people with money. And, well, it largely is. If you go to any big sporting event and look at the journalist section you'll see a bunch of people with lenses about the size of my arm, and cost about 4x the price of my car. For one lens. Then if you look at discussion boards online, most of what I see is about gear: which body, which lens, which upgrade, what this year's model does that last year's didn't. It puts people off of a hobby that can actually be done with very little. You can take good photos on a camera that sells used for about a hundred dollars, or on the phone you already own, and edit them in software that costs nothing (except some time to learn).

Musicians have a name for it

Gear Acquisition Syndrome, or GAS, is "the musicians' unrelenting urge to buy and own instruments and equipment as an anticipated catalyst of creative energy and bringer of happiness". That's from Jan-Peter Herbst and Jonas Menze, who wrote a whole book about it in 2021. They also write: "The urge is directed by the belief that acquiring another instrument will make one a better player."^[3] With "camera" in place of "instrument", that describes a good chunk of photography YouTube and TikTok.

I've played guitar since I was about 11 and got into photography around 15. I get the urge to build up your gear library, who doesn't want new toys? It brings immediate satisfaction, fun, enjoyment, inspiration, while doing so with your existing gear is something of a chore before you actually enjoy it again. Plus - think of all the things new gear can do your current gear can't! But that is where creativity come into play, or sometimes, swallowing that urge and focusing on what your gear CAN do, putting new gear into the future. You'll enjoy your new gear even more with more skills and experience gained from your old gear.

My camera came out in 2010

Almost every photo in my gallery was shot on a Canon EOS 60D (though I think a couple linger from my old 350D, and a couple of phone shots are there, too). Canon announced it on 26 August 2010, an 18 megapixel APS-C DSLR with a screen that flips out, and it went on sale in the US at $1,099 for the body, or $1,399 with an 18-135mm kit lens.^[4][5] That was a lot of money in 2010. This week the used camera shop MPB lists 60D bodies from $97 to $214.^[6]

Canon has replaced it three times since, with the 70D, the 80D and then the 90D in August 2019, and Wikipedia now lists a mirrorless camera, the EOS R10, as the 90D's successor, and that's excluding the myriad of better series out there.^[7][8] At my old workplace, I had the opportunity to use a Canon EOS 5D Mark III. Here's a brief comparison:^[7][17]

Prices and weights are for the body alone, without a lens.

The gallery has ~500 photos in it, about a decade of self-taught landscapes, urban decay and pets, and I'm happy with them. I'm sure that a fancier camera and lens would have improved the shots, sure, but the grass is always greener. The 60D doesn't even have GPS, so I've been putting the photo map together by hand. It shoots 5.3 frames a second and tops out at ISO 6400 (12800 if you push it), which is plenty for a hill at sunset or a patient dog, and not what I'd take to a football match under floodlights (not that I'd go to a football match, anyhow).^[7] Most of a landscape photo is being in the right place when the light is good and pointing the thing in a sensible direction. The lenses I have are nothing fancy. I make use of a kit lens and a cheap telephoto lens I got for around $100. The only thing I need is more time, not gear!

The camera you already have

Rise Above Research, who somehow came up with a number, estimate that around 1.95 trillion photos will be taken worldwide in 2026, with smartphones accounting for the vast majority.^[9]

In 2009 the photographer Chase Jarvis published a book of pictures taken on his iPhone, a visual notebook of a year of his life, and called it The Best Camera Is The One That's With You.^[10]

Editing for nothing

Snapseed is Google's photo editor for phones. The App Store listing promises "30+ pro tools and filters" with "no subscriptions, purchases, ads, or watermarks".^[11] It edits raw files too: DNG files on Android, and on an iPhone or iPad raw files from a list of cameras that includes the 60D (and my old 350D).^[12] Opening a raw file goes straight to the Develop tool, with exposure, shadows, highlights, contrast, structure, temperature and tint.

On a computer there's darktable, which calls itself "an open source photography workflow application and raw developer", free under the GPL, with builds for Windows, macOS and Linux.^[13][14] Its editing is "fully non-destructive", so the original raw file is never changed and the full image is only worked out on export. Its camera support page lists the 60D with white balance presets and a noise profile.^[15] RawTherapee is the other big free one, also GPL, also on all three systems.^[16]

Sharing photos doesn't have to cost anything either. The full-size files for my gallery are hosted for free.

If you're starting out

This is what I'd tell a friend who wanted to get into photography and thought they couldn't afford it:

• Use your phone for a month first. Take far too many photos, delete most of them, as is tradition.
• Put Snapseed on it and edit the ones you keep.
• If you still want a camera with a viewfinder and lenses that come off, look at used bodies a few generations old, or borrow a friends. Check darktable's camera support page before you buy, so you know the raw files will open.

If this gets you out with an old camera or your phone, show me what you took in the comments, and tell me what it was shot on.

References

1. The Franklin Institute. Kodak Brownie camera. Collections (accessed September 2026). https://fi.edu/en/science-and-education/collection/kodak-brownie-camera
2. $1 in 1900 is worth $39.88 today. CPI Inflation Calculator, Official Data Foundation, based on US Bureau of Labor Statistics data (accessed September 2026). https://www.officialdata.org/us/inflation/1900?amount=1
3. Herbst, J.-P., & Menze, J. (2021). Gear Acquisition Syndrome: Consumption of instruments and technology in popular music. University of Huddersfield Press. https://doi.org/10.5920/GearAcquisition.fulltext
4. Canon EOS 60D DSLR announced and previewed. (2010, 26 August). DPReview. https://www.dpreview.com/articles/1692435631/canoneos60d/
5. Miller, R. (2010, 26 August). Canon EOS 60D: 18 megapixels and 1080p video flexes its articulating screen this September. Engadget. https://www.engadget.com/2010-08-26-canon-eos-60d-18-megapixels-and-1080p-video-flexes-its-articula.html
6. MPB. Used Canon EOS 60D (accessed 25 September 2026). https://www.mpb.com/en-us/product/canon-eos-60d
7. Canon EOS 60D. Wikipedia (accessed September 2026). https://en.wikipedia.org/wiki/Canon_EOS_60D
8. Canon EOS 90D. Wikipedia (accessed September 2026). https://en.wikipedia.org/wiki/Canon_EOS_90D
9. Rise Above Research. (2026, February). 2026 worldwide image capture forecast: 2025–2030. https://riseaboveresearch.com/rar-reports/2026-worldwide-image-capture-forecast-2025-2030/
10. Jarvis, C. (2009). The Best Camera Is The One That's With You: iPhone photography. New Riders. https://www.peachpit.com/store/best-camera-is-the-one-thats-with-you-iphone-photography-9780321684783
11. Google LLC. Snapseed. App Store (accessed September 2026). https://apps.apple.com/us/app/snapseed/id439438619
12. Google. Develop: RAW editing. Snapseed Help (accessed September 2026). https://support.google.com/snapseed/answer/6312515?hl=en
13. darktable. About (accessed September 2026). https://www.darktable.org/about/
14. darktable. Install (version 5.6.1, accessed September 2026). https://www.darktable.org/install/
15. darktable. Camera support (accessed September 2026). https://www.darktable.org/resources/camera-support/
16. RawTherapee (accessed September 2026). https://www.rawtherapee.com/
17. Canon EOS 5D Mark III. Wikipedia (accessed September 2026). https://en.wikipedia.org/wiki/Canon_EOS_5D_Mark_III
18. MPB. Used Canon EOS 5D Mark III (accessed 26 September 2026). https://www.mpb.com/en-us/product/canon-eos-5d-mark-iii

What a sketch is

Each one is a function, in essence. It gets a width, a height and a random number generator, and hands back two more functions: step, which moves things along, and draw, which paints. Here's the offline-reading one with some fat trimmed:

def('reading-this-site-offline', function (w, h, rnd) {
    var t = 0;   // frame counter; the engine advances this, not the browser's clock
    return {
        // one cycle is 300 frames: 90 online, then 210 offline, then round again
        step: function () { t = (t + 1) % 300; },
        draw: function (ctx) {
            var off = t > 90;
            // the wifi symbol: three arcs, faded out once "off"
            for (var i = 0; i < 3; i++) {
                ctx.strokeStyle = off ? alpha('#ffffff', 0.15) : C.ink;
                ctx.beginPath(); ctx.arc(w * 0.2, h * 0.5, 10 + i * 10, 3.6, 5.8); ctx.stroke();
            }
            // a little page, sliding left to right, looping every 100 frames
            var k = phase(t * 1.5, 100);
            rect(ctx, w * 0.62 + w * 0.22 * k, h * 0.42, 14, 18, C.b);
            // the caption underneath, swapping with the wifi state
            text(ctx, off ? 'offline, still reading' : 'online', w * 0.5, h * 0.9, 9, off ? C.b : C.dim, 700, 'center');
        }
    };
});

The engine

First, the engine finds the cards. Cards on this site get rendered by five or six different scripts at different moments (the blog listing, the series pages, the related posts under each article, the homepage), and my first version made each of those scripts call the engine once it had finished rendering. Now the engine watches the page for anything being added and hooks up any card it has a sketch for.

Second, it steps sixty times a second whatever the screen is doing. The sketches count frames, and my first version stepped once per requestAnimationFrame, which is fine on a 60 Hz monitor but silly on a 120 Hz one: every ant ran at double speed, every tide went out in half the time. In a throttled background tab they crawled instead. So the engine keeps its own clock now and steps once per sixtieth of a second of real time, with a cap so a tab that's been hidden for a minute doesn't try to catch up all in one go.

Third, it doesn't load until it's needed. The three files come to 41 KB compressed, which is a lot to send a phone for a hover effect it can't trigger (no mouse, no hover, nothing to do). So no page includes them any more. The first time a mouse moves, and only if the browser says hovering is a thing it can do and you haven't asked for reduced motion, the shared script fetches all three. On a desktop they turn up about a second before you could want them, which is close enough for me, and phones don't store them at all.

Keeping them from rotting

There are two checks. A commit gets refused if any post in the listing has no sketch, so I can't publish without drawing one, to make sure I don't forget (something I do frequently without lots of checkpoints for myself!). And the test suite builds every sketch, runs it for 120 frames, and fails if it throws or the canvas is still blank at the end.

If you're reading this on a phone you've seen none of it, and that's fine. Nothing to hover with, nothing to download, and the cards are just cards.

Nine posts in, every search strategy in this series has handed you its dials and waited:

• the ant colony wanted an evaporation rate
• the genetic algorithm and its program-breeding cousin wanted mutation rates and population sizes
• simulated annealing put you in charge of the thermostat
• the swarm asked for inertia, own memory and peer pressure
• variable neighbourhood search needed to know how many neighbourhoods to shake through
• CMA-ES adapted its own step size, but still took a population and a starting sigma from you
• differential evolution asked for F and CR
• tabu search, one post ago, wanted its tenure set

Hyper heuristics are inherently meta. It searches for the right way to search, optimizing by choosing the best optimizer.

It's a heuristic for choosing heuristics, and the field calls it a hyper-heuristic.

The problem this demo is chewing on is exam timetabling, which is very close to what I solved during my studies. My PhD went on rostering BT's engineers, and exam timetabling is the same family of problem with students in place of shifts: easy to state, brutal to solve, and impossible to explain to friends and family what makes it so difficult.

Twenty exams, seven slots, three rooms

Any of us who have been to college / university have likely experienced the hellish end of term exam period. Most students stick to specific degree programs, so if you're a 'generic' student, your exams will likely be spaced fairly evenly. If you, however, are doing a triple major of sports, theoretical physics and nursing, you are the most likely to be screwed over, as screwing up people with that specific triple-threat (just you) is less of a failure than messing up everyone who just took generic psychology classes exams.

The instance below is twenty exams, labelled A to T, to be placed into seven timeslots with three rooms available per slot. A seeded random conflict graph decides which pairs of exams share students. From that, one number to shrink. Two conflicting exams in the same slot is a hard constraint violation and costs '100', because somewhere a student is expected to be in two rooms at once.

Conflicting exams in adjacent slots cost 1 to 3 (scaled by how many students they share), because back-to-back exams are legal, but far from ideal. And a fourth exam squeezed into a three-room slot costs 10 per extra exam. That weighting is the standard trick for mixing hard and soft constraints into one objective.

A timetable is not usable until the hard count reaches zero. Everything after that is how an "acceptable" timetables progresses towards "ok", and hopefully, "good", or even better, "optimal".

The manager and the workers

Four low-level heuristics do the standard heuristic solving business. H1 nudge moves one random exam to a random slot. H2 swap picks two exams and trades their slots. H3 worst-to-best finds the exam contributing the most penalty and moves it to the slot where it does the least damage. H4 ruin kicks three random exams to random slots, a small controlled demolition.

Above them sits the manager - it keeps a score for each worker, rewards a worker whenever its move is accepted and improves the cost, and lets every score decay over time. Even the manager's least favourite heuristic keeps getting the odd chance, because a heuristic useless so far might be useful later down the line when the circumstances change.

Cowling, Kendall and Soubeiga used the same pattern to schedule a sales summit^[2], in the paper that named the modern field.

Intentionally Blind

The manager can't see the timetable. It doesn't know what an exam is. It has no idea students exist. It just has four levers to pull and keeps records of what is working well just now, what isn't, and keeps chugging.

All it ever observes is "I pulled lever three and the cost went down by 40" or "I pulled lever one and the move was rejected". That blindness is called the domain barrier, and it sounds like a handicap but it's necessary.

Because the manager only ever hears cost deltas, you can unplug the four timetabling workers, plug in "move a nurse to another shift" and "swap two delivery routes", and the same manager, unchanged, starts learning which of those levers earns its keep. The domain knowledge lives entirely in the cheap, disposable workers. Burke (who was briefly my supervisor at Stirling) and colleagues' survey^[3] maps how far that one trick works.

Great Deluge: the tide going out

The manager decides which move to try. Something still has to decide which results to keep: Gunter Dueck's Great Deluge, published in 1993^[1] does the trick.

Dueck told it as a story about rain - it rains without end, the water rises, and a hiker who wants to end up on a high peak just has one goal: keep their feet dry!

For minimisation, just turn it upside down: cost is depth, and the search is a fish that must stay submerged. The tide goes out, the pool drains, and the fish is herded into the deepest hollow of the reef it can still reach, or mermaids, or whatever, it's a metaphor you get the idea.

Compare the two acceptance rules. Simulated annealing accepts a worsening move with probability exp(−Δ/T), which means choosing a starting temperature, a cooling schedule, and possibly reheats: a small family of dials, as the demo showed. Great Deluge accepts any move that lands at or below the water level. Uphill, downhill, sideways: if you are under the surface, woohoo.

Watch the manager learn

Watch the probability bars first. That row is the manager's opinion of its workers, updating live. Early on, H3 hoovers up credit, because on a messy random timetable the biggest offender is obvious. Then the easy repairs run out, H3 stalls, decay eats its lead, and you can watch H1 and H4 claw back probability as disruption starts earning acceptances that greedy repair cannot find.

Then watch the flood chart. The shaded water is the accepted region, the line is the cost swimming inside it, and whenever the tide presses down on the line, the level is doing the forcing.

The tide, too fast and too slow

Drain the water too fast, by pushing the drop rate to maximum, and the level crashes below any cost the search can actually reach. From then on nothing passes the test at all: the accepted-moves stat falls to zero while the hard-conflict count sits stranded above it, and after 600 rejections in a row the demo gives up and says so. The search hasn't converged. The tide stranded it partway down the reef, which is the deluge's version of quenching hot metal too quickly.

Drain it too slowly and the level stays miles above the cost, the rule accepts almost everything, and the search wanders: plenty of motion, nothing pushing the cost down. The chart shows a cost line bouncing around far below a distant water surface but going nowhere in particular. All the forcing comes from the tide.

Then turn learning to zero. The manager stops keeping score and picks workers uniformly at random: a committee with no memory. Restart a few runs at learning zero, then a few at full learning, and compare how long the hard-conflict counter takes to hit zero.

The ladder up

There is a tradition in this series of me telling you that some dial matters enormously and then making you tune it by hand: mutation rates, evaporation rates, cooling schedules, F and CR, tabu tenures. CMA-ES staged the first escape from that tradition, adapting its own search distribution as it ran; a hyper-heuristic takes the idea to its limit and moves the whole choice of which heuristic to use inside the run.

It doesn't remove the design decisions: I still chose the workers, the reward, the decay and the floor. Turtles, all the way up.

References

1. Dueck, G. (1993). New Optimization Heuristics: The Great Deluge Algorithm and the Record-to-Record Travel. Journal of Computational Physics, 104(1), 86–92. https://doi.org/10.1006/jcph.1993.1010
2. Cowling, P., Kendall, G., & Soubeiga, E. (2001). A Hyperheuristic Approach to Scheduling a Sales Summit. In Practice and Theory of Automated Timetabling III (PATAT 2000), LNCS 2079, 176–190. https://doi.org/10.1007/3-540-44629-X_11
3. Burke, E. K., Gendreau, M., Hyde, M., Kendall, G., Ochoa, G., Özcan, E., & Qu, R. (2013). Hyper-heuristics: a survey of the state of the art. Journal of the Operational Research Society, 64(12), 1695–1724. https://doi.org/10.1057/jors.2013.71

Street photography stirs some interesting questions. It's legal in most public places (though not in the UAE or South Korea, and France is far stricter than visitors expect), but there are lots of legal ways you can be an asshole.

Legality

In most jurisdictions, including the UK and US, photographing people in public spaces is broadly legal. There's a reduced expectation of privacy in public, and the whole documentary tradition depends on that freedom.

"Broadly legal" covers three quite different rules, though, and which one you are under depends on where your feet are. Wikimedia Commons maintains a country-by-country list of consent requirements for pictures of identifiable people, because it has to: it hosts the photographs. Of the 52 jurisdictions it documents, the split runs like this.

Counts from the Wikimedia Commons country-specific consent requirements guideline. It documents 52 jurisdictions out of roughly 195, and says plainly that a country's absence from the list proves nothing: the UAE is not on it, and criminalises photographing a person without consent under Article 44 of Federal Decree-Law 34 of 2021. ^*Iran sits in the top row on one narrow 2007 statute about women-only spaces and private gatherings, which shows how much work "with exceptions" is doing in every row above. None of this is legal advice.

That says nothing about the ethics or social implications, of course. Cultures vary from town to town, never mind across country borders. But legal is the floor, and plenty of legal things are unkind, so treating the law as the whole of the ethics is unwise. The question I care about is "should I, and how", and the law can perhaps give insight into some broad strokes of social acceptance, but not ethics.

The context has changed since the old masters. When Cartier-Bresson or Vivian Maier shot a stranger, that photograph lived in a print, a gallery, a book, or perhaps in a dusty old film cartridge in a chest somewhere. A bounded, slow world.

A photograph now can be uploaded, geolocated, reverse-image-searched and attached to a name in minutes. The same click that once made a fleeting image now makes a searchable, permanent, findable record of a person who never agreed to be in it. So the ethics can't be inherited wholesale from a pre-internet age. And that cuts both ways: we live in the era of Flock cameras, smartphones, and TikTok, Twitch, Facebook and YouTube livestreamers.

A working framework

We can also consider paparazzi, as the worst kind of "well, it's legal" defence. Sure, it's legal, but camping outside someone's home, place of work or school to take non-consensual photos, hoping to make a dollar selling them to some awful supermarket rag, is still a choice you made (who needs AI slop when you can peruse human created slop?).

I think there's something on the other side of this that I haven't heard discussed: photography paralysis. Being so afraid of the first that you never take a plain photograph of human life at all. That's a real loss, because people are some of the most important things we can photograph. Someone just living their life, representing a common interest, feeling and life can echo something shared, capture beautiful moments and collective sadness. There is art in people being people, and capturing a fleeting moment forever. Not capturing that is almost like destroying a beautiful painting or poem.

How I draw my bounds

The dignity test. Before I publish (and often before I shoot), I ask whether the image honours the person or exploits them. A photograph that shows a stranger as interesting, dignified, fully human is a different object from one that invites the viewer to laugh at them, or gawk at them, or one that is really just virtue signalling from me.

If the picture's whole appeal is that the subject is a spectacle, I'll have basically made a zoo and considered that person an animal.

The power gradient. This is the one that does the most work for me. Photograph across or up the power gradient, rarely down it.

A candid of a confident person striding through their city is wonderful. A candid of someone at their most vulnerable (homeless, grieving, drunk, struggling) turns their worst moment into my content. It's the same worry I had about ruin porn: a beautiful photograph of someone's suffering is still a photograph of suffering, and making it beautiful doesn't mean you're not personally benefitting from another's suffering. That doesn't mean you can't take photos of these things, but you need to be more creative in your photos. Avoiding faces or other identifying marks (tattoos, nametags) means you can show, for example, homelessness in Las Vegas, without it being exploitation. It can even be powerful, eye-opening, educational and a rallying cry.

The ask-after option. You don't always have to choose between a stolen candid and a stiff posed shot. Shoot first, then approach, show them the frame, offer to delete it or to send it. It turns a taking into something closer to a collaboration, and people are delighted more often than not in my experience. It doesn't cost the "purity" of the unaware candid but it does buy a clear conscience and, often, can make someone's day (and maybe get you an extra Instagram follower if that's something you care about).

Consider obvious boundaries. Is the person you want to photograph on their own property, e.g. through a window or somewhere else they would normally expect privacy? Or are you in their home with them - if so, consider asking - taking photos inside someone's home can feel like an invasion of privacy to them.

Use the discomfort

Documentary photography of any kind requires you to have an opinion about your subject, not merely a technically good composition. The discomfort I feel raising a camera at a stranger keeps me asking the dignity question and checking the power gradient, so don't squash it down entirely, it can be informative, or at least help you check yourself.

So I don't want to talk myself out of the hesitation, and I don't want you to either. The best street photography was made by people who clearly loved their subjects, and you can see it in the frames: the humans in them are seen, not caught.

Keep the hesitation. Ask the dignity question, check the gradient, leave other people's kids out of it, and take the photograph when the answers come out right. So, Wheaton's Law.

Some of the frames

I started my Ph.D. in 2015 under Dr. Jingpeng Li, a kindly and intelligent man who was very patient with me going down the wrong roads and floundering over fairly fundamental and basic math. He had some interesting ideas on how to modify existing algorithms to improve their capabilities, and let me toy with them to begin to understand some of that fundamental math, as well as get a feel for how search algorithms, at their core, actually function.

Most search algorithms take a working solution, change one thing, and keep the change if it helped. That works until the solution is tangled, at which point every single change makes things worse and the search sits there, surrounded by improvements it cannot reach because each one requires three simultaneous moves.

Ruin and recreate knocks a hole in the solution and rebuilds the solution. This is the method behind a rostering paper I wrote in 2018^[1] with colleagues at Stirling, York and BT, using a variant of Jingpeng's own algorithm: Evolutionary Ruin and Stochastic Recreate^[2], where the wrecking ball is aimed by fitness rather than swung blindly. The demo below is that method on a simple roster. Part of the series on research, live.

The demo

Each row is a person, each column a day. Cells are the shift they are working. The rows under the grid are the demand check: for each day and shift, how many staff over or under the requirement, so -2 on the night row means two short and +1 means one spare. A working cell carries the same figure when its shift is out of balance, which marks the allocations that are surplus and the ones holding up a short day. The panel under the stats scores each constraint separately: the two hard ones, which have to hold, and the two soft ones the last five per cent of the fitness is spent on.

One step is one full ruin and recreate, under the same simulated annealing control the paper uses. The ruined cells flash before they are refilled, so you can see the size and shape of the hole. The default strategy is the paper's: score every allocation, then demolish the probabilistically weakest. Switch to the blind strategies to see what the aiming is worth: scattered cells, a person's whole fortnight, or everybody's Tuesday.

Why breaking it on purpose works

A roster is full of constraints that interlock. Moving one person off Tuesday night leaves Tuesday night short, so the move is rejected, so the person stays. Every individual change is bad, and yet the roster as a whole is nowhere near as good as it could be. That is a local optimum with high walls (you try to climb them and only fall right back down, you need to jump out of there entirely).

Removing thirty assignments at once does not have to be an improvement, because the recreate step is not obliged to put them back where they were. It rebuilds greedily against the fitness function, which means it can discover an arrangement that no sequence of single moves would have reached, since each intermediate step in that sequence would have been rejected.

You can see this in action in the demo: small ruins converge smoothly and stall early. Large ruins jump around and take longer per step, since there is more to rebuild. The paper's answer is this: if you know which allocations are weak, a small, well-aimed ruin can do the work of a large blind one.

Aim the wrecking ball, or swing it harder

I measured all four strategies on this instance. The score is the median number of steps to first reach a fitness of 99.5, over five seeds, with a 200-step cap; "never" means the median run did not get there at all. Lower is faster:

The fitness-guided ruin is fastest when the holes are small, eight or ten steps with a demolition of only eight cells, and gets no benefit from swinging harder, because it was never relying on volume in the first place. The blind strategies are the mirror image: they need size. Scattered cells come good from sixteen cells up; ruining everybody's Tuesday is the single fastest setting in the table at eighty cells, seven whole columns of the roster at a time.

Whole-people ruin at small sizes is the table's disaster zone, and the reason generalises. Eight or sixteen cells is one person's fortnight: freeing a single row while the other eleven stay fixed leaves the recreate step boxed in by everything it did not touch. Structured damage only pays when the structure being suspended is big enough to rearrange.

So the destroy operator and its size are not two parameters, they are one decision: aim, or swing big. The paper's selection operator decides, and its reward is reaching the same quality with a fraction of the demolition, which matters when every ruined cell costs rebuild time. Blind volume gets there too, eventually, by suspending so many constraints at once that the rebuild has room to rearrange everything.

What the demo is not

The real problem had a great deal more to it: engineers with different skills and a first-skill preference, shift rules dictating lengths, start-time windows and shift counts, alternating week patterns, UK working-time law, and two consecutive rest days per week as a soft target. The instances ran to around 200 employees, drawn from a domain of roughly 25,000 field engineers. The fitness function works with eight hard constraints and four soft ones, weighted 95 to 5.

What survives compression is the shape of the method, which is the part that transfers. Score it, break the weak parts, rebuild, and let the temperature decide what to keep.

References

1. Reid, K. N., Li, J., Veerapen, N., Swan, J., McCormick, A., Kern, M., & Owusu, G. (2018). Shift Scheduling and Employee Rostering: An Evolutionary Ruin & Stochastic Recreate Solution. In 2018 10th Computer Science and Electronic Engineering (CEEC), 19–23. IEEE. Google Scholar
2. Li, J., Bai, R., Shen, Y., & Qu, R. (2015). Search with evolutionary ruin and stochastic rebuild: A theoretic framework and a case study on exam timetabling. European Journal of Operational Research, 242(3), 798–806.
3. Schrimpf, G., Schneider, J., Stamm-Wilbrandt, H., & Dueck, G. (2000). Record Breaking Optimization Results Using the Ruin and Recreate Principle. Journal of Computational Physics, 159(2), 139–171. https://doi.org/10.1006/jcph.1999.6413

I'm doing a master's in organizational leadership, and one of the first things I've learned is that leadership is not a status only found in specific roles, it's something we all do every day. Leading projects, leading people, making decisions on next steps, risk assessments, and practicing your social skills. One such example I have been thinking much about recently is providing feedback - generally, but also when you give the same feedback multiple times to someone. You find they nod, say it was fair, thank you for being direct, and then do the thing again the following week.

It's something perplexing: you are specific in the feedback given, offer help, and mean it. You write the steps down, in order, in an email they could find again. You ask if there was anything in the way, and nothing changes.

As the saying goes: you can lead a horse to water. But that hardly solves the problem, it just describes it. How many times must you take the horse to water? Should you fill up a bowl and raise it to its mouth? A PowerPoint on "drinking water"? Should you bring in a horse expert to assist?

The proverb

It is the oldest English proverb still in daily use, recorded as early as 1175 in the Old English Homilies:^[1]

Hwa is thet mei thet hors wettrien the him self nule drinken?

— Old English Homilies, c. 1175 (who can give water to the horse that will not drink of its own accord?)

John Heywood reshaped it into the version everyone knows by 1546. This is not a new problem among people.

The drinking is not yours to do, and no amount of leading better or putting in more effort as a leader will make a horse drink. It can be deeply irritating when the horse is now standing in the river up to its knees, you're gesturing at the water, and it just stares at you.

The version I like adds a third clause. Charlie "Tremendous" Jones, an American insurance-agency-executive-turned-motivational-speaker best known for the bestseller Life Is Tremendous, used to put it this way:

You may not be able to make a horse drink, but you can put salt in his oats and make him thirsty.

— Charlie "Tremendous" Jones

You cannot make the horse drink, but you CAN make it thirsty, you can check the water is fit to drink, and you can find out whether the animal in front of you is a horse that isn't thirsty or a horse that is ill.

First, check the water

Before anything else: was the feedback actually constructive? In Kluger and DeNisi's 1996 meta-analysis, which pooled 607 effect sizes from feedback studies, feedback improved performance on average, but in over a third of cases it made performance worse.^[2] The feedback that hurt was the kind that pulled attention towards the self (who you are, how you compare) rather than the task (what to do differently, next time, at this step). No one wants to hear what is wrong with their character, but many are happy to hear how they can improve the technique, the strategy or the steps.

It's always satisfying when a scientific study rings true. I know that I have this problem when I receive feedback: I feel attacked, if it's about my underlying drive or passion, but when I am given feedback on specific actions or tasks, I feel motivated. So, now I try to do the following when giving other people feedback: can they repeat back, in their own words, what they are meant to do differently, by when, and how they will know they've done it? If the answer to any of the three is a shrug, then I did not lead a horse to water, I led a horse to a description of water. "Be more proactive" is a high-level descriptor and nowhere near specific enough.

Second, check whether the horse is thirsty

The second question is whether the person thinks there is anything to fix. Prochaska and DiClemente's stages of change model, built on decades of watching people quit smoking and fail to, puts a stage before "thinking about it": precontemplation, where the person does not see a problem at all.^[3] Advice delivered to someone in precontemplation is like telling someone it will eventually rain in a city fifty miles away, at least to them. The smoking example also rings true for me: I remember begging my parents to stop smoking, even threatening to start myself if they didn't stop. Ultimately, a person needs to want to change a habit, otherwise any well-meaning, well-informed, kind and considered advice is utterly for naught.

"Yeah, fair, thanks for saying" is what a person says when they agree and intend to act, and it is also what they say to make the meeting end while being polite. This means we can't really settle on that as the finale to the feedback conversation. Klein and colleagues' meta-analysis of goal commitment says: goals predict performance when people are committed to them, and barely at all when they are not, and commitment is not the same as having heard the goal read aloud.^[4]

Worse, pushing harder at this stage makes it worse. Miller and Rollnick, the motivational interviewing people, call the helper's urge to fix the "righting reflex," and their central finding, replicated across addiction, health and now management, is that when you argue for change, the other person starts arguing against it, and hearing themselves argue against it makes them believe it.^[5] Brehm named the underlying push-back in 1966: reactance, the instinct to reclaim a freedom someone is trying to take.^[6] You can't tell a horse it's thirsty and expect it to believe you.

Third, check whether the horse can drink

Someone agreeing and then not doing is what inspired me to write this, and it is usually read as a will problem when it is often a capability or environment problem. Landsberg's skill/will matrix is a two-by-two from a coaching book.^[7] Low skill needs teaching while low will needs reasoning, and low both needs direction. Getting these wrong is like trying to inspire a fish to walk.

How to know if it's a "will" problem:

• Have I seen them do it once? If I have never watched the person do the thing, even badly, I do not know whether they can. "Write clearer tickets" assumes they know what a clear ticket looks like. Show them one. Then watch them write one.
• What is paying them to do the opposite? If they are incentivized or otherwise follow a protocol established by a higher authority or by tradition, they may ignore your feedback.
• Is there room? Asking someone drowning in tickets to also improve how they write tickets is asking for the improvement to be ignored.

Salting the oats

Given a horse that is capable, not obviously ill, and not thirsty, here are some of my thoughts and learnings compiled together:

1. Listen to them, don't assume. Ask them to say what they'll do, and write down what they said, not what you wanted them to say. A plan in someone else's words is often very different from what you think they know. Locke and Latham's goal-setting work has been saying this for forty years: participation in setting the goal raises commitment to it.^[8]
2. One behaviour, one fortnight, one checkpoint. "Be a better communicator" is useless. "Send the summary before the meeting, for the next three meetings, and we'll look at it on the 20th" is tangible, measurable and clear - the outcome is "becoming a better communicator" but the steps are clearly stated.
3. If they don't see shortcomings they don't see a problem to fix. If the report is late and you fix it for them at eleven at night, the report is never late from their standpoint.
4. Changing a person's behaviour is difficult. Changing a template, a checklist, a pairing session, a definition of done, is straightforward and easy.
5. "This is the third time we've had this conversation. What's going on?" and let silence reign until an answer comes. Listening is important.
6. "We agreed X by the 20th." A dated, agreed expectation turns the fourth conversation from an argument about memory into a paper trail.

How far to go

A failure of leadership is waiting too long. When behaviours become patterns that have been that way for years, you are fighting both the problem and established tradition. You can always try one more thing, one more framing, one more month. So the leader who doesn't decide in advance ends up deciding by exhaustion, which is late, and usually unkind to the team, to yourself, and to your colleague.

Set the budget before the third conversation. What decides it is not how much you like the person (something I know we all struggle with). It's the cost of letting the issue stand:

• You. The hours, obviously, but also the attention. A stuck person takes up a share of a leader's head that is wildly out of proportion to their share of the team.
• The others. Every hour spent on the horse that won't is an hour taken from the horses that would.
• The standard. What you tolerate for long enough becomes the definition of acceptable. And that can also spread: "Well, Jon takes 3 days on these tasks, why should I deprioritize it too?"
• The person. Every month they spend in a role they are not going to grow into is a month they are not spending somewhere they might. Endless patience inhibits their growth, prevents them from achieving their goals (and possibly stymieing their future income, prospects, benefits, etc.).

Then, how do you read which kind of stuck you're looking at?

1. They try, and fail. They attempt the thing, it comes out wrong, they attempt it again. This is a skill gap and it is - believe it or not - the good kind of stuck. Keep going, teach more concretely, lengthen the horizon. At least the horse is trying to drink!
2. They agree, and don't. Nod, thanks, nothing. This is will or environment, and the response is a single change of approach, followed, if nothing shifts, by the written expectation with a date on it.
3. They dispute the premise. "I don't think that's a problem." Either they're in precontemplation, or you are wrong. Stop advising and focus on listening. Then go and check with somebody who has no stake in it whether the water you're offering is actually water.

When to give up

Giving up on an approach is not giving up on a person. Most of what I've described above is giving up: on explanation, on the plan in your words, on absorbing the consequence. Surrendering is often surrendering the current strategy for a better one.

Giving up on the person is very different. Informal feedback, then explicit written expectation, then a formal process, then parting. The leader's obligation at each rung is the same: the person must never be surprised. If someone learns at a formal meeting that their job is at risk and this is news to them, the leader failed. And if you're not their leader but are providing feedback, sometimes giving up is giving up your part in the process to someone else - their leader, their mentor, the person themself.

I hate this part.

I'd rather run the sixth conversation than the formal one, and I think it's something many people avoid out of feelings of not wanting to be the bad guy. Ultimately, you can only control that which is your own action, not the outcome. Some wise words by an idol of mine, a physically crippled man who escaped slavery and learned much about our control over our personal destinies:

The things in our control are opinion, pursuit, desire, aversion, and, in a word, whatever are our own actions. The things not in our control are body, property, reputation, command, and, in one word, whatever are not our own actions.

— Epictetus, Enchiridion

You can lead the horse to water, you can make sure the water is clean, and you can show it how to drink, but no matter how much you want to, out of compassion, kindness or care, there is a point where you have to accept that you are not the horse and you cannot force it to drink. The servant leadership test is whether the people you lead grow. If they are not growing with you after a fair run, the kind act may be to help them find where they will, perhaps a sister team, or through a friend you know from college. I personally believe leaders should help colleagues beyond their current role and company, and encourage growth, because paths can cross many times, so help them even if not from inside your own team or organization. And remember, this cannot be a surprise, this is only after a long period of trying and goal setting.

A flow chart

References

1. Old English Homilies, c. 1175. John Heywood, A Dialogue Conteinyng the Nomber in Effect of All the Prouerbes in the Englishe Tongue (1546). Discussed at phrases.org.uk
2. Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284. 607 effect sizes; mean d = 0.41; over one third of interventions reduced performance.
3. Prochaska, J. O., & DiClemente, C. C. (1983). Stages and processes of self-change of smoking: Toward an integrative model of change. Journal of Consulting and Clinical Psychology, 51(3), 390–395.
4. Klein, H. J., Wesson, M. J., Hollenbeck, J. R., & Alge, B. J. (1999). Goal commitment and the goal-setting process: Conceptual clarification and empirical synthesis. Journal of Applied Psychology, 84(6), 885–896. Meta-analysis of 83 samples. pubmed.ncbi.nlm.nih.gov
5. Miller, W. R., & Rollnick, S. (2013). Motivational Interviewing: Helping People Change (3rd ed.). Guilford Press. The righting reflex, ambivalence, and why arguing for change produces arguments against it. Summary at ncbi.nlm.nih.gov
6. Brehm, J. W. (1966). A Theory of Psychological Reactance. Academic Press. Reviewed fifty years on in Steindl et al. (2015), Zeitschrift für Psychologie, 223(4), 205–214.
7. Landsberg, M. (1996). The Tao of Coaching. HarperCollins. Source of the skill/will matrix (direct, guide, excite, delegate).
8. Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705–717.

The keyboard path

The first element injected into every page is what is known as a skip link. Press Tab on any page here and "Skip to main content" appears before anything else. In short, this allows you to press Enter and get past any navigation fluff at the top of the page. It's invisible to mouse users, and is an old trick that is often forgotten in 'modern' websites.

The Ctrl+K command palette got the full dialog treatment: focus moves into the input when it opens, Tab is trapped (it moves the selection instead of escaping the dialog), Escape closes it, and results are given to screen readers through an aria-live region. The filter buttons on the blog and gallery use aria-pressed, so "Photography, toggle button, pressed" comes through instead of a mystery link.

The theme toggle swaps its label between "Switch to light mode" and "Switch to dark mode", so the button always says what it will do, not what it did. It has no aria-pressed, on purpose: a label that changes plus a pressed state gets read out as "Switch to light mode, toggle button, pressed", which leaves you to work out whether pressed means light or dark.

The screen reader path

Links that open a new tab get a screen-reader-only annotation, WCAG technique G201, applied automatically:

// Appends screen-reader-only "(opens in new tab)" to target="_blank" links
var span = document.createElement('span');
span.className = 'sr-only';
span.textContent = ' (opens in new tab)';
link.appendChild(span);

A MutationObserver reapplies it to links injected later (related posts, footers), because on this site half the content arrives after first paint. For a screen reader user, a link that teleports you into a new tab with no warning is awful.

Images carry real alternative text - I imagine not having it feels almost painful for screen-reader users, they likely feel they're missing out on important content and don't know what it is, like missing pages in a book. I aim for a description, not caption: the photo of my dog is "A picture of my puppy, Brutus", and the chart images in data posts describe the actual bars and numbers, so the information is actually useful, instead of "a photo" or "the results shown as an image". The interactive posts give every canvas role="img" and an aria-label saying what the simulation shows. A label is a thin substitute for "here is a live algorithm race you cannot see", so I'm open to suggestions on how to make it a better experience.

The vestibular path

Every script that moves something checks prefers-reduced-motion (eight of them at the last count), and the stylesheet has 28 media blocks for it. The section headings that rise into place as you scroll are pure CSS (animation-timeline: view()), and with reduced motion on they just appear, already visible, as does the script fallback for browsers without it. The homepage stats display their final number instead of counting up. The interactive demos mount paused, the carousels start paused, the hover sketches on the blog cards never load, the hero's evolving headline is drawn as plain type, and the theme switch cross-fades instead of washing across the page from the button. The site has to be complete with the animation subtracted, and only then is the animation allowed.

Checkpoints

Habits decay, so the audit script checks every page for the a11y basics (alt text, heading structure, link health, metadata) on every push, a contrast test checks every text colour and the focus ring against WCAG AA in both themes, and a headless browser presses Tab for real to make sure the focus ring shows up, measures the contrast on the rendered pages, and checks the demos work from a keyboard without flooding a screen reader. CI fails loudly when a new post shows how bad my memory is at maintaining these standards without a checklist. I love automating stuff, I'd rather put in extra effort early to save effort later.

I'd love to hear from an actual screen reader user. If you use assistive tech and something here fights you, the contact page works, and I fix reported things fast.

You judge how likely a thing is by how easily examples come to mind. The examples that come easily are the vivid ones, the ones the news showed you. That, in a nutshell, is the availability heuristic, and it runs your sense of risk, your politics, and a fair amount of your fear.

The shortcut

The availability heuristic, named by Tversky and Kahneman, is the brain's habit of estimating how frequent or likely something is by how readily instances spring to mind^[1]. Most of the time it's a decent shortcut: common things usually are easier to recall than rare ones, so "easy to remember" tracks "actually frequent" well enough to get you through a Tuesday.

It fails when ease of recall is driven by something other than frequency. In the modern world that is constant: vividness, emotional intensity and media coverage all make an event easier to recall without making it any more common.

So the events that dominate our sense of risk are the rare, dramatic, well-covered ones. Plane crashes get wall-to-wall coverage, and car crashes, which kill vastly more people, are too routine to report, so flying feels more dangerous while being far safer per mile.

The risks with no pictures

Shark attacks get a summer of headlines out of a handful of incidents, while the far deadlier risks (heart disease, the drive to the beach) produce no images at all. Terrorism reshapes whole societies' fears and budgets while killing a tiny fraction of the number who die of the mundane, unphotogenic causes nobody panics about. That last one makes me angry.

The mismatch between what we fear and what actually harms us is, in large part, the availability heuristic reading the news as if it were a frequency table^[2].

The news is an availability machine

The news isn't lying, which is the trap. News reports the unusual by definition: "plane lands safely" is not a story, "plane crashes" is. That newsworthiness filter guarantees that what reaches you is the rare and dramatic rather than the common and deadly.

So the news stocks your memory with the vivid, atypical events that the availability heuristic will then treat as frequent ones. You get a mental sample of the world selected for drama and against representativeness, and your sense of risk is built on that sample.

This is a cousin of survivorship bias, and of the habit of analysing whatever data happened to be lying around: in all of them, a filtered sample gets mistaken for the whole picture.

Then there is doomscrolling. An infinite feed that tailors towards what you enjoy - the algorithm literally learns based on how quickly you scroll, or don't, from one item to the next, whether you comment, react, share. So when you show an interest in, well, interesting things that are, say, unusual, such as landslides, violence, politics, terrorism videos, whatever - you begin to see MORE of that because the algorithm learns what you interact with. Therefore, you start to think "this is how things are".

The same wiring that helped an ancestor remember which mushroom caused a friend to die, or where there were dangerous animals, is now allowing you to identify dangerous patterns in a carefully crafted information feed, fed to you from your phone.

What to do about a bias you can't turn off

You can't just "stop" this by knowing it exists, but you can install a correction: when you notice fear, or a judgement about risk, ask where the example came from. Did this feel likely because you have base-rate data, or because a headline handed you a picture? That one question separates "this is common" from "this is merely memorable".

The boring denominators

The deeper practice is going after base rates, the boring denominators, instead of anecdotes. How many flights per crash? How many people drive, versus how many die driving? What's the trend, rather than the latest incident?

Numbers are the antidote because they're unmoved by vividness. A statistic does not care how frightening your mental image is (notice how often journalists say "thousands" without the "out of millions"). So the frightening image is your cue to go and find the statistic.

Your intuition has been fed a curated diet of the dramatic, so when it hands you a confident sense of how risky the world is, treat the confidence as a symptom rather than a measurement.

Next time a headline frightens you, go and find the denominator. It's usually two searches away, and nowhere near as interesting as the headline.

References

1. Tversky, A., & Kahneman, D. (1973). Availability: A heuristic for judging frequency and probability. Cognitive Psychology, 5(2), 207–232. https://doi.org/10.1016/0010-0285(73)90033-9
2. Slovic, P. (1987). Perception of risk. Science, 236(4799), 280–285. https://doi.org/10.1126/science.3563507

There are dozens of named cognitive biases, and this is the start of a series on them - I find them fascinating and I'm using this series as a means to learn about some psychology 101, and share what I've learned. One such bias sits under the others and is something of a basis for others. Confirmation bias is the tendency to seek, notice, remember and believe evidence that supports what we already think, and to ignore, discount or forget evidence that doesn't. It's how your uncle can keep bringing up the same political arguments at Thanksgiving dinner, forgetting what you've already said that clearly counters his argument - he won't remember, because the information you provide is irrelevant to him.

Evidence

The comforting model of the mind is a courtroom: you gather evidence, weigh it impartially, and reach a verdict. The actual mind runs the process backwards most of the time. You start with the verdict (the belief you already hold, or want to hold) and then work as its defence lawyer, scouring for supporting evidence and cross-examining anything inconsistent into silence.

You search for confirming information. You read ambiguous evidence as supporting your side. You remember the hits and forget the misses. You let confirming evidence in over a doorstep and make disconfirming evidence climb a wall^[1].

Setting the stage

I want to put two scenes down here, because I am going to keep returning to them throughout the series (or try to, if I can bend the examples enough each time!). Every bias in this run is easier to see when it happens to the same people in the same rooms, so: meet the dinner table and the dashboard. They will be back!

Why intelligence hinders you

Being clever, educated or numerate does not protect you (despite what most of us tell ourselves), and can make things worse. A smarter person is a better lawyer: quicker at generating persuasive arguments for what they already believe, better at spotting the flaw in inconvenient evidence, more fluent at explaining away the anomaly. It's funny how we assume intelligent people are wiser. There's a reason wisdom and intelligence are separate ability scores in pathfinder and dungeons and dragons: intelligence is knowing that a tomato is a fruit, wisdom is not adding it to your cereal.

Studies find that greater numeracy and reasoning ability can increase polarisation on politically charged questions, because people put their cognitive horsepower to work for their existing side rather than for the truth^[2]. The ignorant are overconfident, fine, we all knew that. The capable are better at rationalising, which is not a great thing for society.

The one habit that helps (apparently)

Willpower does nothing here. "Be less biased" is an instruction on the level of "be taller". Confirmation bias runs below awareness, so telling yourself to be fair-minded mostly produces a more confident lawyer. The thing that helps is structural: go and find the strongest version of the opposing case.

Before settling a belief that matters, hunt down the best argument against it, from the cleverest people who disagree, and try to state their case so well they would sign off on your summary. That is steelmanning, and it is the discipline behind how I try to write (and mostly fail at).

Ask "what would change my mind?" and get worried if the answer is "nothing". Watch for the flinch, too. That little urge to wave a piece of evidence away is strong and itself deeply biased - you're doing that with your "tribe", your "side" and your "argument" in mind, not actually fairly comparing evidence. Your defence lawyer has found something and would like it kept out of the record.

But, even with that...

It won't make you unbiased. Nothing does. But - it will insert a step where you point your intelligence towards something more than building walls.

You cannot trust your sense of having weighed the evidence fairly. It is a horrible fact to know about yourself and we're kind of stuck with that fact. What's worse, knowing this doesn't solve the problem. It's like recognizing you having a proclivity for a certain flavor of ice cream doesn't make you like all of them equally suddenly. But maybe it can help you try new flavors, if you take that next step. This metaphor is a bit stretched: what I mean is, try thinking of other perspectives thoroughly. Intellectual empathy; emotional, too.

Go and read the Nickerson review. It came out in 1998.

References

1. Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175
2. Kahan, D. M., et al. (2017). Motivated numeracy and enlightened self-government. Behavioural Public Policy, 1(1), 54–86. https://doi.org/10.1017/bpp.2016.2

Eight posts in, every escape from a trap in this series has been some flavour of randomness:

• simulated annealing gambles: while it's hot it accepts bad moves at random and hopes for betterment
• the ant colony wanders its way out
• the genetic algorithm and its program-breeding cousin mutate their way out
• the swarm jitters
• CMA-ES and differential evolution sample their way out of trouble
• even variable neighbourhood search, the most systematic escape artist so far, opens every escape with a random shake

Tabu search is the odd one out, escaping traps by remembering them: it keeps a short list of places it has just been and refuses to go back, even when going back looks like the best move on the board.

Fred Glover sketched the idea in 1986^[1] and built it out properly in 1989^[2]. The core of it is simple: run a greedy search, but make the recent past forbidden. That one addition turns the dumbest search strategy going into something that can walk out of a local optimum, cross a plateau, and keep improving. It needs no temperature and no coin to flip, which makes it something of an oddity in this series.

Greed with amnesia

Steepest descent is the greediest search there is: look at every neighbour of your current solution, take the best one, repeat. It's fast, it's simple, and it's doomed: sooner or later it reaches a solution where no neighbour is better, and there it sits. I covered that failure in the annealing post, where the cure was randomness on a cooling schedule.

Allow sideways moves (neighbours that are no worse, a common tweak for crossing flat regions) and the search steps from board A to board B, surveys the neighbourhood, and concludes that the best move from B is... back to A.

So it goes back, and then the best move from A is back to B. It will do this until all the coal mines and oil refineries run dry.

This is cycling, the signature failure of memoryless search, where every step the algorithm wakes up fresh with no idea it has been here before. Whether three times or over nine thousand times.

A ban list and a countdown

In tabu search, after each move, an attribute of the position you just left (in our demo, the square a queen just vacated) goes on a ban list for a handful of iterations, called its tenure.

While it's on the list, the search may not move a queen back there, however tempting the move looks, and with the recent past fenced off the only direction available is somewhere new. So the ban list turns "best move" into "best move I haven't just made", a small change that punches through plateaus which hold a hill climber forever.

There is one safety valve: if a banned move would produce the best solution ever seen, the ban is waived. This is the aspiration criterion, and it exists because tabu attributes are deliberately crude: banning a square bans every future board that would use it, including, occasionally, a record-breaking one. When the forbidden move beats anything in the run so far, take it.

Glover's tug-of-war

Glover framed the wider method as a balance between intensification (search hard near the good solutions you already have) and diversification (force yourself into regions you have never visited). Serious implementations layer on frequency memory (penalise moves you make too often), long-term restarts, and cleverer aspiration rules. It all grows from the same seed: the search takes notes on itself, and then obeys them. The demo here is a simple version.

Eight queens, no peace

The playground this time is the N-Queens problem: place N queens on an N×N chessboard so that none attacks another, meaning no two share a row, column, or diagonal. Our queens start one per column, so columns are handled for free, and a move relocates one queen within her own column. The score is the number of attacking pairs, and zero means solved.

It's a lovely testbed for tabu search because the landscape is nothing but plateaus. Near the end you are almost always sitting at one or two conflicts, surrounded by equally mediocre neighbours. That's the terrain where greedy search paces in circles, like me trying to find my car in the multistory car park outside my work (I have the memory of a pigeon with some things).

Below, a tabu search is solving the board live, and red lines join queens that currently attack each other.

The frost is the memory: when a queen leaves a square that square ices over, and the frost fades as its tenure counts down, so you're watching the algorithm's short-term memory dissolve in real time. The readout underneath lists the banned squares with their countdowns. When the search overrides a ban because the move would break a record, an "aspiration!" badge lights up next to the toolbar.

Tenure, at zero and at twenty

Switch the memory off. The search becomes plain steepest descent with sideways moves, and on a lucky board it solves anyway. Most of the time it drops to one or two conflicts and then starts pacing, and the "revisited states" counter begins to climb as it bounces between the same handful of boards, forever, at whatever speed you set. The chart flatlines with a tiny shimmer, which is the cycle itself.

Flip the memory back on and the search usually punches out within a few dozen steps. One counter, and it makes the case for tabu search better than I can.

Set tenure to 0. A ban that expires instantly is not a ban, so this is hill climbing again, cycle and all, even with the memory checkbox ticked. Tenure is the memory. The checkbox just decides whether anyone writes to it.

Set tenure to 20. Now the search forbids so much of its own neighbourhood that it can barely exploit anything. It reaches a promising region, is immediately banned from refining it, and lurches away, sometimes abandoning a one-conflict board it could have finished in a step or two.

Grow the board. Bigger boards mean bigger neighbourhoods and longer plateaus, and the sweet-spot tenure grows with them. Around 7 suits the 8×8 board, and at 16×16 you'll want noticeably more. The literature's rule of thumb is that tenure should scale with problem size.

Remembering the good and remembering the bad

There's some interesting debates I have in my mind about this concept, because it's somewhat philosophical. Our human memories are odd, we often think of periods of time that are difficult and we dislike fondly later in life, like times in the military - we remember the fun moments and forget the 5am wake up time, endless PT and drills. But we also remember specific jarring, horrible moments with great clarity - PTSD is a big warning marker left with the intent to protect us, and let us remember what was bad.

Maybe there's something to take from this: it's actually efficient to remember the bad things, but at some point to let them go, and perhaps similar can be said about remembering the good things, when trying to move towards a better future.

References

1. Glover, F. (1986). Future paths for integer programming and links to artificial intelligence. Computers & Operations Research, 13(5), 533–549. https://doi.org/10.1016/0305-0548(86)90048-1
2. Glover, F. (1989). Tabu Search - Part I. ORSA Journal on Computing, 1(3), 190–206. https://doi.org/10.1287/ijoc.1.3.190

Genetic programming evolves programs. What gets bred is the program itself rather than the settings of a program: a working formula nobody wrote, assembled out of worse ones. I've published with this one: the belt-laying search in the Factorio post is a genetic programming system, and it came out ahead of both other methods in the paper.

This is part eight of the series. So far:

• ants have foraged
• triangles have evolved
• a search has cooled its way out of bad valleys
• a swarm has chased its own best guess
• a stuck search has widened its neighbourhood until it came unstuck
• a cloud of samples has learned the shape of a valley
• a population has built its moves out of its own spread

In part two every individual was a fixed-length list of numbers: fifty triangles, ten genes each, five hundred dials for evolution to twiddle. The shape of that genome was decided by me, in advance, forever. Genetic programming throws that away. The genome is a tree of code, and the size and shape of the tree are up for grabs too. Evolution gets the structure as well as the settings.

An infinite haystack

The demo below does symbolic regression. I sample forty points from a hidden target curve, and the population's job is to find an expression that fits them. Individuals are trees built from plus, minus, times, a protected divide (dividing by nearly zero returns 1 instead of exploding), sine, cosine, the variable x, and random constants.

Fitness is mean squared error over the sample points, the lower the better. A formula that outputs infinity or NaN anywhere gets a score so bad it amounts to a death sentence.

Part two's genome was a box with five hundred dials: finite and fixed. The space of all expressions is infinite, and almost all of it is garbage. A random tree comes out like cos(x/x)*0.3-x. There's no gradient to follow and no obvious notion of "nearby".

And yet a population of these things, bred and culled, finds working programs in seconds - which is kinda crazy.

Trading subtrees

The engine is the one you already know from part two: tournament selection, crossover, mutation, and a couple of elites carried into each new generation untouched. What changes is what crossover means. With trees, crossover picks a random subtree in each parent and swaps them. It works because subtrees are meaningful units in a way that random substrings of characters never are: sin(2*x) is a working component. Evolution here is trading parts, not shuffling letters.

The starting population is grown "ramped half-and-half", a mix of bushy full trees and scraggly random ones at several depths (GP-speak for seeding the pool with variety). I also cap tree depth at eight, partly for your device's sake and partly because, as you'll see, GP will take every inch you give it.

Watch a program grow

Below, a population of formulas is evolving against the data. The dots are the samples from the hidden curve, and the green line is the best formula so far, redrawn each generation so you can watch it snap toward the data.

The node diagram is that formula's actual code: blue nodes are operators, amber leaves are x and constants. The readout underneath prints it as maths you can read. Generations tick by a few per second so the motion stays watchable.

The parsimony tax

Set parsimony to zero. Keep one eye on the average size chart. Error improves, then plateaus, while average size climbs and keeps climbing until it slams into the depth cap.

The population is now hauling around programs of a hundred-plus nodes that fit the data no better than a fifteen-node formula from fifty generations ago. This is bloat, the most famous disease of genetic programming and the reason the parsimony slider exists. Why a population would choose to bloat gets its own section below.

Now slam parsimony to maximum. The opposite catastrophe: every node now costs more fitness than accuracy is worth, so the population collapses to tiny, dumb expressions: a bare constant, a lonely x. They fit the data terribly, but cheaply. Parsimony is a tax, and like any tax, set it too high and nobody builds anything.

Shrink the population to twenty. The run crawls or stalls. With so few trees, the pool of subtrees for crossover to trade is tiny, and once the population homogenises, mutation has to invent every new part from scratch.

And drop mutation to zero while you're at it. Same lesson as part two, sharper here: crossover can only recombine subtrees that already exist somewhere in the population. Once the interesting ones are gone, they are gone.

Bloat is evolved defensive code

Subtree crossover is destructive: graft a random subtree into a random point of a working program and you'll usually break it.

Imagine two parents of equal fitness, one lean, one padded with introns: code that computes something and then multiplies it by zero, or adds a quantity and subtracts it again. When crossover hits the padded parent, odds are it lands in the padding, and the child behaves just like its parent. When it hits the lean one, it hits something that mattered.

Padded parents therefore produce more viable children. Padding spreads. Average size marches upward while fitness stands perfectly still.

The population is evolving resistance to its own genetic operators. No line of my code rewards defensive padding: it falls out of selection pressure, the same way real genomes are stuffed with non-coding DNA and real codebases accrete dead branches nobody dares delete (you don't want to see some of the folders I don't sync to GitHub here locally...).

The trees are, in effect, growing bubble wrap to survive crossover.

Where GP turns up in practice

Symbolic regression is the flagship application and it earns its keep in science. Fit a neural network to experimental data and you get a black box that interpolates. Fit an expression tree and you get an equation a human can read and argue with.

GP-based systems have rediscovered conservation laws from raw motion-capture data, and they get used to propose compact models in physics, materials science and ecology. When you want insight rather than raw predictive power, a ten-node formula beats a million-weight network, contrary to what many machine learning specialists would say.

NASA's ST5 mission needed a small satellite antenna with an awkward radiation pattern, and an evolutionary algorithm designed one^[3]. The result looks like a bent paperclip. No engineer would have drawn it, and it outperformed the conventionally designed alternative.

It flew in 2006. There is evolved hardware in space!

John Koza launched the field with his 1992 book^[1] and then spent years compiling "human-competitive" results: GP runs that reinvented patented circuits and produced patentable designs of their own.

Every one of them lived in a domain with a fast, automatic, well-behaved fitness function. GP never became a general way to write software, and the field guide I recommend below^[2] is refreshingly frank about why. Where a score is cheap to compute and means what you want, GP is a live option. Everywhere else, the fitness function is the really hard part.

Structure is a freedom

Part two's lesson was that the fitness function is where the meaning lives: change what you reward and you change what evolves. Give evolution control over structure as well and it will use that control in every way that pays.

It builds the subtree that fits your curve. It also builds padding to shield that subtree from your own crossover operator, and for the same reason: both help its children survive.

Zero the parsimony and watch the trees fatten in real time, then tax them back down to stumps. Sit through thirty flat generations of the error chart until it falls off a cliff, because two mediocre parents happened to be holding complementary halves of the answer.

An infinite search space, almost all of it garbage, explored by trading parts and taxing waste. It should not work anything like as well as it does.

References

1. Koza, J. R. (1992). Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press. ISBN 978-0262111706.
2. Poli, R., Langdon, W. B., & McPhee, N. F. (2008). A Field Guide to Genetic Programming. Freely available at http://www.gp-field-guide.org.uk
3. Lohn, J. D., Hornby, G. S., & Linden, D. S. (2005). An evolved antenna for deployment on NASA's Space Technology 5 mission. In Genetic Programming Theory and Practice II (pp. 301–315). Springer.

This is part seven of the Algorithms, Live series. So far:

• ants have foraged
• triangles have evolved
• a search has cooled its way out of bad valleys
• a swarm has chased its own best guess
• a stuck search has widened its neighbourhood until it came unstuck
• a cloud of samples has learned the shape of a valley

Today's algorithm is for hard continuous problems: differential evolution. It's one of the simpler of the evolutionary family, a few lines of arithmetic wrapped in a loop. I was first properly introduced to it by Prof. Cedric Gondro, at Michigan State University, as it's one of his favorite algorithms, so we were considering its use for genomic prediction problems.

Subtraction as a search strategy

Here is DE/rand/1/bin^[1]: keep a population of points scattered over the landscape. To improve any one of them:

1. Pick three other members at random: \(a\), \(b\), and \(c\).
2. Build a mutant: take the difference vector from \(c\) to \(b\), scale it by \(F\), and add it to \(a\). In symbols, \(a + F(b - c).\)
3. Crossover: build a trial point by taking each coordinate from the mutant with probability \(\mathit{CR}\), otherwise keeping the parent's coordinate. (One coordinate is always taken from the mutant, so the trial is never a clone of the parent.)
4. Greedy selection: score the trial. If it beats the parent, it takes the parent's seat in the population. If not, it is thrown away and the parent stays.

Do that once for every member to create a generation. Repeat!

Every other optimiser in this series had to decide how big a move to make: simulated annealing had a cooling schedule, the genetic algorithm had a mutation rate, gradient descent has a learning rate. DE samples a difference between two of its own members and uses that as the move.

Early in a run the population is sprayed across the whole landscape, so the differences are long and the moves are large. As the population drains into a valley, its members bunch together, the differences between them shrink, and the move size shrinks too.

The difference vectors shrink, and they also point somewhere useful. If the population has settled along a curved valley, the differences between members mostly run along that valley, so mutants get launched along it too.

The population becomes a rough map of the terrain it's sitting on. The state you already have to keep doubles as the search geometry.

Watch the geometry

You probably get this now if you're reading through the series, but here goes: the demo below runs DE on a choice of four classic test landscapes. The coloured ground is the fitness surface (deeper blue is worse, the ground fades toward the background where it is low, and a small cross marks the global minimum). The blue dots are the population, and the green ring is the best point found so far.

Each generation, one member of the population gets its donor construction drawn in full: the three amber dots are \(a\), \(b\), and \(c\), the dashed amber arrow is the difference vector from \(c\) to \(b\), and the red arrow is that same vector scaled by \(F\) and grafted onto \(a\), ending at the mutant (the hollow red circle). The red diamond is the trial point after crossover, ringed if it won its duel with the parent.

Then compare the arrows at generation two with the arrows at generation sixty. Same code, same \(F\), and the moves have shrunk by orders of magnitude because the population did.

F, CR, and a population of eight

Drop \(\boldsymbol{F}\) to 0.1 on the Egg carton. Every move is now a tenth of a population difference, so members can only creep to the nearest dimple. The spread collapses early, usually before anyone has found the central basin.

Then the trap springs. A huddled population can only make tiny difference vectors, so the step size follows the spread down and the search is welded shut. Self-scaling, run in reverse. The best-fitness line goes flat while the spread line dives, and that is premature convergence.

Now push \(\boldsymbol{F}\) to 1.8. Every mutant overshoots the region the population spans, so the population becomes a spray. The spread line stops falling, the mean step stays huge, and best fitness improves only by luck. DE needs the population to contract for its thermometer to read cooler. At \(F\) this high, contraction never begins.

Set \(\boldsymbol{\mathit{CR}}\) near zero on the Banana valley. Each trial now differs from its parent in a single coordinate, so the search moves like a rook: horizontally or vertically, never diagonally. The banana valley is curved, and progress along it needs both coordinates to change together.

Watch the population inch along in tiny axis-aligned stair steps. Then push \(\mathit{CR}\) back up to 0.9 and watch it slide along the valley in coordinated diagonal moves. \(\mathit{CR}\) is the knob that decides whether DE moves one coordinate at a time or as whole vectors.

Shrink the population to 8. The difference vectors are now drawn from a pool of a few dozen possible pairs, so the same handful of directions gets recycled over and over.

Diversity is essential and a tiny population runs out of it almost immediately: fast convergence, frequently to the wrong dimple. The genetic algorithm in part two had the same failure for the same reason, a gene pool too small to explore with.

The missing knob

Put this next to the genetic algorithm from part two and the family resemblance is obvious: a population, mutation, crossover, selection. The GA shuffled bits and letters. DE does arithmetic on real vectors. The GA made mutation from coin flips and a fixed rate you had to tune. DE makes mutation out of the population itself, so the operator adapts as the population does.

And the GA ran a selection lottery where fit parents got more tickets. DE holds one duel per seat, child against its own parent, winner sits. The best member can only be replaced by something better.

Its sophisticated cousin CMA-ES learns an explicit statistical model of where good moves live and updates it with some fairly serious mathematics. DE gets a decent share of the same benefit, steps that stretch along the good directions and shrink as the search closes in, using subtraction.

It is dumber than CMA-ES generally, but very much stands on its own, regardless!

Where DE turns up in practice

Storn and Price introduced it in 1997^[1], it won early optimisation competitions, and it is still a default first try in engineering optimisation: antenna design, power systems, chemical process tuning, anywhere someone has a black-box simulator with a dozen real-valued dials. The survey literature on its variants runs to hundreds of papers^[2].

You can buy adaptivity: learned models, schedules, meta-parameters. Or you can arrange the algorithm so the state it already keeps is the adaptation. DE gets self-scaling search out of a subtraction.

Go and set the Egg carton running, click somewhere daft to strand the population in a corner, and watch the arrows do the rest.

References

1. Storn, R., & Price, K. (1997). Differential Evolution - A Simple and Efficient Heuristic for global Optimization over Continuous Spaces. Journal of Global Optimization, 11(4), 341–359. https://doi.org/10.1023/A:1008202821328
2. Das, S., & Suganthan, P. N. (2011). Differential Evolution: A Survey of the State-of-the-Art. IEEE Transactions on Evolutionary Computation, 15(1), 4–31. https://doi.org/10.1109/TEVC.2010.2059031

There is a shape of problem that many optimisers struggle with: a long, thin, tilted valley.

Every algorithm in this series so far has moved in steps whose size and shape were settled in advance:

• the ants of part 1 took steps along edges and left scent behind
• the genetic algorithm of part 2 took steps in genome space, one mutation at a time
• simulated annealing in part 3 had a step size you could watch shrink as the temperature fell

This one learns the shape of the neighbourhood worth searching, and restretches itself to match: CMA-ES.

Steps versus shapes

Valleys like that are everywhere in real problems (the technical word is ill-conditioned). An optimiser proposing moves in random directions wastes almost all of them climbing walls while gradient descent zigzags across the floor of possible solutions.

This requires an optimiser that notices "movement along this axis keeps paying off, movement along that one never does".

CMA-ES keeps a sampling distribution, a fuzzy ellipse of probability. Each generation it draws a cloud of candidate points from the ellipse, scores them, and shifts the centre toward the better half.

It also updates the ellipse's shape. Directions the winners recently moved in get widened. Directions that never produced a winner get squeezed. It saves the historic data of wins in the covariance matrix.

Instead of a fixed cooling schedule, CMA-ES watches its own recent movement. If successive generations keep marching the same way, the steps were too small, so sigma grows. If the path folds back on itself, the steps were too big, so sigma shrinks. Exploration vs exploitation.

Hansen and Ostermeier formalised all of this in 2001^[1], and the design is careful enough that the standard version has no parameters you are expected to tune. But, in this version, for fun, there are sliders so you can override that.

The loop on a napkin

1. Sample a cloud of candidate points from a Gaussian: centre (mean), scale (sigma), shape (covariance).
2. Score every point and rank them.
3. Move the mean toward a weighted average of the best few, best weighted most.
4. Stretch the covariance along the directions those winners came from.
5. Grow or shrink sigma depending on whether recent moves kept pointing the same way.
6. Go to step 1.

Watch it

Below, CMA-ES is minimising a landscape drawn as contour shading (darker is worse, the red cross is the global optimum). Blue dots are one generation's samples, the amber rings are the search ellipse at one and two sigma, and the green dot with the fading tail is the mean and the route it took.

Start with the default narrow valley and watch the ellipse for the first few seconds. It begins as a circle, tips over to match the valley's angle, and stretches. Click or drag anywhere on the map to plant the start point somewhere new.

The step-size rule

On the smooth valleys the run still works, it just crawls. The step-size rule notices the steady progress and inflates sigma back to something sensible, which you can watch happening on the right-hand chart.

On the egg carton a tiny sigma is fatal. The cloud fits inside a single dimple, every sample agrees the local floor is the best thing going, and the run settles into the nearest dent.

If the sigma is huge, the first generations are chaos.

Then watch the sigma chart. The path the mean traces keeps folding back on itself, the adaptation rule reads that as "steps too big", and sigma falls generation after generation until the cloud shrinks onto something worth exploiting. Thrashing and then recovery is the self recovering step size.

Drop lambda to 6 and each generation's ranking is built from almost nothing. The mean update gets noisy and the covariance estimate gets noisier.

Back to the narrow valley, default settings, and this time watch only the amber ellipse and the stretch readout. Circle, tilt, stretch, and then a needle sliding along the valley floor with the stretch number climbing into the hundreds.

Every other algorithm I've covered in this series would be paying the valley's toll on every single move, but CMA-ES redrew its own geometry so there is no toll left to pay!

A memory of directions

What I like most about CMA-ES is where it keeps what it has learned. The ants kept their knowledge in the world, as scent on edges. The genetic algorithm kept it in a population, spread implicitly across genomes: variety, but no geometry. Simulated annealing had one knob, a step size, and its step size only goes down.

CMA-ES keeps a small, dense summary: one point for "where", one number for "how far", and one matrix for "which way". The matrix is the difference. It is a running record of the directions that recently succeeded, updated every generation, and it turns a search that happens in the landscape into one that builds a model of the landscape's local shape.

Where it gets used

For continuous black-box problems, where the objective is a simulator or a physical measurement or a hyperparameter score with no gradient in sight, CMA-ES is the default serious tool up to a few hundred dimensions (yes, you read that correctly!!).

It has tuned aircraft geometry, robot gaits, controller parameters, and the settings of other learning systems. Hansen's tutorial^[2].

When progress stalls, the instinct is to push harder in the direction you were already going. CMA-ES keeps an estimate of which directions have been rewarding you lately, and spends your effort along those, even when they aren't the ones you set off in. Which is surprisingly good life advice to get from a pile of linear algebra.

References

1. Hansen, N., & Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9(2), 159–195. https://doi.org/10.1162/106365601750190398
2. Hansen, N. (2016). The CMA Evolution Strategy: A Tutorial. arXiv preprint arXiv:1604.00772. https://doi.org/10.48550/arXiv.1604.00772

Factorio is a game I've loved since it was released. It's my most-played game on Steam, and there's something so deeply satisfying about playing a game all about optimization, exploring and solving problems. When I started thinking about the psychology behind this, I realized that video games, puzzles, board games and the like mirror a satisfaction I find when programming, conducting research and solving real-world problems. It scratches an itch in a particular way unfound in other walks of life: defeating challenges beyond a grind in an RPG, lifting a heavier weight at the gym or accomplishing projects based more on time than difficulty. It's the slow accrual of problems solved into a portfolio of solutions working in tandem, so you can zoom out to the map and say "look at my base! it's a bit of a mess, but it's mine, and I love it."

I decided to bridge the gap between the video game and the research, both of which scratched that itch in my mind. I brought up the idea with one of my supervisors at Michigan State University (MSU) at the time, Prof. Wolfgang Banzhaf. He thought it sounded interesting, and liked my point that a video game is a better teaching instrument than a textbook, since this visually shows success and failure, and the mechanics of individual algorithms live in front of you. He said I should go for it, so I enlisted a couple of my colleagues (now co-authors) and we worked on it together.

Watch it lay the belts

Two places this demo is not the paper

The scoring here is not identical to the published version. The paper scored a candidate by running it in the game, which returns nothing at all until the route lands. That cliff is survivable for a method that lays whole paths in one move, and hopeless for one that edits a single cell at a time: annealing would sit at zero for thousands of evaluations and look broken. So the demo adds partial credit for reaching further east, and a small bonus for using fewer belts once connected. It is a shaped objective, and shaped objectives can be gamed, so this is really just for demo purposes, not a realistic solution (mostly since "east" isn't always the goal).

The second departure is in how annealing picks what to change. The paper varies its neighbour selection with iteration progress. Here, two thirds of edits are at the frontier, meaning the cell where the current route gives up. The reason is arithmetic: on a twelve by twelve grid, a uniformly chosen cell is almost always somewhere the route never reaches, so most edits are invisible to the score and the search spends its budget rearranging scenery. Both changes exist to make the comparison fair to annealing, which is the direction a bias should run if you are going to have one.

So: this is a pretty and intentionally non-representative demo of the paper.

What the paper actually did

The paper behind this is The Factory Must Grow: Automation in Factorio, written with my co-authors Iliya Miralavy, Stephen Kelly, Wolfgang Banzhaf and Cedric Gondro, and presented at GECCO in 2021. The starting point was that Factorio is a factory simulator that has already done the expensive part of building a benchmark: it has consistent physics, a fixed belt throughput, and a community of players who have spent years discovering what good layouts look like, so how would our algorithms fare?

We defined the logistic transport belt problem formally, as an integer program with hard constraints on what can occupy a cell and how underground pairs must line up, and two soft constraints to be maximised: items delivered to the receiver, and items accepted from the chest. Then we built the 'Factorio Optimizer Interface', which uses the game's Lua modding hooks and the RCON protocol to place a candidate layout in a running Factorio server and read back how many items the input inserter took and how many reached the output chest. Because it talks over a socket and reads matrices from disk, the optimiser can be written in anything: ours were in Java, Python and C++. The evaluation function is the game itself.

Three optimisers were compared on six instances, being three grid sizes at three by three, six by six and twelve by twelve, each with and without obstacles. Parallel simulated annealing, a genetic programming system whose operators place and connect belt runs rather than editing cells, and an evolutionary reinforcement learning approach. The small instances are solvable by anything. The twelve by twelve with obstacles is where things really got interesting.

Genetic programming did well here because its operators fit the problem: laying a run of belts and connecting two points are single moves, so one step of the search covers ground that costs annealing a long chain of individually pointless edits.

The model

Section 4 of the paper states the problem formally. Two hard constraints, two soft ones, and a pair of binary decision variables. Throughput, routing and flow is absent, for a reason I will come back to.

A position \(p\) is a coordinate pair in the grid. Directions are \(d \in \{0,1,2,3\}\), meaning north, east, south and west. Belts and underground belts are drawn from \(L_b\) and \(L_u\), and \(\phi_p\) is 1 when position \(p\) holds an obstacle and 0 otherwise. The decisions are whether to place a given belt type, at a given position, facing a given way:

The objective maximises the sum of the two soft constraints, each a parameterised item count scaled by a weight (not as in how heavy it is, but a weighting as in scaling), where \(o\) is the number of items output and \(i\) the number input:

HC1 allows a position to hold one logistic object or one obstacle, never both and never two. HC2 requires at least one item to be placed at all. A solution failing either scores zero; otherwise fitness lands between 0 and 1.

HC1 is the constraint that makes this combinatorial. Every cell you spend is a cell you cannot spend on anything else, and the obstacle term means the grid's difficulty is set before the search starts. HC2 exists to rule out the empty solution, which would otherwise be feasible and uninformative.

There is no flow conservation here, no belt capacity, no constraint saying a route has to connect. The underground pairing rule, the one that requires an entrance and exit in the same row or column, facing the same way, at most four cells apart, is described in the paper as a property of the game rather than written as an inequality.

All of that is missing because the model does not evaluate solutions. The game does. A candidate is built in Factorio through the interface, run, and scored on what the receiving chest actually contains, so the physics stays where it already worked instead of being restated as constraints that could disagree with it. That is the paper's real trick, and it is why the formal model can afford to be this short.

It also explains the weights. With \(w_2 = 0\) the objective would count only what arrives, and every layout that fails to connect would score identically at zero, leaving a search no way to tell a near miss from an empty grid. Because \(SC_2\) pays for items merely taken out of the input chest, a belt that starts correctly and dead-ends still earns something. That is a modelling decision rather than an algorithmic one, and it is the difference between a landscape a local search can climb and a cliff it cannot.

The niche

Optimisation research runs on benchmark sets, and benchmark sets have a known failure mode: they get old, methods get tuned against them, and after a decade you cannot tell whether a new result is progress or overfitting to a fixed library of instances. Games are an interesting escape from that, because a game that people play is a simulator someone else has already paid to build, validate and keep interesting.

Factorio is a good one for the specific reason that its difficulty is combinatorial rather than reflexive. Nothing here depends on reaction time or hidden information, like many games: it is placement under constraints, with a throughput number at the end, which is the same shape as facility layout, circuit routing and warehouse design in real life. It also comes with something no synthetic benchmark has: a large population of humans who have independently optimised the same problem for fun, and whose solutions you can compare against.

This is one problem inside one game, with a single objective and an evaluation function that a commercial studio can change in a patch. A benchmark you do not control is a benchmark that can be revised out from under your published results. And the community solutions that make it appealing are also a contamination risk, since a method tuned by someone who knows the good patterns is not discovering them. Games are a useful source of problems, not a replacement for the discipline of saying what your benchmark can and cannot show.

References

1. Reid, K. N., Miralavy, I., Kelly, S., Banzhaf, W., & Gondro, C. (2021). The Factory Must Grow: Automation in Factorio. Proceedings of the Genetic and Evolutionary Computation Conference Companion. https://arxiv.org/abs/2102.04871

The algorithms themselves need to be separate, I wasn't planning on making a new JMetal, DEAP or similar. Part of the series on how this site is built.

The CSS owns the colours

I always enjoyed color theory in computer science. The RGB values, pixels (did you know it stands for "picture element"?), bitmaps vs JPEGs vs other formats, it's super interesting to me. On this website, the theme toggle cannot restyle a canvas the way it restyles text, and the first version of the ant post hardcoded its colours in JavaScript and looked wrong in one theme or the other. Every widget's colours live on its wrapper element as CSS custom properties, set for both themes in one place in the stylesheet, and the little framework, js/kr-viz.js, reads them and hands them to the drawing code as ctx.colors. Before it existed, every post carried its own copy of the function that reads them: sixteen copies, in five slightly different versions. Here's the one that's left, and the observer that rereads it:

var TOKENS = ['s1', 's2', 's3', 's4', 's5', 's6', 's7', 's8',
              'ink', 'muted', 'grid', 'surface', 'border', 'cell'];

function readColors(el) {
  var cs = getComputedStyle(el), out = {};
  for (var i = 0; i < TOKENS.length; i++) {
    out[TOKENS[i]] = cs.getPropertyValue('--viz-' + TOKENS[i]).trim();
  }
  return out;
}

new MutationObserver(function () {
  colors = readColors(root);
  ctx.colors = colors;
  redraw();
}).observe(document.documentElement, {attributes: true, attributeFilter: ['data-theme']});

The observer redraws the moment data-theme gets toggled, so a running simulation changes palette mid-run. This has no practical value whatsoever.

Randomness must be reproducible

Math.random() is banned from the demos. Every widget gets its randomness from ctx.rng(), which is mulberry32, a seeded generator inside the framework:

function mulberry32(seed) {
  var a = seed >>> 0;
  return function () {
    a |= 0; a = (a + 0x6D2B79F5) | 0;
    var t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

When a reader reports that the race chart does something odd around 90,000 evaluations, I can watch the same 90,000 evaluations they did. "New cities" buttons just bump the seed.

Phones are the real test

No canvas ever gets a fixed rendered height now: the framework computes height from measured width, so narrow screens get a taller aspect instead of a squashed ribbon. It also caps the pixel density at 2, because a 3x phone would otherwise redraw nine pixels for every one it shows, every frame, for detail nobody can see at that size. Side-by-side panes stack vertically below ~480px, and sliders become full-width rows on small screens.

A canvas only gets touch-action: none if it has drag interaction, so chart canvases never trap a reader mid-scroll.

And because rules I merely remember are rules I eventually break, the test suite enforces them. The smoke test auto-discovers every blog page containing a <canvas> tag (no registration list to forget to update) and loads each at 360px wide in a headless browser. It fails the build on horizontal overflow, a canvas wider than the viewport, a canvas squashed under 100px tall, or any console error.

The chassis checklist

What every new interactive post starts from:

• A .kr-viz wrapper with empty slots for the toolbar, the sliders and the stat tiles, which the framework fills. The --viz-* palette comes from the stylesheet; a post that wants other series colours sets them on its own wrapper, in both themes.
• One call: KRViz.mount(target, {canvases, buttons, controls, stats, charts, init, step, draw}). The post keeps init, step and draw, the algorithm and the drawing, as plain functions you can read in view-source. The framework does the rest: it sizes the canvases, rereads the colours on a theme flip, and runs the loop.
• ctx.rng() for every random draw, from a seed in the options.
• Speed in steps per second, the same in every demo; the framework works out how many steps to run in each frame.
• role="img" and a real aria-label on every canvas.

The idea: if the site can't fetch, fetch into the site

A normal website with live data has a server that queries things when visitors arrive. A static site can't do that, but it has a loophole: the site is a git repository, and anything that can commit to the repository can change the site. So instead of fetching data per visit, a scheduled job fetches it once a week and commits the results as plain JSON files. The site stays static. Visitors' browsers read those JSON files with an ordinary fetch: same origin, no keys, no rate limits, nothing that can fail separately from the site itself.

The whole pipeline is one workflow file and three Python scripts. The workflow checks out the repository, runs the two fetch scripts, checks what they wrote, and commits whatever changed (the comments are trimmed out here):

name: Refresh live data

on:
  schedule:
    - cron: "0 6 * * 1"   # weekly, Mondays 06:00 UTC
  workflow_dispatch:

permissions:
  contents: write

concurrency:
  group: live-data
  cancel-in-progress: false

jobs:
  refresh:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    env:
      FILES: >-
        data/lastfm.json data/now.json data/lastfm-history.json
        data/topalbums.json data/books.json data/reading.json
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Fetch listening and currently-reading
        id: now
        continue-on-error: true
        env:
          LASTFM_API_KEY: ${{ secrets.LASTFM_API_KEY }}
        run: python .github/scripts/refresh_now.py

      - name: Refresh the read shelf
        id: books
        continue-on-error: true
        run: python .github/scripts/refresh_books.py

      - name: Validate the data files
        id: validate
        run: python .github/scripts/validate_data.py $FILES

      - name: Commit what refreshed
        if: always() && steps.validate.outcome == 'success'
        run: |
          if git diff --quiet -- $FILES; then
            echo "No change."
            exit 0
          fi
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git add -- $FILES
          git commit -m "Refresh live data (Last.fm, Goodreads)"
          git pull --rebase origin "$GITHUB_REF_NAME"
          git push origin "HEAD:$GITHUB_REF_NAME"

      - name: Fail if any source did not refresh
        if: always()
        env:
          NOW: ${{ steps.now.outcome }}
          BOOKS: ${{ steps.books.outcome }}
          VALID: ${{ steps.validate.outcome }}
        run: |
          status=0
          if [ "$NOW" != success ]; then
            echo "::error::Last.fm / currently-reading refresh: $NOW (see 'Fetch listening and currently-reading')"
            status=1
          fi
          if [ "$BOOKS" != success ]; then
            echo "::error::Read-shelf refresh: $BOOKS (see 'Refresh the read shelf')"
            status=1
          fi
          if [ "$VALID" != success ]; then
            echo "::error::Data validation: $VALID; nothing was committed"
            status=1
          fi
          exit $status

The workflow_dispatch line adds a manual "run now" button in GitHub's interface (which you will want the first dozen times), while the diff check means runs where nothing changed produce no commit at all.

Most of the rest is there for when something breaks. Both fetch steps are allowed to fail (continue-on-error), so a Goodreads outage doesn't stop the Last.fm numbers from landing, and the last step turns the run red afterwards so I still hear about it. Nothing is committed unless every file passes its schema in data/schema/, because the pages read these files with no fallback for a wrong shape. The robot and I both push to the same branch, hence the git pull --rebase before its push, and the concurrency group makes a manual run wait for a scheduled one instead of both committing the same files at once.

What the script gathers

The first script, refresh_now.py, makes a handful of HTTP requests. From the Last.fm API: my total play count, the most recent track, and my top albums of the past three months, which feed the homepage stat, the "now playing" strip, and the record crate on the music page respectively. From Goodreads, which shut its API years ago but still publishes RSS feeds: my currently-reading shelf, parsed straight out of the XML.

The second, refresh_books.py, reads the feed of my read shelf the same way and merges it into data/books.json, so a book I mark as read on Goodreads turns up on this site's reading pages by itself. The feed only reports the latest time I finished a book, so when I re-read something the script holds on to the earlier dates itself, because they exist nowhere else. It also writes data/reading.json, the books-read and reviews figures on the homepage. Everything goes into JSON files under data/, most of them a few kilobytes, apart from books.json, which holds every book I've marked read and weighs about 83 KB.

Each run appends that week's play count to a rolling log, capped at ninety entries, which at one measurement a week is close to two years of history.

What the homepage shows

The strip under the hero is a single row of up to four things, and only two of them come from the robot. The book and the track are read straight out of now.json. The latest short story comes from stories.json, which is an empty list until the first story goes up, so for now the strip leaves that card out. The latest blog post isn't in the strip at all, because the homepage has a whole section of its own for those.

Last.fm distinguishes a track that is playing from one that was played, so the label switches between "Now playing" and "Last played" depending on whether I happen to be listening when you open the page, and the first version gets a little animated equaliser beside it. It is not accurate, because it is updated so infrequently, but it gives some flavor to the site and over time it's likely representative of me. Maybe.

The third item, my most recent Bluesky post, is in no JSON file at all. Your browser fetches it from Bluesky's public API while the page loads. I've been tempted to remove it, as recently I've only posted new blogs to social media, but I hope I will return to posting random thoughts there again soon, making this a bit more useful than another place to see a new blog post (which is already on the index page!)

Design decisions

The homepage HTML contains a hardcoded scrobble count, and the JavaScript replaces it with the fresh value from the JSON after load. If the fetch fails, or JavaScript is off, the visitor sees a slightly stale number instead of a blank. The robot improves the site but it fails elegantly.

Goodreads' RSS goes down more often than Last.fm's API. When a source fails, the script keeps whatever the previous run wrote rather than blanking it: better yesterday's truth than today's error. A failure in one source never stops the others from updating, of course.

The Last.fm key lives in the repository's encrypted secrets and reaches the script as an environment variable. It appears in no file and no log for security reasons (not that I'd be massively put-out if someone broke into my goodreads or last.fm).

The bot's commits are labelled as bot commits, touch only the data files, and say what they did. My git history has a weekly heartbeat in it now: a tidy row of "Refresh live data" commits, one every Monday, each one the robot clocking in so I don't have to manually update stuff.

What else this pattern is good for

Anything that changes slowly and comes from somewhere with an API or a feed: your latest posts elsewhere, sports scores, weather, stars on a project, prices you're tracking, a "days since" counter for whatever you are currently ashamed of. The recipe is always the same three steps: a script that fetches and writes JSON, a workflow that runs it on a schedule and commits, and a page that reads the JSON with a fallback. If the data changes faster than a schedule can keep up with (live chat, comment counts), a static site is the wrong tool and no robot can easily fix that. Everything slower works just fine though.