Leading a Horse to Water

I'm doing a master's in organizational leadership, and one of the first things I've learned is that leadership is not a status only found in specific roles, it's something we all do every day. Leading projects, leading people, making decisions on next steps, risk assessments, and practicing your social skills. One such example I have been thinking much about recently is providing feedback - generally, but also when you give the same feedback multiple times to someone. You find they nod, say it was fair, thank you for being direct, and then do the thing again the following week.

It's something perplexing: you are specific in the feedback given, offer help, and mean it. You write the steps down, in order, in an email they could find again. You ask if there was anything in the way, and nothing changes.

As the saying goes: you can lead a horse to water. But that hardly solves the problem, it just describes it. How many times must you take the horse to water? Should you fill up a bowl and raise it to its mouth? A PowerPoint on "drinking water"? Should you bring in a horse expert to assist?

The Proverb

It is the oldest English proverb still in daily use, recorded as early as 1175 in the Old English Homilies:^[1]

Hwa is thet mei thet hors wettrien the him self nule drinken?

— Old English Homilies, c. 1175 (who can give water to the horse that will not drink of its own accord?)

John Heywood reshaped it into the version everyone knows by 1546. This is not a new problem among people.

The drinking is not yours to do, and no amount of leading better or putting in more effort as a leader will make a horse drink. It can be deeply irritating when the horse is now standing in the river up to its knees, you're gesturing at the water, and it just stares at you.

The version I like adds a third clause. Charlie "Tremendous" Jones, an American insurance-agency-executive-turned-motivational-speaker best known for the bestseller Life Is Tremendous, used to put it this way:

You may not be able to make a horse drink, but you can put salt in his oats and make him thirsty.

— Charlie "Tremendous" Jones

You cannot make the horse drink, but you CAN make it thirsty, you can check the water is fit to drink, and you can find out whether the animal in front of you is a horse that isn't thirsty or a horse that is ill.

First, check the water

Before anything else: was the feedback actually constructive? In Kluger and DeNisi's 1996 meta-analysis, which pooled 607 effect sizes from feedback studies, feedback improved performance on average, but in over a third of cases it made performance worse.^[2] The feedback that hurt was the kind that pulled attention towards the self (who you are, how you compare) rather than the task (what to do differently, next time, at this step). No one wants to hear what is wrong with their character, but many are happy to hear how they can improve the technique, the strategy or the steps.

It's always satisfying when a scientific study rings true. I know that I have this problem when I receive feedback: I feel attacked, if it's about my underlying drive or passion, but when I am given feedback on specific actions or tasks, I feel empowered. So, now I try to do the following when giving other people feedback: can they repeat back, in their own words, what they are meant to do differently, by when, and how they will know they've done it? If the answer to any of the three is a shrug, then I did not lead a horse to water, I led a horse to a description of water. "Be more proactive" is a high-level descriptor and nowhere near specific enough.

Second, check whether the horse is thirsty

The second question is whether the person thinks there is anything to fix. Prochaska and DiClemente's stages of change model, built on decades of watching people quit smoking and fail to, puts a stage before "thinking about it": precontemplation, where the person does not see a problem at all.^[3] Advice delivered to someone in precontemplation is like telling someone it will eventually rain in a city fifty miles away, at least to them. The smoking example also rings true for me: I remember begging my parents to stop smoking, even threatening to start myself if they didn't stop. Ultimately, a person needs to want to change a habit, otherwise any well-meaning, well-informed, kind and considered advice is utterly for naught.

"Yeah, fair, thanks for saying" is what a person says when they agree and intend to act, and it is also what they say to make the meeting end while being polite. This means we can't really settle on that as the finale to the feedback conversation. Klein and colleagues' meta-analysis of goal commitment says: goals predict performance when people are committed to them, and barely at all when they are not, and commitment is not the same as having heard the goal read aloud.^[4]

Worse, pushing harder at this stage makes it worse. Miller and Rollnick, the motivational interviewing people, call the helper's urge to fix the "righting reflex," and their central finding, replicated across addiction, health and now management, is that when you argue for change, the other person starts arguing against it, and hearing themselves argue against it makes them believe it.^[5] Brehm named the underlying push-back in 1966: reactance, the instinct to reclaim a freedom someone is trying to take.^[6] You can't tell a horse it's thirsty and expect it to believe you.

Third, check whether the horse can drink

Someone agreeing and then not doing is what inspired me to write this, and it is usually read as a will problem when it is often a capability or environment problem. Landsberg's skill/will matrix is a two-by-two from a coaching book.^[7] Low skill needs teaching while low will needs reasoning, and low both needs direction. Getting these wrong is like trying to inspire a fish to walk.

How to know if it's a "will" problem:

• Have I seen them do it once? If I have never watched the person do the thing, even badly, I do not know whether they can. "Write clearer tickets" assumes they know what a clear ticket looks like. Show them one. Then watch them write one.
• What is paying them to do the opposite? If they are incentivized or otherwise follow a protocol established by a higher authority or by tradition, they may ignore your feedback.
• Is there room? Asking someone drowning in tickets to also improve how they write tickets is asking for the improvement to be ignored.

Salting the oats

Given a horse that is capable, not obviously ill, and not thirsty, here are some of my thoughts and learnings compiled together:

1. Listen to them, don't assume. Ask them to say what they'll do, and write down what they said, not what you wanted them to say. A plan in someone else's words is often very different from what you think they know. Locke and Latham's goal-setting work has been saying this for forty years: participation in setting the goal raises commitment to it.^[8]
2. One behaviour, one fortnight, one checkpoint. "Be a better communicator" is useless. "Send the summary before the meeting, for the next three meetings, and we'll look at it on the 20th" is tangible, measurable and clear - the outcome is "becoming a better communicator" but the steps are clearly stated.
3. If they don't see shortcomings they don't see a problem to fix. If the report is late and you fix it for them at eleven at night, the report is never late from their standpoint.
4. Changing a person's behaviour is difficult. Changing a template, a checklist, a pairing session, a definition of done, is straightforward and easy.
5. "This is the third time we've had this conversation. What's going on?" and let silence reign until an answer comes. Listening is important.
6. "We agreed X by the 20th." A dated, agreed expectation turns the fourth conversation from an argument about memory into a paper trail.

How far to go

A failure of leadership is waiting too long. When behaviours become patterns that have been that way for years, you are fighting both the problem and established tradition. You can always try one more thing, one more framing, one more month. So the leader who doesn't decide in advance ends up deciding by exhaustion, which is late, and usually unkind to the team, to yourself, and to your colleague.

Set the budget before the third conversation. What decides it is not how much you like the person (something I know we all struggle with). It's the cost of letting the issue stand:

• You. The hours, obviously, but also the attention. A stuck person takes up a share of a leader's head that is wildly out of proportion to their share of the team.
• The others. Every hour spent on the horse that won't is an hour taken from the horses that would.
• The standard. What you tolerate for long enough becomes the definition of acceptable. And that can also spread: "Well, Jon takes 3 days on these tasks, why should I deprioritize it too?"
• The person. Every month they spend in a role they are not going to grow into is a month they are not spending somewhere they might. Endless patience inhibits their growth, prevents them from achieving their goals (and possibly stymieing their future income, prospects, benefits, etc.).

Then, how do you read which kind of stuck you're looking at?

1. They try, and fail. They attempt the thing, it comes out wrong, they attempt it again. This is a skill gap and it is - believe it or not - the good kind of stuck. Keep going, teach more concretely, lengthen the horizon. At least the horse is trying to drink!
2. They agree, and don't. Nod, thanks, nothing. This is will or environment, and the response is a single change of approach, followed, if nothing shifts, by the written expectation with a date on it.
3. They dispute the premise. "I don't think that's a problem." Either they're in precontemplation, or you are wrong. Stop advising and focus on listening. Then go and check with somebody who has no stake in it whether the water you're offering is actually water.

When to give up

Giving up on an approach is not giving up on a person. Most of what I've described above is giving up: on explanation, on the plan in your words, on absorbing the consequence. Surrendering is often surrendering the current strategy for a better one.

Giving up on the person is very different. Informal feedback, then explicit written expectation, then a formal process, then parting. The leader's obligation at each rung is the same: the person must never be surprised. If someone learns at a formal meeting that their job is at risk and this is news to them, the leader failed. Additionally, if you're not their leader but are providing feedback, sometimes giving up is giving up your part in the process to someone else - their leader, their mentor, the person themself.

I hate this part.

I'd rather run the sixth conversation than the formal one, and I think it's something many people avoid out of feelings of not wanting to be the bad guy. Ultimately, you can only control that which is your own action, not the outcome. Some wise words by an idol of mine, a physically crippled man who escaped slavery and learned much about our control over our personal destinies:

The things in our control are opinion, pursuit, desire, aversion, and, in a word, whatever are our own actions. The things not in our control are body, property, reputation, command, and, in one word, whatever are not our own actions.

— Epictetus, Enchiridion

You can lead the horse to water, you can make sure the water is clean, and you can show it how to drink, but no matter how much you want to, out of compassion, kindness or care, there is a point where you have to accept that you are not the horse and you cannot force it to drink. The servant leadership test is whether the people you lead grow. If they are not growing with you after a fair run, the kind act may be to help them find where they will, perhaps a sister team, or through a friend you know from college. I personally believe leaders should help colleagues beyond their current role and company, and encourage growth, because paths can cross many times, so help them even if not from inside your own team or organization. And remember, this cannot be a surprise, this is only after a long period of trying and goal setting.

A Flow Chart

References

1. Old English Homilies, c. 1175. John Heywood, A Dialogue Conteinyng the Nomber in Effect of All the Prouerbes in the Englishe Tongue (1546). Discussed at phrases.org.uk
2. Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284. 607 effect sizes; mean d = 0.41; over one third of interventions reduced performance.
3. Prochaska, J. O., & DiClemente, C. C. (1983). Stages and processes of self-change of smoking: Toward an integrative model of change. Journal of Consulting and Clinical Psychology, 51(3), 390–395.
4. Klein, H. J., Wesson, M. J., Hollenbeck, J. R., & Alge, B. J. (1999). Goal commitment and the goal-setting process: Conceptual clarification and empirical synthesis. Journal of Applied Psychology, 84(6), 885–896. Meta-analysis of 83 samples. pubmed.ncbi.nlm.nih.gov
5. Miller, W. R., & Rollnick, S. (2013). Motivational Interviewing: Helping People Change (3rd ed.). Guilford Press. The righting reflex, ambivalence, and why arguing for change produces arguments against it. Summary at ncbi.nlm.nih.gov
6. Brehm, J. W. (1966). A Theory of Psychological Reactance. Academic Press. Reviewed fifty years on in Steindl et al. (2015), Zeitschrift für Psychologie, 223(4), 205–214.
7. Landsberg, M. (1996). The Tao of Coaching. HarperCollins. Source of the skill/will matrix (direct, guide, excite, delegate).
8. Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705–717.

I have banged on before about accessibility working best as a starting constraint rather than a retrofit. This site is where I get to practise that habit continually: my core philosophy is that if something is inaccessible to some it may as well be inaccessible to all. I make sure of this with a pile of small habits, and an audit script that catches the bits I inevitably miss. Part of the series on how this site is built.

The keyboard path

The first element injected into every page is what is known as a skip link. Press Tab on any page here and "Skip to main content" appears before anything else. In short, this allows you to press Enter and get past any navigation fluff at the top of the page. It's invisible to mouse users, and is an old trick that is often forgotten in 'modern' websites.

The Ctrl+K command palette got the full dialog treatment: focus moves into the input when it opens, Tab is trapped (it moves the selection instead of escaping the dialog), Escape closes it, and results are given to screen readers through an aria-live region. The filter buttons on the blog and gallery use aria-pressed, so "Photography, toggle button, pressed" comes through instead of a mystery link.

The theme toggle updates aria-pressed and swaps its label between "Switch to light mode" and "Switch to dark mode", so the button always says what it will do, not what it did.

The screen reader path

Links that open a new tab get a screen-reader-only annotation, WCAG technique G201, applied automatically:

// Appends screen-reader-only "(opens in new tab)" to target="_blank" links
var span = document.createElement('span');
span.className = 'sr-only';
span.textContent = ' (opens in new tab)';
link.appendChild(span);

A MutationObserver reapplies it to links injected later (related posts, footers), because on this site half the content arrives after first paint. For a screen reader user, a link that teleports you into a new tab with no warning is awful.

Images carry real alternative text - I imagine not having it feels almost painful for screen-reader users, they likely feel they're missing out on important content and don't know what it is, like missing pages in a book. I aim for a description, not caption: the photo of my dog is "A picture of my puppy, Brutus", and the chart images in data posts describe the actual bars and numbers, so the information is actually useful, instead of "a photo" or "the results shown as an image". The interactive posts give every canvas role="img" and an aria-label saying what the simulation shows. A label is a thin substitute for "here is a live algorithm race you cannot see", so I'm open to suggestions on how to make it a better experience.

The vestibular path

Three separate systems check prefers-reduced-motion: the scroll-reveal animations (everything just appears, already visible), the homepage stats (they display their final number instead of counting up), and the CSS transitions (three media blocks calm them down). The site has to be complete with the animation subtracted, and only then is the animation allowed.

Checkpoints

Habits decay, so the audit script checks every page for the a11y basics (alt text, heading structure, link health, metadata) on every push, and CI fails loudly when a new post shows how bad my memory is at maintaining these standards without a checklist. I love automating stuff, I'd rather put in extra effort early to save effort later.

I'd love to hear from an actual screen reader user. If you use assistive tech and something here fights you, the contact page works, and I fix reported things fast.

The news is a list of the least representative things that happened yesterday, which you might think - great, that's interesting! The problem is, our minds calibrate to work out how dangerous the world is with this information. Quick: which is more dangerous, flying or driving? Sharks or vending machines? Terrorism or heart disease? If your gut said "flying, sharks, terrorism", your gut is confidently and dramatically wrong.

You judge how likely a thing is by how easily examples come to mind. The examples that come easily are the vivid ones, the ones the news showed you. That, in a nutshell, is the availability heuristic, and it runs your sense of risk, your politics, and a fair amount of your fear.

The shortcut

The availability heuristic, named by Tversky and Kahneman, is the brain's habit of estimating how frequent or likely something is by how readily instances spring to mind^[1]. Most of the time it's a decent shortcut: common things usually are easier to recall than rare ones, so "easy to remember" tracks "actually frequent" well enough to get you through a Tuesday.

It fails when ease of recall is driven by something other than frequency. In the modern world that is constant: vividness, emotional intensity and media coverage all make an event easier to recall without making it any more common.

So the events that dominate our sense of risk are the rare, dramatic, well-covered ones. Plane crashes get wall-to-wall coverage, and car crashes, which kill vastly more people, are too routine to report, so flying feels more dangerous while being far safer per mile.

The risks with no pictures

Shark attacks get a summer of headlines out of a handful of incidents, while the far deadlier risks (heart disease, the drive to the beach) produce no images at all. Terrorism reshapes whole societies' fears and budgets while killing a tiny fraction of the number who die of the mundane, unphotogenic causes nobody panics about. That last one makes me angry.

The mismatch between what we fear and what actually harms us is, in large part, the availability heuristic reading the news as if it were a frequency table^[2].

The news is an availability machine

The news isn't lying, which is the trap. News reports the unusual by definition: "plane lands safely" is not a story, "plane crashes" is. That newsworthiness filter guarantees that what reaches you is the rare and dramatic rather than the common and deadly.

So the news stocks your memory with the vivid, atypical events that the availability heuristic will then treat as frequent ones. You get a mental sample of the world selected for drama and against representativeness, and your sense of risk is built on that sample.

This is a cousin of survivorship bias, and of the habit of analysing whatever data happened to be lying around: in all of them, a filtered sample gets mistaken for the whole picture.

Then there is doomscrolling. An infinite feed that tailors towards what you enjoy - the algorithm literally learns based on how quickly you scroll, or don't, from one item to the next, whether you comment, react, share. So when you show an interest in, well, interesting things that are, say, unusual, such as landslides, violence, politics, terrorism videos, whatever - you begin to see MORE of that because the algorithm learns what you interact with. Therefore, you start to think "this is how things are".

The same wiring that helped an ancestor remember which mushroom caused a friend to die, or where there were dangerous animals, is now allowing you to identify dangerous patterns in a carefully crafted information feed, fed to you from your phone.

What to do about a bias you can't turn off

You can't just "stop" this by knowing it exists, but you can install a correction: when you notice fear, or a judgement about risk, ask where the example came from. Did this feel likely because you have base-rate data, or because a headline handed you a picture? That one question separates "this is common" from "this is merely memorable".

The boring denominators

The deeper practice is going after base rates, the boring denominators, instead of anecdotes. How many flights per crash? How many people drive, versus how many die driving? What's the trend, rather than the latest incident?

Numbers are the antidote because they're unmoved by vividness. A statistic does not care how frightening your mental image is (notice how often journalists say "thousands" without the "out of millions"). So the frightening image is your cue to go and find the statistic.

Your intuition has been fed a curated diet of the dramatic, so when it hands you a confident sense of how risky the world is, treat the confidence as a symptom rather than a measurement.

Next time a headline frightens you, go and find the denominator. It's usually two searches away, and nowhere near as interesting as the headline.

References

1. Tversky, A., & Kahneman, D. (1973). Availability: A heuristic for judging frequency and probability. Cognitive Psychology, 5(2), 207–232. https://doi.org/10.1016/0010-0285(73)90033-9
2. Slovic, P. (1987). Perception of risk. Science, 236(4799), 280–285. https://doi.org/10.1126/science.3563507

Back to all posts

Nobody googles "reasons I am wrong about this". We google "reasons I am right about this", and we smugly nod at how wise we are.

There are dozens of named cognitive biases, and this is the start of a series on them - I find them fascinating and I'm using this series as a means to learn about some psychology 101, and share what I've learned. One such bias sits under the others and is something of a basis for others. Confirmation bias is the tendency to seek, notice, remember and believe evidence that supports what we already think, and to ignore, discount or forget evidence that doesn't. It's how your uncle can keep bringing up the same political arguments at Thanksgiving dinner, forgetting what you've already said that clearly counters his argument - he won't remember, because the information you provide is irrelevant to him.

Evidence

The comforting model of the mind is a courtroom: you gather evidence, weigh it impartially, and reach a verdict. The actual mind runs the process backwards most of the time. You start with the verdict (the belief you already hold, or want to hold) and then work as its defence lawyer, scouring for supporting evidence and cross-examining anything inconsistent into silence.

You search for confirming information. You read ambiguous evidence as supporting your side. You remember the hits and forget the misses. You let confirming evidence in over a doorstep and make disconfirming evidence climb a wall^[1].

Setting the Stage

I want to put two scenes down here, because I am going to keep returning to them throughout the series (or try to, if I can bend the examples enough each time!). Every bias in this run is easier to see when it happens to the same people in the same rooms, so: meet the dinner table and the dashboard. They will be back!

Why intelligence hinders you

Being clever, educated or numerate does not protect you (despite what most of us tell ourselves), and can make things worse. A smarter person is a better lawyer: quicker at generating persuasive arguments for what they already believe, better at spotting the flaw in inconvenient evidence, more fluent at explaining away the anomaly. It's funny how we assume intelligent people are wiser. There's a reason wisdom and intelligence are separate ability scores in pathfinder and dungeons a dragons: intelligence is knowing that a tomato is a fruit, wisdom is not adding it to your cereal.

Studies find that greater numeracy and reasoning ability can increase polarisation on politically charged questions, because people put their cognitive horsepower to work for their existing side rather than for the truth^[2]. The ignorant are overconfident, fine, we all knew that. The capable are better at rationalising, which is not a great thing for society.

The one habit that helps (apparently)

Willpower does nothing here. "Be less biased" is an instruction on the level of "be taller". Confirmation bias runs below awareness, so telling yourself to be fair-minded mostly produces a more confident lawyer. The thing that helps is structural: go and find the strongest version of the opposing case.

Before settling a belief that matters, hunt down the best argument against it, from the cleverest people who disagree, and try to state their case so well they would sign off on your summary. That is steelmanning, and it is the discipline behind how I try to write (and mostly fail at).

Ask "what would change my mind?" and get worried if the answer is "nothing". Watch for the flinch, too. That little urge to wave a piece of evidence away is strong and itself deeply bias - you're doing that with your "tribe", your "side" and your "argument" in mind, not actually fairly comparing evidence. Your defence lawyer has found something and would like it kept out of the record.

But, even with that...

It won't make you unbiased. Nothing does. But - it will insert a step where you point your intelligence towards something more than building walls.

You cannot trust your sense of having weighed the evidence fairly. It is a horrible fact to know about yourself and we're kind of stuck with that fact. What's worse, knowing this doesn't solve the problem. It's like recognizing you having a proclivity for a certain flavor of ice cream doesn't make you like all of them equally suddenly. But maybe it can help you try new flavors, if you take that next step. This metaphor is a bit stretched: what I mean is, try thinking of other perspectives thoroughly. Intellectual empathy; emotional, too.

Go and read the Nickerson review. It came out in 1998.

References

1. Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175
2. Kahan, D. M., et al. (2017). Motivated numeracy and enlightened self-government. Behavioural Public Policy, 1(1), 54–86. https://doi.org/10.1017/bpp.2016.2

Back to all posts

Tabu Search, Live

Eight posts in, every escape from a trap in this series has been some flavour of randomness:

• simulated annealing gambles: while it's hot it accepts bad moves at random and hopes for betterment
• the ant colony wanders its way out
• the genetic algorithm and its program-breeding cousin mutate their way out
• the swarm jitters
• CMA-ES and differential evolution sample their way out of trouble
• even variable neighbourhood search, the most systematic escape artist so far, opens every escape with a random shake

Tabu search is the odd one out, escaping traps by remembering them: it keeps a short list of places it has just been and refuses to go back, even when going back looks like the best move on the board.

Fred Glover sketched the idea in 1986^[1] and built it out properly in 1989^[2]. The core of it is simple: run a greedy search, but make the recent past forbidden. That one addition turns the dumbest search strategy going into something that can walk out of a local optimum, cross a plateau, and keep improving. It needs no temperature and no coin to flip, which makes it something of an oddity in this series.

Greed with amnesia

Steepest descent is the greediest search there is: look at every neighbour of your current solution, take the best one, repeat. It's fast, it's simple, and it's doomed: sooner or later it reaches a solution where no neighbour is better, and there it sits. That was covered that failure in the annealing post, where the cure was randomness on a cooling schedule.

Allow sideways moves (neighbours that are no worse, a common tweak for crossing flat regions) and the search steps from board A to board B, surveys the neighbourhood, and concludes that the best move from B is... back to A.

So it goes back, and then the best move from A is back to B. It will do this until all the coal mines and oil refineries run dry.

This is cycling, the signature failure of memoryless search, where every step the algorithm wakes up fresh with no idea it has been here before. Whether three times or over nine thousand times.

A ban list and a countdown

In tabu search, after each move, an attribute of the position you just left (in our demo, the square a queen just vacated) goes on a ban list for a handful of iterations, called its tenure.

While it's on the list, the search may not move a queen back there, however tempting the move looks, and with the recent past fenced off the only direction available is somewhere new. So the ban list turns "best move" into "best move I haven't just made", a small change that punches through plateaus which hold a hill climber forever.

There is one safety valve: if a banned move would produce the best solution ever seen, the ban is waived. This is the aspiration criterion, and it exists because tabu attributes are deliberately crude: banning a square bans every future board that would use it, including, occasionally, a record-breaking one. When the forbidden move beats anything in the run so far, take it.

Glover's tug-of-war

Glover framed the wider method as a balance between intensification (search hard near the good solutions you already have) and diversification (force yourself into regions you have never visited). Serious implementations layer on frequency memory (penalise moves you make too often), long-term restarts, and cleverer aspiration rules. It all grows from the same seed: the search takes notes on itself, and then obeys them. The demo here is a simple version.

Eight queens, no peace

The playground this time is the N-Queens problem: place N queens on an N×N chessboard so that none attacks another, meaning no two share a row, column, or diagonal. Our queens start one per column, so columns are handled for free, and a move relocates one queen within her own column. The score is the number of attacking pairs, and zero means solved.

It's a lovely testbed for tabu search because the landscape is nothing but plateaus. Near the end you are almost always sitting at one or two conflicts, surrounded by equally mediocre neighbours. That's the terrain where greedy search paces in circles, like me trying to find my car in the multistory car park outside my work (I have the memory of a pigeon with some things).

Below, a tabu search is solving the board live, and red lines join queens that currently attack each other.

The frost is the memory: when a queen leaves a square that square ices over, and the frost fades as its tenure counts down, so you're watching the algorithm's short-term memory dissolve in real time. The readout underneath lists the banned squares with their countdowns. When the search overrides a ban because the move would break a record, an "aspiration!" badge lights up next to the toolbar.

Tenure, at zero and at twenty

Switch the memory off. The search becomes plain steepest descent with sideways moves, and on a lucky board it solves anyway. Most of the time it drops to one or two conflicts and then starts pacing, and the "revisited states" counter begins to climb as it bounces between the same handful of boards, forever, at whatever speed you set. The chart flatlines with a tiny shimmer, which is the cycle itself.

Flip the memory back on and the search usually punches out within a few dozen steps. One counter, and it makes the case for tabu search better than I can.

Set tenure to 0. A ban that expires instantly is not a ban, so this is hill climbing again, cycle and all, even with the memory checkbox ticked. Tenure is the memory. The checkbox just decides whether anyone writes to it.

Set tenure to 20. Now the search forbids so much of its own neighbourhood that it can barely exploit anything. It reaches a promising region, is immediately banned from refining it, and lurches away, sometimes abandoning a one-conflict board it could have finished in a step or two.

Grow the board. Bigger boards mean bigger neighbourhoods and longer plateaus, and the sweet-spot tenure grows with them. Around 7 suits the 8×8 board, and at 16×16 you'll want noticeably more. The literature's rule of thumb is that tenure should scale with problem size.

Remembering the good and remembering the bad

There's some interesting debates I have in my mind about this concept, because it's somewhat philosophical. Our human memories are odd, we often think of periods of time that are difficult and we dislike fondly later in life, like times in the military - we remember the fun moments and forget the 5am wake up time, endless PT and drills. But we also remember specific jarring, horrible moments with great clarity - PTSD is exactly a big warning marker left with the intent to protect us, and let us remember what was bad.

Maybe there's something to take from this: it's actually efficient to remember the bad things, but at some point to let them go, and perhaps similar can be said about remembering the good things, when trying to move towards a better future.

References

1. Glover, F. (1986). Future paths for integer programming and links to artificial intelligence. Computers & Operations Research, 13(5), 533–549. https://doi.org/10.1016/0305-0548(86)90048-1
2. Glover, F. (1989). Tabu Search - Part I. ORSA Journal on Computing, 1(3), 190–206. https://doi.org/10.1287/ijoc.1.3.190

Genetic Programming, Live

Genetic programming evolves programs. What gets bred is the program itself rather than the settings of a program: a working formula nobody wrote, assembled out of worse ones. I've published with this one: the belt-laying search in the Factorio post is a genetic programming system, and it came out ahead of both other methods in the paper.

This is part eight of the series. So far:

• ants have foraged
• triangles have evolved
• a search has cooled its way out of bad valleys
• a swarm has chased its own best guess
• a stuck search has widened its neighbourhood until it came unstuck
• a cloud of samples has learned the shape of a valley
• a population has built its moves out of its own spread

In part two every individual was a fixed-length list of numbers: fifty triangles, ten genes each, five hundred dials for evolution to twiddle. The shape of that genome was decided by me, in advance, forever. Genetic programming throws that away. The genome is a tree of code, and the size and shape of the tree are up for grabs too. Evolution gets the structure as well as the settings.

An infinite haystack

The demo below does symbolic regression. I sample forty points from a hidden target curve, and the population's job is to find an expression that fits them. Individuals are trees built from plus, minus, times, a protected divide (dividing by nearly zero returns 1 instead of exploding), sine, cosine, the variable x, and random constants.

Fitness is mean squared error over the sample points with the lower the better. A formula that outputs infinity or NaN anywhere gets a score so bad it amounts to a death sentence.

Part two's genome was a box with five hundred dials: finite and fixed. The space of all expressions is infinite, and almost all of it is garbage. A random tree comes out like cos(x/x)*0.3-x. There's no gradient to follow and no obvious notion of "nearby".

And yet a population of these things, bred and culled, finds working programs in seconds - which is kinda crazy.

Trading subtrees

The engine is the one you already know from part two: tournament selection, crossover, mutation, and a couple of elites carried into each new generation untouched. What changes is what crossover means. With trees, crossover picks a random subtree in each parent and swaps them. It works because subtrees are meaningful units in a way that random substrings of characters never are: sin(2*x) is a working component. Evolution here is trading parts, not shuffling letters.

The starting population is grown "ramped half-and-half", a mix of bushy full trees and scraggly random ones at several depths (GP-speak for seeding the pool with variety). I also cap tree depth at eight, partly for your device's sake and partly because, as you'll see, GP will take every inch you give it.

Watch a program grow

Below, a population of formulas is evolving against the data. The dots are the samples from the hidden curve, and the green line is the best formula so far, redrawn each generation so you can watch it snap toward the data.

The node diagram is that formula's actual code: blue nodes are operators, amber leaves are x and constants. The readout underneath prints it as maths you can read. Generations tick by a few per second so the motion stays watchable.

The parsimony tax

Set parsimony to zero. Keep one eye on the average size chart. Error improves, then plateaus, while average size climbs and keeps climbing until it slams into the depth cap.

The population is now hauling around programs of a hundred-plus nodes that fit the data no better than a fifteen-node formula from fifty generations ago. This is bloat, the most famous disease of genetic programming and the reason the parsimony slider exists. Why a population would choose to bloat gets its own section below.

Now slam parsimony to maximum. The opposite catastrophe --- every node now costs more fitness than accuracy is worth, so the population collapses to tiny, dumb expressions: a bare constant, a lonely x. They fit the data terribly, but cheaply. Parsimony is a tax, and like any tax, set it too high and nobody builds anything.

Shrink the population to twenty. The run crawls or stalls. With so few trees, the pool of subtrees for crossover to trade is tiny, and once the population homogenises, mutation has to invent every new part from scratch.

And drop mutation to zero while you're at it. Same lesson as part two, sharper here: crossover can only recombine subtrees that already exist somewhere in the population. Once the interesting ones are gone, they are gone.

Bloat is evolved defensive code

Subtree crossover is destructive: graft a random subtree into a random point of a working program and you'll usually break it.

Imagine two parents of equal fitness, one lean, one padded with introns: code that computes something and then multiplies it by zero, or adds a quantity and subtracts it again. When crossover hits the padded parent, odds are it lands in the padding, and the child behaves just like its parent. When it hits the lean one, it hits something that mattered.

Padded parents therefore produce more viable children. Padding spreads. Average size marches upward while fitness stands perfectly still.

The population is evolving resistance to its own genetic operators. No line of my code rewards defensive padding: it falls out of selection pressure, the same way real genomes are stuffed with non-coding DNA and real codebases accrete dead branches nobody dares delete (you don't want to see some of the folders I don't sync to GitHub here locally...).

The trees are, in effect, growing bubble wrap to survive crossover.

Where GP turns up in practice

Symbolic regression is the flagship application and it earns its keep in science. Fit a neural network to experimental data and you get a black box that interpolates. Fit an expression tree and you get an equation a human can read and argue with.

GP-based systems have rediscovered conservation laws from raw motion-capture data, and they get used to propose compact models in physics, materials science and ecology. When you want insight rather than raw predictive power, a ten-node formula beats a million-weight network, contrary to what many machine learning specialists would say.

NASA's ST5 mission needed a small satellite antenna with an awkward radiation pattern, and an evolutionary algorithm designed one^[3]. The result looks like a bent paperclip. No engineer would have drawn it, and it outperformed the conventionally designed alternative.

It flew in 2006. There is evolved hardware in space!

John Koza launched the field with his 1992 book^[1] and then spent years compiling "human-competitive" results: GP runs that reinvented patented circuits and produced patentable designs of their own.

Every one of them lived in a domain with a fast, automatic, well-behaved fitness function. GP never became a general way to write software, and the field guide I recommend below^[2] is refreshingly frank about why. Where a score is cheap to compute and means what you want, GP is a live option. Everywhere else, the fitness function is the really hard part.

Structure is a freedom

Part two's lesson was that the fitness function is where the meaning lives: change what you reward and you change what evolves. Give evolution control over structure as well and it will use that control in every way that pays.

It builds the subtree that fits your curve. It also builds padding to shield that subtree from your own crossover operator, and for the same reason: both help its children survive.

Zero the parsimony and watch the trees fatten in real time, then tax them back down to stumps. Sit through thirty flat generations of the error chart until it falls off a cliff, because two mediocre parents happened to be holding complementary halves of the answer.

An infinite search space, almost all of it garbage, explored by trading parts and taxing waste. It should not work anything like as well as it does.

References

1. Koza, J. R. (1992). Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press. ISBN 978-0262111706.
2. Poli, R., Langdon, W. B., & McPhee, N. F. (2008). A Field Guide to Genetic Programming. Freely available at http://www.gp-field-guide.org.uk
3. Lohn, J. D., Hornby, G. S., & Linden, D. S. (2005). An evolved antenna for deployment on NASA's Space Technology 5 mission. In Genetic Programming Theory and Practice II (pp. 301–315). Springer.

Differential Evolution, Live

This is part seven of the Algorithms, Live series. So far:

• ants have foraged
• triangles have evolved
• a search has cooled its way out of bad valleys
• a swarm has chased its own best guess
• a stuck search has widened its neighbourhood until it came unstuck
• a cloud of samples has learned the shape of a valley

Today's algorithm is for hard continuous problems: differential evolution. It's one of the simpler of the evolutionary family, a few lines of arithmetic wrapped in a loop. I was first properly introduced to it by Prof. Cedric Gondro, at Michigan State University, as it's one of his favorite algorithms, so we were considering its use for genomic prediction problems.

Subtraction as a search strategy

Here is DE/rand/1/bin^[1]: keep a population of points scattered over the landscape. To improve any one of them:

1. Pick three other members at random: \(a\), \(b\), and \(c\).
2. Build a mutant: take the difference vector from \(c\) to \(b\), scale it by \(F\), and add it to \(a\). In symbols, \(a + F(b - c).\)
3. Crossover: build a trial point by taking each coordinate from the mutant with probability \(\mathit{CR}\), otherwise keeping the parent's coordinate. (One coordinate is always taken from the mutant, so the trial is never a clone of the parent.)
4. Greedy selection: score the trial. If it beats the parent, it takes the parent's seat in the population. If not, it is thrown away and the parent stays.

Do that once for every member to create a generation. Repeat!

Every other optimiser in this series had to decide how big a move to make: simulated annealing had a cooling schedule, the genetic algorithm had a mutation rate, gradient descent has a learning rate. DE samples a difference between two of its own members and uses that as the move.

Early in a run the population is sprayed across the whole landscape, so the differences are long and the moves are large. As the population drains into a valley, its members bunch together, the differences between them shrink, and the move size shrinks too.

The difference vectors shrink, and they also point somewhere useful. If the population has settled along a curved valley, the differences between members mostly run along that valley, so mutants get launched along it too.

The population becomes a rough map of the terrain it's sitting on. The state you already have to keep doubles as the search geometry.

Watch the geometry

You probably get this now if you're reading through the series, but here goes: the demo below runs DE on a choice of four classic test landscapes. The coloured ground is the fitness surface (deeper blue is worse, the ground fades toward the background where it is low, and a small cross marks the global minimum). The blue dots are the population, and the green ring is the best point found so far.

Each generation, one member of the population gets its donor construction drawn in full: the three amber dots are \(a\), \(b\), and \(c\), the dashed amber arrow is the difference vector from \(c\) to \(b\), and the red arrow is that same vector scaled by \(F\) and grafted onto \(a\), ending at the mutant (the hollow red circle). The red diamond is the trial point after crossover, ringed if it won its duel with the parent.

Then compare the arrows at generation two with the arrows at generation sixty. Same code, same \(F\), and the moves have shrunk by orders of magnitude because the population did.

F, CR, and a population of eight

Drop \(\boldsymbol{F}\) to 0.1 on the Egg carton. Every move is now a tenth of a population difference, so members can only creep to the nearest dimple. The spread collapses early, usually before anyone has found the central basin.

Then the trap springs. A huddled population can only make tiny difference vectors, so the step size follows the spread down and the search is welded shut. Self-scaling, run in reverse. The best-fitness line goes flat while the spread line dives, and that is premature convergence.

Now push \(\boldsymbol{F}\) to 1.8. Every mutant overshoots the region the population spans, so the population becomes a spray. The spread line stops falling, the mean step stays huge, and best fitness improves only by luck. DE needs the population to contract for its thermometer to read cooler. At \(F\) this high, contraction never begins.

Set \(\boldsymbol{\mathit{CR}}\) near zero on the Banana valley. Each trial now differs from its parent in a single coordinate, so the search moves like a rook: horizontally or vertically, never diagonally. The banana valley is curved, and progress along it needs both coordinates to change together.

Watch the population inch along in tiny axis-aligned stair steps. Then push \(\mathit{CR}\) back up to 0.9 and watch it slide along the valley in coordinated diagonal moves. \(\mathit{CR}\) is the knob that decides whether DE moves one coordinate at a time or as whole vectors.

Shrink the population to 8. The difference vectors are now drawn from a pool of a few dozen possible pairs, so the same handful of directions gets recycled over and over.

Diversity is essential and a tiny population runs out of it almost immediately: fast convergence, frequently to the wrong dimple. The genetic algorithm in part two had the same failure for the same reason, a gene pool too small to explore with.

The missing knob

Put this next to the genetic algorithm from part two and the family resemblance is obvious: a population, mutation, crossover, selection. The GA shuffled bits and letters. DE does arithmetic on real vectors. The GA made mutation from coin flips and a fixed rate you had to tune. DE makes mutation out of the population itself, so the operator adapts as the population does.

And the GA ran a selection lottery where fit parents got more tickets. DE holds one duel per seat, child against its own parent, winner sits. The best member can only be replaced by something better.

Its sophisticated cousin CMA-ES learns an explicit statistical model of where good moves live and updates it with some fairly serious mathematics. DE gets a decent share of the same benefit, steps that stretch along the good directions and shrink as the search closes in, using subtraction.

It is dumber than CMA-ES generally, but very much stands on its own, regardless!

Where DE turns up in practice

Storn and Price introduced it in 1997^[1], it won early optimisation competitions, and it is still a default first try in engineering optimisation: antenna design, power systems, chemical process tuning, anywhere someone has a black-box simulator with a dozen real-valued dials. The survey literature on its variants runs to hundreds of papers^[2].

You can buy adaptivity: learned models, schedules, meta-parameters. Or you can arrange the algorithm so the state it already keeps is the adaptation. DE gets self-scaling search out of a subtraction.

Go and set the Egg carton running, click somewhere daft to strand the population in a corner, and watch the arrows do the rest.

References

1. Storn, R., & Price, K. (1997). Differential Evolution - A Simple and Efficient Heuristic for global Optimization over Continuous Spaces. Journal of Global Optimization, 11(4), 341–359. https://doi.org/10.1023/A:1008202821328
2. Das, S., & Suganthan, P. N. (2011). Differential Evolution: A Survey of the State-of-the-Art. IEEE Transactions on Evolutionary Computation, 15(1), 4–31. https://doi.org/10.1109/TEVC.2010.2059031

CMA-ES, Live

There is a shape of problem that many optimisers struggle with: a long, thin, tilted valley.

Every algorithm in this series so far has moved in steps whose size and shape were settled in advance:

• the ants of part 1 took steps along edges and left scent behind
• the genetic algorithm of part 2 took steps in genome space, one mutation at a time
• simulated annealing in part 3 had a step size you could watch shrink as the temperature fell

This one learns the shape of the neighbourhood worth searching, and restretches itself to match: CMA-ES

Steps versus shapes

Valleys like that are everywhere in real problems (the technical word is ill-conditioned). An optimiser proposing moves in random directions wastes almost all of them climbing walls while gradient descent zigzags across the floor of possible solutions.

This requires an optimiser that notices "movement along this axis keeps paying off, movement along that one never does".

CMA-ES keeps a sampling distribution, a fuzzy ellipse of probability. Each generation it draws a cloud of candidate points from the ellipse, scores them, and shifts the centre toward the better half.

It also updates the ellipse's shape. Directions the winners recently moved in get widened. Directions that never produced a winner get squeezed. It saves the historic data of wins in the covariance matrix.

Instead of a fixed cooling schedule, CMA-ES watches its own recent movement. If successive generations keep marching the same way, the steps were too small, so sigma grows. If the path folds back on itself, the steps were too big, so sigma shrinks. Exploration vs exploitation.

Hansen and Ostermeier formalised all of this in 2001^[1], and the design is careful enough that the standard version has no parameters you are expected to tune. But, in this version, for fun, there are sliders so you can override that.

The loop on a napkin

1. Sample a cloud of candidate points from a Gaussian: centre (mean), scale (sigma), shape (covariance).
2. Score every point and rank them.
3. Move the mean toward a weighted average of the best few, best weighted most.
4. Stretch the covariance along the directions those winners came from.
5. Grow or shrink sigma depending on whether recent moves kept pointing the same way.
6. Go to step 1.

Watch it

Below, CMA-ES is minimising a landscape drawn as contour shading (darker is worse, the red cross is the global optimum). Blue dots are one generation's samples, the amber rings are the search ellipse at one and two sigma, and the green dot with the fading tail is the mean and the route it took.

Start with the default narrow valley and watch the ellipse for the first few seconds. It begins as a circle, tips over to match the valley's angle, and stretches. Click or drag anywhere on the map to plant the start point somewhere new.

The step-size rule

On the smooth valleys the run still works, it just crawls. The step-size rule notices the steady progress and inflates sigma back to something sensible, which you can watch happening on the right-hand chart.

On the egg carton a tiny sigma is fatal. The cloud fits inside a single dimple, every sample agrees the local floor is the best thing going, and the run settles into the nearest dent.

If the sigma is huge, the first generations are chaos.

Then watch the sigma chart. The path the mean traces keeps folding back on itself, the adaptation rule reads that as "steps too big", and sigma falls generation after generation until the cloud shrinks onto something worth exploiting. Thrashing and then recovery is the self recovering step size.

Drop lambda to 6 and each generation's ranking is built from almost nothing. The mean update gets noisy and the covariance estimate gets noisier.

Back to the narrow valley, default settings, and this time watch only the amber ellipse and the stretch readout. Circle, tilt, stretch, and then a needle sliding along the valley floor with the stretch number climbing into the hundreds.

Every other algorithm I've covered in this series would be paying the valley's toll on every single move, but CMA-ES redrew its own geometry so there is no toll left to pay!

A memory of directions

What I like most about CMA-ES is where it keeps what it has learned. The ants kept their knowledge in the world, as scent on edges. The genetic algorithm kept it in a population, spread implicitly across genomes: variety, but no geometry. Simulated annealing had one knob, a step size, and its step size only goes down.

CMA-ES keeps a small, dense summary: one point for "where", one number for "how far", and one matrix for "which way". The matrix is the difference. It is a running record of the directions that recently succeeded, updated every generation, and it turns a search that happens in the landscape into one that builds a model of the landscape's local shape.

Where it gets used

For continuous black-box problems, where the objective is a simulator or a physical measurement or a hyperparameter score with no gradient in sight, CMA-ES is the default serious tool up to a few hundred dimensions (yes, you read that correctly!!).

It has tuned aircraft geometry, robot gaits, controller parameters, and the settings of other learning systems. Hansen's tutorial^[2].

When progress stalls, the instinct is to push harder in the direction you were already going. CMA-ES keeps an estimate of which directions have been rewarding you lately, and spend your effort along those, even when they aren't the ones you set off in. Which is surprisingly good life advice to get from a pile of linear algebra.

References

1. Hansen, N., & Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9(2), 159–195. https://doi.org/10.1162/106365601750190398
2. Hansen, N. (2016). The CMA Evolution Strategy: A Tutorial. arXiv preprint arXiv:1604.00772. https://doi.org/10.48550/arXiv.1604.00772

Factorio, Live: a Game as an Optimisation Benchmark

Factorio is a game I've loved since it was released. It's my most-played game on Steam, and there's something so deeply satisfying about playing a game all about optimization, exploring and solving problems. When I started thinking about the psychology behind this, I realized that video games, puzzles, board games and the like mirror a satisfaction I find when programming, conducting research and solving real-world problems. It scratches an itch in a particular way unfound in other walks of life: defeating challenges beyond a grind in an RPG, lifting a heavier weight at the gym or accomplishing projects based more on time than difficulty. It's the slow accrual of problems solved into a portfolio of solutions working in tandem, so you can zoom out to the map and say "look at my base! it's a bit of a mess, but it's mine, and I love it."

I decided to bridge the gap between the video game and the research, both of which scratched that itch in my mind. I brought up the idea with one of my supervisors at Michigan State University (MSU) at the time, Prof. Wolfgang Banzhaf. He thought it sounded interesting, and liked my point that a video game is a better teaching instrument than a textbook, since this visually shows success and failure, and the mechanics of individual algorithms live in front of you. He said I should go for it, so I enlisted a couple of my colleauges (now co-authors) and we worked on it together.

Watch it lay the belts

Two places this demo is not the paper

The scoring here is not identical to the published version. The paper scored a candidate by running it in the game, which returns nothing at all until the route lands. That cliff is survivable for a method that lays whole paths in one move, and hopeless for one that edits a single cell at a time: annealing would sit at zero for thousands of evaluations and look broken. So the demo adds partial credit for reaching further east, and a small bonus for using fewer belts once connected. It is a shaped objective, and shaped objectives can be gamed, so this is really just for demo purposes, not a realistic solution (mostly since "east" isn't always the goal).

The second departure is in how annealing picks what to change. The paper varies its neighbour selection with iteration progress. Here, two thirds of edits are at the frontier, meaning the cell where the current route gives up. The reason is arithmetic: on a twelve by twelve grid, a uniformly chosen cell is almost always somewhere the route never reaches, so most edits are invisible to the score and the search spends its budget rearranging scenery. Both changes exist to make the comparison fair to annealing, which is the direction a bias should run if you are going to have one.

So: this is a pretty and intentionally non-representative demo of the paper.

What the paper actually did

The paper behind this is The Factory Must Grow: Automation in Factorio, written with my co-authors Iliya Miralavy, Stephen Kelly, Wolfgang Banzhaf and Cedric Gondro, and presented at GECCO in 2021. The starting point was that Factorio is a factory simulator that has already done the expensive part of building a benchmark: it has consistent physics, a fixed belt throughput, and a community of players who have spent years discovering what good layouts look like, so how would our algorithms fair?

We defined the logistic transport belt problem formally, as an integer program with hard constraints on what can occupy a cell and how underground pairs must line up, and two soft constraints to be maximised: items delivered to the receiver, and items accepted from the chest. Then we built the 'Factorio Optimizer Interface', which uses the game's Lua modding hooks and the RCON protocol to place a candidate layout in a running Factorio server and read back how many items the input inserter took and how many reached the output chest. Because it talks over a socket and reads matrices from disk, the optimiser can be written in anything: ours were in Java, Python and C++. The evaluation function is the game itself.

Three optimisers were compared on six instances, being three grid sizes at three by three, six by six and twelve by twelve, each with and without obstacles. Parallel simulated annealing, a genetic programming system whose operators place and connect belt runs rather than editing cells, and an evolutionary reinforcement learning approach. The small instances are solvable by anything. The twelve by twelve with obstacles is where things really got interesting.

Genetic programming did well here because its operators speak the language of the problem: laying a run of belts and connecting two points are single moves, so one step of the search covers ground that costs annealing a long chain of individually pointless edits.

The model

Section 4 of the paper states the problem formally. Two hard constraints, two soft ones, and a pair of binary decision variables. Throughput, routing and flow is absent, for a reason I will come back to.

A position \(p\) is a coordinate pair in the grid. Directions are \(d \in \{0,1,2,3\}\), meaning north, east, south and west. Belts and underground belts are drawn from \(L_b\) and \(L_u\), and \(\phi_p\) is 1 when position \(p\) holds an obstacle and 0 otherwise. The decisions are whether to place a given belt type, at a given position, facing a given way:

The objective maximises the sum of the two soft constraints, each a parameterised item count scaled by a weight (not as in how heavy it is, but a weighting as in scaling), where \(o\) is the number of items output and \(i\) the number input:

HC1 allows a position to hold one logistic object or one obstacle, never both and never two. HC2 requires at least one item to be placed at all. A solution failing either scores zero; otherwise fitness lands between 0 and 1.

HC1 is the constraint that makes this combinatorial. Every cell you spend is a cell you cannot spend on anything else, and the obstacle term means the grid's difficulty is set before the search starts. HC2 exists to rule out the empty solution, which would otherwise be feasible and uninformative.

There is no flow conservation here, no belt capacity, no constraint saying a route has to connect. The underground pairing rule, the one that requires an entrance and exit in the same row or column, facing the same way, at most four cells apart, is described in the paper as a property of the game rather than written as an inequality.

All of that is missing because the model does not evaluate solutions. The game does. A candidate is built in Factorio through the interface, run, and scored on what the receiving chest actually contains, so the physics stays where it already worked instead of being restated as constraints that could disagree with it. That is the paper's real trick, and it is why the formal model can afford to be four lines long.

It also explains the weights. With \(w_2 = 0\) the objective would count only what arrives, and every layout that fails to connect would score identically at zero, leaving a search no way to tell a near miss from an empty grid. Because \(SC_2\) pays for items merely taken out of the input chest, a belt that starts correctly and dead-ends still earns something. That is a modelling decision rather than an algorithmic one, and it is the difference between a landscape a local search can climb and a cliff it cannot.

The niche

Optimisation research runs on benchmark sets, and benchmark sets have a known failure mode: they get old, methods get tuned against them, and after a decade you cannot tell whether a new result is progress or overfitting to a fixed library of instances. Games are an interesting escape from that, because a game that people play is a simulator someone else has already paid to build, validate and keep interesting.

Factorio is a good one for the specific reason that its difficulty is combinatorial rather than reflexive. Nothing here depends on reaction time or hidden information., like many games: it is placement under constraints, with a throughput number at the end, which is the same shape as facility layout, circuit routing and warehouse design in real life. It also comes with something no synthetic benchmark has: a large population of humans who have independently optimised the same problem for fun, and whose solutions you can compare against.

This is one problem inside one game, with a single objective and an evaluation function that a commercial studio can change in a patch. A benchmark you do not control is a benchmark that can be revised out from under your published results. And the community solutions that make it appealing are also a contamination risk, since a method tuned by someone who knows the good patterns is not discovering them. Games are a useful source of problems, not a replacement for the discipline of saying what your benchmark can and cannot show.

References

1. Reid, K. N., Miralavy, I., Kelly, S., Banzhaf, W., & Gondro, C. (2021). The Factory Must Grow: Automation in Factorio. Proceedings of the Genetic and Evolutionary Computation Conference Companion. https://arxiv.org/abs/2102.04871

Someone on Cyberspace (new social media website that has such a cool vibe to it) recently asked me how I set up the simulators on my website, if I used a specific library. I responded saying how it's raw javascript, but that got me thinking: I intend to keep making these kind of educational and fun blogs, so why not make a little framework to make it easier on myself?

The algorithms themselves need to be separate, I wasn't planning on making a new JMetal, DEAP or similar. Part of the series on how this site is built.

The CSS owns the colours

I always enjoyed color theory in computer science. The RGB values, pixels (did you know it stands for "picture element"?), bitmaps vs JPEGs vs other formats, it's super interesting to me. On this website, the theme toggle cannot restyle it the way it restyles text, and the first version of the ant post hardcoded its colours in JavaScript and looked wrong in one theme or the other. Every widget's colours live on its wrapper element as CSS custom properties, with a dark-theme override block, and the drawing code reads them fresh at draw time:

function vizColors(el) {
  var cs = getComputedStyle(el);
  function v(name) { return cs.getPropertyValue(name).trim(); }
  return {
    series: [v('--viz-s1'), v('--viz-s2'), v('--viz-s3'), v('--viz-s4')],
    ink: v('--viz-ink'), muted: v('--viz-muted'),
    grid: v('--viz-grid'), surface: v('--viz-surface')
  };
}

new MutationObserver(function () { draw(); })
  .observe(document.documentElement, { attributes: true, attributeFilter: ['data-theme'] });

The observer redraws the moment data-theme gets toggled, so a running simulation changes palette mid-run. This has no practical value whatsoever.

Randomness must be reproducible

Math.random() is banned from the demos. Every widget gets its randomness from mulberry32, a seeded generator:

function mulberry32(seed) {
  var a = seed >>> 0;
  return function () {
    a |= 0; a = (a + 0x6D2B79F5) | 0;
    var t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

When a reader reports that the race chart does something odd around 90,000 evaluations, I can watch the same 90,000 evaluations they did. "New cities" buttons just bump the seed.

Phones are the real test

No canvas ever gets a fixed rendered height now: height is computed from measured width, so narrow screens get a taller aspect instead of a squashed ribbon. Side-by-side panes stack vertically below ~480px, and sliders become full-width rows on small screens.

A canvas only gets touch-action: none if it has drag interaction, so chart canvases never trap a reader mid-scroll.

And because rules I merely remember are rules I eventually break, the test suite enforces them. The smoke test auto-discovers every blog page containing a <canvas> tag (no registration list to forget to update) and loads each at 360px wide in a headless browser. It fails the build on horizontal overflow, a canvas wider than the viewport, a canvas squashed under 100px tall, or any console error.

The chassis checklist

What every new interactive post starts from:

• A wrapper div carrying the --viz-* palette, light values plus a dark override block.
• vizColors() read at draw time, plus a MutationObserver for theme flips.
• fitCanvas() handling devicePixelRatio and re-running on resize.
• mulberry32 for every random draw, seeds visible near the top of the file.
• A requestAnimationFrame loop with a speed slider controlling work per frame, not frame rate.
• role="img" and a real aria-label on every canvas.
• A standalone single-file version of the demos in blog/downloads/.

My homepage shows my scrobble count, the book I'm working reading through, the last song I played, and whatever I last posted on Bluesky. This is a static site, meaning it can't just run dynamic scripts on a whim. I set up a system to dynamically update my HTML weekly, however, and it works for free. Part of the series on how this site is built.

The idea: if the site can't fetch, fetch into the site

A normal website with live data has a server that queries things when visitors arrive. A static site can't do that, but it has a loophole: the site is a git repository, and anything that can commit to the repository can change the site. So instead of fetching data per visit, a scheduled job fetches it once a week and commits the results as plain JSON files. The site stays static. Visitors' browsers read those JSON files with an ordinary fetch: same origin, no keys, no rate limits, nothing that can fail separately from the site itself.

The whole pipeline is one workflow file and one Python script. The workflow is short: check out the repository, run the script, and commit if anything changed:

name: Refresh live data

on:
  schedule:
    - cron: "0 6 * * 1"   # weekly, Mondays 06:00 UTC
  workflow_dispatch:

permissions:
  contents: write

jobs:
  refresh:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Fetch live data
        env:
          LASTFM_API_KEY: ${{ secrets.LASTFM_API_KEY }}
        run: python .github/scripts/refresh_now.py

      - name: Commit if changed
        run: |
          FILES="data/lastfm.json data/now.json data/lastfm-history.json data/topalbums.json"
          if git diff --quiet $FILES; then
            echo "No change."
            exit 0
          fi
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git add $FILES
          git commit -m "Refresh live data (Last.fm, Goodreads)"
          git push

The workflow_dispatch line adds a manual "run now" button in GitHub's interface (which you will want the first dozen times), while the diff check means runs where nothing changed produce no commit at all.

What the script gathers

The Python script makes a handful of HTTP requests. From the Last.fm API: my total play count, the most recent track, and my top albums of the past three months, which feed the homepage stat, the "now playing" strip, and the album wall on the music page respectively. From Goodreads, which shut its API years ago but still publishes RSS feeds: my currently-reading shelf, parsed straight out of the XML. Everything goes into a small JSON files under data/, the biggest a few kilobytes.

Each run appends that week's play count to a rolling log, capped at ninety entries, which at one measurement a week is close to two years of history.

What the homepage shows

The strip under the hero is a single row of five things, and only three of them come from the robot. The book and the track are read straight out of now.json. The latest blog post and the latest short story come from posts.json and stories.json, which are written by build scripts when I publish rather than by the Action on a schedule.

Last.fm distinguishes a track that is playing from one that was played, so the label switches between "Now playing" and "Last played" depending on whether I happen to be listening when you open the page, and the first version gets a little animated equaliser beside it. It is not accurate, because it is updated so infrequently, but it gives some flavor to the site and over time it's likely representative of me. Maybe.

The fifth item, my most recent Bluesky post, is in no JSON file at all. Your browser fetches it from Bluesky's public API while the page loads. I've been tempted to remove it, as recently I've only posted new blogs to social media, but I hope I will return to posting random thoughts there again soon, making this a bit more useful than another place to see a new blog post (which is already on the index page!)

Design decisions

The homepage HTML contains a hardcoded scrobble count, and the JavaScript replaces it with the fresh value from the JSON after load. If the fetch fails, or JavaScript is off, the visitor sees a slightly stale number instead of a blank. The robot improves the site but it fails elegantly.

Goodreads' RSS goes down more often than Last.fm's API. When a source fails, the script keeps whatever the previous run wrote rather than blanking it: better yesterday's truth than today's error. A failure in one source never stops the others from updating, of course.

The Last.fm key lives in the repository's encrypted secrets and reaches the script as an environment variable. It appears in no file and no log for security reasons (not that I'd be massively put-out if someone broke into my goodreads or last.fm).

The bot's commits are labelled as bot commits, touch only the data files, and say what they did. My git history has a weekly heartbeat in it now: a tidy row of "Refresh live data" commits, one every Monday, each one the robot clocking in so I don't have to manually update stuff.

What else this pattern is good for

Anything that changes slowly and comes from somewhere with an API or a feed: your latest posts elsewhere, sports scores, weather, stars on a project, prices you're tracking, a "days since" counter for whatever you are currently ashamed of. The recipe is always the same three steps: a script that fetches and writes JSON, a workflow that runs it on a schedule and commits, and a page that reads the JSON with a fallback. If the data changes faster than a schedule can keep up with (live chat, comment counts), a static site is the wrong tool and no robot can easily fix that. Everything slower works just fine though.

Back to all posts

Variable Neighbourhood Search, Live

Every algorithm in this series so far has had its own way of not getting stuck:

• the genetic algorithm kept a whole population, so one individual getting stuck didn't matter
• simulated annealing tolerated bad moves while it was hot
• the ant colony let old trails evaporate
• the particle swarm carried enough momentum to sail over a dip before it noticed

This fifth one runs without a temperature schedule, a population, a pheromone trail, or a velocity. When it gets stuck, it changes what "nearby" means. This one hits home because, like many of the algorithms I am blogging about, I am making use of code I wrote years ago - but this one is special, because I used VNS for my first ever published research paper.

What does 'stuck' mean?

The playing field is the same Travelling Salesman Problem as the annealing post: a scatter of cities, and we want the shortest round trip that visits each city once.

Local search on the TSP is very simple: take any tour, try a small edit, keep the edit if the tour got shorter, repeat. The classic small edit is 2-opt: snip two edges, reconnect them the other way, which reverses a stretch of the tour and uncrosses one of those ugly X shapes. Do that greedily and a random tangle collapses into something pretty good, pretty quickly.

But, inevitably, it stops. You reach a tour where no single 2-opt move helps, a local optimum, and are unable to move. In the annealing post I treated that as a fact about the tour: it's stuck. What is stuck is the tour plus this particular definition of nearby.

A 2-opt local optimum only means that none of the roughly n² tours reachable by one segment reversal is shorter. Offer a different move, say lifting three consecutive cities out and reinserting them elsewhere (Or-opt), and the same tour suddenly has a completely different set of neighbours, some of which may well be better.

Mladenović and Hansen built this idea in 1997^[1], called it variable neighbourhood search, and it has been a fixture of the routing and scheduling literature ever since.

A local optimum is a property of the neighbourhood, not of the solution.

I'm gonna need a bigger hammer.

Basic VNS keeps a list of neighbourhood structures, ordered from smallest to largest, and climbs that list on an "as needed" basis. Here is one VNS setup:

• N1, 2-opt kick: reverse one random segment. The lightest nudge.
• N2, Or-opt: pull out a chain of one to three cities and reinsert it somewhere else.
• N3, segment exchange: swap two short segments of the tour with each other.
• N4, double bridge: cut the tour in four places and reassemble the quarters in a different order. A big kick. It changes four edges at once.

That is to say: descend with plain first-improvement 2-opt until you hit a local optimum. That's your incumbent, so set k to 1. Now: shake the incumbent with a random move from neighbourhood N_k, descend again with 2-opt from wherever that landed, and compare. If the re-descended tour beats the incumbent, move there and reset k to 1. If it doesn't, throw it away and increment k, up to k_max, then wrap back round.

This runs without a temperature, an acceptance probability, or a crossover operator. Hansen and Mladenović later distilled the design principles in a survey which is a great starting point ^[2].

This means the algorithm, by its very nature, will reach for a bigger hammer, as needed. The moment anything works, it drops straight back to the smallest hammer.

Watch it climb the ladder

Below, VNS is solving a seeded random TSP instance live, and the lit badge under the map is the neighbourhood currently in use. Watch it during a rough patch. N1 lights, fails. N2 lights, fails. N3 lights, and then an escape lands, and it snaps straight back to N1. The right-hand chart records that as a sawtooth: every tooth is the search getting stuck, escalating, and being rescued. Amber flashes on the map are the edges the latest shake or improvement just changed.

The Sliders

Tick "plain local search". This switches the shaking off entirely: descend with 2-opt, then stop, and the tour untangles, hits its first local optimum, and freezes there forever, while the k chart flatlines like in that medical drama my partner watches.

Set k max to 1. Now the search still shakes, but only ever with the smallest kick, a single random segment reversal. Watch the escapes counter crawl: a small kick barely leaves the current basin, so the 2-opt descent usually slides straight back into the tour it just left.

The gap stat twitches and recovers, twitches and recovers, going nowhere. It is all motion and no progress.

Now give it the full ladder, k max at 4, and watch the cost side. Every time N4 fires, the gap stat spikes hard, because a double bridge tears four edges out of a good tour, and the descent then spends a long run of moves stitching the damage back together.

That work is only worth it when nothing smaller can move the search, so VNS keeps the big hammer as a last resort.

Change what nearby means

I like the contrast with the earlier posts in this series: simulated annealing keeps one move type and varies when it will accept a worse solution. The genetic algorithm keeps a population, so the question of any one solution being stuck barely arises.

VNS, in its basic form, never accepts a worse solution at all, runs a single solution, and the only thing it varies is the move set. Three completely different answers to the same trap.

References

1. Mladenović, N., & Hansen, P. (1997). Variable neighborhood search. Computers & Operations Research, 24(11), 1097–1100. https://doi.org/10.1016/S0305-0548(97)00031-2
2. Hansen, P., & Mladenović, N. (2001). Variable neighborhood search: Principles and applications. European Journal of Operational Research, 130(3), 449–467. https://doi.org/10.1016/S0377-2217(00)00100-4

Particle Swarm, Live

In 1995, a social psychologist and an electrical engineer sat down to animate a flock of birds and accidentally invented one of the most popular optimisers of the next two decades, which is a story I love. I really believe that error is one of the most fruitful forms of novelty creation. Below, you can fly the result yourself. It's the sibling of the ant colony post: another swarm, another crowd of simple agents solving a problem none of them can see individually, with the communication channel turned inside out.

In Ant Colony Optimisation, the ants coordinate through the environment, and no ant knows anything whatsoever. The knowledge lived in the pheromone map, written and read by thousands of insects who are basically automatons. Particles do the opposite, and each one privately remembers the best spot it has ever stood on, while the whole swarm shares a single piece of gossip: the best spot any member has ever found. That's the entire communication system, and it runs without a trail, a map, or a leader, just a telepathic link of where to find the dropped ice cream or sugar granules.

The flock that forgot how to fly

James Kennedy was a social psychologist and Russell Eberhart was an electrical engineer. Their 1995 paper^[1] opens with birds rather than mathematics. They wanted to simulate the choreography of a flock, in the tradition of the boids models, where each simulated bird matches its neighbours' speed, avoids collisions, and stays near the roost. They started deleting rules to see which ones mattered.

Collision avoidance went, velocity matching went, nearest-neighbour tracking went. What survived would not fool a birdwatcher, or a pigeon, or a cat, just dots drifting toward places that had scored well. But the dots were doing something more interesting than flocking: they were finding the optimum of whatever function defined "good". The choreography fell away and the optimiser remained, and Kennedy and Eberhart named the agents particles, "birds" probably no longer felt right.

The version below includes one refinement from 1998, when Shi and Eberhart added the inertia weight^[2]: a knob controlling how much of its current velocity each particle carries into the next step.

Three kinds of pull

Every step, each particle updates its velocity as a weighted sum of three forces, which, to me at least, feel kind of like personality traits:

• Habit (w): keep doing what you were doing. The inertia weight scales the particle's current velocity.
• Own memory (c1): a pull back toward the best place you have ever found. Call it self-trust, or nostalgia.
• Peer pressure (c2): a pull toward the best place anyone has ever found. Conformism, sheep-ery.

The two social pulls are each multiplied by a fresh random number every step, so no two particles are likely to respond the same. That randomness is what keeps the swarm from collapsing into a single deterministic trajectory.

The result is clamped to a maximum speed, the particle moves, the function gets evaluated at the new spot, memories get updated, and the loop repeats. That's it!

Watch the swarm hunt

Below, a swarm is minimising a two-dimensional landscape, where warm colours are low ground (better) and the dashed ring marks the true optimum, which the swarm does not know about. Watch the characteristic dance: particles overshoot the best-known point, get hauled back, overshoot the other way, and the whole cloud contracts around the answer like a slowing pendulum.

Turning the swarm into a stampede

Crank inertia past 1. Above w = 1 a particle's velocity grows every step instead of decaying, so the only thing stopping the swarm from leaving the map is the speed clamp. The swarm boils, with particles orbiting the best-known point at full speed, permanently overshooting, while the spread chart flatlines high and the fitness chart stalls on momentum.

Now do the opposite: inertia near 0, peer pressure at 3. This is the stampede: with no momentum to resist, every particle dives straight at the first decent global best, the spread chart crashes within a few dozen iterations, and the swarm becomes one throbbing dot.

On the bowl this looks like efficiency, but switch to the egg carton and watch the same setting get eaten: the swarm piles into whichever dimple it saw first, the true optimum sits two ridges away, and nobody has enough independence left to go and look. Premature convergence.

Set own memory (c1) to zero. Pure conformism: particles have no attachment to their own discoveries, and the global best is the only guidance.

Set peer pressure (c2) to zero. Now nobody shares anything, and you get thirty hermits running thirty independent local searches, each rattling around its own personal best and improving it a little.

Now tune it back

Then go back to the classic setting (w = 0.72, c1 = c2 = 1.5, numbers the field converged on after years of tuning) and watch the dance it buys you: overshoot, return, contract, with just enough disagreement to keep a few scouts out wide while the core closes in. On the egg carton it will still fail sometimes. The demo isn't broken, that just shows that it's not perfect, and there are still papers being published solving more complex landscapes with PSO.

Personality knobs

Kennedy never stopped pointing out that this is a model of social learning first and an optimiser second: individuals keep private experience, the group broadcasts its best result, and behaviour is a compromise between habit, self-trust and conformism.

Every failure mode you just created is a recognisable social failure. The high-conformism swarm is an echo chamber stampeding to its first confident answer. The zero-gossip swarm is a room full of talented people who never talk. The overheated swarm is a committee that keeps relitigating and never makes an actual decision (sounds like many meetings I've been in).

Annealing, evolution, ants, particles

The through-line of the series:

• simulated annealing was one searcher learning when to accept a worse answer
• genetic algorithms were a population improving by dying
• the ants stored the answer in the world
• and the particles store it in each other

Four strategies, one shared problem: explore enough to find the right valley, exploit enough to reach the bottom of it. See some interesting overlaps in strategy, parameters, landscape advantages and disadvantages?

Anyway. Go scoop up the swarm, drop it in a corner of the egg carton, and watch thirty dots argue their way out. Or fail to.

References

1. Kennedy, J., & Eberhart, R. (1995). Particle swarm optimization. Proceedings of ICNN'95 - International Conference on Neural Networks, 4, 1942–1948. https://doi.org/10.1109/ICNN.1995.488968
2. Shi, Y., & Eberhart, R. (1998). A modified particle swarm optimizer. 1998 IEEE International Conference on Evolutionary Computation Proceedings, IEEE World Congress on Computational Intelligence, 69–73. https://doi.org/10.1109/ICEC.1998.699146

50 Blogs: An Infographic

This is post number fifty. That felt like the kind of milestone that deserved a party, and since this is a data science blog at heart, the party includes confetti, a banner and a bunch of unnecessary plots.

Everything below is drawn live in your browser from the same JSON file that runs the blog itself, so the infographic *should* stay up to date. If you are reading this long after publication and the numbers have crept past fifty, you can consider this post "blog emeritus".

Fifty is also the first point where the archive is big enough to start exhibiting some shape, behaviors and such. So between the charts I have tried to be straight with myself about what this blog turned into: what I enjoy writing about most, what people read, and the discovery that, uh, they're not always teh same thing.

One hundred and thirty-two thousand words. That's kind of crazy. That's a decent-sized novel and a half, mostly about whatever my brain wouldn't let go of and I didn't want to bore my wife or colleagues with (or at least not more than I already had). The fourth tile is the one that I have mixed feelings about: I like having so many draft ideas, some just bullet points or a few scattered paragraphs, it makes it easier to choose a topic I feel like when I have 15 minutes to write. But it also means my drafts just grow and grow, and I already know there are about 5 blogs I wrote out, went to revise, decided I didn't like them and deleted them. So I sometimes wonder how many drafts won't make it to the end, but I suppose that just comes with the process.

The publishing rhythm

I started in April: an opening sprint, a spring lull while I travelled to Colorado, and then July happened. Admittedly, a lot of my posts are shorter format, or reuse code I wrote years ago for demonstration purposes, making fifteen blogs in one month less crazy than it sounds. August is only a week old, but I don't think it'll grow much looking at my calendar.

Zoom in: this is every day since the first post, one cell per day. GitHub feel.

Mondays are the most common publishing day, which makes sense, because I like to write at the weekend mostly then try to publish when I think people are more likely to click while bored on a commute or during a lunch break. 12 July, when four posts went out in one day, was not intentional. I had planned to publish 1 and draft the others but I accidentally had them in the queue for publishing, thankfully they were mostly done.

Tags and what they say about me

Tags are categories of a sort, and the totals appear to be something of a personality test. Technology and personal are tied at the top, which fits thematically about what this site is: half machine and half diary. Books outnumbering television which appears once doesn't surprise me: I read a lot more than I watch TV or movies.

I started this blog assuming it would be somewhere I could talk vaguely about whatever I feel like, but would probably end up just being a professional portfolio of some kind. I've since learned that the posts I am most eager to write, and proudest of afterwards, are the discursive ones: the philosophy and the social questions. They're the most fun, encourage the most discussion with friends and people who read them.

The clearest evidence is that Everyday Ethics became a series retroactively. I wrote four essays about responsibility, prevention, tolerance, and supply chains over three months without once noticing they were the same project, and only recognized that once I hit blog 50. I've added other blogs into series, and made pages dedicated to series now, too. I think this is healthy: some topics deserve returning to, or expanding upon, or link nicely together so readers might enjoy them in a specific order.

The same tags get more interesting when you spread them over time. April was books and AI, May and June drifted personal, and July was the month the technology tag ate the blog, because that was the month I rebuilt half the site and wrote about what I did (and more are in drafts!). This wasn't planned at all: topics tend to come and go on whims.

Tags by hours

There is one more way to slice the tags: by reading time instead of post count. Counted this way the leaderboard flips: technology has the most posts but they are quick reads. The personal essays are where the hours actually live, with books close behind.

Fifty dots

Every post on this site is a dot. Time on X, word count on Y, and its main tag for the colour. The band between one and three thousand words is my default length it seems. The dot at the top is the accessibility post at 9,015 words. The one scraping the floor is an 802-word story about ending up in Italy through mistaken identity.

The first twenty-five posts have a median of 2,621 words. The second twenty-five have 1,818. I am expressing brevity over time, unintentionally.

The pile of words

What gets read

These are all-time page views from analytics for the most-read posts, and lets address the Cthaeh in the room: one post about a fantasy novel that has not been published in fifteen years has collected more readers than everything else on this site combined, several times over. Yeah, one post went viral, so my stats are a bit iffy.

I wrote it in one long sitting. The 9,015-word accessibility chapter I laboured over doesn't even crack the top five in views (reminds me of my thesis now that I think of it). There is a lesson in there about audiences, and I am choosing not to learn it, because I will just continue to write whatever I feel like regardless.

Every post with tracked views, positioned by how many words I wrote against how many people read them. The vertical axis is logarithmic (because of the Rothfuss article). If effort predicted readership this would be a rising diagonal line. It is not a rising diagonal line.

A footnote for fellow data people: the Rothfuss number is actually two numbers. I renamed that post once ("fourteen years of silence" became "fifteen years of silence", I somehow failed to count and published before realizing my mistake), and its old address kept collecting readers for months. The figure above merges both.

Elsewhere on the internet

There is also a readership I can only see from a distance. On Reddit, there is a discussion thread about the Rothfuss blog, and it has now been viewed 300,000 times.

For scale, that is roughly eight times every page view this site has ever recorded, across all fifty posts and every other page, combined. The most-read thing connected to this blog is a forum thread about the blog. Shows how often people read a headline and discuss it on social media, and how few actually click the source material.

A decent share of the twenty-eight thousand almost certainly walked over from there, which makes my biggest referrer a comment section full of people waiting for a book.

The small print

Across the 118 days between the first post and the fiftieth, the blog averaged 1,122 published words per day, counting the days I wrote nothing. I think this is largely due to the fact I write some every day, or at least edit, and when I write I tend to write a lot, perhaps an entire draft, and create a couple other drafts I have for ideas for future posts at once.

Titles run from 16 characters ("Ant Colony, Live") to the 105-character Rothfuss monument, and only two of the fifty end in question marks, so apparently I prefer telling to asking.

What I've learned doing this

What I love writing and what gets read are different. The views chart is ruled by a post about a fantasy novel, and the essays I care most about live in its shadow.

The mailbag tells the other half of it. The one email about a non-Rothfuss post was about the most discursive thing on this site, from a reader on another continent who found it by googling an idea (also neat that I come up on a Google search!). Search traffic arrives in crowds for the topical stuff, and the philosophical stuff gets found by one person at a time, but they actually write in and discuss the ideas with me. That was so nice.

Volume was never the goal: I'd rather have a few people engage than thousands who don't.

So the next fifty will lean harder into the discursive essays, because I enjoy it. The drafts folder is stacked with cognitive biases, ethics, and questions about meaning, far more than gadgets and code. The technology posts will keep coming, because rebuilding this site is apparently a hobby I document compulsively (twelve parts and counting).

The queue

About those 135 drafts: they range from nearly finished essays to single sentences I clearly believed were profound at the time (even if on second read they're awful!).

Four of them are planned as entire series, and the queue leans discursive: biases, ethics, meaning, and the occasional myth that needs debunking. At the current pace the queue would last well over a year without a single new idea arriving, and new ideas arrive weekly, so the realistic projection is that the drafts folder will continue to outgrow published posts.

The mailbag

The best part of the Rothfuss emails is that the two most thoughtful Rothfuss letters take opposite sides. One reader argued, carefully and at length, that a fandom's broken trust should carry consequences, that admiration and trust are different things, and that the charity chapter still stings years later.

Another wrote that anyone who has wrestled a long, enormous endeavour knows how daunting it must be, and that the entitled anger aimed at the author misses how fortunate we are to have the first two books at all.

One reader in India googled "the individualization of responsibility", stayed to say the site was organised enough to borrow from someday, and promised to cite the post on WhatsApp to friends and family. Loved waking up to that email. He also has his own blog, which is a great read!

Another, an Austrian living in Germany, checks the web every so often for a Doors of Stone release date (in vain, as they put it, till date), landed on the Rothfuss piece instead, and then read half the M*A*S*H post before remembering they had never seen M*A*S*H but kept reading anyway. How nice is that?

Thank you

Read a random post

Fifty posts in, people are reading: subscribing by email, arriving from search engines at two in the morning, following the RSS feed, leaving comments, and sending the occasional message that starts "I read your post about..." and makes my whole day.

Being read turns thinking out loud into a conversation, and I am grateful for every one of you, from the person who has read all fifty to the twenty-eight thousand who came for Rothfuss and left.

Thank you for reading. Here is to the next fifty!

I cannot stand light mode for pretty much anything. Many of my hobbies, my full time job, my masters and my honorary research position all require screentime. Wearing anti-glare glasses helps, but light mode just burns my retinas actively. So, one of the first things I did when making my website an active project rather than a passive one, was adding dark mode. What the tutorials tend to skip for "adding dark mode" is you pick dark mode, you click a link, and the next page blasts you with a white screen for a quarter of a second before remembering your choice. In a dark room, at night, that flash is scorching, and it happens way too often, even on big-name sites. This post covers how this site's theme system avoids it. Part of the series on how this site is built.

The process

The site's theme is a single attribute on the root element: <html data-theme="dark"> or <html data-theme="light">. The stylesheet's light styles are the defaults, and dark mode is a set of override rules scoped to that attribute. These are the first two rules of this site's, verbatim:

body {
  color: #252525;
  background-color: #ffffff; }

[data-theme="dark"] body {
  background-color: #1a1a1a;
  color: #e2e2e2; }

The toggle button in the header flips the attribute, saves the choice to localStorage under a key (kr-theme), and everything restyles instantly, no reload. A brief cross-fade class softens the swap, and the browser's theme-color meta tag (which tints the address bar on phones) is updated to match. All of that is a 60-line JavaScript file.

/**
 * theme.js — Site-wide dark/light mode toggle.
 *
 * Responsibilities:
 *   - Read stored preference from localStorage on DOMContentLoaded
 *   - Default to dark mode when there is no stored preference
 *   - Apply data-theme="dark"|"light" to <html>
 *   - Keep the #theme-toggle button aria state in sync
 *   - Persist choice across page loads
 *
 * Flash-of-wrong-theme (FODT) prevention is handled by a tiny inline
 * script injected into every page's <head> (see the kr-theme check).
 */
(function () {
  var STORAGE_KEY = 'kr-theme';
  var DEFAULT_THEME = 'dark';

  function stored() {
    try { return localStorage.getItem(STORAGE_KEY); } catch (e) { return null; }
  }

  function save(t) {
    try { localStorage.setItem(STORAGE_KEY, t); } catch (e) {}
  }

  function apply(theme) {
    document.documentElement.setAttribute('data-theme', theme);
    var dark = theme === 'dark';

    // Keep the browser UI (mobile address bar etc.) matching the page background.
    var meta = document.querySelector('meta[name="theme-color"]');
    if (meta) meta.setAttribute('content', dark ? '#1a1a1a' : '#ffffff');

    var btn = document.getElementById('theme-toggle');
    if (!btn) return;
    btn.setAttribute('aria-pressed', String(dark));
    btn.setAttribute('aria-label', dark ? 'Switch to light mode' : 'Switch to dark mode');
    btn.setAttribute('title',      dark ? 'Switch to light mode' : 'Switch to dark mode');
  }

  document.addEventListener('DOMContentLoaded', function () {
    apply(stored() || DEFAULT_THEME);

    // Delegated click — works even if button is injected after this script runs
    var switchTimer = null;
    document.addEventListener('click', function (e) {
      var btn = e.target.closest ? e.target.closest('#theme-toggle') : null;
      if (!btn) return;
      var next = document.documentElement.getAttribute('data-theme') === 'dark' ? 'light' : 'dark';
      // Cross-fade the swap (CSS scopes transitions to .theme-switching)
      document.documentElement.classList.add('theme-switching');
      if (switchTimer) clearTimeout(switchTimer);
      switchTimer = setTimeout(function () {
        document.documentElement.classList.remove('theme-switching');
      }, 350);
      save(next);
      apply(next);
    });
  });
}());

The hard part

The flash happens because of an ordering problem. The browser starts painting your page as soon as it has the HTML and CSS; your theme JavaScript, if it's a normal script loaded at the end, runs after that first paint. So the sequence in dark mode is: paint the default (light) page, run the script, discover the visitor wanted dark, restyle. Well, that's no good.

The fix is to cheat the ordering. The very first thing in this site's <head>, before the stylesheet, before anything, is this inline script (unsquashed here for reading; the shipped version is one line):

<script>
(function () {
  var t;
  try { t = localStorage.getItem('kr-theme'); } catch (e) {}
  if (!t) t = 'dark';
  document.documentElement.setAttribute('data-theme', t);
})();
</script>

Inline scripts in the head are render-blocking: the browser stops and runs them before painting a single pixel. Normally that's why performance guides tell you never to put scripts there. I broke the rules here because I made sure it's super efficient (it runs in well under a millisecond) and by doing the one job that must happen before paint: stamping the saved theme onto the root element to prevent harming the corneas of my few thousand readers (although the idea of my code having the power to burn eyeballs across the globe is kind of cool sounding). By the time the browser paints, the attribute is already correct, and there is nothing to flash. The try/catch is for browsers that block localStorage entirely; they get the default, which on this site is dark, on the fact that a light flash in a dark room is worse than a dark flash in a bright one.

The extras: images and iframes

Images with backgrounds don't theme themselves. My post about music listening data includes charts rendered as images, with axis labels and backgrounds that only look right on the right theme. The fix is low-tech and bulletproof: export the chart twice, once per theme, put both in the page, and let three CSS rules choose:

<img class="theme-img-light" src="https://www.kenreid.co.uk/blog/chart-light.png" alt="Top artists chart">
<img class="theme-img-dark" src="https://www.kenreid.co.uk/blog/chart-dark.png" alt="Top artists chart">

/* in the stylesheet */
.theme-img-dark { display: none; }
[data-theme="dark"] .theme-img-light { display: none; }
[data-theme="dark"] .theme-img-dark { display: block; }

Two images, one visible at a time, swapping instantly with the toggle. It costs a second download for the chart a visitor actually views, which for the occasional plot is nothing. (For photographs none of this applies; a photo is a photo on any background, I don't use frames in photos of dark or white)

Iframes are other people's pages. The comments section is an embedded giscus frame (the free-comments-via-GitHub-Discussions setup has a post of its own), and my CSS cannot reach inside it. giscus's answer is a message-passing API: when the toggle flips, the site posts a message into the frame telling it which stylesheet to use, and the comments swap themes in step with the page around them. Every embedded widget needs this conversation, which is one more argument for embedding as few widgets as possible.

// runs after the toggle is clicked; giscusTheme() returns the stylesheet for the new mode
var frame = document.querySelector('iframe.giscus-frame');
frame.contentWindow.postMessage(
  { giscus: { setConfig: { theme: giscusTheme() } } },
  'https://giscus.app'
);

Respect the choice, remember the choice

The remaining design decision is what to do for a first-time visitor with no saved preference. The fashionable answer is to read the operating system's setting via a media query, and it's a good answer. This site instead defaults everyone to dark and makes the toggle obvious, partly because the photography looks better on dark, partly because the site had dark-first visuals long before it had a toggle, but mostly because I hate light theme and I force it upon everyone by default, because I care about your optical health, but you can opt into the pain if you prefer.

Back to all posts

Simulated Annealing, Live

The star of the show is simulated annealing, which I love the metaphor for. When I went to Disney some 12 years ago, I saw a live performance of an artist shaping molten glass into the shape of Mickey Mouse. It's a fascinating art, known as annealing: where you, a blacksmith or glassblower, heat up your material till it's incredibly hot, often a striking red color, then shape it as you please, and cool it strategically. As it cools, it hardens, so you have a more flexible material the hotter it is, and less so the cooler it is. This post shows a metaphor of this in optimization known as Simulated Annealing, it's a sixty-line algorithm that solves problems by exploring a search space of a problem (shaping the material that is wet and goopy), then cooling down as better solutions are found, modifying the medium less and less until you have a solution.

The problem: greed gets stuck

Imagine a marble rolling on a bumpy curve, trying to find the lowest point. The obvious strategy is "only ever move downhill". This is the spirit of gradient descent and hill climbing, but the first dip the marble lands in, it stays in. Nearby there may be a valley ten times deeper, but it will never see it, because getting there requires briefly going up.

Metallurgists solved this problem for atoms centuries before computer scientists had it: cool molten metal too fast and its atoms freeze wherever they happen to be, full of internal stress: a brittle mess stuck in a bad arrangement. But cool it slowly, annealing it, and the atoms, still hot enough to jiggle out of bad positions, gradually settle into a low-energy crystal. Randomness, early on, is what lets the system escape bad arrangements; the slow cooling is what lets it eventually commit to a good one.

In 1983, Kirkpatrick, Gelatt and Vecchi published the algorithmic version in Science^[1] (Černý found it independently^[2]), building on a Monte Carlo method physicists had used since 1953.^[3] The recipe: search like a hill climber, but when a proposed move is worse, don't always refuse. Accept it with probability P = exp(−Δ/T), where Δ is how much worse and T is a temperature you slowly lower. Hot: almost anything goes, the search bounces freely across the landscape. Cold: only improvements survive, and the algorithm hardens into a plain hill climber.

Play with it: the marble and the thermostat

Below are two marbles on the same bumpy curve, started at the same spot. The green one is a pure hill climber: it only accepts downhill moves. The blue one anneals: you hold its thermostat. Run it a few times. Watch the greedy marble find the nearest dip and retire there, while the hot marble rattles across the whole curve; then drag the temperature down (or let auto-cool do it) and watch the blue marble crystallise, usually somewhere much deeper. The small dots mark the best point each marble has ever found. The strip under the curve records your run as you go: the temperature you are holding in amber and, in blue, the fraction of uphill moves the annealer is accepting, which is P = exp(−Δ/T) made visible.

The blue marble isn't smarter, it's just temporarily tolerant of bad moves, on a schedule. And that single tweak converts the world's dumbest algorithm into one that, given a sensible cooling schedule, provably converges toward global optima and, more importantly, finds excellent solutions to real industrial problems in practice.

The main event: four algorithms, one map

Travelling Salesman Problem: given a set of cities, find the shortest round trip that visits each once. It's the classic NP-hard problem, the number of possible tours explodes factorially (30 cities is already ~10³⁰ tours), and it's the kind of discrete, gradient-free terrain I wrote about in the previous post.

The racers, each in its lane colour:

• Simulated annealing: as above; proposes 2-opt moves (pick two edges of the tour, reconnect them the other way^[4]), accepting bad ones on the cooling schedule.
• Hill climber, same 2-opt moves, zero tolerance: improvements only.
• Genetic algorithm: a population of 40 tours; tournament selection, order crossover, swap mutation, elitism. The napkin algorithm from my GA post.
• Random search, shuffles a fresh tour every try and keeps the best. The floor. Somebody has to be it.

The race is scored in evaluations, not iterations. Every time any algorithm asks "how long is this tour?", its meter ticks. The GA evaluates 40 tours per generation, so it gets fewer generations; the annealer evaluates one tour per step, so it gets many steps.

What to watch for

Every race is different (the map is random, and so are the algorithms), but a few patterns recur:

The hill climber sprints, then flatlines. Early on, greed looks brilliant: almost every 2-opt move on a random tour is an improvement, so the green line dives. Then it hits a tour where no single 2-opt move helps, a local optimum, and the line goes horizontal forever. Sometimes it gets lucky and the flatline is a good tour.

The annealer loses the early race, but that's by design. While hot, it accepts terrible moves, and its curve dawdles above the hill climber's. It's supposed to look bad early. It's spending the early budget on exploration, when the cooled annealer keeps improving long after the hill climber has frozen, and the blue line crosses below the green one. If you set cooling speed too high in the playground, you can watch that advantage die: quench it, and it becomes a hill climber with a lot more code! The telemetry strip under the chart shows it happening: while the amber temperature line is up the blue acceptance trace rides high too, and the moment amber flattens near zero, blue dies with it, leaving the annealer greedy from then on.

The GA is the tortoise. Forty tours sharing a budget means it always looks slow per-evaluation, and on a problem this small, 2-opt moves are so effective that the population approach rarely wins the sprint. But watch how steadily it improves, and how rarely it gets truly stuck: the population is insurance against the trap that eats the hill climber. On bigger, nastier, more constrained problems, that insurance is essential, which is why my own field lives there.

Random search is bad. It might find a great solution but it doesn't follow any strategy to improve upon. Every (real) metaheuristic beats random.

Why Simulated Annealing is still relevant

Simulated annealing is over forty years old, has no learned components, no data, no gradients, and can be setup in a couple of minutes. It is also, right now, in production: routing chips, scheduling factories and hospitals, packing containers, planning telescope observations. Its longevity comes from the same property that made the marble demo work: it makes almost no assumptions about the problem. Anything you can score and perturb, you can anneal, which in practice means nearly everything with discrete parts and hard constraints, the terrain where gradient methods can't get a foothold. It requires no training data or understanding of a problem, just some means to measure "goodness".

I also like the general philosophy of it: explore wildly at the beginning, become pickier over time, and always remember the most important solution found. It's one of the best examples of "exploration vs. exploitation", a concept applicable to all optimization problems, and to life in general: explore while you can and make use of your search to find the best for you. Settling too early or too late is unwise, and always be willing to consider a better option.

  /* ============================================================
  Simulated Annealing, Live, all demo code for this post.
  No dependencies. Everything renders to <canvas> and reads its
  colours from the CSS custom properties on.kr-viz, so the
  site's light/dark theme toggle restyles the demos live.
  ============================================================ */
  (function() {
    'use strict';

    function mulberry32(seed) {
      var a = seed >>> 0;
      return function() {
        a |= 0;
        a = (a + 0x6D2B79F5) | 0;
        var t = Math.imul(a ^ (a >>> 15), 1 | a);
        t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
        return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
      };
    }

    function vizColors(el) {
      var cs = getComputedStyle(el);

      function v(name) {
        return cs.getPropertyValue(name).trim();
      }
      return {
        series: [v('--viz-s1'), v('--viz-s2'), v('--viz-s3'), v('--viz-s4')],
        ink: v('--viz-ink'),
        muted: v('--viz-muted'),
        grid: v('--viz-grid'),
        surface: v('--viz-surface')
      };
    }

    function fitCanvas(cv) {
      var dpr = window.devicePixelRatio || 1;
      // the height attribute is the desktop height; on narrow screens clamp tall
      // canvases to a sane aspect (never below 180px) while short telemetry
      // strips keep their declared height
      var w = cv.clientWidth;
      var h = Math.min(parseInt(cv.getAttribute('height'), 10), Math.max(180, Math.round(w * 0.75)));
      cv.style.height = h + 'px';
      cv.width = Math.round(w * dpr);
      cv.height = Math.round(h * dpr);
      var ctx = cv.getContext('2d');
      ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
      return ctx;
    }

    /* ============================================================
    Widget A: the marble and the thermostat
    ============================================================ */
    (function marbleDemo() {
      var wrap = document.getElementById('curve-demo');
      if (!wrap) return;
      var cv = document.getElementById('curveCanvas');
      var runBtn = document.getElementById('curveRun');
      var resetBtn = document.getElementById('curveReset');
      var tempSlider = document.getElementById('curveTemp');
      var tempVal = document.getElementById('curveTempVal');
      var autoCool = document.getElementById('curveAuto');
      var strip = document.getElementById('curveStrip');
      var ctx = fitCanvas(cv);
      var stripCtx = fitCanvas(strip);
      var STRIP_N = 480; // rolling telemetry window, ~8s of frames
      var running = false,
        rafId = null;
      var rng = mulberry32(12345);

      function f(x) {
        return 0.32 * Math.sin(x * 9) +
          0.18 * Math.sin(x * 23 + 1.7) +
          0.10 * Math.sin(x * 41 + 0.4) +
          1.6 * (x - 0.72) * (x - 0.72);
      }
      var F_MIN = -0.55,
        F_MAX = 1.15; // plotting range

      var T_MAX = 0.4;
      var state;

      function reset() {
        var x0 = 0.05 + rng() * 0.9;
        state = {
          hc: {
            x: x0,
            best: x0,
            bestF: f(x0)
          },
          sa: {
            x: x0,
            best: x0,
            bestF: f(x0)
          },
          T: (tempSlider.value / 100) * T_MAX,
          hist: [],
          accEma: null,
          dEma: null
        };
        tempSlider.value = Math.round(state.T / T_MAX * 100);
        draw();
      }

      function px(x, W) {
        return 14 + x * (W - 28);
      }

      function py(y, H) {
        return 16 + (1 - (y - F_MIN) / (F_MAX - F_MIN)) * (H - 44);
      }

      function step() {
        var SIGMA = 0.02,
          STEPS = 26;
        var upProp = 0,
          upAcc = 0,
          upSum = 0;
        for (var i = 0; i < STEPS; i++) {
          // hill climber
          var nx = Math.min(1, Math.max(0, state.hc.x + (rng() * 2 - 1) * SIGMA));
          if (f(nx) <= f(state.hc.x)) state.hc.x = nx;
          if (f(state.hc.x) < state.hc.bestF) {
            state.hc.bestF = f(state.hc.x);
            state.hc.best = state.hc.x;
          }
          // annealer (uphill proposals are tallied for the telemetry strip)
          var sx = Math.min(1, Math.max(0, state.sa.x + (rng() * 2 - 1) * SIGMA));
          var d = f(sx) - f(state.sa.x);
          if (d <= 0) {
            state.sa.x = sx;
          } else {
            upProp++;
            upSum += d;
            if (state.T > 1e-6 && rng() < Math.exp(-d / state.T)) {
              state.sa.x = sx;
              upAcc++;
            }
          }
          if (f(state.sa.x) < state.sa.bestF) {
            state.sa.bestF = f(state.sa.x);
            state.sa.best = state.sa.x;
          }
          if (autoCool.checked) {
            state.T *= 0.99965;
            tempSlider.value = Math.round(state.T / T_MAX * 100);
          }
        }
        // smooth the tallies into the telemetry (EMA so the traces don't jitter)
        if (upProp) {
          var K = 0.15;
          var rate = upAcc / upProp;
          state.accEma = state.accEma === null ? rate : state.accEma + K * (rate - state.accEma);
          var dbar = upSum / upProp;
          state.dEma = state.dEma === null ? dbar : state.dEma + K * (dbar - state.dEma);
        }
        state.hist.push({
          t: state.T / T_MAX,
          a: state.accEma === null ? 1 : state.accEma
        });
        if (state.hist.length > STRIP_N) state.hist.shift();
      }

      function draw() {
        var c = vizColors(wrap);
        var W = cv.clientWidth,
          H = cv.clientHeight;
        ctx.clearRect(0, 0, W, H);
        // curve
        ctx.beginPath();
        for (var i = 0; i <= 300; i++) {
          var x = i / 300,
            X = px(x, W),
            Y = py(f(x), H);
          if (i === 0) ctx.moveTo(X, Y);
          else ctx.lineTo(X, Y);
        }
        ctx.strokeStyle = c.muted;
        ctx.lineWidth = 2;
        ctx.stroke();
        // best markers
        [
          ['hc', c.series[1]],
          ['sa', c.series[0]]
        ].forEach(function(pair) {
          var a = state[pair[0]];
          ctx.beginPath();
          ctx.arc(px(a.best, W), py(f(a.best), H), 3.5, 0, Math.PI * 2);
          ctx.fillStyle = pair[1];
          ctx.globalAlpha = 0.45;
          ctx.fill();
          ctx.globalAlpha = 1;
        });
        // marbles (annealer drawn second, with a surface ring so overlaps stay legible)
        [
          ['hc', c.series[1]],
          ['sa', c.series[0]]
        ].forEach(function(pair) {
          var a = state[pair[0]];
          ctx.beginPath();
          ctx.arc(px(a.x, W), py(f(a.x), H) - 8, 8, 0, Math.PI * 2);
          ctx.fillStyle = pair[1];
          ctx.fill();
          ctx.lineWidth = 2;
          ctx.strokeStyle = c.surface;
          ctx.stroke();
        });
        // readouts (text in ink, swatch carries identity)
        ctx.font = '12px Poppins, sans-serif';
        ctx.fillStyle = c.series[0];
        ctx.fillRect(14, 10, 9, 9);
        ctx.fillStyle = c.ink;
        ctx.fillText('annealer best: ' + state.sa.bestF.toFixed(3), 28, 19);
        ctx.fillStyle = c.series[1];
        ctx.fillRect(14, 26, 9, 9);
        ctx.fillStyle = c.ink;
        ctx.fillText('greedy best: ' + state.hc.bestF.toFixed(3), 28, 35);
        tempVal.textContent = 'T = ' + state.T.toFixed(3);
        drawStrip();
      }

      // the flight recorder: temperature (amber) and share of uphill proposals
      // accepted (blue) over a rolling window, both on one 0-100% scale
      function drawStrip() {
        var c = vizColors(wrap);
        var W = strip.clientWidth,
          H = strip.clientHeight;
        var padL = 14,
          padR = 78,
          padT = 10,
          padB = 8;
        stripCtx.clearRect(0, 0, W, H);

        function X(i) {
          return padL + (i / (STRIP_N - 1)) * (W - padL - padR);
        }

        function Y(v) {
          return padT + (1 - v) * (H - padT - padB);
        }
        stripCtx.font = '11px Poppins, sans-serif';
        [0, 0.5, 1].forEach(function(v) {
          stripCtx.strokeStyle = c.grid;
          stripCtx.lineWidth = 1;
          stripCtx.beginPath();
          stripCtx.moveTo(padL, Y(v));
          stripCtx.lineTo(W - padR, Y(v));
          stripCtx.stroke();
        });
        var start = STRIP_N - state.hist.length;
        [
          ['t', c.series[2]],
          ['a', c.series[0]]
        ].forEach(function(pair) {
          if (!state.hist.length) return;
          stripCtx.beginPath();
          state.hist.forEach(function(h, i) {
            var x = X(start + i),
              y = Y(h[pair[0]]);
            if (i === 0) stripCtx.moveTo(x, y);
            else stripCtx.lineTo(x, y);
          });
          stripCtx.strokeStyle = pair[1];
          stripCtx.lineWidth = 2;
          stripCtx.stroke();
        });
        // live end labels (ink text, coloured dot), nudged apart if they collide
        var labels = [{
          y: Y(state.T / T_MAX),
          color: c.series[2],
          text: 'T ' + Math.round(state.T / T_MAX * 100) + '%'
        }];
        if (state.accEma !== null) labels.push({
          y: Y(state.accEma),
          color: c.series[0],
          text: 'uphill ' + Math.round(state.accEma * 100) + '%'
        });
        labels.sort(function(p, q) {
          return p.y - q.y;
        });
        labels.forEach(function(l) {
          l.y = Math.max(padT + 5, Math.min(H - padB - 3, l.y));
        });
        if (labels.length === 2 && labels[1].y - labels[0].y < 13) labels[0].y = labels[1].y - 13;
        labels.forEach(function(l) {
          var x = W - padR + 6;
          stripCtx.beginPath();
          stripCtx.arc(x + 4, l.y, 3.5, 0, Math.PI * 2);
          stripCtx.fillStyle = l.color;
          stripCtx.fill();
          stripCtx.fillStyle = c.ink;
          stripCtx.fillText(l.text, x + 11, l.y + 4);
        });
        // the Metropolis rule with live numbers, for a typical uphill step
        // (surface-coloured halo keeps it legible when the traces run hot)
        if (state.dEma !== null) {
          var p = state.T > 1e-6 ? Math.exp(-state.dEma / state.T) : 0;
          var txt = 'typical uphill Δ ' + state.dEma.toFixed(3) +
            ' · P(accept) ' + Math.round(p * 100) + '%';
          stripCtx.strokeStyle = c.surface;
          stripCtx.lineWidth = 3;
          stripCtx.strokeText(txt, padL + 4, padT + 12);
          stripCtx.fillStyle = c.muted;
          stripCtx.fillText(txt, padL + 4, padT + 12);
        }
      }

      function loop() {
        if (!running) return;
        state.T = (tempSlider.value / 100) * T_MAX;
        step();
        draw();
        rafId = requestAnimationFrame(loop);
      }

      runBtn.addEventListener('click', function() {
        running = !running;
        runBtn.textContent = running ? 'Pause' : 'Run';
        if (running) loop();
        else cancelAnimationFrame(rafId);
      });
      resetBtn.addEventListener('click', function() {
        reset();
      });
      tempSlider.addEventListener('input', function() {
        state.T = (tempSlider.value / 100) * T_MAX;
        if (!running) draw();
      });
      window.addEventListener('resize', function() {
        ctx = fitCanvas(cv);
        stripCtx = fitCanvas(strip);
        draw();
      });
      new MutationObserver(function() {
          if (!running) draw();
        })
        .observe(document.documentElement, {
          attributes: true,
          attributeFilter: ['data-theme']
        });
      reset();
    })();

    /* ============================================================
    Widget B: the four-way TSP race
    ============================================================ */
    (function raceDemo() {
      var wrap = document.getElementById('race-demo');
      if (!wrap) return;
      var NAMES = ['Simulated annealing', 'Hill climber', 'Genetic algorithm', 'Random search'];
      var SHORT = ['SA', 'Hill', 'GA', 'Rand'];
      var MAX_EVALS = 200000;
      var runBtn = document.getElementById('raceRun');
      var resetBtn = document.getElementById('raceReset');
      var speedSlider = document.getElementById('raceSpeed');
      var statusEl = document.getElementById('raceStatus');
      var tourCanvases = Array.prototype.slice.call(wrap.querySelectorAll('.tourCanvas'));
      var scoreEls = Array.prototype.slice.call(wrap.querySelectorAll('.sa-score'));
      var chartCv = document.getElementById('raceChart');
      var stripCv = document.getElementById('raceStrip');
      var tooltip = document.getElementById('raceTooltip');
      var tableBody = document.getElementById('raceTable');
      var pgCities = document.getElementById('pgCities'),
        pgCitiesVal = document.getElementById('pgCitiesVal');
      var pgCool = document.getElementById('pgCool'),
        pgCoolVal = document.getElementById('pgCoolVal');
      var pgMut = document.getElementById('pgMut'),
        pgMutVal = document.getElementById('pgMutVal');

      var tourCtx = tourCanvases.map(fitCanvas);
      var chartCtx = fitCanvas(chartCv);
      var stripCtx = fitCanvas(stripCv);
      var seed = 20260706,
        rng = mulberry32(seed);
      var cities = [],
        algs = [],
        running = false,
        rafId = null,
        finished = false;

      function tourLen(t) {
        var L = 0;
        for (var i = 0; i < t.length; i++) {
          var a = cities[t[i]],
            b = cities[t[(i + 1) % t.length]];
          L += Math.hypot(a[0] - b[0], a[1] - b[1]);
        }
        return L;
      }

      function randTour(r) {
        var t = [];
        for (var i = 0; i < cities.length; i++) t.push(i);
        for (var j = t.length - 1; j > 0; j--) {
          var k = Math.floor(r() * (j + 1)),
            tmp = t[j];
          t[j] = t[k];
          t[k] = tmp;
        }
        return t;
      }

      function twoOpt(t, r) {
        var n = t.length;
        var i = 1 + Math.floor(r() * (n - 2));
        var j = i + 1 + Math.floor(r() * (n - i - 1));
        var nt = t.slice(0, i).concat(t.slice(i, j + 1).reverse(), t.slice(j + 1));
        return nt;
      }

      function makeAlgs() {
        var r1 = mulberry32(seed + 1),
          r2 = mulberry32(seed + 2),
          r3 = mulberry32(seed + 3),
          r4 = mulberry32(seed + 4);
        var start = randTour(mulberry32(seed + 9));
        var startLen = tourLen(start);
        var coolSpeed = parseInt(pgCool.value, 10); // decay exponent over full budget
        var mutRate = parseInt(pgMut.value, 10) / 100; // per-child swap probability
        var T0 = 2 * startLen / cities.length;

        // Each algorithm: { evals, best, bestLen, hist:[{e,len}], step(budget) }
        function record(a) {
          a.hist.push({
            e: a.evals,
            len: a.bestLen
          });
        }

        var sa = {
          evals: 0,
          cur: start.slice(),
          curLen: startLen,
          best: start.slice(),
          bestLen: startLen,
          hist: [],
          pstrip: [],
          upP: 0,
          upA: 0,
          tempAt: function(e) {
            // deterministic schedule: T as a fraction of T0 after e evaluations
            return Math.exp(-coolSpeed * e / MAX_EVALS);
          },
          rng: r1,
          step: function(budget) {
            for (var i = 0; i < budget && this.evals < MAX_EVALS; i++) {
              var cand = twoOpt(this.cur, this.rng);
              var len = tourLen(cand);
              this.evals++;
              var T = T0 * this.tempAt(this.evals);
              var d = len - this.curLen;
              if (d <= 0) {
                this.cur = cand;
                this.curLen = len;
              } else {
                this.upP++;
                if (T > 1e-9 && this.rng() < Math.exp(-d / T)) {
                  this.cur = cand;
                  this.curLen = len;
                  this.upA++;
                }
              }
              if (len < this.bestLen) {
                this.best = cand;
                this.bestLen = len;
              }
            }
          }
        };
        var hc = {
          evals: 0,
          cur: start.slice(),
          curLen: startLen,
          best: start.slice(),
          bestLen: startLen,
          hist: [],
          rng: r2,
          step: function(budget) {
            for (var i = 0; i < budget && this.evals < MAX_EVALS; i++) {
              var cand = twoOpt(this.cur, this.rng);
              var len = tourLen(cand);
              this.evals++;
              if (len <= this.curLen) {
                this.cur = cand;
                this.curLen = len;
              }
              if (len < this.bestLen) {
                this.best = cand;
                this.bestLen = len;
              }
            }
          }
        };
        var ga = (function() {
          var POP = 40,
            ELITE = 2,
            TOUR = 3;
          var pop = [],
            fit = [];
          for (var i = 0; i < POP; i++) pop.push(randTour(r3));
          var g = {
            evals: 0,
            best: null,
            bestLen: Infinity,
            hist: [],
            rng: r3,
            pending: 0
          };

          function evalPop() {
            fit = pop.map(function(t) {
              g.evals++;
              var L = tourLen(t);
              if (L < g.bestLen) {
                g.bestLen = L;
                g.best = t.slice();
              }
              return L;
            });
          }
          evalPop();

          function pick() {
            var bi = -1,
              bf = Infinity;
            for (var k = 0; k < TOUR; k++) {
              var c = Math.floor(g.rng() * POP);
              if (fit[c] < bf) {
                bf = fit[c];
                bi = c;
              }
            }
            return pop[bi];
          }

          function ox(p1, p2) {
            var n = p1.length,
              i = Math.floor(g.rng() * n),
              j = Math.floor(g.rng() * n);
            if (i > j) {
              var t = i;
              i = j;
              j = t;
            }
            var child = new Array(n),
              used = {};
            for (var k = i; k <= j; k++) {
              child[k] = p1[k];
              used[p1[k]] = true;
            }
            var pos = (j + 1) % n;
            for (var m = 0; m < n; m++) {
              var gene = p2[(j + 1 + m) % n];
              if (!used[gene]) {
                child[pos] = gene;
                pos = (pos + 1) % n;
              }
            }
            return child;
          }
          g.step = function(budget) {
            this.pending += budget;
            while (this.pending >= POP && this.evals < MAX_EVALS) {
              var next = [];
              // elitism: carry the best tours forward untouched
              var order = fit.map(function(f, idx) {
                return [f, idx];
              }).sort(function(a, b) {
                return a[0] - b[0];
              });
              for (var e = 0; e < ELITE; e++) next.push(pop[order[e][1]].slice());
              while (next.length < POP) {
                var child = ox(pick(), pick());
                if (g.rng() < mutRate) {
                  var a = Math.floor(g.rng() * child.length),
                    b = Math.floor(g.rng() * child.length);
                  var tmp = child[a];
                  child[a] = child[b];
                  child[b] = tmp;
                }
                next.push(child);
              }
              pop = next;
              evalPop();
              this.pending -= POP;
            }
          };
          return g;
        })();
        var rs = {
          evals: 0,
          best: start.slice(),
          bestLen: startLen,
          hist: [],
          rng: r4,
          step: function(budget) {
            for (var i = 0; i < budget && this.evals < MAX_EVALS; i++) {
              var cand = randTour(this.rng);
              var len = tourLen(cand);
              this.evals++;
              if (len < this.bestLen) {
                this.best = cand;
                this.bestLen = len;
              }
            }
          }
        };
        [sa, hc, ga, rs].forEach(record);
        return [sa, hc, ga, rs];
      }

      function newCities(n) {
        rng = mulberry32(seed);
        cities = [];
        for (var i = 0; i < n; i++) cities.push([0.06 + rng() * 0.88, 0.08 + rng() * 0.84]);
      }

      function resetRace(keepCities) {
        running = false;
        finished = false;
        cancelAnimationFrame(rafId);
        runBtn.textContent = 'Start race';
        if (!keepCities) newCities(parseInt(pgCities.value, 10));
        algs = makeAlgs();
        statusEl.textContent = 'Press start, or click any map to add a city.';
        drawAll();
      }

      function drawTour(idx) {
        var c = vizColors(wrap);
        var ctx = tourCtx[idx],
          cv = tourCanvases[idx];
        var W = cv.clientWidth,
          H = cv.clientHeight;
        ctx.clearRect(0, 0, W, H);
        ctx.strokeStyle = c.grid;
        ctx.strokeRect(0.5, 0.5, W - 1, H - 1);
        var a = algs[idx];
        // tour path
        ctx.beginPath();
        for (var i = 0; i <= a.best.length; i++) {
          var p = cities[a.best[i % a.best.length]];
          var X = p[0] * W,
            Y = p[1] * H;
          if (i === 0) ctx.moveTo(X, Y);
          else ctx.lineTo(X, Y);
        }
        ctx.strokeStyle = c.series[idx];
        ctx.lineWidth = 2;
        ctx.stroke();
        // cities
        for (var j = 0; j < cities.length; j++) {
          ctx.beginPath();
          ctx.arc(cities[j][0] * W, cities[j][1] * H, 3, 0, Math.PI * 2);
          ctx.fillStyle = c.ink;
          ctx.fill();
        }
        scoreEls[idx].textContent = a.bestLen.toFixed(2);
      }

      function drawChart() {
        var c = vizColors(wrap);
        var W = chartCv.clientWidth,
          H = chartCv.clientHeight;
        var ctx = chartCtx;
        var padL = 44,
          padR = 70,
          padT = 12,
          padB = 26;
        ctx.clearRect(0, 0, W, H);
        var maxLen = 0,
          minLen = Infinity;
        algs.forEach(function(a) {
          a.hist.forEach(function(h) {
            if (h.len > maxLen) maxLen = h.len;
            if (h.len < minLen) minLen = h.len;
          });
        });
        if (!isFinite(minLen)) return;
        var lo = minLen * 0.95,
          hi = maxLen * 1.02;

        function X(e) {
          return padL + (e / MAX_EVALS) * (W - padL - padR);
        }

        function Y(v) {
          return padT + (1 - (v - lo) / (hi - lo)) * (H - padT - padB);
        }
        // grid + y ticks
        ctx.font = '11px Poppins, sans-serif';
        for (var g = 0; g <= 4; g++) {
          var v = lo + (g / 4) * (hi - lo),
            y = Y(v);
          ctx.strokeStyle = c.grid;
          ctx.lineWidth = 1;
          ctx.beginPath();
          ctx.moveTo(padL, y);
          ctx.lineTo(W - padR, y);
          ctx.stroke();
          ctx.fillStyle = c.muted;
          ctx.textAlign = 'right';
          ctx.fillText(v.toFixed(1), padL - 6, y + 4);
        }
        // x ticks ("evals" suffix on the last tick names the axis)
        ctx.textAlign = 'center';
        for (var t = 0; t <= 4; t++) {
          var e = (t / 4) * MAX_EVALS;
          ctx.fillStyle = c.muted;
          ctx.fillText((e / 1000) + 'k' + (t === 4 ? ' evals' : ''), X(e), H - 8);
        }
        ctx.textAlign = 'left';
        // series (best-so-far is a step function; draw as steps)
        algs.forEach(function(a, i) {
          ctx.beginPath();
          var prevY = null;
          a.hist.forEach(function(h, k) {
            var x = X(h.e),
              y = Y(h.len);
            if (k === 0) ctx.moveTo(x, y);
            else {
              ctx.lineTo(x, prevY);
              ctx.lineTo(x, y);
            }
            prevY = y;
          });
          // extend to current eval count
          ctx.lineTo(X(a.evals), prevY);
          ctx.strokeStyle = c.series[i];
          ctx.lineWidth = 2;
          ctx.stroke();
        });
        // direct labels at line ends (ink text + colored dot), collision-nudged
        var ends = algs.map(function(a, i) {
            return {
              i: i,
              y: Y(a.bestLen),
              len: a.bestLen
            };
          })
          .sort(function(p, q) {
            return p.y - q.y;
          });
        var lastY = -Infinity;
        ends.forEach(function(e) {
          lastY = e.y = Math.max(e.y, lastY + 13);
        });
        var overflow = lastY - (H - padB - 4);
        if (overflow > 0) ends.forEach(function(e) {
          e.y -= overflow;
        });
        ends.forEach(function(e) {
          var y = e.y;
          var x = W - padR + 6;
          ctx.beginPath();
          ctx.arc(x + 4, y, 3.5, 0, Math.PI * 2);
          ctx.fillStyle = c.series[e.i];
          ctx.fill();
          ctx.fillStyle = c.ink;
          ctx.fillText(SHORT[e.i], x + 11, y + 4);
        });
      }

      // the annealer's telemetry, sharing the race chart's x-axis: planned
      // cooling schedule (faint amber), temperature spent so far (solid amber)
      // and observed share of uphill moves accepted (blue), all on 0-100%
      function drawStrip() {
        var c = vizColors(wrap);
        var W = stripCv.clientWidth,
          H = stripCv.clientHeight;
        var ctx = stripCtx;
        var padL = 44,
          padR = 70,
          padT = 10,
          padB = 10;
        ctx.clearRect(0, 0, W, H);
        var sa = algs[0];

        function X(e) {
          return padL + (e / MAX_EVALS) * (W - padL - padR);
        }

        function Y(v) {
          return padT + (1 - v) * (H - padT - padB);
        }
        ctx.font = '11px Poppins, sans-serif';
        [0, 0.5, 1].forEach(function(v) {
          ctx.strokeStyle = c.grid;
          ctx.lineWidth = 1;
          ctx.beginPath();
          ctx.moveTo(padL, Y(v));
          ctx.lineTo(W - padR, Y(v));
          ctx.stroke();
          ctx.fillStyle = c.muted;
          ctx.textAlign = 'right';
          ctx.fillText(Math.round(v * 100) + '%', padL - 6, Y(v) + 4);
        });
        ctx.textAlign = 'left';
        // planned cooling schedule, faint, across the whole budget
        ctx.beginPath();
        for (var i = 0; i <= 120; i++) {
          var e = (i / 120) * MAX_EVALS;
          if (i === 0) ctx.moveTo(X(e), Y(sa.tempAt(e)));
          else ctx.lineTo(X(e), Y(sa.tempAt(e)));
        }
        ctx.strokeStyle = c.series[2];
        ctx.globalAlpha = 0.3;
        ctx.lineWidth = 2;
        ctx.stroke();
        ctx.globalAlpha = 1;
        // temperature actually spent, solid, up to the current evaluation
        if (sa.evals > 0) {
          ctx.beginPath();
          var segs = Math.max(2, Math.round(120 * sa.evals / MAX_EVALS));
          for (var k = 0; k <= segs; k++) {
            var e2 = (k / segs) * sa.evals;
            if (k === 0) ctx.moveTo(X(e2), Y(sa.tempAt(e2)));
            else ctx.lineTo(X(e2), Y(sa.tempAt(e2)));
          }
          ctx.strokeStyle = c.series[2];
          ctx.lineWidth = 2;
          ctx.stroke();
        }
        // observed share of uphill moves accepted
        if (sa.pstrip.length) {
          ctx.beginPath();
          sa.pstrip.forEach(function(h, k2) {
            if (k2 === 0) ctx.moveTo(X(h.e), Y(h.a));
            else ctx.lineTo(X(h.e), Y(h.a));
          });
          ctx.strokeStyle = c.series[0];
          ctx.lineWidth = 2;
          ctx.stroke();
        }
        // live end labels, matching the race chart's style
        var labels = [{
          y: Y(sa.tempAt(sa.evals)),
          color: c.series[2],
          text: 'T ' + Math.round(sa.tempAt(sa.evals) * 100) + '%'
        }];
        if (sa.pstrip.length) {
          var lastA = sa.pstrip[sa.pstrip.length - 1].a;
          labels.push({
            y: Y(lastA),
            color: c.series[0],
            text: 'uphill ' + Math.round(lastA * 100) + '%'
          });
        }
        labels.sort(function(p, q) {
          return p.y - q.y;
        });
        labels.forEach(function(l) {
          l.y = Math.max(padT + 5, Math.min(H - padB - 3, l.y));
        });
        if (labels.length === 2 && labels[1].y - labels[0].y < 13) labels[0].y = labels[1].y - 13;
        labels.forEach(function(l) {
          var x = W - padR + 6;
          ctx.beginPath();
          ctx.arc(x + 4, l.y, 3.5, 0, Math.PI * 2);
          ctx.fillStyle = l.color;
          ctx.fill();
          ctx.fillStyle = c.ink;
          ctx.fillText(l.text, x + 11, l.y + 4);
        });
      }

      function drawTable() {
        var c = vizColors(wrap);
        var leader = Math.min.apply(null, algs.map(function(a) {
          return a.bestLen;
        }));
        var rows = algs.map(function(a, i) {
            return {
              i: i,
              len: a.bestLen
            };
          })
          .sort(function(p, q) {
            return p.len - q.len;
          });
        tableBody.innerHTML = rows.map(function(r) {
          var gap = ((r.len / leader - 1) * 100);
          return '<div class="kr-stat"><span class="kr-stat-lbl">' +
            '<span class="kr-swatch" style="background:' + c.series[r.i] + ';"></span>' +
            NAMES[r.i] + '</span><span class="kr-stat-val">' + r.len.toFixed(2) +
            ' <span style="color:var(--viz-muted); font-weight:400;">' +
            (gap < 0.005 ? 'leader' : '+' + gap.toFixed(1) + '%') + '</span></span></div>';
        }).join('');
      }

      function drawAll() {
        algs.forEach(function(_, i) {
          drawTour(i);
        });
        drawChart();
        drawStrip();
        drawTable();
      }

      var recordEvery = MAX_EVALS / 400;

      function loop() {
        if (!running) return;
        var budget = parseInt(speedSlider.value, 10);
        algs.forEach(function(a) {
          var before = a.evals;
          a.step(budget);
          if (Math.floor(a.evals / recordEvery) > Math.floor(before / recordEvery)) {
            a.hist.push({
              e: a.evals,
              len: a.bestLen
            });
            if (a.pstrip) {
              a.pstrip.push({
                e: a.evals,
                a: a.upP ? a.upA / a.upP : 0
              });
              a.upP = 0;
              a.upA = 0;
            }
          }
        });
        drawAll();
        var done = algs.every(function(a) {
          return a.evals >= MAX_EVALS;
        });
        if (done) {
          running = false;
          finished = true;
          runBtn.textContent = 'Start race';
          var leader = algs.slice().sort(function(p, q) {
            return p.bestLen - q.bestLen;
          })[0];
          statusEl.textContent = 'Race finished. ' + NAMES[algs.indexOf(leader)] + ' wins. Press "New cities" to go again.';
          algs.forEach(function(a) {
            a.hist.push({
              e: a.evals,
              len: a.bestLen
            });
            if (a.pstrip) {
              a.pstrip.push({
                e: a.evals,
                a: a.upP ? a.upA / a.upP : 0
              });
              a.upP = 0;
              a.upA = 0;
            }
          });
          drawAll();
          return;
        }
        statusEl.textContent = Math.round(algs[0].evals / 1000) + 'k / ' + (MAX_EVALS / 1000) + 'k evaluations';
        rafId = requestAnimationFrame(loop);
      }

      runBtn.addEventListener('click', function() {
        if (finished) resetRace(true);
        running = !running;
        runBtn.textContent = running ? 'Pause' : 'Resume';
        if (running) loop();
        else {
          cancelAnimationFrame(rafId);
          statusEl.textContent = 'Paused.';
        }
      });
      resetBtn.addEventListener('click', function() {
        seed += 7;
        resetRace(false);
      });

      tourCanvases.forEach(function(cv) {
        cv.addEventListener('click', function(ev) {
          if (cities.length >= 80) return;
          var rect = cv.getBoundingClientRect();
          cities.push([(ev.clientX - rect.left) / rect.width, (ev.clientY - rect.top) / rect.height]);
          resetRace(true);
          statusEl.textContent = cities.length + ' cities, race reset. Press start.';
        });
      });

      pgCities.addEventListener('input', function() {
        pgCitiesVal.textContent = pgCities.value;
      });
      pgCities.addEventListener('change', function() {
        resetRace(false);
      });

      function updateCool() {
        pgCoolVal.textContent = (parseInt(pgCool.value, 10) / 10).toFixed(1) + '×';
      }

      function updateMut() {
        pgMutVal.textContent = pgMut.value + '%';
      }
      pgCool.addEventListener('input', updateCool);
      pgCool.addEventListener('change', function() {
        resetRace(true);
      });
      pgMut.addEventListener('input', updateMut);
      pgMut.addEventListener('change', function() {
        resetRace(true);
      });
      updateCool();
      updateMut();

      // chart hover tooltip
      chartCv.addEventListener('mousemove', function(ev) {
        var rect = chartCv.getBoundingClientRect();
        var padL = 44,
          padR = 62;
        var frac = (ev.clientX - rect.left - padL) / (rect.width - padL - padR);
        if (frac < 0 || frac > 1) {
          tooltip.style.display = 'none';
          return;
        }
        var e = frac * MAX_EVALS;
        var c = vizColors(wrap);
        var lines = algs.map(function(a, i) {
          var best = null;
          for (var k = 0; k < a.hist.length; k++) {
            if (a.hist[k].e <= e) best = a.hist[k].len;
            else break;
          }
          return '<span class="kr-swatch" style="background:' + c.series[i] + '; margin-right:6px;"></span>' +
            SHORT[i] + ': ' + (best === null ? ', ' : best.toFixed(2));
        });
        var accAt = null;
        for (var k3 = 0; k3 < algs[0].pstrip.length; k3++) {
          if (algs[0].pstrip[k3].e <= e) accAt = algs[0].pstrip[k3].a;
          else break;
        }
        tooltip.innerHTML = '<strong>' + Math.round(e / 1000) + 'k evals</strong><br>' + lines.join('<br>') +
          '<br><span style="color:var(--viz-muted);">SA telemetry: T ' + Math.round(algs[0].tempAt(e) * 100) + '%' +
          (accAt === null ? '' : ', uphill ' + Math.round(accAt * 100) + '%') + '</span>';
        tooltip.style.display = 'block';
        var tx = ev.clientX - rect.left + 14;
        if (tx + 130 > rect.width) tx = ev.clientX - rect.left - 144;
        tooltip.style.left = tx + 'px';
        tooltip.style.top = (ev.clientY - rect.top + 10) + 'px';
      });
      chartCv.addEventListener('mouseleave', function() {
        tooltip.style.display = 'none';
      });

      window.addEventListener('resize', function() {
        tourCtx = tourCanvases.map(fitCanvas);
        chartCtx = fitCanvas(chartCv);
        stripCtx = fitCanvas(stripCv);
        drawAll();
      });
      new MutationObserver(function() {
          drawAll();
        })
        .observe(document.documentElement, {
          attributes: true,
          attributeFilter: ['data-theme']
        });

      newCities(parseInt(pgCities.value, 10));
      resetRace(true);
    })();
  })();
 

References

1. Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983). Optimization by simulated annealing. Science, 220(4598), 671–680. https://doi.org/10.1126/science.220.4598.671
2. Černý, V. (1985). Thermodynamical approach to the travelling salesman problem: An efficient simulation algorithm. Journal of Optimization Theory and Applications, 45, 41–51. https://doi.org/10.1007/BF00940812
3. Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., & Teller, E. (1953). Equation of state calculations by fast computing machines. The Journal of Chemical Physics, 21(6), 1087–1092. https://doi.org/10.1063/1.1699114
4. Croes, G. A. (1958). A method for solving traveling-salesman problems. Operations Research, 6(6), 791–812. https://doi.org/10.1287/opre.6.6.791

Evolution, Live

A while ago I set an ant colony foraging in your browser. Watching an optimiser work beats reading about one, by a mile, so here is the sequel, for the algorithm that is even more watchable, because it is even more alive. This post evolves a solution in front of you, out of a population that breeds, mutates, and dies, generation by generation, on your CPU.

The whole algorithm on a napkin

The entire loop:

1. Make a population of random candidate solutions.
2. Score each one with the fitness function.
3. Select parents, biased toward the fitter ones.
4. Cross them over to make children, mutate the children a little.
5. Replace the old population with the new one (keeping a few elites).
6. Go to step 2. Repeat until bored or converged.

Watch it evolve

Below, a population is evolving toward a target. Each little picture is one individual; the big one is the current champion. Press play and watch the champion sharpen out of noise, generation by generation. Play with the parameters in the sliders to see how they effect the outcome.

Three ways to stall evolution

Three experiments to try:

Crank selection pressure to maximum. The population will rocket toward the best early solution and then stop improving, sometimes far from a good answer. You've just caused premature convergence: everyone became a copy of one lucky ancestor, and with no diversity left, crossover has nothing to recombine and mutation alone can't dig out.

Set mutation to zero. Evolution grinds to a halt the moment the population homogenises, because crossover can only ever remix genes that already exist. No new material, no progress. Then push mutation to maximum and watch the opposite disaster: the champion thrashes randomly and never settles, because you've turned evolution into a random search.

Shrink the population to a handful. Small populations converge fast and badly; they're a tiny gene pool, prone to the same fragility as any small, inbred group. This happens in real life often when a natural disaster strikes, or a small number of a species is separated (e.g. a pond splits in two, or high winds migrate some animals across distance).

The thesis

Gradient descent, the engine under modern AI, needs a smooth, differentiable landscape to roll down. The genetic algorithm in your browser needs no such thing. It will happily optimise a target you can only score, never differentiate: a timetable, a circuit layout, a wing shape evaluated by a physics simulator. Whole classes of real problems have no gradient to descend, and on those, evolution isn't a quaint biological metaphor, nor a glorified random number generator, it's the only tool that works.

And the fitness function, the thing you were choosing every time you moved a slider, is where all the meaning lives. Change what you reward and you change what evolves, completely. I've watched evolutionary systems find gleeful, alien loopholes in a fitness function, technically-correct solutions no human would ever propose, and every one of them was a small lesson in being careful what you optimise for. It's the same lesson the paperclip thought-experiment provides.

So play with it. Melt it, starve it, inbreed it, and get a better feel for the importance of parameter selection and genetic algorithms.

 /* ============================================================
    Evolution, Live — the interactive demo for this post.
    No dependencies. A genetic algorithm evolves paintings of a target letter
    (K by default, any of A-Z from the toolbar select):
    each individual is 50 translucent triangles (10 genes apiece: six vertex
    coordinates, rgb, alpha), fitness is one minus the mean per-pixel
    difference from a 48x48 target render. Tournament selection, per-triangle
    crossover, per-gene mutation, optional two-slot elitism. Chart colours
    come from the CSS custom properties on .kr-viz, so the site's light/dark
    toggle restyles the widget live; the paintings themselves keep fixed
    colours because the target never changes.
    ============================================================ */
 (function () {
   'use strict';
   var wrap = document.getElementById('evo-demo');
   if (!wrap) return;
   var cv = document.getElementById('evoCanvas');
   var chart = document.getElementById('evoChart');

   function mulberry32(seed) {
     var a = seed >>> 0;
     return function () {
       a |= 0; a = (a + 0x6D2B79F5) | 0;
       var t = Math.imul(a ^ (a >>> 15), 1 | a);
       t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
       return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
     };
   }
   function vizColors(el) {
     var cs = getComputedStyle(el);
     function v(n) { return cs.getPropertyValue(n).trim(); }
     return { s1: v('--viz-s1'), s2: v('--viz-s2'), s3: v('--viz-s3'), s4: v('--viz-s4'),
              ink: v('--viz-ink'), muted: v('--viz-muted'), grid: v('--viz-grid'), surface: v('--viz-surface') };
   }
   var W = 0, H = 0, ctx = null, WC = 0, HC = 0, ctxC = null;
   function fitOne(canvas, h) {
     var dpr = window.devicePixelRatio || 1;
     var w = canvas.clientWidth || 600;
     canvas.style.height = h + 'px';
     canvas.width = Math.round(w * dpr); canvas.height = Math.round(h * dpr);
     var g = canvas.getContext('2d'); g.setTransform(dpr, 0, 0, dpr, 0, 0);
     return { g: g, w: w, h: h };
   }
   function fitCanvas() {
     // height tracks width: champion beside the grid on desktop, champion
     // above the grid (and a taller canvas) on phones
     var w = cv.clientWidth || 600;
     var mainH = w < 480 ? Math.round(Math.min(600, w * 1.30))
                         : Math.round(Math.max(260, Math.min(340, w * 0.5)));
     var m = fitOne(cv, mainH); ctx = m.g; W = m.w; H = m.h;
     var c = fitOne(chart, w < 480 ? 170 : 120); ctxC = c.g; WC = c.w; HC = c.h;
   }

   // ---- tunables ----
   var TRIS = 50, GENES = TRIS * 10, EVAL = 48;   // genome + fitness render size
   var EVAL_BUDGET = 120, MAX_GENS_FRAME = 2;     // work per animation frame; low
                                                  // enough that the sharpening is
                                                  // watchable rather than instant
   var ELITES = 2;
   var BG = '#1e1e21', TARGET_INK = '#fc6060';    // fixed painting colours

   // ---- state ----
   var seedBase = 20260718, reseeds = 0, rng = mulberry32(seedBase);
   var pop = [];                    // [{genes, fit, cv}], eval canvas per individual
   var gen = 0, evals = 0, bestEver = 0, bestIdx = 0;
   var hist = [], stride = 1, sinceSample = 0;    // chart history, decimated as it grows
   var running = true, raf = null;
   var target = null, targetData = null;

   // control refs
   var runBtn = document.getElementById('evoRun');
   var reseedBtn = document.getElementById('evoReseed');
   var popS = document.getElementById('evoPop'), popV = document.getElementById('evoPopVal');
   var mutS = document.getElementById('evoMut'), mutV = document.getElementById('evoMutVal');
   var pressS = document.getElementById('evoPress'), pressV = document.getElementById('evoPressVal');
   var eliteC = document.getElementById('evoElite');
   var letterSel = document.getElementById('evoLetter');
   var statusEl = document.getElementById('evoStatus');
   var genEl = document.getElementById('evoGen'), bestEl = document.getElementById('evoBest');
   var avgEl = document.getElementById('evoAvg'), divEl = document.getElementById('evoDiv');
   var evalsEl = document.getElementById('evoEvals');

   // ---- the target: one bold letter (K by default), re-rendered on change ----
   function makeTarget(letter) {
     target = document.createElement('canvas');
     target.width = EVAL; target.height = EVAL;
     var g = target.getContext('2d', { willReadFrequently: true });
     g.fillStyle = BG; g.fillRect(0, 0, EVAL, EVAL);
     g.fillStyle = TARGET_INK;
     var size = 46;
     g.font = '900 ' + size + 'px Arial, sans-serif';
     var w = g.measureText(letter).width;   // shrink-to-fit so W and M don't clip
     if (w > EVAL - 6) {
       size = Math.floor(size * (EVAL - 6) / w);
       g.font = '900 ' + size + 'px Arial, sans-serif';
     }
     g.textAlign = 'center'; g.textBaseline = 'middle';
     g.fillText(letter, EVAL / 2, EVAL / 2 + 2);
     targetData = g.getImageData(0, 0, EVAL, EVAL).data;
   }

   // ---- genome plumbing ----
   function renderGenome(genes, g, size) {
     g.fillStyle = BG; g.fillRect(0, 0, size, size);
     for (var t = 0; t < TRIS; t++) {
       var i = t * 10;
       g.fillStyle = 'rgba(' + (genes[i + 6] * 255 | 0) + ',' + (genes[i + 7] * 255 | 0) + ',' +
                     (genes[i + 8] * 255 | 0) + ',' + (genes[i + 9] * 0.85).toFixed(3) + ')';
       g.beginPath();
       g.moveTo(genes[i] * size, genes[i + 1] * size);
       g.lineTo(genes[i + 2] * size, genes[i + 3] * size);
       g.lineTo(genes[i + 4] * size, genes[i + 5] * size);
       g.closePath(); g.fill();
     }
   }
   function evaluate(ind) {
     var g = ind.cv.getContext('2d', { willReadFrequently: true });
     renderGenome(ind.genes, g, EVAL);
     var d = g.getImageData(0, 0, EVAL, EVAL).data, sum = 0;
     for (var p = 0; p < d.length; p += 4) {
       sum += Math.abs(d[p] - targetData[p]) + Math.abs(d[p + 1] - targetData[p + 1]) +
              Math.abs(d[p + 2] - targetData[p + 2]);
     }
     ind.fit = 1 - sum / (EVAL * EVAL * 3 * 255);
     evals++;
   }
   function newIndividual(genes) {
     var c = document.createElement('canvas');
     c.width = EVAL; c.height = EVAL;
     var ind = { genes: genes, fit: 0, cv: c };
     evaluate(ind);
     return ind;
   }
   function randomGenes() {
     var g = new Float32Array(GENES);
     for (var i = 0; i < GENES; i++) g[i] = rng();
     return g;
   }

   // ---- the algorithm ----
   function tournament(T) {
     var bi = (rng() * pop.length) | 0;
     for (var k = 1; k < T; k++) {
       var j = (rng() * pop.length) | 0;
       if (pop[j].fit > pop[bi].fit) bi = j;
     }
     return pop[bi];
   }
   function breed(pa, pb, pMut, dMut) {
     var g = new Float32Array(GENES);
     for (var t = 0; t < TRIS; t++) {      // crossover: whole triangles from either parent
       var src = rng() < 0.5 ? pa.genes : pb.genes;
       for (var k = 0; k < 10; k++) g[t * 10 + k] = src[t * 10 + k];
     }
     for (var i = 0; i < GENES; i++) {     // mutation: per-gene nudge, clamped to [0,1]
       if (rng() < pMut) {
         var v = g[i] + (rng() * 2 - 1) * dMut;
         g[i] = v < 0 ? 0 : v > 1 ? 1 : v;
       }
     }
     return g;
   }
   function stepGeneration() {
     var n = pop.length;
     // slider mappings: pressure 1 = uniform random parents (pure drift),
     // 10 = the whole population in every tournament (winner takes all)
     var s = +pressS.value;
     var T = Math.max(1, Math.round(1 + (n - 1) * Math.pow((s - 1) / 9, 2)));
     var mv = +mutS.value / 100;
     var pMut = Math.pow(mv, 1.5) * 0.35;
     var dMut = 0.05 + 0.45 * mv;
     var elite = eliteC.checked ? Math.min(ELITES, n) : 0;
     pop.sort(function (a, b) { return b.fit - a.fit; });
     var next = pop.slice(0, elite);
     while (next.length < n) next.push(newIndividual(breed(tournament(T), tournament(T), pMut, dMut)));
     pop = next;
     gen++;
     sample();
   }
   function generationCost() {
     return Math.max(1, pop.length - (eliteC.checked ? Math.min(ELITES, pop.length) : 0));
   }
   function runBudget() {
     var spent = 0, gens = 0;
     while (gens < MAX_GENS_FRAME && spent + generationCost() <= EVAL_BUDGET) {
       stepGeneration();
       spent += generationCost(); gens++;
     }
   }

   // ---- stats + chart history ----
   function popStats() {
     var n = pop.length, sum = 0, best = -1, bi = 0;
     for (var i = 0; i < n; i++) {
       sum += pop[i].fit;
       if (pop[i].fit > best) { best = pop[i].fit; bi = i; }
     }
     bestIdx = bi;
     // diversity: mean per-gene standard deviation, normalised so a fully
     // random population reads ~100% (std of uniform [0,1] is 1/sqrt(12))
     var m, m2, gsum = 0;
     for (var gI = 0; gI < GENES; gI++) {
       m = 0; m2 = 0;
       for (var j = 0; j < n; j++) { var v = pop[j].genes[gI]; m += v; m2 += v * v; }
       m /= n;
       gsum += Math.sqrt(Math.max(0, m2 / n - m * m));
     }
     var div = Math.min(1, (gsum / GENES) / 0.2887);
     return { best: best, avg: sum / n, div: div };
   }
   function sample() {
     if (++sinceSample < stride) return;
     sinceSample = 0;
     var st = popStats();
     if (st.best > bestEver) bestEver = st.best;
     hist.push({ g: gen, best: st.best, avg: st.avg, div: st.div });
     if (hist.length >= 480) {   // decimate so the chart array stays bounded
       hist = hist.filter(function (_, i) { return i % 2 === 0; });
       stride *= 2;
     }
   }

   // ---- drawing ----
   function layout() {
     var pad = 10, lab = 16;
     if (W < 480) {
       var s = Math.round(Math.min(W - 2 * pad, H * 0.55));
       var gy = pad + lab + s + pad + lab;
       return { champ: { x: (W - s) / 2, y: pad + lab, s: s },
                grid: { x: pad, y: gy, w: W - 2 * pad, h: H - gy - pad } };
     }
     var side = H - 2 * pad - lab;
     var gx = pad + side + pad + 8;
     return { champ: { x: pad, y: pad + lab, s: side },
              grid: { x: gx, y: pad + lab, w: W - gx - pad, h: side } };
   }
   function label(text, x, y, c, align) {
     ctx.fillStyle = c;
     ctx.font = '600 10px Poppins, sans-serif';
     ctx.textAlign = align || 'start'; ctx.textBaseline = 'alphabetic';
     ctx.fillText(text, x, y);
     ctx.textAlign = 'start';
   }
   function draw() {
     var c = vizColors(wrap);
     var L = layout();
     var st = popStats();   // refreshes bestIdx so the champion pane never lags
     ctx.clearRect(0, 0, W, H);
     ctx.fillStyle = c.surface; ctx.fillRect(0, 0, W, H);

     // champion pane: the best individual, rendered as vectors so it stays crisp
     label('CHAMPION', L.champ.x, L.champ.y - 5, c.muted);
     ctx.save();
     ctx.translate(L.champ.x, L.champ.y);
     renderGenome(pop[bestIdx].genes, ctx, L.champ.s);
     ctx.restore();
     ctx.strokeStyle = c.grid; ctx.lineWidth = 1;
     ctx.strokeRect(L.champ.x + 0.5, L.champ.y + 0.5, L.champ.s - 1, L.champ.s - 1);

     // target inset, top-right corner of the champion pane
     var ts = Math.max(34, Math.round(L.champ.s * 0.24));
     var tx = L.champ.x + L.champ.s - ts - 6, ty = L.champ.y + 6;
     ctx.drawImage(target, tx, ty, ts, ts);
     ctx.strokeStyle = c.muted; ctx.strokeRect(tx + 0.5, ty + 0.5, ts - 1, ts - 1);
     label('TARGET', L.champ.x + L.champ.s, L.champ.y - 5, c.muted, 'right');

     // population grid: every individual's evaluation render, champion ringed
     label('POPULATION', L.grid.x, L.grid.y - 5, c.muted);
     var n = pop.length;
     var cols = Math.max(1, Math.ceil(Math.sqrt(n * L.grid.w / Math.max(1, L.grid.h))));
     var rows = Math.ceil(n / cols);
     var cell = Math.floor(Math.min(L.grid.w / cols, L.grid.h / rows));
     for (var i = 0; i < n; i++) {
       var x = L.grid.x + (i % cols) * cell, y = L.grid.y + ((i / cols) | 0) * cell;
       ctx.drawImage(pop[i].cv, x + 1, y + 1, cell - 2, cell - 2);
       if (i === bestIdx) {
         ctx.strokeStyle = c.s2; ctx.lineWidth = 2;
         ctx.strokeRect(x + 1.5, y + 1.5, cell - 3, cell - 3);
         ctx.lineWidth = 1;
       }
     }
     drawChart(c);
     updateStats(st);
   }
   function drawChart(c) {
     ctxC.clearRect(0, 0, WC, HC);
     ctxC.fillStyle = c.surface; ctxC.fillRect(0, 0, WC, HC);
     var x0 = 34, x1 = WC - 62, y0 = 8, y1 = HC - 16;
     ctxC.font = '9px Poppins, sans-serif';
     [0, 0.5, 1].forEach(function (p) {
       var y = y1 - (y1 - y0) * p;
       ctxC.strokeStyle = c.grid; ctxC.lineWidth = 1;
       ctxC.beginPath(); ctxC.moveTo(x0, y + 0.5); ctxC.lineTo(x1, y + 0.5); ctxC.stroke();
       ctxC.fillStyle = c.muted; ctxC.textAlign = 'right';
       ctxC.fillText(Math.round(p * 100) + '%', x0 - 5, y + 3);
     });
     ctxC.textAlign = 'center';
     ctxC.fillStyle = c.muted;
     ctxC.fillText('generations', (x0 + x1) / 2, HC - 4);
     ctxC.textAlign = 'start';
     if (hist.length < 2) return;
     var g0 = hist[0].g, g1 = hist[hist.length - 1].g;
     function X(gi) { return x0 + (x1 - x0) * ((gi - g0) / Math.max(1, g1 - g0)); }
     function Y(v) { return y1 - (y1 - y0) * v; }
     var series = [ { k: 'best', col: c.s2, name: 'best' },
                    { k: 'avg', col: c.s1, name: 'average' },
                    { k: 'div', col: c.s3, name: 'diversity' } ];
     // series lines, labelled at their right-hand ends (labels nudged apart)
     var ends = [];
     series.forEach(function (s) {
       ctxC.beginPath();
       for (var i = 0; i < hist.length; i++) {
         var x = X(hist[i].g), y = Y(hist[i][s.k]);
         if (i === 0) ctxC.moveTo(x, y); else ctxC.lineTo(x, y);
       }
       ctxC.strokeStyle = s.col; ctxC.lineWidth = 1.6; ctxC.stroke();
       ends.push({ y: Y(hist[hist.length - 1][s.k]), col: s.col, name: s.name });
     });
     ends.sort(function (a, b) { return a.y - b.y; });
     for (var e = 1; e < ends.length; e++) {
       if (ends[e].y - ends[e - 1].y < 10) ends[e].y = ends[e - 1].y + 10;
     }
     ends.forEach(function (e2) {
       ctxC.fillStyle = e2.col;
       ctxC.fillText(e2.name, x1 + 5, Math.max(y0 + 6, Math.min(y1, e2.y)) + 3);
     });
   }
   function fmtPct(v) { return (v * 100).toFixed(1) + '%'; }
   function updateStats(st) {
     genEl.textContent = gen;
     bestEl.textContent = fmtPct(Math.max(bestEver, st.best));
     avgEl.textContent = fmtPct(st.avg);
     divEl.textContent = Math.round(st.div * 100) + '%';
     evalsEl.textContent = evals >= 10000 ? (evals / 1000).toFixed(0) + 'k' : evals;
   }

   // ---- run control ----
   function loop() {
     if (!running) return;
     runBudget();
     draw();
     raf = requestAnimationFrame(loop);
   }
   function start() { if (running) raf = requestAnimationFrame(loop); }
   function resetSim() {
     rng = mulberry32(seedBase + reseeds * 9973);
     evals = 0; gen = 0; bestEver = 0; bestIdx = 0;
     hist = []; stride = 1; sinceSample = 0;
     pop = [];
     var n = +popS.value;
     for (var i = 0; i < n; i++) pop.push(newIndividual(randomGenes()));
     sample();
   }

   runBtn.addEventListener('click', function () {
     running = !running;
     runBtn.textContent = running ? 'Pause' : 'Play';
     statusEl.textContent = running ? '' : 'Paused.';
     if (running) start(); else if (raf) cancelAnimationFrame(raf);
   });
   reseedBtn.addEventListener('click', function () {
     reseeds++; resetSim();
     if (!running) draw();
   });
   function setPop() {
     popV.textContent = popS.value;
     var n = +popS.value;
     if (n === pop.length) return;
     pop.sort(function (a, b) { return b.fit - a.fit; });
     if (n < pop.length) pop = pop.slice(0, n);
     else while (pop.length < n) {   // grow with mutated copies of survivors
       var p = pop[(rng() * pop.length) | 0];
       pop.push(newIndividual(breed(p, p, 0.1, 0.3)));
     }
     if (!running) draw();
   }
   function setMut() { mutV.textContent = mutS.value + '%'; }
   function setPress() { pressV.textContent = pressS.value; }
   popS.addEventListener('input', setPop);
   mutS.addEventListener('input', setMut);
   pressS.addEventListener('input', setPress);
   letterSel.addEventListener('change', function () {
     // a new target is a new fitness landscape: re-render it and start over
     makeTarget(letterSel.value);
     resetSim();
     if (!running) draw();
   });

   window.addEventListener('resize', function () { fitCanvas(); if (!running) draw(); });
   new MutationObserver(function () { draw(); }).observe(document.documentElement, { attributes: true, attributeFilter: ['data-theme'] });

   // ---- init ----
   for (var li = 0; li < 26; li++) {
     var opt = document.createElement('option');
     opt.value = opt.textContent = String.fromCharCode(65 + li);
     if (opt.value === 'K') opt.selected = true;
     letterSel.appendChild(opt);
   }
   makeTarget('K');
   fitCanvas();
   setMut(); setPress(); popV.textContent = popS.value;
   resetSim();
   draw();
   start();
 })();
 

When I publish a post here, the words are roughly half the file. The other half is metadata: eighty-odd lines in the page's head, plus entries in two site-wide XML files. Skipping it all would change nothing about the reading experience, but it's necessary and is checked by a machine. This post describes it. Part of the series on how this site is built.

The card in the chat window

Open Graph: five tags giving the post's title, description, preview image, address, and type. When someone pastes your link into a group chat or a social feed, the platform reads these tags to build the preview card. Without them, your link renders as bare blue text or, worse, a card assembled from whatever the scraper found first. With them, every share has a title, summary sentence, and one of my photographs at full card width. Twitter's near-identical tag family sits alongside for the platforms that read that instead.

<meta property="og:title" content="The Invisible Half of a Blog Post">
<meta property="og:description" content="Every post on this blog carries around eighty lines
      of metadata nobody reads: social preview cards, structured data, feeds, citation tags.">
<meta property="og:image" content="https://github.com/DrKenReid/DrKenReid.github.io/releases/download/photos-v1/78.png">
<meta property="og:url" content="https://www.kenreid.co.uk/blog/invisible-half-of-a-blog-post.html">
<meta property="og:type" content="article">

The description gets written by a human (me), because it's the sentence that decides whether a stranger clicks. And the image tag points at a full-resolution photo, not a thumbnail, because preview scrapers resize down gracefully and up horribly.

The librarian's copy

Below the social tags sits a block of JSON-LD: the same facts (headline, author, date, image) restated in schema.org's vocabulary, which is the format Google actually parses for rich results. Where the Open Graph tags talk to chat apps, this block talks to crawlers, and it's how I explain that this page is an article by me.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "The Invisible Half of a Blog Post",
  "author": { "@type": "Person", "name": "Ken Reid", "url": "https://www.kenreid.co.uk" },
  "datePublished": "2026-08-04",
  "image": "https://www.kenreid.co.uk/img/photography/hero/78.webp"
}
</script>

Each post carries citation tags (the format Google Scholar reads) and Dublin Core tags (the library world's metadata standard). I spent years in academia, the tags cost six lines, and if someone ever does want to cite the scrobbles post in their media-studies dissertation, their reference manager will fill every field correctly. It's courteous and costs me nothing but a couple of seconds, and might give another citation or two to my Google Scholar account!

<meta name="citation_title" content="The Invisible Half of a Blog Post">
<meta name="citation_author" content="Reid, Kenneth N.">
<meta name="citation_publication_date" content="2026/08/04">
<meta name="DC.title" content="The Invisible Half of a Blog Post">
<meta name="DC.creator" content="Kenneth N. Reid">
<meta name="DC.date" content="2026-08-04">

The site-wide ledgers

Two XML files round out every publish. The sitemap is a list of every page, so crawlers miss nothing. The RSS feed is how readers with feed readers (a small, excellent demographic) get new posts delivered without visiting, and it's essential infrastructure here. RSS predates the social platforms and will outlive several of them.

<!-- feed.xml -->
<item>
  <title>The Invisible Half of a Blog Post</title>
  <link>https://www.kenreid.co.uk/blog/invisible-half-of-a-blog-post.html</link>
  <pubDate>Tue, 04 Aug 2026 00:00:00 +0000</pubDate>
</item>

<!-- sitemap.xml -->
<url><loc>https://www.kenreid.co.uk/blog/invisible-half-of-a-blog-post.html</loc>
     <changefreq>yearly</changefreq><priority>0.8</priority></url>

Each post also declares a canonical URL: the www and non-www versions of your site, plus any URL with tracking junk appended, all count as different pages to a search engine, splitting your modest search presence into fragments. One tag per page declares the official address and the fragments reunite.

<link rel="canonical" href="https://www.kenreid.co.uk/blog/invisible-half-of-a-blog-post.html">

The real problem is discipline

Nothing above is hard. The hard part is that it's eighty lines of near-identical boilerplate on every post, and near-identical is the operative curse: copy the head from the last post, forget to change the og:image, and next week your post about cats is shared around wearing a photograph of a dog.

So: automate the checking, not necessarily the writing. My audit script validates every page's metadata (descriptions present and sensibly sized, canonical matching the filename, preview images existing, JSON-LD parsing, feed and sitemap complete) on every push. Those eighty lines stay correct because a robot reads them. Without that, my genuine recommendation would be to keep less metadata: only what you can maintain, because wrong metadata is worse than none.

Try it now, if you're on a keyboard: press Ctrl+K (Cmd+K on a Mac), or just /. A search box appears over the page. Type "abandoned" and you'll get the blog post about ruins and the gallery filtered to abandoned buildings; type "map" and you're one Enter away from the photo map. Every page, every post, and every photo tag on this site is reachable from that box, from anywhere. It's the interface programmers know from their editors, transplanted onto a personal website, and this post explains how little it took: vanilla JavaScript, no search service, no library. Part of the series on how this site is built.

What it searches, and what it doesn't

The index has three sources, in descending order of hardcoded-ness. First, the site's ten pages, written directly into the script with a subtitle each ("490+ photos", "571 saved passages"); they change a few times a year, so a hardcoded list is simply correct. Second, every blog post, fetched from the same JSON file that runs the blog listing, so the palette knows each post's title, tags, and excerpt without any separate index to maintain; publish a post, and the palette knows it. Third, the gallery's ten photo tags, each deep-linking to the gallery pre-filtered, which is what makes "wildlife" a destination rather than a word.

The palette doesn't search inside post text, and it doesn't index the 571 quotes or 492 photos individually. Full-text search of a static site is doable (ship an index file, use a library like Lunr) but the index grows with every word you write, and I certainly wouldn't want paragraph-level results on a personal site. Titles, tags, and excerpts answer "take me to the thing I half-remember". The quote wall also gets one entry, not 571.

Fuzzy

Real fuzzy-search libraries use sophisticated string mathematics. Mine uses a five-tier ladder: a whole-string match scores 120, a match at the start of the text 100, a match at a word boundary 80, a match anywhere 60, and letters-in-the-right-order (the "phto" case) 25. Each source gets a weighting on top (a title match beats a tag match beats an excerpt match), everything above zero is sorted, and the top ten render.

Arrow keys move the selection and wrap at the ends; Tab is trapped so focus can't escape the dialog; Escape closes; a screen-reader announcement reports the result count as you type; and the input is focused synchronously on opening, because mobile keyboards only appear if the focus happens inside the user's gesture.

Why bother, on a site this small?

Part of the answer is who visits: this site's readers skew technical, their muscle memory already knows Ctrl+K. Part is that the palette compounds with the site's growth: at ten pages it's a nicety, at 50 posts it's the fastest way to find anything, and every post published makes it slightly more useful at zero marginal cost. And part of it, in the spirit of candour that runs through this series: it's the kind of detail that's simply pleasing to have on one's own website. It scratches an itch for me, and it was fun to implement.

One keydown listener for the shortcut, a dialog built on first open (not before; most visitors never press it), a scoring function, and a list that re-renders per keystroke. Resist the URL that offers to do it for you as a hosted service; for a site whose whole index fits in one JSON fetch, the dependency would outweigh the feature.

Take the code

The whole thing is below, copy-paste ready. The JavaScript is the complete js/palette.js from this site: swap the PAGES array for your own pages, point the two fetch calls at whatever JSON you have (or delete them, along with the sources they feed), and it will run on any static site with no build step. The CSS is the full set of palette styles, dark theme included; the [data-theme="dark"] selectors assume a theme attribute on <html>, so adapt those to however your site handles dark mode.

/**
 * palette.js — site-wide command palette (Ctrl/Cmd+K, or '/').
 *
 * Fuzzy search across pages, blog posts (data/posts.json), photo tags,
 * and map places (deep links to gallery.html?tag=... and map.html?region=...).
 * Vanilla JS, builds its DOM on first open, loaded with defer on every page.
 */
(() => {
    const prefix = /\/blog\//.test(window.location.pathname) ? '../' : './';

    const PAGES = [
        { title: 'Home', sub: 'Intro and latest posts', url: 'index.html' },
        { title: 'About', sub: 'Who I am', url: 'about.html' },
        { title: 'Data Science', sub: 'Projects & publications', url: 'data_science.html' },
        { title: 'Photography', sub: '490+ photos', url: 'gallery.html' },
        { title: 'Photo Map', sub: 'Photographs by place', url: 'map.html' },
        { title: 'Music', sub: 'Guitar & listening stats', url: 'music.html' },
        { title: 'Literature', sub: 'Reviews & reading stats', url: 'literature.html' },
        { title: 'Quote Wall', sub: '571 saved passages', url: 'quotes.html' },
        { title: 'Blog', sub: 'All posts', url: 'blog.html' },
        { title: 'Contact', sub: 'Get in touch', url: 'contact.html' }
    ];

    const PHOTO_TAGS = ['wildlife', 'portrait', 'bw', 'architecture', 'abandoned',
        'urban', 'nature', 'silhouette', 'landscape', 'winter'];

    let overlay = null;
    let input = null;
    let list = null;
    let items = [];
    let active = 0;
    let posts = null;
    let places = null;
    let indexRequested = false;

    // Score ladder: exact 120, prefix 100, word boundary 80, anywhere 60,
    // letters-in-the-right-order 25, no match 0.
    function score(query, text) {
        if (!text) return 0;
        const q = query.toLowerCase();
        const t = text.toLowerCase();
        if (t === q) return 120;
        const idx = t.indexOf(q);
        if (idx === 0) return 100;
        if (idx > 0) return t[idx - 1] === ' ' ? 80 : 60;
        let ti = 0;
        for (const ch of q) {
            ti = t.indexOf(ch, ti);
            if (ti === -1) return 0;
            ti++;
        }
        return 25;
    }

    function collectResults(query) {
        const q = query.trim();
        const results = [];
        const add = (s, kind, title, sub, href) => {
            if (s > 0) results.push({ score: s, kind, title, sub, href });
        };

        // With an empty query the palette shows the pages as a menu.
        for (const p of PAGES) {
            add(q ? Math.max(score(q, p.title), score(q, p.sub) * 0.5) : 10,
                'Page', p.title, p.sub, prefix + p.url);
        }
        if (q) {
            for (const p of posts || []) {
                const tags = (p.tags || []).join(' ');
                add(Math.max(score(q, p.title), score(q, tags) * 0.8, score(q, p.excerpt || '') * 0.4),
                    'Post', p.title, (p.tags || []).join(' · '), prefix + (p.url || ''));
            }
            for (const t of PHOTO_TAGS) {
                add(score(q, t) * 0.9, 'Photos', `${t[0].toUpperCase()}${t.slice(1)} photos`,
                    'Gallery filter', `${prefix}gallery.html?tag=${t}`);
            }
            for (const pl of places || []) {
                add(score(q, pl.name) * 0.9, 'Place', pl.name,
                    `${pl.count} photo${pl.count === 1 ? '' : 's'} on the map`,
                    `${prefix}map.html?region=${encodeURIComponent(pl.name)}`);
            }
        }
        return results.sort((a, b) => b.score - a.score).slice(0, 10);
    }

    function render(query) {
        items = collectResults(query);
        active = 0;
        overlay.querySelector('.kr-palette-live').textContent =
            items.length ? `${items.length} result${items.length === 1 ? '' : 's'}` : 'No results';
        list.innerHTML = items.length
            ? items.map((r, i) =>
                `<a href="https://www.kenreid.co.uk/blog/${r.href}" class="kr-palette-item${i === 0 ? ' active' : ''}" data-i="${i}">` +
                `<span class="kr-palette-kind">${r.kind}</span>` +
                `<span class="kr-palette-text"><span class="kr-palette-title">${r.title}</span>` +
                (r.sub ? `<span class="kr-palette-sub">${r.sub}</span>` : '') +
                '</span></a>').join('')
            : '<div class="kr-palette-empty">No matches. Try a post title, page, photo tag, or place.</div>';
    }

    function setActive(i) {
        const els = list.querySelectorAll('.kr-palette-item');
        if (!els.length) return;
        active = (i + els.length) % els.length;
        els.forEach((el, j) => el.classList.toggle('active', j === active));
        els[active].scrollIntoView({ block: 'nearest' });
    }

    function build() {
        overlay = document.createElement('div');
        overlay.className = 'kr-palette-overlay';
        overlay.innerHTML =
            '<div class="kr-palette" role="dialog" aria-modal="true" aria-label="Site search">' +
            '<input type="text" class="kr-palette-input" placeholder="Search posts, pages, photos…" aria-label="Search site" role="combobox" aria-expanded="true" aria-autocomplete="list">' +
            '<div class="kr-palette-list" role="listbox"></div>' +
            '<div class="kr-palette-live sr-only" aria-live="polite"></div>' +
            '<div class="kr-palette-foot"><span>&uarr;&darr; navigate</span><span>&crarr; open</span><span>esc close</span></div>' +
            '</div>';
        document.body.appendChild(overlay);
        input = overlay.querySelector('.kr-palette-input');
        list = overlay.querySelector('.kr-palette-list');

        overlay.addEventListener('mousedown', (e) => {
            if (e.target === overlay) close();
        });
        input.addEventListener('input', () => render(input.value));
        input.addEventListener('keydown', (e) => {
            if (e.key === 'ArrowDown') { e.preventDefault(); setActive(active + 1); }
            else if (e.key === 'ArrowUp') { e.preventDefault(); setActive(active - 1); }
            else if (e.key === 'Tab') {
                // Focus trap: the input is the palette's single focus stop;
                // Tab moves the selection instead of leaving the dialog.
                e.preventDefault();
                setActive(active + (e.shiftKey ? -1 : 1));
            } else if (e.key === 'Enter') {
                e.preventDefault();
                if (items[active]) window.location.href = items[active].href;
            } else if (e.key === 'Escape') close();
        });
        list.addEventListener('mousemove', (e) => {
            const el = e.target.closest('.kr-palette-item');
            if (el) setActive(Number(el.dataset.i));
        });
    }

    function loadIndex() {
        if (indexRequested) return;
        indexRequested = true;
        const grab = (url, apply) => fetch(prefix + url)
            .then((r) => r.json())
            .then((data) => { apply(data); render(input.value); })
            .catch(() => { /* source stays empty; pages still work */ });
        grab('data/posts.json', (data) => { posts = data; });
        grab('data/photo-locations.json', (data) => {
            places = (data.regions || []).map((r) => ({ name: r.name, count: (r.photos || []).length }));
        });
    }

    function open() {
        if (!overlay) build();
        loadIndex();
        overlay.classList.add('is-open');
        document.body.classList.add('kr-palette-open');
        input.value = '';
        render('');
        // Focus synchronously: mobile browsers only raise the soft keyboard
        // when focus happens inside the user gesture.
        input.focus();
        setTimeout(() => input.focus(), 30);
    }

    function close() {
        overlay.classList.remove('is-open');
        document.body.classList.remove('kr-palette-open');
    }

    const isOpen = () => Boolean(overlay) && overlay.classList.contains('is-open');

    document.addEventListener('keydown', (e) => {
        if ((e.ctrlKey || e.metaKey) && e.key.toLowerCase() === 'k') {
            e.preventDefault();
            if (isOpen()) close(); else open();
        } else if (e.key === '/' && !isOpen()) {
            const t = e.target;
            const typing = t && (t.tagName === 'INPUT' || t.tagName === 'TEXTAREA' || t.isContentEditable);
            if (!typing) { e.preventDefault(); open(); }
        } else if (e.key === 'Escape' && isOpen()) {
            close();
        }
    });

    // Nav hint: the header is injected by shared-components.js, so wait
    // for #nav to exist before appending the Search pill.
    document.addEventListener('DOMContentLoaded', () => {
        const tryInsert = () => {
            const nav = document.getElementById('nav');
            if (!nav || document.querySelector('.kr-palette-hint')) return Boolean(nav);
            const li = document.createElement('li');
            li.innerHTML = '<a href="#" class="kr-palette-hint" role="button" aria-label="Search the site (Ctrl+K)">' +
                '<svg viewBox="0 0 24 24" width="14" height="14" aria-hidden="true"><path fill="currentColor" d="M15.5 14h-.79l-.28-.27a6.5 6.5 0 1 0-.7.7l.27.28v.79l5 4.99L20.49 19zm-6 0A4.5 4.5 0 1 1 14 9.5 4.5 4.5 0 0 1 9.5 14z"/></svg>' +
                '<span class="kr-palette-word">Search</span>' +
                '<span class="kr-palette-kbd">Ctrl K</span></a>';
            li.querySelector('a').addEventListener('click', (e) => {
                e.preventDefault();
                open();
            });
            nav.appendChild(li);
            return true;
        };
        if (!tryInsert()) {
            const mo = new MutationObserver(() => {
                if (tryInsert()) mo.disconnect();
            });
            mo.observe(document.body, { childList: true, subtree: true });
        }
    });
})();

/* Command palette (Ctrl/Cmd+K — js/palette.js) */
.kr-palette-overlay {
  position: fixed;
  inset: 0;
  z-index: 6000;
  background: rgba(10, 10, 10, 0.55);
  backdrop-filter: blur(3px);
  display: none;
  align-items: flex-start;
  justify-content: center;
  padding: 12vh 16px 0; }

.kr-palette-overlay.is-open {
  display: flex; }

body.kr-palette-open {
  overflow: hidden; }

.kr-palette {
  width: 100%;
  max-width: 560px;
  background: #ffffff;
  border: 1px solid #e2e2e2;
  border-radius: 14px;
  box-shadow: 0 24px 70px rgba(0, 0, 0, 0.45);
  overflow: hidden; }

.kr-palette-input {
  width: 100%;
  border: 0;
  outline: none;
  background: transparent;
  padding: 18px 20px;
  font-size: 17px;
  font-family: "Poppins", sans-serif;
  color: #252525;
  border-bottom: 1px solid #ececec;
  box-shadow: none !important; }

.kr-palette-list {
  max-height: 46vh;
  overflow-y: auto;
  padding: 8px; }

.kr-palette-item {
  display: flex;
  align-items: center;
  gap: 12px;
  padding: 10px 12px;
  border-radius: 8px;
  text-decoration: none; }

.kr-palette-item.active {
  background: rgba(252, 96, 96, 0.1); }

.kr-palette-kind {
  flex-shrink: 0;
  width: 52px;
  font-size: 10px;
  font-weight: 600;
  letter-spacing: 0.08em;
  text-transform: uppercase;
  color: #999; }

.kr-palette-text {
  display: flex;
  flex-direction: column;
  min-width: 0; }

.kr-palette-title {
  font-size: 14.5px;
  font-weight: 500;
  color: #252525;
  white-space: nowrap;
  overflow: hidden;
  text-overflow: ellipsis; }

.kr-palette-item.active .kr-palette-title {
  color: #c53030; }

.kr-palette-sub {
  font-size: 12px;
  color: #8a8a8a;
  white-space: nowrap;
  overflow: hidden;
  text-overflow: ellipsis; }

.kr-palette-empty {
  padding: 24px 16px;
  text-align: center;
  font-size: 14px;
  color: #8a8a8a; }

.kr-palette-foot {
  display: flex;
  gap: 18px;
  justify-content: center;
  padding: 10px;
  border-top: 1px solid #ececec;
  font-size: 11px;
  color: #9a9a9a; }

[data-theme="dark"] .kr-palette {
  background: #222222;
  border-color: #3a3a3a; }

[data-theme="dark"] .kr-palette-input {
  color: #e2e2e2;
  border-bottom-color: #333; }

[data-theme="dark"] .kr-palette-item.active {
  background: rgba(252, 96, 96, 0.14); }

[data-theme="dark"] .kr-palette-title {
  color: #e2e2e2; }

[data-theme="dark"] .kr-palette-item.active .kr-palette-title {
  color: #fc6060; }

[data-theme="dark"] .kr-palette-foot {
  border-top-color: #333; }

/* Nav hint pill */
.kr-palette-hint {
  display: inline-flex;
  align-items: center;
  gap: 7px; }

.kr-palette-kbd {
  font-size: 10.5px;
  font-weight: 600;
  letter-spacing: 0.05em;
  border: 1px solid currentColor;
  border-radius: 5px;
  padding: 1px 6px;
  opacity: 0.75; }

.kr-palette-word {
  display: none; }

/* In the mobile hamburger menu the hint becomes a plain "Search" item
   (keyboard shortcut label makes no sense on touch) */
@media (max-width: 991px) {
  .kr-palette-kbd {
    display: none; }
  .kr-palette-word {
    display: inline; } }

Back to all posts

On the banks of the Clyde in Glasgow there is a statue of Dolores Ibárruri, La Pasionaria, arms raised, with an inscription that reads "better to die on your feet than live forever on your knees". It honours the Glaswegians who volunteered to fight Franco's fascists in Spain in the 1930s, many of whom did not come home. In late July 2026, a short walk upriver at Glasgow Green, people at a rally were filmed performing Nazi salutes, and sadly more than one or two.

The rally was organised by a group calling itself Unite the Clans, and around 1,700 people, by Police Scotland's count, turned up. So did counter-demonstrations organised by Stand Up To Racism and Women Against The Far Right. By the end of the day, footage circulating online showed the salutes, a non-white delivery worker being assaulted, and a crowd chanting "send them home" at a black passerby. Police Scotland made fifteen arrests on the day, on charges including possession of an offensive weapon, assault, and breach of the peace, and said plainly that some attendees had arrived intending "pre-meditated disorder, violence and intimidation". The First Minister called it "a demonstration of far-right hatred". A Chief Superintendent called the scenes "nothing short of disgraceful" and promised those involved that officers would be knocking on their doors. All of this happened while Glasgow was hosting Commonwealth Games delegations from seventy-four nations.

I'm from Scotland. I grew up on the story that Scotland was always the "open arms" country, and the English were the racist, anti-immigrant fannies down south. I remember when I was younger, a teenager, I had a lot of arguments with friends and online on forums about freedom of speech. I was always told that Americans have "true" freedom of speech and can't be arrested for anything they say, while in the UK we are limited in what we can say. But the only examples these people could come up with were slurs, or inciting violence. They argued that "true freedom" should allow anyone to speak and say anything, and it has some merit, at least at first glance. This brings us to the topic of this article: the paradox of tolerance, which I think is widely misunderstood by both the people who invoke it and the people who dismiss it.

What Popper actually wrote

Karl Popper was a Viennese philosopher of Jewish descent who fled Austria ahead of the Anschluss and spent the war years teaching in New Zealand, writing what he called his "war effort": a philosophical defence of open societies against their enemies. The Open Society and Its Enemies came out in 1945. The famous passage is not in the main text, it sits in a footnote to chapter seven, as if Popper considered the point almost too obvious to argue at length:

Unlimited tolerance must lead to the disappearance of tolerance. If we extend unlimited tolerance even to those who are intolerant, if we are not prepared to defend a tolerant society against the onslaught of the intolerant, then the tolerant will be destroyed, and tolerance with them.

— Karl Popper, The Open Society and Its Enemies

It fits on a placard and it appears in my social media feeds every time something like Glasgow Green happens. But Popper kept writing:

In this formulation, I do not imply, for instance, that we should always suppress the utterance of intolerant philosophies; as long as we can counter them by rational argument and keep them in check by public opinion, suppression would certainly be most unwise. But we should claim the right to suppress them if necessary even by force; for it may easily turn out that they are not prepared to meet us on the level of rational argument, but begin by denouncing all argument; they may forbid their followers to listen to rational argument, because it is deceptive, and teach them to answer arguments by the use of their fists or pistols. We should therefore claim, in the name of tolerance, the right not to tolerate the intolerant.

— Karl Popper, The Open Society and Its Enemies

Popper is not saying "silence anyone whose views you find intolerant". He is saying almost the opposite: while a movement is willing to argue, argue back, and suppression is "most unwise". The right to suppress is held in reserve for the moment a movement exits the argument, when it answers speech with fists.

The peace treaty framing

The cleanest way out of the paradox is to stop treating tolerance as a virtue and start treating it as an agreement, a reframing I owe to Yonatan Zunger's essay "Tolerance is not a moral precept", which calls tolerance a peace treaty, and reminds us that a peace treaty is not a suicide pact. A virtue is something you must extend to everyone unconditionally, which is how you end up handing a microphone to people who want you dead. An agreement, however, is different. Tolerance is a mutual pact: I accept your right to live as you choose, and you accept mine. Like any pact, it protects the people who keep it. Someone who performs a Nazi salute in a public park is not exercising the pact; they are announcing, in the most legible gesture the twentieth century produced, that they have left it. The paradox dissolves once you see it this way. You are not being intolerant when you refuse to tolerate fascism, any more than you are being violent when you defend yourself from a punch.

I like this framing because it answers the smug gotcha ("so much for the tolerant left!") without any philosophical hand-waving, and because it matches how every other agreement in life works. Nobody thinks a referee is betraying the spirit of football by ever sending players off.

Where the threshold sits

Where is the line?

Drawing the line too tightly is of course problematic, where every uncomfortable opinion becomes "intolerance" and suddenly a government can use "hate speech" as a silencing tool. This is the slippery slope the free speech absolutists worry about. Whoever holds the definition of "intolerable" today will not hold it forever, and a rule you would not want your worst enemy to wield is a rule you should hesitate to create. It's why Popper's default is argument and public opinion, with suppression as a reserve power rather than a first resort.

Drawing the line so loosely that nothing short of actual violence ever crosses it, by which point the argument phase is long over, is also problematic. Weimar Germany had a liberal constitution, a functioning press, and elections, and a movement that used all three while openly promising to abolish them. The people who watched that happen did not conclude that more debate was needed. Post-war Germany clearly learned this lesson, as exhibited by their new laws: the Nazi salute is a criminal offence, parties hostile to the constitutional order can be banned, and this framework (they call it militant democracy) has coexisted with one of the freest societies in Europe for eighty years. The slippery slope has guardrails you can build.

A salute that means "the Holocaust was good, actually", performed by a crowd that assaulted a delivery worker for his skin colour, is not a contribution to public discourse that we must meet with rational argument. It is the fists-instead-of-arguments moment Popper described. The counter-protesters, the fifteen arrests, and the door-knocking that Police Scotland promised are society using proportionate, lawful force to defend the conditions under which everyone else gets to keep arguing. But I don't think this is the end of it.

What Scotland's law actually does

People are often surprised to learn that the UK, unlike Germany, does not ban the Nazi salute as a gesture. What we have instead is a patchwork: public order offences covering threatening and abusive behaviour, and in Scotland the Hate Crime and Public Order (Scotland) Act 2021, which since 2024 has criminalised behaviour intended to stir up hatred against protected groups. Whether a given salute is criminal depends on context: who saw it, what accompanied it, what it was intended to do. This context-sensitivity is frustrating to people who want a bright line, but it is also recognisably Popper's structure translated into statute: the gesture alone is speech, the gesture as part of an intimidation campaign is conduct, and the law reserves its force for the latter. A teacher showing a class what the salute is and educating on the fascism that it implies is very different from a crowd doing it at an anti-immigration (or really, anti-black) rally.

Whether the patchwork is enough is a live political question in Scotland right now. What I am confident about is the direction of the burden. The question is not "can we justify restricting the fascists?" It is "can a society that watched the twentieth century happen justify treating open fascist organising as ordinary politics?" Eighty years of German experience says restriction is compatible with freedom. Ten years of watching the far right grow on unmoderated platforms says indifference is not.

The view from the receiving end

On the Saturday afternoon of the rally, a 31-year-old delivery driver named Muhammad Jawad Yaqoob picked up an order from the Palm Tree Kitchen on Bridge Street, just south of the river, as dozens of demonstrators went past chanting "get them out". He later told the BBC what happened next: a man shouted "you brownie, you Asian, you Pakistani" at him, and another threw a traffic cone at him while the group cheered. He ran behind a nearby restaurant. Then his phone buzzed: Uber's app, knowing nothing, was asking him to complete the delivery. So he did. He delivered the food, went back to ask the shop for CCTV footage (it had none), and reported the attack to the police anyway.

By the time he got home, footage of the attack had spread across social media, as far as Pakistan, where he is from. Days later he had still not returned to work, though he knows he will have to: "I have bills to pay so I can't just sit at home." He told the BBC that until that weekend he had found Glasgow a friendly place. "Everybody gives me tips on my deliveries and says hello, greeting me with smiley faces. But after this incident I feel scared, I don't feel like I know this version of Glasgow. Police have not protected me and there has been no further action." Fifteen arrests on the day is the aggregate response working roughly as designed. The man who threw the cone was, as of the BBC's report, still only "being investigated". Society-level self-defence and individual protection are not the same thing, and the gap between them is where people like Yaqoob actually live.

Dr Zubir Ahmed, the Labour MP for Glasgow South West, asked the question this whole essay has been circling: "How did we get here? How have we reached a point where people in one of the most diverse communities in Scotland now think twice about stepping outside?" His answer is accurate: "We've been sold a story about ourselves: that racism here in Scotland isn't really as bad." And: "We fix nothing by fooling ourselves that we're better than others." A self-image inherited from your grandparents is not a property you personally possess; it is a standard you either meet or fail. The Justice Secretary, Neil Gray, promised that "people will be brought to justice". Yaqoob, whose attacker cheered and walked away, is still the one attacked in the street, still afraid to go to his job, while we discuss political philosophy. That makes me feel deep shame in the systems in Scotland that fail to protect and help people like Yaqoob thrive. It makes me feel a deep sadness for what I thought I missed: a Scotland that is welcoming, arms open to people who aren't part of its white majority. It makes me wonder if I'd return with my partner, who, unlike me, is not Scottish and is not white. Would Scotland be a welcoming place for her like I thought, or would she be attacked on the street?

The part that is ours to do

Popper's first line of defence was "rational argument and public opinion", which is to say: us. The Overton window does not move because bad people are strong. It moves because ordinary people recalibrate what they will let pass in a pub, a group chat, a family dinner, a football stand. Every "it's just banter" is a data point in someone's model of what this society permits. The counter-demonstration is the visible version of this, and Glasgow's showed up in numbers.

La Pasionaria's statue has stood on the Clyde since 1980, paid for by public subscription because the city council of the day would not fund it, which is its own reminder that Glasgow's anti-fascism was always the people's project before it was official. The men it commemorates did not go to Spain because they had read the footnotes of a book that would not be written for another decade. They went because they understood, without needing the philosophy, that some movements do not stay in the argument phase, and that the time to stop them is while stopping them is still cheap. The salutes at Glasgow Green are a test of whether we still understand it.

Hero image: Arthur Dooley's La Pasionaria memorial to the International Brigades on Custom House Quay, Glasgow. Photograph by Alex Liivet, via Wikimedia Commons (CC0, public domain).

Back to all posts