Brain cells in a dish that play games
Scientists can grow real brain cells (neurons) on a small chip covered in tiny metal contacts called electrodes. An electrode can do two things: give the cells a small electrical zap, or listen to the cells firing. Neurons talk to each other with quick electrical blips called spikes.
In 2022 a team in Australia connected a dish like this to the game Pong. Zaps on one side of the dish told the cells where the ball was, and spikes on the other side moved the paddle. When the paddle hit the ball, the cells got a neat, regular pattern of zaps. When it missed, they got a few seconds of random static. Within minutes, the cells were hitting the ball more often.
Why would that work? There's no reward. The idea, called the free energy principle, is that living things try to keep their world predictable. A regular drumbeat is easy to predict. Random static never is. The only way for the cells to avoid the static is to stop doing whatever caused it.
The experiment
Here a computer pretends to be the dish: 100 fake neurons wired up like the real thing. The game is a tiny Doom arena. You stand in the middle, monsters (imps) walk toward you, and there are only three buttons: turn left, turn right, fire.
The cells "see" the game through 7 electrodes, one for each direction the nearest imp could be in (far left to far right). The closer the imp, the faster the zaps. Whichever of the three "button" areas of the dish fires the most picks the move.
Try this
- Press Run, then 16× to speed up time.
- Watch the dish (next to the game on a wide screen, below it on a narrow one). The lines are connections, and thicker means stronger. At first they are all the same, so the cells pick moves at random.
- After a minute or two, look at the table further down: each row should light up one button. Imp on the left means turn left, imp in the center means fire.
- Now that it has learned, press Pause, pick the Noise brush, and click the "center" row twice. You just made the cells forget how to shoot. Press Run and watch them relearn it.
- Click New culture, pick No feedback, and run again. Without the drumbeat and static, it never learns.
Why would a dish play at all?
In 2022, Kagan and colleagues grew cortical neurons on a high-density multi-electrode array and connected them to Pong. The ball's position was delivered as stimulation to a sensory area of the dish, and activity in two motor areas moved the paddle. There was no dopamine and no reward. The only feedback was the stimulation that followed each rally. A hit was followed by predictable stimulation. A miss was followed by a few seconds of random stimulation at random electrodes. Within minutes of play, cultures given this feedback hit the ball more often than cultures without it.
The account they give is the free energy principle. A self-organizing system that persists has to keep its sensory states within bounds, and it does that by minimizing surprise about its inputs. (Strictly, it minimizes variational free energy, a quantity that is always at least as large as surprise, so pushing it down pushes surprise down too. Section 2 makes this precise.) A culture has two ways to do that. It can update its internal model so the input it gets becomes less surprising. Or it can act so that it gets input it can already predict. Random stimulation cannot be predicted by any model, so the only way to avoid it is to stop doing whatever triggers it.
I wanted to see that mechanism working, so I built it. Below is a simulated culture of 100 spiking neurons on a virtual electrode array, connected to a small Doom arena. Run it first, then the rest of the post goes through the math one piece at a time.
The experiment
The arena is a stripped-down version of ViZDoom's Defend the Center. You stand in the middle, imps walk in from every side, and the only moves are turn left, turn right, and fire. The direction of the nearest imp is place-coded onto one of 7 stimulating electrodes, from far left (L3) to far right (R3), and its distance is rate-coded: closer means faster pulses. Each 50 ms tick, the action is whichever of the 3 motor regions fires the most. A kill triggers 100 ms of regular 100 Hz pulses on every electrode. A hit triggers 400 ms of noise, and a missed shot triggers 150 ms of noise. The game pauses during feedback, as in the Pong setup.
Press Run and switch to 16× if you're impatient. For the first minute or so the culture flails. The green dots in the surprise chart start at 1 bit, because the culture has not learned that the pattern is a pattern yet. Once they drop, the policy table snaps into shape and the kill rate climbs. Try the other feedback modes on a new culture, and try lowering the noise entropy.
Graduate notes
Two framing notes before the math. First, the free energy principle is a claim about any system with a Markov blanket that persists: such a system can be described as minimizing variational free energy \(F \ge -\ln p(o)\) with respect to its internal states (perception) and active states (action). Here the "blanket" is concrete. Sensory states are the 7 stimulating electrodes, active states are the 3 motor read-out regions, and internal states are membrane potentials and synaptic weights.
Second, the original paper's use of the word "sentience" was criticized in later commentary, and alternative explanations of its learning effect have been proposed. Nothing here depends on that framing. The question is narrower: does surprise-gated plasticity produce the behavior, and what does it look like as inference?
1. How undecided is the culture?
Imagine guessing what the cells will do next. If they always fire when the imp is in the center, guessing is easy. If they pick left, right or fire completely at random, like rock-paper-scissors, guessing is as hard as it gets.
Scientists measure "how hard to guess" with a number called entropy, counted in bits. One bit is the uncertainty in a fair coin flip. A totally random choice between 3 options is about 1.58 bits. A choice that's always the same is 0 bits.
Before learning, every connection in the dish is the same strength, so nothing tips the balance. The cells are as undecided as possible. That starting point is called maximum entropy: when you know nothing, assume nothing.
1. Before learning: the maximum-entropy prior
For each state \(s\), meaning each of the 7 sectors the nearest imp can be in, the culture has a policy \(p(a \mid s)\) over the three actions \(a \in \{\text{Left}, \text{Right}, \text{Fire}\}\). How committed that policy is gets measured by its entropy:
$$H(a \mid s) = -\sum_{a} p(a \mid s)\,\log_2 p(a \mid s).$$
Jaynes' maximum entropy principle says that if you know nothing beyond the constraints, you should pick the distribution with the largest entropy that satisfies them. With no constraints except that probabilities sum to one, that is the uniform distribution, and \(H = \log_2 3 \approx 1.585\) bits.
The simulation does not impose this. It falls out of symmetry. Every sensory neuron starts with the same weight \(w_0\) onto every motor neuron, and its outgoing weights share a fixed synaptic budget, \(\sum_j w_{ij} = N_M w_0\). With nothing to break the tie, the three motor regions receive the same drive, and which one fires most on a given tick is decided by intrinsic noise. An unconnected culture really is a max-entropy model: its spontaneous activity samples every action equally.
The quantity \(-\log_2 p(a \mid s)\) is the surprise of an action. Entropy is just the expected surprise. Drag the point below to feel how the two trade off.
Graduate notes
Maximum entropy does more than pick the uniform prior. Maximize \(H(p)\) subject to a constraint on the expected drive, \(\sum_a p(a)\,\bar W_{sa} = c\), and the Lagrangian gives a Gibbs distribution:
$$p(a \mid s) = \frac{e^{\beta \bar W_{sa}}}{\sum_{a'} e^{\beta \bar W_{sa'}}},$$
where \(\beta\) is the Lagrange multiplier, or inverse temperature. The culture's winner-take-all readout is only approximately softmax, but the "culture temperature" slider above plays exactly this role. It is the same \(\beta\) as in the Ising model at criticality. As \(\beta \to 0\), every state returns to \(\log_2 3\) bits whatever the weights. As \(\beta \to \infty\), the policy becomes the argmax. Learning changes the constraint (the weights). Temperature sets how sharply the policy follows it.
Note that the policy shown in the table is estimated from 24 probe trials per state, with plasticity off and the imp at mid-range. Its entropy is therefore a plug-in estimate, biased downward by roughly \((|A|-1)/(2N\ln 2) \approx 0.06\) bits (the Miller–Madow correction).
2. Surprise: what you can learn and what you can't
Surprise is how shocked you are by what happens. If your friend says "hi" every morning, it stops surprising you after a few days. That surprise went away because you learned the pattern.
Static is different. It's random every time, so no amount of learning makes it predictable. That leftover surprise never goes away. Scientists call it the static's entropy, the same "how hard to guess" number from before.
So the cells face two kinds of surprise. The kind they can get rid of by learning, which is the drumbeat. And the kind they can only get rid of by changing what they do so the static stops coming. That second route is why a dish ends up playing the game.
2. Surprise: what the culture can and cannot learn away
The culture also has a model of its input. In plain terms, it keeps a few "templates" of what feedback usually looks like. Each template is a grid saying how likely each electrode is to pulse in each 5 ms bin. When a burst \(x\) arrives, the culture checks how well its templates explain it, then nudges the best-fitting template toward what it just saw. Written out (a mixture of Bernoulli templates, where each template \(k\) has weight \(\pi_k\) and per-bin pulse probabilities \(\theta_{kb}\)):
$$q(x) = \sum_{k} \pi_k \prod_{b} \theta_{kb}^{\,x_b}\,(1-\theta_{kb})^{1-x_b}.$$
The surprise of a burst is \(-\log_2 q(x)\). I report it per bin as \(u = -\tfrac{1}{B}\log_2 q(x)\), so \(u = 1\) bit means the culture's prediction of each bin is no better than a coin flip, and \(u = 0\) means it saw it coming.
The important identity is how expected surprise splits. In words: average surprise = how random the world really is + how wrong your model is. The second part is the KL divergence, a measure of the gap between the true distribution \(p\) and the model's prediction \(q\). It is zero only when the model is exactly right. Formally:
$$\underbrace{\mathbb{E}_{x\sim p}\!\left[-\log_2 q(x)\right]}_{\text{expected surprise}} \;=\; \underbrace{H(p)}_{\text{entropy of the world}} \;+\; \underbrace{D_{\mathrm{KL}}(p \,\|\, q)}_{\text{model error}}.$$
Learning, meaning perception, can drive the KL term to zero and nothing more. For the pattern, \(H(p) = 0\): every bin is determined, so once the template is learned the surprise falls to about 0.03 bits (it is not exactly zero because templates are clipped to [0.02, 0.98]). For noise that pulses each bin with probability \(\rho\), the entropy is
$$H(\rho) = -\rho\log_2\rho - (1-\rho)\log_2(1-\rho),$$
which is 1 bit per bin at \(\rho = 0.5\). No model of any size can predict it better. The free energy principle wraps both routes into one quantity. In words: free energy = surprise + how wrong the culture's explanation of the burst is, so it can never be smaller than the surprise itself. With hidden causes \(z\), here which template generated the burst,
$$F = \mathbb{E}_{q(z)}\!\left[\log q(z) - \log p(x, z)\right] = -\log p(x) + D_{\mathrm{KL}}\!\left(q(z)\,\|\,p(z \mid x)\right) \;\ge\; -\log p(x).$$
Updating the model tightens the bound. The only way left to lower the surprise itself is to change which \(x\) arrives, which means acting. In this setup, the irreducible entropy of the noise is the whole lever.
Graduate notes
The culture's model is a mixture of \(K = 3\) Bernoulli templates with Laplace-smoothed weights \(\pi_k = (n_k + 1)/(N + K)\), learned by online hard EM. After each burst, the template with the highest posterior responsibility \(z^* = \arg\max_k \pi_k\,p(x \mid k)\) moves toward \(x\) with step 0.25. With a point-mass \(q(z) = \delta_{z^*}\) the bound reads \(F = -\ln p(x, z^*) \ge -\ln p(x)\), because the entropy of \(q\) is zero. The simulation reports the exact mixture surprise \(-\log_2 \sum_k \pi_k p(x \mid k)\) rather than \(F\). The two differ by \(D_{\mathrm{KL}}(\delta_{z^*} \,\|\, p(z \mid x))\), which is close to zero once pattern and noise land in separate templates.
Two consequences show up in the live chart. Every template starts at \(\theta = 0.5\), so the first few patterns cost a full 1 bit per bin and are treated as noise. Early learning is slowed until the culture discovers the pattern is a pattern. Templates are clipped to \([0.02, 0.98]\), which sets a floor of \(-\log_2 0.98 \approx 0.029\) bits per bin. That floor is the 0.03 you see.
3. How one connection learns
Before the whole dish, zoom in on a single connection. Neuron A sends a signal to an output neuron. If A fires just before the output fires, A probably helped make it fire. That's worth remembering, so the connection gets a small green tag. If the output fires first and A fires after, A couldn't have caused it, so the connection gets a red tag instead. This timing rule is called STDP (spike-timing-dependent plasticity). In short: "fire together, wire together", but order matters.
A tag is not learning yet. It's a sticky note that says "I was involved just now", and it fades within about a second. Learning only happens when feedback arrives while the tag is still there. The drumbeat turns a green tag into a stronger connection. Static shuffles tagged connections back toward average.
The important part is that each connection only uses what it can "see" at its own location: when its own input fired, when the output fired, and the feedback that reaches every connection. Nobody tells it whether it was right.
3. One synapse at a time: STDP
Everything the culture learns happens at individual synapses, and each one only uses information available at its own location. The first ingredient is spike-timing-dependent plasticity (STDP), first measured in cultured hippocampal neurons by Bi and Poo. In plain English: if the input fires just before the output, tag the synapse positively. If it fires just after, tag it negatively. The closer in time, the bigger the tag. With \(\Delta t = t_{\text{post}} - t_{\text{pre}}\):
$$\Delta e = \begin{cases} +A_+\, e^{-\Delta t/\tau} & \Delta t > 0 \quad (\text{pre before post}) \\[2pt] -A_-\, e^{\,\Delta t/\tau} & \Delta t < 0 \quad (\text{post before pre}) \end{cases} \qquad \tau = 20\text{ ms},\; A_+ = 1,\; A_- = 0.5.$$
A synapse doesn't need to store spike times to do this. It keeps two decaying traces: \(x\), which jumps by 1 at each presynaptic spike, and \(y\), which jumps by 1 at each postsynaptic spike. Both decay with \(\tau = 20\) ms. When the output fires, the tag gains \(x\). When the input fires, the tag loses \(A_- y\). Summed over spikes, that is exactly the window above.
The result \(e\) is an eligibility trace, not a weight change. It decays over about half a second (by a factor of 0.9 per 50 ms tick) and only becomes a weight change if a third signal arrives in time: the feedback surprise \(u\), broadcast to every synapse. In the brain, neuromodulators such as dopamine are thought to play this role for reward-driven learning. Here the third factor carries predictability rather than reward. The figure below is one output neuron with two inputs. A fires at a timing you choose. B fires at random. Watch which synapse gets tagged, then deliver feedback.
Two things to notice. First, a weight never moves on its own: STDP only writes tags, and a weight changes when feedback arrives. Turn on auto-feedback to watch the weights evolve. Second, the demo starts with weakening a bit stronger than strengthening (\(A_- = 1.5\)), the textbook competitive regime (Song, Miller & Abbott, 2000). An uncorrelated input like B lands on both sides of the window at random, so its tag averages out negative and it fades away, while A, which reliably fires first, keeps growing. Inputs that predict the output win. Inputs that don't, lose.
The culture itself can't use that setting. Click Culture (0.5): with strengthening dominant, B's chance coincidences net out positive and it grows alongside A. I tested the competitive window in the full game (8 seeds, 20 minutes, pattern + noise). With \(A_- = 1.2\) the culture reached 29 kills per minute, and with \(A_- = 1.5\) it stayed at chance (5), against 109 with \(A_- = 0.5\). The reason is rate coding. A sensory electrode drives its neurons at 60–160 Hz for the whole tick, so every motor spike is followed by more input spikes, and "output first" pairings are about as common as "input first". A weakening-dominated window then cancels the useful tag along with the noise. The culture filters out uncorrelated input in two other ways instead: it ignores tags below 20% of the strongest one, and each sensory neuron's outgoing weights share a fixed budget, so strengthening one connection takes from the others.
Graduate notes
The trace implementation is all-to-all pairwise STDP: \(e = \sum_{\text{post}} \sum_{\text{pre}} W(t_{\text{post}} - t_{\text{pre}})\) with \(W\) the window above. Its integral is \(\int W = \tau (A_+ - A_-) = 10\) ms, which is positive. So even uncorrelated Poisson firing builds a net positive tag at rate \(\nu_{\text{pre}}\nu_{\text{post}}\,\tau(A_+ - A_-)\). In the demo, B fires at 20 Hz against an output near 6.7 Hz, which gives \(20 \times 6.7 \times 0.01 \approx 1.3\) tag units per second, or a steady \(\approx 0.64\) after the 475 ms decay. Measured over 30 runs it is 0.66, against about 2.0 for A at \(\Delta t = +10\) ms. With \(A_- = 1.5\) the integral is \(-10\) ms and B's tag measures \(-0.54\). Over 20 simulated seconds of auto-feedback (10 runs each), B ends at 2.72 with \(A_- = 0.5\), 1.25 with \(A_- = 1\), and 0.21 with \(A_- = 1.5\), while A saturates at the cap of 3 in every case.
Why competitive STDP fails for the culture: for an input firing at rate \(\nu_{\text{pre}}\) whose spikes raise the output's rate by a causal kernel \(c(\Delta t)\) over a baseline \(\nu_{\text{post}}\), the expected tag rate is \(\nu_{\text{pre}}\nu_{\text{post}} \int W + \nu_{\text{pre}} \int W(\Delta t)\,c(\Delta t)\,d\Delta t\). The first term is the chance term. The second is the causal one. With sustained high-rate input, the chance term dominates both, so its sign (that of \(\int W\)) decides everything: positive, and every active pathway is strengthened; negative, and the useful pathway is erased too. In the demo the output fires only when driven by A, so the causal term wins regardless. The culture therefore keeps \(\int W > 0\) and removes the chance term downstream, with the eligibility threshold and the synaptic budget.
Gating STDP traces by a delayed global signal is the standard answer to the distal-reward problem (Izhikevich, 2007). The family is reviewed as neuromodulated STDP and three-factor learning rules (Frémaux & Gerstner, 2016). The difference here is only the third factor: \(1-u\) and \(u\), derived from the predictability of the feedback, instead of a reward prediction error.
4. The drumbeat says "more of that", the static says "forget that"
Every connection between two neurons has a strength, like the volume knob on a speaker. Strong connections make the next cell more likely to fire. When the cells pick a move, the connections that carried that decision are briefly "tagged".
- Drumbeat after a move: the tagged connections get stronger. Next time, the cells are more likely to do the same thing. They get less undecided.
- Static after a move: the tagged connections get shuffled back toward average. The cells don't learn the right move. They just forget the wrong one and try again at random. They get more undecided.
Repeat this thousands of times and every situation ends up with the move that brings the drumbeat. Below is one situation on its own: the imp is dead ahead, so only Fire works.
4. How pattern and noise move \(H(a \mid s)\)
Now connect surprise to behavior. While the game runs, every synapse keeps an STDP eligibility trace \(e_{ij}\): a presynaptic spike just before a postsynaptic spike adds to it, and the reverse subtracts. The trace decays by a factor of 0.9 per tick, so it remembers roughly the last half second of what the culture did. In other words, it tags the connections that just carried a decision. When a feedback burst ends, the culture scores its surprise \(u\) and applies one rule. In plain English: tagged connections get stronger if the feedback was predictable, and get shuffled back toward average if it was random. In symbols:
$$\Delta w_{ij} = \underbrace{\eta\,(1-u)\,\hat e_{ij}}_{\text{consolidate}} \;+\; \underbrace{u\,d\,|\hat e_{ij}|\left(\kappa\,(\bar w_i - w_{ij}) + \sigma w_0\,\xi_{ij}\right)}_{\text{scramble}}, \qquad \xi_{ij}\sim\mathcal N(0,1)$$
after which each sensory neuron's outgoing weights are rescaled back to their budget. Here \(\hat e\) is the trace normalized to its largest value with weak entries zeroed, \(\bar w_i\) is the mean of neuron \(i\)'s outgoing weights, and \(d\) is the noise duration relative to a hit burst. \(\xi_{ij}\) is a fresh random number from a standard bell curve, which is the "shuffle". \(\eta\), \(\kappa\), \(\sigma\) set how big each effect is. There is no reward variable. The same stimulation-driven gate \(u\) decides which way plasticity goes.
Predictable input (\(u \approx 0\)) sharpens. The synapses that just carried the decision get stronger. The winner-take-all readout acts roughly like a softmax over the region-to-region weights, \(p(a \mid s) \propto e^{\beta \bar W_{sa}}\), so adding \(\delta\) to the taken action multiplies its probability by \(e^{\beta\delta}\) before renormalizing. Repeat that and the policy walks into the corner of the action that keeps producing the pattern, and \(H(a \mid s)\) falls.
Unpredictable input (\(u \approx 1\)) flattens. The pull toward \(\bar w_i\) moves the recently used weights back toward uniform. In policy terms that is close to mixing with the uniform distribution \(U\), \(p' \approx (1-\kappa')\,p + \kappa' U\). Intuitively, blending anything with "completely random" can only make it more random. The math name for that property is concavity of entropy:
$$H\big((1-\kappa')\,p + \kappa' U\big) \;\ge\; (1-\kappa')\,H(p) + \kappa' \log_2 3 \;\ge\; H(p).$$
Noise can only push a state toward maximum entropy (plus whatever the jitter adds). It does not say which action would have been right. It erases the commitment and lets spontaneous activity sample again.
Put the two in a loop and each state ends up sharpened toward the actions that lead to predictable input, while the actions that lead to noise keep getting dissolved and resampled. Below is a single state where the imp is dead ahead, so only Fire produces the pattern. It is the same rule on just three weights.
Graduate notes
When does consolidation lower entropy? With softmax logits, raising the taken action's logit by \(\delta\) gives
$$\frac{\partial H}{\partial \delta} = -\beta\, p_a \left(\ln p_a + H\right) \qquad (H \text{ in nats}),$$
so entropy falls exactly when \(-\ln p_a < H\), that is, when the reinforced action is less surprising than average. Reinforcing a rare action first raises entropy (the point moves toward the center of the simplex) before lowering it. Pattern feedback is not monotone in \(H\). Noise is: the concavity bound above holds for any mixing toward \(U\).
It is a policy-gradient rule with a twist. In a winner-take-all network, the STDP trace on the winning pathway approximates \(\nabla_W \ln \pi(a \mid s)\). The consolidation term is then REINFORCE, \(\Delta W \propto R\,\nabla \ln \pi\), with \(R = 1 - u \ge 0\) and no baseline. That explains the pattern-only failure: with non-negative returns and no baseline, every action that is sampled gets pushed up, and the variance that produces drives rich-get-richer lock-in. The noise term is not a negative reward. It is a state-local contraction toward the uniform policy, which is the direction of the gradient of the entropy bonus used in maximum-entropy RL. The whole rule reads as REINFORCE with \(R = 1-u\), plus entropy regularization whose strength is gated by \(u\). Note that eligibility traces with per-tick decay \(\lambda = 0.9\) play the role of TD(\(\lambda\)) credit assignment for the turns that precede a kill.
A crude threshold. Suppose the culture has locked onto a wrong action \(a\) with weight advantage \(\Delta = w_a - \bar w\). Each noise burst after \(a\) adds \(\eta(1-u)\) and removes \(u\,\kappa\,d\,\Delta\). The lock-in is stable while
$$\eta(1-u) > u\,\kappa\,d\,\Delta \quad\Longleftrightarrow\quad u < u^* = \frac{\eta}{\eta + \kappa\,d\,\Delta}.$$
In the toy (\(\eta = \kappa = 0.3\), \(d = 1\), saturated \(\Delta = 2\)) this gives \(u^* \approx 0.33\). The empirical transition is 0.4–0.6, because the wrong action rarely saturates before Fire wins the race. In the game, \(d < 1\) for misses and the trace spans many ticks, which pushes \(u^*\) up toward the observed ~0.8.
Here's the surprising part. If the "static" isn't really random, for example a buzz that's mostly quiet with an occasional blip, the cells can partly predict it. And something you can predict counts as a little bit of drumbeat. So a predictable punishment actually strengthens the mistake a bit. We tested this in the full game below. Make the static less random, and learning doesn't just slow down. It flips. The cells confidently learn the wrong moves.
That last point is the clearest prediction the model makes, so I tested it in the full game. Holding everything else fixed and lowering only the density of the noise bursts, learning stops below roughly 0.8 bits per bin of noise entropy. The threshold in the full game is higher than in the one-state toy because a hit's eligibility trace spans many ticks of mistakes, and partial consolidation of all of them adds up.
5. Keeping score with tally marks
Here's another way to picture what the cells are doing. Imagine each situation has a scoreboard with three columns: Left, Right, Fire. Everyone starts with one tally in each column, so no move is favored.
- Drumbeat after a move: add a tally to that move's column. Your best guess shifts toward it.
- Static: erase a fraction of the tallies you've collected, back toward the starting one-each. Your guess moves back to undecided, and because you now have fewer tallies, the next new tally changes your mind more.
That second point is the clever part. Static doesn't just punish. It makes the cells more open-minded in that situation, so they're ready to learn something new. Statisticians call this way of updating beliefs with evidence Bayesian inference.
5. The Bayesian view
There is a cleaner way to say all of this: keep a tally for each action. Formally, treat each state's policy as an unknown probability distribution over the three actions (a categorical distribution) and describe the culture's belief about it with a Dirichlet distribution. That is the standard way to hold a belief about probabilities, and its parameters \(\alpha\) behave exactly like tally counts. The culture's best guess, its predictive policy, is each count divided by the total:
$$p(a \mid s) \sim \mathrm{Dir}(\alpha_s), \qquad \hat p(a \mid s) = \frac{\alpha_{s,a}}{\sum_{a'} \alpha_{s,a'}}.$$
The prior \(\alpha_0 = (1,1,1)\) is flat over the whole simplex, and its predictive distribution is uniform: the max-entropy culture again. A pattern after action \(a\) is evidence that \(a\) keeps the world predictable, so it is a count: \(\alpha_{s,a} \leftarrow \alpha_{s,a} + 1\). Noise is not evidence for anything. It is the world telling you your model of this state is stale. The Bayesian move for that is forgetting, which shrinks the counts back toward the prior:
$$\alpha_s \leftarrow \alpha_0 + (1-\kappa)\,(\alpha_s - \alpha_0).$$
That is a tempered posterior: the accumulated likelihood raised to the power \(1-\kappa\). It does two things at once. It moves the predictive mean toward uniform, so entropy goes up. And it lowers the total count \(\sum \alpha\), so the next observation moves the belief further. Noise does not just undo a habit. It reopens learning in that state, which is exploration without anyone asking for it.
Graduate notes
Scaling the accumulated counts by \(1-\kappa\) is the same as raising the likelihood to the power \(1-\kappa\): a power, or tempered, posterior. Applied once per noise burst, it is exponential forgetting. This is the standard Bayesian answer to a non-stationary world, and it mirrors the volatility effect in human learning: in changeable environments the learning rate goes up (Behrens et al., 2007). Read that way, noise feedback is evidence that the state's action–outcome mapping is volatile, and the culture responds by raising its effective learning rate \(1/(\sum\alpha + 1)\) there.
The synaptic analogue is close but not exact. The budget \(\sum_j w_{ij} = N_M w_0\) is a normalization of the predictive mean, with no free parameter for \(\sum \alpha\). In the culture, the role of confidence is played by the gap between weights relative to \(1/\beta\). A proper hierarchical version would make the forgetting rate itself a learned quantity.
One nice consequence: if the cells never get a drumbeat or static in some situation, its scoreboard never changes and it stays undecided forever. In one of our 8 test runs, the "far left" row stayed completely random after everything else was learned. No information means no learning.
6. Did it really work?
We ran 8 separate cultures for each kind of feedback, 20 minutes of game time each.
The culture is not literally a Dirichlet model. Its synaptic budget fixes the total weight, so it cannot store \(\sum \alpha\) as a separate number. Its confidence lives in how far the weights sit from uniform relative to the readout's sharpness \(\beta\). The directions match, though: pattern behaves like an observation and noise behaves like forgetting toward the prior. One prediction carries over directly. A state that never gets informative feedback stays at the prior. Occasionally one of the far states (L3 or R3) stays near 1.58 bits long after the others have converged. It happened in 1 of the 8 seeds in Figure 7. That state's trace is usually too old to be consolidated by the time a kill arrives, and random turning there is cheap enough that it rarely ends in a hit. No data, no update.
Active inference makes the loop explicit by choosing actions with probability \(p(a \mid s) \propto \exp(-\gamma\,G(s,a))\), where \(G\) is the expected free energy of what follows: roughly, how surprising you expect the world to be after taking that action. Low \(G\) actions get picked more. In this game the only thing that differs between outcomes is how predictable the feedback is, so roughly
$$G(s,a) \approx P(\text{noise} \mid s,a)\,H_{\text{noise}} + P(\text{pattern} \mid s,a)\,H_{\text{pattern}}, \qquad H_{\text{pattern}} \approx 0.$$
The culture never computes \(G\). It gets to a similar fixed point by sharpening after low-surprise outcomes and diffusing after high-surprise ones. The equation also explains Figure 5 at a glance. As \(H_{\text{noise}}\) shrinks, \(G\) stops distinguishing good actions from bad ones, and the culture has nothing to avoid.
6. Results across seeds
All four feedback conditions, 8 cultures each (seeds 1–8), 20 minutes of play, with the parameters used on this page.
Graduate notes
In active inference, expected free energy splits into risk and ambiguity:
$$G(s,a) = \underbrace{D_{\mathrm{KL}}\big(q(o \mid s,a)\,\|\,\tilde p(o)\big)}_{\text{risk}} + \underbrace{\mathbb{E}_{q(s' \mid s,a)}\,H\big[p(o \mid s')\big]}_{\text{ambiguity}}.$$
The culture has no explicit preferences \(\tilde p(o)\). Nothing in its model says kills are good. So the operative term is ambiguity, the expected entropy of the outcome, which is the approximation for \(G\) used above. This also says what the culture lacks. With no epistemic (information-gain) term, its only source of exploration is forgetting under noise. That is enough here, because the noise is triggered exactly by the states that need re-exploring.
Methods. Culture: 7×10 sensory and 3×10 motor neurons; \(\Delta t = 5\) ms; leak 0.75; threshold 2.5; escape-noise temperature 0.6 with \(p_{\max} = 0.5\); lateral inhibition 0.5 per spike; STDP trace \(\tau = 20\) ms with LTD/LTP ratio 0.5; eligibility decay 0.9 per 50 ms tick with threshold 0.2; \(\eta = 0.08\), \(\kappa = 0.35\), \(\sigma = 0.25\); \(w_0 = 0.12\); sensory spontaneous rate 0.5 Hz; stimulation 60–160 Hz by proximity. Game: 2 imps, speed 0.14 units/tick, turn 12°/tick, crosshair ±8°. These values came from a random search over 48 configurations (4 seeds × 25 min each), scored on final kill rate, so treat them as one working point, not a robust optimum. Code is in src/culture.js and src/experiment.js. scripts/make-results.mjs regenerates every number on this page.
You need both. The drumbeat alone teaches "do more of whatever you were doing", even when that's wrong, so most cultures get stuck in bad habits. Static alone only makes the cells forget, so they stay random. Together, the drumbeat locks in what works and the static erases what doesn't.
The honest fine print
- This is a computer simulation, not real cells. Real dishes have hundreds of thousands of neurons. This one has 100.
- I had to tune the settings to make it work. Some of the rules here are my best guess at what real cells do, not proven facts.
- The cells never get points, but the scientist still decides which events get the drumbeat. So the "reward" hasn't vanished. It's hidden in that choice.
Words to know
- Neuron: a brain cell. It sends quick electrical blips called spikes.
- Electrode: a tiny metal contact that can zap cells or listen to them.
- Culture: a dish of living cells grown in a lab.
- Synapse / connection strength: how strongly one neuron can make another fire.
- Entropy: how hard something is to guess, measured in bits.
- Surprise: how unexpected one particular event was.
- Policy: the chance of each move in each situation, or "what the cells tend to do".
- Probability: how likely something is, from 0 (never) to 1 (always).
- Bit: the unit for uncertainty. One bit is the uncertainty of a fair coin flip, or one yes/no question you'd need to ask.
- Density (of static): what fraction of the tiny time slots get a zap. 50% is the most random. Near 0% or 100% is predictable.
- Prior: what you believe before seeing any evidence. Here, "every move is equally good".
- Bayesian inference: updating your beliefs as evidence comes in.
Each condition fails for a different reason. Pattern-only feedback makes whatever the culture happened to be doing when kills arrived stronger everywhere, and since hits are silent nothing ever unlearns it. Noise-only feedback can only push toward uniform, and with nothing sharpening, the culture stays a max-entropy model. Both together give the full Bayesian loop: evidence where the world is predictable, forgetting where it is not.
What this is and isn't
- It is a toy. 100 neurons instead of hundreds of thousands, a place code I chose, and parameters I tuned with a random search. Two choices mattered a lot. Lateral inhibition between motor regions concentrates the eligibility trace on the chosen action. Low spontaneous firing in sensory regions stops a habit learned in one state from leaking into the others.
- The surprise gate is my modeling choice. A real culture has no explicit \(u\). The free energy account claims something with this effect emerges from ordinary plasticity. A simpler reading of the same data is that noise disrupts recently potentiated synapses while structured stimulation does not. This model is close to that reading, with the gate made explicit so the entropy story can be measured.
- "No reward" has a caveat. The culture never gets a reward value, but the experimenter decides which events get which stimulation. The reward is built into that mapping, not removed. What the experiment shows is that predictability alone is enough of a currency to carry it.
References
- Kagan, B. J. et al. (2022). In vitro neurons learn and exhibit sentience when embodied in a simulated game-world. Neuron, 110(23).
- Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11(2).
- Isomura, T. & Friston, K. (2018). In vitro neural networks minimise variational free energy. Scientific Reports, 8.
- Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical Review, 106(4).
- Bi, G. & Poo, M. (1998). Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. Journal of Neuroscience, 18(24).
- Izhikevich, E. M. (2007). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cerebral Cortex, 17(10).
- Frémaux, N. & Gerstner, W. (2016). Neuromodulated spike-timing-dependent plasticity, and theory of three-factor learning rules. Frontiers in Neural Circuits, 9.
- Song, S., Miller, K. D. & Abbott, L. F. (2000). Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nature Neuroscience, 3(9).
- Kempka, M. et al. (2016). ViZDoom: A Doom-based AI research platform for visual reinforcement learning. IEEE CIG.