Q-learning

Now we replace the hand-written table with one that fills itself. Tabular Q-learning, shared across the whole colony - 16 states × 4 actions.

Reward

The only guidance we give the learning:

let r = -0.01;            // every step costs (efficiency)
if (ev.picked)    r += 1; // sugar picked up
if (ev.delivered) r += 10;// sugar delivered to the nest  ← the real goal

Action selection (ε-greedy) and learning rule

decide(s) {
  this.visits[s]++;
  if (Math.random() < this.eps)                 // explore
    return Math.floor(Math.random() * 4);
  return argmax(this.Q[s]);                      // exploit (greedy move)
}

update(s, a, r, sNext, alpha) {
  const maxNext = Math.max(...this.Q[sNext]);
  this.Q[s][a] += alpha * (r + this.gamma * maxNext - this.Q[s][a]);
  this.eps = Math.max(0.05, this.eps * 0.9996); // exploration decays over time
}

The update needs the previous decision. So each ant remembers prevS/prevA and the reward accumulated since its last move, rAccum. Per tick: first update(prevS, prevA, rAccum, s), then decide anew.

The money moment

Put the learned policy next to the hand-written one. For each experienced state: does argmax(Q[s]) match hardAction(s)?

const match = argmax(Q[s]) === hardAction(s);
// count only states visited often enough - the colony knows nothing about the unseen

In the lab the metrics panel shows a “Policy match” ratio like 4/5 ✓. The machine found your rules itself - purely from reward.

Honest note on convergence: We only compare the reward-decisive states - the ones where reward genuinely determines the action (carrying → ZUM_BAU, seeing sugar → ZUM_ZUCKER). In states where any navigation is roughly equally good (only “at nest”, only “trail”), the Q-values stay close together and argmax is essentially random. That is not a bug but a property of reward shaping: learning is reliable only where the reward distinguishes.

Agent lesson: Three things become tangible here.

1. That is the difference between algorithm and learning: same box, but in stage 1 prescribed, in stage 2 formed from experience.
2. The colony learns only about situations it experiences. Rare states stay empty - the bridge to “a model is only as good as its data”.
3. ε governs the explore/exploit dilemma: too little exploration → the colony freezes in a mediocre strategy.

→ In the lab: pick the “Q-learning” brain and watch “Policy match”