Back to Blog

Small Tricks for Small Models

·Geoff·13 min read
devlogaillm

As with the last post, I started this one in September and only now got around to releasing it. The tests below were run in September 2026, and the numbers are from those runs.

In the last post I explained why I keep the rules of NextOath in plain code and only let the model grade plans and tell the story. That solves one problem. It doesn't solve the next one: getting a small model to reliably give the game the answers it needs.

In my testing, the bigger hosted models mostly behaved. Small ones — the cheap, fast ones you can run on your own machine — have bad habits, and I designed around them. The patterns below are the ones I've kept.

For each one: the problem, the target, what I changed, and how close that got.

How I tested. NextOath's prompts were written and tested on small local models, ending on Gemma 4 E4B. Production runs on hosted services, but the patterns were tested and shaped on the small ones, so that's where they're measured. Each test calls the model the way the game does — same prompt structure, JSON mode, same temperatures — on three models:

  • Gemma 4 E2B, the smallest,
  • Llama 3.1 8B, an older, widely used 8-billion-parameter model,
  • Gemma 4 E4B, the largest, and the one I ended on.

They don't fail the same way. Some patterns help all three, some only the smaller ones, and I'll say which. Every target is worked out from probability, not measured. The test code and every raw reply are in the companion repo.

1. Deal from a deck, don't ask for variety

The problem. Every NextOath game starts with the players building a world together by answering questions — how does this world handle trade, what do people worship, who's allowed to speak at council. The model writes those questions, and I want each one to explore a different part of the world.

Left to itself, it doesn't. It circles the same few ideas. Showing it the earlier questions doesn't fix that — a small model seems to read the history as "this is what we talk about here," not "pick something new."

The target. I played 20 short games of two questions each: 40 questions. There are 20 possible topics. If every question landed on a topic picked completely at random, you'd see about 17 different topics across the 40. And two random picks in the same game would share a topic only about 1 time in 20.

Before. With the prompt I used to use, which let the model choose the subject, all three models covered only 8 or 9 of the 20 topics. Gemma 4 E2B asked two questions about the same topic in more than half its games.

What I changed. Each game gets its own shuffled deck of topics, and each question is dealt the next card. The model still writes the question; it just doesn't pick the subject. I also swapped the old format example — which showed a real question about "resource scarcity" the model loved to echo — for one that describes the answer instead (more on that in the next pattern).

const deck = shuffle(TOPICS, seedFrom(gameId)); // same game, same order
const topic = deck[questionNumber % deck.length];
// prompt: "Write one question about ${topic} in this world."

How it did.

Dumbbell chart of distinct topics used out of 20. Asked for variety vs shuffled topic deck: Gemma 4 E2B 8 to 13, Llama 3.1 8B 8 to 16, Gemma 4 E4B 9 to 14. Random sampling would give about 17.4.
Dumbbell chart of games that repeated a topic, out of 20. Asked for variety vs shuffled topic deck: Gemma 4 E2B 11 to 2, Llama 3.1 8B 6 to 0, Gemma 4 E4B 5 to 1. Random sampling would give about 1.

Better on all three models: many more topics, and repeats in a game dropped to between 0 and 2 out of 20, close to the 1 you'd get picking at random.

It isn't perfect, and it's worth knowing why. The deck controls what you ask about, not what the model writes about. When I checked whether each question actually stayed on its dealt topic, it did 65–83% of the time depending on the model; the rest drifted to a neighboring topic. That's where the few remaining repeats come from.

One more honest note: I'd expected it to be faster, since there's less to decide. The prompt did get about 13% shorter, but the replies weren't quicker — almost all the time goes into writing the answer. The reason to deal from a deck is variety, not speed.

2. Describe the answer in your example; don't show one

The problem. Small models copy your format example. My old referee prompt showed the model the shape of its answer with real values in it — every one of its five yes/no criteria set to false:

{ "ontopic": false, "specific": false, "imaginative": false,
  "grounded": false, "memorable": false }

And the model would mark genuinely strong answers down, especially on the subjective criteria like "imaginative" and "memorable."

The target. I gave it 10 answers that clearly pass all five criteria. A good judge scores every one of them at least 4 out of 5, leaving room for one honest judgment call on a subjective criterion.

What I changed. Every slot in the example describes the answer instead of showing one, and the instructions ask for "yes" or "no":

{ "ontopic": "yes/no — does it try to answer?",
  "specific": "yes/no — names a person, place, or event?" }

How it did.

Dumbbell chart of strong answers scored 4 or more of 5, out of 10. Example shows false vs example describes the answer: Gemma 4 E2B 7 to 9, Llama 3.1 8B 7 both ways, Gemma 4 E4B 10 both ways. A fair judge scores all 10.

On Gemma 4 E2B this moved almost exactly the way I hoped: the answers it had marked down on "imaginative" and "memorable" went from 3 each to 1, though ten answers is too few to call that certain. The bigger Gemma didn't copy the example at all. And Llama 3.1 didn't move: one answer rose, one fell — a reminder that a fix for one model is a guess for the next. It didn't make the judge soft either: I also gave it 10 weak answers and 10 borderline ones. No weak answer reached 4 of 5 under either prompt, on any model, and the borderline ones stayed in the middle and didn't rise.

I still write every example this way, because it costs nothing on the models that don't need it and fixes the ones that do. But test it on yours.

3. Ask for words, not numbers

The problem. Sometimes the game needs an amount — how common is this kind of land, how strong is this storm. The obvious move is to ask for a number between 0 and 1. That's what I used to do. Here's why I stopped: small models pile their answers at the ends of the scale, or come back with "about half."

The target. Twenty things with an obvious answer, five at each of four levels — from rare (a dragon in a quiet farming village) through uncommon and common to everywhere (sand in a desert). A good answer puts all 20 at the right level.

What I changed. The model picks a word — rare, uncommon, common, everywhere — and code turns the word into a number, with a little randomness inside a range.

const RANGES = { rare: [0.01, 0.05], uncommon: [0.05, 0.15],
                 common: [0.15, 0.30], everywhere: [0.30, 0.45] };
const amount = pickInRange(RANGES[word], seed); // seed from the game id

How it did.

Dumbbell chart of things placed at the right level, out of 20. Asked for a number vs asked for a word: Gemma 4 E2B 10 to 17, Llama 3.1 8B 10 to 15, Gemma 4 E4B 13 both ways. A perfect answer is 20.

The two smaller models jumped by 7 and 5. Asked for numbers, Gemma 4 E2B never once landed in the middle two levels — everything was near 0 or near 1. Gemma 4 E4B held up either way, so for it this is insurance rather than a fix. I keep it anyway: it also means the arithmetic lives in my code, where I can see it and seed it for replays.

4. Don't throw out answers that are almost right

The problem. A small model often gives you an answer that's clearly right but not in exactly the shape you asked for. A strict validator rejects it, you retry the call, and you pay twice for an answer that was fine.

The target. Every reply with a usable answer in it gets used.

Before. I ran every yes/no and pick-one answer from these tests — 260 per model — through a strict check (the answer must be exactly one of the allowed words). On Gemma 4 E4B, it threw out 40. Every one was the same shape:

"specific": "yes — it names the 'Bell-Tongue,' the 'Ose,' and the 'tide-wardens.'"

It answered, and then kept going with its reasoning. The smaller models didn't do this at all. The bigger model was the chatty one.

What I changed. Read the first word, and accept what the model obviously meant.

// "yes — because…" -> yes ;  "Good." -> good ;  "goodness" -> no match
const word = String(value).trim().toLowerCase().match(/^[a-z]+/)?.[0];
return OPTIONS.includes(word) ? word : SAFE_DEFAULT;

Match the whole first word, not "starts with" — otherwise "goodness" quietly becomes "good."

How it did. The forgiving check kept every usable answer; on Gemma 4 E4B that meant recovering 40 of 260, about one in six answers a strict check would have thrown away.

Two things I'd add. Log every time you fall back to the default — a parser that can't fail can quietly return its default every time and look perfectly healthy. And if a field's absence means "nothing happens," drop what you don't recognize instead of rounding it to a real value. Being forgiving should never create something the model didn't ask for.

5. Don't pass one call's guess into the next

The problem. One of my prompts used to run in two steps. A first call checked a player's plan and guessed what kind of move it was. A second call judged the plan — and was told the first call's guess, "use this unless the plan clearly says otherwise."

What happened. The second model took the hint almost every time, including when the first one was wrong:

Stacked bar chart of what the second model did when the first model guessed wrong. Gemma 4 E2B: followed the guess 5 of 5. Llama 3.1 8B: followed it 3, fixed it 3, made another wrong read 2, of 8. Gemma 4 E4B: followed it 3, made another wrong read 1, of 4.

That chart flatters the effect, though. The wrong-guess plans are the ambiguous ones, and the judge alone misread most of them the same way. The fairer count is the same 30 plans judged with and without the guess: in the same model order as the chart, the judge moved toward the guess on 1, 1, and 3 plans, and away from it on 0, 1, and 0. Both Gemmas leaned toward the guess, Llama came out even, and 30 plans are too few to call it firm.

Still, it was enough for me to stop handing the second call the first call's answer. When one call's output becomes another's default, errors don't cancel out — they carry through.

What I changed. Each call now works from the player's own words. If two steps both need to decide something, let them decide it independently, and if they disagree, that disagreement is useful information, not something to paper over.

6. Temperature is a dial per model, not one number

A short one. Every prompt in NextOath has a temperature level — low for judging, higher for writing stories and questions — and each level maps to a number per model. Smaller models get a cooler range. Some model makers recommend a single value for everything; Google recommends 1.0 for Gemma 4, so that's what I use for it.

Does it make a difference? On Gemma 4 E4B, not one I could see. I ran the judge from pattern 5 twice, with the same prompts and the same call path; only the temperature changed. The old judge caught 6 of 10 "take something from an enemy" plans at 0.1 and 5 of 10 at the recommended 1.0. The fixed judge went the other way: 21 of 30 plans right when cold, 23 of 30 warm. With ten plans per kind, a one- or two-point gap is noise. So the reason to set temperature per model isn't that cooler is better. It's that the makers' recommendations differ, and a value that suits one model can be wrong for another. Check it on yours rather than assuming.

Bonus: pin the fix

This one isn't about the model at all, but it kept a lot of my LLM fixes from quietly coming undone.

The problem. You find a subtle bug and fix it with one line. Months later someone tidies up that function and the line disappears, because nothing about it looks important.

What I do. Add a lint rule that fails the build if the line is missing. I use ast-grep for this. A normal lint rule asks "is this code bad?" This one asks "is the good code still here?" The rule only proves the guard is there; you still want a test proving it works. These rules are cheap enough that I add one whenever I fix something I don't want to fix twice.

One pattern I cut

I also used to ask a few yes/no questions instead of "pick one of these three," to stop the model from ignoring one option. The problem is real: with the old "pick one" field, Gemma 4 E4B caught only 5 of 10 plans that took something from an enemy. But the yes/no version didn't fix it consistently. On the same test it lifted Gemma 4 E4B from 5 to 8, dropped Llama 3.1 from 9 to 7, and made Gemma 4 E2B catch none of them. So it isn't in this post as a fix. If you use pick-one fields, count how often each option comes back.

Try them yourself

The tests behind every number here are in llm-on-a-leash, along with every raw reply. Install Ollama, run the script, and it downloads the model it needs. Your model will probably slip up differently from mine, and that's sort of the point.

If you want the short version to keep open while you work, PATTERNS.md is the field guide: each pattern's symptom, fix, and how to check it worked on your model.

This is how I handled it, not the only way. If you've got something better, tell me and I'll add it alongside mine.

P.S. A pattern that one model doesn't need isn't wasted. Most of these cost nothing on the models that behave, and they're why the ones that don't behave still work.

Follow our changelog for updates, and don't hesitate to reach out via our support page with ideas, feedback or questions.