maia is Gemma 4 fine-tuned to know Andorra. It took six attempts. Teaching style was easy; teaching facts was not. The current models remember far more of what they were taught, but they do not yet know more about what they were not taught.
338 questions about facts in the training data, in wording it never saw. The answer must contain the expected keywords.
926 questions about Andorra, the same test as every other model. The best model in the ranking scores 71.4 %.
Do not read the 43.3 % as it stands: that gain is inflated. On questions with no similar practice question, the gain is +0.8 points, and that is not significant.
Each attempt changed one thing and was measured against the base model it started from.
1 · E4B, 17,885 conversations
Fluent Catalan with an Andorran tone, and more confident errors. Benchmark 30.3 → 31.0 %, no real change. Open questions fell from 9.1 to 5.6 %.
Lesson: seeing a fact once in a sentence does not make it retrievable.
2 · E4B, 18,738 conversations
284 answers that cited an invisible document were rewritten, and 943 basic questions were added. Benchmark 30.0 %. Open questions stayed at 5.8 %.
Lesson: 5 % more facts does not move the needle.
3 · E4B pilot, 50 facts
Each fact was asked 11 different ways, with answers worded differently each time. Recall on 50 new questions rose from 6 to 66 %. A new error appeared: it mixed up similar facts.
Lesson: repetition with variation works, and similar facts need contrast examples.
4 · E4B, 28,679 conversations
830 verified facts, 124 contrast groups. Recall rose from 7 to 36 %. Benchmark 30.3 → 32.9 %, +2.6 points with a 95 % interval of −0.6 to +5.9, so not significant.
Lesson: the taught facts stick, and the open questions no longer get worse.
5 · 12B, 28,679 conversations
Benchmark 35.0 % against 37.4 % for its base. It answered with a sentence instead of a letter in 67 of 604 multiple-choice questions, because the training data had no multiple-choice conversation at all.
Lesson: fine-tuning can erase a format the base model already knew.
6 · 12B, 30,288 conversations
1,609 multiple-choice practice questions were added. No answer without a letter now. Recall rose from 12 to 47 %. Benchmark 37.4 → 43.3 %, but most of that gain is not knowledge.
Lesson: the filter against similar questions was too loose.
Everything starts from a public corpus about Andorra, with a provenance record for every source. Benchmark questions and the articles the benchmark reserves are kept out of training.
18,738 conversations written from the corpus and basic questions. 9,941 from the canon: 830 atomic facts, each asked about 11 ways, plus contrast between facts that are easy to confuse. 1,609 multiple-choice questions on the same facts (the thin red block), answered with the letter only.
| Method | QLoRA, Unsloth |
|---|---|
| GPU | 1 NVIDIA L4, spot |
| LoRA r / alpha | 32 / 32 |
| Epochs | 2 |
| Learning rate | 2e-4 |
| Batch | E4B 4×4 · 12B 2×8 |
| Steps | E4B 3,586 · 12B 3,786 |
| Loss | answers only |
Before publishing, the data was swept for emails, phone numbers and identity documents, and none were found. The multiple-choice wrong options are invented, and some named full names found nowhere else in the corpus. Andorra has about 85,000 people, so such a name could match a real private person.
43 of those conversations were trained on but are not published, so the public file has 30,245 of the 30,288. The names were picked by hand and not checked against sources, so the filter is not proof that the data is clean.
One command starts everything on Google Cloud. Nothing runs on a laptop.
They are cheap, but Google can reclaim them at any time. In the first full run, the VM was reclaimed at step 831, and no L4 was free in that whole region. The run lost about 18 minutes of GPU and resumed from a checkpoint in another region.
An interruption in the middle of a build left corrupt files, and the quantization step failed at link time. It now rebuilds from clean. The watchdog only changes zone when a VM fails to start, so a zone that keeps interrupting a running VM still costs time.
The multiple-choice practice was written from the facts, without seeing the benchmark. But both ask about the same facts, so many practice questions look like benchmark questions. Only near-copies were dropped, with a word overlap of 0.5 or more. That limit was too loose.
Gain against Gemma 4 12B, in points, with its 95 % interval (paired bootstrap). The white mark is zero. An interval that crosses it is not significant. Similarity is the word overlap with the closest practice question.
Recall of taught facts is real, because those 338 questions are kept out of training. The format is fixed too: no answer without a letter, against 6 for the base model and 67 in the earlier attempt.
These models are experimental. They still make things up. Asked about the Querimònia, which appears nowhere in the training data, the 12B confidently describes something else. Trap questions did not improve: 6.0 → 5.5 %, within the noise. A cleaner figure needs a stricter filter and a new training run.
Models, data and code, with one tagged version per training run.