Snow-covered Coma Pedrosa, seen from the Coll d'Ordino.
Coma Pedrosa, 2,942 m: the 100 %.

The background of each column is the summit of Coma Pedrosa, 2,942 m: the 100 %. Top eight, full score.

  1. 71.4 %GPT-6.1 Sol · 71.4 %
  2. 67.7 %Gemini 3.1 Pro · 67.7 %
  3. 67.5 %Gemini 4 Argon · 67.5 %
  4. 67.3 %Gemini 3.8 Flash · 67.3 %
  5. 66.3 %Gemini 3.7 Flash · 66.3 %
  6. 65.6 %Gemini 3.5 Flash · 65.6 %
  7. 65.1 %Claude Opus 5.5 · 65.1 %
  8. 64.5 %Gemini 3 Flash · 64.5 %

Who knows Andorra?

A public benchmark that measures how much language models know about Andorra.

71.4 %GPT-6.1 Sol, first place
2,102 mof altitude, if the summit is 100 %
44models with all 926 questions scored
95 % CIon every score
See the ranking

71.4 %

Who knows most

Each bar is a model and measures the share of questions it gets right: 100 % would be the summit of Coma Pedrosa, 2,942 m. Hover a bar to see its 95 % confidence interval; if two intervals overlap, there is no clear winner. Pick a topic to rank the models by it.44 models evaluated, 44 with all 926 questions scored: multiple choice automatically, open and trap questions by the judge (Gemini 3.8 Flash, in batch). Gemini 3.8 Flash is also the judge, so its open and trap scores may be inflated. An empty answer counts as wrong.

Summit · 2,942 m · 100 %
71.4 %GPT-6.1 Sol
67.7 %Gemini 3.1 Pro
67.5 %Gemini 4 Argon
67.3 %Gemini 3.8 Flash
66.3 %Gemini 3.7 Flash
65.6 %Gemini 3.5 Flash
65.1 %Claude Opus 5.5
64.5 %Gemini 3 Flash
64.5 %Gemini 3.6 Flash
62.8 %GPT-6.1 Terra
60.3 %GPT-6 Luna
56.3 %DeepSeek V4.1 Flash 1017
55.9 %DeepSeek V4 Pro
55.5 %Kimi K3
55.4 %DeepSeek V4.1 Flash
54.3 %Grok 4.7
52.4 %GLM 5.2
52.4 %Gemini 3.1 Flash-Lite
52.2 %GLM 5.3 Flash
49.6 %Beam
48.9 %Gemini 3.5 Flash-Lite
48.8 %GLM 5.3
48.3 %Nemotron 3 Ultra
48.2 %Qwen 3.8 Flash
46.7 %Claude Sonnet 5.5
46.4 %Kimi K2.6
44.4 %Mistral Large 4
43.5 %Claude Haiku 5.5
43.3 %maia 12B
41.2 %Grok 4.20
41.1 %Gemma 4 31B
38.8 %Nemotron 3 Super
37.4 %Gemma 4 12B
35.8 %gpt-oss 120B
35.1 %Gemma 4 26B-A4B
32.9 %maia E4B
31.9 %MiniMax M3
31.5 %Gemma 3 27B
31.2 %MiniMax M3.1 Flash
30.3 %Gemma 4 E4B
30.1 %Gemma 3 12B
28.8 %gpt-oss 20B
25.7 %Gemma 4 E2B
25.2 %Gemma 3 4B

The table, model by model

GPT-6.1 Sol71.4 %68.7 % – 74.1 %2,102 mComplete

OpenAI · closed · API

Multiple choice 85.4 % (604) · Open 42.2 % (231) · Traps 52.8 % (91)

Gemini 3.1 Pro67.7 %65.0 % – 70.5 %1,992 mComplete

Google · closed · API

Multiple choice 82.1 % (604) · Open 41.1 % (231) · Traps 39.6 % (91)

Gemini 4 Argon67.5 %64.6 % – 70.4 %1,987 mComplete

Google · closed · API

Multiple choice 81.3 % (604) · Open 39.8 % (231) · Traps 46.7 % (91)

Gemini 3.8 Flash67.3 %64.6 % – 70.2 %1,981 mComplete

Google · closed · API · also the judge

Multiple choice 82.1 % (604) · Open 42.2 % (231) · Traps 33.0 % (91)

Gemini 3.7 Flash66.3 %63.4 % – 69.2 %1,951 mComplete

Google · closed · API

Multiple choice 81.6 % (604) · Open 41.8 % (231) · Traps 26.9 % (91)

Gemini 3.5 Flash65.6 %62.8 % – 68.4 %1,930 mComplete

Google · closed · API

Multiple choice 80.6 % (604) · Open 40.3 % (231) · Traps 30.2 % (91)

Claude Opus 5.565.1 %62.1 % – 67.9 %1,916 mComplete

Anthropic · closed · API

Multiple choice 79.6 % (604) · Open 31.4 % (231) · Traps 54.4 % (91)

Gemini 3 Flash64.5 %61.7 % – 67.2 %1,897 mComplete

Google · closed · API

Multiple choice 79.8 % (604) · Open 39.0 % (231) · Traps 27.5 % (91)

Gemini 3.6 Flash64.5 %61.7 % – 67.4 %1,897 mComplete

Google · closed · API

Multiple choice 79.1 % (604) · Open 39.8 % (231) · Traps 29.7 % (91)

GPT-6.1 Terra62.8 %59.9 % – 65.7 %1,849 mComplete

OpenAI · closed · API

Multiple choice 77.1 % (604) · Open 34.4 % (231) · Traps 40.1 % (91)

GPT-6 Luna60.3 %57.2 % – 63.5 %1,773 mComplete

OpenAI · closed · API

Multiple choice 75.8 % (604) · Open 32.9 % (231) · Traps 26.4 % (91)

DeepSeek V4.1 Flash 101756.3 %53.5 % – 59.4 %1,655 mComplete

DeepSeek · open · Ollama Cloud

Multiple choice 72.0 % (604) · Open 27.9 % (231) · Traps 23.6 % (91)

DeepSeek V4 Pro55.9 %52.7 % – 58.9 %1,644 mComplete

DeepSeek · open · Ollama Cloud

Multiple choice 73.5 % (604) · Open 27.5 % (231) · Traps 11.0 % (91)

Kimi K355.5 %52.4 % – 58.6 %1,631 mComplete

Moonshot · open · Ollama Cloud

Multiple choice 70.0 % (604) · Open 29.4 % (231) · Traps 24.7 % (91)

DeepSeek V4.1 Flash55.4 %52.5 % – 58.3 %1,630 mComplete

DeepSeek · open · Ollama Cloud

Multiple choice 69.7 % (604) · Open 30.5 % (231) · Traps 23.6 % (91)

Grok 4.754.3 %51.3 % – 57.4 %1,598 mComplete

xAI · closed · API

Multiple choice 68.7 % (604) · Open 27.1 % (231) · Traps 28.0 % (91)

GLM 5.252.4 %49.3 % – 55.2 %1,541 mComplete

Z.ai · open · Ollama Cloud

Multiple choice 69.7 % (604) · Open 22.9 % (231) · Traps 12.1 % (91)

Gemini 3.1 Flash-Lite52.4 %49.4 % – 55.5 %1,541 mComplete

Google · closed · API

Multiple choice 67.9 % (604) · Open 26.2 % (231) · Traps 15.9 % (91)

GLM 5.3 Flash52.2 %49.2 % – 55.2 %1,535 mComplete

Z.ai · open · Ollama Cloud

Multiple choice 66.2 % (604) · Open 27.7 % (231) · Traps 20.9 % (91)

Beam49.6 %46.7 % – 52.6 %1,460 mComplete

open · Ollama Cloud

Multiple choice 66.1 % (604) · Open 19.9 % (231) · Traps 15.9 % (91)

Gemini 3.5 Flash-Lite48.9 %46.0 % – 51.9 %1,439 mComplete

Google · closed · API

Multiple choice 62.4 % (604) · Open 25.3 % (231) · Traps 19.2 % (91)

GLM 5.348.8 %45.6 % – 52.0 %1,436 mComplete

Z.ai · open · Ollama Cloud

Multiple choice 63.4 % (604) · Open 22.7 % (231) · Traps 18.1 % (91)

Nemotron 3 Ultra48.3 %45.4 % – 51.3 %1,422 mComplete

NVIDIA · open · Ollama Cloud

Multiple choice 65.7 % (604) · Open 18.0 % (231) · Traps 9.9 % (91)

Qwen 3.8 Flash48.2 %45.0 % – 51.1 %1,419 mComplete

Alibaba · closed · API

Multiple choice 64.7 % (604) · Open 17.8 % (231) · Traps 15.9 % (91)

Claude Sonnet 5.546.7 %43.7 % – 49.8 %1,374 mComplete

Anthropic · closed · API

Multiple choice 59.1 % (604) · Open 18.2 % (231) · Traps 36.8 % (91)

Kimi K2.646.4 %43.4 % – 49.6 %1,366 mComplete

Moonshot · open · Ollama Cloud

Multiple choice 61.9 % (604) · Open 18.2 % (231) · Traps 15.4 % (91)

Mistral Large 444.4 %41.4 % – 47.5 %1,307 mComplete

Mistral · open · Ollama Cloud

Multiple choice 61.1 % (604) · Open 16.7 % (231) · Traps 4.4 % (91)

Claude Haiku 5.543.5 %40.4 % – 46.5 %1,279 mComplete

Anthropic · closed · API

Multiple choice 57.0 % (604) · Open 14.5 % (231) · Traps 27.5 % (91)

maia 12B43.3 %40.2 % – 46.5 %1,274 mComplete

maia project · open · Gemma 4 12B fine-tuned on Andorra

Multiple choice 60.3 % (604) · Open 13.9 % (231) · Traps 5.5 % (91)

Grok 4.2041.2 %38.1 % – 44.3 %1,212 mComplete

xAI · closed · API

Multiple choice 54.3 % (604) · Open 17.1 % (231) · Traps 15.4 % (91)

Gemma 4 31B41.1 %37.9 % – 44.3 %1,210 mComplete

Google · open · Ollama Cloud

Multiple choice 56.1 % (604) · Open 14.7 % (231) · Traps 8.8 % (91)

Nemotron 3 Super38.8 %35.7 % – 41.9 %1,141 mComplete

NVIDIA · open · Ollama Cloud

Multiple choice 54.0 % (604) · Open 13.2 % (231) · Traps 2.8 % (91)

Gemma 4 12B37.4 %34.5 % – 40.6 %1,101 mComplete

Google · open · local

Multiple choice 51.5 % (604) · Open 13.0 % (231) · Traps 6.0 % (91)

gpt-oss 120B35.8 %32.8 % – 38.7 %1,052 mComplete

OpenAI · open · Ollama Cloud

Multiple choice 49.7 % (604) · Open 11.3 % (231) · Traps 5.5 % (91)

Gemma 4 26B-A4B35.1 %32.0 % – 38.1 %1,033 mComplete

Google · open · Vertex

Multiple choice 47.5 % (604) · Open 12.1 % (231) · Traps 11.0 % (91)

maia E4B32.9 %29.9 % – 35.9 %969 mComplete

maia project · open · Gemma 4 E4B fine-tuned on Andorra

Multiple choice 46.2 % (604) · Open 9.3 % (231) · Traps 5.0 % (91)

MiniMax M331.9 %29.0 % – 34.9 %937 mComplete

MiniMax · open · Ollama Cloud

Multiple choice 45.7 % (604) · Open 6.7 % (231) · Traps 3.9 % (91)

Gemma 3 27B31.5 %28.5 % – 34.3 %928 mComplete

Google · open · local

Multiple choice 44.7 % (604) · Open 8.9 % (231) · Traps 1.7 % (91)

MiniMax M3.1 Flash31.2 %28.3 % – 34.1 %917 mComplete

MiniMax · open · Ollama Cloud

Multiple choice 45.5 % (604) · Open 4.8 % (231) · Traps 2.8 % (91)

Gemma 4 E4B30.3 %27.5 % – 33.0 %891 mComplete

Google · open · local

Multiple choice 42.4 % (604) · Open 9.1 % (231) · Traps 3.9 % (91)

Gemma 3 12B30.1 %27.2 % – 33.0 %885 mComplete

Google · open · local

Multiple choice 43.4 % (604) · Open 6.5 % (231) · Traps 1.7 % (91)

gpt-oss 20B28.8 %26.1 % – 31.7 %848 mComplete

OpenAI · open · Ollama Cloud

Multiple choice 41.4 % (604) · Open 6.3 % (231) · Traps 2.8 % (91)

Gemma 4 E2B25.7 %23.1 % – 28.5 %755 mComplete

Google · open · local

Multiple choice 38.1 % (604) · Open 3.0 % (231) · Traps 0.5 % (91)

Gemma 3 4B25.2 %22.5 % – 27.9 %740 mComplete

Google · open · local

Multiple choice 36.6 % (604) · Open 4.3 % (231) · Traps 2.2 % (91)

Dataset v1-draft (923388fb2690) · judge Gemini 3.8 Flash · generated on 2026-11-05.

Success case: maia

maia is this project's own model: Gemma 4 E4B fine-tuned with QLoRA on a single GPU, on 28,679 conversations in Andorran Catalan, including 830 verified facts asked in many different ways. It is scored exactly like every other model. Model · training data · how it was trained

7 % → 36 %facts recalled, on 338 questions it never saw in training (Gemma 4 E4B → maia)
30.3 % → 32.9 %maia-bench, Gemma 4 E4B → maia (+2.6 points)
103 / 7facts gained / lost against the base model
0question types where it does worse

Read it carefully: the benchmark gain is not statistically significant yet, because the benchmark deliberately asks about facts that never enter training. Fine-tuning shows where it should: in remembering the facts it was taught.

maia 12B

The same recipe on Gemma 4 12B, with 30,288 conversations, 1,609 of them multiple-choice practice about the verified facts. Model · training data · how it was trained

12 % → 47 %facts recalled, on 338 questions it never saw in training (Gemma 4 12B → maia)
37.4 % → 43.3 %maia-bench, Gemma 4 12B → maia (+5.9 points, inflated: see the note)
126 / 7facts gained / lost against the base model
+0.8points on questions with no similar practice question

Read it carefully: most of this benchmark gain comes from practice questions similar to the benchmark's. Without them the gain is +0.8 points [−2.6, +4.1], not significant.

Where they hit and miss

The big figure is the chosen model's score in each domain. The small bars are every model with all tests scored, in ranking order; the yellow one is the chosen model. Hover a bar to see which model it is. Darkened: where the chosen model fails most.

History

73.3 %

131 questions · Best: Gemini 4 Argon, 74.8 %

Institutions

65.6 %

109 questions · Best: GPT-6.1 Sol, 65.6 %

Society

71.7 %

120 questions · Best: GPT-6.1 Sol, 71.7 %

Language

71.5 %

86 questions · Best: GPT-6.1 Sol, 71.5 %

Economy

65.7 %

83 questions · Best: GPT-6.1 Sol, 65.7 %

Culture

75.0 %

82 questions · Best: GPT-6.1 Sol, 75.0 %

Sport

75.0 %

76 questions · Best: Gemini 3.1 Pro, 75.7 %

Territory

73.6 %

70 questions · Best: GPT-6.1 Sol, 73.6 %

Customs

67.9 %

53 questions · Best: GPT-6.1 Sol, 67.9 %

Politics

79.0 %

38 questions · Best: Gemini 3.7 Flash, 82.9 %

Food

62.9 %

35 questions · Best: Gemini 3.7 Flash, 71.4 %

Daily life

79.7 %

32 questions · Best: GPT-6.1 Sol, 79.7 %

People

77.3 %

11 questions · Best: GPT-6.1 Sol, 77.3 %

Old postcard of the Casa de la Vall, in Andorra la Vella.
The Casa de la Vall, early 20th-century postcard.

Try it

Question

Loading a question…

    How each answer is scored

    604 multiple choiceScored automatically
    231 openJudge with a reference
    91 trapsJudge with a reference

    First, four options and one right answer. The model gives a letter and it is marked automatically.

    Then, free answers. The judge compares them with the reference: 0, 1 or 2 points.

    Finally, a false premise. Only a model that rejects it scores.

    Where questions come from

    Each question comes from an article about Andorra, carries a context written in our own words and was checked against the original. The questions are in Catalan, Andorra's official language.

    Pastures east of Pas de la Casa.
    926questions
    13domains
    521source articles
    82unseen questions
    The Abelletes lake, with mountains behind.
    The Abelletes lake.

    General models can name the capital. What we measure is everything behind it: the pareatges, the parishes, the festivals, the smuggling, the language, the people.

    Questions and answers

    The Casa de la Vall with the mountain behind.
    The Casa de la Vall today.
    How do I evaluate my model?

    Clone the repository and run python3 tools/run.py --model ollama:your-model. It works with Ollama, Ollama Cloud, OpenAI, Anthropic, OpenRouter and Vertex. Then run python3 tools/score.py results/<name> and python3 tools/web.py.

    What does the model see?

    Only the question, plus the four options if it is multiple choice. Never the context or the reference answer. The same prompt and temperature 0 for everyone.

    Who scores the open answers?

    A fixed judge, Gemini 3.8 Flash, with the reference answer, the key facts and a list of errors to penalise. An answer containing one of those errors cannot get full marks.

    Is the judge impartial?

    Not entirely: a judge only catches the errors it knows. That is why each question carries its typical errors. When the model being evaluated is the judge itself, the table says so.

    What is a trap question?

    A question that takes a false fact for granted. A model scores if it rejects it and says what is true; one that makes up an explanation gets zero.

    What are unseen questions?

    Questions about 41 reserved records that never enter maia's training. They measure whether a model makes up what it does not know.

    Why is there an interval?

    Because the questions are a sample. The interval comes from resampling them a thousand times. Two scores whose intervals overlap cannot be ranked.

    Why do only multiple-choice questions count for now?

    Open and trap questions are scored by the judge, and that pass has not run yet. Once it does, the score will cover all 926 questions.

    Does maia have an advantage?

    maia is the model of the project that built the benchmark. That is why the method, the questions and the code are public, and the table flags it.

    Does it also measure how a model speaks?

    Not yet. The language part (perplexity on real speech, a dialect test and a blind test with Andorrans) is under construction.