regenold GmbH has published the results of its 2026 EU AI Act Q&A Challenge (arXiv 2611.04546). Five specialist AI assistants and three off-the-shelf baselines answered 110 expert-curated questions about the EU AI Act. Ansvar placed second overall in both test modes and scored above every baseline. In single-turn mode our answers met every required point more often than any other system's, and our answers cited the most of the expected provisions at the sub-article level.
We are a two-person company, and we did not see the questions in advance. This post covers what the challenge tested, where we came out ahead, where we fell short, and what we are changing.
What regenold tested#
The organisers curated 110 open questions about Regulation (EU) 2024/1689, as it stood on 1 May 2026. Each question came with an exemplary answer, up to five correctness criteria and the Articles or Annexes a good answer should cite. Some are one-line definitions. Others are multi-actor scenarios, such as a private contractor that fine-tunes a licensed AI system to run a gas supply service for a local government.
Every system returned an answer and a list of references in a fixed format,
for example Article 50.4 or Annex IV.2. regenold scored eight things:
answer correctness (loose and strict), answer length, reference correctness
(loose and strict), unneeded references, tone and response time. The overall
score is the geometric mean of the eight, so one weak dimension pulls the
whole score down.
The test ran in two modes. Easy mode asked each question once. Hard mode placed the question after a nine-turn conversation, then told the system its answer might contain hallucinations and asked it to think again. The answer after that pushback was the one scored.
The baselines were GPT-5.5 with web search, GPT-5.5 with basic retrieval over the Act's own text, and a search-integrated assistant from a major US company. An LLM judge (GPT-5.4) graded the criteria three times per answer, and a human reviewer checked a sample of 50 judgements against it.
The results#
| System | Single-turn | Multi-turn |
|---|---|---|
| Pharos Labs | 91.65 | 91.50 |
| Ansvar | 86.64 | 86.29 |
| GPT-5.5 + retrieval | 84.23 | 85.08 |
| GPT-5.5 + web search | 82.64 | 83.30 |
| Antifragile AI | 75.83 | 73.94 |
| Search-integrated assistant | 70.69 | 75.75 |
| Anonymous company | 64.94 | 65.99 |
| Anonymous university lab | 61.40 | 60.38 |
Overall scores from Figures 3 and 4 of the paper. They combine eight metrics and are not the share of correct answers.
Only two of the five specialist entrants beat all three baselines in both modes: Pharos Labs and Ansvar. The paper's authors draw the lesson that building for a domain does not by itself produce better answers than a well-configured general model.
Where we came out ahead#
Two numbers matter most to us, because they measure whether an answer covers the required points and names the paragraph a reviewer can check.
Answers that met every criterion. In single-turn mode, 90.3% of our answers satisfied every correctness criterion for their question, the highest of all eight systems. In multi-turn mode we scored 87.0%, second to the retrieval baseline's 89.1% and ahead of Pharos Labs' 86.4%.
Expected provisions cited at the sub-article level. The strict reference score counts whether a system cites the operative paragraph, such as Article 5.1 rather than Article 5. We found 95.7% of the expected references in single-turn mode, the highest in the challenge, and 93.5% in multi-turn mode. The GPT-5.5 baselines found between 82.3% and 90.5%.
Citations are what Ansvar is built around. When an agent names the exact paragraph, a reviewer can open it and check the claim. The benchmark measures how many of the expected provisions a system found. It does not check that every cited provision supports the sentence it sits next to, and we will not claim more than it measured.
The multi-turn pushback moved us little. Our overall score fell by 0.35 points between the modes.
Where we fell short#
Pharos Labs won on focus and speed.
| Metric | Ansvar single | Pharos single | Ansvar multi | Pharos multi |
|---|---|---|---|---|
| Answer conciseness | 77.3 | 83.1 | 75.6 | 83.5 |
| Reference conciseness | 57.5 | 83.4 | 65.5 | 83.4 |
| Response speed | 88.1 | 94.4 | 82.6 | 95.2 |
Our answers cited more provisions than the expected set. Some of those extras add useful context, and the paper's authors note that the benchmark cannot tell a relevant extra from an irrelevant one. But a reviewer has to read every reference you give them, and a shorter list of the provisions that carry the answer is easier to check.
Reference conciseness is the largest single gap. If we had matched Pharos Labs on that one metric with everything else unchanged, our single-turn score would have been about 90.8 against their 91.65. That is arithmetic on the published numbers, not a forecast: trimming references can also cost coverage.
The paper also lists two of our misses. One answer described "limited risk" without saying the Act gives that term no legal definition. Another cited Article 5 where the expected reference was Article 5.1.
What we are changing#
- Separate the provisions that carry the answer from supporting ones. Lead with the paragraphs that support the conclusion, and add others only when they change what the reader should do.
- Prefer the operative paragraph. Cite Article 5.1, not Article 5, whenever the paragraph is known.
- Find where the seconds go. Our response-speed score trails the leader by six to thirteen points.
- Rerun the public benchmark as a regression test. regenold has released the questions and the scoring code on GitHub. We will keep a fresh set of unseen questions alongside it, so that we measure improvement, not memorisation.
What the result does and does not show#
The challenge tested one regulation, 110 questions, and the system we submitted between May and August 2026. Ansvar's product is the MCP gateway: you connect your own AI agent to it, and it serves the law, regulation and standards corpora with citations. We do not sell a question-answering assistant. For the challenge we built a dedicated EU AI Act agent on the same EU regulation corpus, because regenold needed an endpoint that returns an answer and a reference list. Entrants did not have to disclose their underlying models, so the paper cannot say how much of a score came from the model and how much from everything around it. The spread shown for each score reflects three judge runs, not a confidence interval for new questions.
The study also confirms a point we care about: a professional tone is not enough to show an answer is correct. Every system in the challenge, including the weakest, answered in a polished regulatory register, while their overall scores differed by about 30 points. If you are choosing an AI assistant for regulatory work, ask for results on questions the vendor did not write.
Thanks to Marco Virgolin, Thomas Haszprunar and Maximilian Regenold for running the challenge, writing the questions and publishing both the results and the code.
Key takeaways#
- Ansvar placed second of eight in regenold's 2026 EU AI Act Q&A Challenge, in both single-turn and multi-turn mode.
- We were one of two specialist entrants to beat all three off-the-shelf baselines in both modes.
- In single-turn mode we had the highest share of answers meeting every criterion (90.3%) and the highest strict reference coverage (95.7%).
- Most of our gap to first place is in reference focus, answer length and speed, and that is where we are working.