📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva LLM, trained from scratch on extensive Italian data, outperforms multilingual models but scores near chance on Italian academic tests. This raises questions about the scale of native-language investment needed for true language understanding.
Italy’s Minerva project, a large-scale European sovereign language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, despite its extensive training and open data approach.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research and supercomputing infrastructure, built models ranging from 350 million to 7 billion parameters. It trained on a dataset of 2.5 trillion tokens, roughly half Italian, and openly published its weights and data, setting a technical benchmark for European sovereignty efforts.
However, when evaluated on the INVALSI Italian school exams, Minerva-3B scored just 4.9%, a result considered near chance for the test format. Researchers concluded that, despite the large dataset and native-language focus, the overall scale of parameters and dataset size remains a key factor in handling complex language tasks, including academic content.
This empirical finding underscores a core challenge: substantial native-language investment and data alone may not suffice to produce models with deep country-specific knowledge or academic competence at current parameter scales.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
EarlySincere 4K AI Smart Glasses with Photochromic Lenses, Integrated ChatGPT AI Model, Real-Time Translation, HD Video Recording, Long Battery Life, Bluetooth 5.4 Wireless
👓【𝟒𝐊 𝐕𝐈𝐃𝐄𝐎 𝐑𝐄𝐂𝐎𝐑𝐃𝐈𝐍𝐆 𝐆𝐋𝐀𝐒𝐒𝐄𝐒 𝐖𝐈𝐓𝐇 𝐄𝐈𝐒】 This advanced wearable camera empowers you to capture the wonders of the…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
From Weights to Wisdom: The Complete Guide to Running and Adapting Opensource AI Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AI training data storage solutions
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-LLM Strategies
The results from Minerva suggest that simply training large models on native-language data may not achieve the desired depth of country-specific knowledge or academic performance. This challenges assumptions that scale alone guarantees language understanding, emphasizing the need for strategic investment in model size and training data.
For policymakers and AI strategists across Europe, this highlights the importance of realistic expectations regarding native-language AI capabilities and the potential need for larger investments or alternative architectures to meet national AI goals.
Background on European Sovereign LLM Development
European countries have debated the best approaches to developing sovereign language models, with some, like Portugal, adopting continuation pre-training on smaller datasets (e.g., the AMÁLIA project), and others, like Italy, opting for training from scratch on massive datasets, as exemplified by Minerva.
The Italian approach involved extensive institutional support, including funding from Italy’s national AI strategy, use of the CINECA supercomputer, and open publishing of models and data. Despite technical successes, empirical evaluation reveals significant challenges in achieving complex language understanding at current model scales, raising questions about the efficacy of scale versus data quality.
“While our models outperform multilingual counterparts in some benchmarks, the near-chance results on INVALSI tests reveal limitations in capturing complex academic content.”
— Research team behind Minerva
Unresolved Questions About Model Scaling and Performance
It remains unclear what specific model size, training data composition, or architectural modifications are necessary to significantly improve performance on complex, country-specific tasks. The ongoing research aims to refine these factors, but definitive thresholds or strategies have yet to be established.
Next Steps for European Native-Language AI Development
The Minerva team plans to continue iterative training and evaluation, exploring larger models, different data mixes, and architectural adjustments. Policymakers and researchers will need to reassess native-language investment levels and strategies, potentially scaling up resources to meet the empirical challenges identified.
Key Questions
Why did Minerva score so low on the Italian school exams?
The evaluation suggests that, despite large-scale native-language training, the current model scale and dataset size are insufficient for mastering complex academic content, highlighting the importance of model size and training strategies.
Does this mean training from scratch is ineffective?
Not necessarily; it indicates that scale and dataset size are critical factors. Further research is needed to determine the optimal balance between training from scratch and other approaches.
What does this imply for other European countries developing sovereign LLMs?
It suggests that significant investment in model size and native-language data is essential, and that expectations should be calibrated regarding the achievable depth of knowledge at current scales.
Will larger models improve performance on academic benchmarks?
Likely, but empirical evidence is still emerging. The ongoing research aims to identify the scale and training strategies needed to reach desired performance levels.
Source: ThorstenMeyerAI.com
