27 February 2023

LLaMA matches leading models on public data with far fewer parameters

LLaMA, a collection of foundation language models from 7B to 65B parameters trained exclusively on publicly available data, was submitted to arXiv on 27 February 2023. The 13B model outperformed GPT-3 at 175B parameters on most benchmarks, and the weights were released to the research community.

Language understanding Language models Transformers and attention Deep learning Capability threshold Independently validated

Background

Large language models, which learn to predict and generate text from enormous collections of data, had grown steadily more capable through the early 2020s. The leading examples, including GPT-3 at 175B parameters, Chinchilla-70B, and PaLM-540B, were trained on datasets that were either proprietary or not publicly accessible. That meant most researchers could not replicate the training, audit the data, or easily build on the results.

What happened

Hugo Touvron and thirteen colleagues submitted LLaMA to arXiv on 27 February 2023. The work introduced a collection of foundation language models ranging from 7B to 65B parameters, all trained on trillions of tokens drawn exclusively from publicly available datasets.

The headline result was striking. LLaMA-13B, despite having far fewer parameters than GPT-3, outperformed GPT-3 (175B) on most benchmarks. The largest model, LLaMA-65B, was competitive with Chinchilla-70B and PaLM-540B. That meant the gap between public and proprietary training data was smaller than the field had widely assumed, and that careful use of publicly available text could match models built on restricted sources.

The team released all models to the research community. That decision mattered as much as the results: researchers gained access to a set of competitive weights built entirely without proprietary data, lowering a significant barrier to replication and further research.

Why it mattered

LLaMA demonstrated that proprietary datasets were not necessary to reach competitive performance, with the 13B model outperforming GPT-3 despite having far fewer parameters. By releasing the weights, the work gave researchers access to a set of competitive foundation models trained entirely on publicly available data.

Sources

LLaMA: Open and Efficient Foundation Language Models. arxiv.org. Primary source
LLaMA: Open and Efficient Foundation Language Models. arxiv.org. Primary source

Cite this page

AI Achievements. (2023). LLaMA matches leading models on public data with far fewer parameters. Retrieved 2026-08-29, from https://achievements.ai/milestone/llama-matches-leading-models-on-public-data-with-far-fewer

@misc{achievements_llama_matches_leading_models_on_public_data_with_far_fewer,
  title  = {LLaMA matches leading models on public data with far fewer parameters},
  author = {{AI Achievements}},
  year   = {2023},
  url    = {https://achievements.ai/milestone/llama-matches-leading-models-on-public-data-with-far-fewer}
}