Microsoft Research released Turing Natural Language Generation (T-NLG), a 17-billion-parameter language model
In February 2020, Microsoft Research announced Turing Natural Language Generation (T-NLG), a 17-billion-parameter autoregressive language model trained using the Megatron-LM framework. At the time of release it was the largest publicly disclosed language model and achieved state-of-the-art results on question-answering and summarisation benchmarks.
Background
By late 2019, large language models, which learn to predict text by training on enormous amounts of written material, had been growing fast. OpenAI’s GPT-2, released in 2019, had 1.5 billion parameters, the individual numerical weights a model adjusts during training. Google had been pushing further with its own internal work. The race was becoming a matter of compute and scale as much as architectural ingenuity.
Training a model at that scale was not straightforward. The weights alone could exceed the memory of a single graphics processing unit, and simply adding more GPUs did not solve the problem without careful engineering. NVIDIA had published work on this in late 2019, describing Megatron-LM, a framework that split a model’s layers across multiple GPUs so that training could proceed in parallel without the whole thing collapsing into a communication bottleneck. That made very large transformer models, neural networks that read an entire sequence of text at once rather than one word at a time, practically trainable for the first time at scales beyond a few billion parameters.
Microsoft Research had been watching these developments closely. The company had the compute infrastructure and the research staff to attempt something at the outer edge of what the hardware could support.
What happened
In February 2020, Microsoft Research published details of Turing Natural Language Generation, known as T-NLG, a language model with 17 billion parameters built on the transformer architecture. Corby Rosset and Nebojsa Jojic of Microsoft Research were among the people who presented the work. The model was trained using NVIDIA’s Megatron-LM framework, which handled the problem of fitting the model across multiple GPUs through a technique called model parallelism, where different parts of the network live on different processors simultaneously.
At 17 billion parameters, T-NLG was the largest publicly disclosed language model at the time. On question-answering and summarisation benchmarks it achieved leading results, outperforming previous models on several standard tests. The model could generate free-form text, complete sentences, answer questions directly, and condense longer passages into shorter summaries.
Building it required close collaboration between Microsoft and NVIDIA, with Megatron-LM doing much of the heavy lifting on the infrastructure side. The result was less a single algorithmic breakthrough than a demonstration of what careful engineering at scale could produce. The gap between T-NLG and GPT-2 was not incremental; going from 1.5 billion to 17 billion parameters in roughly a year reflected how quickly institutions with serious hardware budgets were moving.
Why it mattered
T-NLG demonstrated that scaling transformer language models beyond 10 billion parameters yielded measurable gains on downstream natural language generation tasks, reinforcing the scaling hypothesis that was driving the field. It also signalled Microsoft's intent to compete directly with Google and OpenAI in large-scale language modelling, a strategic posture that would later produce the company's partnership with OpenAI and integration of large models into commercial products.
People
Corby Rosset Microsoft Research, Nebojsa Jojic Microsoft Research
Organisations
Sources
- Turing-NLG: A 17-billion-parameter language model by Microsoft.Microsoft Research.Official
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.arXiv / NVIDIA.Primary source
- Microsoft's monster language AI is almost as big as Google's.MIT Technology Review.Secondary
Cite this page
AI Achievements. (2020). Microsoft Research released Turing Natural Language Generation (T-NLG), a 17-billion-parameter language model. Retrieved 2026-08-22, from https://achievements.ai/milestone/turning-natural-language-generation-t-nlg
@misc{achievements_turning_natural_language_generation_t_nlg,
title = {Microsoft Research released Turing Natural Language Generation (T-NLG), a 17-billion-parameter language model},
author = {{AI Achievements}},
year = {2020},
url = {https://achievements.ai/milestone/turning-natural-language-generation-t-nlg}
}