Language understanding
AI milestones in language understanding, part of ai research.
13 milestones
Mixtral 8x7B: sparse mixture-of-experts model released under Apache 2.0
Mixtral 8x7B, a sparse mixture-of-experts language model, was published in January 2024. It outperforms or matches Llama 2 70B and GPT-3.5 on most evaluated benchmarks while using only 13B active parameters per token, and both base and instruction-tuned versions were released under the Apache 2.0 licence.
LLaMA matches leading models on public data with far fewer parameters
LLaMA, a collection of foundation language models from 7B to 65B parameters trained exclusively on publicly available data, was submitted to arXiv on 27 February 2023. The 13B model outperformed GPT-3 at 175B parameters on most benchmarks, and the weights were released to the research community.
Self-Instruct: language models taught to follow instructions using synthetic self-generated data
Yizhong Wang and colleagues introduced Self-Instruct, a method for training language models to follow instructions using synthetic data the models generate themselves, closing much of the gap with InstructGPT-001 on evaluated tasks while using far less human annotation.
BLOOM: open-access 176B-parameter multilingual language model
The BigScience Workshop released BLOOM, a 176-billion-parameter language model trained across 46 natural and 13 programming languages, made freely available under the Responsible AI License at a scale that had only recently begun to become accessible, and had not previously been available with multilingual training data.
Flan-PaLM: instruction finetuning scaled across tasks, model sizes and families
Researchers showed that finetuning large language models on instruction-phrased datasets improves performance across benchmarks, with Flan-PaLM 540B trained on 1.8K tasks scoring 75.2% on five-shot MMLU and outperforming its base model by 9.4% on average.
InstructGPT: aligning language models with human feedback at scale
Researchers showed that fine-tuning GPT-3 with human feedback produced a 1.3B parameter model whose outputs labellers preferred over those of the 175B GPT-3, pointing toward a practical method for aligning language models more closely with expressed human preferences.
Instruction tuning lets a 137B model match or beat GPT-3 zero-shot
Jason Wei and colleagues showed that finetuning a 137B language model on over 60 NLP tasks described via natural language instruction templates produced a model, FLAN, that beat zero-shot GPT-3 on 20 of 25 tasks and surpassed few-shot GPT-3 on several benchmarks.
Microsoft Research released Turing Natural Language Generation (T-NLG), a 17-billion-parameter language model
In February 2020, Microsoft Research announced Turing Natural Language Generation (T-NLG), a 17-billion-parameter autoregressive language model trained using the Megatron-LM framework. At the time of release it was the largest publicly disclosed language model and achieved state-of-the-art results on question-answering and summarisation benchmarks.
BERT introduces masked bidirectional pre-training for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova introduced BERT, a language model that pre-trains on both left and right context simultaneously, achieving new best results on eleven natural language processing tasks.
Andrew M. Dai and Quoc V. Le Introduced Semi-Supervised Sequence Learning
In November 2015, Andrew M. Dai and Quoc V. Le at Google Brain published 'Semi-Supervised Sequence Learning', showing that pre-training recurrent neural networks with unsupervised objectives, language modelling or sequence autoencoding, before supervised fine-tuning improved text classification accuracy and training stability, anticipating the pre-train-then-fine-tune paradigm later adopted widely in NLP.
Bahdanau, Cho and Bengio introduce soft attention for neural machine translation
Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio proposed an attention mechanism letting neural translation models search source sentences dynamically, rather than compressing everything into a single fixed-length vector, achieving performance comparable to phrase-based systems on English-to-French translation.
Jabberwacky Chatbot Developed by Rollo Carpenter
In 1988, British programmer Rollo Carpenter began developing Jabberwacky, a chatbot that simulated conversation by storing and retrieving lines from prior user exchanges rather than using fixed scripted responses, with the aim of exploring machine-based natural language interaction.
HAM-ANS Natural Language Dialog System (University of Hamburg, 1983)
In 1983, Wolfgang Hoeppner, Katharina Morik, and Heinz Marburger at the University of Hamburg presented HAM-ANS, a natural language dialog system enabling users to query databases in ordinary German, demonstrated at the GWAI-83 workshop and published in Springer's Lecture Notes in Computer Science series.