Self-supervised learning
6 milestones used this technique.
DeepMind's AlphaFold 2 Achieves Highest-Accuracy Results at CASP14 Protein Structure Prediction Competition
In November–December 2020, DeepMind's AlphaFold 2 system achieved a median Global Distance Test score of approximately 92.4 across all CASP14 targets, far surpassing the next-best group, in a result that computational biologists described as largely solving the 50-year-old protein-folding problem for single-chain proteins.
OpenAI released GPT-3 via private beta API
In May–June 2020, OpenAI published the GPT-3 language model in a paper by Tom B. Brown and colleagues, and began distributing private beta API access. GPT-3's 175 billion parameters made it substantially larger than any publicly described language model at the time, enabling strong few-shot performance across diverse language tasks.
OpenAI Released the Full 1.5-Billion-Parameter GPT-2 Model
In November 2019, OpenAI released the full 1.5-billion-parameter version of GPT-2, completing a staged release the organisation had begun in February 2019 with a smaller variant, citing concerns about potential misuse of a model capable of generating coherent long-form text.
OpenAI Releases GPT-1: Improving Language Understanding by Generative Pre-Training
In June 2018, Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever at OpenAI published 'Improving Language Understanding by Generative Pre-Training', introducing GPT-1, a 117-million-parameter Transformer pretrained on BooksCorpus via unsupervised language modelling and fine-tuned on downstream tasks, outperforming task-specific models on several NLP benchmarks.
Andrew M. Dai and Quoc V. Le Introduced Semi-Supervised Sequence Learning
In November 2015, Andrew M. Dai and Quoc V. Le at Google Brain published 'Semi-Supervised Sequence Learning', showing that pre-training recurrent neural networks with unsupervised objectives, language modelling or sequence autoencoding, before supervised fine-tuning improved text classification accuracy and training stability, anticipating the pre-train-then-fine-tune paradigm later adopted widely in NLP.
Xinlei Chen, Abhinav Shrivastava and Abhinav Gupta at Carnegie Mellon University present NEIL (Never-Ending Image Learner) at ICCV 2013
In December 2013, Xinlei Chen, Abhinav Shrivastava and Abhinav Gupta at Carnegie Mellon University presented NEIL (Never-Ending Image Learner) at ICCV 2013, a continuously running system that autonomously mined semantic relationships between visual concepts from unlabelled web images without human supervision.