Foundational method
An architecture, algorithm or technique that later work was built on.
65 milestones.
Mixtral 8x7B: sparse mixture-of-experts model released under Apache 2.0
Mixtral 8x7B, a sparse mixture-of-experts language model, was published in January 2024. It outperforms or matches Llama 2 70B and GPT-3.5 on most evaluated benchmarks while using only 13B active parameters per token, and both base and instruction-tuned versions were released under the Apache 2.0 licence.
3D Gaussian Splatting achieves real-time novel-view synthesis at 1080p
Bernhard Kerbl and colleagues applied 3D Gaussian splatting to novel-view synthesis with end-to-end optimisation and adaptive density control, achieving high-quality results at 1080p resolution and real-time frame rates, without the slow neural rendering that previous approaches required.
FlashAttention-2 roughly doubles attention speed on A100 GPUs
Tri Dao's FlashAttention-2 improved GPU work partitioning to deliver roughly 2x the speed of FlashAttention, reaching 50–73% of the theoretical maximum FLOPs/s in the configurations benchmarked in the paper, and up to 225 TFLOPs/s per A100 in the specific GPT-style training configurations reported.
Process supervision outperforms outcome supervision for LLM reasoning
Researchers showed that giving a model step-by-step feedback on its reasoning, rather than judging only the final answer, raised the solve rate to 78% on a representative subset of the MATH benchmark, and released 800,000 human feedback labels to support further work.
Direct Preference Optimisation reduces need for RL in LM alignment
Rafael Rafailov and colleagues introduced Direct Preference Optimization (DPO), a method that aligns language models with human preferences using only a simple classification loss, bypassing the complex reinforcement learning pipeline that existing approaches required.
QLoRA enables finetuning of 65B-parameter models on a single 48GB GPU
Tim Dettmers and colleagues showed that a 65-billion-parameter language model could be finetuned on a single 48GB GPU without measurable quality loss on the benchmarks tested, by combining 4-bit quantisation with low-rank adapter training.
ControlNet adds structured spatial conditioning to pretrained text-to-image diffusion models
Lvmin Zhang, Anyi Rao and Maneesh Agrawala introduced ControlNet, a neural network architecture that adds spatial conditioning controls such as edges, depth, segmentation and human pose to large pretrained text-to-image diffusion models without degrading their existing capabilities.
Self-Instruct: language models taught to follow instructions using synthetic self-generated data
Yizhong Wang and colleagues introduced Self-Instruct, a method for training language models to follow instructions using synthetic data the models generate themselves, closing much of the gap with InstructGPT-001 on evaluated tasks while using far less human annotation.
Flan-PaLM: instruction finetuning scaled across tasks, model sizes and families
Researchers showed that finetuning large language models on instruction-phrased datasets improves performance across benchmarks, with Flan-PaLM 540B trained on 1.8K tasks scoring 75.2% on five-shot MMLU and outperforming its base model by 9.4% on average.
ReAct: LLMs learn to interleave reasoning traces and actions
Shunyu Yao and colleagues introduced ReAct, a method letting large language models interleave reasoning traces with real actions such as querying Wikipedia, reducing hallucination and outperforming prior methods on question answering, fact verification and interactive decision-making tasks.
Flamingo: few-shot visual language model for interleaved images, video and text
Researchers introduced Flamingo, a family of Visual Language Models that could handle interleaved images, video and text, achieving state-of-the-art few-shot performance on many benchmarks without task-specific fine-tuning.
Anthropic paper applies RLHF to train helpful and harmless assistants
Researchers applied preference modelling and reinforcement learning from human feedback to fine-tune language models as assistants, finding that alignment training improved performance on almost all NLP evaluations tested, without measurable loss of specialised skills in the models studied.
SayCan: filtering language model suggestions through robot skill affordances
A team of researchers showed that combining a large language model with pretrained low-level robot skills lets a mobile manipulator complete long-horizon tasks from abstract natural language instructions, with the robot's physical capabilities constraining what the model is allowed to suggest.
InstructGPT: aligning language models with human feedback at scale
Researchers showed that fine-tuning GPT-3 with human feedback produced a 1.3B parameter model whose outputs labellers preferred over those of the 175B GPT-3, pointing toward a practical method for aligning language models more closely with expressed human preferences.
Chain-of-thought prompting improves step-by-step reasoning in large language models
Jason Wei and colleagues showed that prompting a large language model with worked examples of intermediate reasoning steps dramatically improved its performance on arithmetic, commonsense, and symbolic tasks, reaching state-of-the-art accuracy on a maths benchmark.
Multiresolution hash encoding cuts neural graphics training to seconds on a single GPU
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller introduced a multiresolution hash encoding that trains neural graphics primitives in seconds and renders at 1920×1080 in tens of milliseconds, achieving a combined speedup of several orders of magnitude when the encoding and optimised CUDA kernels are used together.
S4: structured state spaces advance long-sequence modelling
Albert Gu, Karan Goel, and Christopher Ré introduced S4, a sequence model based on structured state spaces that achieved state-of-the-art results on the Long Range Arena benchmark and solved the Path-X task of length 16,000 steps that all prior work had failed.
Instruction tuning lets a 137B model match or beat GPT-3 zero-shot
Jason Wei and colleagues showed that finetuning a 137B language model on over 60 NLP tasks described via natural language instruction templates produced a model, FLAN, that beat zero-shot GPT-3 on 20 of 25 tasks and surpassed few-shot GPT-3 on several benchmarks.
OpenAI releases CLIP and DALL-E on the same day
OpenAI announced CLIP and DALL-E together on 5 January 2021. CLIP was trained on 400 million image and text pairs collected from the internet, and matches the accuracy of a ResNet-50 on ImageNet without using any of that benchmark's 1.28 million labelled examples. DALL-E generates images from a written description, using a version of GPT-3 adapted to produce pictures.
Pneumatic-logic soft robot without electronic components, University of California San Diego, 2021
In 2021, Michael T. Tolley and colleagues at the University of California San Diego published a soft walking robot controlled entirely by pneumatic logic circuits embedded in its body, with no electronic components, demonstrating autonomous gait and environmental responsiveness through fluidic computation alone.
Soft Robotic Gripper Modelled on Pole Bean Tendrils Developed at University of Georgia
In December 2020, researchers at the University of Georgia developed a soft robotic gripper modelled on the twining behaviour of pole bean tendrils. The 3-inch device uses a single pneumatic actuator and an embedded fibre-optic sensor to grasp objects as small as 1 millimetre in diameter and characterise surface properties during contact.
Once-for-All: Train One Network and Specialize It for Efficient Deployment
Han Cai, Chuang Gan, Tianhao Chen, and Song Han at MIT published Once-for-All at ICLR 2020, presenting a method to train a single neural network once and then derive specialised sub-networks for diverse hardware platforms without retraining, reducing the computational cost of neural architecture search by orders of magnitude.
BERT introduces masked bidirectional pre-training for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova introduced BERT, a language model that pre-trains on both left and right context simultaneously, achieving new best results on eleven natural language processing tasks.
Facebook AI Research Published StarSpace: Embed All The Things!
In September 2017, Ledell Wu and colleagues at Facebook AI Research published StarSpace (arXiv:1709.03856), a general-purpose neural embedding model capable of learning entity representations across tasks including text classification, ranking, and collaborative filtering, without task-specific architecture changes.
Vaswani et al. introduce the Transformer, a sequence model built on self-attention alone
Ashish Vaswani and colleagues proposed the Transformer, a sequence model whose encoding and decoding relied on multi-head self-attention rather than recurrence or convolutions. It set new translation benchmarks while training faster than existing models.
FAISS: Facebook AI Research Library for Efficient Similarity Search
In 2017, Jeff Johnson, Matthijs Douze, and Hervé Jégou at Facebook AI Research published FAISS (Facebook AI Similarity Search), a library enabling efficient nearest-neighbour search across datasets of billions of vectors, with GPU acceleration substantially reducing search time compared to prior methods.
WaveNet: A Generative Model for Raw Audio, by DeepMind
In September 2016, researchers at Google DeepMind published WaveNet, a deep generative model that synthesises raw audio waveforms sample-by-sample using dilated causal convolutions. In evaluations on English and Mandarin speech, WaveNet reduced the gap between human speech and machine synthesis by more than 50 per cent compared with the best previous text-to-speech systems.
Deep residual networks make training at 100+ layers practical with identity shortcuts
Kaiming He and colleagues introduced residual learning, letting networks train at depths of up to 152 layers. Their ResNet won first place on five tracks at the ILSVRC and COCO 2015 competitions, including ImageNet classification, detection, localisation, and COCO detection and segmentation.
Andrew M. Dai and Quoc V. Le Introduced Semi-Supervised Sequence Learning
In November 2015, Andrew M. Dai and Quoc V. Le at Google Brain published 'Semi-Supervised Sequence Learning', showing that pre-training recurrent neural networks with unsupervised objectives, language modelling or sequence autoencoding, before supervised fine-tuning improved text classification accuracy and training stability, anticipating the pre-train-then-fine-tune paradigm later adopted widely in NLP.
DeepMind's DQN learns to play Atari games from raw pixels
DeepMind's Deep Q-Networks algorithm learned to play Atari 2600 games directly from raw pixels, matching or exceeding the score of a human tester on roughly half of the games tested, without any prior knowledge of the rules.
Kingma and Ba present the Adam optimiser
Diederik P. Kingma and Jimmy Ba presented Adam, an optimisation algorithm combining and extending existing adaptive gradient methods, adapting learning rates using bias-corrected estimates of the first and second moments of gradients, making it efficient and broadly practical, particularly where heavy hyperparameter tuning is not feasible.
Neural Turing Machine Introduced by Alex Graves, Greg Wayne, and Ivo Danihelka
In October 2014, Alex Graves, Greg Wayne, and Ivo Danihelka at Google DeepMind published 'Neural Turing Machines', a preprint proposing a neural network architecture augmented with an external memory matrix and differentiable read/write operations, enabling the system to learn algorithms such as sorting and copying from examples alone.
Bahdanau, Cho and Bengio introduce soft attention for neural machine translation
Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio proposed an attention mechanism letting neural translation models search source sentences dynamically, rather than compressing everything into a single fixed-length vector, achieving performance comparable to phrase-based systems on English-to-French translation.
Goodfellow and colleagues propose generative adversarial networks
Ian Goodfellow and seven co-authors proposed training two neural networks against each other: one generating samples, one judging them. The setup, requiring only backpropagation and no Markov chains, could recover the training data distribution under idealised theoretical assumptions.
Dropout introduced as a regularisation method for neural networks
Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov introduced dropout, a method that randomly omits units during training to stop neural networks from overfitting. It set new records on several specific speech and object recognition benchmarks at the time of publication.
First DNA Robot Capable of Bipedal Motion
In April 2004, chemists William Sherman and Nadrian Seeman at New York University reported a bipedal DNA robot whose two 10-nanometre legs walked along a single-stranded DNA track through the sequential addition of 'set' and 'unset' strands, demonstrating programmable nanoscale locomotion.
TEXTAL System for AI-Assisted Automated Protein Model Building
In 2003, Thomas R. Ioerger and James C. Sacchettini at Texas A&M University described TEXTAL, a pattern-recognition system that automatically traced atomic models through crystallographic electron density maps, substantially reducing the manual labour required in protein structure determination.
Machine Learning Detection of Least Significant Bit Steganography
Around 2003, researchers published work applying machine learning classifiers to the detection of least significant bit steganography in digital images, training models to distinguish unaltered carrier images from those embedding hidden data in both uncompressed and compressed formats.
Bag of Words Applied to Computer Vision (Visual Vocabulary / Bag of Visual Words)
Josef Sivic and Andrew Zisserman at the University of Oxford applied the Bag of Words text-retrieval model to visual features in their 2003 ICCV paper 'Video Google', representing image regions as a vocabulary of visual words to enable efficient object retrieval from video.
A Neural Probabilistic Language Model by Yoshua Bengio and Colleagues
In 2003, Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin at the Université de Montréal published 'A Neural Probabilistic Language Model' in JMLR, demonstrating that a feed-forward neural network trained on word sequences could learn distributed word representations and outperform n-gram models on perplexity benchmarks.
Structured Light for Robust Correspondence in Active Stereo Vision
In 2002, Li Zhang, Brian Curless, and Steven M. Seitz at the University of Washington presented a method using structured light patterns projected onto scenes to establish robust stereo correspondences, enabling reliable 3D reconstruction under conditions where passive stereo fails.
Torch Machine Learning Library
In 2002, Ronan Collobert, Samy Bengio, and Johnny Mariéthoz at IDIAP Research Institute published a paper introducing Torch, a modular C++ and Lua-scriptable machine learning library that unified a range of algorithms, including support vector machines and neural networks, under a common object-oriented framework.
Berners-Lee Proposes the Semantic Web
In 1999–2001, Tim Berners-Lee, director of the World Wide Web Consortium, outlined the Semantic Web vision, a machine-readable extension of the existing web in which data would carry explicit meaning, culminating in a widely cited May 2001 Scientific American article co-authored with James Hendler and Ora Lassila.
Long Short-Term Memory Introduced by Sepp Hochreiter and Jürgen Schmidhuber
In 1997, Sepp Hochreiter at Technische Universität München and Jürgen Schmidhuber at IDSIA published 'Long Short-Term Memory' in Neural Computation, introducing a recurrent neural network architecture with gated memory cells that could learn dependencies across long sequences without suffering from the vanishing gradient problem.
BackRub Web Crawler Introduces Link-Based Page Ranking at Stanford
In March 1996, Larry Page, a PhD student at Stanford University, launched the BackRub web crawler to analyse the backlink structure of the web, laying the algorithmic foundation for what would become Google's PageRank system.
ALICE Chatbot Created by Richard S. Wallace
In 1995, Richard S. Wallace, an independent AI researcher, created ALICE (Artificial Linguistic Internet Computer Entity), a natural-language chatbot that used a pattern-matching markup language called AIML to generate contextually plausible conversational responses, later influencing a generation of open-source chatbot development.
Elephants Don't Play Chess by Rodney Brooks
In 1990, Rodney Brooks of MIT's Artificial Intelligence Laboratory published 'Elephants Don't Play Chess' in Robotics and Autonomous Systems, arguing that classical symbolic AI was fundamentally misconceived and that intelligence emerges from direct physical interaction with the environment rather than from abstract symbol manipulation.
Rodney Brooks Publishes 'Elephants Don't Play Chess', Articulating Nouvelle AI
In 1990, Rodney Brooks of MIT published 'Elephants Don't Play Chess' in Robotics and Autonomous Systems, arguing that intelligent behaviour could emerge from direct sensorimotor coupling with the environment without internal symbolic representations, formalising the nouvelle AI research programme.
Jabberwacky Chatbot Developed by Rollo Carpenter
In 1988, British programmer Rollo Carpenter began developing Jabberwacky, a chatbot that simulated conversation by storing and retrieving lines from prior user exchanges rather than using fixed scripted responses, with the aim of exploring machine-based natural language interaction.
IBM TJ Watson Research Center Publishes Statistical Approach to Machine Translation
In August 1988, researchers at IBM Thomas J. Watson Research Center, including Peter F. Brown, John Cocke, Stephen A. Della Pietra, Vincent J. Della Pietra, Fredrick Jelinek, Robert L. Mercer, and Paul S. Roossin, presented a statistical framework for machine translation at COLING 1988, replacing rule-based linguistics with probabilistic models trained on bilingual text corpora.
Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference Published by Judea Pearl
In 1988, Judea Pearl of UCLA published Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference with Morgan Kaufmann, providing a systematic framework for representing and computing with uncertain knowledge using Bayesian networks and belief propagation algorithms.
W. Daniel Hillis Proposes the Connection Machine Architecture
In 1985, W. Daniel Hillis of MIT and Thinking Machines Corporation completed his doctoral dissertation introducing the Connection Machine, a massively parallel architecture connecting 65,536 single-bit processors to accelerate symbolic and artificial-intelligence computation, realised as the CM-1 system.
SOAR Cognitive Architecture: Doctoral Dissertations by John E. Laird and Paul S. Rosenbloom, Supervised by Allen Newell
In 1983, John E. Laird and Paul S. Rosenbloom completed doctoral dissertations at Carnegie Mellon University under Allen Newell, introducing SOAR, a cognitive architecture designed to support a broad range of intelligent tasks through a unified problem-space model and a chunking-based learning mechanism.
Primal Sketch Theory of Early Visual Representation Described by David Marr
David Marr, working at MIT's Artificial Intelligence Laboratory, formalised the primal sketch as the first stage of his three-level theory of visual processing, published posthumously in 'Vision' (1982). The model proposed that the visual system constructs a symbolic, viewer-centred description of intensity changes and local geometry before any object recognition takes place.
Blackboard Model Description by Lee Erman, Richard Hayes-Roth, Victor Lesser and D. Raj Reddy
In May 1980, Lee Erman, Richard Hayes-Roth, Victor Lesser and D. Raj Reddy published 'The Hearsay-II Speech-Understanding System: Integrating Knowledge to Resolve Uncertainty' in Artificial Intelligence, vol. 14, providing the canonical description of the blackboard model as a structured framework for cooperative problem-solving among independent knowledge sources.
MOLGEN: AI Planning and Constraint Satisfaction for Molecular Biology Experiment Design
In 1978, Mark Stefik at Stanford University's Heuristic Programming Project developed MOLGEN, an expert system that applied AI planning and constraint-satisfaction techniques to the design of molecular biology experiments, demonstrating that structured reasoning could automate complex scientific problem-solving in genetics and cloning.
Meta-Level Knowledge in Expert Systems: Davis and Lenat's Formalisation at Stanford
In 1977, Randall Davis and Douglas Lenat at Stanford University published research formalising meta-level knowledge, a system's explicit representations of its own knowledge and reasoning strategies, and demonstrated its application in the MYCIN and AM expert systems to improve inference control and self-directed learning.
ABSTRIPS Developed by Earl Sacerdoti at SRI International
In 1974, Earl Sacerdoti at SRI International published ABSTRIPS, an extension of the STRIPS planning system that organised problem-solving into a hierarchy of abstraction spaces, allowing a planner to resolve high-level constraints before committing to fine-grained detail.
Human Associative Memory (HAM) Model Published by John R. Anderson and Gordon H. Bower
In 1973, John R. Anderson and Gordon H. Bower, both at Stanford University, published 'Human Associative Memory', introducing the HAM model, a propositional network architecture representing semantic memory as binary trees, providing a computationally explicit theory of human memory that influenced subsequent cognitive architectures.
Prolog Logic Programming Language Created by Alain Colmerauer and Philippe Roussel
In 1972, Alain Colmerauer and Philippe Roussel at the University of Aix-Marseille created Prolog (Programmation en Logique), a declarative programming language grounded in first-order predicate logic, enabling computers to reason over symbolic knowledge without requiring procedural step-by-step instructions.
Augmented Transition Networks Introduced by William A. Woods
In 1970, William A. Woods of Bolt Beranek and Newman published 'Transition Network Grammars for Natural Language Analysis' in Communications of the ACM, introducing Augmented Transition Networks (ATNs) as a formalism for parsing natural language by extending finite-state transition networks with recursion and registers, enabling more expressive grammatical coverage.
Backpropagation Described by Arthur E. Bryson Jr. and Yu-Chi Ho
In 1969, Arthur E. Bryson Jr. and Yu-Chi Ho of Harvard University described a gradient-based optimisation procedure for multi-stage dynamic systems in their textbook Applied Optimal Control, presenting what is now recognised as an early statement of the backpropagation principle in a supervised-learning context.
A* Search Algorithm Published by Hart, Nilsson, and Raphael at Stanford Research Institute
In 1968, Peter E. Hart, Nils J. Nilsson, and Bertram Raphael at the Stanford Research Institute published 'A Formal Basis for the Heuristic Determination of Minimum Cost Paths', introducing the A* search algorithm, which finds shortest paths in graphs efficiently by combining actual path cost with a heuristic estimate of remaining cost.
General Problem Solver introduced by Newell, Shaw and Simon
In 1959, Allen Newell and Herbert A. Simon at the RAND Corporation and Carnegie Institute of Technology, with J. C. Shaw, presented the General Problem Solver, a computer program that separated problem-solving strategy from domain knowledge using means–ends analysis.
Programming a Computer for Playing Chess, Claude Shannon
In March 1950, Claude Shannon, then at Bell Telephone Laboratories, published 'Programming a Computer for Playing Chess' in Philosophical Magazine, outlining two strategic approaches, exhaustive search (Type A) and selective heuristic search (Type B), that framed computer chess research for decades.