AI milestones of the 2020s
55 documented milestones spanning 2020 to 2024. Mostly capability and method entries.
Nobel Prizes in Physics and Chemistry are awarded for machine learning
The Royal Swedish Academy of Sciences awarded the 2024 Nobel Prize in Physics to John Hopfield and Geoffrey Hinton on 8 October for foundational discoveries enabling machine learning with artificial neural networks. The Chemistry prize followed the next day, half to David Baker for computational protein design and half jointly to Demis Hassabis and John Jumper for protein structure prediction.
Government servicesOpenAI releases o1, a model trained to reason before answering
OpenAI released o1-preview and o1-mini on 12 September 2024, models trained by reinforcement learning to produce an extended chain of thought before answering. On the 2024 AIME examination GPT-4o averaged 12 per cent of problems; o1 averaged 74 per cent with a single attempt and 93 per cent when re-ranking a thousand samples.
Reasoning and planningThe EU Artificial Intelligence Act enters into force
Regulation (EU) 2024/1689 entered into force on 1 August 2024, twenty days after publication in the Official Journal. It is the first comprehensive legal framework for artificial intelligence adopted anywhere, sorting systems into risk tiers and attaching obligations that scale with the harm a system could do.
Regulation and policyGPT-4o handles speech, vision and text in one network
OpenAI released GPT-4o on 13 May 2024, trained end to end across text, vision and audio so that every input and output passes through one network. It responds to speech in as little as 232 milliseconds and 320 on average, against roughly 2.8 seconds for the three separate models it replaced.
Architectures and modelsAlphaFold 3 predicts how proteins interact with DNA, RNA and drugs
Google DeepMind and Isomorphic Labs published AlphaFold 3 in Nature on 8 May 2024. Where AlphaFold 2 predicted the shape of a protein, this predicts the joint structure of complexes containing proteins, nucleic acids, small molecules and ions, using a diffusion architecture that produces atom coordinates directly.
BiologyMixtral 8x7B: sparse mixture-of-experts model released under Apache 2.0
Mixtral 8x7B, a sparse mixture-of-experts language model, was published in January 2024. It outperforms or matches Llama 2 70B and GPT-3.5 on most evaluated benchmarks while using only 13B active parameters per token, and both base and instruction-tuned versions were released under the Apache 2.0 licence.
Language understandingGNoME predicts 2.2 million crystal structures, and the claim is disputed
Google DeepMind published GNoME in Nature on 29 November 2023, reporting 2.2 million predicted crystal structures of which 381,000 were judged stable, and describing this as an order of magnitude expansion in the stable materials known. A companion paper from Berkeley Lab reported a robotic laboratory synthesising 41 of the predicted compounds.
Materials scienceGraphCast forecasts ten days of global weather in under a minute
Google DeepMind published GraphCast in Science on 14 November 2023. Trained on about forty years of reanalysis data, it predicts hundreds of atmospheric variables ten days ahead at quarter-degree resolution, and beat the European Centre's operational forecast on 90 per cent of 1,380 verification targets. A forecast that takes hours on a supercomputer takes it under a minute.
Climate and weatherAlphaMissense scores every possible missense variant in the human proteome
Google DeepMind published AlphaMissense in Science on 19 September 2023, an adaptation of AlphaFold that scores how likely a single amino acid change is to cause disease. It classified 89 per cent of all 71 million possible missense variants, against roughly 0.1 per cent confirmed by laboratory or clinical work.
Genomics and genetics3D Gaussian Splatting achieves real-time novel-view synthesis at 1080p
Bernhard Kerbl and colleagues applied 3D Gaussian splatting to novel-view synthesis with end-to-end optimisation and adaptive density control, achieving high-quality results at 1080p resolution and real-time frame rates, without the slow neural rendering that previous approaches required.
Machine perceptionFlashAttention-2 roughly doubles attention speed on A100 GPUs
Tri Dao's FlashAttention-2 improved GPU work partitioning to deliver roughly 2x the speed of FlashAttention, reaching 50–73% of the theoretical maximum FLOPs/s in the configurations benchmarked in the paper, and up to 225 TFLOPs/s per A100 in the specific GPT-style training configurations reported.
Compute and infrastructureRFdiffusion designs proteins that do not exist in nature
Watson, Juergens and colleagues at the Baker Lab published RFdiffusion in Nature on 11 July 2023. By fine-tuning the RoseTTAFold prediction network on denoising tasks, they turned a model that reads protein structures into one that invents them. Hundreds of the designs were then made in the laboratory.
BiologyPangu-Weather is the first learned model to beat operational forecasting
Bi and colleagues at Huawei Cloud published Pangu-Weather in Nature on 5 July 2023. Trained on 39 years of recorded atmospheric data, it was the first machine-learned model reported to beat the operational numerical forecast on accuracy as well as speed, producing a global forecast in about 1.4 seconds on one graphics processor.
Climate and weatherProcess supervision outperforms outcome supervision for LLM reasoning
Researchers showed that giving a model step-by-step feedback on its reasoning, rather than judging only the final answer, raised the solve rate to 78% on a representative subset of the MATH benchmark, and released 800,000 human feedback labels to support further work.
Reasoning and planningDirect Preference Optimisation reduces need for RL in LM alignment
Rafael Rafailov and colleagues introduced Direct Preference Optimization (DPO), a method that aligns language models with human preferences using only a simple classification loss, bypassing the complex reinforcement learning pipeline that existing approaches required.
Learning methodsQLoRA enables finetuning of 65B-parameter models on a single 48GB GPU
Tim Dettmers and colleagues showed that a 65-billion-parameter language model could be finetuned on a single 48GB GPU without measurable quality loss on the benchmarks tested, by combining 4-bit quantisation with low-rank adapter training.
Learning methodsESMFold predicts structure without alignments and folds 617 million proteins
Lin and colleagues at Meta published ESMFold in Science on 16 March 2023. It predicts a protein's structure from its sequence alone, with no search for related sequences, because the evolutionary information such a search supplies is already inside a language model of 15 billion parameters. They used that speed to fold over 617 million metagenomic proteins.
BiologyOpenAI releases GPT-4, the first of its models to accept images
OpenAI released GPT-4 on 14 March 2023, a model accepting both image and text input and producing text. Its technical report gives human-level performance on a range of professional and academic examinations, including a simulated bar exam in roughly the top ten per cent of test takers.
Architectures and modelsLLaMA matches leading models on public data with far fewer parameters
LLaMA, a collection of foundation language models from 7B to 65B parameters trained exclusively on publicly available data, was submitted to arXiv on 27 February 2023. The 13B model outperformed GPT-3 at 175B parameters on most benchmarks, and the weights were released to the research community.
Language understandingControlNet adds structured spatial conditioning to pretrained text-to-image diffusion models
Lvmin Zhang, Anyi Rao and Maneesh Agrawala introduced ControlNet, a neural network architecture that adds spatial conditioning controls such as edges, depth, segmentation and human pose to large pretrained text-to-image diffusion models without degrading their existing capabilities.
Architectures and modelsSelf-Instruct: language models taught to follow instructions using synthetic self-generated data
Yizhong Wang and colleagues introduced Self-Instruct, a method for training language models to follow instructions using synthetic data the models generate themselves, closing much of the gap with InstructGPT-001 on evaluated tasks while using far less human annotation.
Language understandingOpenAI launches ChatGPT as a free research preview
OpenAI released ChatGPT on 30 November 2022 as a free research preview built on GPT-3.5. It reached one million users within five days, and analysts at UBS estimated roughly 100 million monthly users by January 2023, a pace that had taken TikTok nine months and Instagram about two and a half years.
Software engineeringBLOOM: open-access 176B-parameter multilingual language model
The BigScience Workshop released BLOOM, a 176-billion-parameter language model trained across 46 natural and 13 programming languages, made freely available under the Responsible AI License at a scale that had only recently begun to become accessible, and had not previously been available with multilingual training data.
Language understandingFlan-PaLM: instruction finetuning scaled across tasks, model sizes and families
Researchers showed that finetuning large language models on instruction-phrased datasets improves performance across benchmarks, with Flan-PaLM 540B trained on 1.8K tasks scoring 75.2% on five-shot MMLU and outperforming its base model by 9.4% on average.
Language understandingReAct: LLMs learn to interleave reasoning traces and actions
Shunyu Yao and colleagues introduced ReAct, a method letting large language models interleave reasoning traces with real actions such as querying Wikipedia, reducing hallucination and outperforming prior methods on question answering, fact verification and interactive decision-making tasks.
Reasoning and planningStable Diffusion is released with downloadable weights
Stable Diffusion 1.4 was released publicly on 22 August 2022 with its weights freely downloadable, by Stability AI with CompVis at LMU Munich, Runway, EleutherAI and LAION. Because the diffusion runs in a compressed latent space, it generates 512 by 512 images in around 6.9 gigabytes of video memory, within reach of a consumer graphics card.
Architectures and modelsGitHub Copilot becomes generally available as a paid product
GitHub made Copilot generally available on 21 June 2022 at ten dollars a month, free to verified students and to maintainers of popular open source projects. It had run as a technical preview since June 2021 and suggests code inside the editor, drawing on Codex, a model trained on public code.
Software engineeringFlamingo: few-shot visual language model for interleaved images, video and text
Researchers introduced Flamingo, a family of Visual Language Models that could handle interleaved images, video and text, achieving state-of-the-art few-shot performance on many benchmarks without task-specific fine-tuning.
Machine perceptionAnthropic paper applies RLHF to train helpful and harmless assistants
Researchers applied preference modelling and reinforcement learning from human feedback to fine-tune language models as assistants, finding that alignment training improved performance on almost all NLP evaluations tested, without measurable loss of specialised skills in the models studied.
Alignment and safetySayCan: filtering language model suggestions through robot skill affordances
A team of researchers showed that combining a large language model with pretrained low-level robot skills lets a mobile manipulator complete long-horizon tasks from abstract natural language instructions, with the robot's physical capabilities constraining what the model is allowed to suggest.
Manipulation and dexterityInstructGPT: aligning language models with human feedback at scale
Researchers showed that fine-tuning GPT-3 with human feedback produced a 1.3B parameter model whose outputs labellers preferred over those of the 175B GPT-3, pointing toward a practical method for aligning language models more closely with expressed human preferences.
Language understandingChain-of-thought prompting improves step-by-step reasoning in large language models
Jason Wei and colleagues showed that prompting a large language model with worked examples of intermediate reasoning steps dramatically improved its performance on arithmetic, commonsense, and symbolic tasks, reaching state-of-the-art accuracy on a maths benchmark.
Reasoning and planningMultiresolution hash encoding cuts neural graphics training to seconds on a single GPU
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller introduced a multiresolution hash encoding that trains neural graphics primitives in seconds and renders at 1920×1080 in tens of milliseconds, achieving a combined speedup of several orders of magnitude when the encoding and optimised CUDA kernels are used together.
Architectures and modelsS4: structured state spaces advance long-sequence modelling
Albert Gu, Karan Goel, and Christopher Ré introduced S4, a sequence model based on structured state spaces that achieved state-of-the-art results on the Long Range Arena benchmark and solved the Path-X task of length 16,000 steps that all prior work had failed.
Architectures and modelsInstruction tuning lets a 137B model match or beat GPT-3 zero-shot
Jason Wei and colleagues showed that finetuning a 137B language model on over 60 NLP tasks described via natural language instruction templates produced a model, FLAN, that beat zero-shot GPT-3 on 20 of 25 tasks and surpassed few-shot GPT-3 on several benchmarks.
Language understandingRoseTTAFold reproduces protein structure prediction and releases the code
Baek and colleagues at the Institute for Protein Design published RoseTTAFold in Science on 15 July 2021, having rebuilt protein structure prediction from the ideas DeepMind had described at CASP14 eight months earlier without releasing. Its accuracy approached AlphaFold 2 without matching it, and it ran in about ten minutes on a gaming computer.
BiologyUMass Amherst researchers demonstrate protein-nanowire memristors for neuromorphic computing
In June 2021, researchers at the University of Massachusetts Amherst published findings in Nature Communications showing that protein nanowires harvested from the bacterium Geobacter sulfurreducens can function as memristors, enabling brain-inspired computation at ultralow power without a conventional battery.
Compute and infrastructureBrainGate researchers decode imagined handwriting from neural signals to enable high-speed text communication
In May 2021, Francis R. Willett and colleagues at Stanford University and Howard Hughes Medical Institute published results showing that a BrainGate2 intracortical electrode array could decode imagined handwriting movements in a paralysed person at 90 characters per minute with 94.1% raw accuracy, substantially exceeding prior neural-interface typing rates.
Prosthetics and assistive technologyCornell University researchers demonstrate electrically actuated microscale origami robots with onboard CMOS control
On 17 March 2021, researchers at Cornell University published a demonstration of self-folding microscale robots, roughly 100–250 microns in size, driven by platinum-based shape-memory actuators and controlled by onboard complementary metal-oxide-semiconductor (CMOS) circuits, enabling untethered, electrically commanded origami-style locomotion at the micron scale.
Manipulation and dexterityOpenAI releases CLIP and DALL-E on the same day
OpenAI announced CLIP and DALL-E together on 5 January 2021. CLIP was trained on 400 million image and text pairs collected from the internet, and matches the accuracy of a ResNet-50 on ImageNet without using any of that benchmark's 1.28 million labelled examples. DALL-E generates images from a written description, using a version of GPT-3 adapted to produce pictures.
Architectures and modelsReinforcement learning framework proposed to model T-cell adaptive immune response
In 2021, researchers published in Physical Review Research a theoretical framework proposing that T-cell receptor signalling during adaptive immunity can be formally described as a reinforcement learning process, connecting immunological learning to established machine-learning theory.
BiologyGoogle Announces TPU v4 Tensor Processing Unit
Google announced its fourth-generation Tensor Processing Unit (TPU v4) at Google I/O in 2021. The chip, designed specifically for large-scale machine-learning workloads, offered substantially higher performance than its predecessor and was made available to researchers via Google Cloud.
Compute and infrastructureCavatappi Artificial Muscles from Drawing, Twisting, and Coiling Polymer Tubes
In 2021, researchers led by Tzu-Ching Shyu at the University of Illinois at Urbana-Champaign published findings in Science Robotics describing cavatappi artificial muscles, twisted polymer actuators made from polyethylene-based tubes, capable of contracting up to 45% of their length, lifting approximately 300 times their weight per unit volume, and actuating within 23 milliseconds per 1% length change.
Manipulation and dexterityPneumatic-logic soft robot without electronic components, University of California San Diego, 2021
In 2021, Michael T. Tolley and colleagues at the University of California San Diego published a soft walking robot controlled entirely by pneumatic logic circuits embedded in its body, with no electronic components, demonstrating autonomous gait and environmental responsiveness through fluidic computation alone.
Manipulation and dexteritySwinburne University of Technology Researchers Demonstrate Optical Neuromorphic Processor Using Micro-Comb Photonic Chip
In January 2021, a Swinburne University of Technology-led international team published results in Nature demonstrating an optical neuromorphic processor built on a photonic micro-comb chip, achieving a processing speed of 10 TOPS from a single integrated device and performing image classification tasks at high speed.
Compute and infrastructureMilad Abolhasani and colleagues demonstrated Artificial Chemist 2.0, an autonomous flow chemistry platform for quantum dot synthesis
In December 2020, Milad Abolhasani's group at North Carolina State University published Artificial Chemist 2.0, an autonomous flow chemistry system combining machine learning with robotic synthesis to navigate a space of approximately 20 million quantum dot formulations and produce a target material within roughly 30 minutes of initiating a search.
ChemistrySoft Robotic Gripper Modelled on Pole Bean Tendrils Developed at University of Georgia
In December 2020, researchers at the University of Georgia developed a soft robotic gripper modelled on the twining behaviour of pole bean tendrils. The 3-inch device uses a single pneumatic actuator and an embedded fibre-optic sensor to grasp objects as small as 1 millimetre in diameter and characterise surface properties during contact.
Manipulation and dexterityDeepMind's AlphaFold 2 Achieves Highest-Accuracy Results at CASP14 Protein Structure Prediction Competition
In November–December 2020, DeepMind's AlphaFold 2 system achieved a median Global Distance Test score of approximately 92.4 across all CASP14 targets, far surpassing the next-best group, in a result that computational biologists described as largely solving the 50-year-old protein-folding problem for single-chain proteins.
BiologyWaymo One Launches Fully Driverless Rides to the General Public in Phoenix, Arizona
On 8 October 2020, Waymo opened its Waymo One ride-hailing service to the general public in the greater Phoenix, Arizona area, operating without a safety driver in the vehicle, the first time a commercial autonomous vehicle service had done so at public scale.
Autonomous vehiclesCURIAL: An AI System to Detect COVID-19 in Emergency Department Patients Using Routine Blood Tests and Vital Signs
In July 2020, researchers at the University of Oxford, led by Dr Andrew Soltan and Professor David Clifton, announced CURIAL, a machine-learning model trained on routine blood tests and vital signs from 115,000 hospital presentations that could identify COVID-19 patients in emergency departments within one hour and with accuracy exceeding 90%.
DiagnosisOpenAI released GPT-3 via private beta API
In May–June 2020, OpenAI published the GPT-3 language model in a paper by Tom B. Brown and colleagues, and began distributing private beta API access. GPT-3's 175 billion parameters made it substantially larger than any publicly described language model at the time, enabling strong few-shot performance across diverse language tasks.
Architectures and modelsOnce-for-All: Train One Network and Specialize It for Efficient Deployment
Han Cai, Chuang Gan, Tianhao Chen, and Song Han at MIT published Once-for-All at ICLR 2020, presenting a method to train a single neural network once and then derive specialised sub-networks for diverse hardware platforms without retraining, reducing the computational cost of neural architecture search by orders of magnitude.
Architectures and modelsMIT Researchers Use Machine Learning to Identify Halicin, an Antibiotic Effective Against Drug-Resistant Bacteria
On 20 February 2020, James Collins and colleagues at MIT published research in Cell describing a deep-learning model trained to predict antibiotic activity; the model identified halicin, a compound previously investigated for diabetes treatment, as a potent broad-spectrum antibiotic capable of killing several drug-resistant bacterial strains.
Drug discoveryMicrosoft Research released Turing Natural Language Generation (T-NLG), a 17-billion-parameter language model
In February 2020, Microsoft Research announced Turing Natural Language Generation (T-NLG), a 17-billion-parameter autoregressive language model trained using the Megatron-LM framework. At the time of release it was the largest publicly disclosed language model and achieved state-of-the-art results on question-answering and summarisation benchmarks.
Language understandingFacebook AI Research Publishes GrokNet, a Unified Computer Vision Model for Commerce Understanding
In 2020, researchers at Facebook AI published GrokNet, a unified deep learning system for product understanding in commerce settings, capable of recognising object categories, attributes such as colour and material, and brand information from product images at scale across Facebook Shops.
Retail and commerce